Staff were answering routine operational questions — refund rules, contract terms, escalation paths, product exceptions — by searching across SharePoint, shared drives, email and personal notes. Answers varied by who was asked and when. New employees took months to become productive. Some staff had started pasting internal content into public AI tools, creating a data-handling risk nobody could audit.
The decision
Leadership had to decide whether to sanction AI at all, and if so, under what controls. The goal was not "an AI chatbot". It was faster, more consistent answers with a clear source of truth and an audit trail — while reducing the unsanctioned use that was already happening. That decision set the boundaries for everything that followed: which questions the assistant would answer, which documents it could read, and what it must refuse.
What the assistant had to work from
- Content estate: policies, procedures, contracts, product guides and internal FAQs spread across several systems, with duplicate and superseded versions.
- Access structure: role-based permissions that differed by team, region and document owner.
- Question patterns: a mix of simple lookups, multi-document questions and cases where two policies genuinely conflicted.
- Risk boundaries: legal advice, pricing exceptions and employee-relations questions that required a human owner rather than an answer.
Why the model was not the hard part
- Ownership was unclear. Many documents had no named owner, no review date and no obvious authority. Retrieval cannot fix content that should not be in the corpus.
- Permissions are part of the answer. An assistant that retrieves a document the user is not allowed to see is a security incident, not a relevance problem.
- Stale content outranks current content. Without version metadata, the most similar passage on a topic is often the outdated one.
- Confidence is invisible to users. A fluent paragraph looks the same whether it is grounded in three approved sources or invented. The interface has to make evidence visible.
How the pilot was built
- Scope the use case and the risk. We listed the questions staff actually asked, who asked them, and what harm would follow from a wrong answer. High-risk topics were explicitly out of scope for the first release.
- Audit content and permissions. We identified document owners, removed superseded versions, mapped access controls and defined which sources were approved for retrieval. This work took longer than the model configuration, and it determined whether the pilot was trustworthy.
- Build retrieval that respects structure. Policies, procedures and tables were chunked with headings and metadata intact, so a retrieved passage carried its section, version and effective date.
- Generate with evidence. The assistant used hybrid retrieval, reranking and citation-first responses. When evidence was weak or conflicting, it was designed to say so rather than guess.
- Evaluate before launch. We built a question set with known correct answers and scored retrieval recall, answer groundedness, citation accuracy and appropriate refusal. Latency and cost per query were tracked alongside quality.
- Pilot with human review. A small group used the assistant in daily work. Feedback, thumbs-down reasons and escalated questions fed a weekly improvement cycle.
What this changes
- The team gained one auditable place to ask. Routine questions moved out of inboxes and personal notes into a system where the source could be seen and challenged.
- Shadow AI use had a sanctioned alternative. Staff were no longer choosing between a slow search and an unapproved public tool.
- Answers became easier to trust and easier to dispute. Citations turned a confident paragraph into a claim with a page reference attached.
- The organisation gained a way to measure AI quality. The evaluation set can be rerun whenever documents or models change, which is what separates an assistant that improves from one that quietly degrades.
What we delivered
- A permission-aware pilot assistant with citations and a clear refusal path.
- An evaluation harness covering retrieval, groundedness, citation accuracy and refusal.
- A content governance playbook covering ownership, review cadence and retirement of stale material.
- A rollout plan with training, escalation routes and usage monitoring.
- Guardrails for sensitive data, including redaction and audit logging.
Where this applies
This fits operations, HR, legal, finance and support teams whose knowledge lives in documents rather than databases. The minimum inputs are a document corpus with identifiable owners, an identity and permission model, and a set of real questions with known correct answers. The first deliverable is usually the content and permission audit plus the evaluation set — because an assistant is only as trustworthy as the corpus and the test you can rerun against it.
This is an illustrative case study built on an anonymized dataset; it does not report results from a specific client engagement. The approach is tool-agnostic; the right stack depends on data residency, identity management, security requirements and total cost.
