October 11, 2026AI & Automation
Retrieval-augmented generation — RAG — is the most common way to build an AI assistant over an organisation's own knowledge. Instead of asking a language model to remember your policies, you retrieve the relevant passages at question time and ask the model to answer from those passages, with citations.
It is a good pattern. It is also frequently oversold. The model is rarely the bottleneck; content quality, access control and evaluation usually are.
When RAG is the right tool
RAG fits when the answer lives in documents or records that change, when the user needs a source, and when the cost of a wrong answer is reduced by showing evidence. Typical examples: policy and procedure assistants, contract and procurement Q&A, support knowledge bases and internal onboarding guides.
It is a weaker fit when the task requires calculation over structured data, when the answer must be perfectly consistent every time, or when the underlying content is contradictory and unowned. In those cases, fix the data or the process first. An assistant that confidently summarises two conflicting policies is not an improvement.
RAG is not the only option
| Approach | Best for | Main limitation |
|---|---|---|
| Keyword or semantic search | Finding a document or passage | User still has to read and interpret it |
| RAG | Answering questions from changing content with citations | Depends on retrieval quality and clean sources |
| Fine-tuning | Style, format or domain-specific behaviour | Poor way to inject frequently changing facts |
| Text-to-SQL or tools | Questions over structured data | Requires governed schemas and guardrails |
The architecture, in plain terms
- Ingestion. Pull documents from approved sources on a schedule. Record owner, version, effective date and access scope.
- Parsing and chunking. Preserve headings, tables and section boundaries. A chunk that starts mid-sentence loses the context that makes retrieval useful.
- Indexing. Store embeddings and metadata. Metadata — department, effective date, classification — is what allows filtering and permission-aware retrieval.
- Retrieval. Combine semantic and keyword search, then rerank the best candidates. Pure vector search often misses exact identifiers, product codes and names.
- Generation. Pass the retrieved passages to the model with instructions to answer only from evidence, cite each claim and refuse when evidence is insufficient.
- Presentation. Show citations, effective dates and a feedback path. Users need to verify, not just trust.
Where projects fail
Content is stale or contradictory
Most organisations have three versions of the same policy and no clear owner. RAG will retrieve whichever is most similar, not whichever is current. The project needs a content audit and a governance process, not just an index.
Permissions are ignored
If the retriever does not respect the user's access rights, the assistant can become a data-leak channel. Identity-aware retrieval is a requirement, not a later enhancement.
Chunking destroys meaning
Splitting by fixed character count breaks tables, exceptions and cross-references. Structure-aware chunking costs more upfront and saves months of bad answers.
Nobody evaluates it
A demo with ten questions proves nothing. You need a test set with known answers, including questions the assistant should refuse. Then you can measure whether a change helped or hurt.
How to evaluate a RAG system
- Retrieval recall: did the correct passage appear in the top results?
- Groundedness: is every claim supported by a cited passage?
- Citation accuracy: do the links point to the passage that actually supports the answer?
- Refusal quality: does it decline when evidence is missing or conflicting?
- Task success: can a user complete the real task faster and more accurately?
- Cost and latency: can the system operate at the volume and budget the business needs?
Build versus buy
Managed platforms get you a pilot quickly. Custom builds give you control over chunking, permissions and evaluation. The right choice depends on how sensitive the content is, how much integration is required and whether the assistant is a differentiator or a utility. In either case, insist on exportable evaluation results and a clear data-processing agreement.
A practical starting checklist
- Pick one high-volume, low-risk use case with a named business owner.
- Inventory the sources; retire what is outdated before indexing it.
- Define the questions the assistant must answer and the ones it must refuse.
- Build the evaluation set before the interface.
- Launch to a small group with human review and a feedback loop.
- Monitor quality, cost and usage weekly; expand only when the metrics hold.
RAG is a useful architecture, but it is not a shortcut around information governance. The organisations that get value are the ones that treat content, permissions and evaluation as the product — and the language model as one component among several.
