RAG explained: turning your documents into an AI assistant that gets it right.

Retrieval-augmented generation, or RAG, gets pitched as the answer to "we want an AI that knows our company documents." That is roughly right, but the gap between a genuinely useful RAG system and a chatbot that occasionally hallucinates your own policy back at you is entirely in the implementation.

What RAG actually is, without the jargon

A language model on its own only knows what it was trained on — it has never seen your internal wiki, your product specs or last month's pricing update. RAG bridges that gap in two steps: first it searches your actual documents for the passages relevant to the question, then it hands those passages to the model along with the question, so the answer is generated from your real content instead of the model's general training data.

That is the core difference from just pasting documents into a chat window: RAG searches and retrieves the right passage at answer time, from a source that can be updated independently of the model itself.

How it works, at a practical level

Documents get broken into chunks and converted into embeddings — numerical representations that let a system find passages by meaning, not just keyword match. A question comes in, the system retrieves the most relevant chunks from that index, and the model generates an answer grounded in exactly those chunks, typically with a citation back to the source document.

The quality of the whole system rides on the retrieval step. A model asked to answer from irrelevant or poorly chunked context will confidently produce a wrong answer — this is where most weak RAG implementations actually fail, not in the language model itself.

There is a second, less obvious failure that explains a lot of disappointing results: position. The "Lost in the Middle" study found that models answer most reliably when the relevant passage sits near the beginning or the end of the supplied context, and measurably worse when it is buried in the middle — even in models explicitly built for long contexts. Retrieving the right document is not enough if you then bury it inside forty pages of "just in case" context. Retrieving less, and ranking it properly, routinely beats retrieving more.

Where it is genuinely useful for a business

Internal knowledge bases are the clearest fit: support staff querying policy documents, engineers searching technical documentation, sales teams pulling accurate product specs mid-call. Customer-facing support that needs to answer from your actual documentation — not generic knowledge — is another strong use case, provided the answers stay grounded and cite their source.

It is a poor fit for anything requiring reasoning or calculation beyond what is written down, and a poor fit for data that changes faster than the index can realistically be refreshed.

What separates a good RAG system from a bad one

Document chunking that respects actual structure (not arbitrary character counts), a retrieval step that is evaluated and tuned rather than assumed to work, visible source citations so a human can verify the answer, and a process for keeping the index current as documents change. Access control matters too — a RAG system that surfaces confidential documents to the wrong audience is a real risk, not a hypothetical one.

The single highest-leverage technique published so far is contextual retrieval: prefixing each chunk with a short description of where it sits in the parent document before embedding it, which Anthropic measured as cutting failed retrievals by 49%, or 67% when combined with a reranking step. It costs a one-off preprocessing pass over your corpus. That is the kind of unglamorous detail that separates a system people trust from one they quietly stop using — and it is what we build into a Custom Text AI Model rather than leaving to defaults.

Sources

The mechanism and the measured figures above come from these primary sources, checked July 2026.

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401) ↗

  The 2020 paper that named RAG. Introduces combining a pre-trained generator with a dense vector index as non-parametric memory, and finds RAG models produce more specific, diverse and factual language than a parametric-only baseline.

• Liu et al. — Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172) ↗

  Finds accuracy is highest when relevant information sits at the start or end of the context and degrades significantly in the middle, including in models built for long contexts — the reason retrieval ranking matters more than context length.

• Anthropic — Introducing Contextual Retrieval ↗

  Measured results for prefixing chunks with document context before embedding: a 49% reduction in failed retrievals, rising to 67% when combined with reranking.