Retrieval-augmented generation (RAG)
A model can't answer from documents it was never shown. RAG is the pattern for finding the small handful of relevant documents at request time and handing only those to the model — instead of trying to cram an entire knowledge base into the prompt.
3 min read
The problem RAG solves
A model only knows two things: what it learned during training, and whatever you put in its context window for this specific request. A support bot for a product with 400 pages of documentation can't have all 400 pages memorized correctly, and can't have all 400 pages pasted into every system prompt either — that's slow, expensive, and past a certain size the model starts missing things buried in the middle regardless of the context window's advertised limit. RAG is the answer to "how do I let the model draw on a large body of information without shipping all of it on every request."
The retrieve-then-generate pattern
The "retrieval" step is a similarity search over embeddings — numeric vectors positioned so that semantically related text ends up near each other in vector space. "Reset your password" and "I forgot my login" land close together even though they share almost no words, which is the whole point: retrieval isn't keyword matching, it's meaning matching.
RAG vs. just stuffing more into the context window
Larger context windows make it tempting to skip retrieval and paste everything in. Two real problems with that: cost (you're paying for every token on every single turn, not once), and accuracy — models are measurably worse at using information buried in the middle of a very long context than information near the start or end, an effect often called "lost in the middle." Retrieval keeps the prompt small and dense with only what's actually relevant, which tends to produce better answers, not just cheaper ones. As a rule of thumb: RAG earns its complexity once the knowledge base is bigger than comfortably fits in a prompt, or changes often enough that baking it into a static system prompt would mean editing that prompt constantly.
Chunking is where most RAG quality problems actually live
Splitting documents into chunks sounds like a footnote but is usually the single biggest lever on answer quality. Chunks too large dilute the embedding (a 2,000-word chunk's vector represents an average of everything in it, drowning out the one relevant sentence). Chunks too small lose surrounding context a human would need to actually understand the fragment. A common, effective refinement is prepending a short, model-generated summary of the chunk's surrounding context to each chunk before embedding it — giving the retrieval step more to match against without bloating what gets sent to the generation step.
RAG is a context tool, not a magic fix for correctness
Structurally, a retrieval step is exactly the context-tool pattern from earlier in this domain: read-only, safe to auto-run, feeds real data back into the model's context before it answers. RAG doesn't make hallucination impossible — a bad retrieval (wrong chunks, or no chunks) still leaves the model guessing, just now with wrong material to guess from instead of none. It narrows the problem; it doesn't eliminate it.
Further reading
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. A product has 400 pages of documentation. Why is pasting all 400 pages into the system prompt on every request usually a worse approach than RAG?
2. In the retrieve-then-generate pattern, what does the retrieval step actually search over?
3. Why is chunking often described as the biggest lever on RAG answer quality?
4. Does adding RAG to an agent guarantee it can no longer hallucinate?