Retrieval-Augmented Generation in Plain Terms

A model's parameters are fixed once training ends, so anything private, recent or specific to your situation has to arrive some other way. Retrieval is that other way: fetch material when the request comes in, and put it into the input. The idea is simple. The failures are not, because they happen at several different points and look identical from the outside.

The stages

Between a question and a grounded answer there are usually six stages.

Source material. Documents, records, pages, tickets. Whatever holds the answer.

Preparation and indexing. The material is cleaned, often split into smaller passages, and stored so it can be searched. If a document never made it in, nothing downstream can find it.

Retrieval. A search returns candidates for this question. Usually more than you will use.

Ranking and filtering. The candidates are ordered, and most are dropped.

Context assembly. What survives is placed into the input alongside the instruction, the conversation so far, and anything else the application adds.

Generation. The model produces an answer from what it was handed.

Not every system has all six as separate components.

Small ones collapse several into one search call. But the stages are worth separating in your head, because when the answer is wrong, exactly one of them usually lost the evidence.

Retrieval is not the same as context assembly

This is the distinction the term "RAG" tends to hide.

Retrieval finds candidate material. Context assembly decides what the model actually receives.

A search can return the correct passage at rank four, and the assembly step can keep only the top three. Retrieval succeeded. The model never saw it. Nothing in the logs of the search engine will look wrong, because nothing about the search was wrong.

Keeping these apart changes the first question you ask when an answer is bad. Not "why did the model get this wrong", but "did the evidence reach it at all".

What "search" means here

RAG is often described as though it means a vector database. It does not. Vector search is one retrieval mechanism among several, and the choice between them is a real design decision.

Lexical retrieval matches the words themselves. Ask for an error code, a part number or a surname, and matching the literal string is exactly what you want. This is mature technology and it does not stop being useful because embeddings exist.

Vector retrieval compares embeddings, which are numeric representations positioned so that related text lands near related text. This is what lets a query about "small dog" reach a passage that only says "puppy".

Hybrid retrieval runs both and combines the results, which is common precisely because the two fail in different places.

Reranking is a second pass over the candidates the first pass returned, usually with a slower and more accurate model, to reorder them before anything is kept.

Lexical search misses a paraphrase. Vector search misses an exact identifier, and will happily return something topically adjacent instead of admitting it found nothing. Neither weakness is a bug; they are properties of what each method compares.

Similar is not correct

Vector retrieval ranks by proximity in an embedding space. Proximity reflects patterns in whatever the embedding model was trained on, which is not the same as relevance to your question and is certainly not the same as truth.

Two passages can be close in an embedding space and still answer different questions. A policy that was replaced last month sits near the policy that replaced it, because they are about the same subject. That is the retrieval working as designed, and it is also how a system serves a confidently outdated answer.

Treat the embedding model as a replaceable component rather than a fixed source of truth. Different models arrange the space differently, so the right one is the one whose near neighbours match your material, measured on your own queries rather than on a public leaderboard.

The honest boundary

Retrieval can provide evidence to the model. It does not guarantee that the evidence is relevant, sufficient, current or correct, nor that the model will use it correctly.

You will read that retrieval fixes hallucination. It does not. It changes where the problem lives. A model given a wrong document will now be wrong with a citation attached, which is harder to catch, not easier.

Where it actually fails

When a grounded answer comes back wrong, walk the stages in order:

• the material was never indexed
• it was indexed but the search did not return it
• it was returned but ranked below the cut
• it survived ranking but was dropped during assembly, for length or another reason
• it reached the model and the model did not use it

Only the last of those is a model problem. The rest are retrieval and assembly problems, and they are fixable with ordinary engineering. This page owns one question: was the needed evidence indexed, retrieved, ranked and selected? Whether the final answer was any good is a separate question, and it is answered by evaluation rather than here.

Keeping the material worth retrieving

Retrieval quality has an upstream dependency that is easy to neglect, because it looks like documentation work rather than engineering.

Content tends to grow faster than structure. As a store fills with near-duplicates and drifting names, searches start returning something related instead of the right thing, and the outputs stay fluent while getting quietly worse. A small store with clear ownership and scope usually beats a large one without.

Two habits carry most of the weight. Give each unit an owner and a scope, so somebody can answer whether a page is still true. And check freshness deliberately, because a store that was correct when it was built will not stay correct on its own.

One stage sits even further upstream. If the text was corrupted on the way in, no amount of retrieval quality recovers it. That is the subject of why OCR quietly breaks document AI.

What this changes

Before adding a bigger model or a cleverer prompt, find out whether the evidence reached the input at all. Log what was retrieved and what was actually assembled, because those are different lists and the gap between them is where a surprising number of failures live.

For how the assembled material competes for limited space, see context windows.