What is RAG (Retrieval-Augmented Generation)? — one analogy

1The analogy

Ask a bank teller "if I close this account, what happens to the interest?" and they don't answer from memory of every rule on file. They pull up the product's terms and the customer's own transaction history on screen first. Only then do they walk through what's on screen.

The same order can be built for an AI asked about company policy or a personal document. Instead of letting it answer the instant a question comes in, the relevant passage gets pulled from a pile of material first and placed right next to the question. The model reads whatever came along and writes the answer. Putting a fetch step in front like this is RAG.

A teller can change and the terms on screen stay the same. When the rules get revised, the screen's document gets swapped instead of retraining the teller. The advantage of this approach is that the answer follows once the material side gets updated.

2In detail

A step gets inserted between question and answer

Normally, an AI answers a question using only what got baked in during training. Add RAG and a lookup step gets inserted in between. A question comes in, a pile of material gets searched first for passages that look relevant, and those passages get handed to the model alongside the question.

To the model, the question just got a bit longer. It's as if "answer using the following material" and a few paragraphs of terms got tacked on in front of "if I close this account, what happens to the interest." So there's no need to retrain the model at all.

Since the model knows which passage it looked at while writing, a source can be attached at the end of the answer too. Being able to open the original and check it matters a great deal for real work.

Material gets split fine and searched

Paste an entire document in whole and it's like stacking a pile of paperwork in front of the counter. So material gets pre-split into paragraph-sized chunks. Only a handful of those chunks get pulled in for a given question.

Two approaches usually get mixed for picking chunks. One finds chunks where the same words show up literally; the other finds chunks close in meaning. Search by meaning and a question about "closing an account" also pulls in a passage that says "early withdrawal." Using both together catches more of what a literal match alone would miss.

Where a chunk gets cut matters a lot for answer quality too. Slice straight through the middle of a table and only bare numbers survive; cut too large and a pile of unrelated sentences tags along.

The model doesn't have to change

There's another way to put the same content into a model: additional training. That's the equivalent of retraining the teller. It takes time and money, and it has to happen all over again whenever a rule changes.

RAG swaps the screen instead of retraining anyone. A notice posted today shows up in the answer right away, and a mistaken document just gets deleted. Who's allowed to see which document can be managed on the material side too, so different departments can be shown different scopes.

That's why RAG suits knowledge that changes often, and additional training suits something that needs to be baked into behavior — tone, output format. Plenty of setups use both together.

Where it commonly goes wrong

The most common problem is pulling in the wrong chunk. If another customer's terms happen to be up on screen, however careful the teller is, the guidance comes out wrong. This happens when material is disorganized or several versions of a similar document are mixed together.

Pulling in too much is a problem too. There's a cap on how much fits in view at once, and mixing in unrelated passages buries the sentence that actually matters. A well-chosen three or four chunks usually beats a carelessly grabbed twenty.

Sometimes the fetched material simply doesn't contain the answer at all. Without being told to say "it's not in the material" for cases like this, the model fills the gap with whatever it already knows. It can look like an answer written from the source when it really wasn't.

It pairs with grounding

RAG is the method for fetching material, and grounding is the separate act of tying each sentence of the answer to that material. At the counter, RAG is putting the terms up on screen; grounding is making sure the guidance never strays from what's on that screen.

Even with material sitting right there, tacking on a sentence from a guess means grounding has slipped. Fetching material well and making sure the model only speaks within what got fetched are two separate things that both need tending.

3More precisely

RAG is a structure that joins a retriever to a generator. Documents get split into chunks and converted into number bundles that carry meaning, stored ahead of time; a question gets converted the same way to pull out the nearest chunks. Slot the pulled chunks into the prompt and the rest runs like ordinary generation. The model's weights never change at all — only the input changes.

The analogy breaks down somewhere. A teller knows what they don't know and pulls up the screen because of it; in this setup, the lookup mostly fires automatically for every single question. A chunk tags along even for a question that never needed material, and when a question genuinely needs material and the search misses, the model still writes an answer using only whatever chunk it happened to be handed. It's a teller who never questions what's on screen — so search quality becomes the answer's quality directly. That's why real work checks the retrieval step before touching the answer at all: pull twenty sample questions, check whether the chunk with the right answer actually gets fetched, and at what rank. If this step is shaky, no amount of polishing the instructions afterward makes the answer any better.

4Try it yourself

5Common misconceptions

It's easy to think adding RAG makes hallucination disappear, but actually fetching the wrong material still produces a confident wrong answer with a source attached to it.
It's easy to think RAG teaches the model new knowledge, but actually the model stays exactly the same — it's just given material to work from at answer time.
It's easy to think more documents is always better, but actually the more disorganized material piles up, the more likely the wrong chunk gets pulled in.

7One-line summary

In shortRAG is like a teller pulling up the terms on screen instead of answering from memory — fetching relevant material before answering and writing with it close at hand.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02