When should I use RAG instead of fine-tuning?

When the answer depends on facts that change, or that you need to cite. Retrieval keeps documents outside the model, so updating a fact is an edit rather than a training run, and the answer can point at its source.

Why does RAG still get answers wrong?

Usually retrieval, not generation. If the right passage was never fetched, the model answers from its own patterns instead, just as fluently. Most RAG debugging is checking what was actually retrieved before blaming the model.

Do I still need RAG with a very large context window?

Often yes. You can paste more now, but you pay for every token, latency grows, and accuracy sags for facts buried in the middle. Retrieval is also how you get citations and per-user permissions, which context size does not give you.

What is a vector database, and do I need one?

It is a store that indexes embeddings so you can find the nearest ones fast across millions of chunks. Below a few thousand chunks you can get away with a database extension or even plain keyword search, and many teams start there before adding one.

How big should the chunks be?

A few hundred tokens is the usual starting point, often around 200 to 500, with a little overlap and splits placed on real boundaries like headings. Too large and the chunk matches everything weakly, too small and it loses the context that makes it understandable.

What does it mean for an answer to be grounded?

Every factual claim in the answer can be traced back to a specific retrieved passage rather than to the model's memory. Teams score this as groundedness or faithfulness, and a low score means the model is filling gaps the retrieved text did not cover.

Is giving an AI web search the same thing as RAG?

It is the same shape with a different corpus. The search engine is the retriever, the web is the index, and the fetched pages are pasted into the context before the model writes. That is why AI search products can cite links and answer about this morning's news.

Does RAG keep my documents private?

Your documents stay in your own store, but the passages that get retrieved are sent to the model inside the prompt, so they do leave your system at answer time. Check your provider's data terms, and filter retrieval by permission so nobody retrieves what they cannot open.

Plain-language explainer

RAG, explained interactively

What is retrieval-augmented generation (RAG)?

RAG is how an AI answers from your documents instead of only its training. When you ask a question, the system searches your content for the most relevant passages, pastes them into the model's context, and asks the model to answer using them. The model never memorized your data. It reads the retrieved text at answer time. That is why RAG can cite sources and stay current, and why most RAG failures are really retrieval failures: if the right passage was not fetched, the model cannot use it.

Last reviewed August 4, 2026

Do not just read it. Operate the mechanism yourself in a short interactive lesson.

See it work: RAG: retrieval as a callback to similarity →

Free, no code, no signup.

How does a RAG pipeline work, step by step?

A RAG system has two halves that run at different times. The first half is indexing, and it happens before anyone asks anything. You ingest the documents, split each one into chunks of a few hundred tokens, run every chunk through an embedding model that turns text into a list of numbers, and store those numbers next to the original text in a searchable index. Nothing is being taught to the model here. You are building a library with a good catalogue.

The second half runs when a question arrives. The question is turned into numbers by the same embedding model, the index is searched for the passages whose numbers sit closest to it, and the top handful, usually five to twenty, come back. An optional reranking step reorders them more carefully. Then comes the step the name is about: augmentation. The chosen passages are pasted into the prompt above the question, with an instruction along the lines of 'answer using only the text below, and cite which passage you used'. The model generates its answer from that.

Notice what never happens: no weights change, no training run starts, nothing about the model is different from one question to the next. RAG is plumbing around a frozen model. The interactive lesson on this site lets you turn that crank yourself, running a query, watching which passages come back, and seeing the answer change when a wrong passage is swapped into the context.

Ingest, chunk, embed, index: done ahead of time, this builds the searchable library.
Retrieve, augment, generate: done per question, this picks passages and answers from them.

Why do products use RAG instead of retraining the model?

Retraining bakes information into the weights, and weights are a terrible filing cabinet. A training run costs real money and takes time, so your knowledge is always as old as the last run. You cannot point at where an answer came from, because the fact is smeared across billions of numbers rather than stored anywhere. You cannot reliably delete one fact when a customer asks you to. And you cannot give two employees different access, since the model that learned a document has learned it for everyone.

Retrieval fixes all four by keeping the documents outside the model. Publishing a corrected policy is an edit and a reindex, not a training run, and the next question uses it. The answer can quote the passage it used, which is what makes an answer auditable. Deleting a document removes it from the index. And because retrieval is a search you control, you can filter it by permission first, so a user only ever gets passages they are allowed to see.

None of that makes fine-tuning useless. Fine-tuning is how you change behavior: a house tone of voice, a strict output format, a domain vocabulary, a refusal style. Knowledge and behavior are different problems, and plenty of serious systems do both, fine-tuning the manner and retrieving the matter.

What each approach is actually good at. Costs are relative and illustrative, not price quotes.

What you need	RAG (retrieval)	Fine-tuning	Paste it all in context
Add new facts	Yes, as soon as they are indexed	Yes, but only via a new training run	Yes, per request
Correct one fact	Edit the document, reindex	Retrain or patch	Re-paste it every time
Cite the source	Yes, at passage level	No	Only if you ask, and it may drift
Per-user permissions	Filter at retrieval time	Not possible inside weights	Whatever you chose to paste
Shape tone and format	Weakly, through the prompt	This is its strength	Weakly, through the prompt
Cost per question	Low, a few thousand tokens	Low per call, paid up front	Highest, you pay for every token

What is retrieval, actually?

The default answer in 2026 is embedding search, also called dense or semantic search. An embedding model reads a chunk of text and returns a long list of numbers that encodes its meaning, so passages about similar things land near each other in that number space. Search means embedding the question and asking which stored vectors are closest, usually by cosine similarity. The payoff is that 'how many days off do I get' can find a passage titled 'annual leave entitlement' even though the two share no words. A vector database is just a store built to make that nearest-neighbour lookup fast over millions of chunks.

Meaning-based search has a blind spot: exact strings. Part numbers, error codes, invoice IDs, surnames and rare jargon are precisely where embeddings blur things together. Old-fashioned keyword scoring, usually BM25, is excellent at exactly that. So mature systems run hybrid retrieval, both methods at once, and merge the two ranked lists. Published comparisons consistently report a real lift from the combination rather than from either half. One 2026 e-commerce benchmark write-up reported roughly 91% recall in the top ten results for hybrid search, against about 78% for embeddings alone and 65% for keywords alone. Treat the exact figures as illustrative of the direction, not as constants.

The last common piece is reranking. A reranker, typically a cross-encoder, reads the question and one candidate passage together and scores the pair properly instead of comparing two independent vectors. That is far more accurate and far too slow to run over a whole corpus, so the standard pattern is cheap retrieval of fifty to a hundred candidates, then reranking down to the five you actually paste. It is usually the highest-value thing you can add to a mediocre pipeline.

Where does RAG break, and why is it almost always retrieval?

The generator can only work with what it was handed, so retrieval sets the ceiling on quality. If the passage holding the answer never came back, no prompt wording rescues you: the model will answer from its own general patterns, in exactly the same confident tone it uses when it is right. This is why the first move in debugging a bad RAG answer is never to edit the prompt. It is to look at the passages that were actually retrieved and ask whether a careful human could have answered from them.

The failure modes are boringly mechanical, which is good news, because mechanical problems have fixes. Most of them come from how the documents were cut up and what got lost on the way in.

One failure mode deserves separate mention because it is not a retrieval miss: conflicting sources. When the index holds last year's policy and this year's, retrieval happily returns both, and the model picks one with no way to know which is current. Dates in the metadata, deduplication, and instructions to prefer the newest source are the practical answers. Also remember that retrieved text is untrusted input. If a document can be edited by someone else, it can carry instructions aimed at your model, which is why grounding data should never be treated as a trusted part of the prompt.

Split answers: the fact sits across a chunk boundary, so neither chunk is convincing on its own.
Oversized chunks: one long chunk covers five topics, so its vector matches everything weakly and nothing strongly.
Flattened tables: a table turned into a run of text loses the link between headers and values, a widely cited cause of silent enterprise failures.
Missing context: a chunk says 'this does not apply in that case' with no sign of which document or section it came from.
Vocabulary gaps: users ask in their words, the documents use internal jargon, and pure embedding search never bridges it.
Stale index: the document was updated, the index was not, and the system is confidently quoting a deleted paragraph.

Does RAG still make sense with million-token context windows?

Context windows got genuinely huge. By 2026 a one million token window is ordinary at the frontier, some models advertise more, and that really did kill the weakest reason for RAG, which was that documents simply did not fit. If your whole knowledge base is a couple of handbooks and it changes rarely, pasting it in and skipping the infrastructure is now a sensible choice, and prompt caching makes repeating it cheaper.

What size did not fix is cost, latency and accuracy at scale. You pay for input tokens on every single question, so a million-token prompt is orders of magnitude more expensive per answer than a few retrieved passages, and it is markedly slower to first word. Accuracy also sags for facts buried in the middle of a very long input, a well documented effect, so more context is not automatically more correct. And plenty of real corpora, a company wiki, a support archive, a code monorepo, run to hundreds of millions of tokens, which no window holds.

So the honest 2026 framing is not RAG versus long context, it is RAG feeding long context. Retrieval narrows a huge corpus to a generous, relevant slice, and the big window lets you be less surgical about it: you can afford fifty passages instead of five, which forgives a lot of imperfect ranking. You keep the things a window never gave you, per-user permission filtering, citations, and an index you can update this afternoon.

Illustrative token math for one question against a knowledge base of about one million tokens.

Approach	Input tokens per question	Relative input cost	What it costs you
Paste the entire base	~1,000,000	~250x	Slowest, and impossible once the base outgrows the window
Retrieve 8 passages of ~500 tokens	~4,000	1x	Fast and citable, but ranking has to be good
Retrieve 50 passages into a long window	~25,000	~6x	The common 2026 middle ground, forgiving of imperfect ranking
A 100 million token corpus	Does not fit	n/a	Retrieval is the only option available

How do teams tell whether their RAG is working?

The useful habit is to score the two halves separately, because they fail for different reasons and only one of them is fixable by prompting. On the retrieval side, context recall asks whether the passage that contains the answer was retrieved at all, and context precision asks how much irrelevant text tagged along with it. On the generation side, groundedness, often called faithfulness, asks whether every claim in the answer is actually supported by the retrieved text, and answer relevancy asks whether the response addressed the question that was asked.

In practice that becomes a small, unglamorous evaluation set. Collect fifty to a couple of hundred real questions, record which source passage should answer each one, and measure recall in the top k results first. Retrieval quality is the ceiling, so tuning the prompt while recall is poor is wasted work. Once retrieval is healthy, groundedness is usually scored by asking a separate model to check each claim against the passages, with a human spot check to keep the judge honest. Open frameworks such as RAGAS, DeepEval, TruLens and Arize Phoenix package these metrics so you do not have to build the harness yourself.

Then there is the part that only production teaches you. Log the retrieved chunks alongside every answer, so a complaint can be traced instead of argued about. Track how often nothing good was found, because that rate is your real content gap. And let the system say it does not know when the passages do not support an answer. An honest refusal is cheap. A fluent, well-cited, wrong answer is what erodes trust in the whole product.

What people get wrong

RAG means the model was trained on your data. It is not; it reads retrieved text at the moment of answering.
A wrong RAG answer means a weak model. Far more often the retrieval missed the right passage.
RAG replaces fine-tuning. They solve different problems: RAG adds knowledge, fine-tuning shapes behavior.
Adding RAG eliminates hallucination. It reduces it when retrieval is good, and it introduces new errors when the retrieved passage is wrong or out of date.

Where you see it in real products

Support bots answer from a help center using RAG.
Internal 'chat with your docs' tools retrieve from a private knowledge base.
AI search engines fetch web pages, then write an answer that cites them.
Coding assistants pull the relevant files and snippets out of your repository before proposing an edit.

Common questions

When should I use RAG instead of fine-tuning?: When the answer depends on facts that change, or that you need to cite. Retrieval keeps documents outside the model, so updating a fact is an edit rather than a training run, and the answer can point at its source.
Why does RAG still get answers wrong?: Usually retrieval, not generation. If the right passage was never fetched, the model answers from its own patterns instead, just as fluently. Most RAG debugging is checking what was actually retrieved before blaming the model.
Do I still need RAG with a very large context window?: Often yes. You can paste more now, but you pay for every token, latency grows, and accuracy sags for facts buried in the middle. Retrieval is also how you get citations and per-user permissions, which context size does not give you.
What is a vector database, and do I need one?: It is a store that indexes embeddings so you can find the nearest ones fast across millions of chunks. Below a few thousand chunks you can get away with a database extension or even plain keyword search, and many teams start there before adding one.
How big should the chunks be?: A few hundred tokens is the usual starting point, often around 200 to 500, with a little overlap and splits placed on real boundaries like headings. Too large and the chunk matches everything weakly, too small and it loses the context that makes it understandable.
What does it mean for an answer to be grounded?: Every factual claim in the answer can be traced back to a specific retrieved passage rather than to the model's memory. Teams score this as groundedness or faithfulness, and a low score means the model is filling gaps the retrieved text did not cover.
Is giving an AI web search the same thing as RAG?: It is the same shape with a different corpus. The search engine is the retriever, the web is the index, and the fetched pages are pasted into the context before the model writes. That is why AI search products can cite links and answer about this morning's news.
Does RAG keep my documents private?: Your documents stay in your own store, but the passages that get retrieved are sent to the model inside the prompt, so they do leave your system at answer time. Check your provider's data terms, and filter retrieval by permission so nobody retrieves what they cannot open.

Related explainers

Embeddings AI agents Context engineering Hallucination RAG vs fine-tuning

One idea at a time, in your inbox

New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.

We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.

Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.