Plain-language explainer
RAG or fine-tuning: how to actually choose
RAG (retrieval-augmented generation) gives a frozen model the right documents to read at answer time; fine-tuning changes the model's weights with new training examples. Use RAG for facts that change, private data and citations. Use fine-tuning for behavior: tone, format and narrow skills. Many production systems use both.
Should I use RAG or fine-tuning?
Ask what you are missing. If the model lacks facts, use retrieval: keep the documents outside the model and fetch the relevant ones into the prompt, so a wrong fact is an edit and answers can cite a source. If the model has the facts but gets the format, tone or task wrong, fine-tune: it buys consistency and shorter prompts. Knowledge is retrieval, behavior is fine-tuning. Most teams need retrieval, and the ones who reach for fine-tuning first usually end up doing both.
Last reviewed
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model reads at answer time | The model's weights |
| Best for | Facts that change, private documents, citations | Tone, format, a narrow skill |
| Update cost | Add or delete a document; no training run | A new training run and a new eval |
| Failure mode | A wrong or stale snippet is retrieved and the answer is confidently wrong | Loses general ability; hard to audit what it learned |
| Needs | An index, an embedding model, a retrieval eval | Labeled examples, GPU time, an eval set |
Do not just read it. Operate the mechanism yourself in a short interactive lesson.
See it work: RAG: retrieval as a callback to similarity →Free, no code, no signup.
Ten seconds, no lesson: play with the standalone widget, and embed it in your own page.
How does each one actually work?
RAG stands for retrieval augmented generation, and it is less exotic than it sounds. You split your documents into chunks, run every chunk through an embedding model, and store the resulting vectors in an index. When a question arrives, you embed the question too, find the stored chunks whose vectors sit nearest to it, and paste the best few into the prompt above the question. The model answers by reading them, exactly as if you had pasted the text by hand. Its weights never change. The knowledge lives outside the model, in documents you can edit, version, and delete.
Fine-tuning is further training. You collect hundreds or thousands of example pairs, each an input plus the output you wish the model had produced, and run a training job that nudges the model's weights toward those examples. The most common recipe, LoRA, freezes the original weights and trains a small adapter on top, which is why it fits on a single rented GPU. What changes is the model itself: how it phrases things, which format it defaults to, what it refuses. Nothing extra is attached at question time.
Where the change lives explains almost every tradeoff on this page. Retrieval adds knowledge per request and can point at its source. Fine-tuning bakes behavior in, invisible and always on. The interactive lesson on this site lets you turn the retrieval crank yourself: embed a question, watch the nearest chunks surface, and see the answer change when the pasted context changes.
Which one fits which problem?
The fastest way to choose is to name the failure you are seeing. If the model does not know something, that is a knowledge gap, and knowledge gaps are retrieval's job. If the model knows enough but keeps answering in the wrong shape, tone, or format, that is a behavior gap, and behavior is what fine-tuning changes. Teams that skip this diagnosis often fine-tune against a knowledge gap, which is the expensive way to learn that facts do not stick well in weights.
One honest default comes first: before either, try a better prompt with two or three worked examples. Prompting is the cheapest lever and ships in minutes, and it is the baseline you need anyway to prove that RAG or fine-tuning improved anything at all.
| Your goal | Reach for | Why |
|---|---|---|
| Answer questions from your own documents | RAG | Facts stay editable and answers can cite the chunk they used |
| Information that changes weekly | RAG | Editing a document updates the next answer, no training run |
| Citations or an audit trail | RAG | Retrieval can show its source, weights cannot |
| A house tone that must hold everywhere | Fine-tuning | Style lives in weights, and examples teach it better than instructions |
| Strict output format at high volume | Fine-tuning | A tuned small model holds the format with a short prompt |
| Both problems at once | Both | Tune the behavior, retrieve the facts |
What does each one cost?
RAG's bill is mostly per request. The setup is cheap: embedding models price at pennies per million tokens as of mid-2026, so indexing even a large document set usually costs a few dollars, and a vector index ranges from free, with pgvector inside a Postgres you already run, to a few hundred dollars a month on a managed service at scale. The part that grows is the recurring one. Every request now carries a couple of thousand retrieved tokens, so each call costs a little more and waits on a retrieval step, forever.
Fine-tuning inverts the shape. You pay up front, then each request gets cheaper, because the behavior no longer needs a long prompt. A LoRA fine-tune of a small open model is reported to cost a few dollars of rented GPU time per run, hosted services price small-model training at well under a dollar per million training tokens as of mid-2026, and a full fine-tune of a large model still runs to thousands. The bigger up-front cost is rarely the GPU bill. It is writing hundreds of clean training examples, plus the eval set that tells you whether the run helped.
The rule of thumb in 2026 cost write-ups: fine-tuning starts paying for itself when one narrow, stable task runs at high volume, the case where a tuned small model replaces an expensive general one on millions of requests. Below that, retrieval wins on total cost almost by default.
| What you pay for | Rough cost | How often |
|---|---|---|
| Embedding 1M tokens of documents | $0.02 to $0.13 | once per document version |
| Vector index hosting | free (pgvector) to hundreds per month | monthly |
| About 2,000 retrieved tokens per request | fractions of a cent at mid-tier prices | every request |
| LoRA fine-tune of a 7B open model | a few dollars of GPU time, reported | per training run |
| Hosted fine-tuning of a small model | under $1 per million training tokens | per training run |
| Full fine-tune of a large model | thousands of dollars | per training run |
When do you use both?
In production the question stops being either-or fast. The standard mature pattern is a division of labor. Fine-tune a model for behavior: the format, the domain vocabulary, the refusal rules. Then put it behind a retrieval pipeline that supplies the facts at request time. Practitioner write-ups through 2025 and 2026 describe this combination as the default for serious deployments, often with a small open model carrying the tuned behavior while retrieval carries the knowledge.
The research points the same way. A Microsoft team compared the two head to head for injecting new knowledge and found retrieval consistently better at getting a model to answer questions about facts it had never trained on. From the other side, a Berkeley technique called RAFT, retrieval augmented fine-tuning, tunes the model specifically to be good at RAG: it trains on questions paired with retrieved documents, some deliberately irrelevant, so the model learns to quote the right chunk and ignore the distractors.
If you run both, keep the jobs separate in your head and in your evals. Behavior regressions point at the tuned weights. Wrong facts point at the retrieval step. A system where you cannot tell which one failed is a system you cannot fix.
How does each one fail?
RAG fails at the retrieval step, quietly. If the search returns the wrong chunks, the model answers fluently from bad context, and the failure looks exactly like a success. Long context adds its own problems: models attend less reliably to text buried in the middle of a big prompt, and reasoning across many retrieved documents is still weak. Most real RAG bugs trace to unglamorous causes. Bad chunking. A stale index nobody re-embedded. A question phrased differently from every document that answers it.
Fine-tuning fails inside the weights, where you cannot look. The classic failure is catastrophic forgetting: train hard on a narrow task and the model gets measurably worse at the general instruction following and reasoning it used to handle. The second is staleness. Whatever you trained is frozen at training time, and changing it means building data and running the job again. Facts trained in are the worst case: unreliable to recall, impossible to cite, and expensive to remove.
The defense is the same for both: a written eval set you run before and after every change. A retrieval bug and a forgetting regression are both invisible in a demo and obvious in a benchmark of a hundred of your real questions.
What are the wrong reasons to fine-tune?
Fine-tuning is the glamorous option, which is why it attracts the wrong projects. These are the reasons that come up most and hold up least.
- To teach the model our documents. Fine-tuning mostly teaches form, and the facts that do stick are unreliable, uncitable, and frozen. A model that must answer from your documents should read them at request time.
- Our prompt is getting long. A long stable prompt is what prompt caching is for. Providers discount a repeated prefix heavily, which removes most of the cost argument for baking instructions into weights.
- To make the model smarter. Fine-tuning specializes, it does not add capability. Tuned on your task, a model gets better at your task and often a little worse at everything else.
- We have not really tried prompting. A page of clear instructions with three worked examples is the baseline. A surprising share of fine-tuning projects turn out to be a prompt that nobody wrote.
- We have training data but no evals. Without an eval set you cannot see forgetting, regressions, or whether the run helped at all. Measure first, train second.
What people get wrong
- Fine-tuning teaches the model your documents. It mostly teaches a style. Facts trained in this way are hard to update and impossible to cite.
- RAG is the cheap option and fine-tuning is the serious one. Retrieval is what production systems run; fine-tuning is a specialization on top.
- It is one or the other. Fine-tune the behavior, retrieve the facts. They solve different problems and compose.
- A chatbot on your own data needs fine-tuning. A chatbot over documents is the textbook RAG case. Fine-tuning only enters if the tone or output format will not hold.
Where you see it in real products
- Support assistants that answer from a help center and link the article they used.
- Company assistants tuned to a house style, still fetching the live document.
- Narrow classifiers fine-tuned for a fixed output format at high volume.
- Coding assistants that retrieve your repository into context instead of training on it.
Common questions
- Which one is cheaper?
- Retrieval costs more per request, because you send the fetched documents with every call. Fine-tuning costs more up front and then makes each request cheaper by shortening the prompt. At low volume retrieval wins easily; at very high volume on one narrow task, a fine-tuned small model can be the cheaper answer.
- How do I keep answers current with each one?
- With retrieval, you update the document and the next answer is current. With fine-tuning, current information means another training run, which is why nobody uses it for anything that changes weekly.
- What should I try before either of them?
- A better prompt with two or three examples, and a check that the model is actually failing at what you think. A large share of problems blamed on missing knowledge turn out to be a vague instruction or a retrieval step that returned the wrong passage.
- Should I use RAG or fine-tuning for a chatbot on my own data?
- Start with RAG. A chatbot that answers from your documents is the textbook retrieval case: facts stay editable, answers can cite their source, and you need no training data to launch. Add fine-tuning later, and only if the bot's tone or output format refuses to hold through prompting.
- Does fine-tuning add new knowledge to a model?
- Weakly and unreliably. Studies that compared the two directly, including a well-cited Microsoft experiment, found retrieval consistently better at getting a model to answer questions about facts it had not trained on. Fine-tuning shines at behavior: format, tone, and task shape. Knowledge that must be right belongs in documents the system can fetch and cite.
- What is RAFT, or retrieval augmented fine-tuning?
- A technique from Berkeley researchers that fine-tunes a model to be better at RAG. The training data pairs each question with retrieved documents, some deliberately irrelevant, so the model learns to answer from the right chunk and ignore distractors. It is the clearest example of the two approaches composing instead of competing.
- How much training data do I need to fine-tune?
- Less than most people expect for style and format. Providers have reported visible gains from as few as fifty to a hundred good examples, and a few hundred to a few thousand is a common range for a narrow task. Quality dominates quantity: a small set of clean, consistent examples beats a big scraped one.
Related explainers
More in Building on it, and trusting it
One idea at a time, in your inbox
New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.
We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.