RAG or Fine Tuning: Which One Do You Actually Need?
Sergio JimenezOctober 21, 20265 min read
Teams reach for fine tuning when they should reach for retrieval, and the mistake is expensive in a specific way: you spend six weeks and a training budget discovering that the model still does not know last Tuesday's policy change.
The two techniques solve different problems. Here is the distinction that matters.
The short answer
| You want the model to | Use |
|---|---|
| Know facts it was never trained on | RAG |
| Know facts that change weekly | RAG |
| Cite a source for what it says | RAG |
| Answer in a consistent format or voice | Fine tuning |
| Follow a specialised task reliably | Fine tuning |
| Be cheaper per call at high volume | Fine tuning |
| Both of the above | Both, in that order |
The rule that survives contact with reality: RAG teaches knowledge, fine tuning teaches behaviour. If your problem is that the model does not know something, training will not fix it reliably. If your problem is that the model knows but answers badly, retrieval will not fix it either.
What RAG actually is
Retrieval augmented generation means finding the relevant passages from your own content at question time and putting them in front of the model along with the question. The model reasons over text it can see rather than over what it memorised.
What you get: facts that are current the moment you update a document, citations that let a reader verify an answer, no training cost, and the ability to remove a document and have the system immediately stop citing it. That last property matters more than people expect, because it is what makes the system governable.
What it costs: every call carries the retrieved passages, so your token bill is higher per request, and quality depends almost entirely on retrieval. A RAG system that returns the wrong three paragraphs produces a confident wrong answer, and the model is not the thing at fault.
What fine tuning actually is
Continuing to train a model on your examples so its default behaviour changes. You are not adding a knowledge base; you are shifting what the model does by reflex.
It is excellent at: producing output in a rigid structure, matching a house style without a long prompt explaining it, handling a narrow classification task better than a general model, and cutting cost at scale because a small fine tuned model can match a large general one on one specific job.
It is bad at: facts. Training does not create a reliable lookup table. The model will produce something that looks like your data and is sometimes wrong, with no way to tell which. It is also static: the knowledge is frozen at training time, and updating means training again.
The test that decides it
Ask one question. If a human expert had the right document in front of them, could they answer correctly?
If yes, your problem is retrieval. Get the right document in front of the model. Fine tuning will not help and may hurt, because a model trained on your documents becomes more confident about them without becoming more accurate.
If no, because answering requires a skill or a judgement rather than a fact, then you have a behaviour problem and fine tuning is the right tool.
What we try before either
Most problems presented as "we need fine tuning" are solved by prompting, and it is worth an afternoon to find out. A clear system prompt with three or four worked examples closes the majority of format and tone gaps at zero training cost and with no new infrastructure.
Our order of escalation, which has not changed in two years:
- Prompt engineering with examples. Hours. Solves most format and tone problems.
- RAG. Weeks. Solves every knowledge problem, and is the only option if facts change.
- Fine tuning. Weeks plus training cost. For behaviour that prompting cannot reach, or for cost at scale.
Skipping to step three because it sounds more serious is the most common and most expensive mistake in applied AI.
When you genuinely want both
A mature system often uses both, and the division of labour is clean: fine tune for the shape of the answer, retrieve for the content of it.
A support agent is the canonical example. Fine tune it so every reply follows your escalation policy, uses your tone and always ends with the next step. Retrieve the current product documentation so the facts inside that structure are right today. Neither technique could do the other's job, and together they do what neither does alone. If you are costing this out, our breakdown of what an AI agent costs to build and run covers where each sits in the budget.
Frequently asked questions
What is the difference between RAG and fine tuning?
RAG supplies knowledge at question time by retrieving relevant passages and showing them to the model. Fine tuning changes the model's default behaviour by training it on examples. RAG teaches knowledge, fine tuning teaches behaviour. They solve different problems and are not substitutes.
Should I use RAG or fine tuning for my company documents?
RAG, in almost every case. Documents change, and a fine tuned model's knowledge is frozen at training time and cannot cite a source. Use RAG so an updated document is reflected immediately and every answer can be traced back to the passage it came from.
Is fine tuning cheaper than RAG?
Cheaper per call, more expensive to set up and to maintain. Fine tuning costs training time and money up front but needs no retrieved context in the prompt, so each request is smaller. At very high volume on a narrow task that economics wins. At low or moderate volume it rarely pays back.
Can you use RAG and fine tuning together?
Yes, and mature systems usually do. Fine tune for the shape of the answer, its structure, tone and policy, and retrieve for the content, the facts that must be current. A support agent that follows your escalation rules by reflex while citing today's documentation is exactly this combination.
Why does my RAG system give wrong answers?
Nearly always retrieval, not the model. If the wrong passages are retrieved, the model answers confidently from the wrong source. Check what is actually being retrieved before changing the prompt or the model: the usual culprits are chunks that are too large, documents with several contradictory versions, and no filter on recency.
Do I need fine tuning to make the model sound like my brand?
Try prompting first, with three or four worked examples in the system prompt. That closes most tone gaps in an afternoon at no training cost. Fine tune only when prompting has genuinely failed, or when the prompt needed to carry the style has grown long enough to be expensive on every call.
Where to start
Take ten real questions your users ask. For each, note whether a person with the right document could answer it. Count them. That ratio tells you which technique you need, and it takes an hour.
If the answer is mixed, or the retrieval you already have is returning the wrong passages, send us the ten questions and we will tell you which of the three steps you are actually on.
