Default to prompting, add retrieval when the model lacks knowledge, and fine-tune only when you need changed behaviour. These solve different problems and are not competing options. Most teams reach for fine-tuning to fix what is actually a retrieval gap, then spend weeks discovering it.
A support team wants their assistant to answer questions about a product that shipped last month, in the company’s house tone, following a fixed escalation policy. That single request contains all three problems, and each needs a different tool.
The new product facts are a knowledge gap. The house tone is a behaviour gap. The escalation policy is an instruction-following problem. Treating them as one decision is how projects go wrong.
At a glance
- Prompting changes instructions; retrieval changes available knowledge; fine-tuning changes behaviour
- Knowledge that changes weekly belongs in retrieval, never in weights
- Fine-tuning is a poor and expensive way to store facts
- Prompting is instantly reversible; fine-tuning is not
- The three combine well and are routinely used together
- Measure a prompted baseline before costing anything else
What does each approach actually change?
| Approach | Changes | Good at | Bad at |
|---|---|---|---|
| Prompting | The instructions in context | Format, task framing, rapid iteration | Consistency at scale, token cost per call |
| Retrieval | The facts in context | Fresh knowledge, citations, access control | Style, tone, deep behavioural change |
| Fine-tuning | The model weights | Tone, format conformance, task-specific behaviour | Storing facts, anything that changes often |
The single most useful distinction: retrieval changes what the model knows, fine-tuning changes how the model acts. Almost every misapplication comes from confusing those two.
Which variables actually decide?
Five, and the first two settle most cases on their own.
- Does the gap involve facts or behaviour? Facts point to retrieval. Behaviour points to fine-tuning. Both point to both.
- How often does the requirement change? Weekly change rules out fine-tuning, because every change forces a retrain and a revalidation cycle.
- How many labelled examples exist? Under about 100, few-shot prompting usually matches an adapter without the overfitting risk.
- Is per-call token cost or per-project cost the binding constraint? Prompting moves cost to every call forever; fine-tuning front-loads it.
- Do you need citations? Only retrieval can point at a source. Fine-tuned knowledge cannot be attributed and cannot be audited.
Figure 1 — Decision order, cheapest to rule out first
- Knowledge
Retrieval. Do not fine-tune facts in.Fresh, citable, access-controllable - Behaviour
Continue below.Prompting may still be enough
- No
Do that first. It frequently ends the project.Hours of work, zero commitment - Yes, and it fell short
Continue below.Now you have a baseline to beat
- Yes
Fine-tune, and keep retrieval for the facts.Front-loaded cost, cheaper per call - No
Stay with prompting and collect examples.Revisit when the data exists
Note where fine-tuning sits: three questions deep, reachable only after retrieval and prompting have both been ruled out. That ordering reflects how often it turns out to be the wrong tool.
Where does the cost actually land?
The three approaches distribute cost along completely different axes, which is why comparing them on a single number is misleading.
Cost shape, not cost figures
Prices change too fast to quote responsibly, so compare the shape of each cost curve instead. This is the part that does not go stale.
| Cost type | Prompting | Retrieval | Fine-tuning |
|---|---|---|---|
| Up-front | Near zero | Index build, chunking work | Data curation, training run |
| Per call | High: instructions in every prompt | Moderate: retrieved context adds tokens | Low: behaviour is in the weights |
| Per change | Near zero | Re-embed changed documents only | Full retrain plus revalidation |
| Ongoing | None | Index storage and maintenance | Adapter versioning per base model |
The crossover follows from the per-call row. At low volume, prompting’s per-call overhead never accumulates enough to justify a training run. At high volume, a long system prompt repeated millions of times becomes the dominant line item and fine-tuning starts paying for itself.
Three scenarios, three different answers
Internal documentation assistant
Thousands of documents, updated constantly, answers must cite sources.
Retrieval, and nothing else. Fine-tuning would bake in facts that go stale within days and would remove the citations that make the answers trustworthy. This case is not close.
High-volume classification
One narrow task, millions of calls a month, a fixed label set, thousands of labelled examples already sitting in a database.
Fine-tune a small model. Every element favours it: the requirement is stable, the data exists, the volume amortises the training cost, and a smaller fine-tuned model frequently beats a larger prompted one on a narrow task.
Brand-voice content drafting
Output must sound like the house style, and the underlying facts come from a product database.
Both. Retrieval supplies the facts, a fine-tuned adapter supplies the voice. This combination is common and gets missed by teams framing the decision as a choice.
What does being wrong cost?
Reversibility differs sharply, and it is rarely priced in.
| Approach | Undo cost | Failure mode when wrong |
|---|---|---|
| Prompting | Edit a string, redeploy | Inconsistency across calls |
| Retrieval | Re-chunk and re-embed | Wrong context retrieved; visible and debuggable |
| Fine-tuning | Retrain, or revert to base and lose the work | Degraded general capability; often invisible until production |
Fine-tuning’s failure mode is the dangerous one because it is silent. A model that has quietly lost general reasoning while gaining your task behaviour passes your task eval and fails on everything you forgot to test.
Common misapplications
Fine-tuning to inject knowledge
The most expensive mistake in this space. Weights are an inefficient, unauditable, un-updatable place to store facts. It sometimes appears to work on the training distribution and generalises poorly.
Prompting around a genuine behaviour gap
The opposite error. When a system prompt has grown past a thousand words of increasingly specific rules and still produces inconsistent output, the prompt is doing a job fine-tuning does better and cheaper per call.
Skipping the baseline
Without a measured prompted baseline you cannot tell whether fine-tuning helped, and you will not know whether the project was necessary. This step costs hours and regularly cancels weeks of work.
What do experienced teams do differently?
They separate the three concerns architecturally so the decision stays revisable.
Prompts live in version control, retrieval sits behind an interface, and any adapter is pinned to a base model version. When one layer needs to change, the others are untouched, and switching approaches later is a contained change rather than a rewrite.
They also start every project with retrieval plus prompting, treating fine-tuning as an optimisation applied after the system already works. Optimising something that works is straightforward; debugging three simultaneous unknowns is not.
A short glossary
- Few-shot prompting
- Including several worked examples in the prompt so the model infers the pattern without any weight changes.
- Retrieval-augmented generation
- Fetching relevant documents at query time and placing them in context so the model answers from supplied sources.
- Parameter-efficient fine-tuning
- Training a small set of additional parameters, such as low-rank adapters, while the base model stays frozen.
- Prompted baseline
- The measured performance of a strong prompt on your evaluation set, used as the bar any heavier approach must clear.
- Catastrophic forgetting
- Loss of general capability caused by adapting a model too aggressively toward a narrow task.
Key takeaways
- Retrieval changes what the model knows; fine-tuning changes how it acts. Confusing these causes most wasted effort.
- Knowledge that changes weekly belongs in a retrieval index, never in weights.
- Fine-tuning is reachable only after prompting and retrieval have both been tried and measured.
- Prompting front-loads nothing and charges per call; fine-tuning front-loads cost and charges little per call.
- Below roughly 100 labelled examples, few-shot prompting usually matches an adapter without overfitting risk.
- Only retrieval produces citations, which matters for any auditable or regulated use.
- Above all: measure a prompted baseline first, because it frequently makes the rest of the project unnecessary.
Frequently asked questions
Can fine-tuning replace RAG?
Not for knowledge. Fine-tuning stores information in weights, where it cannot be updated without retraining, cannot be cited, and cannot respect per-user access control. For facts that change or need attribution, retrieval is the correct tool and fine-tuning is an expensive substitute that degrades over time.
Should I fine-tune or write a longer prompt?
Write the prompt first and measure it. If a well-constructed prompt closes the gap, you are finished. Consider fine-tuning when the prompt has grown to many hundreds of words of specific rules, output is still inconsistent, and call volume is high enough that per-call token cost matters.
How many examples do I need before fine-tuning is worth it?
Roughly 100 as a lower bound, though quality matters far more than quantity. Below that, few-shot prompting typically performs comparably without the overfitting risk. A few hundred carefully curated examples generally beats tens of thousands of scraped ones for behaviour adaptation.
Can I use retrieval and fine-tuning together?
Yes, and it is often the correct architecture. Retrieval supplies current facts with citations while a fine-tuned adapter supplies tone, format and task-specific behaviour. Teams framing this as an either-or choice frequently pick one and then struggle with the problem the other would have solved.
What is the biggest hidden cost of fine-tuning?
Revalidation. Every requirement change forces a retrain, and every retrain needs a regression check against general capability, not just task performance. Teams budget for the training run and forget the evaluation cycle that must follow it each time.
Does fine-tuning make a model worse at other tasks?
It can, through catastrophic forgetting, and the degradation is often invisible because task-specific evaluations still pass. Run general-capability prompts unrelated to your task before and after training, and treat any meaningful drop as a cost rather than an acceptable side effect.
Is prompting always cheaper than fine-tuning?
Only at low volume. Prompting has near-zero up-front cost but charges for the instruction tokens on every single call. At high call volume that recurring overhead can exceed a one-off training run, which is where fine-tuning a smaller model starts to pay for itself.
Which approach should a new project start with?
Prompting plus retrieval. Get the system working end to end with facts flowing correctly and a strong prompt, then measure. Fine-tuning is an optimisation applied to something that already works, and introducing it early means debugging several unknowns simultaneously.
References
- Hu, E. J., et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.
- Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314, 2023. Source for the finding that small high-quality datasets outperform large scraped ones.
- Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401.
