RAG injects knowledge; fine-tuning changes behaviour. If the facts change faster than you'd realistically retrain — weekly prices, daily tickets, living documentation — or if you need answers with citations, RAG wins. If the problem is output format, tone, or a narrow task repeated millions of times, fine-tuning wins, usually on a small model. Most teams who ask for fine-tuning have a retrieval problem, because "the model doesn't know about X" is a knowledge gap, and fine-tuning is a poor way to install knowledge.
The two failure modes people conflate
Knowledge gaps: the model doesn't know your API surface, your catalogue, this week's pricing. Behaviour gaps: the model knows enough but responds in the wrong shape — too verbose, wrong JSON layout, misses your severity taxonomy, ignores house style.
Fine-tuning on new facts works badly for the first gap. A model fine-tuned on your docs will parrot phrasing from the training set while still hallucinating the details you actually need, and research on knowledge injection consistently finds fine-tuned facts less reliable than the same facts placed in context. Worse, you take on catastrophic forgetting risk and a retraining pipeline whose cadence now has to match your data's change rate. Retrieval puts the fact in front of the model at answer time, verbatim, with a provenance trail.
The reverse also holds: prompting your way to a strict output format works until it doesn't. When you're spending 800 tokens of system prompt per request policing format, and the model still drifts on 2% of requests, a fine-tune that bakes the behaviour in is cheaper and more reliable.
The decision table
| Axis | Favors RAG | Favors fine-tuning |
|---|---|---|
| Data volatility | Changes daily/weekly | Stable for months |
| Provenance | Citations required | Nobody asks "says who" |
| Training data | Few labelled examples exist | 1,000+ good input/output pairs |
| Latency | Can afford +100–500ms retrieval | Every millisecond counts |
| Task shape | Open-ended Q&A over a corpus | Narrow, repetitive transformation |
| Cost profile | Low volume, spiky | High volume, steady (amortises training) |
| Failure audit | "Why did it say that" must be answerable | Best-effort is acceptable |
Count the columns and you'll usually land on RAG plus prompting as the default — which matches deployment reality. Fine-tuning earns its keep at the margins: massive volume, tight latency, small models.
What fine-tuning is actually for
- Format and style adherence. JSON layouts, report templates, brand voice. A LoRA on a 7–8B model with 1,000–5,000 examples typically nails what a frontier model needed a page of system prompt for.
- Classification and routing at volume. A fine-tuned small model matching a frontier model's accuracy on your label set, at roughly 1/20th the per-token cost and much lower latency.
- Distillation. Have the big model label 10,000 examples, train the small one on them — the standard path to cheap specialist models.
- Local tool-calling reliability. Small open-weight models are mediocre at emitting your exact tool-call schema; a modest fine-tune fixes this better than prompt engineering does.
The economics are gentler than their reputation: QLoRA on a 24GB consumer GPU handles a 7–8B model overnight, and a rented A100 run for the same job costs $5–20. The real cost is assembling the 1,000+ good examples — budget days for data, minutes for training.
What RAG costs you
RAG's price is an eternal pipeline. Chunking, embedding, index refreshes, and — the part that dominates production error rates — retrieval quality. In most RAG systems I've debugged, the majority of wrong answers trace to the wrong context being retrieved, not to the model misreading good context. That means owning an evaluation harness for retrieval itself, which is its own discipline covered in embeddings search in production. Add 100–500ms latency per retrieval hop and an index whose staleness you now monitor like any other cache.
Hybrids that actually work
- RAFT (retrieval-aware fine-tuning). Fine-tune the model on examples that include retrieved context — some relevant, some distractors — so it learns to use and ignore retrieval appropriately. The RAFT paper showed solid domain gains, and the pattern holds up in practice: you're fine-tuning behaviour with respect to retrieval, not knowledge.
- Fine-tuned retrieval components. Keep the generator frozen; fine-tune the embedding model or reranker on your click/label data. Often the best-value fine-tune in the whole stack, and it ships without touching the LLM.
- Small fine-tuned router in front. A distilled classifier decides which pipeline (or which model) handles each query — cheap, fast, and it composes with everything else.
The pattern across all three: retrieval carries the facts, tuned weights carry the behaviour. They're complements, not rivals.
What I'd do
Start with a strong base model, retrieval, and a disciplined prompt; get an eval set before changing anything. Reach for fine-tuning when at least two of these are true: the task is stable and narrow, you have (or can distil) 1,000+ quality examples, volume or latency makes a small model economically attractive, or format drift persists after honest prompt work. And if the goal is a private stack, the same ladder applies with open weights — retrieval over your documents with a solid 8–14B model via Ollama before you ever schedule a training run. Teams regret premature fine-tunes far more often than they regret shipping month one on RAG.