RAG vs. Fine-Tuning: Choosing the Right Approach

When a model doesn't know enough about your domain, two options come up almost immediately: give it the right information at request time with retrieval-augmented generation (RAG), or change the model itself with fine-tuning. Teams often frame this as a one-time architectural choice. In practice it's a question about what kind of gap you have, and the answer is frequently “retrieval first, fine-tuning maybe later, for a narrower reason.”

This guide walks through what each approach actually changes, the failure modes of each, and a checklist you can run before committing engineering time.

What each approach actually changes

RAG changes the input. At request time you search a corpus — documents, tickets, product data — and place the most relevant passages into the prompt. The model's weights are untouched; it reasons over text you hand it. Update the corpus and the next request sees the new information.

Fine-tuning changes the model. You train on examples of inputs and desired outputs, and the weights shift toward producing outputs like those examples. It is good at teaching behavior: a consistent format, a house style, a classification scheme, the shape of a good answer. It is a poor and expensive way to teach facts, because facts absorbed into weights are hard to update, hard to attribute, and easy to blend into confident-sounding errors.

When RAG is the better fit

Retrieval is usually the right starting point when your knowledge changes, when answers need to be traceable to a source, or when different users are allowed to see different documents.

• Freshness matters. Pricing, policies, inventory, and internal docs change weekly. Re-indexing a document is cheap; retraining a model is not.
• Answers need citations. Because the source passages are in the prompt, you can ask the model to cite them and verify those citations programmatically.
• Access control is per user. You can filter retrieval by tenant, role, or document permissions. Weights can't enforce that boundary.
• The corpus is large or long-tailed. No training run will reliably memorize thousands of rarely asked details, but a search index can surface them on demand.

RAG has its own failure modes, and most of them live in the retrieval step rather than the model. Poor chunking splits the answer across passages, weak queries miss the relevant document, and too many loosely related passages dilute the context. When a RAG system gives a wrong answer, check whether the right passage was retrieved before you touch the prompt.

When fine-tuning earns its cost

Fine-tuning becomes worth considering when you have a stable, well-defined task, a meaningful number of high-quality examples, and evidence that prompting alone has plateaued.

• Consistent output shape. A classifier with a fixed label set or an extractor that must always emit the same structure can become more reliable with targeted examples.
• Tone and style at scale. If every response must follow a specific voice and long style instructions are eating your context budget, fine-tuning can move that behavior into the model.
• Latency and cost. A smaller fine-tuned model can sometimes match a larger general model on a narrow task, which may reduce per-request cost and latency. Verify this on your own eval set; it is not guaranteed.

Risks to plan for

Fine-tuning introduces a training pipeline, a dataset you must curate and version, and a model artifact you must evaluate and redeploy whenever the base model changes. Narrow training data can also degrade general capabilities the product quietly relied on. Budget for that ongoing ownership, not just the first training run.

Side-by-side comparison

Combining both approaches

The approaches are not mutually exclusive. A common mature pattern is a RAG system whose generator has been fine-tuned to use retrieved context well: cite passages in a fixed format, say “I don't know” when the context doesn't contain the answer, and stay within a defined tone. Retrieval supplies the facts; fine-tuning supplies the discipline. Reach for this only after the RAG baseline is measured and the remaining failures are clearly behavioral.

Whatever you choose, make the decision with the same eval set run against each candidate. The snippet below shows the shape of that comparison. The functions are placeholders for your own application code.

1from collections.abc import Callable2from dataclasses import dataclass3 4# Placeholders for your own code: an eval set loader, a grader,5# and one answer function per candidate approach.6from app.evals import Case, grade, load_cases7from app.pipelines import answer_prompt_only, answer_with_rag8 9 10@dataclass11class Result:12    name: str13    pass_rate: float14    failed_ids: list[str]15 16 17def evaluate(name: str, answer: Callable[[str], str], cases: list[Case]) -> Result:18    failed = [c.id for c in cases if not grade(c, answer(c.question))]19    return Result(name, 1 - len(failed) / len(cases), failed)20 21 22if __name__ == "__main__":23    cases = load_cases("evals/product_qa.jsonl")24    # Start with the cheapest candidate; add a fine-tuned one only if needed.25    for name, fn in {"prompt_only": answer_prompt_only, "rag": answer_with_rag}.items():26        r = evaluate(name, fn, cases)27        print(f"{r.name:<12} pass={r.pass_rate:.0%} failed={r.failed_ids[:5]}")

A short decision checklist

1. Write down the specific failure you're trying to fix, with three to five real examples.
2. Classify each example: missing knowledge, wrong behavior or format, or both.
3. Build a small eval set (even 30–50 cases is a useful start) and measure a prompt-only baseline.
4. If failures are mostly missing knowledge, build a RAG baseline and check retrieval quality separately from answer quality.
5. If failures are mostly behavioral, iterate on instructions and few-shot examples before considering training.
6. Consider fine-tuning only when the task is stable, you have a curated example set, and prompting has measurably plateaued.
7. Whatever you ship, keep the eval set and rerun it whenever prompts, data, or models change.