Fine-Tuning vs RAG vs Prompting: LoRA, SFT, DPO and Distillation

When to prompt, use RAG or fine-tune an LLM: supervised fine-tuning, LoRA and QLoRA, preference tuning with DPO, distillation and serving adapters.

Advanced 3 min read · Updated 29 October 2026

There are three main ways to adapt a foundation model to your product: prompting (change the instructions), retrieval-augmented generation (change the context), and fine-tuning (change the weights). A common interview question is "would you fine-tune or use RAG?" The short answer: RAG for knowledge that's private or changing, fine-tuning for behaviour (format, style, a narrow skill) or to make a smaller model do a big model's job cheaply, and prompting first for everything.

Choosing the approach

A decision path for adapting a model. Most products stop at prompting plus RAG; fine-tuning is for behaviour, consistency or cost at volume, after evaluation shows the simpler options fall short.

Need	Prompting	RAG	Fine-tuning
Answer from private or fresh documents	No	Yes	Poor: facts go stale and can't be cited or permission-filtered
Consistent output format or tone	Often	No	Yes
Narrow skill (classify our tickets, our SQL dialect)	Sometimes	Partly	Yes
Cut cost and latency at high volume	No	No	Yes: distil into a smaller model
Shorter prompts (drop long examples)	No	No	Yes
Time to first version	Hours	Days	Days to weeks, plus data work

Kinds of fine-tuning

Method	Training data	Use
Supervised fine-tuning (SFT)	Input and ideal output pairs	Formats, styles, task skills
Preference tuning (DPO and RLHF-style methods)	Pairs of a preferred and a rejected answer	Tone, helpfulness, following policy
Distillation	A strong model's outputs on your inputs	Make a small model match a big one on your task
Continued pretraining	Large amounts of raw domain text	Specialist vocabulary (legal, medical); rarely needed
Reinforcement fine-tuning	Tasks with a verifiable grader (tests pass, answer correct)	Reasoning-heavy tasks with checkable outcomes

Parameter-efficient fine-tuning: LoRA and QLoRA

Full fine-tuning updates every weight, which needs a lot of GPU memory and produces a full copy of the model per task. LoRA freezes the base model and trains small low-rank adapter matrices added to some layers, often well under 1% of the parameters. QLoRA does the same on top of a 4-bit quantized base model, cutting memory further.

LoRA serving: one shared base model in GPU memory, with small per-task or per-customer adapters loaded alongside it and selected per request.

Adapters are small, so many can be stored and swapped cheaply; serving stacks can batch requests for different adapters on one base model.
Training fits on far fewer GPUs, which makes per-task or even per-customer models feasible.
Quality is usually close to full fine-tuning for format and task adaptation.

The fine-tuning pipeline

Define the task and eval set first, with the base model's score as the baseline; see LLM evaluation.
Collect data: production logs (with consent and PII removed), expert-written examples, or a strong model's outputs reviewed by humans. Hundreds to a few thousand high-quality examples often beat many more noisy ones.
Clean and split: deduplicate, remove eval items from training (contamination), balance cases.
Train (LoRA first), tracking validation loss and eval scores per checkpoint.
Evaluate against the baseline on the task and on general behaviour and safety, since tuning can erode abilities the task doesn't exercise.
Deploy behind the gateway with canary traffic; version the adapter with its data and base model; see LLMOps.

Risks and costs

Base model upgrades: an adapter is tied to its base model; a new base means retraining and re-evaluating.
Forgetting and safety regressions: narrow tuning can weaken general skills or refusals; test them.
Data leakage: a model can memorise training examples; never train on secrets or data users didn't agree to.
Ownership: you now own a model lifecycle (data, training, evaluation, serving) instead of only prompts.

A worked cost argument for distillation

Task        : classify 20M support messages/month into 40 intents
Large model : ~600 tokens/message (prompt with examples) → 12B tokens/month
Fine-tuned small model : ~120 tokens (no examples needed) on a self-hosted GPU
Result      : eval accuracy 94% (large) vs 93% (tuned small) on the golden set;
              cost and latency down by an order of magnitude → ship the small model,
              route low-confidence cases to the large model

Where it’s used

High-volume classification and extraction

Distilled small models replace large-model calls, with the large model as a fallback.

Brand voice and formats

SFT for a consistent writing style or output structure that prompting couldn't hold.

Multi-tenant products

Per-customer LoRA adapters served on one shared base model.

In an interview

Use RAG for knowledge and fine-tuning for behaviour or cost, after trying prompts.
Explain LoRA and why adapters make serving many variants cheap.
Start with an eval set and baseline; check for regressions outside the task.
Note the lifecycle cost: data, retraining on base model upgrades, versioning.

Frequently asked questions

Should I fine-tune an LLM or use RAG?

Use RAG for knowledge that is private or changes, since it stays fresh, cited and permission-filtered. Fine-tune for behaviour such as format, style or a narrow skill, or to make a smaller model do a task cheaply. Try better prompts first.

What is LoRA?

Low-rank adaptation: the base model is frozen and small adapter matrices are trained instead, using far less memory. Many adapters can be served on one shared base model.