Fine-Tuning vs. RAG vs. Prompting: How to Adapt an LLM
Three levers, not one decision
A base large language model knows a great deal about the world and nothing about your world — your products, your policies, your tickets, your tone. Closing that gap is the central engineering task of every enterprise AI project, and there are exactly three levers to pull: change the prompt, change the context you retrieve at query time, or change the weights through training. Teams get into trouble when they treat these as a ranking — fine-tuning at the top, prompting at the bottom — and reach for the heaviest tool first.
They are not a ranking. They solve different problems, and the right architecture usually combines two of them. This guide describes what each lever actually does, the failure mode it addresses, and how to decide without burning a quarter on the wrong approach.
Prompting: cheapest, fastest, and often enough
Prompting means shaping behavior entirely through the text you send at inference time — instructions, examples, and formatting rules — without touching the model or adding a retrieval system. It is the highest-leverage place to start because it is instant, free of infrastructure, and fully reversible.
The techniques that matter in practice:
- Clear instructions and role framing. Specify the task, the audience, the format, and the constraints explicitly. Most "the model is bad at this" problems are underspecified prompts.
- Few-shot examples. Include two to five worked examples of input and desired output. This teaches format and edge-case handling far more reliably than prose.
- Structured output. Ask for JSON against a schema, or use the provider's structured-output mode, when the result feeds another system.
- Decomposition. Break a hard task into steps, or into a small chain of prompts, rather than demanding one giant leap.
Prompting cannot do two things: it cannot give the model facts it was never trained on, and it cannot permanently change deeply ingrained behavior. When you hit those walls, you add retrieval or training — but only then.
RAG: give the model the right facts at query time
Retrieval-augmented generation keeps your knowledge in an external store — a vector index, a search engine, or a database — and fetches the relevant passages at query time, inserting them into the prompt so the model answers from them. The model's weights never change; you are changing what it reads before it answers.
RAG is the correct tool when the problem is knowledge: the model needs to answer from information it does not contain. That covers most enterprise use cases — support answers from a knowledge base, policy lookups, answering over contracts or tickets. Its structural advantages are the reason it dominates:
- Freshness. Update a document and the next answer reflects it. No retraining.
- Auditability. The system can cite the passage it used, which is decisive for regulated decisions and for containing hallucination.
- Access control. Because data stays in a store you own, you can filter retrieval by the requesting user's permissions, so people only see passages they are entitled to. Baked-in weights cannot enforce row-level access.
- Data privacy. Sensitive content stays in your boundary until the moment of use, and can be redacted on the way to the model — the job of a privacy layer like the intSignal AI Gateway.
RAG's quality is bounded by retrieval quality. If the right passage is not retrieved, the model cannot use it, and a fluent wrong answer is the result. Investment in chunking, embeddings, and re-ranking is where RAG projects succeed or fail.
Fine-tuning: change behavior, not knowledge
Fine-tuning continues training a base model on your examples, adjusting its weights so the new patterns become part of how it responds. The critical mental model: fine-tuning is excellent at teaching behavior, form, and style, and a poor and expensive way to teach facts.
Fine-tuning earns its cost when:
- You need a consistent format, tone, or structure that prompting cannot reliably produce across thousands of calls.
- You are encoding a specialized task — classification into your taxonomy, extraction into your schema, a domain-specific style of reasoning — where you have a good set of labeled examples.
- You want to shrink cost and latency by teaching a smaller model to match a larger one's behavior on a narrow task (distillation), so you can run the cheaper model in production.
The tradeoffs are real. Fine-tuning needs a curated, high-quality dataset; garbage examples produce a confidently wrong model. The result is frozen at training time, so new facts mean a new training run. And fine-tuned weights cannot enforce access control. Modern parameter-efficient methods such as LoRA lower the compute cost considerably, but the data-curation cost remains the real expense.
A side-by-side comparison
| Dimension | Prompting | RAG | Fine-tuning |
|---|---|---|---|
| Solves | Behavior via instructions | Missing knowledge | Ingrained behavior/style |
| Changes | Nothing persistent | External data store | Model weights |
| Freshness | Instant | Instant per document | Requires retraining |
| Setup cost | Lowest | Moderate | Highest |
| Access control | N/A | Per-document at retrieval | Not enforceable |
| Best for | Format, decomposition | Facts, citations, Q&A | Consistent task behavior |
Read the table as a diagnosis tool. Match the row that names your actual problem — missing facts point to RAG, inconsistent form points to fine-tuning, underspecified instructions point right back at the prompt.
How to choose — and why you often combine
A pragmatic decision path:
- Start with prompting. Push it hard before adding infrastructure. A large share of "we need to fine-tune" conclusions dissolve under a better prompt with good few-shot examples.
- Add RAG when the gap is knowledge. If the model needs facts from your corpus, retrieval is almost always the answer — and the governance benefits make it the safe default for regulated data.
- Fine-tune when the gap is behavior. Once retrieval is solid but the model still will not hold a format or a specialized task consistently, fine-tune for that behavior — ideally on a smaller model to cut inference cost.
The strongest production systems combine them: a fine-tuned model that reliably produces your output format, fed by RAG that supplies current, permission-filtered facts, driven by a disciplined prompt. These are complementary layers, not rival camps.
Where to start
Pick one real use case and walk the ladder in order. Write the best prompt you can and measure it against a held-out set of real inputs. If it fails on knowledge, add retrieval and measure again. Only if it still fails on consistent behavior do you reach for training. Skipping to step three is the most common and most expensive mistake in enterprise AI.
If you want help matching the architecture to the problem — and building the retrieval, evaluation, and governance around it so the result holds up in production — our machine learning and AI team does exactly this. Talk to our team and start with the lightest tool that solves the problem, not the heaviest one that sounds impressive.
