Every LLM project reaches the same fork: the base model does not know your business. Teams then reach for the most sophisticated-sounding fix, which is usually fine-tuning, when a cheaper option would have met the requirement. The three customization paths solve different problems, and choosing well is mostly about naming which problem you actually have.
Three tools, three problems
- Prompting solves an instruction problem. If the model has the knowledge and capability but needs direction on task, tone, format, or rules, that belongs in the prompt. Modern models follow long, structured instructions well, and large context windows hold substantial reference material.
- Retrieval-augmented generation solves a knowledge problem. If the model needs facts it was never trained on, your product catalog, your policies, this morning's ticket, RAG fetches the relevant material at request time and hands it to the model as context. The knowledge stays current, permissioned, and auditable.
- Fine-tuning solves a behavior problem. If the model must reliably produce a specific style or structured format, or follow domain conventions that are awkward to specify in instructions, training on examples changes the default behavior. What fine-tuning does not do well is add facts: it teaches how to answer, not what is true.
The decision sequence
Work the options in cost order. First, exhaust prompting: a rigorous prompt with good examples, measured against an evaluation set, is the baseline every fancier approach must beat. Second, if failures trace to missing or stale knowledge, add retrieval. Third, if failures persist in format or style even with the right knowledge in context, and you have hundreds to thousands of high-quality examples, consider fine-tuning. Many production systems land on prompting plus RAG and never need more.
What each path really costs
Prompting is cheap to change and ships in days; its risk is brittleness, which an evaluation set catches. RAG's cost is mostly the unglamorous data work: cleaning documents, chunking sensibly, keeping the index synchronized with the source of truth, and carrying permissions through to retrieval so users cannot ask the assistant for documents they may not read. Retrieval quality, not the model, is where most RAG systems fail.
Fine-tuning's cost is the dataset and the lifecycle. Curating examples is real work, the tuned model is tied to a base model you now must track, and every future change means retraining and re-evaluating. Meanwhile a tuned model still hallucinates facts, which surprises teams who reached for tuning to fix accuracy.
Common failure patterns
- Fine-tuning on a few dozen examples and getting a model that is slightly different but not better. Below hundreds of quality examples, prompting with examples usually wins.
- Building RAG over documents nobody curated. Retrieval faithfully surfaces the obsolete policy next to the current one, and the model blends them.
- Skipping the evaluation set, which makes every comparison an opinion. You cannot choose between paths you cannot score.
- Treating the choice as permanent. Healthy systems evolve: start with prompting, add retrieval when knowledge gaps appear, tune later if a behavior gap survives.
The question to ask before any of it
Write down ten real inputs and the outputs you would consider correct. If you cannot, the project is not ready for any customization path, because none of them can be aimed. If you can, you have the seed of an evaluation set, and the choice between prompting, RAG, and fine-tuning becomes an engineering decision with numbers attached instead of a fashion decision.