Retrieval, Fine-Tuning, or Both?
This choice is made badly more often than any other in custom LLM projects, usually because fine-tuning sounds more impressive. Twelve questions about your actual requirements will point to the right architecture — which for most organisations is not the one they assumed.
A higher score points towards retrieval; a lower score towards fine-tuning. Scores in the middle usually mean retrieval first, with fine-tuning considered later if a specific gap remains.
RAG or Fine-Tuning?
Higher scores point to retrieval. Lower scores point to fine-tuning.
A guide to structure the architecture conversation, not a substitute for technical assessment of your specific data and requirements.
What Each Approach Actually Does
The distinction is often described as knowledge versus behaviour, and that framing holds up well in practice. Confusing the two is what produces expensive projects that answer the wrong problem.
Retrieval gives the model knowledge
Retrieval-augmented generation fetches relevant documents at query time and passes them to the model as context. The model does not learn your information — it reads it each time — which means updating your knowledge is as simple as updating a document.
Fine-tuning changes behaviour
Fine-tuning adjusts the model’s weights using examples of the input and output you want. It is effective for teaching a consistent style, format or task pattern, and ineffective as a way of teaching facts, which is the most common misuse.
Both together is legitimate but rarely first
Mature systems sometimes fine-tune for consistent output format and use retrieval for current information. That combination is real, but attempting it initially means debugging two variables at once. Retrieval first is almost always the correct starting point.
The Four Things That Decide It
Twelve questions across four dimensions. Data volatility carries the most weight, because it is the factor most likely to make fine-tuning the wrong answer outright.
Data volatility
How often the underlying information changes. Frequently changing knowledge strongly favours retrieval, because re-training on every change is impractical.
Output requirements
Whether you need factual answers grounded in documents, or consistent behaviour, tone and structure across every response.
Traceability
Whether users need to see where an answer came from. Citation is straightforward with retrieval and effectively impossible with fine-tuning alone.
Resources and data availability
Whether you have thousands of high-quality training examples and the budget to produce and maintain them.
The Four Misconceptions That Cost the Most
These come up in nearly every architecture conversation, and each one has led organisations to spend heavily on the wrong approach.
"We will fine-tune the model on our documents"
This is the single most expensive misconception in the field. Fine-tuning on a document corpus teaches the model to imitate the style of those documents, not to recall their contents reliably. Ask it a specific factual question and it will produce something that reads correctly and may be entirely invented.
- Fine-tuning teaches patterns and style, not reliable factual recall
- Facts learned in training cannot be cited or verified
- Updating a single fact requires re-training the entire model
- Retrieval is the correct approach for document-grounded answers
"Fine-tuning will be cheaper than paying for context"
Occasionally true at very high query volumes, but the comparison usually omits the cost of producing training data, the engineering effort, and re-training whenever the model provider deprecates a version. Reducing retrieved context is a far cheaper way to cut token spend.
- Training data preparation is usually the dominant cost
- Model deprecation forces periodic re-training you must budget for
- Context reduction achieves similar savings for a fraction of the effort
- Compare total cost of ownership, not per-query token cost alone
"Fine-tuning will make it more accurate"
It makes output more consistent, which is not the same thing. If the model is producing factually wrong answers, that is usually a retrieval problem — it was not given the right information — and fine-tuning will make the wrong answers more consistently formatted rather than correct.
- Diagnose whether failures are retrieval failures or generation failures
- Retrieval failures are fixed by better retrieval, not by training
- Fine-tuning improves format and tone consistency reliably
- Measure both retrieval quality and answer quality separately
"We need a fine-tuned model to sound like us"
Sometimes true, but prompt engineering with good examples achieves most of this for a tiny fraction of the cost and effort. Try the cheap approach thoroughly before committing to the expensive one — the gap is frequently smaller than expected.
- Try few-shot prompting with strong examples first
- A detailed system prompt handles most tone requirements
- Fine-tune only if prompting demonstrably falls short after real effort
- Measure the difference rather than assuming it
Next Steps
LLM Cost Calculator
Architecture drives cost. Model what your chosen approach will run to.
Model the cost →RAG Architecture in Australia
How retrieval systems are built and deployed for Australian organisations.
Read the guide →Private LLM Readiness Assessment
Check your data and governance foundations before building either.
Assess readiness →Frequently Asked Questions
Because the questions are weighted towards the conditions that make retrieval the right choice, and those conditions describe the large majority of organisational use cases: information that changes, answers that need to be traceable to a source, and a limited supply of high-quality labelled training examples. Fine-tuning is genuinely valuable, but its ideal conditions — stable data, no citation requirement, thousands of clean examples, and a specific behaviour that prompting cannot achieve — are considerably rarer than the enthusiasm for it suggests.
Yes, and it is a legitimate architecture for mature systems. The usual pattern is fine-tuning for consistent output structure or a specialised task format, with retrieval supplying current factual grounding. The caution is one of sequencing rather than principle: doing both from the start means debugging two variables simultaneously, and you will not know which component is responsible for a given failure. Build retrieval first, measure it properly, and add fine-tuning only if a specific, identified gap remains.
For meaningful behavioural change, plan on at least several hundred high-quality examples and expect the results to improve materially into the low thousands. Quality matters considerably more than volume — a few hundred carefully constructed, consistent examples typically outperform several thousand noisy ones. The practical constraint is that producing them requires subject-matter expert time, which is expensive and scarce. Organisations that cannot commit that time reliably should not plan on fine-tuning.
It changes deployment considerations more than the architecture choice. Both approaches can be run in a private or sovereign environment. Retrieval has a practical advantage for confidential data: because the information stays in your own document store and is only passed as context at query time, it never becomes embedded in model weights, and access controls can be enforced at retrieval time so users only see documents they are entitled to. With fine-tuning, confidential information becomes part of the model itself and cannot be selectively withheld from particular users.
Almost certainly not, and this is the most common wrong turn in these projects. Diagnose the failure first by checking whether the correct information was actually retrieved and passed to the model. In the substantial majority of cases it was not, which makes it a retrieval problem — poor chunking, weak embeddings, no re-ranking, or a query the retrieval step handled badly. Fine-tuning cannot fix information the model never received. Fix retrieval, measure again, and only then consider whether a generation problem remains.
A functional retrieval prototype over a modest document corpus can be built in days, with a production-grade system including permissions, evaluation and monitoring taking weeks to a few months depending on scale and integration requirements. Fine-tuning is dominated not by the training run — which is often hours — but by producing and validating the training data, which commonly takes months of subject-matter expert time. That data preparation burden is the reason fine-tuning projects overrun far more often than retrieval projects do.
Not Sure the Answer Fits?
Tell us your use case and we will tell you which architecture we would choose and why — including when the honest answer is that you do not need a custom model at all.