RAG or fine-tuning? Choosing the right way to teach an LLM about your business

One of the first questions in almost every generative AI project is how to make a general-purpose language model useful for a specific organisation. The two most common answers are retrieval-augmented generation (RAG) and fine-tuning. They are often discussed as alternatives, but they address different needs.

What RAG does

RAG leaves the model unchanged. When a question arrives, the system searches your documents for relevant passages and gives them to the model along with the question. The model answers using that supplied context.

• Knowledge stays current: update a document and the next answer reflects it
• Answers can cite the source passages they came from
• Access controls can be enforced at retrieval time
• No model training is required

What fine-tuning does

Fine-tuning adjusts the model's weights using examples. It is best at changing how a model behaves: its tone, its output format, the way it handles a specialised task. It is a poor way to add factual knowledge that changes over time, because every update requires retraining and the model cannot tell you where an answer came from.

• Consistent style, structure or classification behaviour
• Smaller, faster models for a narrow task
• Domain-specific language patterns the base model handles poorly

A simple decision rule

In practice, better prompting and structured outputs solve many behaviour problems without any fine-tuning at all. We usually exhaust those options first, because they are cheaper to iterate and easier to maintain.

Where RAG projects go wrong

Most RAG quality problems are retrieval problems, not model problems. If the right passage is not retrieved, no model can produce the right answer. Common fixes include:

1. Chunking documents along their natural structure rather than fixed character counts
2. Combining keyword and semantic search
3. Re-ranking retrieved passages before they reach the model
4. Rewriting vague user questions into better search queries
5. Cleaning duplicate and outdated content from the source corpus

When to use both

Some systems benefit from a fine-tuned model that is very good at a particular output format, combined with RAG for up-to-date facts. A claims summarisation tool might fine-tune for a consistent report structure while retrieving the policy wording relevant to each claim.

Measure before you decide

Whichever path you take, build an evaluation set from real questions first. It turns a subjective debate into an engineering decision, and it keeps paying off every time you change a prompt, a model or a retrieval setting.