RAG vs fine-tuning for enterprise knowledge

When a client asks for "an AI assistant that knows our documentation," the underlying question is almost always about retrieval-augmented generation (RAG), not fine-tuning — even though the two get conflated constantly.

What each approach actually changes

Fine-tuning changes the model's weights based on examples you provide. It is good at teaching a model a style, a format, or a narrow task pattern it should follow consistently. It is not a good way to teach a model facts, because those facts get baked into the weights at a point in time and cannot be audited, updated individually, or traced back to a source.

RAG leaves the model's weights alone and instead retrieves relevant passages from your own content at query time, then asks the model to answer using only that retrieved material. This has three practical advantages for enterprise knowledge assistants:

• Freshness. Update the source document and the assistant's next answer reflects it — no retraining required.

• Traceability. You can show which passage the answer came from, which matters for trust and for catching the times the model gets it wrong.

• Access control. Retrieval can respect the same permissions your documents already have, so a user only ever retrieves — and is answered from — content they are allowed to see.

Where fine-tuning still helps

Fine-tuning is not obsolete. It earns its place when you need the model to consistently follow a narrow output format (a specific JSON schema, a house writing style), or when a task requires reasoning patterns that are hard to convey through a prompt alone. In practice, we most often see fine-tuning used on top of a RAG pipeline — for example, fine-tuning a smaller model to reliably extract structured fields from retrieved passages — rather than as a replacement for it.

The parts that actually determine quality

Most RAG failures are not model failures. They come from three earlier decisions:

1. Chunking. Splitting documents into pieces that are too large dilutes relevance; too small loses context. This has to be tuned per document type, not applied uniformly.

2. Retrieval strategy. Pure semantic (embedding) search misses exact terms — product codes, clause numbers, acronyms. Hybrid search, combining semantic and keyword matching, consistently outperforms either alone for enterprise content.

3. Refusal behaviour. An assistant that always attempts an answer, even with weak retrieved evidence, will eventually state something confidently wrong. Explicitly instructing the model to decline when the retrieved evidence is insufficient — and testing that it actually does — is not optional.

Evaluate before you trust it

Before any assistant goes in front of real users, we build a small evaluation set of representative questions with known-correct answers, including questions the assistant should refuse. We run this set after every meaningful change to the retrieval pipeline or prompt, and track groundedness (does the answer match the retrieved source) separately from helpfulness (is the answer actually useful). The two are not the same metric, and optimizing for one alone tends to produce an assistant that is either evasive or confidently wrong.