Chapter 01

Why RAG?

A trained language model is a frozen snapshot of the internet. That single fact causes three problems — and retrieval solves all three at once.

The three limits of a bare LLM

🌀 Hallucination

When it doesn't know, it generates something plausible and confident — but wrong. There's no built-in notion of "I'm not sure."

🕰️ Stale knowledge

Training ended on a cutoff date. Anything newer — a release, a price, last week's incident — simply isn't in the weights.

🔒 No private data

Your wiki, codebase, contracts, and tickets were never in training. The model can't answer questions about data it has never seen.

The core idea

Instead of relying on what the model memorized, RAG fetches relevant text at query time and puts it directly in the prompt. The model then answers from that supplied context — like an open-book exam instead of a closed-book one.

See the difference. Same question, two setups:

Prompt “What's our refund window for enterprise plans?”

Answer: “Enterprise plans typically offer a 30-day refund window.”
— A confident guess. The model has never seen your policy, so it pattern-matches to something generic. It might be wrong, and it can't tell you where it got that.

Retrieved billing_policy.md §4: "Enterprise: pro-rated refunds within 45 days of renewal."

Answer: “Enterprise plans get pro-rated refunds within 45 days of renewal [billing_policy.md §4].”
— Grounded in your actual policy, and it cites the source so a human can verify.

What retrieval buys you

Factual grounding

Answers are tied to real, supplied documents — the single biggest lever against hallucination.

Fresh & private knowledge

Update the knowledge base, not the model. New docs are searchable the moment they're indexed — no retraining, no fine-tuning.

Citations & trust

Because the source is known, you can show “according to [doc]”, which is often a hard product requirement.

Cost control

Retrieving a few relevant chunks is far cheaper than stuffing an entire corpus into every prompt — and cheaper than fine-tuning.

RAG vs the alternatives

Approach	Adds new knowledge?	Stays current?	Cites sources?	Cost to update
Prompt only	Only what you paste in	—	—	Free
Fine-tuning	Yes, baked in	❌ frozen again	❌	High (retrain)
Long context	Yes, per request	✅	Partial	High (tokens/request)
RAG	✅ from a live index	✅ re-index only	✅	Low (add docs)

Fine-tuning vs RAG: fine-tuning teaches the model new behaviour or style; RAG gives it new facts. They're complementary — many systems use both. When someone asks “should we fine-tune?”, the answer for a knowledge problem is almost always “RAG first.”

Where this is heading → Even long-context models (1M+ tokens) don't kill RAG; they change it. You still need to select what to put in the window. In 2026 the field increasingly calls this context engineering — RAG is the original, and still the workhorse, form of it. We'll get there in Chapter 5.