Chapter 06

Evaluation & pitfalls

A RAG system has two things that can go wrong — retrieval and generation — so you have to measure both. “It looks good in the demo” is how RAG projects quietly fail in production.

The RAG triad

Three questions cover most failure modes. Together they separate a retrieval problem from a generation problem — which tells you what to fix.

Context relevance

Did retrieval fetch passages that actually relate to the question? Bad → fix chunking / retrieval / reranking.

Faithfulness

Is the answer supported by the retrieved context, with nothing made up? Bad → the model is hallucinating past its sources.

Answer relevance

Does the answer actually address the user's question? Bad → prompt/generation issue.

Metrics that pin down where it breaks

Metric	Measures	Stage
Context precision	Of retrieved chunks, how many are relevant (and are the relevant ones ranked high)?	Retrieval
Context recall	Of the chunks needed to answer, how many did we actually retrieve?	Retrieval
Faithfulness	Are the answer's claims grounded in the retrieved context?	Generation
Answer relevance	Does the answer address the question directly?	Generation

Debugging shortcut: low recall means the answer was never retrievable — no prompt fixes that. Good retrieval but low faithfulness means the model is ignoring its context — tighten the prompt or lower the amount of context. Diagnose retrieval first; it's upstream of everything.

How to actually measure it

Golden set

Curate question → ideal-answer (and ideal-sources) pairs. Your regression test — run it on every change to the pipeline.

LLM-as-judge

Use a strong model to score faithfulness and relevance at scale. Correlates well with humans and is far cheaper than manual review.

Frameworks

RAGAS, TruLens, and similar compute the triad metrics for you from your questions, contexts, and answers.

Retrieval-only metrics

Evaluate the retriever in isolation with recall@k / precision@k / MRR before you even involve the LLM.

Faithfulness with an LLM judge

The core check: does every claim in the answer trace back to the retrieved context? A structured judge call turns that into a score.

import anthropic
client = anthropic.Anthropic()

def faithfulness(answer: str, context: str) -> dict:
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=512,
        system=(
            "You are a strict RAG evaluator. Decide whether EVERY claim in the answer "
            "is supported by the context. Reply as JSON: "
            '{"supported": true|false, "unsupported_claims": [...], "score": 0-1}.'
        ),
        messages=[{
            "role": "user",
            "content": f"CONTEXT:\n{context}\n\nANSWER:\n{answer}",
        }],
    )
    return resp.content[0].text  # parse the JSON verdict

# Run this across your golden set and track the average score over time.

Failure modes to watch for

🕳️ Missing content

The answer isn't in the corpus at all — the system should say “I don't know”, not invent one.

📉 Retrieval miss

The right chunk exists but ranked below top-k. Fix with hybrid search, reranking, or better chunking.

🗑️ Lost in the middle

Relevant context is buried among noise; models attend less to the middle of long prompts. Retrieve less, rerank harder.

🎭 Ignored context

The right context was supplied but the model answered from its own priors anyway. Strengthen the “use only the context” instruction.

🧩 Bad chunking

Answers split across chunk boundaries, or chunks mixing topics. Revisit chunk size/overlap and structure-aware splitting.

🥫 Stale index

Docs changed but the index didn't. Schedule re-indexing for anything that updates.

The loop that matters: build a golden set early, measure the triad, change one thing (chunk size, hybrid weights, reranker, prompt), re-measure. RAG quality is won by iteration against real questions — not by guessing.

Wrap-up

You now have the whole arc: why retrieval beats a bare model, the pipeline, the retrieval stack (chunking, embeddings, hybrid search, reranking), the advanced frontier, and how to evaluate it. That's a production-grade mental model of RAG — and of the broader move toward context engineering.