Chapter 02

The RAG pipeline

RAG has two halves that run at different times: an offline indexing phase that happens once (and on every update), and an online query phase that runs on every question. Understanding the split is the whole game.

Phase 1 — Indexing (offline)

Done ahead of time. You turn a pile of documents into a searchable index of vectors. Do it once; refresh it when the source data changes.

📄

Documents

PDFs, wikis, code, tickets

✂️

Chunk

split into passages

🔢

Embed

text → vectors

🗄️

Vector store

index for search

Phase 2 — Query (online)

Runs per request. The user's question is embedded the same way, matched against the index, and the best passages are folded into the prompt before the model answers.

❓

Query

user question

🔢

Embed

same model as index

🔎

Retrieve

top-k nearest

🧩

Augment

context + question

🤖

Generate

grounded answer

💬

Answer

+ citations

The golden rule: the query and the documents must be embedded by the same model. The vectors only share a meaningful space if they came from the same encoder — mixing models makes similarity scores meaningless.

What each stage does

Chunk

Split long documents into passages small enough to embed and to fit several into a prompt. Chunk size is a real quality knob → Chapter 3.

Embed

Map each chunk to a vector where “close” means “similar in meaning”. A dedicated embedding model, not the chat model.

Store

Index vectors for fast approximate-nearest-neighbour search — Pinecone, pgvector, Chroma, Weaviate, FAISS.

Retrieve

Embed the query, find the top-k closest chunks. Often improved with hybrid search + reranking → Chapter 4.

Augment

Insert the retrieved chunks into the prompt with instructions to answer only from them and to cite sources.

Generate

The LLM writes the final answer, grounded in the supplied context — ideally with inline citations.

The generation step, in code

Retrieval hands you a list of chunks. The “augment + generate” step is just a well-structured prompt to the chat model. Here it is with Claude:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

def answer(question: str, chunks: list[str]) -> str:
    # 1. AUGMENT — fold retrieved passages into the prompt, numbered for citations
    context = "\n\n".join(f"[{i}] {c}" for i, c in enumerate(chunks))

    # 2. GENERATE — instruct the model to stay grounded in the context
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        system=(
            "Answer the question using ONLY the provided context. "
            "Cite the sources you used as [n]. "
            "If the answer is not in the context, say you don't know."
        ),
        messages=[{
            "role": "user",
            "content": f"<context>\n{context}\n</context>\n\nQuestion: {question}",
        }],
    )
    return resp.content[0].text

Why the strict system prompt? Grounding isn't automatic — you have to ask for it. “Use only the context”, “cite sources”, and “say you don't know” are what turn a retrieval pipeline into a trustworthy one.

Where latency & cost hide

Retrieval adds a round trip before generation. The usual levers:

Latency: embedding the query + the vector search + a larger prompt all add time. Cache embeddings, cache frequent answers, and keep top-k small.
Cost: more retrieved chunks = more input tokens per request. Retrieve broadly, then rerank down to a few high-value chunks (Chapter 4) instead of stuffing everything in.
Freshness: the index is only as current as your last re-index — schedule updates for data that changes.