Chapter 03
Chunking & embeddings
Retrieval quality is decided before a single query runs — in how you split documents and how you turn text into vectors. Get these wrong and no amount of clever prompting recovers it.
Why chunk at all?
You can't embed a 40-page PDF as one vector — you'd lose all detail, and you couldn't fit it into a prompt. So documents are split into chunks: passages small enough to embed precisely and to pack several into the model's context.
Chunk size is a genuine trade-off:
Chunks too small
Precise matches, but each fragment lacks context — a sentence pulled out of its section can be misleading or unanswerable.
Chunks too large
Rich context, but the embedding blurs multiple topics together, retrieval gets noisier, and you burn more tokens per chunk.
Try it. Drag to change the chunk size and watch the same document split differently. Notice how overlap keeps ideas from being cut in half.
→ ? chunks. Overlap shown as the faint repeated words between blocks.
Chunking strategies
Fixed-size
Every N tokens/words, with overlap. Simple, fast, and a perfectly good baseline.
Recursive / structural
Split on natural boundaries — paragraphs, headings, sentences — before falling back to size. Keeps ideas intact.
Semantic
Split where the topic shifts (detected via embedding similarity between sentences). Chunks map to coherent ideas.
Document-aware
Respect the format: code by function, Markdown by section, tables as units. Structure carries meaning — don't shred it.
What is an embedding?
An embedding is a list of numbers — a vector — that represents the meaning of a piece of text. The trick: texts with similar meaning get vectors that sit close together in that high-dimensional space, even if they share no words. “How do I reset my password?” lands near “I forgot my login credentials.”
Meaning becomes geometry. A simplified 2-D view of an embedding space — related documents cluster; the query lands nearest the cluster that answers it.
text-embedding-3, Cohere, or open models like sentence-transformers. Pick
one and use it for both indexing and querying.Embed and search, in code
Embed the chunks once at index time, embed the query at request time, then rank by cosine similarity. Here it's a tiny NumPy search for clarity — in production swap in a real vector DB.
import numpy as np
import voyageai # Anthropic's recommended embeddings provider
vo = voyageai.Client() # reads VOYAGE_API_KEY
def embed(texts, kind):
# input_type distinguishes stored documents from search queries
return np.array(vo.embed(texts, model="voyage-3", input_type=kind).embeddings)
# --- index time (offline) ---
doc_vecs = embed(chunks, kind="document") # shape: (n_chunks, dim)
# --- query time (online) ---
def retrieve(query, k=5):
q = embed([query], kind="query")[0]
# cosine similarity (vectors are L2-normalised by voyage)
scores = doc_vecs @ q
top = np.argsort(scores)[::-1][:k]
return [chunks[i] for i in top]