Chapter 04

Retrieval & reranking

Basic vector search is a solid start, but it misses exact terms and returns “close-ish” results. Two techniques close most of the gap: hybrid search and reranking.

Dense vs sparse retrieval

Dense (semantic)

Embedding similarity. Understands meaning and paraphrase — “can't sign in” matches “login failure”. But it can miss exact identifiers, product codes, or rare terms.

Sparse (lexical / BM25)

Classic keyword scoring. Nails exact matches — error codes, names, SKUs, function names — but is blind to synonyms and rephrasing.

Hybrid search fuses both

Run dense and sparse in parallel, then combine their rankings (commonly with Reciprocal Rank Fusion). Hybrid consistently beats either method alone — it captures conceptual meaning and exact keywords.

🔢 Dense

meaning

🔤 Sparse

keywords

→

🔀 Fuse (RRF)

merged ranking

→

🎯 Rerank

top-N by relevance

Reranking: the biggest single lever

Fast retrieval optimises for recall — cast a wide net, grab ~20 candidates. A reranker (a cross-encoder that reads the query and each passage together) then re-scores them for true relevance and keeps the best few. It's slower per item, so you only run it on the shortlist — but the top-5 after reranking are far better than the top-5 from search alone.

Watch it work. These 6 passages came back from vector search, ranked by embedding score. Hit rerank — a cross-encoder re-reads each against the query and the genuinely relevant ones rise.

Query “How do I get a refund on an annual enterprise plan?”

Hybrid + rerank, in code

import numpy as np, voyageai
from rank_bm25 import BM25Okapi

vo = voyageai.Client()

def hybrid_retrieve(query, chunks, doc_vecs, bm25, k=20):
    # dense: cosine similarity ranking
    q_vec = np.array(vo.embed([query], model="voyage-3", input_type="query").embeddings)[0]
    dense = np.argsort(doc_vecs @ q_vec)[::-1]
    # sparse: BM25 keyword ranking
    sparse = np.argsort(bm25.get_scores(query.split()))[::-1]
    # Reciprocal Rank Fusion — reward chunks ranked high by EITHER method
    scores, C = {}, 60
    for rank, i in enumerate(dense[:k]):  scores[i] = scores.get(i, 0) + 1 / (C + rank)
    for rank, i in enumerate(sparse[:k]): scores[i] = scores.get(i, 0) + 1 / (C + rank)
    fused = sorted(scores, key=scores.get, reverse=True)[:k]
    return [chunks[i] for i in fused]

def rerank(query, candidates, top_n=5):
    # cross-encoder reads (query, passage) pairs together — high precision
    r = vo.rerank(query, candidates, model="rerank-2", top_k=top_n)
    return [candidates[res.index] for res in r.results]

candidates = hybrid_retrieve(query, chunks, doc_vecs, bm25)  # ~20, high recall
context    = rerank(query, candidates, top_n=5)              # 5, high precision

Rule of thumb: retrieve wide (k≈20–50 with hybrid), rerank narrow (top 3–8). It's the cheapest quality win in RAG — one extra model call that dramatically cuts irrelevant context before it reaches the LLM.

Other retrieval tricks

Query rewriting

Have an LLM expand or rephrase the question (or generate several variants) before retrieving — catches more relevant chunks.

Metadata filtering

Attach metadata (date, source, product) to chunks and filter before or during search — “only the 2026 docs”.

Small-to-big

Retrieve on small precise chunks, then feed the model their larger parent sections for full context.

HyDE

Generate a hypothetical answer, embed that, and search with it — often closer to the target docs than the raw question.