Embeddings Search: From Cosine Demo to Production

Naive vector search — fixed 500-token chunks, cosine similarity, top-k — plateaus around 60–70% recall on real query traffic. Production quality comes from three upgrades applied in order: structure-aware chunking, hybrid BM25-plus-vector retrieval with rank fusion, and a cross-encoder reranker. Each step is roughly a day of work and each buys mid-to-high single-digit recall points. The prerequisite for all of them is the thing teams skip: an evaluation set of your own queries, because without it every one of these changes is a guess.

Build the eval set before touching anything

A hundred real queries from your logs, each labelled with the documents that should come back, stored as JSONL. Budget 2–3 hours of labelling. Metrics: recall@10 (did the right doc make the candidate set) and MRR (how high did it land). That's the whole harness — maybe 80 lines of Python — and it converts retrieval work from vibes to engineering. It also tells you when to stop: if recall@10 is already 0.9 on your actual queries, a fashionable new embedding model is a lateral move, and you'll know that in ten minutes instead of after the migration. The same harness later feeds your RAG-level evals, where retrieval misses masquerade as model hallucinations. Refresh the set quarterly from fresh logs — query distributions drift, and an eval set frozen in January quietly stops representing June's traffic.

Chunking: structure beats token counts

Fixed-size chunking cuts sentences in half and welds unrelated sections together. Split on document structure instead — headings, sections, list boundaries — targeting roughly 200–500 tokens per chunk, and keep tables and code blocks whole; a bisected table is garbage in both halves.

The highest-ROI trick in the whole pipeline is embarrassingly simple: prepend the document title and heading path to every chunk. A chunk that reads "Restore procedure: run the following…" embeds poorly; the same chunk as "BorgBackup docs > Disaster recovery > Restore procedure: run the following…" carries its context into the vector. It costs ~15 tokens per chunk and reliably lifts recall several points on documentation-shaped corpora. Once you chunk on structure, sliding-window overlap mostly stops paying for itself — structure boundaries were the thing overlap was compensating for.

Hybrid: BM25 catches what vectors miss

Embeddings encode meaning, and that's exactly why they fail on strings whose meaning is the string: error codes, function names, version numbers, product names. A query for ERR_CONN_REFUSED or sonarr wants exact lexical match, and cosine similarity gives you semantically-adjacent mush instead. Building search across this directory's 3,550 apps made this vivid — vector-only search returned "similar media automation tools" for queries that were literally an app's name.

Run BM25 and vector search in parallel and fuse with Reciprocal Rank Fusion:

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.__getitem__, reverse=True)

RRF needs no score normalisation and no tuning beyond the conventional k=60 — which is precisely why it's the right default. Learned or weighted fusion can beat it, but in my experience rarely by enough to justify the maintenance, and a bad weight silently wrecks one query class while improving another.

Reranking: top-50 in, top-5 out

First-stage retrieval (either kind) is a compression trick — the document was embedded once, blind to the query. A cross-encoder reranker reads the actual query and each candidate together and scores the pair, which is simply a better-informed judgment. Retrieve 50 candidates with hybrid search, rerank, keep the top 5–10.

Open-weight rerankers in the BGE family (see the model zoo around sentence-transformers) run 50 pairs in tens of milliseconds on a modest GPU, or a few hundred on CPU. In most pipelines I've measured, the reranker is the single largest quality jump — bigger than any embedding-model upgrade — because it fixes the "right doc at rank 23" problem that first-stage retrieval structurally has. Cache scores by (query, doc) hash; real traffic repeats itself.

What the upgrade path looks like

Representative numbers from a documentation-corpus pipeline I ran — yours will differ, which is what your eval set is for:

Note what's absent from the table: swapping the vector database. Storage choice moves latency and ops burden, not recall. pgvector with HNSW indexes is comfortable into the low millions of vectors; past that, or when you need heavy metadata filtering and multi-tenancy, a dedicated engine like Qdrant earns its place — the decision ladder is in choosing a vector database. Choose it last, after the pipeline above, because it's the component that changes your results the least.

What I'd do

Week one: eval set, then structure-aware chunking with heading prefixes — measure. Week two: add BM25 alongside vectors with RRF — measure. Week three: bolt on an open-weight cross-encoder over the top-50 — measure. Skip any step whose predecessor already hit your quality bar, and revisit the embedding model only if the eval says lexical-vs-semantic isn't your problem. The pattern worth internalising: every upgrade that mattered here was about what gets compared and when — not about the database, and not about the fashionable model of the month.