Pyserini

Last reviewed Sep 13, 2026 · 851 words

There is exactly one situation where a self-hoster should install Pyserini: you have a corpus of your own text, you want to know how well BM25 or a particular embedding model retrieves from it, and you would rather measure that in an afternoon than guess. It is a 2,161-star Python library from the University of Waterloo's Castorini group that wraps Anserini (Java, Lucene) for keyword retrieval and Faiss for dense vector retrieval, and its stated purpose is reproducible information-retrieval research. No daemon, no HTTP API, no admin page, no search box. For a site or app search, pick a server from the search category instead, and I say which below.

Install it as a lab bench, not as infrastructure

The install is pip install pyserini plus a JDK, since the sparse side calls into Lucene through a Java bridge (Java 21 at last check; the README pins the exact version). Faiss comes from conda or a wheel and is only needed for dense retrieval. Plan on 2 GB of RAM as the floor, and on tens of gigabytes of disk if you touch any prebuilt index: the library can download ready-made indexes for MS MARCO, BEIR, Wikipedia and other standard collections, and those are what make it useful for benchmarking without a build step.

from pyserini.search.lucene import LuceneSearcher
searcher = LuceneSearcher.from_prebuilt_index('msmarco-v1-passage')
for hit in searcher.search('what causes tides', k=5):
    print(hit.docid, round(hit.score, 2))

That is the whole loop. Swap LuceneSearcher for FaissSearcher with a query encoder and you are running dense retrieval over the same collection, which is the comparison the toolkit was built to make easy.

Indexing your own corpus takes one command and a JSONL file

Write your documents as one JSON object per line with id and contents fields, then:

python -m pyserini.index.lucene \
  --collection JsonCollection --input ./corpus \
  --index ./indexes/mine --generator DefaultLuceneDocumentGenerator \
  --threads 4 --storePositions --storeDocvectors --storeRaw

On a laptop this indexes roughly a million short passages in minutes. The --storeRaw flag matters: without it the searcher returns IDs and scores only, and you will be joining back to your source data yourself. For dense indexes, pyserini.encode runs a Hugging Face encoder over the same JSONL and pyserini.index.faiss builds the vector index, so one corpus format feeds both pipelines.

The thing it is good at: settling the sparse-versus-dense argument with numbers

Every homelab search project eventually hits the question of whether embeddings are worth the GPU. Pyserini answers it. Because the standard collections ship with relevance judgements and the toolkit includes trec_eval bindings, you can run BM25, a dense model and a hybrid fusion over your own queries and get nDCG@10 and recall numbers in the same session. My experience matches the published results: BM25 remains hard to beat on exact-term queries (part numbers, error strings, names), dense models win on paraphrase, and the hybrid of the two beats either by a few points almost everywhere. Knowing that for your data, rather than in general, changes what you deploy. The embeddings in production piece covers what happens after you have decided.

What to run once you have decided

Pyserini is where you find out; it is not what you leave running. For a site or app search with typo tolerance and a REST API, Meilisearch or Typesense are the right shape, each installs from one binary or container and each holds a few million documents in a couple of gigabytes. For heavy keyword search over large corpora where you want Lucene itself under a server, Apache Solr is the direct sibling: the same indexing engine Pyserini calls, with an HTTP API and a query parser you can hand to other people. YaCy, which this listing's slug points at, is a different animal again, a peer-to-peer web crawler and search engine that indexes the public web rather than your documents. If you came here for that, the YaCy listing is the one you want.

What it is not

It is not a vector database (no updates, no filtering, no replication), not an embedding service, and not something to put behind a web form. Indexes are files on disk that you rebuild rather than mutate, which is exactly right for research and exactly wrong for a product. The documentation is written for graduate students, so expect commands that assume you know what MS MARCO is and why a Recall@1000 figure is interesting.

What I'd do

Use Pyserini for a day. Index the real corpus, run BM25 against the best small embedding model you can serve, and score both on 30 queries you wrote yourself. Then take the answer to Meilisearch if keyword search won, or to a hybrid setup in a real vector store if dense retrieval earned its GPU. The toolkit's value is that it makes the measuring step cheap enough that you actually do it; the only mistake is treating the bench as the building.