Tantivy

Last reviewed Aug 26, 2026 · 828 words

You will almost certainly never run Tantivy. It is a library in the same sense Lucene is, and it lives inside things you might run: Quickwit for logs, ParadeDB for full-text search inside Postgres, and a string of smaller engines. The reason to understand it is that it explains why those tools are fast and cheap on memory, and why none of them is a drop-in for Elasticsearch despite the catalogue listing that as the thing it replaces.

A search library is an engine without a car around it

Elasticsearch is Lucene plus a server: a REST API, a cluster, replication, a query DSL, security, dashboards. Tantivy is the Lucene part, written in Rust, MIT licensed, 15,983 GitHub stars. It gives you a schema, an index on disk, an indexing writer and a searcher, and nothing that listens on a port. That is the catalogue's one drawback, "library, not a turnkey server", and it is not a flaw; it is the design. Every product built on it makes its own choices about the server, and those choices are what you evaluate when picking what to self-host.

Where it turns up in a self-hosted stack

Quickwit is the flagship: log and trace search built on Tantivy with object storage as the backing store, positioned as the cheaper Elasticsearch for observability data. ParadeDB's pg_search extension embeds Tantivy in PostgreSQL, so BM25-ranked full-text search becomes a Postgres index type and you skip the separate search server entirely. Several smaller engines use it as well. The ones people assume use it and do not: Meilisearch has its own engine, and Typesense is C++. So "Tantivy-based" tells you about indexing performance and memory behaviour, not about the API you will talk to.

Why it is fast and light

Tantivy follows Lucene's architecture: an index is a set of immutable segments, each holding an inverted index with postings compressed and skip-able, plus optional fast fields for sorting and aggregation. Segments are memory-mapped rather than loaded, so the working set is whatever the OS decides to keep hot, and the listed 128 MB minimum is honest for an index of a few gigabytes. Scoring is BM25, the same as modern Lucene. Indexing is multithreaded with a configurable memory budget per writer, and the project has historically benchmarked competitively against Lucene on both indexing and query latency; I will not quote a number because it shifts release to release, but "faster to index than Lucene, similar query speed, far less RAM than a JVM" is the reliable summary.

What you give up against Elasticsearch

There is no cluster and no replication; an index is a directory on one machine, and scaling out is the job of whatever wraps it. There is one writer per index at a time, enforced with a lock file, so concurrent ingest has to be funnelled through a single process. The schema is declared up front, with field types and whether each is indexed, stored or a fast field, and changing it means reindexing. Query features cover term, phrase, fuzzy, range, boolean, faceting and aggregations, but the long tail of Elasticsearch's DSL, the analyzers for 30 languages, the ingest pipelines and the security layer are not there. If you need any of that, you need the product built around it, not the library. Vector search is likewise not Tantivy's job; the embeddings in production piece covers how full-text and vector retrieval get combined.

If you actually embed it

For a Rust application, tantivy is a crate: define a schema, open or create an index in a directory, add documents through an IndexWriter, commit, and query through a searcher with the built-in query parser. Python bindings exist as tantivy-py and are the practical route for a small Flask or FastAPI service that needs real full-text search over 100,000 documents without running another daemon. A tantivy-cli binary can build and query an index from the command line, which is enough for a scripted "index this directory of Markdown and grep it properly" job. The difficulty rating of Hard is fair: it is a well-documented library, but you are writing the server.

What I'd do

Do not install Tantivy. Decide what you are searching and install the wrapper that fits: Quickwit for logs and traces at volume, ParadeDB if the data already lives in Postgres and you want one database rather than two, Meilisearch for the search box on a website or app. Embed the library only if you are writing a Rust or Python application where a search daemon would be the only extra process, and then keep the index on local disk with one writer. Everything else in the search category is a decision about the car, and Tantivy is the engine you get for free inside the good ones.