Narsil benchmarks: one engine, embedded or on a server

Narsil runs two ways from a single codebase. You can embed it inside your application process like a library, and you can run it as a search server that scales across machines. This page measures both, because portability is the goal: the engine that indexes a few thousand documents inside a browser tab is the same engine that answers queries behind an HTTP API.

Every number and every chart on this page comes from a recorded run. A script reads the latest run of each suite and fills the tables below from the raw results, while a second script draws each figure into the run's own directory. A continuous-integration check fails the build if the page or a figure ever differs from those recordings. Each section links to the run it came from so you can read the per-engine detail and reproduce the figures yourself.

Search servers: keyword, vector, and hybrid retrieval

The first comparison runs over HTTP against six production search engines on BEIR datasets, the datasets and metrics that the published information-retrieval leaderboards use. Each engine ingests the corpus, answers the dataset's test queries, writes a TREC run file, and gets scored with pytrec_eval, the same tool the BEIR leaderboard uses. The comparison runs three tracks. The keyword track scores BM25 ranking. The vector track scores dense nearest-neighbour search. The hybrid track scores keyword and vector combined. On the vector and hybrid tracks every engine receives identical precomputed vectors from one fixed embedding model, so the comparison measures the index and holds the embedder constant.

Narsil calibrates its BM25 against the Anserini reference configuration, so the rest of the comparison stands on a trusted baseline. The setup, the pinned engine versions, and the datasets all come from the recorded run.

• Run. These figures come from run 20260928T002255Z, recorded on 2026-09-28 from commit 5dc940c70724. The raw per-engine results and the full comparison are in the run report.
• Datasets. The harness measured SciFact (5,183 documents), NFCorpus (3,633 documents), DBpedia entities 100K (100,000 documents), and DBpedia entities 1M (995,000 documents). SciFact is loaded and hash-verified through ir_datasets, with sentence-transformers/all-MiniLM-L6-v2 vectors at 384 dimensions; NFCorpus is loaded and hash-verified through ir_datasets, with sentence-transformers/all-MiniLM-L6-v2 vectors at 384 dimensions; DBpedia entities 100K is read from a published dataset artifact pinned by its SHA-256, with text-embedding-ada-002 vectors at 1,536 dimensions; DBpedia entities 1M is read from a published dataset artifact pinned by its SHA-256, with text-embedding-ada-002 vectors at 1,536 dimensions.
• Engines. The comparison runs Narsil 0.3.0 against Elasticsearch 9.5.4, Meilisearch 1.54.0, OpenSearch 3.8.0, Qdrant 1.19.1, Typesense 30.2, and Weaviate 1.39.5, and every engine runs from a pinned image.
• Equal conditions. Every engine receives the same 20 GiB memory cap, the same run depth of 1,000, and the same run-file ordering. The harness tests one engine at a time, so each engine has the machine to itself. Each Java engine divides that cap between its heap and the memory outside it. Elasticsearch reports a 10 GiB heap and OpenSearch reports a 6 GiB heap.
• Load. The harness measures throughput at 1, 2, 4, 8, 16, 32, and 64 concurrent clients, with one pass per level and 3 passes at each engine's peak level. The tables report the median peak pass with a 95% bootstrap interval. The load generator shares the machine with the engine under test, so its client processes take CPU time that the engine could otherwise use. The harness measures every engine under that same arrangement.
• Narsil threads. Narsil started at the engine defaults, and the harness read 7 worker threads, 7 request threads receiving requests, and the benchmark index scaled out across them.
• Narsil vector search. Narsil's server reports that it searches vector graphs through its native search core in C, which npm installs with the package on Node.js for macOS, Linux, and Windows.
• Machine. The engines ran on 3 machines: Narsil, Elasticsearch, and OpenSearch on GCP c3-standard-8, us-central1-a, which reports Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz and Linux 7.0.0-1011-gcp x86_64, Meilisearch on GCP c3-standard-8, us-west1-a, which reports Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz and Linux 7.0.0-1011-gcp x86_64, and Qdrant, Typesense, and Weaviate on GCP c3-standard-8, us-east1-b, which reports Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz and Linux 7.0.0-1011-gcp x86_64.
• BM25 calibration. Narsil indexes each corpus with BM25 k1=0.9 and b=0.4, the Anserini reference configuration.

Each track below opens with two bar charts per dataset, ranking quality and peak throughput, followed by the table behind them. Where a run measured more than one concurrency level, a line chart then follows each engine across those levels, and the setup block above says which levels a run measured. A latency profile at the end of the track shows how far each engine's tail stretches at its own peak. That under-load latency comes from the throughput passes, because their five-second windows hold several times more samples than the serial track, so p99.9 rests on more than the slowest one or two requests.

Keyword track

Narsil's BM25 is calibrated to the Anserini reference, so it ranks with the Lucene engines on these graded judgements and finishes a little ahead of them on both datasets. Typesense and Meilisearch apply their own documented ranking models in place of BM25, which places them lower here.

SciFact.

NFCorpus.

DBpedia entities 100K.

DBpedia entities 1M.

Vector track

Every engine indexes the identical vectors and tunes its search effort up to the same matched recall point against the exact nearest neighbours. Retrieval quality is therefore equal across engines by construction, so this track compares speed at that point. The throughput differences at a few thousand vectors reflect per-request handling at this corpus size, since every engine sits near full recall at a modest search effort. Where the run recorded throughput at every step of the recall sweep, a further chart draws queries per second against recall, so you can see how much throughput each engine gives up for the last point of recall.

On SciFact, every engine tunes its search effort to reach ann_recall@10 of at least 0.99 against the exact neighbours, and each returns the same ranking, so nDCG@10 is 0.6239 and Recall@100 is 0.9227 across the field. On NFCorpus, every engine tunes its search effort to reach ann_recall@10 of at least 0.99 against the exact neighbours, and each returns the same ranking, so nDCG@10 is 0.3145 and Recall@100 is 0.3094 across the field. On DBpedia entities 100K, every engine tunes its search effort to reach ann_recall@10 of at least 0.99 against the exact neighbours. The set carries no relevance judgements, so it reports recall, latency, and throughput and no ranking quality. On DBpedia entities 1M, every engine tunes its search effort to reach ann_recall@10 of at least 0.99 against the exact neighbours. The set carries no relevance judgements, so it reports recall, latency, and throughput and no ranking quality.

SciFact.

NFCorpus.

DBpedia entities 100K.

DBpedia entities 1M.

Hybrid track

Hybrid fusion combines the keyword and vector rankings, and the fusion method differs per engine, so ranking quality varies again.

SciFact.

NFCorpus.

DBpedia entities 100K.

DBpedia entities 1M.

Recommended production settings

The tracks above hold every engine at full float so that the comparison isolates the index. Each engine also recommends a quantisation for production, and this second pass runs the vector and hybrid tracks under those settings. The tables name the quantisation each engine applied once a run records it. Every engine still tunes its search effort to the same recall target, so compression differs by engine on purpose while the accuracy bar stays fixed. The equal-precision figures above remain the headline.

SciFact.

NFCorpus.

DBpedia entities 100K.

DBpedia entities 1M.

SciFact.

NFCorpus.

DBpedia entities 100K.

DBpedia entities 1M.

A note on latency

Throughput under concurrent load is the headline speed measure here, because single-query latency cannot separate these engines at a few thousand documents. Narsil reports its server-side query time in floating milliseconds, so its sub-millisecond searches are recorded exactly. Elasticsearch, OpenSearch, Meilisearch, and Typesense report whole milliseconds, so their sub-millisecond searches fall below what their own timers can resolve, and the latency charts leave out the points those timers floor to zero. Weaviate exposes no server-side query time, so only its client round-trip is recorded and it is absent from the server-side latency charts. The linked run report carries the full latency tables, both server-side and client round-trip.

Embedded search: in-process against Orama and MiniSearch

The same engine also runs as a library inside one Node.js process, with no server and no network, against Orama and MiniSearch. This is the embedded class, where Narsil indexes and queries in the same process as your application code. The speed tiers run on a BEIR corpus, and ranking quality is scored on BEIR SciFact with its human relevance judgements. All three engines use the same Lucene English stop words and default BM25 parameters. Each one stems English with its own implementation, which no shared setting overrides, so a ranking gap between them carries both the ranking and the stemmer.

• Run. These figures come from run 20260928T003010Z, recorded on 2026-09-28 from commit 5dc940c70724. The full per-scale tables are in the run report.
• Engines. The comparison runs Narsil 0.3.0 against Orama 3.1.18 and MiniSearch 7.2.0, all inside one Node.js process.
• Threads. Every engine answers on one thread. Narsil runs with workers.enabled off, so it holds no worker copies here, and the server comparison above is where its worker threads take part.
• Vector search path. Narsil searches vector graphs through WebAssembly in this suite, which is the path that it takes in a browser. The suite sets NARSIL_SEARCH_BACKEND=wasm, so these figures exclude the native search core that npm installs with the package on Node.js.
• Machine. GCP c3-standard-8, us-west1-a hosted this run, and it reports Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz, 31GB of memory, Node.js v24.21.0, and Linux x64.
• Speed corpus. The indexing and query tiers run on BEIR FiQA, 50,000 documents, measured at 1,000, 10,000, and 50,000 documents.
• Relevance dataset. Ranking quality is scored on BEIR SciFact, 5,183 documents and 300 judged queries, verified by archive checksum 536e14446a0b.

Ranking quality

Ranking quality on BEIR SciFact, higher is better:

Indexing and query speed

The suite records indexing throughput, query latency, and memory at each corpus scale, and it measures filtered search where the engine supports it. Memory counts the JavaScript heap plus the memory the process holds outside it, so a vector an engine keeps in a typed array or in WebAssembly memory counts the same as one it keeps as JavaScript objects. Narsil runs this track with trackPositions: false, because the suite measures no feature that reads term positions, and with workers.enabled off, because Orama and MiniSearch answer on one thread and this table compares one thread. The server comparison above runs Narsil with its defaults, which keep positions on and hold a worker copy on every thread.

Insert throughput at each scale, documents per second:

Search latency at each scale, p50 milliseconds:

Heap plus external memory at each scale, megabytes:

Filtered search latency at 50,000 documents, p50 milliseconds:

Vector search

Narsil carries vector search in the same embedded engine. MiniSearch has no vector support, so this tier compares Narsil against Orama.

Embedded vector search on BEIR SciFact:

Embedded vector search on BEIR NFCorpus:

Reproduce these numbers

• Search servers. The only requirement is Docker. From benchmarks/server/, run ./run-all.sh. The harness builds the Narsil server from this repository, embeds every corpus once into a shared cache, runs each engine one at a time under equal precision and again under its recommended production settings, and writes a fresh run directory under benchmarks/server/results/runs/. The server benchmark README covers the configuration and the large-dataset path.
• Embedded libraries. From the repository root, run pnpm build, then pnpm --filter benchmarks bench. The in-process benchmark README lists the tiers and the single-tier commands.
• This page. After a run, python3 benchmarks/writeup/charts.py draws every figure into the run's own charts/ directory, and python3 benchmarks/writeup/generate.py rewrites the tables above from the latest recorded run of each suite. python3 benchmarks/writeup/generate.py --check verifies that the page and its figures match those runs, and continuous integration runs the same check.

Absolute numbers move with the hardware, so the value is in the comparison between engines measured on the same machine in the same run.