Narsil benchmarks: one engine, embedded or on a server

Narsil runs two ways from a single codebase. You can embed it inside your application process like a library, and you can run it as a search server that scales across machines. This page measures both, because portability is the goal: the engine that indexes a few thousand documents inside a browser tab is the same engine that answers queries behind an HTTP API.

Every number on this page is generated from a recorded run rather than typed by hand. A script reads the latest run of each suite and fills the tables and charts below from the raw results, and a continuous-integration check fails the build if this page ever drifts from those recordings. The bars are horizontal and scaled to the best value in each group, so a full bar marks the leader and the rest sit in proportion. Every chart on this page reads higher-is-better, and each section links to the run it came from so you can read the per-engine detail and reproduce the figures yourself.

Search servers: keyword, vector, and hybrid retrieval

The first comparison runs over HTTP against six production search engines on BEIR datasets, the datasets and metrics that the published information-retrieval leaderboards use. Each engine ingests the corpus, answers the dataset's test queries, writes a TREC run file, and gets scored with pytrec_eval, the same tool the BEIR leaderboard uses. The comparison runs three tracks. The keyword track scores BM25 ranking. The vector track scores dense nearest-neighbour search. The hybrid track scores keyword and vector combined. On the vector and hybrid tracks every engine receives identical precomputed vectors from one fixed embedding model, so the comparison measures the index and holds the embedder constant.

Narsil calibrates its BM25 against the Anserini reference configuration, so the rest of the comparison stands on a trusted baseline. The setup, the pinned engine versions, and the datasets all come from the recorded run.

• Run. These figures come from run 20260804T184221Z, recorded on 2026-08-04 from commit ad93b7f4fe58. The raw per-engine results and the full comparison are in the run report.
• Datasets. The run covers SciFact (5,183 documents) and NFCorpus (3,633 documents), each loaded and hash-verified through ir_datasets.
• Engines. The comparison runs Narsil 0.2.2 against Elasticsearch 9.5.0, Meilisearch 1.52.0, OpenSearch 3.7.0, Qdrant 1.18.3, Typesense 30.2, and Weaviate 1.39.0, and every engine runs from a pinned image.
• Equal conditions. Every engine receives the same 8.6 GB memory cap, the same run depth of 1,000, and the same run-file ordering, and the engines run one at a time so latency never contends.
• Machine. The run executed on GCP c3-standard-8, us-central1-a, which reports Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz and Linux 6.17.0-1021-gcp x86_64.
• BM25 calibration. Narsil indexes each corpus with BM25 k1=0.9 and b=0.4, the Anserini reference configuration.

Keyword track

Narsil's BM25 is calibrated to the Anserini reference, so it ranks with the Lucene engines on these graded judgements and finishes a little ahead of them on both datasets. Typesense and Meilisearch apply their own documented ranking models rather than BM25, which places them lower here.

nDCG@10 on SciFact, higher is better:

Narsil        ██████████████████████████████ 0.6814
Elasticsearch █████████████████████████████▉ 0.6789
OpenSearch    █████████████████████████████▉ 0.6789
Meilisearch   ████████████████▌              0.3748
Typesense     ████████████████▍              0.3728

Peak throughput on SciFact, queries per second, higher is better:

Narsil        ██████████████████████████████ 958 QPS
OpenSearch    ███████████████████████████▌   878 QPS
Elasticsearch ██████████████████████████▍    841 QPS
Meilisearch   █████████████████████████▋     818 QPS
Typesense     █████▉                         189 QPS

nDCG@10 on NFCorpus, higher is better:

Narsil        ██████████████████████████████ 0.3278
Elasticsearch █████████████████████████████▍ 0.3206
OpenSearch    █████████████████████████████▍ 0.3206
Meilisearch   ███████████████████████▍       0.2550
Typesense     ████████████████▋              0.1817

Peak throughput on NFCorpus, queries per second, higher is better:

Narsil        ██████████████████████████████ 1,089 QPS
Elasticsearch ██████████████████████████▉    975 QPS
OpenSearch    ██████████████████████████▊    969 QPS
Meilisearch   ████████████████████████▋      893 QPS
Typesense     ███████████████████████▌       852 QPS

Vector track

Every engine indexes the identical vectors and tunes its search effort up to the same matched recall point against the exact nearest neighbours. Retrieval quality is therefore equal across engines by construction, so this track compares speed at that point. The throughput differences at a few thousand vectors reflect per-request handling at this corpus size, since every engine sits near full recall at a modest search effort.

On SciFact, every engine tunes its search effort to reach ann_recall@10 of at least 0.99 against the exact neighbours, and each returns the same ranking, so nDCG@10 is 0.6239 and Recall@100 is 0.9227 across the field.

Peak throughput on SciFact at matched recall, queries per second, higher is better:

OpenSearch    ██████████████████████████████ 730 QPS
Qdrant        ████████████████████████████▊  698 QPS
Elasticsearch ████████████████████████████▍  690 QPS
Weaviate      ██████████████████████████▏    637 QPS
Narsil        ██████████▋                    259 QPS

On NFCorpus, every engine tunes its search effort to reach ann_recall@10 of at least 0.99 against the exact neighbours, and each returns the same ranking, so nDCG@10 is 0.3145 and Recall@100 is 0.3094 across the field.

Peak throughput on NFCorpus at matched recall, queries per second, higher is better:

Elasticsearch ██████████████████████████████ 715 QPS
OpenSearch    █████████████████████████████▊ 710 QPS
Qdrant        █████████████████████████████▌ 703 QPS
Weaviate      ██████████████████████████▌    632 QPS
Narsil        ███████████▎                   267 QPS

Hybrid track

Hybrid fusion combines the keyword and vector rankings, and the fusion method differs per engine, so ranking quality varies again.

nDCG@10 on SciFact, higher is better:

Qdrant        ██████████████████████████████ 0.7155
Elasticsearch █████████████████████████████▋ 0.7053
OpenSearch    █████████████████████████████▋ 0.7053
Narsil        █████████████████████████████▌ 0.7026
Weaviate      ████████████████████████████▉  0.6885

Peak throughput on SciFact, queries per second, higher is better:

Qdrant        ██████████████████████████████ 668 QPS
OpenSearch    █████████████████████████████▌ 656 QPS
Elasticsearch ████████████████████████████▉  642 QPS
Weaviate      ███████████████████████▏       516 QPS
Narsil        ████████████▏                  269 QPS

nDCG@10 on NFCorpus, higher is better:

Narsil        ██████████████████████████████ 0.3560
OpenSearch    █████████████████████████████▋ 0.3521
Elasticsearch █████████████████████████████▋ 0.3516
Qdrant        █████████████████████████████▋ 0.3515
Weaviate      ████████████████████████████▉  0.3427

Peak throughput on NFCorpus, queries per second, higher is better:

Qdrant        ██████████████████████████████ 683 QPS
OpenSearch    █████████████████████████████▉ 681 QPS
Elasticsearch █████████████████████████████▌ 672 QPS
Weaviate      ████████████████████████       547 QPS
Narsil        ███████████▌                   263 QPS

A note on latency

Throughput under concurrent load is the headline speed measure here, because single-query latency cannot separate these engines at a few thousand documents. Narsil reports its server-side query time in floating milliseconds, so its sub-millisecond searches are recorded exactly. Elasticsearch, OpenSearch, Meilisearch, and Typesense report whole milliseconds, so their sub-millisecond searches fall below what their own timers can resolve. Weaviate exposes no server-side query time, so only its client round-trip is recorded. The linked run report carries the full latency tables, both server-side and client round-trip.

Embedded search: in-process against Orama and MiniSearch

The same engine also runs as a library inside one Node.js process, with no server and no network, against Orama and MiniSearch. This is the embedded class, where Narsil indexes and queries in the same process as your application code. The speed tiers run on a BEIR corpus, and ranking quality is scored on BEIR SciFact with its human relevance judgements. All three engines use the same Lucene English stop words and default BM25 parameters. Each one stems English with its own implementation, which no shared setting overrides, so a ranking gap between them carries both the ranking and the stemmer.

• Run. These figures come from run 20260804T180208Z, recorded on 2026-08-04 from commit ad93b7f4fe58. The full per-scale tables are in the run report.
• Engines. The comparison runs Narsil 0.2.2 against Orama 3.1.18 and MiniSearch 7.2.0, all inside one Node.js process.
• Machine. The run executed on GCP c3-standard-8, us-central1-a, which reports Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz, 31GB of memory, Node.js v24.19.0, and Linux x64.
• Speed corpus. The indexing and query tiers run on BEIR FiQA, 50,000 documents, measured at 1,000, 10,000, and 50,000 documents.
• Relevance dataset. Ranking quality is scored on BEIR SciFact, 5,183 documents and 300 judged queries, verified by archive checksum 536e14446a0b.

Ranking quality

Ranking quality on BEIR SciFact, nDCG@10, higher is better:

Narsil     ██████████████████████████████ 0.6840
Orama      ███████████████████▏           0.4351
MiniSearch ███████████                    0.2506

Indexing and query speed

The suite records indexing throughput, query latency, and resident memory at each corpus scale, and it measures filtered search where the engine supports it.

Insert throughput at 50,000 documents, documents per second, higher is better:

Narsil     ██████████████████████████████ 8,903 docs/s
MiniSearch ████████████████████▍          6,063 docs/s
Orama      ████████████▏                  3,611 docs/s

Insert throughput at each scale, documents per second:

Search latency at each scale, p50 milliseconds:

Resident memory at each scale, megabytes:

Filtered search latency at 50,000 documents, p50 milliseconds:

Vector search

Narsil carries vector search in the same embedded engine. MiniSearch has no vector support, so this tier compares Narsil against Orama.

Embedded vector search on BEIR SciFact:

Embedded vector search on BEIR NFCorpus:

Reproduce these numbers

• Search servers. The only requirement is Docker. From benchmarks/server/, run ./run-all.sh. The harness builds the Narsil server from this repository, embeds every corpus once into a shared cache, runs each engine one at a time, and writes a fresh run directory under benchmarks/server/results/runs/. The server benchmark README covers the configuration and the large-dataset path.
• Embedded libraries. From the repository root, run pnpm build, then pnpm --filter benchmarks bench. The in-process benchmark README lists the tiers and the single-tier commands.
• This page. After a run, python3 benchmarks/writeup/generate.py rewrites the tables and charts above from the latest recorded run of each suite. python3 benchmarks/writeup/generate.py --check verifies that the page matches those runs, and continuous integration runs the same check.

Absolute numbers move with the hardware, so the value is in the comparison between engines measured on the same machine in the same run.