Partitions and workers

Partitioning and workers are how one Narsil engine scales beyond a single thread. Documents route to partitions deterministically, searches fan out and merge, and worker threads keep the main thread free. This page builds one file, logs.ts, step by step. You create a partitioned index, load log records, reshape the index while it stays online, widen each partition, move search onto worker threads, and share storage across instances. Each step highlights exactly what changed.

Create a partitioned index

An index starts with the partition count set by partitions.maxPartitions, which defaults to 1. Documents route to partitions by FNV-1a hash of their id, which keeps routing deterministic across processes and languages. Start logs.ts with an engine and a logs index split across four partitions.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 250_000 },
})

Each partition holds up to maxDocsPerPartition documents. When the index reaches maxDocsPerPartition * partitionCount documents, further inserts fail with PARTITION_CAPACITY_EXCEEDED, so the four partitions above hold a million records between them before you must reshape or widen them.

Load log records

With the index in place, load a batch of log records. Each record carries its own id, and the hash of that id determines which partition holds it, so the same record always routes the same way.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 250_000 },
})
 
await narsil.insertBatch('logs', [ 
  { id: 'evt-90001', message: 'Checkout completed for order 4821', service: 'payments', level: 'info', timestamp: 1_720_051_200 },
  { id: 'evt-90002', message: 'Retrying charge after gateway timeout', service: 'payments', level: 'warn', timestamp: 1_720_051_260 },
  { id: 'evt-90003', message: 'Cart abandoned before payment', service: 'checkout', level: 'info', timestamp: 1_720_051_320 },
])

insertBatch returns partial results, so one malformed record never aborts the load. Every success still applies, and every failure comes back with its id and error.

Reshape the index online

As the corpus grows, four partitions may no longer spread the load well. rebalance(indexName, newPartitionCount) reshapes the index to a new partition count while it stays online. Writes arriving during the reshape buffer in a write-ahead queue and replay in order when it completes, and queries keep running throughout. Reshape logs from four partitions to eight.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 250_000 },
})
 
await narsil.insertBatch('logs', [
  { id: 'evt-90001', message: 'Checkout completed for order 4821', service: 'payments', level: 'info', timestamp: 1_720_051_200 },
  { id: 'evt-90002', message: 'Retrying charge after gateway timeout', service: 'payments', level: 'warn', timestamp: 1_720_051_260 },
  { id: 'evt-90003', message: 'Cart abandoned before payment', service: 'checkout', level: 'info', timestamp: 1_720_051_320 },
])
 
await narsil.rebalance('logs', 8)

A reshape is not free. While it runs, p95 latency can reach about 25ms, compared with about 11ms in steady state. Schedule reshapes during low-traffic windows, or pre-size the index so a mid-load reshape never becomes necessary. Inside one Node.js thread, going from 1 to 20 partitions costs about 14% of insert throughput and 27% of p95 search latency and gains nothing, so keep maxPartitions low for single-process deployments.

Widen each partition

Reshaping changes how many partitions hold the corpus. updatePartitionConfig(indexName, { maxDocsPerPartition }) changes how much each partition holds. Raise the per-partition ceiling so the eight partitions carry more records before they hit capacity.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 250_000 },
})
 
await narsil.insertBatch('logs', [
  { id: 'evt-90001', message: 'Checkout completed for order 4821', service: 'payments', level: 'info', timestamp: 1_720_051_200 },
  { id: 'evt-90002', message: 'Retrying charge after gateway timeout', service: 'payments', level: 'warn', timestamp: 1_720_051_260 },
  { id: 'evt-90003', message: 'Cart abandoned before payment', service: 'checkout', level: 'info', timestamp: 1_720_051_320 },
])
 
await narsil.rebalance('logs', 8)
 
await narsil.updatePartitionConfig('logs', { maxDocsPerPartition: 500_000 })

With eight partitions each holding 500,000 records, the index carries four million records before further inserts fail with PARTITION_CAPACITY_EXCEEDED.

Move search onto worker threads

Everything so far runs on the main thread. Search can move off it through worker threads on Node.js and Bun, or Web Workers in browsers and Deno. You enable this in createNarsil. Switch the engine at the top of logs.ts to start with a worker pool.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({ 
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 250_000 },
})
 
await narsil.insertBatch('logs', [
  { id: 'evt-90001', message: 'Checkout completed for order 4821', service: 'payments', level: 'info', timestamp: 1_720_051_200 },
  { id: 'evt-90002', message: 'Retrying charge after gateway timeout', service: 'payments', level: 'warn', timestamp: 1_720_051_260 },
  { id: 'evt-90003', message: 'Cart abandoned before payment', service: 'checkout', level: 'info', timestamp: 1_720_051_320 },
])
 
await narsil.rebalance('logs', 8)
 
await narsil.updatePartitionConfig('logs', { maxDocsPerPartition: 500_000 })

With workers enabled, the engine starts in direct mode and moves to the pool once any index passes promotionThreshold documents or all indexes together pass totalPromotionThreshold. The API remains identical before and after promotion. Promotion emits the workerPromote event, and a crashed worker emits workerCrash while the engine reassigns its indexes. Worker heap usage appears in getMemoryStats().

Several engine instances can share one persistence backend. When they do, the invalidation adapter publishes which partitions changed, so the others evict stale cache. Give the engine a filesystem persistence backend and a matching invalidation adapter targeting the same directory.

logs.ts

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
import { createFilesystemInvalidation } from '@delali/narsil/invalidation/filesystem'
 
const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
  persistence: createFilesystemPersistence({ directory: './narsil-data' }), 
  invalidation: createFilesystemInvalidation({ directory: './narsil-data', pollInterval: 1000 }),
})
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 250_000 },
})
 
await narsil.insertBatch('logs', [
  { id: 'evt-90001', message: 'Checkout completed for order 4821', service: 'payments', level: 'info', timestamp: 1_720_051_200 },
  { id: 'evt-90002', message: 'Retrying charge after gateway timeout', service: 'payments', level: 'warn', timestamp: 1_720_051_260 },
  { id: 'evt-90003', message: 'Cart abandoned before payment', service: 'checkout', level: 'info', timestamp: 1_720_051_320 },
])
 
await narsil.rebalance('logs', 8)
 
await narsil.updatePartitionConfig('logs', { maxDocsPerPartition: 500_000 })

The filesystem adapter coordinates processes on one machine through marker files. In the browser, createBroadcastChannelInvalidation() from @delali/narsil/invalidation/broadcast-channel coordinates tabs through a BroadcastChannel instead. The invalidation channel also carries partition statistics for the broadcast scoring mode. A custom adapter satisfies publish(event), subscribe(handler), and shutdown().

Report memory

Narsil reports memory at three levels. Each call inspects the running index rather than continuing the script, so run them on their own. The figures below are a snapshot of a populated four-partition logs index, so the counts and byte estimates reflect that index rather than a fixed number.

getStats covers one index and returns the document and partition counts, the estimated memory footprint, and the stored schema.

const stats = narsil.getStats('logs')
 
console.log(JSON.stringify(stats, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))

getPartitionStats breaks that estimate down per partition, so you can see which partition uses the most memory.

const perPartition = narsil.getPartitionStats('logs')
 
console.log(JSON.stringify(perPartition, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))

getMemoryStats returns a runtime snapshot. It reads live V8 heap usage, so it is async; await it. When the engine runs a worker pool, the workers array holds each worker's heap.

const runtime = await narsil.getMemoryStats()
 
console.log(JSON.stringify(runtime, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))

The estimates compare indexes inside one process. They exclude allocator and runtime overhead, so size host memory from process-level numbers instead.

Create a partitioned index

Load log records

Reshape the index online

Widen each partition

Move search onto worker threads

Share storage across instances

Report memory