Partitions and workers

Partitioning and workers are how one Narsil engine scales beyond a single thread. The engine routes each document to a partition deterministically, fans a search out across the partitions and merges the results, and runs that work on worker threads to keep the main thread free. This page builds one file, logs.ts, step by step. You create a partitioned index, load log records, ask for a warning before the partitions fill, reshape the index from the handler that receives that warning, widen each partition, move search onto worker threads, and share storage across instances. Each step highlights exactly what changed.

Create a partitioned index

partitions.maxPartitions sets the partition count an index starts with, and it defaults to 1. The engine sends each document to a partition by hashing its id with , which keeps that choice identical across processes and languages. Start logs.ts with an engine and a logs index of four partitions.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000 },
})

Each partition holds up to maxDocsPerPartition documents. When the index reaches maxDocsPerPartition * partitionCount documents, further inserts fail with PARTITION_CAPACITY_EXCEEDED, so the four partitions above hold 100,000 records between them before you must reshape or widen them.

Load log records

With the index in place, load a batch of log records. Each record carries its own id, and the hash of that id decides which partition holds it, so the same record always routes the same way. RECORD_COUNT is the one number to change if you want a smaller or larger load.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000 },
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
const loaded = await narsil.insertBatch('logs', records)
 
console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)

insertBatch returns partial results, so one malformed record never aborts the load. Every success still applies, and every failure comes back with its id and error, which is why the line above prints both counts.

The id field is optional, and leaving it out changes what you can do with the record later. For a record without one, the engine generates a UUID v7 (or calls the idGenerator you configured) and routes on that value, so the record still reaches a partition. What you give up is control of the key: you cannot work out which partition holds a given record, loading the same record twice stores it twice instead of failing with DOC_ALREADY_EXISTS, and update and remove need an id you kept from the insert. insert returns the resolved id and insertBatch lists the ids it resolved under succeeded. Pass your own id whenever the record has a natural key, as the event id does here.

Get a warning before the partitions fill

Hitting PARTITION_CAPACITY_EXCEEDED means writes are already failing. The watermark fraction gives you warning ahead of that point: set it above 0 and at most 1, and the engine emits partitionWatermark once the index crosses watermark * capacity documents. Add an 80 per cent watermark and a handler that reports the crossing.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 }, 
})
 
narsil.on('partitionWatermark', (payload) => { 
  console.log(
    `${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents across ${payload.partitionCount} partitions`,
  )
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
const loaded = await narsil.insertBatch('logs', records)
 
console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)

The report prints before the load summary because the engine runs the check inside insertBatch, before that call resolves. A single insert is checked the same way, so the warning always arrives on the write that crosses the line.

One crossing produces one event. The engine latches the event against the capacity it fired at and emits nothing further until a later check runs with the count back below the threshold, or with a larger capacity in place. The check also runs after every rebalance and every partition config change, so a handler that reshapes the index runs once per crossing and runs again only when the larger capacity still falls short.

Reshape the index online

Reporting the crossing tells you the index is filling up. Reshaping it is what buys the room. rebalance(indexName, newPartitionCount) moves the index to a new partition count while it stays online. Writes arriving during the reshape buffer in a write-ahead queue and replay in order when it completes, and queries keep running throughout.

A reshape cannot exceed maxPartitions, so going from four partitions to eight takes two calls: raise the ceiling with updatePartitionConfig, then reshape into it. A target above the ceiling fails with PARTITION_CAPACITY_EXCEEDED. growPartitions below does both and reports the result, and the handler hands it double the partition count the index has now.

Everything the reshape needs belongs inside that one function, because a rebalance holds the index's partition config for as long as it runs. A second updatePartitionConfig call from anywhere else during that window fails with PARTITION_REBALANCING_BACKPRESSURE. Keeping the whole job in growPartitions means only one caller ever touches the config, and the handler fires and forgets.

The handler returns nothing, and the engine ignores whatever a handler returns, so the promise growPartitions produces needs its own .catch to report failures. The script itself never waits on the reshape: it finishes the load and ends, and the reshape keeps the process alive until it completes.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})
 
const growPartitions = async (indexName: string, partitionCount: number): Promise<void> => { 
  await narsil.updatePartitionConfig(indexName, { maxPartitions: partitionCount })
  await narsil.rebalance(indexName, partitionCount)
  const stats = narsil.getStats(indexName)
  console.log(`${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions`)
}
 
narsil.on('partitionWatermark', (payload) => {
  console.log( 
    `${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents across ${payload.partitionCount} partitions`,
  )
  console.log(`${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents, making room`) 
  growPartitions(payload.indexName, payload.partitionCount * 2).catch((error: Error) => {
    console.error(`could not make room in ${payload.indexName}:`, error.message)
  })
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
const loaded = await narsil.insertBatch('logs', records)
 
console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)

The load summary prints in the middle because the reshape is still running when insertBatch resolves. The 85,000 records end up across eight partitions, and capacity doubles to 200,000, so the watermark does not fire again. The engine also emits partitionRebalance with the old and new counts, though it fires when the partition move finishes rather than when buffered writes have replayed, so treat it as progress rather than as the all-clear.

A reshape costs latency while it runs

While a reshape runs, latency can reach about 25ms, against about 11ms in steady state. Schedule reshapes for low-traffic windows, or pre-size the index so that a mid-load reshape never becomes necessary. Inside one Node.js thread, going from 1 to 20 partitions costs about 14 per cent of insert throughput and 27 per cent of p95 search latency and gains nothing, so keep maxPartitions low for single-process deployments.

Widen each partition

Reshaping changes how many partitions hold the corpus. maxDocsPerPartition changes how much each partition holds, and updatePartitionConfig takes both limits in one call. Raise the per-partition ceiling in the same call that raises the partition ceiling, so that making room grows the index in both directions at once.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})
 
const MAX_DOCS_PER_PARTITION = 50_000
 
const growPartitions = async (indexName: string, partitionCount: number): Promise<void> => {
  await narsil.updatePartitionConfig(indexName, { maxPartitions: partitionCount }) 
  await narsil.updatePartitionConfig(indexName, { 
    maxPartitions: partitionCount,
    maxDocsPerPartition: MAX_DOCS_PER_PARTITION,
  })
  await narsil.rebalance(indexName, partitionCount)
  const stats = narsil.getStats(indexName)
  console.log(`${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions`) 
  console.log( 
    `${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions, with room for ${stats.partitionCount * MAX_DOCS_PER_PARTITION}`,
  )
}
 
narsil.on('partitionWatermark', (payload) => {
  console.log(`${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents, making room`)
  growPartitions(payload.indexName, payload.partitionCount * 2).catch((error: Error) => {
    console.error(`could not make room in ${payload.indexName}:`, error.message)
  })
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
const loaded = await narsil.insertBatch('logs', records)
 
console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)

Raising maxDocsPerPartition before the rebalance is safe because the engine checks each change against the index as it stands now, and 50,000 documents across the four partitions of the moment is already more room than the 85,000 records need.

updatePartitionConfig accepts maxDocsPerPartition, maxPartitions, and watermark. A value that is not a positive integer, or a watermark outside its range, fails with CONFIG_INVALID. A maxPartitions below the current partition count fails with PARTITION_CAPACITY_EXCEEDED, and so does a maxDocsPerPartition whose capacity across the current partitions falls below the documents already stored. The second rule is what rejects the change in capacity-check.ts, a file of its own that starts the index at the eight partitions the reshape produced and then tries to shrink each one to 10,000, because eight partitions of 10,000 leave nowhere for 5,000 of the records.

capacity-check.ts

import { createNarsil, NarsilError } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
await narsil.insertBatch('logs', records)
 
try {
  await narsil.updatePartitionConfig('logs', { maxDocsPerPartition: 10_000 })
} catch (error) {
  if (error instanceof NarsilError) {
    console.log(`${error.code}: ${error.message}`)
  }
}

PARTITION_CAPACITY_EXCEEDED: New capacity (80000) is less than current document count (85000)

The engine writes every accepted limit into durability metadata, so a persisted index keeps them through recovery.

Move search onto worker threads

Everything so far runs on the main thread. Search can move off it through worker threads on Node.js and Bun, or Web Workers in browsers and Deno. You enable this in createNarsil. Switch the engine at the top of logs.ts to start with a worker pool, and report the promotions that fail.

logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil() 
const narsil = await createNarsil({ 
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})
 
narsil.on('workerPromoteFailure', (payload) => { 
  console.warn(`promotion failed (${payload.reason}, retryable: ${payload.retryable}):`, payload.error.message)
})
 
const MAX_DOCS_PER_PARTITION = 50_000
 
const growPartitions = async (indexName: string, partitionCount: number): Promise<void> => {
  await narsil.updatePartitionConfig(indexName, {
    maxPartitions: partitionCount,
    maxDocsPerPartition: MAX_DOCS_PER_PARTITION,
  })
  await narsil.rebalance(indexName, partitionCount)
  const stats = narsil.getStats(indexName)
  console.log(
    `${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions, with room for ${stats.partitionCount * MAX_DOCS_PER_PARTITION}`,
  )
}
 
narsil.on('partitionWatermark', (payload) => {
  console.log(`${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents, making room`)
  growPartitions(payload.indexName, payload.partitionCount * 2).catch((error: Error) => {
    console.error(`could not make room in ${payload.indexName}:`, error.message)
  })
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
const loaded = await narsil.insertBatch('logs', records)
 
console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)

The pool prints nothing of its own, so logs.ts reports exactly what it reported before. What changes is that the process now stays up after the reshape, because the worker threads keep it alive. Call shutdown() when your process is done with the engine, or stop the script with Ctrl-C.

With workers enabled, the engine starts in direct mode and moves to the pool once any index passes promotionThreshold documents or all indexes together pass totalPromotionThreshold. The API remains identical before and after promotion. A promoted index answers exactly as it did on the main thread, and search-logs.ts shows it: the same pool, the eight partitions the reshape produced, and the 21,250 records in four that carry the gateway timeout message.

search-logs.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
await narsil.insertBatch('logs', records)
 
const matches = await narsil.query('logs', { term: 'gateway' })
 
console.log(`${matches.count} matches for gateway`)
 
await narsil.shutdown()

21250 matches for gateway

Promotion emits the workerPromote event, and a crashed worker emits workerCrash while the engine reassigns its indexes. Worker heap usage appears in getMemoryStats().

Which indexes can promote

A worker receives an index's configuration as a copy, and a copy holds data rather than code, so an index must satisfy three conditions before it can run on a worker at all. A tokenizer given as an instance cannot cross the thread boundary. Neither can a stopWords function. A worker also starts with English alone and gains another language only after a module registers one inside it, so any other language needs workers.bootstrapModule, a module that every worker imports at startup.

That module is a normal file, and it registers whatever your indexes name. Write it once as register-analysis.ts.

import { registerLanguage, registerStopWords, registerTokenizer } from '@delali/narsil'
import { french } from '@delali/narsil/languages/french'
 
registerLanguage(french)
 
registerStopWords('log-noise', (defaults) => new Set([...defaults, 'trace', 'debug']))
 
registerTokenizer('service-paths', {
  tokenize: (text) =>
    text
      .toLowerCase()
      .split(/[\s/]+/)
      .map((token, position) => ({ token, position })),
})

A second file, analysed-logs.ts, names that module twice. It imports the module so that the main thread holds the registrations, and it passes the same module URL as workers.bootstrapModule so that every worker imports it at startup. A French index with a named tokeniser and a named stop word set then promotes like any English one.

analysed-logs.ts

import { createNarsil } from '@delali/narsil'
import './register-analysis.ts'
 
const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 2,
    promotionThreshold: 2,
    bootstrapModule: new URL('./register-analysis.ts', import.meta.url).href,
  },
})
 
await narsil.createIndex('logs-fr', {
  schema: { message: 'string' },
  language: 'french',
  stopWords: 'log-noise',
  tokenizer: 'service-paths',
})
 
await narsil.insertBatch('logs-fr', [
  { id: 'evt-90101', message: 'Paiement accepté pour la commande 4821' },
  { id: 'evt-90102', message: 'Nouvelle tentative après expiration de la passerelle' },
  { id: 'evt-90103', message: 'Panier abandonné avant le paiement' },
])
 
const results = await narsil.query('logs-fr', { term: 'paiement' })
 
console.log(results.count)
 
await narsil.shutdown()

The French analyser stems Paiement and paiement to the same root, so the query matches two of the three records. Because the registrations reach the workers, this index counts towards promotion like any English one, and three documents carry it past the threshold of two. Without the bootstrapModule line, the engine excludes the same index from the pool.

An excluded index keeps answering on the main thread while the eligible ones promote, and the engine reports it once through workerPromoteFailure with retryable: false. The same event covers a promotion attempt that fails outright. A deterministic cause, such as a bootstrap module that never registers the language an index names, blocks further attempts and reports retryable: false. A transient failure reports retryable: true, and the engine tries again at the next threshold check.

Registering tokenisers and stop word sets by name is what makes an index with custom analysis eligible, and language support covers both registries.

Several engine instances can share one persistence backend. When they do, the invalidation adapter publishes which partitions changed, so the others reload instead of serving stale data. Give the engine a filesystem persistence backend, a matching invalidation adapter targeting the same directory, and the snapshot durability tier.

logs.ts

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
import { createFilesystemInvalidation } from '@delali/narsil/invalidation/filesystem'
 
const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
  persistence: createFilesystemPersistence({ directory: './narsil-data' }), 
  invalidation: createFilesystemInvalidation({ directory: './narsil-data', pollInterval: 1000 }),
  durability: { tier: 'snapshot' },
})
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})
 
narsil.on('workerPromoteFailure', (payload) => {
  console.warn(`promotion failed (${payload.reason}, retryable: ${payload.retryable}):`, payload.error.message)
})
 
narsil.on('invalidationError', (payload) => { 
  console.error('invalidation failed:', payload.error)
})
 
const MAX_DOCS_PER_PARTITION = 50_000
 
const growPartitions = async (indexName: string, partitionCount: number): Promise<void> => {
  await narsil.updatePartitionConfig(indexName, {
    maxPartitions: partitionCount,
    maxDocsPerPartition: MAX_DOCS_PER_PARTITION,
  })
  await narsil.rebalance(indexName, partitionCount)
  const stats = narsil.getStats(indexName)
  console.log(
    `${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions, with room for ${stats.partitionCount * MAX_DOCS_PER_PARTITION}`,
  )
}
 
narsil.on('partitionWatermark', (payload) => {
  console.log(`${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents, making room`)
  growPartitions(payload.indexName, payload.partitionCount * 2).catch((error: Error) => {
    console.error(`could not make room in ${payload.indexName}:`, error.message)
  })
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
const loaded = await narsil.insertBatch('logs', records)
 
console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)

Persistence changes nothing that logs.ts prints, so a second file shows what it bought you. Leave logs.ts running and start reader.ts, which opens the same directory with the same three options and never inserts anything.

reader.ts

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
import { createFilesystemInvalidation } from '@delali/narsil/invalidation/filesystem'
 
const reader = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  invalidation: createFilesystemInvalidation({ directory: './narsil-data', pollInterval: 1000 }),
  durability: { tier: 'snapshot' },
})
 
const stats = reader.getStats('logs')
 
console.log(`reader sees ${stats.documentCount} documents across ${stats.partitionCount} partitions`)
 
await reader.shutdown()

The reader sees the reshaped eight-partition index because the rebalance checkpointed it, and it keeps polling for later changes. Both processes hold the directory at once, which is the arrangement the snapshot tier exists for. Delete ./narsil-data before running logs.ts again, since createIndex on a directory that already holds the index fails with INDEX_ALREADY_EXISTS.

The durability line is what makes the rest of that config legal. A filesystem persistence adapter runs the write-ahead-log tier by default, and a write-ahead log requires exclusive use of its directory, so it leaves the other instances nothing to read. tier: 'snapshot' moves the same adapter onto snapshot persistence, which every instance can read. Without the line, createNarsil rejects the combination with CONFIG_INVALID. Persistence and durability covers both tiers and what each costs on a crash.

Invalidation failures never surface on the call that triggered them, which is why the invalidationError handler is part of the config rather than an extra. The filesystem adapter coordinates processes on one machine through marker files. In the browser, createBroadcastChannelInvalidation() from @delali/narsil/invalidation/broadcast-channel coordinates tabs through a BroadcastChannel instead. The invalidation channel also carries partition statistics for the broadcast scoring mode. A custom adapter satisfies publish(event), subscribe(handler), and shutdown().

Report memory

Narsil reports memory at three levels. These calls inspect the index rather than continue logs.ts, so each one gets a file of its own. All three run the same pool, start the index at the eight partitions the reshape produced, and hold off on reporting until the pool has taken the index, because a promotion still in flight leaves the worker figures empty.

getStats covers one index and returns the document and partition counts, the estimated memory footprint, and the stored schema.

index-stats.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})
 
const promoted = new Promise<void>((resolve) => {
  narsil.on('workerPromote', () => resolve())
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
await narsil.insertBatch('logs', records)
 
await promoted
 
const indexStats = narsil.getStats('logs')
 
console.log(JSON.stringify(indexStats, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))
 
await narsil.shutdown()

getPartitionStats breaks that estimate down per partition so that you can see which partition uses the most memory. The busiest partition holds six documents more than the quietest, which is how evenly the hash spread 85,000 ids across eight partitions.

partition-stats.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})
 
const promoted = new Promise<void>((resolve) => {
  narsil.on('workerPromote', () => resolve())
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
await narsil.insertBatch('logs', records)
 
await promoted
 
const perPartition = narsil.getPartitionStats('logs')
 
console.log(JSON.stringify(perPartition, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))
 
await narsil.shutdown()

getMemoryStats returns a runtime snapshot. It reads live V8 heap usage, so it is async; await it. When the engine runs a worker pool, the workers array holds each worker's heap. These figures move between runs, between machines, and with how recently V8 collected, so treat the numbers below as one sample rather than a constant.

memory-stats.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})
 
await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})
 
const promoted = new Promise<void>((resolve) => {
  narsil.on('workerPromote', () => resolve())
})
 
const RECORD_COUNT = 85_000
 
const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]
 
const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})
 
await narsil.insertBatch('logs', records)
 
await promoted
 
const runtime = await narsil.getMemoryStats()
 
console.log(JSON.stringify(runtime, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))
 
await narsil.shutdown()

Compare the two kinds of number. estimatedIndexBytes puts the index structures at 66MB, while each worker reports a V8 heap in the hundreds of megabytes, because a worker holds its own copy of the index on top of everything V8 allocated during the load and has not yet collected. The estimates compare indexes inside one process. They exclude allocator and runtime overhead, so size host memory from process-level numbers instead.