> Complete content index: https://narsil.sondelali.com/llms.txt

# Partitions and workers

> Shard an index across partitions, reshape it online, move search onto worker threads, and coordinate instances that share storage.

URL: https://narsil.sondelali.com/docs/partitions-and-workers
Section: Operations
Version: 0.2 (@delali/narsil@0.2.2, latest)

Partitioning and workers are how one Narsil engine scales beyond a single thread. The engine routes each document to a partition deterministically, fans a search out across the partitions and merges the results, and runs that work on worker threads to keep the main thread free. This page builds one file, `logs.ts`, step by step. You create a partitioned index, load log records, ask for a warning before the partitions fill, reshape the index from the handler that receives that warning, widen each partition, move search onto worker threads, and share storage across instances. Each step highlights exactly what changed.

## Create a partitioned index

`partitions.maxPartitions` sets the partition count an index starts with, and it defaults to 1. The engine sends each document to a partition by hashing its id with <Tooltip tip="FNV-1a is a fast non-cryptographic hash function that turns a string into a number. Hashing each document id gives every document a stable, evenly spread partition assignment.">FNV-1a</Tooltip>, which keeps that choice identical across processes and languages. Start `logs.ts` with an engine and a `logs` index of four partitions.

```ts title="logs.ts"

const narsil = await createNarsil()

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000 },
})
```

Each partition holds up to `maxDocsPerPartition` documents. When the index reaches `maxDocsPerPartition * partitionCount` documents, further inserts fail with `PARTITION_CAPACITY_EXCEEDED`, so the four partitions above hold 100,000 records between them before you must reshape or widen them.

## Load log records

With the index in place, load a batch of log records. Each record carries its own `id`, and the hash of that id decides which partition holds it, so the same record always routes the same way. `RECORD_COUNT` is the one number to change if you want a smaller or larger load.

```ts title="logs.ts"

const narsil = await createNarsil()

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000 },
})

const RECORD_COUNT = 85_000 // [!code ++:23]

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

const loaded = await narsil.insertBatch('logs', records)

console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)
```

```txt result
loaded 85000 records, 0 failed
```

`insertBatch` returns partial results, so one malformed record never aborts the load. Every success still applies, and every failure comes back with its id and error, which is why the line above prints both counts.

The `id` field is optional, and leaving it out changes what you can do with the record later. For a record without one, the engine generates a UUID v7 (or calls the `idGenerator` you configured) and routes on that value, so the record still reaches a partition. What you give up is control of the key: you cannot work out which partition holds a given record, loading the same record twice stores it twice instead of failing with `DOC_ALREADY_EXISTS`, and `update` and `remove` need an id you kept from the insert. `insert` returns the resolved id and `insertBatch` lists the ids it resolved under `succeeded`. Pass your own id whenever the record has a natural key, as the event id does here.

## Get a warning before the partitions fill

Hitting `PARTITION_CAPACITY_EXCEEDED` means writes are already failing. The **watermark** fraction gives you warning ahead of that point: set it above 0 and at most 1, and the engine emits `partitionWatermark` once the index crosses `watermark * capacity` documents. Add an 80 per cent watermark and a handler that reports the crossing.

```ts title="logs.ts"

const narsil = await createNarsil()

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 }, // [!code ++]
})

narsil.on('partitionWatermark', (payload) => { // [!code ++:5]
  console.log(
    `${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents across ${payload.partitionCount} partitions`,
  )
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

const loaded = await narsil.insertBatch('logs', records)

console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)
```

```txt result
logs holds 85000 of 100000 documents across 4 partitions
loaded 85000 records, 0 failed
```

The report prints before the load summary because the engine runs the check inside `insertBatch`, before that call resolves. A single insert is checked the same way, so the warning always arrives on the write that crosses the line.

One crossing produces one event. The engine latches the event against the capacity it fired at and emits nothing further until a later check runs with the count back below the threshold, or with a larger capacity in place. The check also runs after every rebalance and every partition config change, so a handler that reshapes the index runs once per crossing and runs again only when the larger capacity still falls short.

## Reshape the index online

Reporting the crossing tells you the index is filling up. Reshaping it is what buys the room. `rebalance(indexName, newPartitionCount)` moves the index to a new partition count while it stays online. Writes arriving during the reshape buffer in a write-ahead queue and replay in order when it completes, and queries keep running throughout.

A reshape cannot exceed `maxPartitions`, so going from four partitions to eight takes two calls: raise the ceiling with `updatePartitionConfig`, then reshape into it. A target above the ceiling fails with `PARTITION_CAPACITY_EXCEEDED`. `growPartitions` below does both and reports the result, and the handler hands it double the partition count the index has now.

Everything the reshape needs belongs inside that one function, because a rebalance holds the index's partition config for as long as it runs. A second `updatePartitionConfig` call from anywhere else during that window fails with `PARTITION_REBALANCING_BACKPRESSURE`. Keeping the whole job in `growPartitions` means only one caller ever touches the config, and the handler fires and forgets.

The handler returns nothing, and the engine ignores whatever a handler returns, so the promise `growPartitions` produces needs its own `.catch` to report failures. The script itself never waits on the reshape: it finishes the load and ends, and the reshape keeps the process alive until it completes.

```ts title="logs.ts"

const narsil = await createNarsil()

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})

const growPartitions = async (indexName: string, partitionCount: number): Promise<void> => { // [!code ++:7]
  await narsil.updatePartitionConfig(indexName, { maxPartitions: partitionCount })
  await narsil.rebalance(indexName, partitionCount)
  const stats = narsil.getStats(indexName)
  console.log(`${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions`)
}

narsil.on('partitionWatermark', (payload) => {
  console.log( // [!code --:3]
    `${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents across ${payload.partitionCount} partitions`,
  )
  console.log(`${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents, making room`) // [!code ++:4]
  growPartitions(payload.indexName, payload.partitionCount * 2).catch((error: Error) => {
    console.error(`could not make room in ${payload.indexName}:`, error.message)
  })
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

const loaded = await narsil.insertBatch('logs', records)

console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)
```

```txt result
logs holds 85000 of 100000 documents, making room
loaded 85000 records, 0 failed
logs now holds 85000 documents across 8 partitions
```

The load summary prints in the middle because the reshape is still running when `insertBatch` resolves. The 85,000 records end up across eight partitions, and capacity doubles to 200,000, so the watermark does not fire again. The engine also emits `partitionRebalance` with the old and new counts, though it fires when the partition move finishes rather than when buffered writes have replayed, so treat it as progress rather than as the all-clear.

<Callout type="warning" title="A reshape costs latency while it runs">
  While a reshape runs, <Tooltip tip="p95 latency is the time under which 95 per cent of requests complete. It shows the slow tail of performance rather than the average.">p95</Tooltip> latency can reach about 25ms, against about 11ms in steady state. Schedule reshapes for low-traffic windows, or pre-size the index so that a mid-load reshape never becomes necessary. Inside one Node.js thread, going from 1 to 20 partitions costs about 14 per cent of insert throughput and 27 per cent of p95 search latency and gains nothing, so keep `maxPartitions` low for single-process deployments.
</Callout>

## Widen each partition

Reshaping changes how many partitions hold the corpus. `maxDocsPerPartition` changes how much each partition holds, and `updatePartitionConfig` takes both limits in one call. Raise the per-partition ceiling in the same call that raises the partition ceiling, so that making room grows the index in both directions at once.

```ts title="logs.ts"

const narsil = await createNarsil()

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})

const MAX_DOCS_PER_PARTITION = 50_000 // [!code ++:2]

const growPartitions = async (indexName: string, partitionCount: number): Promise<void> => {
  await narsil.updatePartitionConfig(indexName, { maxPartitions: partitionCount }) // [!code --]
  await narsil.updatePartitionConfig(indexName, { // [!code ++:4]
    maxPartitions: partitionCount,
    maxDocsPerPartition: MAX_DOCS_PER_PARTITION,
  })
  await narsil.rebalance(indexName, partitionCount)
  const stats = narsil.getStats(indexName)
  console.log(`${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions`) // [!code --]
  console.log( // [!code ++:3]
    `${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions, with room for ${stats.partitionCount * MAX_DOCS_PER_PARTITION}`,
  )
}

narsil.on('partitionWatermark', (payload) => {
  console.log(`${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents, making room`)
  growPartitions(payload.indexName, payload.partitionCount * 2).catch((error: Error) => {
    console.error(`could not make room in ${payload.indexName}:`, error.message)
  })
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

const loaded = await narsil.insertBatch('logs', records)

console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)
```

```txt result
logs holds 85000 of 100000 documents, making room
loaded 85000 records, 0 failed
logs now holds 85000 documents across 8 partitions, with room for 400000
```

Raising `maxDocsPerPartition` before the rebalance is safe because the engine checks each change against the index as it stands now, and 50,000 documents across the four partitions of the moment is already more room than the 85,000 records need.

`updatePartitionConfig` accepts `maxDocsPerPartition`, `maxPartitions`, and `watermark`. A value that is not a positive integer, or a watermark outside its range, fails with `CONFIG_INVALID`. A `maxPartitions` below the current partition count fails with `PARTITION_CAPACITY_EXCEEDED`, and so does a `maxDocsPerPartition` whose capacity across the current partitions falls below the documents already stored. The second rule is what rejects the change in `capacity-check.ts`, a file of its own that starts the index at the eight partitions the reshape produced and then tries to shrink each one to 10,000, because eight partitions of 10,000 leave nowhere for 5,000 of the records.

```ts title="capacity-check.ts"

const narsil = await createNarsil()

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

await narsil.insertBatch('logs', records)

try {
  await narsil.updatePartitionConfig('logs', { maxDocsPerPartition: 10_000 })
} catch (error) {
  if (error instanceof NarsilError) {
    console.log(`${error.code}: ${error.message}`)
  }
}
```

```txt result open
PARTITION_CAPACITY_EXCEEDED: New capacity (80000) is less than current document count (85000)
```

The engine writes every accepted limit into durability metadata, so a persisted index keeps them through recovery.

## Move search onto worker threads

Everything so far runs on the main thread. Search can move off it through worker threads on Node.js and Bun, or Web Workers in browsers and Deno. You enable this in `createNarsil`. Switch the engine at the top of `logs.ts` to start with a worker pool, and report the promotions that fail.

```ts title="logs.ts"

const narsil = await createNarsil() // [!code --]
const narsil = await createNarsil({ // [!code ++:8]
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})

narsil.on('workerPromoteFailure', (payload) => { // [!code ++:3]
  console.warn(`promotion failed (${payload.reason}, retryable: ${payload.retryable}):`, payload.error.message)
})

const MAX_DOCS_PER_PARTITION = 50_000

const growPartitions = async (indexName: string, partitionCount: number): Promise<void> => {
  await narsil.updatePartitionConfig(indexName, {
    maxPartitions: partitionCount,
    maxDocsPerPartition: MAX_DOCS_PER_PARTITION,
  })
  await narsil.rebalance(indexName, partitionCount)
  const stats = narsil.getStats(indexName)
  console.log(
    `${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions, with room for ${stats.partitionCount * MAX_DOCS_PER_PARTITION}`,
  )
}

narsil.on('partitionWatermark', (payload) => {
  console.log(`${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents, making room`)
  growPartitions(payload.indexName, payload.partitionCount * 2).catch((error: Error) => {
    console.error(`could not make room in ${payload.indexName}:`, error.message)
  })
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

const loaded = await narsil.insertBatch('logs', records)

console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)
```

The pool prints nothing of its own, so `logs.ts` reports exactly what it reported before. What changes is that the process now stays up after the reshape, because the worker threads keep it alive. Call `shutdown()` when your process is done with the engine, or stop the script with Ctrl-C.

With workers enabled, the engine starts in direct mode and moves to the pool once any index passes `promotionThreshold` documents or all indexes together pass `totalPromotionThreshold`. The API remains identical before and after promotion. A promoted index answers exactly as it did on the main thread, and `search-logs.ts` shows it: the same pool, the eight partitions the reshape produced, and the 21,250 records in four that carry the gateway timeout message.

```ts title="search-logs.ts"

const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

await narsil.insertBatch('logs', records)

const matches = await narsil.query('logs', { term: 'gateway' })

console.log(`${matches.count} matches for gateway`)

await narsil.shutdown()
```

```txt result open
21250 matches for gateway
```

Promotion emits the `workerPromote` event, and a crashed worker emits `workerCrash` while the engine reassigns its indexes. Worker heap usage appears in `getMemoryStats()`.

### Which indexes can promote

A worker receives an index's configuration as a copy, and a copy holds data rather than code, so an index must satisfy three conditions before it can run on a worker at all. A `tokenizer` given as an instance cannot cross the thread boundary. Neither can a `stopWords` function. A worker also starts with English alone and gains another language only after a module registers one inside it, so any other language needs `workers.bootstrapModule`, a module that every worker imports at startup.

That module is a normal file, and it registers whatever your indexes name. Write it once as `register-analysis.ts`.

```ts title="register-analysis.ts"

registerLanguage(french)

registerStopWords('log-noise', (defaults) => new Set([...defaults, 'trace', 'debug']))

registerTokenizer('service-paths', {
  tokenize: (text) =>
    text
      .toLowerCase()
      .split(/[\s/]+/)
      .map((token, position) => ({ token, position })),
})
```

A second file, `analysed-logs.ts`, names that module twice. It imports the module so that the main thread holds the registrations, and it passes the same module URL as `workers.bootstrapModule` so that every worker imports it at startup. A French index with a named tokeniser and a named stop word set then promotes like any English one.

```ts title="analysed-logs.ts"

import './register-analysis.ts'

const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 2,
    promotionThreshold: 2,
    bootstrapModule: new URL('./register-analysis.ts', import.meta.url).href,
  },
})

await narsil.createIndex('logs-fr', {
  schema: { message: 'string' },
  language: 'french',
  stopWords: 'log-noise',
  tokenizer: 'service-paths',
})

await narsil.insertBatch('logs-fr', [
  { id: 'evt-90101', message: 'Paiement accepté pour la commande 4821' },
  { id: 'evt-90102', message: 'Nouvelle tentative après expiration de la passerelle' },
  { id: 'evt-90103', message: 'Panier abandonné avant le paiement' },
])

const results = await narsil.query('logs-fr', { term: 'paiement' })

console.log(results.count)

await narsil.shutdown()
```

```ts result
2
```

The French analyser stems `Paiement` and `paiement` to the same root, so the query matches two of the three records. Because the registrations reach the workers, this index counts towards promotion like any English one, and three documents carry it past the threshold of two. Without the `bootstrapModule` line, the engine excludes the same index from the pool.

An excluded index keeps answering on the main thread while the eligible ones promote, and the engine reports it once through `workerPromoteFailure` with `retryable: false`. The same event covers a promotion attempt that fails outright. A deterministic cause, such as a bootstrap module that never registers the language an index names, blocks further attempts and reports `retryable: false`. A transient failure reports `retryable: true`, and the engine tries again at the next threshold check.

Registering tokenisers and stop word sets by name is what makes an index with custom analysis eligible, and [language support](/docs/language-support) covers both registries.

## Share storage across instances

Several engine instances can share one persistence backend. When they do, the invalidation adapter publishes which partitions changed, so the others reload instead of serving stale data. Give the engine a filesystem persistence backend, a matching invalidation adapter targeting the same directory, and the snapshot durability tier.

```ts title="logs.ts"

import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem' // [!code ++:2]

const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
  persistence: createFilesystemPersistence({ directory: './narsil-data' }), // [!code ++:3]
  invalidation: createFilesystemInvalidation({ directory: './narsil-data', pollInterval: 1000 }),
  durability: { tier: 'snapshot' },
})

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})

narsil.on('workerPromoteFailure', (payload) => {
  console.warn(`promotion failed (${payload.reason}, retryable: ${payload.retryable}):`, payload.error.message)
})

narsil.on('invalidationError', (payload) => { // [!code ++:3]
  console.error('invalidation failed:', payload.error)
})

const MAX_DOCS_PER_PARTITION = 50_000

const growPartitions = async (indexName: string, partitionCount: number): Promise<void> => {
  await narsil.updatePartitionConfig(indexName, {
    maxPartitions: partitionCount,
    maxDocsPerPartition: MAX_DOCS_PER_PARTITION,
  })
  await narsil.rebalance(indexName, partitionCount)
  const stats = narsil.getStats(indexName)
  console.log(
    `${indexName} now holds ${stats.documentCount} documents across ${stats.partitionCount} partitions, with room for ${stats.partitionCount * MAX_DOCS_PER_PARTITION}`,
  )
}

narsil.on('partitionWatermark', (payload) => {
  console.log(`${payload.indexName} holds ${payload.documentCount} of ${payload.capacity} documents, making room`)
  growPartitions(payload.indexName, payload.partitionCount * 2).catch((error: Error) => {
    console.error(`could not make room in ${payload.indexName}:`, error.message)
  })
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

const loaded = await narsil.insertBatch('logs', records)

console.log(`loaded ${loaded.succeeded.length} records, ${loaded.failed.length} failed`)
```

Persistence changes nothing that `logs.ts` prints, so a second file shows what it bought you. Leave `logs.ts` running and start `reader.ts`, which opens the same directory with the same three options and never inserts anything.

```ts title="reader.ts"

const reader = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  invalidation: createFilesystemInvalidation({ directory: './narsil-data', pollInterval: 1000 }),
  durability: { tier: 'snapshot' },
})

const stats = reader.getStats('logs')

console.log(`reader sees ${stats.documentCount} documents across ${stats.partitionCount} partitions`)

await reader.shutdown()
```

```txt result
reader sees 85000 documents across 8 partitions
```

The reader sees the reshaped eight-partition index because the rebalance checkpointed it, and it keeps polling for later changes. Both processes hold the directory at once, which is the arrangement the snapshot tier exists for. Delete `./narsil-data` before running `logs.ts` again, since `createIndex` on a directory that already holds the index fails with `INDEX_ALREADY_EXISTS`.

The `durability` line is what makes the rest of that config legal. A filesystem persistence adapter runs the write-ahead-log tier by default, and a write-ahead log requires exclusive use of its directory, so it leaves the other instances nothing to read. `tier: 'snapshot'` moves the same adapter onto snapshot persistence, which every instance can read. Without the line, `createNarsil` rejects the combination with `CONFIG_INVALID`. [Persistence and durability](/docs/persistence-and-durability) covers both tiers and what each costs on a crash.

Invalidation failures never surface on the call that triggered them, which is why the `invalidationError` handler is part of the config rather than an extra. The filesystem adapter coordinates processes on one machine through marker files. In the browser, `createBroadcastChannelInvalidation()` from `@delali/narsil/invalidation/broadcast-channel` coordinates tabs through a BroadcastChannel instead. The invalidation channel also carries partition statistics for the `broadcast` scoring mode. A custom adapter satisfies `publish(event)`, `subscribe(handler)`, and `shutdown()`.

## Report memory

Narsil reports memory at three levels. These calls inspect the index rather than continue `logs.ts`, so each one gets a file of its own. All three run the same pool, start the index at the eight partitions the reshape produced, and hold off on reporting until the pool has taken the index, because a promotion still in flight leaves the worker figures empty.

`getStats` covers one index and returns the document and partition counts, the estimated memory footprint, and the stored schema.

```ts title="index-stats.ts"

const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})

const promoted = new Promise<void>((resolve) => {
  narsil.on('workerPromote', () => resolve())
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

await narsil.insertBatch('logs', records)

await promoted

const indexStats = narsil.getStats('logs')

console.log(JSON.stringify(indexStats, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))

await narsil.shutdown()
```

```json result
{
  "documentCount": 85000,
  "partitionCount": 8,
  "estimatedMemoryBytes": 65978720,
  "language": "english",
  "schema": {
    "message": "string",
    "service": "enum",
    "level": "enum",
    "timestamp": "number"
  }
}
```

`getPartitionStats` breaks that estimate down per partition so that you can see which partition uses the most memory. The busiest partition holds six documents more than the quietest, which is how evenly the hash spread 85,000 ids across eight partitions.

```ts title="partition-stats.ts"

const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})

const promoted = new Promise<void>((resolve) => {
  narsil.on('workerPromote', () => resolve())
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

await narsil.insertBatch('logs', records)

await promoted

const perPartition = narsil.getPartitionStats('logs')

console.log(JSON.stringify(perPartition, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))

await narsil.shutdown()
```

```json result
[
  {
    "partitionId": 0,
    "documentCount": 10625,
    "estimatedMemoryBytes": 8247364
  },
  {
    "partitionId": 1,
    "documentCount": 10622,
    "estimatedMemoryBytes": 8244988
  },
  {
    "partitionId": 2,
    "documentCount": 10628,
    "estimatedMemoryBytes": 8249572
  },
  {
    "partitionId": 3,
    "documentCount": 10625,
    "estimatedMemoryBytes": 8247436
  },
  {
    "partitionId": 4,
    "documentCount": 10625,
    "estimatedMemoryBytes": 8247436
  },
  {
    "partitionId": 5,
    "documentCount": 10628,
    "estimatedMemoryBytes": 8249572
  },
  {
    "partitionId": 6,
    "documentCount": 10622,
    "estimatedMemoryBytes": 8244988
  },
  {
    "partitionId": 7,
    "documentCount": 10625,
    "estimatedMemoryBytes": 8247364
  }
]
```

`getMemoryStats` returns a runtime snapshot. It reads live V8 heap usage, so it is async; await it. When the engine runs a worker pool, the `workers` array holds each worker's heap. These figures move between runs, between machines, and with how recently V8 collected, so treat the numbers below as one sample rather than a constant.

```ts title="memory-stats.ts"

const narsil = await createNarsil({
  workers: {
    enabled: true,
    count: 4,
    promotionThreshold: 10_000,
    totalPromotionThreshold: 50_000,
  },
})

await narsil.createIndex('logs', {
  schema: {
    message: 'string',
    service: 'enum',
    level: 'enum',
    timestamp: 'number',
  },
  partitions: { maxPartitions: 8, maxDocsPerPartition: 50_000 },
})

const promoted = new Promise<void>((resolve) => {
  narsil.on('workerPromote', () => resolve())
})

const RECORD_COUNT = 85_000

const EVENT_TYPES = [
  { service: 'payments', level: 'info', message: 'Checkout completed for order' },
  { service: 'payments', level: 'warn', message: 'Gateway timed out, retrying charge for order' },
  { service: 'checkout', level: 'info', message: 'Cart abandoned before payment for order' },
  { service: 'shipping', level: 'error', message: 'Address validation failed for order' },
]

const records = Array.from({ length: RECORD_COUNT }, (_, index) => {
  const eventType = EVENT_TYPES[index % EVENT_TYPES.length]
  return {
    id: `evt-${90_001 + index}`,
    message: `${eventType.message} ${4_821 + index}`,
    service: eventType.service,
    level: eventType.level,
    timestamp: 1_720_051_200 + index * 60,
  }
})

await narsil.insertBatch('logs', records)

await promoted

const runtime = await narsil.getMemoryStats()

console.log(JSON.stringify(runtime, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))

await narsil.shutdown()
```

```json result
{
  "process": {
    "heapUsed": 264025136,
    "heapTotal": 440958976,
    "external": 4157860,
    "rss": 2055651328
  },
  "estimatedIndexBytes": 65978720,
  "workers": [
    {
      "workerId": 0,
      "heapUsed": 290320440,
      "heapTotal": 410075136,
      "external": 4925803
    },
    {
      "workerId": 1,
      "heapUsed": 324561608,
      "heapTotal": 408649728,
      "external": 4925803
    },
    {
      "workerId": 2,
      "heapUsed": 295956280,
      "heapTotal": 413237248,
      "external": 4925803
    },
    {
      "workerId": 3,
      "heapUsed": 296301272,
      "heapTotal": 413761536,
      "external": 4925803
    }
  ]
}
```

Compare the two kinds of number. `estimatedIndexBytes` puts the index structures at 66MB, while each worker reports a V8 heap in the hundreds of megabytes, because a worker holds its own copy of the index on top of everything V8 allocated during the load and has not yet collected. The estimates compare indexes inside one process. They exclude allocator and runtime overhead, so size host memory from process-level numbers instead.