> Complete content index: https://narsil.sondelali.com/llms.txt

# Persistence and durability

> Persist indexes through pluggable adapters, close the crash window with a write-ahead log, and move whole indexes as portable snapshots.

URL: https://narsil.sondelali.com/docs/persistence-and-durability
Section: Operations
Version: 0.2 (@delali/narsil@0.2.2, latest)

Narsil writes indexes to storage through a pluggable persistence adapter, and the adapter you choose sets how durable each write is. A filesystem adapter runs the write-ahead-log tier, where a mutation reaches disk before the call that made it resolves. Every other adapter runs the snapshot tier, where the engine writes the whole index on a checkpoint. This page builds one file, `store.ts`, step by step. You start with an in-memory engine, add filesystem persistence, tune the log and the checkpoint schedule, and force a checkpoint by hand. Each step highlights exactly what changed.

## Start with a plain engine

Begin `store.ts` with an engine and a `products` index that holds a small catalogue. Nothing reaches disk yet, so the whole index remains in memory and disappears when the process exits.

```ts title="store.ts"

const narsil = await createNarsil()

await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})

await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})
```

## Add filesystem persistence

The package includes three adapters:

| Adapter | Import | Environment |
| --- | --- | --- |
| Memory | `@delali/narsil/adapters/memory` | Works everywhere and suits tests. |
| Filesystem | `@delali/narsil/adapters/filesystem` | Runs on Node.js, Bun, and Deno. |
| IndexedDB | `@delali/narsil/adapters/indexeddb` | Runs in browsers. |

Give the engine a filesystem adapter. That single line turns on the write-ahead-log tier: every insert, update, and removal appends to a log in `./narsil-data` before the call resolves, and `createNarsil` replays that log before it hands you the engine. The `products` index now survives a restart with every acknowledged write intact.

```ts title="store.ts"

import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem' // [!code ++]

const narsil = await createNarsil() // [!code --]
const narsil = await createNarsil({ // [!code ++:3]
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
})

await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})

await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})
```

A custom backend satisfies the `PersistenceAdapter` interface: `save`, `load`, `delete`, and `list`, all returning promises. The serialisation format is `.nrsl`, a 32-byte header followed by a <Tooltip tip="MessagePack is a compact binary serialisation format with the same data model as JSON, so structured data encodes smaller and parses faster.">MessagePack</Tooltip> payload. The format is cross-language portable and specified in the repository's [spec package](https://github.com/assetcorp/narsil/blob/main/packages/spec/envelope.md), so another language implementation can read and write the same files.

## How much a crash can cost you

Each adapter belongs to one of two tiers, and the tier sets how much a hard crash costs you.

| Storage | Tier | What survives a crash |
| --- | --- | --- |
| A filesystem adapter, or `durability.directory` | Write-ahead log | Every write your caller saw succeed. |
| Memory, IndexedDB, or any other adapter | Snapshot | Everything up to the last checkpoint. |

On the write-ahead-log tier the log holds each mutation, checkpoints capture the index state so that recovery has a shorter log to replay, and recovery replays the log over the newest checkpoint.

On the snapshot tier there is no log. The engine writes the whole index through the adapter on a checkpoint trigger, either the `durability.checkpointIntervalMs` timer (five minutes by default) or `durability.checkpointMutationThreshold` mutations (100,000 by default), whichever comes first. Writes made after the last checkpoint are lost on a crash, and `shutdown()` does not checkpoint on its way out, so call `checkpoint(indexName)` before a planned stop.

Set `durability.tier` to pick a tier yourself. `'snapshot'` puts any adapter on the snapshot tier, a filesystem one included, which is what several processes sharing one directory need, because a write-ahead log requires exclusive use of its directory. `'wal'` requires a directory, either from `durability.directory` or from a filesystem adapter.

## Tune the log and the checkpoint schedule

Add a `durability` block to choose the acknowledgement contract and the checkpoint schedule. The block needs no `directory` of its own here, because the filesystem adapter already names one.

```ts title="store.ts"

const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  durability: { // [!code ++:5]
    mode: 'sync',
    checkpointIntervalMs: 300_000,
    checkpointMutationThreshold: 100_000,
  },
})

await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})

await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})
```

In `sync` mode (the default) a write is not acknowledged until it is on disk, so a crash never loses a write that already returned success to your caller. In `async` mode writes acknowledge immediately while the log flushes every `flushIntervalMs` (one second by default), which is faster but can lose the final interval on a hard crash. Two more knobs shape the log itself: `segmentMaxBytes` rolls a new segment at 64 MiB, and `compactionThreshold` compacts after twelve checkpoint segments.

`createNarsil` checks every durability value before it starts anything, and an invalid one fails with `CONFIG_INVALID`:

- The engine rejects a `tier` other than `'wal'` or `'snapshot'`, and a `mode` other than `'sync'` or `'async'`.
- It rejects a number that is not finite, a negative interval, and a size or threshold below 1. An interval of `0` is valid and turns that timer off.
- `'snapshot'` needs a persistence adapter to write through, and `'wal'` needs a directory it can resolve.
- `'snapshot'` rejects every write-ahead-log field, meaning `directory`, `mode`, `flushIntervalMs`, `segmentMaxBytes`, and `compactionThreshold`, because none of them applies to a tier with no log.

## Force a checkpoint

Checkpoints run on the schedule above, and `checkpoint(indexName)` runs one immediately. Call it before a planned shutdown or a backup.

```ts title="store.ts"

const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  durability: {
    mode: 'sync',
    checkpointIntervalMs: 300_000,
    checkpointMutationThreshold: 100_000,
  },
})

await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})

await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

await narsil.checkpoint('products') // [!code ++]
```

The call resolves once the checkpoint reaches storage. With no durability configured it does nothing, so the same code is safe whether or not the engine has an adapter.

## Share one directory between processes

A write-ahead log requires exclusive use of its directory, so two engines pointed at the same one would corrupt each other's log. Force the snapshot tier when several processes read and write the same storage. This is a separate program, `shared-store.ts`, holding the same catalogue in a directory of its own with the tier pinned. Every process that shares that directory runs this configuration.

```ts title="shared-store.ts"

const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-shared' }),
  durability: { tier: 'snapshot', checkpointIntervalMs: 30_000 },
})

await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})

await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

await narsil.checkpoint('products')
```

An invalidation adapter then reports each instance's writes to the others, so none of them serves stale data. [Partitions and workers](/docs/partitions-and-workers) covers that side, and it uses the same `tier: 'snapshot'` line, because `createNarsil` rejects an invalidation adapter on any other tier with `CONFIG_INVALID`.

## What recovery restores

`createNarsil` runs recovery before it resolves, so the first call after start-up runs against the recovered state. Recovery brings back each index's documents and its full configuration: the partition limits (`maxDocsPerPartition`, `maxPartitions`, and `watermark`), the scoring default, position tracking, strictness, the required fields, the vector promotion settings, the stop words and tokeniser, and the named embedding adapter bindings.

Recovery also checks each index's analysis. An index records the revision of the language module that built its terms, and recovery compares that against the revision the module carries now. A difference marks the index stale, which is what upgrading `@delali/narsil` past a release that corrects a stemmer or a stop word list produces. The index keeps answering, its results carry `analysisStale: true`, and by default the engine rebuilds its terms in the background from the documents already stored, then writes the new revision and takes a checkpoint. [Language support](/docs/language-support#keep-an-index-current-when-a-language-changes) covers the `analysis` config and `rebuildAnalysis`.

A checkpoint holds data, never code, so custom analysis persists under a name instead. Register a tokeniser with `registerTokenizer` or a stop word set with `registerStopWords`, then name it in the index config; recovery reads the name from the metadata and rebinds the implementation from the registry. Register the names before you call `createNarsil`, because recovery runs inside that call. An index config that hands a durable engine a tokeniser instance or a stop word function fails with `CONFIG_INVALID` at `createIndex`, which reports the problem while you write the code rather than at the next recovery. [Language support](/docs/language-support) shows both registries.

## Report background write failures

Checkpoints and log appends run in the background, so a failed one never surfaces on the call that triggered it. The `durabilityError` event is the engine's channel for those failures. Register the handler once at start-up in any deployment that persists data, which in `store.ts` means the line right after the engine exists.

```ts title="store.ts"

const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  durability: {
    mode: 'sync',
    checkpointIntervalMs: 300_000,
    checkpointMutationThreshold: 100_000,
  },
})

narsil.on('durabilityError', (payload) => { // [!code ++:3]
  console.error('durability write failed:', payload.error)
})

await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})

await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

await narsil.checkpoint('products')
```

## Snapshots and restore

`snapshot(indexName)` serialises a whole index, including its documents, schema, and vector data, into one portable byte array. `restore(indexName, data)` rebuilds an index from those bytes, replacing the index if it already exists. This is a separate maintenance script, `backup.ts`, and it reads what `store.ts` already wrote, so run `store.ts` first. It opens the same persisted engine so that recovery restores the `products` index, then writes that index to a file and reads it back.

```ts title="backup.ts"

const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
})

const bytes = await narsil.snapshot('products')
await writeFile('./products.nrsl', bytes)

const saved = await readFile('./products.nrsl')
await narsil.restore('products', new Uint8Array(saved))
```

Snapshots use the same cross-language `.nrsl` envelope as persistence, so one engine's snapshot restores in another process, another machine, or another language implementation. Restoring bytes from an incompatible envelope version fails with `ENVELOPE_VERSION_MISMATCH`.