Persistence and durability

Narsil writes indexes to storage through a pluggable persistence adapter, and the adapter you choose sets how durable each write is. A filesystem adapter runs the write-ahead-log tier, where a mutation reaches disk before the call that made it resolves. Every other adapter runs the snapshot tier, where the engine writes the whole index on a checkpoint. This page builds one file, store.ts, step by step. You start with an in-memory engine, add filesystem persistence, tune the log and the checkpoint schedule, and force a checkpoint by hand. Each step highlights exactly what changed.

Start with a plain engine

Begin store.ts with an engine and a products index that holds a small catalogue. Nothing reaches disk yet, so the whole index remains in memory and disappears when the process exits.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

Add filesystem persistence

The package includes three adapters:

Give the engine a filesystem adapter. That single line turns on the write-ahead-log tier: every insert, update, and removal appends to a log in ./narsil-data before the call resolves, and createNarsil replays that log before it hands you the engine. The products index now survives a restart with every acknowledged write intact.

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil() 
const narsil = await createNarsil({ 
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
})
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

A custom backend satisfies the PersistenceAdapter interface: save, load, delete, and list, all returning promises. The serialisation format is .nrsl, a 32-byte header followed by a MessagePack payload. The format is cross-language portable and specified in the repository's spec package, so another language implementation can read and write the same files.

How much a crash can cost you

Each adapter belongs to one of two tiers, and the tier sets how much a hard crash costs you.

On the write-ahead-log tier the log holds each mutation, checkpoints capture the index state so that recovery has a shorter log to replay, and recovery replays the log over the newest checkpoint.

On the snapshot tier there is no log. The engine writes the whole index through the adapter on a checkpoint trigger, either the durability.checkpointIntervalMs timer (five minutes by default) or durability.checkpointMutationThreshold mutations (100,000 by default), whichever comes first. Writes made after the last checkpoint are lost on a crash, and shutdown() does not checkpoint on its way out, so call checkpoint(indexName) before a planned stop.

Set durability.tier to pick a tier yourself. 'snapshot' puts any adapter on the snapshot tier, a filesystem one included, which is what several processes sharing one directory need, because a write-ahead log requires exclusive use of its directory. 'wal' requires a directory, either from durability.directory or from a filesystem adapter.

Tune the log and the checkpoint schedule

Add a durability block to choose the acknowledgement contract and the checkpoint schedule. The block needs no directory of its own here, because the filesystem adapter already names one.

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  durability: { 
    mode: 'sync',
    checkpointIntervalMs: 300_000,
    checkpointMutationThreshold: 100_000,
  },
})
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

In sync mode (the default) a write is not acknowledged until it is on disk, so a crash never loses a write that already returned success to your caller. In async mode writes acknowledge immediately while the log flushes every flushIntervalMs (one second by default), which is faster but can lose the final interval on a hard crash. Two more knobs shape the log itself: segmentMaxBytes rolls a new segment at 64 MiB, and compactionThreshold compacts after twelve checkpoint segments.

createNarsil checks every durability value before it starts anything, and an invalid one fails with CONFIG_INVALID:

• The engine rejects a tier other than 'wal' or 'snapshot', and a mode other than 'sync' or 'async'.
• It rejects a number that is not finite, a negative interval, and a size or threshold below 1. An interval of 0 is valid and turns that timer off.
• 'snapshot' needs a persistence adapter to write through, and 'wal' needs a directory it can resolve.
• 'snapshot' rejects every write-ahead-log field, meaning directory, mode, flushIntervalMs, segmentMaxBytes, and compactionThreshold, because none of them applies to a tier with no log.

Force a checkpoint

Checkpoints run on the schedule above, and checkpoint(indexName) runs one immediately. Call it before a planned shutdown or a backup.

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  durability: {
    mode: 'sync',
    checkpointIntervalMs: 300_000,
    checkpointMutationThreshold: 100_000,
  },
})
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})
 
await narsil.checkpoint('products') 

The call resolves once the checkpoint reaches storage. With no durability configured it does nothing, so the same code is safe whether or not the engine has an adapter.

Share one directory between processes

A write-ahead log requires exclusive use of its directory, so two engines pointed at the same one would corrupt each other's log. Force the snapshot tier when several processes read and write the same storage. This is a separate program, shared-store.ts, holding the same catalogue in a directory of its own with the tier pinned. Every process that shares that directory runs this configuration.

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-shared' }),
  durability: { tier: 'snapshot', checkpointIntervalMs: 30_000 },
})
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})
 
await narsil.checkpoint('products')

An invalidation adapter then reports each instance's writes to the others, so none of them serves stale data. Partitions and workers covers that side, and it uses the same tier: 'snapshot' line, because createNarsil rejects an invalidation adapter on any other tier with CONFIG_INVALID.

What recovery restores

createNarsil runs recovery before it resolves, so the first call after start-up runs against the recovered state. Recovery brings back each index's documents and its full configuration: the partition limits (maxDocsPerPartition, maxPartitions, and watermark), the scoring default, position tracking, strictness, the required fields, the vector promotion settings, the stop words and tokeniser, and the named embedding adapter bindings.

Recovery also checks each index's analysis. An index records the revision of the language module that built its terms, and recovery compares that against the revision the module carries now. A difference marks the index stale, which is what upgrading @delali/narsil past a release that corrects a stemmer or a stop word list produces. The index keeps answering, its results carry analysisStale: true, and by default the engine rebuilds its terms in the background from the documents already stored, then writes the new revision and takes a checkpoint. Language support covers the analysis config and rebuildAnalysis.

A checkpoint holds data, never code, so custom analysis persists under a name instead. Register a tokeniser with registerTokenizer or a stop word set with registerStopWords, then name it in the index config; recovery reads the name from the metadata and rebinds the implementation from the registry. Register the names before you call createNarsil, because recovery runs inside that call. An index config that hands a durable engine a tokeniser instance or a stop word function fails with CONFIG_INVALID at createIndex, which reports the problem while you write the code rather than at the next recovery. Language support shows both registries.

Report background write failures

Checkpoints and log appends run in the background, so a failed one never surfaces on the call that triggered it. The durabilityError event is the engine's channel for those failures. Register the handler once at start-up in any deployment that persists data, which in store.ts means the line right after the engine exists.

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  durability: {
    mode: 'sync',
    checkpointIntervalMs: 300_000,
    checkpointMutationThreshold: 100_000,
  },
})
 
narsil.on('durabilityError', (payload) => { 
  console.error('durability write failed:', payload.error)
})
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})
 
await narsil.checkpoint('products')

Snapshots and restore

snapshot(indexName) serialises a whole index, including its documents, schema, and vector data, into one portable byte array. restore(indexName, data) rebuilds an index from those bytes, replacing the index if it already exists. This is a separate maintenance script, backup.ts, and it reads what store.ts already wrote, so run store.ts first. It opens the same persisted engine so that recovery restores the products index, then writes that index to a file and reads it back.

import { readFile, writeFile } from 'node:fs/promises'
import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
})
 
const bytes = await narsil.snapshot('products')
await writeFile('./products.nrsl', bytes)
 
const saved = await readFile('./products.nrsl')
await narsil.restore('products', new Uint8Array(saved))

Snapshots use the same cross-language .nrsl envelope as persistence, so one engine's snapshot restores in another process, another machine, or another language implementation. Restoring bytes from an incompatible envelope version fails with ENVELOPE_VERSION_MISMATCH.