Persistence and durability

Narsil separates two storage concerns. Persistence stores serialised partitions through a pluggable adapter on a debounced schedule. Durability adds a write-ahead log so a crash never loses acknowledged writes. This page builds one file, store.ts, step by step. You start with an in-memory engine, add filesystem persistence, then add durability. Each step highlights exactly what changed.

Start with a plain engine

Begin store.ts with an engine and a products index that holds a small catalogue. Nothing is written to disk yet, so the whole index remains in memory and is lost when the process exits.

store.ts

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

Add filesystem persistence

The package includes three adapters:

Adapter	Import	Environment
Memory	`@delali/narsil/adapters/memory`	Works everywhere and suits tests.
Filesystem	`@delali/narsil/adapters/filesystem`	Runs on Node.js, Bun, and Deno.
IndexedDB	`@delali/narsil/adapters/indexeddb`	Runs in browsers.

Give the engine a filesystem adapter and a flush schedule. Flushing is debounced: dirty partitions serialise on the flush.interval timer or after flush.mutationThreshold mutations, whichever fires first. The products index now survives a restart because its partitions are written to ./narsil-data.

store.ts

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil() 
const narsil = await createNarsil({ 
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  flush: { interval: 5000, mutationThreshold: 100 },
})
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

A custom backend satisfies the PersistenceAdapter interface: save, load, delete, and list, all returning promises. The serialisation format is .nrsl, a 32-byte header followed by a MessagePack payload. The format is cross-language portable and specified in the repository's spec package, so another language implementation can read and write the same files.

Add durability

Debounced persistence can lose the writes made after the last flush. Durability closes that window: every mutation appends to a write-ahead log before it is acknowledged, periodic checkpoints capture the index state, and recovery replays the log over the newest checkpoint. Add a durability block that stores its log alongside the persisted partitions.

store.ts

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  flush: { interval: 5000, mutationThreshold: 100 },
  durability: { 
    directory: './narsil-data',
    mode: 'sync',
  },
})
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})

In sync mode (the default) a write is not acknowledged until it is on disk, so a crash never loses a write that already returned success to your caller. In async mode writes acknowledge immediately while the log flushes on an interval, which is faster but can lose the final interval on a hard crash.

createNarsil runs recovery before it resolves, so indexes, documents, and named embedding adapter bindings are restored before the first call.

Checkpoints keep recovery fast by capturing the current index state, which shortens the log that recovery must replay. The engine writes them on a schedule: one every checkpointIntervalMs (five minutes by default) or after checkpointMutationThreshold mutations (100,000 by default), whichever comes first. Call checkpoint when you want one immediately, such as before a planned shutdown or a backup.

store.ts

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
  flush: { interval: 5000, mutationThreshold: 100 },
  durability: {
    directory: './narsil-data',
    mode: 'sync',
  },
})
 
await narsil.createIndex('products', {
  schema: {
    title: 'string',
    description: 'string',
    price: 'number',
    inStock: 'boolean',
    category: 'enum',
    tags: 'string[]',
  },
  language: 'english',
})
 
await narsil.insert('products', {
  id: 'mechanical-keyboard',
  title: 'Mechanical Keyboard',
  description: 'Cherry MX Brown switches with PBT keycaps and USB-C connection',
  price: 129.99,
  inStock: true,
  category: 'electronics',
  tags: ['peripherals', 'typing', 'mechanical'],
})
 
await narsil.checkpoint('products')

The call resolves once the checkpoint is on disk. With no durability configured, it does nothing, so the same code is safe whether or not durability is enabled.

Report background write failures

Persistence and durability both write in the background, so a failed flush or a failed log append never surfaces on the call that triggered it. The persistenceError and durabilityError events are the only way the engine reports those failures. Subscribe to both in any deployment that persists data, and register these handlers once at start-up.

narsil.on('persistenceError', (payload) => {
  console.error(`flush failed for ${payload.indexName}:`, payload.error)
})
 
narsil.on('durabilityError', (payload) => {
  console.error('write-ahead log failed:', payload.error)
})

Snapshots and restore

snapshot(indexName) serialises a whole index, including its documents, schema, and vector data, into one portable byte array. restore(indexName, data) rebuilds an index from those bytes, replacing the index if it already exists. This is a separate maintenance script, backup.ts. It opens the same persisted engine so recovery restores the products index, then writes that index to a file and reads it back.

backup.ts

import { readFile, writeFile } from 'node:fs/promises'
import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
})
 
const bytes = await narsil.snapshot('products')
await writeFile('./products.nrsl', bytes)
 
const saved = await readFile('./products.nrsl')
await narsil.restore('products', new Uint8Array(saved))

Snapshots use the same cross-language .nrsl envelope as persistence, so one engine's snapshot restores in another process, another machine, or another language implementation. Restoring bytes from an incompatible envelope version fails with ENVELOPE_VERSION_MISMATCH.