Persistence and durability

Narsil holds its indexes in memory, while a persistence adapter writes them to storage so that they come back after a restart. The adapter also sets the durability tier, which sets how many writes you can lose in a crash. You build store.ts here, in which you restart an order index without losing it and inspect the files that a checkpoint writes. You then use three short files to close idle indexes, share a directory between processes, and move an index as one snapshot. Each code block below holds a whole file, with green lines marking the lines that you add at each step and struck lines marking the lines that you remove.

Keep every acknowledged write across a restart

Three adapters come with the package: @delali/narsil/adapters/filesystem for a server process, @delali/narsil/adapters/indexeddb for a browser, and @delali/narsil/adapters/memory for tests. The filesystem adapter puts the engine on the write-ahead-log tier, where the engine appends every insert, update, and removal to a log before the call resolves. The program writes three orders, shuts the engine down, and opens a second engine on the same directory.

import { mkdtempSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const directory = mkdtempSync(join(tmpdir(), 'narsil-orders-'))
 
const openEngine = () => createNarsil({ persistence: createFilesystemPersistence({ directory }) })
 
const first = await openEngine()
 
await first.createIndex('orders', { schema: { customer: 'string', item: 'string', total: 'number' } })
 
await first.insertBatch('orders', [
  { id: 'ord-1001', customer: 'Grace Mensah', item: 'Hydraulic disc brake set', total: 149 },
  { id: 'ord-1002', customer: 'Diego Torres', item: 'Folding bicycle', total: 899 },
  { id: 'ord-1003', customer: 'Mika Ito', item: 'Steel gravel frame', total: 1299 },
])
 
await first.shutdown()
 
const second = await openEngine()
 
console.log(JSON.stringify(second.listIndexes(), null, 2))
 
await second.shutdown()

[
  {
    "name": "orders",
    "documentCount": 3,
    "partitionCount": 1,
    "language": "english",
    "state": "open",
    "reopenCount": 0
  }
]

The second engine recovers the three orders from the directory before createNarsil resolves, so the engine holds them before your first call after the restart. The engine also recovers each index's configuration, its partition limits included, which is why a createIndex for a name that the engine recovered fails with INDEX_ALREADY_EXISTS.

Choose how many writes you can lose in a crash

The tier sets how many writes you can lose in a crash. In a measured run, a child process wrote 500 orders one at a time and then killed itself with SIGKILL. A fresh engine then recovered the following from its directory:

Under sync, the default, the engine forces each write to disk before the call resolves. Under async the engine passes each write to the operating system before the call resolves and forces the log to disk every flushIntervalMs, so the engine loses nothing when the process crashes, although it can lose that last interval when the machine loses power. In one measured run, 2,000 awaited inserts took 14.8 seconds under sync and 0.7 seconds under async. On the snapshot tier the engine keeps no log and writes the whole index at each checkpoint, so you lose every write after the last checkpoint when the process crashes. shutdown() takes no checkpoint on that tier, so call checkpoint(indexName) before a planned stop.

You tune the log and the checkpoint schedule in the durability block. mode defaults to 'sync' and flushIntervalMs to 1,000, while the engine checkpoints every 300,000 ms by default or after 100,000 mutations, whichever comes first. The engine rolls the log to a new segment at segmentMaxBytes, 64 MiB by default, and merges a partition's checkpoint segments once the partition holds more than compactionThreshold of them, 12 by default. createNarsil checks every value before it starts, so a tier other than 'wal' or 'snapshot', a mode other than 'sync' or 'async', a negative interval, or a threshold below 1 fails with CONFIG_INVALID. In this step the program also forces a checkpoint and lists the files under the index.

import { mkdtempSync } from 'node:fs'
import { mkdtempSync, readdirSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
const directory = mkdtempSync(join(tmpdir(), 'narsil-orders-'))
 
const openEngine = () => createNarsil({ persistence: createFilesystemPersistence({ directory }) }) 
const openEngine = () =>
  createNarsil({
    persistence: createFilesystemPersistence({ directory }),
    durability: {
      mode: 'async',
      flushIntervalMs: 250,
      checkpointIntervalMs: 60_000,
      checkpointMutationThreshold: 10_000,
    },
  })
 
const first = await openEngine()
 
await first.createIndex('orders', { schema: { customer: 'string', item: 'string', total: 'number' } })
 
await first.insertBatch('orders', [
  { id: 'ord-1001', customer: 'Grace Mensah', item: 'Hydraulic disc brake set', total: 149 },
  { id: 'ord-1002', customer: 'Diego Torres', item: 'Folding bicycle', total: 899 },
  { id: 'ord-1003', customer: 'Mika Ito', item: 'Steel gravel frame', total: 1299 },
])
 
await first.checkpoint('orders') 
await first.shutdown()
 
const second = await openEngine()
 
console.log(JSON.stringify(second.listIndexes(), null, 2)) 
console.log(JSON.stringify(readdirSync(join(directory, 'orders'), { recursive: true }).sort(), null, 2)) 
 
await second.shutdown()

[
  "manifest",
  "meta",
  "segments",
  "segments/0",
  "segments/0/s0000000000000000",
  "wal",
  "wal/0",
  "wal/0/0000000000000001",
  "wal/0/commit"
]

The engine keeps for each index a manifest, a meta file, and one folder per partition under segments and under wal. At each checkpoint, the engine writes one new segment for each partition, which shortens the log that the engine replays at recovery. The engine also records in the commit file how much of the log it has forced to disk.

The engine reports a failed write to the durability directory through the durabilityError event, so register a handler for it with narsil.on. Where the directory of a live engine goes missing or is replaced, every later insert and checkpoint fails with PERSISTENCE_SAVE_FAILED, while insertBatch lists that code against each document. The engine emits the event once for that loss. Where the process loses write access to an index folder, inserts still resolve, but checkpoint fails with the same code and the engine emits the event. In a measured run, an engine restarted once the permissions came back recovered every order from that folder.

Close the indexes that you are not using

An engine with durability can release an index from memory and load it again on demand. close(indexName) releases one index and open(indexName) loads it, while the engine loads a closed index again by itself for a query or a write on it. With lifecycle the engine releases indexes for you, closing an index that goes idleTimeoutMs without a call, keeping at most maxOpenIndexes open, and keeping its own estimate of their memory under maxOpenBytes. close fails with CONFIG_INVALID on an engine with no durability, because there would be nothing to load the index back from.

import { mkdtempSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({
  durability: { directory: mkdtempSync(join(tmpdir(), 'narsil-desk-')) },
  lifecycle: { maxOpenIndexes: 2, idleTimeoutMs: 400 },
})
 
const states = (): string[] => narsil.listIndexes().map((index) => `${index.name} ${index.state} ${index.reopenCount}`)
 
for (const name of ['orders', 'invoices', 'shipments']) {
  await narsil.createIndex(name, { schema: { customer: 'string', total: 'number' } })
  await narsil.insert(name, { id: `${name}-1`, customer: 'Grace Mensah', total: 149 })
}
 
const afterCreating = states()
 
await narsil.query('orders', { term: 'abena' })
 
const afterQuerying = states()
 
await new Promise((resolve) => setTimeout(resolve, 1_200))
 
console.log(JSON.stringify({ afterCreating, afterQuerying, afterIdling: states() }, null, 2))
 
await narsil.shutdown()

{
  "afterCreating": [
    "orders closed 0",
    "invoices open 0",
    "shipments open 0"
  ],
  "afterQuerying": [
    "orders open 1",
    "invoices closed 0",
    "shipments open 0"
  ],
  "afterIdling": [
    "orders closed 1",
    "invoices closed 0",
    "shipments closed 0"
  ]
}

When you create the third index, the engine closes orders, which has gone longest without a call. The engine loads orders again for the query, which raises its reopenCount to 1, so it closes invoices to stay within maxOpenIndexes. After 1,200 ms with no calls, the engine has closed all three under the idleTimeoutMs of 400. A closed index still appears in listIndexes with the document count that it held at its last checkpoint.

Share one directory between processes

The write-ahead-log tier needs a directory of its own for each engine, so the engine locks its directory as it starts. A second engine that opens a locked directory fails with CONFIG_INVALID until the first engine shuts down, whether the second engine is in the same process or in another. Where the owner dies without shutting down, as a process killed with SIGKILL does, the next engine opens the directory and recovers its documents. To share one store between several processes, use the snapshot tier, where an invalidation adapter publishes each write and every other process reloads the partitions that the write touched. createNarsil refuses an invalidation adapter on the write-ahead-log tier with CONFIG_INVALID, so every process that shares the directory uses this configuration.

import { createNarsil } from '@delali/narsil'
import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
import { createFilesystemInvalidation } from '@delali/narsil/invalidation/filesystem'
 
const narsil = await createNarsil({
  persistence: createFilesystemPersistence({ directory: './narsil-shared' }),
  invalidation: createFilesystemInvalidation({ directory: './narsil-shared' }),
  durability: { tier: 'snapshot' },
})
 
await narsil.shutdown()

Move an index as one snapshot

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('orders', { schema: { customer: 'string', item: 'string', total: 'number' } })
 
await narsil.insertBatch('orders', [
  { id: 'ord-1001', customer: 'Grace Mensah', item: 'Hydraulic disc brake set', total: 149 },
  { id: 'ord-1002', customer: 'Diego Torres', item: 'Folding bicycle', total: 899 },
  { id: 'ord-1003', customer: 'Mika Ito', item: 'Steel gravel frame', total: 1299 },
])
 
const bytes = await narsil.snapshot('orders')
 
await narsil.restore('orders-copy', bytes)
 
console.log(
  JSON.stringify(
    {
      header: new TextDecoder().decode(bytes.subarray(0, 4)),
      copied: await narsil.countDocuments('orders-copy'),
      found: (await narsil.query('orders-copy', { term: 'gravel' })).hits.map((hit) => hit.id),
    },
    null,
    2
  )
)
 
await narsil.shutdown()

{
  "header": "NRSL",
  "copied": 3,
  "found": [
    "ord-1003"
  ]
}

A snapshot is a Uint8Array that opens with the bytes NRSL and holds the whole index in the portable .nrsl format, so you can store it as a backup or restore it in another process or on another machine. restore(indexName, bytes) rebuilds the index under the name that you give it, replacing an index of that name where one exists. restore fails with ENVELOPE_VERSION_MISMATCH for bytes from a newer envelope format, and with ENVELOPE_INVALID_MAGIC for bytes that are no snapshot.

Where to go next

• Partitions and workers covers the partitions that each hold a folder here, and the worker threads that search them.
• Language support covers the language revisions that the engine compares at recovery.
• HTTP server covers the snapshot and restore endpoints that a server exposes.