> Complete content index: https://narsil.sondelali.com/llms.txt

# Observability

> Read what each index holds, report the memory that the engine uses, alert on the events that the engine emits on its own, and probe a server from a load balancer.

URL: https://narsil.sondelali.com/docs/observability
Section: Operations
Version: 0.3 (@delali/narsil@0.3.0, latest)

The engine reports what it holds through three calls, `listIndexes`, `getPartitionStats`, and `getMemoryStats`, while it reports the work that it does on its own through events. You build `monitor.ts` here, in which you load 2,500 support tickets, read what the index holds and what the engine uses, and then send the engine's events to an alert. Each code block below holds the whole file, with green lines marking the lines that you add at each step and struck lines marking the lines that you remove.

## Read what each index holds

`listIndexes` returns one entry for each index, with its document count, its partition count, and whether it is open. `getPartitionStats` returns one entry for each partition of an index, with the engine's estimate of the memory that the partition takes.

```ts title="monitor.ts"
import { createNarsil } from '@delali/narsil'

const narsil = await createNarsil({ workers: { count: 4 } })

await narsil.createIndex('tickets', {
  schema: { subject: 'string', priority: 'enum' },
  partitions: { maxPartitions: 2, maxDocsPerPartition: 1_500, watermark: 0.8 },
})

await narsil.insertBatch(
  'tickets',
  Array.from({ length: 2_500 }, (_unused, position) => ({
    id: `ticket-${position}`,
    subject: `Refund request ${position}`,
    priority: position % 3 === 0 ? 'high' : 'low',
  }))
)

console.log(JSON.stringify({ indexes: narsil.listIndexes(), partitions: narsil.getPartitionStats('tickets') }, null, 2))

await narsil.shutdown()
```

```json result open
{
  "indexes": [
    {
      "name": "tickets",
      "documentCount": 2500,
      "partitionCount": 2,
      "language": "english",
      "state": "open",
      "reopenCount": 0
    }
  ],
  "partitions": [
    {
      "partitionId": 0,
      "documentCount": 1250,
      "estimatedMemoryBytes": 189526
    },
    {
      "partitionId": 1,
      "documentCount": 1250,
      "estimatedMemoryBytes": 189522
    }
  ]
}
```

The engine spreads the 2,500 tickets evenly across the two partitions, each of which takes about 190 KB by the engine's estimate. The engine computes that estimate from a formula, so use it to compare indexes and partitions inside one process, while the process's own heap figures give the real total. `state` holds `closed` once the engine releases an index from memory under its lifecycle limits. Each time the engine loads that index back, it adds one to `reopenCount`.

## Report the memory that the engine uses

`getMemoryStats` adds the process's heap figures to the engine's own. `process.heapLimit` is the size that the heap may reach before Node ends the process, which Node takes from `--max-old-space-size`, from `--max-old-space-size-percentage`, or from the memory that the host reports. `workerCopies` holds, for each index, whether its worker copies are loaded and how many times the engine has loaded them again after an idle spell, while `workers` holds one heap report for each worker thread that holds copies.

```ts title="monitor.ts"
import { createNarsil } from '@delali/narsil'

const narsil = await createNarsil({ workers: { count: 4 } })

await narsil.createIndex('tickets', {
  schema: { subject: 'string', priority: 'enum' },
  partitions: { maxPartitions: 2, maxDocsPerPartition: 1_500, watermark: 0.8 },
})

await narsil.insertBatch(
  'tickets',
  Array.from({ length: 2_500 }, (_unused, position) => ({
    id: `ticket-${position}`,
    subject: `Refund request ${position}`,
    priority: position % 3 === 0 ? 'high' : 'low',
  }))
)

console.log(JSON.stringify({ indexes: narsil.listIndexes(), partitions: narsil.getPartitionStats('tickets') }, null, 2)) // [!code --]
const memory = await narsil.getMemoryStats() // [!code ++:15]

console.log(
  JSON.stringify(
    {
      estimatedIndexBytes: memory.estimatedIndexBytes,
      openIndexCount: memory.openIndexCount,
      closedIndexCount: memory.closedIndexCount,
      workerCopies: memory.workerCopies,
      workerThreads: memory.workers.length,
    },
    null,
    2
  )
)

await narsil.shutdown()
```

```json result open
{
  "estimatedIndexBytes": 379048,
  "openIndexCount": 1,
  "closedIndexCount": 0,
  "workerCopies": [
    {
      "indexName": "tickets",
      "scaledOut": true,
      "reloadCount": 0
    }
  ],
  "workerThreads": 2
}
```

The program prints the figures that stay the same from one machine to the next. In the panel, `estimatedIndexBytes` is the sum of the estimates for every index, while `workers` holds two entries because the engine gives two of its four threads to keyword copies and the other two to vector search. The heap figures depend on the host and the Node version, so the program leaves them out. On the machine that produced these panels, as one sample, `process.heapUsed` stood at about 12 MB of a `heapLimit` of about 4.5 GB, and each worker thread used about 10 MB.

## Alert on the engine's own events

The engine emits an event for work that it does away from any call that you make, so a failure there rejects no promise that you hold. Register a handler with `narsil.on` for each event that should reach an alert, before the engine starts the work behind that event.

| Event | The engine emits it when |
| --- | --- |
| `heapPressure` | The heap reaches nine tenths of its limit during a write or a load. |
| `durabilityError` | A write to the durability directory fails. |
| `invalidationError` | The invalidation channel fails, so the engine can read partitions that another process has changed. |
| `workerCrash` | A worker thread dies, and the engine starts a replacement. |
| `workerPromote` | The engine gives an index worker copies. |
| `workerPromoteFailure` | The engine cannot give an index worker copies, so the index stays on the main thread. |
| `partitionWatermark` | The documents in an index reach the share of its capacity named in `watermark`. |
| `partitionRebalance` | A rebalance finishes moving an index onto a new partition count. |
| `analysisRebuild` | A rebuild of an index's terms starts, completes, or fails. |

In this step the program sends five of those events to an `alert` function, which writes one JSON line for each one.

```ts title="monitor.ts"
import { createNarsil } from '@delali/narsil'

const narsil = await createNarsil({ workers: { count: 4 } })

const alert = (event: string, detail: Record<string, unknown>): void => { // [!code ++:10]
  console.log(JSON.stringify({ event, ...detail }))
}

narsil.on('heapPressure', (payload) => alert('heapPressure', payload))
narsil.on('durabilityError', (payload) => alert('durabilityError', { message: payload.error.message }))
narsil.on('workerCrash', (payload) => alert('workerCrash', { workerId: payload.workerId, indexNames: payload.indexNames }))
narsil.on('workerPromote', (payload) => alert('workerPromote', { workerCount: payload.workerCount }))
narsil.on('partitionWatermark', (payload) => alert('partitionWatermark', payload))

await narsil.createIndex('tickets', {
  schema: { subject: 'string', priority: 'enum' },
  partitions: { maxPartitions: 2, maxDocsPerPartition: 1_500, watermark: 0.8 },
})

await narsil.insertBatch(
  'tickets',
  Array.from({ length: 2_500 }, (_unused, position) => ({
    id: `ticket-${position}`,
    subject: `Refund request ${position}`,
    priority: position % 3 === 0 ? 'high' : 'low',
  }))
)

const memory = await narsil.getMemoryStats() // [!code --]
console.log( // [!code --]
  JSON.stringify( // [!code --]
    { // [!code --]
      estimatedIndexBytes: memory.estimatedIndexBytes, // [!code --]
      openIndexCount: memory.openIndexCount, // [!code --]
      closedIndexCount: memory.closedIndexCount, // [!code --]
      workerCopies: memory.workerCopies, // [!code --]
      workerThreads: memory.workers.length, // [!code --]
    }, // [!code --]
    null, // [!code --]
    2 // [!code --]
  ) // [!code --]
) // [!code --]

await narsil.shutdown()
```

```txt result open
{"event":"workerPromote","workerCount":2}
{"event":"partitionWatermark","indexName":"tickets","documentCount":2500,"capacity":3000,"partitionCount":2}
```

The engine loads two worker copies once the index holds 1,000 tickets. It emits one `partitionWatermark` once the index reaches 80 per cent of its capacity of 3,000 tickets. The engine emits that event once for each crossing, so `alert` writes one line for as long as the index stays above the mark. In a measured run, the engine emitted no second event after 100 further inserts.

### Alert before the heap runs out

The engine measures the heap during each write and each load and emits `heapPressure` once the heap reaches nine tenths of `heapLimit`. In a measured run under `node --max-old-space-size=300`, `heapLimit` stood at 314,572,800 bytes. With batches of 500 tickets, the event fired at 90 per cent of that figure, during the batch that followed the first 37,000 tickets. With batches of 20,000, the event fired at 93 per cent during the third batch; however, the engine carried on writing that batch until the process ran out of memory. A handler for the event can hold back the next batch but not the one that the engine is writing, so on a host whose heap stays close to its limit, keep each batch small enough to fit in the last tenth of the heap.

Set `--max-old-space-size` or `--max-old-space-size-percentage` on any host where the event matters. Without either flag, the engine compares the heap with the headroom that V8 reports, which is about 192 MB above the real ceiling, so on a default heap below about 2 GB the process can end before the event fires. The engine emits the event once for each crossing and arms it again once the free headroom recovers to two tenths.

## Probe a server

A server built with `createServer` answers `GET /livez` with 200 while the process serves HTTP, and `GET /readyz` with 200 once the engine is ready. Point a load balancer's health check at `/readyz`, so that the load balancer sends traffic to a server only once its engine is ready. Neither probe passes through the `onRequest` hook, so a load balancer can call them without a token. `GET /stats/memory` returns the report from `getMemoryStats` with one more field, `requestThreads`, which is the number of worker threads that receive requests and answer searches from their own copies. On an 11-core machine with the default worker budget, `requestThreads` held 8. `GET /indexes/{name}/stats` returns the counts and the memory estimate for one index.

## Where to go next

- [Partitions and workers](/docs/partitions-and-workers) covers the partition limits and the worker copies behind these figures.
- [Persistence and durability](/docs/persistence-and-durability) covers the lifecycle limits under which the engine closes an index, and the durability failures behind `durabilityError`.