Observability

The engine reports what it holds through three calls, listIndexes, getPartitionStats, and getMemoryStats, while it reports the work that it does on its own through events. You build monitor.ts here, in which you load 2,500 support tickets, read what the index holds and what the engine uses, and then send the engine's events to an alert. Each code block below holds the whole file, with green lines marking the lines that you add at each step and struck lines marking the lines that you remove.

Read what each index holds

listIndexes returns one entry for each index, with its document count, its partition count, and whether it is open. getPartitionStats returns one entry for each partition of an index, with the engine's estimate of the memory that the partition takes.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({ workers: { count: 4 } })
 
await narsil.createIndex('tickets', {
  schema: { subject: 'string', priority: 'enum' },
  partitions: { maxPartitions: 2, maxDocsPerPartition: 1_500, watermark: 0.8 },
})
 
await narsil.insertBatch(
  'tickets',
  Array.from({ length: 2_500 }, (_unused, position) => ({
    id: `ticket-${position}`,
    subject: `Refund request ${position}`,
    priority: position % 3 === 0 ? 'high' : 'low',
  }))
)
 
console.log(JSON.stringify({ indexes: narsil.listIndexes(), partitions: narsil.getPartitionStats('tickets') }, null, 2))
 
await narsil.shutdown()

{
  "indexes": [
    {
      "name": "tickets",
      "documentCount": 2500,
      "partitionCount": 2,
      "language": "english",
      "state": "open",
      "reopenCount": 0
    }
  ],
  "partitions": [
    {
      "partitionId": 0,
      "documentCount": 1250,
      "estimatedMemoryBytes": 189526
    },
    {
      "partitionId": 1,
      "documentCount": 1250,
      "estimatedMemoryBytes": 189522
    }
  ]
}

The engine spreads the 2,500 tickets evenly across the two partitions, each of which takes about 190 KB by the engine's estimate. The engine computes that estimate from a formula, so use it to compare indexes and partitions inside one process, while the process's own heap figures give the real total. state holds closed once the engine releases an index from memory under its lifecycle limits. Each time the engine loads that index back, it adds one to reopenCount.

Report the memory that the engine uses

getMemoryStats adds the process's heap figures to the engine's own. process.heapLimit is the size that the heap may reach before Node ends the process, which Node takes from --max-old-space-size, from --max-old-space-size-percentage, or from the memory that the host reports. workerCopies holds, for each index, whether its worker copies are loaded and how many times the engine has loaded them again after an idle spell, while workers holds one heap report for each worker thread that holds copies.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({ workers: { count: 4 } })
 
await narsil.createIndex('tickets', {
  schema: { subject: 'string', priority: 'enum' },
  partitions: { maxPartitions: 2, maxDocsPerPartition: 1_500, watermark: 0.8 },
})
 
await narsil.insertBatch(
  'tickets',
  Array.from({ length: 2_500 }, (_unused, position) => ({
    id: `ticket-${position}`,
    subject: `Refund request ${position}`,
    priority: position % 3 === 0 ? 'high' : 'low',
  }))
)
 
console.log(JSON.stringify({ indexes: narsil.listIndexes(), partitions: narsil.getPartitionStats('tickets') }, null, 2)) 
const memory = await narsil.getMemoryStats() 
 
console.log(
  JSON.stringify(
    {
      estimatedIndexBytes: memory.estimatedIndexBytes,
      openIndexCount: memory.openIndexCount,
      closedIndexCount: memory.closedIndexCount,
      workerCopies: memory.workerCopies,
      workerThreads: memory.workers.length,
    },
    null,
    2
  )
)
 
await narsil.shutdown()

{
  "estimatedIndexBytes": 379048,
  "openIndexCount": 1,
  "closedIndexCount": 0,
  "workerCopies": [
    {
      "indexName": "tickets",
      "scaledOut": true,
      "reloadCount": 0
    }
  ],
  "workerThreads": 2
}

The program prints the figures that stay the same from one machine to the next. In the panel, estimatedIndexBytes is the sum of the estimates for every index, while workers holds two entries because the engine gives two of its four threads to keyword copies and the other two to vector search. The heap figures depend on the host and the Node version, so the program leaves them out. On the machine that produced these panels, as one sample, process.heapUsed stood at about 12 MB of a heapLimit of about 4.5 GB, and each worker thread used about 10 MB.

Alert on the engine's own events

The engine emits an event for work that it does away from any call that you make, so a failure there rejects no promise that you hold. Register a handler with narsil.on for each event that should reach an alert, before the engine starts the work behind that event.

In this step the program sends five of those events to an alert function, which writes one JSON line for each one.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil({ workers: { count: 4 } })
 
const alert = (event: string, detail: Record<string, unknown>): void => { 
  console.log(JSON.stringify({ event, ...detail }))
}
 
narsil.on('heapPressure', (payload) => alert('heapPressure', payload))
narsil.on('durabilityError', (payload) => alert('durabilityError', { message: payload.error.message }))
narsil.on('workerCrash', (payload) => alert('workerCrash', { workerId: payload.workerId, indexNames: payload.indexNames }))
narsil.on('workerPromote', (payload) => alert('workerPromote', { workerCount: payload.workerCount }))
narsil.on('partitionWatermark', (payload) => alert('partitionWatermark', payload))
 
await narsil.createIndex('tickets', {
  schema: { subject: 'string', priority: 'enum' },
  partitions: { maxPartitions: 2, maxDocsPerPartition: 1_500, watermark: 0.8 },
})
 
await narsil.insertBatch(
  'tickets',
  Array.from({ length: 2_500 }, (_unused, position) => ({
    id: `ticket-${position}`,
    subject: `Refund request ${position}`,
    priority: position % 3 === 0 ? 'high' : 'low',
  }))
)
 
const memory = await narsil.getMemoryStats() 
console.log( 
  JSON.stringify( 
    { 
      estimatedIndexBytes: memory.estimatedIndexBytes, 
      openIndexCount: memory.openIndexCount, 
      closedIndexCount: memory.closedIndexCount, 
      workerCopies: memory.workerCopies, 
      workerThreads: memory.workers.length, 
    }, 
    null, 
    2
  ) 
) 
 
await narsil.shutdown()

{"event":"workerPromote","workerCount":2}
{"event":"partitionWatermark","indexName":"tickets","documentCount":2500,"capacity":3000,"partitionCount":2}

The engine loads two worker copies once the index holds 1,000 tickets. It emits one partitionWatermark once the index reaches 80 per cent of its capacity of 3,000 tickets. The engine emits that event once for each crossing, so alert writes one line for as long as the index stays above the mark. In a measured run, the engine emitted no second event after 100 further inserts.

Alert before the heap runs out

The engine measures the heap during each write and each load and emits heapPressure once the heap reaches nine tenths of heapLimit. In a measured run under node --max-old-space-size=300, heapLimit stood at 314,572,800 bytes. With batches of 500 tickets, the event fired at 90 per cent of that figure, during the batch that followed the first 37,000 tickets. With batches of 20,000, the event fired at 93 per cent during the third batch; however, the engine carried on writing that batch until the process ran out of memory. A handler for the event can hold back the next batch but not the one that the engine is writing, so on a host whose heap stays close to its limit, keep each batch small enough to fit in the last tenth of the heap.

Set --max-old-space-size or --max-old-space-size-percentage on any host where the event matters. Without either flag, the engine compares the heap with the headroom that V8 reports, which is about 192 MB above the real ceiling, so on a default heap below about 2 GB the process can end before the event fires. The engine emits the event once for each crossing and arms it again once the free headroom recovers to two tenths.

Probe a server

A server built with createServer answers GET /livez with 200 while the process serves HTTP, and GET /readyz with 200 once the engine is ready. Point a load balancer's health check at /readyz, so that the load balancer sends traffic to a server only once its engine is ready. Neither probe passes through the onRequest hook, so a load balancer can call them without a token. GET /stats/memory returns the report from getMemoryStats with one more field, requestThreads, which is the number of worker threads that receive requests and answer searches from their own copies. On an 11-core machine with the default worker budget, requestThreads held 8. GET /indexes/{name}/stats returns the counts and the memory estimate for one index.

Where to go next

• Partitions and workers covers the partition limits and the worker copies behind these figures.
• Persistence and durability covers the lifecycle limits under which the engine closes an index, and the durability failures behind durabilityError.