A partition is a slice of an index with its own terms and statistics. An index opens with as many partitions as maxPartitions sets, 1 by default, and the engine keeps that count until you rebalance the index. Worker threads hold copies of an index, so that the engine can answer searches off the main thread and leave it free for the rest of your application. You build logs.ts here, in which you split 85,000 log records across four partitions and then reshape them into eight. You then use search-logs.ts to move searches onto worker threads that stay current with every write. Each code block below holds a whole file, with green lines marking what each step adds and struck lines marking what it takes away.
Split an index across partitions
partitions.maxPartitions sets how many partitions the index opens with, and the engine never adds one on its own. The engine caps each partition at maxDocsPerPartition documents, so the index holds at most that number times the partition count, 100,000 documents here, and a write past that fails with PARTITION_CAPACITY_EXCEEDED. watermark sets the share of that capacity at which the engine emits partitionWatermark, once for each crossing. The program turns worker threads off, so that every search in this file stays on the main thread.
import { createNarsil } from '@delali/narsil'
const RECORD_COUNT = 85_000
const SERVICES = ['checkout', 'gateway', 'inventory', 'search'] as const
const narsil = await createNarsil({ workers: { enabled: false } })
const warnings: string[] = []
narsil.on('partitionWatermark', (event) => {
warnings.push(`${event.indexName} holds ${event.documentCount} of ${event.capacity}`)
})
await narsil.createIndex('logs', {
schema: { message: 'string', service: 'enum', level: 'enum', timestamp: 'number' },
partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})
await narsil.insertBatch(
'logs',
Array.from({ length: RECORD_COUNT }, (_unused, position) => ({
id: `log-${position}`,
message: `Request ${position} completed through the ${SERVICES[position % SERVICES.length]} service`,
service: SERVICES[position % SERVICES.length],
level: position % 50 === 0 ? 'error' : 'info',
timestamp: 1_790_000_000_000 + position * 1_000,
}))
)
console.log(
JSON.stringify(
{ partitions: narsil.getPartitionStats('logs').map((partition) => partition.documentCount), warnings },
null,
2
)
)
await narsil.shutdown(){
"partitions": [
21250,
21250,
21250,
21250
],
"warnings": [
"logs holds 85000 of 100000"
]
}The engine spreads the 85,000 records evenly, 21,250 to a partition, and it emits one warning as the load passes 80 per cent of the 100,000 capacity. When a load goes past the capacity, the engine stores what fits, so from a batch of 90 records into a capacity of 80 it stores 80 and reports PARTITION_CAPACITY_EXCEEDED against each of the other 10.
Reshape the index while it serves queries
updatePartitionConfig raises the limits, and rebalance(indexName, count) moves the documents into the new number of partitions. A rebalance may not go past maxPartitions, and a new maxDocsPerPartition has to leave room for the documents already held, so either step fails with PARTITION_CAPACITY_EXCEEDED otherwise.
import { createNarsil } from '@delali/narsil'
const RECORD_COUNT = 85_000
const SERVICES = ['checkout', 'gateway', 'inventory', 'search'] as const
const narsil = await createNarsil({ workers: { enabled: false } })
const warnings: string[] = []
narsil.on('partitionWatermark', (event) => {
warnings.push(`${event.indexName} holds ${event.documentCount} of ${event.capacity}`)
})
await narsil.createIndex('logs', {
schema: { message: 'string', service: 'enum', level: 'enum', timestamp: 'number' },
partitions: { maxPartitions: 4, maxDocsPerPartition: 25_000, watermark: 0.8 },
})
await narsil.insertBatch(
'logs',
Array.from({ length: RECORD_COUNT }, (_unused, position) => ({
id: `log-${position}`,
message: `Request ${position} completed through the ${SERVICES[position % SERVICES.length]} service`,
service: SERVICES[position % SERVICES.length],
level: position % 50 === 0 ? 'error' : 'info',
timestamp: 1_790_000_000_000 + position * 1_000,
}))
)
await narsil.updatePartitionConfig('logs', { maxPartitions: 8, maxDocsPerPartition: 50_000 })
await narsil.rebalance('logs', 8)
const gateway = await narsil.query('logs', { term: 'gateway', limit: 0 })
console.log(
JSON.stringify(
{ partitions: narsil.getPartitionStats('logs').map((partition) => partition.documentCount), warnings },
{ partitions: narsil.getPartitionStats('logs').map((partition) => partition.documentCount), gateway: gateway.count },
null,
2
)
)
await narsil.shutdown(){
"partitions": [
10624,
10625,
10625,
10624,
10626,
10625,
10625,
10626
],
"gateway": 21250
}After the rebalance, each of the eight partitions holds about 10,625 records, and the search still counts every gateway record. The engine keeps answering while a rebalance moves the documents: in a measured run, 43 queries spaced 10 ms apart during a rebalance of these 85,000 records all returned 21,250, and the rebalance finished in 545 ms.
Search on worker threads
Once an index holds promotionThreshold documents, 1,000 by default, the engine gives it copies on worker threads and sends searches to them. workers.count sets the thread budget, which the engine splits between keyword copies and a pool for vector search, so count: 4 gives each index two keyword copies. With mainCopyQueries: 'none' the engine sends every search to a copy, which is how an engine behind createServer behaves by default, while under the default 'lone' the main copy also answers a search that arrives while no copy is free, on an index of at most 50,000 documents.
import { createNarsil } from '@delali/narsil'
const SERVICES = ['checkout', 'gateway', 'inventory', 'search'] as const
const narsil = await createNarsil({ workers: { count: 4, mainCopyQueries: 'none' } })
const copies: number[] = []
narsil.on('workerPromote', (event) => {
copies.push(event.workerCount)
})
await narsil.createIndex('logs', {
schema: { message: 'string', service: 'enum', level: 'enum', timestamp: 'number' },
})
await narsil.insertBatch(
'logs',
Array.from({ length: 2_000 }, (_unused, position) => ({
id: `log-${position}`,
message: `Request ${position} completed through the ${SERVICES[position % SERVICES.length]} service`,
service: SERVICES[position % SERVICES.length],
level: 'info',
timestamp: 1_790_000_000_000 + position * 1_000,
}))
)
await Promise.all(
Array.from({ length: 50 }, (_unused, position) =>
narsil.insert('logs', {
id: `burst-${position}`,
message: `Payment ${position} declined at the gateway`,
service: 'gateway',
level: 'error',
timestamp: 1_790_010_000_000 + position,
})
)
)
await narsil.waitForWrites('logs')
const declined = await narsil.query('logs', { term: 'declined', limit: 0 })
console.log(JSON.stringify({ copies, declined: declined.count }, null, 2))
await narsil.shutdown(){
"copies": [
2
],
"declined": 50
}The engine gives the index two worker copies once the load passes 1,000 documents. Each copy applies a write after the main copy does, so a search sent straight after a burst of concurrent writes can miss some of them: in a measured run of twelve bursts of 50 concurrent inserts with no wait, every read came back short, and the first held 32 of its 50 writes. waitForWrites(indexName) resolves once every copy has applied every write that returned before the call, which is why the count here holds all 50, and { wait: true } on one write does the same for that write. A read that follows each awaited write in turn came back current in every measured round, so the wait matters for writes that overlap.
The engine drops a copy that nothing has searched for workers.idleTimeoutMs, and the next search loads it again. When a worker thread dies, the engine emits workerCrash, replaces the thread, and keeps answering, which measured runs confirmed with no search failing. An index configured with a tokeniser instance stays on the main thread and reports workerPromoteFailure, because no worker thread can receive code, so register the tokeniser under a name as language support shows, and set workers.bootstrapModule to an absolute path or URL of a module that registers it in every worker.
Where to go next
- Persistence and durability covers the folders that the engine writes for each partition and the directory that several processes can share.
- Full-text search covers the scoring modes that set which partition statistics a query uses.