Indexes and documents

Every index starts from a schema. Narsil validates each document against it at insertion time, so invalid data fails fast with a clear error. This page builds one file, catalog.ts, step by step. You define a schema, load documents, read them back, page through them, and change them. Each step highlights exactly what changed.

Define a schema

The schema supports string, number, boolean, enum, geopoint, vector[N], and the array variants string[], number[], boolean[], and enum[]. Objects nest up to four levels deep. Start catalog.ts with an engine and an articles index.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})

The required list names fields a document must carry; inserts without them fail with DOC_MISSING_REQUIRED_FIELD. Set strict: true to reject documents carrying fields the schema does not declare. Beyond the schema, the index config controls tokenisation and ranking: language selects the analyser, stopWords replaces or transforms the stop word set, tokenizer swaps in your own implementation, and bm25 overrides the k1 and b ranking parameters. surfaceForms makes suggestions and prefix completions return the original spellings from your documents; the default is false, which returns the stemmed index terms. Recording every spelling costs some insert throughput, and the setting persists with the index.

Insert documents

insert(indexName, document, docId?) resolves the document id in a fixed order: an explicit docId argument takes precedence, then a string id field on the document, and otherwise Narsil generates a UUID v7. The method returns the resolved id. Add two articles to catalog.ts, each carrying its own id.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})
 
await narsil.insert('articles', { 
  id: 'scaling-search',
  title: 'Scaling search with partitions',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024,
})
 
await narsil.insert('articles', { 
  id: 'vector-basics',
  title: 'Vector search basics',
  body: 'Dense vectors capture meaning beyond exact keywords.',
  author: { name: 'Kofi Boateng', verified: false },
  publishedYear: 2023,
})

Inserting an id that already exists fails with DOC_ALREADY_EXISTS. An upsert checks has() first and picks the right call; the HTTP server's PUT endpoint wraps that check into one request.

Load many documents in one call

Inserting one document at a time is clear, but loading a corpus that way sends a separate call per document. insertBatch(indexName, documents) takes the whole set in one call and returns partial results: one bad document never aborts the batch, so every success still applies and every failure comes back with its id and error. Swap the two single inserts for a single batch that also loads a third article and one document that fails validation.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})
 
await narsil.insert('articles', { 
  id: 'scaling-search',
  title: 'Scaling search with partitions',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024,
})
 
await narsil.insert('articles', { 
  id: 'vector-basics',
  title: 'Vector search basics',
  body: 'Dense vectors capture meaning beyond exact keywords.',
  author: { name: 'Kofi Boateng', verified: false },
  publishedYear: 2023,
})
 
const result = await narsil.insertBatch('articles', [ 
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])
 
console.log(result) 

The three valid documents are added, and the fourth fails because title must be a string. insertBatch returns the ids that succeeded and, for each failure, the id paired with the NarsilError that stopped it:

{
  succeeded: [ 'scaling-search', 'vector-basics', 'ranking-explained' ],
  failed: [ { docId: 'broken', error: [NarsilError] } ]
}

console.log abbreviates the error as [NarsilError]. That entry's error.code is DOC_VALIDATION_FAILED, and error.details holds { field: 'title', expected: 'string', received: 'number' }.

Large batches process in chunks and yield the event loop between chunks, so searches remain responsive during a bulk load. updateBatch and removeBatch follow the same shape for changing and deleting many documents at once.

Read documents

With the three articles indexed, read them back by id. get returns the document, or undefined for an unknown id. getMultiple returns a Map of the ids that exist. has tests for presence, and countDocuments reports the index size.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})
 
const result = await narsil.insertBatch('articles', [
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])
 
const scaling = await narsil.get('articles', 'scaling-search') 
const found = await narsil.getMultiple('articles', ['scaling-search', 'ranking-explained'])
const exists = await narsil.has('articles', 'scaling-search')
const total = await narsil.countDocuments('articles')
 
console.log(scaling)
console.log(found)
console.log(exists)
console.log(total)

The broken document fails validation because its title is a number, so it never reaches the index and countDocuments reports three.

Page through every document

get and getMultiple need ids you already have. A query with no term returns no hits, even when it carries filters. Neither path shows you what a freshly loaded index contains. By default, listDocuments pages through the stored documents in document-id order without searching. Each page carries a cursor. A cursor is an opaque string that marks the id its page stopped at. Leave it out on the first call, and pass it back on every call after that. Stop once the cursor comes back null. Ask for two documents at a time so that the three articles span two pages.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})
 
const result = await narsil.insertBatch('articles', [
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])
 
const scaling = await narsil.get('articles', 'scaling-search')
const found = await narsil.getMultiple('articles', ['scaling-search', 'ranking-explained'])
const exists = await narsil.has('articles', 'scaling-search')
const total = await narsil.countDocuments('articles')
 
const ids: string[] = [] 
let cursor: string | undefined
 
do {
  const page = await narsil.listDocuments('articles', { limit: 2, cursor })
  console.log(JSON.stringify(page, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))
  for (const entry of page.documents) ids.push(entry.id)
  cursor = page.cursor ?? undefined
} while (cursor !== undefined)

The first page holds the two articles whose ids sort first. total reports the three documents the listing covers. The page carries two of them. The second page returns the last article with cursor: null, which ends the loop. Narsil compares ids by their UTF-16 code units, so '10' sorts ahead of '9'. Pad numeric ids to a fixed width when you want a listing to follow the numbers.

Writes continue while you page. Narsil guarantees three things about them. A document that stays in the index for the whole listing comes back exactly once. Narsil skips a document you remove part-way through. It returns one you insert part-way through once that document's id sorts above the cursor.

The cursor is that id and nothing more. A client that stops paging leaves nothing to expire. A saved cursor still works after a restart, a snapshot restore, and a rebalance. Reaching the last page of a large index costs what reaching the first page costs. limit defaults to 10 and tops out at 10,000. Narsil raises a value below one to one. A cursor Narsil did not issue fails with SEARCH_INVALID_CURSOR. Pass filters to list part of an index, sort to order it by field value instead of by id, and document to cut down what each entry carries. Browsing an index without a search term shows all three.

Update and remove

update replaces the whole document under an id. Internally it removes the old document and inserts the new one, with a fast path when the change touches nothing the index depends on. remove deletes a document by id. Both throw DOC_NOT_FOUND for an unknown id.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})
 
const result = await narsil.insertBatch('articles', [
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])
 
const scaling = await narsil.get('articles', 'scaling-search')
const found = await narsil.getMultiple('articles', ['scaling-search', 'ranking-explained'])
const exists = await narsil.has('articles', 'scaling-search')
const total = await narsil.countDocuments('articles')
 
const ids: string[] = []
let cursor: string | undefined
 
do {
  const page = await narsil.listDocuments('articles', { limit: 2, cursor })
  for (const entry of page.documents) ids.push(entry.id)
  cursor = page.cursor ?? undefined
} while (cursor !== undefined)
 
await narsil.update('articles', 'scaling-search', { 
  title: 'Scaling search with partitions and workers',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024,
})
await narsil.remove('articles', 'vector-basics') 

Manage indexes

These calls inspect the index and then reset it. clear removes every document but keeps the index and its schema, while dropIndex removes the index entirely, including its persisted data.

import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})
 
const result = await narsil.insertBatch('articles', [
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])
 
await narsil.update('articles', 'scaling-search', {
  title: 'Scaling search with partitions and workers',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024,
})
await narsil.remove('articles', 'vector-basics')
 
const indexes = narsil.listIndexes() 
const stats = narsil.getStats('articles')
 
console.log(indexes)
console.log(stats)
 
await narsil.clear('articles')
await narsil.dropIndex('articles')

Call shutdown() when the process is done with the engine; it stops workers, flushes durability state, and rejects further calls.

Handle errors

Every failure throws a NarsilError carrying a stable string code, a readable message, and a details object with the values that produced the failure. errors.ts builds the same index and inserts a number where the schema declares a string.

import { createNarsil, ErrorCodes, NarsilError } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})
 
try {
  await narsil.insert('articles', { title: 42 })
} catch (error) {
  if (error instanceof NarsilError && error.code === ErrorCodes.DOC_VALIDATION_FAILED) {
    console.error(error.message, error.details)
  }
}

Field "title" expected string, got number { field: 'title', expected: 'string', received: 'number' }

The full set of codes is exported as ErrorCodes, covering index lifecycle, document validation, search input, vectors, embeddings, partitions, and configuration.