> Complete content index: https://narsil.sondelali.com/llms.txt

# Indexes and documents

> Define typed schemas, manage indexes, and insert, read, update, and remove documents one at a time or in batches.

URL: https://narsil.sondelali.com/docs/indexes-and-documents
Section: Core concepts
Version: 0.2 (@delali/narsil@0.2.2, latest)

Every index starts from a schema. Narsil validates each document against it at insertion time, so invalid data fails fast with a clear error. This page builds one file, `catalog.ts`, step by step. You define a schema, load documents, read them back, page through them, and change them. Each step highlights exactly what changed.

## Define a schema

The schema supports `string`, `number`, `boolean`, `enum`, `geopoint`, `vector[N]`, and the array variants `string[]`, `number[]`, `boolean[]`, and `enum[]`. Objects nest up to four levels deep. Start `catalog.ts` with an engine and an `articles` index.

```ts title="catalog.ts"

const narsil = await createNarsil()

await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})
```

The `required` list names fields a document must carry; inserts without them fail with `DOC_MISSING_REQUIRED_FIELD`. Set `strict: true` to reject documents carrying fields the schema does not declare. Beyond the schema, the index config controls tokenisation and ranking: `language` selects the <Tooltip tip="An analyser is the processing pipeline that turns raw text into searchable terms, typically by splitting it into tokens, lowercasing them, and stemming them.">analyser</Tooltip>, `stopWords` replaces or transforms the <Tooltip tip="Stop words are common words such as 'the' and 'of' that a search engine drops from the index because they match nearly every document.">stop word</Tooltip> set, `tokenizer` swaps in your own implementation, and `bm25` overrides the `k1` and `b` ranking parameters. `surfaceForms` makes [suggestions and prefix completions](/docs/full-text-search) return the original spellings from your documents; the default is `false`, which returns the stemmed index terms. Recording every spelling costs some insert throughput, and the setting persists with the index.

## Insert documents

`insert(indexName, document, docId?)` resolves the document id in a fixed order: an explicit `docId` argument takes precedence, then a string `id` field on the document, and otherwise Narsil generates a UUID v7. The method returns the resolved id. Add two articles to `catalog.ts`, each carrying its own `id`.

```ts title="catalog.ts"

const narsil = await createNarsil()

await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})

await narsil.insert('articles', { // [!code ++:7]
  id: 'scaling-search',
  title: 'Scaling search with partitions',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024,
})

await narsil.insert('articles', { // [!code ++:7]
  id: 'vector-basics',
  title: 'Vector search basics',
  body: 'Dense vectors capture meaning beyond exact keywords.',
  author: { name: 'Kofi Boateng', verified: false },
  publishedYear: 2023,
})
```

Inserting an id that already exists fails with `DOC_ALREADY_EXISTS`. An upsert checks `has()` first and picks the right call; the HTTP server's PUT endpoint wraps that check into one request.

## Load many documents in one call

Inserting one document at a time is clear, but loading a corpus that way sends a separate call per document. `insertBatch(indexName, documents)` takes the whole set in one call and returns partial results: one bad document never aborts the batch, so every success still applies and every failure comes back with its id and error. Swap the two single inserts for a single batch that also loads a third article and one document that fails validation.

```ts title="catalog.ts"

const narsil = await createNarsil()

await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})

await narsil.insert('articles', { // [!code --:7]
  id: 'scaling-search',
  title: 'Scaling search with partitions',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024,
})

await narsil.insert('articles', { // [!code --:7]
  id: 'vector-basics',
  title: 'Vector search basics',
  body: 'Dense vectors capture meaning beyond exact keywords.',
  author: { name: 'Kofi Boateng', verified: false },
  publishedYear: 2023,
})

const result = await narsil.insertBatch('articles', [ // [!code ++:6]
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])

console.log(result) // [!code ++]
```

The three valid documents are added, and the fourth fails because `title` must be a string. `insertBatch` returns the ids that succeeded and, for each failure, the id paired with the `NarsilError` that stopped it:

```ts result
{
  succeeded: [ 'scaling-search', 'vector-basics', 'ranking-explained' ],
  failed: [ { docId: 'broken', error: [NarsilError] } ]
}
```

`console.log` abbreviates the error as `[NarsilError]`. That entry's `error.code` is `DOC_VALIDATION_FAILED`, and `error.details` holds `{ field: 'title', expected: 'string', received: 'number' }`.

Large batches process in chunks and yield the event loop between chunks, so searches remain responsive during a bulk load. `updateBatch` and `removeBatch` follow the same shape for changing and deleting many documents at once.

## Read documents

With the three articles indexed, read them back by id. `get` returns the document, or `undefined` for an unknown id. `getMultiple` returns a `Map` of the ids that exist. `has` tests for presence, and `countDocuments` reports the index size.

```ts title="catalog.ts"

const narsil = await createNarsil()

await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})

const result = await narsil.insertBatch('articles', [
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])

const scaling = await narsil.get('articles', 'scaling-search') // [!code ++:9]
const found = await narsil.getMultiple('articles', ['scaling-search', 'ranking-explained'])
const exists = await narsil.has('articles', 'scaling-search')
const total = await narsil.countDocuments('articles')

console.log(scaling)
console.log(found)
console.log(exists)
console.log(total)
```

```ts result
{
  id: 'scaling-search',
  title: 'Scaling search with partitions',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024
}
Map(2) {
  'scaling-search' => {
    id: 'scaling-search',
    title: 'Scaling search with partitions',
    body: 'Partitioning spreads one index across many workers.',
    author: { name: 'Emma Wright', verified: true },
    publishedYear: 2024
  },
  'ranking-explained' => {
    id: 'ranking-explained',
    title: 'How BM25 ranking works',
    body: 'BM25 balances term frequency against document length.',
    author: { name: 'Sofia Rossi', verified: true },
    publishedYear: 2025
  }
}
true
3
```

The `broken` document fails validation because its `title` is a number, so it never reaches the index and `countDocuments` reports three.

## Page through every document

`get` and `getMultiple` need ids you already have. A query with no term returns no hits, even when it carries filters. Neither path shows you what a freshly loaded index contains. By default, `listDocuments` pages through the stored documents in document-id order without searching. Each page carries a **cursor**. A cursor is an opaque string that marks the id its page stopped at. Leave it out on the first call, and pass it back on every call after that. Stop once the cursor comes back null. Ask for two documents at a time so that the three articles span two pages.

```ts title="catalog.ts"

const narsil = await createNarsil()

await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})

const result = await narsil.insertBatch('articles', [
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])

const scaling = await narsil.get('articles', 'scaling-search')
const found = await narsil.getMultiple('articles', ['scaling-search', 'ranking-explained'])
const exists = await narsil.has('articles', 'scaling-search')
const total = await narsil.countDocuments('articles')

const ids: string[] = [] // [!code ++:9]
let cursor: string | undefined

do {
  const page = await narsil.listDocuments('articles', { limit: 2, cursor })
  console.log(JSON.stringify(page, (k, v) => (typeof v === 'number' ? Number(v.toFixed(2)) : v), 2))
  for (const entry of page.documents) ids.push(entry.id)
  cursor = page.cursor ?? undefined
} while (cursor !== undefined)
```

```json result
{
  "documents": [
    {
      "id": "ranking-explained",
      "document": {
        "id": "ranking-explained",
        "title": "How BM25 ranking works",
        "body": "BM25 balances term frequency against document length.",
        "author": {
          "name": "Sofia Rossi",
          "verified": true
        },
        "publishedYear": 2025
      }
    },
    {
      "id": "scaling-search",
      "document": {
        "id": "scaling-search",
        "title": "Scaling search with partitions",
        "body": "Partitioning spreads one index across many workers.",
        "author": {
          "name": "Emma Wright",
          "verified": true
        },
        "publishedYear": 2024
      }
    }
  ],
  "cursor": "eyJ2IjoxLCJhIjoic2NhbGluZy1zZWFyY2gifQ==",
  "total": 3,
  "elapsed": 0.16
}
{
  "documents": [
    {
      "id": "vector-basics",
      "document": {
        "id": "vector-basics",
        "title": "Vector search basics",
        "body": "Dense vectors capture meaning beyond exact keywords.",
        "author": {
          "name": "Kofi Boateng",
          "verified": false
        },
        "publishedYear": 2023
      }
    }
  ],
  "cursor": null,
  "total": 3,
  "elapsed": 0.08
}
```

The first page holds the two articles whose ids sort first. `total` reports the three documents the listing covers. The page carries two of them. The second page returns the last article with `cursor: null`, which ends the loop. Narsil compares ids by their UTF-16 code units, so `'10'` sorts ahead of `'9'`. Pad numeric ids to a fixed width when you want a listing to follow the numbers.

Writes continue while you page. Narsil guarantees three things about them. A document that stays in the index for the whole listing comes back exactly once. Narsil skips a document you remove part-way through. It returns one you insert part-way through once that document's id sorts above the cursor.

The cursor is that id and nothing more. A client that stops paging leaves nothing to expire. A saved cursor still works after a restart, a [snapshot restore](/docs/persistence-and-durability), and a [rebalance](/docs/partitions-and-workers). Reaching the last page of a large index costs what reaching the first page costs. `limit` defaults to 10 and tops out at 10,000. Narsil raises a value below one to one. A cursor Narsil did not issue fails with `SEARCH_INVALID_CURSOR`. Pass `filters` to list part of an index, `sort` to order it by field value instead of by id, and `document` to cut down what each entry carries. [Browsing an index without a search term](/docs/filters-facets-and-pagination#browse-an-index-without-a-search-term) shows all three.

## Update and remove

`update` replaces the whole document under an id. Internally it removes the old document and inserts the new one, with a fast path when the change touches nothing the index depends on. `remove` deletes a document by id. Both throw `DOC_NOT_FOUND` for an unknown id.

```ts title="catalog.ts"

const narsil = await createNarsil()

await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})

const result = await narsil.insertBatch('articles', [
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])

const scaling = await narsil.get('articles', 'scaling-search')
const found = await narsil.getMultiple('articles', ['scaling-search', 'ranking-explained'])
const exists = await narsil.has('articles', 'scaling-search')
const total = await narsil.countDocuments('articles')

const ids: string[] = []
let cursor: string | undefined

do {
  const page = await narsil.listDocuments('articles', { limit: 2, cursor })
  for (const entry of page.documents) ids.push(entry.id)
  cursor = page.cursor ?? undefined
} while (cursor !== undefined)

await narsil.update('articles', 'scaling-search', { // [!code ++:6]
  title: 'Scaling search with partitions and workers',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024,
})
await narsil.remove('articles', 'vector-basics') // [!code ++]
```

## Manage indexes

These calls inspect the index and then reset it. `clear` removes every document but keeps the index and its schema, while `dropIndex` removes the index entirely, including its persisted data.

```ts title="catalog.ts"

const narsil = await createNarsil()

await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})

const result = await narsil.insertBatch('articles', [
  { id: 'scaling-search', title: 'Scaling search with partitions', body: 'Partitioning spreads one index across many workers.', author: { name: 'Emma Wright', verified: true }, publishedYear: 2024 },
  { id: 'vector-basics', title: 'Vector search basics', body: 'Dense vectors capture meaning beyond exact keywords.', author: { name: 'Kofi Boateng', verified: false }, publishedYear: 2023 },
  { id: 'ranking-explained', title: 'How BM25 ranking works', body: 'BM25 balances term frequency against document length.', author: { name: 'Sofia Rossi', verified: true }, publishedYear: 2025 },
  { id: 'broken', title: 42 },
])

await narsil.update('articles', 'scaling-search', {
  title: 'Scaling search with partitions and workers',
  body: 'Partitioning spreads one index across many workers.',
  author: { name: 'Emma Wright', verified: true },
  publishedYear: 2024,
})
await narsil.remove('articles', 'vector-basics')

const indexes = narsil.listIndexes() // [!code ++:8]
const stats = narsil.getStats('articles')

console.log(indexes)
console.log(stats)

await narsil.clear('articles')
await narsil.dropIndex('articles')
```

```ts result
[
  {
    name: 'articles',
    documentCount: 2,
    partitionCount: 1,
    language: 'english'
  }
]
{
  documentCount: 2,
  partitionCount: 1,
  estimatedMemoryBytes: 5152,
  language: 'english',
  schema: {
    title: 'string',
    body: 'string',
    author: { name: 'string', verified: 'boolean' },
    publishedYear: 'number'
  }
}
```

Call `shutdown()` when the process is done with the engine; it stops workers, flushes durability state, and rejects further calls.

## Handle errors

Every failure throws a `NarsilError` carrying a stable string `code`, a readable `message`, and a `details` object with the values that produced the failure. `errors.ts` builds the same index and inserts a number where the schema declares a string.

```ts title="errors.ts"

const narsil = await createNarsil()

await narsil.createIndex('articles', {
  schema: {
    title: 'string',
    body: 'string',
    author: {
      name: 'string',
      verified: 'boolean',
    },
    publishedYear: 'number',
  },
  language: 'english',
  required: ['title'],
})

try {
  await narsil.insert('articles', { title: 42 })
} catch (error) {
  if (error instanceof NarsilError && error.code === ErrorCodes.DOC_VALIDATION_FAILED) {
    console.error(error.message, error.details)
  }
}
```

```txt result open
Field "title" expected string, got number { field: 'title', expected: 'string', received: 'number' }
```

The full set of codes is exported as `ErrorCodes`, covering index lifecycle, document validation, search input, vectors, embeddings, partitions, and configuration.