Core concepts

Indexes and documents

Declare a typed schema, store documents under their ids, read them back, page through an index in id order, and change or remove a document.

An index holds your documents under a schema. The engine checks each document against that schema when you write it, so a value of the wrong type fails the write. You build one file here, catalogue.ts, in which you declare a bicycle catalogue's schema, store the bicycles, read them back, page through them, and change two of them. Each code block below holds the whole file, with green lines marking what each step adds and struck lines marking what it takes away.

Declare the schema

The schema maps each field to a type or, for a nested object, to another schema, which the engine accepts up to four levels deep. required names the fields that every document has to hold, while with strict: true the engine rejects a document holding a field that the schema never declares.

catalogue.ts
import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('bicycles', {
  schema: {
    name: 'string',
    summary: 'string',
    maker: { name: 'string', country: 'enum' },
    category: 'enum',
    price: 'number',
    inStock: 'boolean',
    tags: 'string[]',
  },
  required: ['name', 'price'],
  strict: true,
})
 
await narsil.shutdown()

A fifth level of nesting fails createIndex with SCHEMA_DEPTH_EXCEEDED. The other index options change how the engine analyses and scores text. language picks the and defaults to english, stopWords and tokenizer replace that analyser's and tokeniser, and bm25 sets the ranking's k1 and b, which default to 1.2 and 0.75. surfaceForms, which defaults to true, records each spelling that the stemmer changes, so that suggestions and prefix completions return words as your documents spell them.

Store documents under their ids

insert stores one document and returns its id. The engine takes that id from the document's own id field, or from a third docId argument where the document has no id, and it generates a UUID v7 where you give neither. An insert whose docId contradicts the document's own id fails with DOC_VALIDATION_FAILED, because the stored document would then hold a different id from the key that the engine stores it under. An insert under an id that the index already holds fails with DOC_ALREADY_EXISTS. insertBatch stores many documents in one call and reports each failure beside the successes.

catalogue.ts
import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('bicycles', {
  schema: {
    name: 'string',
    summary: 'string',
    maker: { name: 'string', country: 'enum' },
    category: 'enum',
    price: 'number',
    inStock: 'boolean',
    tags: 'string[]',
  },
  required: ['name', 'price'],
  strict: true,
})
 
await narsil.insert('bicycles', { 
  id: 'urban-commuter',
  name: 'Gomes Urban commuter',
  summary: 'A lightweight aluminium commuter bicycle with hydraulic disc brakes.',
  maker: { name: 'Gomes Cycles', country: 'pt' },
  category: 'city',
  price: 749,
  inStock: true,
  tags: ['commuter', 'aluminium'],
})
 
const loaded = await narsil.insertBatch('bicycles', [
  {
    id: 'gravel-explorer',
    name: 'Mori Gravel explorer',
    summary: 'A steel gravel bicycle built for long unpaved rides.',
    maker: { name: 'Mori Works', country: 'jp' },
    category: 'gravel',
    price: 1299,
    inStock: true,
    tags: ['gravel', 'steel'],
  },
  {
    id: 'folding-metro',
    name: 'Obi Metro folding bicycle',
    summary: 'A folding bicycle that fits under a train seat.',
    maker: { name: 'Obi Mobility', country: 'ng' },
    category: 'folding',
    price: 899,
    inStock: false,
    tags: ['folding', 'commuter'],
  },
  {
    id: 'cargo-hauler',
    name: 'Larsson Cargo hauler',
    summary: 'An electric cargo bicycle rated to carry two children.',
    maker: { name: 'Larsson Bikes', country: 'se' },
    category: 'cargo',
    price: 3499,
    inStock: true,
    tags: ['cargo', 'electric'],
  },
  {
    id: 'road-racer',
    name: 'Khoury Road racer',
    summary: 'A carbon road bicycle with electronic shifting.',
    maker: { name: 'Khoury Composites', country: 'lb' },
    category: 'road',
    price: '4250',
    inStock: true,
    tags: ['road', 'carbon'],
  },
])
 
console.log(
  JSON.stringify(
    {
      succeeded: loaded.succeeded,
      failed: loaded.failed.map(({ docId, error }) => ({ docId, code: error.code, details: error.details })),
    },
    null,
    2
  )
)
 
await narsil.shutdown()
{
  "succeeded": [
    "gravel-explorer",
    "folding-metro",
    "cargo-hauler"
  ],
  "failed": [
    {
      "docId": "road-racer",
      "code": "DOC_VALIDATION_FAILED",
      "details": {
        "field": "price",
        "expected": "number",
        "received": "string"
      }
    }
  ]
}

The road racer holds price as the string '4250', so the engine rejects that one document and stores the other three. Its details hold the field, the type that the schema expects, and the type in the document. The engine applies a batch 1,000 documents at a time and yields to the event loop between those chunks, so it can answer a query between two chunks of a large batch. updateBatch takes a list of { docId, document } pairs and removeBatch takes a list of ids, and both return the same { succeeded, failed } shape.

Read documents back by id

get returns the stored document, or undefined for an id that the index lacks. getMultiple returns a Map with an entry for each id that the index holds, while has returns a boolean and countDocuments returns the number of documents in the index. Here the program prints those four answers in place of the batch report.

catalogue.ts
import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('bicycles', {
  schema: {
    name: 'string',
    summary: 'string',
    maker: { name: 'string', country: 'enum' },
    category: 'enum',
    price: 'number',
    inStock: 'boolean',
    tags: 'string[]',
  },
  required: ['name', 'price'],
  strict: true,
})
 
await narsil.insert('bicycles', {
  id: 'urban-commuter',
  name: 'Gomes Urban commuter',
  summary: 'A lightweight aluminium commuter bicycle with hydraulic disc brakes.',
  maker: { name: 'Gomes Cycles', country: 'pt' },
  category: 'city',
  price: 749,
  inStock: true,
  tags: ['commuter', 'aluminium'],
})
 
const loaded = await narsil.insertBatch('bicycles', [ 
await narsil.insertBatch('bicycles', [ 
  {
    id: 'gravel-explorer',
    name: 'Mori Gravel explorer',
    summary: 'A steel gravel bicycle built for long unpaved rides.',
    maker: { name: 'Mori Works', country: 'jp' },
    category: 'gravel',
    price: 1299,
    inStock: true,
    tags: ['gravel', 'steel'],
  },
  {
    id: 'folding-metro',
    name: 'Obi Metro folding bicycle',
    summary: 'A folding bicycle that fits under a train seat.',
    maker: { name: 'Obi Mobility', country: 'ng' },
    category: 'folding',
    price: 899,
    inStock: false,
    tags: ['folding', 'commuter'],
  },
  {
    id: 'cargo-hauler',
    name: 'Larsson Cargo hauler',
    summary: 'An electric cargo bicycle rated to carry two children.',
    maker: { name: 'Larsson Bikes', country: 'se' },
    category: 'cargo',
    price: 3499,
    inStock: true,
    tags: ['cargo', 'electric'],
  },
  {
    id: 'road-racer',
    name: 'Khoury Road racer',
    summary: 'A carbon road bicycle with electronic shifting.',
    maker: { name: 'Khoury Composites', country: 'lb' },
    category: 'road',
    price: '4250',
    inStock: true,
    tags: ['road', 'carbon'],
  },
])
 
console.log(
  JSON.stringify(
    {
      succeeded: loaded.succeeded, 
      failed: loaded.failed.map(({ docId, error }) => ({ docId, code: error.code, details: error.details })), 
      commuter: await narsil.get('bicycles', 'urban-commuter'), 
      found: [...(await narsil.getMultiple('bicycles', ['gravel-explorer', 'road-racer'])).keys()],
      hasCargo: await narsil.has('bicycles', 'cargo-hauler'),
      total: await narsil.countDocuments('bicycles'),
    },
    null,
    2
  )
)
 
await narsil.shutdown()
{
  "commuter": {
    "id": "urban-commuter",
    "name": "Gomes Urban commuter",
    "summary": "A lightweight aluminium commuter bicycle with hydraulic disc brakes.",
    "maker": {
      "name": "Gomes Cycles",
      "country": "pt"
    },
    "category": "city",
    "price": 749,
    "inStock": true,
    "tags": [
      "commuter",
      "aluminium"
    ]
  },
  "found": [
    "gravel-explorer"
  ],
  "hasCargo": true,
  "total": 4
}

The engine holds no road racer, so getMultiple returns one entry for the two ids in the call, and countDocuments reports the four bicycles that the index holds.

Page through every document

A query with no term matches nothing, even under a filter, so listDocuments is the call that shows everything that an index holds. It pages through the stored documents in id order without ranking them. Each page comes with a cursor that you pass back to reach the next page, until the cursor comes back null. document.include cuts each entry down to the fields that you name, while each entry still holds its id beside them.

catalogue.ts
import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('bicycles', {
  schema: {
    name: 'string',
    summary: 'string',
    maker: { name: 'string', country: 'enum' },
    category: 'enum',
    price: 'number',
    inStock: 'boolean',
    tags: 'string[]',
  },
  required: ['name', 'price'],
  strict: true,
})
 
await narsil.insert('bicycles', {
  id: 'urban-commuter',
  name: 'Gomes Urban commuter',
  summary: 'A lightweight aluminium commuter bicycle with hydraulic disc brakes.',
  maker: { name: 'Gomes Cycles', country: 'pt' },
  category: 'city',
  price: 749,
  inStock: true,
  tags: ['commuter', 'aluminium'],
})
 
await narsil.insertBatch('bicycles', [
  {
    id: 'gravel-explorer',
    name: 'Mori Gravel explorer',
    summary: 'A steel gravel bicycle built for long unpaved rides.',
    maker: { name: 'Mori Works', country: 'jp' },
    category: 'gravel',
    price: 1299,
    inStock: true,
    tags: ['gravel', 'steel'],
  },
  {
    id: 'folding-metro',
    name: 'Obi Metro folding bicycle',
    summary: 'A folding bicycle that fits under a train seat.',
    maker: { name: 'Obi Mobility', country: 'ng' },
    category: 'folding',
    price: 899,
    inStock: false,
    tags: ['folding', 'commuter'],
  },
  {
    id: 'cargo-hauler',
    name: 'Larsson Cargo hauler',
    summary: 'An electric cargo bicycle rated to carry two children.',
    maker: { name: 'Larsson Bikes', country: 'se' },
    category: 'cargo',
    price: 3499,
    inStock: true,
    tags: ['cargo', 'electric'],
  },
  {
    id: 'road-racer',
    name: 'Khoury Road racer',
    summary: 'A carbon road bicycle with electronic shifting.',
    maker: { name: 'Khoury Composites', country: 'lb' },
    category: 'road',
    price: '4250',
    inStock: true,
    tags: ['road', 'carbon'],
  },
])
 
console.log( 
  JSON.stringify( 
    { 
      commuter: await narsil.get('bicycles', 'urban-commuter'), 
      found: [...(await narsil.getMultiple('bicycles', ['gravel-explorer', 'road-racer'])).keys()], 
      hasCargo: await narsil.has('bicycles', 'cargo-hauler'), 
      total: await narsil.countDocuments('bicycles'), 
    }, 
    null, 
    2
  ) 
) 
let cursor: string | undefined
do {
  const page = await narsil.listDocuments('bicycles', { limit: 2, cursor, document: { include: ['name', 'price'] } })
  console.log(JSON.stringify(page, (key, value) => (key === 'elapsed' ? undefined : value), 2))
  cursor = page.cursor ?? undefined
} while (cursor !== undefined)
 
await narsil.shutdown()
{
  "documents": [
    {
      "id": "cargo-hauler",
      "document": {
        "name": "Larsson Cargo hauler",
        "price": 3499
      }
    },
    {
      "id": "folding-metro",
      "document": {
        "name": "Obi Metro folding bicycle",
        "price": 899
      }
    }
  ],
  "cursor": "eyJ2Ijo0LCJhIjoiZm9sZGluZy1tZXRybyIsInEiOiIwNTUzN2EwNyJ9",
  "total": 4
}
{
  "documents": [
    {
      "id": "gravel-explorer",
      "document": {
        "name": "Mori Gravel explorer",
        "price": 1299
      }
    },
    {
      "id": "urban-commuter",
      "document": {
        "name": "Gomes Urban commuter",
        "price": 749
      }
    }
  ],
  "cursor": null,
  "total": 4
}

The engine compares ids by their UTF-16 code units, which puts cargo-hauler first and urban-commuter last, and which orders the ids 9, 10, 2, and 100 as 10, 100, 2, and 9. Pad a numeric id to a fixed width where you want a listing to follow the numbers. total counts the four documents that the listing covers. limit defaults to 10 and takes a value from 1 to 10,000, where the engine reads a value below 1 as 1 and fails a value above 10,000 with SEARCH_RESULT_WINDOW_EXCEEDED.

Writes can continue while you page, in which case three rules hold for the listing. A document that stays in the index for the whole listing comes back exactly once, a document that you remove ahead of the cursor stays out of the listing, and a document that you insert comes back only where its id sorts after the cursor. A cursor that the engine did not issue fails with SEARCH_INVALID_CURSOR.

Change a document, then inspect the index

update replaces the whole document under an id, so the new version repeats every field that you want to keep, id included. remove deletes a document by id, and both calls fail with DOC_NOT_FOUND for an id that the index lacks. listIndexes and getStats then report what the index holds.

catalogue.ts
import { createNarsil } from '@delali/narsil'
 
const narsil = await createNarsil()
 
await narsil.createIndex('bicycles', {
  schema: {
    name: 'string',
    summary: 'string',
    maker: { name: 'string', country: 'enum' },
    category: 'enum',
    price: 'number',
    inStock: 'boolean',
    tags: 'string[]',
  },
  required: ['name', 'price'],
  strict: true,
})
 
await narsil.insert('bicycles', {
  id: 'urban-commuter',
  name: 'Gomes Urban commuter',
  summary: 'A lightweight aluminium commuter bicycle with hydraulic disc brakes.',
  maker: { name: 'Gomes Cycles', country: 'pt' },
  category: 'city',
  price: 749,
  inStock: true,
  tags: ['commuter', 'aluminium'],
})
 
await narsil.insertBatch('bicycles', [
  {
    id: 'gravel-explorer',
    name: 'Mori Gravel explorer',
    summary: 'A steel gravel bicycle built for long unpaved rides.',
    maker: { name: 'Mori Works', country: 'jp' },
    category: 'gravel',
    price: 1299,
    inStock: true,
    tags: ['gravel', 'steel'],
  },
  {
    id: 'folding-metro',
    name: 'Obi Metro folding bicycle',
    summary: 'A folding bicycle that fits under a train seat.',
    maker: { name: 'Obi Mobility', country: 'ng' },
    category: 'folding',
    price: 899,
    inStock: false,
    tags: ['folding', 'commuter'],
  },
  {
    id: 'cargo-hauler',
    name: 'Larsson Cargo hauler',
    summary: 'An electric cargo bicycle rated to carry two children.',
    maker: { name: 'Larsson Bikes', country: 'se' },
    category: 'cargo',
    price: 3499,
    inStock: true,
    tags: ['cargo', 'electric'],
  },
  {
    id: 'road-racer',
    name: 'Khoury Road racer',
    summary: 'A carbon road bicycle with electronic shifting.',
    maker: { name: 'Khoury Composites', country: 'lb' },
    category: 'road',
    price: '4250',
    inStock: true,
    tags: ['road', 'carbon'],
  },
])
 
let cursor: string | undefined
do { 
  const page = await narsil.listDocuments('bicycles', { limit: 2, cursor, document: { include: ['name', 'price'] } }) 
  console.log(JSON.stringify(page, (key, value) => (key === 'elapsed' ? undefined : value), 2)) 
  cursor = page.cursor ?? undefined
} while (cursor !== undefined) 
await narsil.update('bicycles', 'folding-metro', { 
  id: 'folding-metro',
  name: 'Obi Metro folding bicycle',
  summary: 'A folding bicycle that fits under a train seat and unfolds in fifteen seconds.',
  maker: { name: 'Obi Mobility', country: 'ng' },
  category: 'folding',
  price: 949,
  inStock: true,
  tags: ['folding', 'commuter'],
})
await narsil.remove('bicycles', 'cargo-hauler')
 
console.log(JSON.stringify({ indexes: narsil.listIndexes(), stats: narsil.getStats('bicycles') }, null, 2))
 
await narsil.shutdown()
{
  "indexes": [
    {
      "name": "bicycles",
      "documentCount": 3,
      "partitionCount": 1,
      "language": "english",
      "state": "open",
      "reopenCount": 0
    }
  ],
  "stats": {
    "documentCount": 3,
    "partitionCount": 1,
    "estimatedMemoryBytes": 14645,
    "language": "english",
    "schema": {
      "name": "string",
      "summary": "string",
      "maker": {
        "name": "string",
        "country": "enum"
      },
      "category": "enum",
      "price": "number",
      "inStock": "boolean",
      "tags": "string[]"
    }
  }
}

The update and the removal leave three bicycles. state is open while the index is in memory and closed once the engine releases it. reopenCount counts the loads since the engine started, and persistence and durability covers when the engine releases an index and loads it again. estimatedMemoryBytes is the engine's own estimate of the memory that the index occupies. clear empties an index and keeps its schema, while dropIndex deletes the index along with its schema.

Call shutdown() before the process exits, because the engine flushes pending writes, ends its workers, and releases its adapters there. Once it starts, the engine fails every further call with INSTANCE_SHUT_DOWN.

Where to go next

  • Full-text search covers the ranking, the typo tolerance, the prefix completion, and the suggestions whose spelling surfaceForms sets.
  • Filters and facets covers the filter expressions that a query and a listing share.
  • Sorting, grouping, and pagination covers the sort order, the grouped results, and the cursor that pages through ranked results.