Core concepts

Language support

Load a language module, index text in French and Chinese, name your own tokeniser and stop words, write a language of your own, and rebuild an index when a language changes.

A language module holds the rules for splitting a language's text into words, dropping the commonest ones, and reducing each word to its , so that a search for one form of a word finds the others. You build articles.ts here, in which you index French and Chinese articles. You then use four short files to name a tokeniser and a stop word set, write a language of your own, and rebuild an index after a language changes. Each code block below holds a whole file, with green lines marking what each step adds and struck lines marking what it takes away.

Load a language and index in it

Every language except English comes from its own entry point, such as @delali/narsil/languages/french, so an application loads only the languages that it uses. registerLanguage makes a module's name valid for any index, while an index that names a language that you never registered fails with LANGUAGE_NOT_SUPPORTED. English is registered already, which is why an index can leave language out.

articles.ts
import { createNarsil, registerLanguage } from '@delali/narsil'
import { french } from '@delali/narsil/languages/french'
 
registerLanguage(french)
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles-fr', {
  schema: { title: 'string', body: 'string' },
  language: 'french',
})
 
await narsil.insertBatch('articles-fr', [
  { id: 'recherche-vectorielle', title: 'La recherche vectorielle expliquée', body: 'Les vecteurs capturent le sens au-delà des mots exacts.' },
  { id: 'cafe-de-quartier', title: 'Un café pour chaque quartier', body: 'Le livreur traverse la ville à vélo avec des cafés chauds.' },
])
 
const countsFor = async (indexName: string, terms: string[]): Promise<Record<string, number>> => {
  const counts: Record<string, number> = {}
  for (const term of terms) counts[term] = (await narsil.query(indexName, { term, limit: 0 })).count
  return counts
}
 
console.log(JSON.stringify(await countsFor('articles-fr', ['vecteur', 'vectoriel', 'cafe', 'vélo', 'velo', 'le']), null, 2))
 
await narsil.shutdown()
{
  "vecteur": 1,
  "vectoriel": 1,
  "cafe": 1,
  "vélo": 1,
  "velo": 0,
  "le": 0
}

The French stemmer reduces vecteurs to vecteur and vectorielle to vectoriel, so a search for either form finds the article, while the index drops le as a stop word. No bundled module folds accents, however. A search for cafe finds café only because the stemmer strips the final é as an ending, while the same stemmer keeps the é in vélo, so a search for velo misses the article that holds vélo. The Spanish stemmer keeps ñ in the same way, so a Spanish index keeps montaña and montana apart.

Index a language written without spaces

The modules for Chinese, Japanese, Korean, Thai, Lao, Khmer, and Burmese cut every run of script into overlapping pairs of characters in place of words, because writers of most of those languages put no spaces between words. You create articles-zh on the same engine as the French index, and each index keeps its own language.

articles.ts
import { createNarsil, registerLanguage } from '@delali/narsil'
import { chinese } from '@delali/narsil/languages/chinese'
import { french } from '@delali/narsil/languages/french'
 
registerLanguage(french)
registerLanguage(chinese) 
 
const narsil = await createNarsil()
 
await narsil.createIndex('articles-fr', {
  schema: { title: 'string', body: 'string' },
  language: 'french',
})
 
await narsil.insertBatch('articles-fr', [
  { id: 'recherche-vectorielle', title: 'La recherche vectorielle expliquée', body: 'Les vecteurs capturent le sens au-delà des mots exacts.' },
  { id: 'cafe-de-quartier', title: 'Un café pour chaque quartier', body: 'Le livreur traverse la ville à vélo avec des cafés chauds.' },
])
 
await narsil.createIndex('articles-zh', { 
  schema: { title: 'string', body: 'string' },
  language: 'chinese',
})
 
await narsil.insert('articles-zh', {
  id: 'shenzhen-metro',
  title: '深圳地铁票价查询',
  body: '乘客可以在车站的自助机上查询票价。',
})
 
const countsFor = async (indexName: string, terms: string[]): Promise<Record<string, number>> => {
  const counts: Record<string, number> = {}
  for (const term of terms) counts[term] = (await narsil.query(indexName, { term, limit: 0 })).count
  return counts
}
 
console.log(JSON.stringify(await countsFor('articles-fr', ['vecteur', 'vectoriel', 'cafe', 'vélo', 'velo', 'le']), null, 2)) 
console.log(JSON.stringify(await countsFor('articles-zh', ['地铁', '铁票', '深圳地铁', '地', '票价']), null, 2)) 
 
await narsil.shutdown()
{
  "地铁": 1,
  "铁票": 1,
  "深圳地铁": 1,
  "地": 0,
  "票价": 1
}

The engine indexes every overlapping pair, so 地铁 and 票价 match, and so does 铁票, a pair that crosses the boundary between two words. A longer term matches where every pair inside it appears, which is why 深圳地铁 matches too, while a single character such as 地 matches nothing, because the engine never indexes one on its own.

One index applies one language to all of its text. The French analyser splits text on every character outside the letters that French uses, so a French index drops the Chinese in a document that mixes the two, and a search for 深圳 on it matches nothing. The English analyser keeps the letters of every script, which makes an English index the safer choice for mixed text, although it matches a Chinese run only as a whole word. The Chinese analyser splits a Latin word at an accented letter in its turn, so a Chinese index finds café under café and under caf but misses it under cafe.

See which languages come with Narsil

Narsil includes 107 language modules. Thirty-seven of them split words, drop stop words, and stem: Arabic, Armenian, Basque, Bulgarian, Catalan, Czech, Danish, Dutch, English, Esperanto, Estonian, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Lithuanian, Nepali, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Sanskrit, Serbian, Slovenian, Spanish, Swahili, Swedish, Tamil, Turkish, and Ukrainian. The seven languages written without spaces use the overlapping pairs above, and the other 63 split words and drop stop words with no stemmer. Twenty-one modules also hold a normaliser, which folds the spellings that a writing system leaves optional before the stemmer reduces a word. Load any of them the way that articles.ts loads French and Chinese.

Name a tokeniser and a stop word set

A tokeniser or a stop word override is code, which a checkpoint cannot store, so a durable engine refuses an inline tokeniser or a stop word function with CONFIG_INVALID, and its message names the register call to use. registerTokenizer and registerStopWords store an implementation under a name, and an index names it where the inline form would go. In catalogue.ts, the tokeniser on the parts index splits on spaces alone, which keeps a product code whole, while the engine drops refurbished from searches on the listings index.

catalogue.ts
import { createNarsil, registerStopWords, registerTokenizer } from '@delali/narsil'
 
registerTokenizer('product-codes', {
  tokenize: (text) =>
    text
      .toLowerCase()
      .split(/\s+/)
      .filter((token) => token.length > 0)
      .map((token, position) => ({ token, position })),
})
 
registerStopWords('catalogue-noise', (defaults) => new Set([...defaults, 'refurbished']))
 
const narsil = await createNarsil()
 
await narsil.createIndex('parts', { schema: { code: 'string' }, tokenizer: 'product-codes' })
await narsil.createIndex('listings', { schema: { title: 'string' }, stopWords: 'catalogue-noise' })
 
await narsil.insert('parts', { id: 'brake-set', code: 'SKU-4821 hydraulic brake set' })
await narsil.insert('listings', { id: 'folding-metro', title: 'Refurbished folding bicycle' })
 
const count = async (indexName: string, term: string): Promise<number> =>
  (await narsil.query(indexName, { term, limit: 0 })).count
 
console.log(
  JSON.stringify(
    {
      wholeCode: await count('parts', 'sku-4821'),
      codeFragment: await count('parts', '4821'),
      droppedWord: await count('listings', 'refurbished'),
      keptWord: await count('listings', 'folding'),
    },
    null,
    2
  )
)
 
await narsil.shutdown()
{
  "wholeCode": 1,
  "codeFragment": 0,
  "droppedWord": 0,
  "keptWord": 1
}

A search on the parts index finds the whole code and misses the fragment 4821, because the tokeniser keeps SKU-4821 as one token, where the default English analyser splits the code at the hyphen so that both searches match. A custom tokeniser replaces the whole pipeline, so the engine stems nothing and drops no stop words on that index, which means that a whitespace tokeniser finds a document holding running under running and misses it under run. Naming a tokeniser or a stop word set that you never registered fails with CONFIG_INVALID, and the error's details.available lists the names that you did register.

Write a language of your own

registerLanguage takes any object that fits the exported LanguageModule type, which holds a name, a revision, a stemmer or null, and a stopWords set, with an optional normalizer and tokenizer. The shortest module extends a bundled one, so in legal.ts you keep the English stemmer and add three words of legal boilerplate to the English stop words.

legal.ts
import { createNarsil, type LanguageModule, registerLanguage } from '@delali/narsil'
import { english } from '@delali/narsil/languages/english'
 
const englishLegal: LanguageModule = {
  ...english,
  name: 'english-legal',
  revision: '1',
  stopWords: new Set([...english.stopWords, 'herein', 'thereof', 'whereas']),
}
 
registerLanguage(englishLegal)
 
const narsil = await createNarsil()
 
await narsil.createIndex('contracts', { schema: { clause: 'string' }, language: 'english-legal' })
 
await narsil.insert('contracts', {
  id: 'clause-12',
  clause: 'Whereas the supplier delivers the goods herein described, payment falls due within thirty days.',
})
 
const count = async (term: string): Promise<number> => (await narsil.query('contracts', { term, limit: 0 })).count
 
console.log(JSON.stringify({ whereas: await count('whereas'), supplier: await count('supplier'), delivering: await count('delivering') }, null, 2))
 
await narsil.shutdown()
{
  "whereas": 0,
  "supplier": 1,
  "delivering": 1
}

Whereas matches nothing, because the index drops it as a stop word, while delivering still finds delivers through the English stemmer. Give a new module the revision '1', and change that string whenever you change its stemmer, normaliser, stop words, or tokeniser rules, because the engine compares that string to find the indexes built under the old rules.

Find the indexes that a language change makes stale

The engine records with each index the revision under which it built that index's terms, and when a durable engine opens the index, it compares that record with the revision that the module carries now. A difference marks the index stale. An upgrade of @delali/narsil that corrects a stemmer changes that module's revision, so the engine marks stale every index built before the upgrade. In revisions.ts you build an index under revision '1', shut the engine down, and open it again under revision '2', which is what such an upgrade does to a bundled module. With rebuild: 'manual' the engine holds its own rebuild back so that the stale state stays visible.

revisions.ts
import { mkdirSync, rmSync } from 'node:fs'
import { createNarsil, registerLanguage } from '@delali/narsil'
import { swahili } from '@delali/narsil/languages/swahili'
 
rmSync('./narsil-revisions', { recursive: true, force: true })
mkdirSync('./narsil-revisions', { recursive: true })
 
registerLanguage({ ...swahili, name: 'swahili-news', revision: '1' })
 
const first = await createNarsil({ durability: { directory: './narsil-revisions' } })
 
await first.createIndex('habari', {
  schema: { title: 'string', body: 'string' },
  language: 'swahili-news',
})
 
await first.insert('habari', {
  id: 'soko-la-mombasa',
  title: 'Soko la Mombasa lafunguliwa upya',
  body: 'Wafanyabiashara wa samaki warudi sokoni baada ya ukarabati.',
})
 
await first.checkpoint('habari')
await first.shutdown()
 
registerLanguage({ ...swahili, name: 'swahili-news', revision: '2' })
 
const second = await createNarsil({
  durability: { directory: './narsil-revisions' },
  analysis: { rebuild: 'manual' },
})
 
const stale = await second.query('habari', { term: 'soko' })
 
console.log(JSON.stringify({ indexes: second.listIndexes(), stale: stale.analysisStale }, null, 2))
 
await second.shutdown()
{
  "indexes": [
    {
      "name": "habari",
      "documentCount": 1,
      "partitionCount": 1,
      "language": "swahili-news",
      "state": "open",
      "reopenCount": 0,
      "analysisStale": true
    }
  ],
  "stale": true
}

A stale index keeps answering from the terms that it holds, while the engine marks each result with analysisStale: true, the same flag that listIndexes reports for the index.

Rebuild the terms

The engine rebuilds a stale index in the background by default. Under rebuild: 'manual' it leaves every stale index as it is, and rebuildAnalysis(indexName) resolves once every partition of one index holds current terms, while the analysisRebuild event reports each rebuild as it starts and as it completes.

revisions.ts
import { mkdirSync, rmSync } from 'node:fs'
import { createNarsil, registerLanguage } from '@delali/narsil'
import { swahili } from '@delali/narsil/languages/swahili'
 
rmSync('./narsil-revisions', { recursive: true, force: true })
mkdirSync('./narsil-revisions', { recursive: true })
 
registerLanguage({ ...swahili, name: 'swahili-news', revision: '1' })
 
const first = await createNarsil({ durability: { directory: './narsil-revisions' } })
 
await first.createIndex('habari', {
  schema: { title: 'string', body: 'string' },
  language: 'swahili-news',
})
 
await first.insert('habari', {
  id: 'soko-la-mombasa',
  title: 'Soko la Mombasa lafunguliwa upya',
  body: 'Wafanyabiashara wa samaki warudi sokoni baada ya ukarabati.',
})
 
await first.checkpoint('habari')
await first.shutdown()
 
registerLanguage({ ...swahili, name: 'swahili-news', revision: '2' })
 
const second = await createNarsil({
  durability: { directory: './narsil-revisions' },
  analysis: { rebuild: 'manual' },
})
 
const progress: string[] = [] 
second.on('analysisRebuild', (event) => {
  progress.push(`${event.status} ${event.partitionsRebuilt}/${event.partitionCount}`)
})
 
const stale = await second.query('habari', { term: 'soko' })
 
for (const index of second.listIndexes()) { 
  if (index.analysisStale) await second.rebuildAnalysis(index.name)
}
 
const current = await second.query('habari', { term: 'soko' })
 
console.log(JSON.stringify({ indexes: second.listIndexes(), stale: stale.analysisStale }, null, 2)) 
console.log( 
  JSON.stringify(
    {
      beforeRebuild: { count: stale.count, analysisStale: stale.analysisStale },
      progress,
      afterRebuild: { count: current.count, analysisStale: current.analysisStale },
    },
    null,
    2
  )
)
 
await second.shutdown()
{
  "beforeRebuild": {
    "count": 1,
    "analysisStale": true
  },
  "progress": [
    "started 0/1",
    "completed 1/1"
  ],
  "afterRebuild": {
    "count": 1
  }
}

The search finds the article before and after the rebuild, and analysisStale is absent from the result once the index holds current terms, because the engine sets the flag only while stale terms remain. A durable engine writes the new revision into the index when the rebuild completes, so a later start finds the index current.

Hear about a stale index at start-up

The engine calls onStaleAnalysis once for each stale index that it finds when it opens a directory, and the callback is the one place where the stored and the current revision appear together. Its second argument starts the rebuild for that index. In stale-watch.ts you open the directory that revisions.ts wrote under a third revision, so run revisions.ts first.

stale-watch.ts
import { createNarsil, registerLanguage } from '@delali/narsil'
import { swahili } from '@delali/narsil/languages/swahili'
 
registerLanguage({ ...swahili, name: 'swahili-news', revision: '3' })
 
const narsil = await createNarsil({
  durability: { directory: './narsil-revisions' },
  analysis: {
    rebuild: 'manual',
    onStaleAnalysis: (index) => {
      console.log(`${index.indexName} was built with ${index.storedRevision}, ${index.language} now reports ${index.currentRevision}`)
    },
  },
})
 
await narsil.shutdown()
habari was built with 2, swahili-news now reports 3

The engine reports that the index holds terms from revision 2, which the rebuild in revisions.ts wrote, while the module now carries revision 3.

Where to go next