Narsil includes language modules for tokenisation, stemming, and stop word removal. Each module is a separate entry point, so your bundle includes only the languages you use. This page builds one file, articles.ts, step by step. You load a language, create an index that names it, add a second index in another language, and override the stop words for one index. A second file then registers a tokeniser and a stop word set under names that indexes reference.
Load a language
A language module exports a language object you pass to registerLanguage. Register it once and its name becomes valid for any index. English is always available, so an English index requires no import or registration. Start articles.ts with a French module and an index that names French.
import { french } from '@delali/narsil/languages/french'
import { createNarsil, registerLanguage } from '@delali/narsil'
const narsil = await createNarsil()
registerLanguage(french)
await narsil.createIndex('articles', {
schema: { title: 'string', body: 'string' },
language: 'french',
})Naming a language that was never registered fails with LANGUAGE_NOT_SUPPORTED. The language option selects the analyser Narsil uses to tokenise and stem every text field on the index, so a French index stems chercher, cherche, and cherchons to the same root and drops French stop words such as le and des.
Index documents in the chosen language
With the module loaded and the index created, add two French articles. Narsil tokenises and stems each text field on the way in, using the French analyser the index selected.
import { french } from '@delali/narsil/languages/french'
import { createNarsil, registerLanguage } from '@delali/narsil'
const narsil = await createNarsil()
registerLanguage(french)
await narsil.createIndex('articles', {
schema: { title: 'string', body: 'string' },
language: 'french',
})
await narsil.insert('articles', {
id: 'recherche-vectorielle',
title: 'La recherche vectorielle expliquée',
body: 'Les vecteurs denses capturent le sens au-delà des mots exacts.',
})
await narsil.insert('articles', {
id: 'partitions',
title: 'Partitionner un index',
body: 'Le partitionnement répartit un index sur plusieurs processus.',
})A French query for vecteurs matches the singular vecteur and the verb form vectorielle, because the stemmer reduces them to the same root before ranking.
Add an index in another language
Each index names its own analyser, so a Swahili index works alongside the French one. A created index keeps the language it was made with, so to add Swahili you register the module and create a second index that names it. Register the Swahili module, create articles-sw, and insert a Swahili document.
import { french } from '@delali/narsil/languages/french'
import { swahili } from '@delali/narsil/languages/swahili'
import { createNarsil, registerLanguage } from '@delali/narsil'
const narsil = await createNarsil()
registerLanguage(french)
registerLanguage(swahili)
await narsil.createIndex('articles', {
schema: { title: 'string', body: 'string' },
language: 'french',
})
await narsil.insert('articles', {
id: 'recherche-vectorielle',
title: 'La recherche vectorielle expliquée',
body: 'Les vecteurs denses capturent le sens au-delà des mots exacts.',
})
await narsil.insert('articles', {
id: 'partitions',
title: 'Partitionner un index',
body: 'Le partitionnement répartit un index sur plusieurs processus.',
})
await narsil.createIndex('articles-sw', {
schema: { title: 'string', body: 'string' },
language: 'swahili',
})
await narsil.insert('articles-sw', {
id: 'utafutaji-wa-vekta',
title: 'Utafutaji wa vekta umefafanuliwa',
body: 'Vekta hunasa maana zaidi ya maneno kamili.',
})Swahili has a full stemmer, so the Swahili index stems and filters stop words the same way the French one does. Each index keeps its own analyser, and the schema, inserts, and queries share the same shape whichever language a field uses.
Coverage
Narsil includes 107 language modules. Thirty-seven of them tokenise, stem, and filter stop words: Arabic, Armenian, Basque, Bulgarian, Catalan, Czech, Danish, Dutch, English, Esperanto, Estonian, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Lithuanian, Nepali, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Sanskrit, Serbian, Slovenian, Spanish, Swahili, Swedish, Tamil, Turkish, and Ukrainian.
Thirty-two of those stemmers come from the Snowball reference sources, and each one reproduces Snowball's own output on all 11.4 million word pairs Snowball publishes. The Bulgarian, Sanskrit, Slovenian, Swahili, and Ukrainian stemmers were written for Narsil, because Snowball covers none of those five.
Burmese, Chinese (Mandarin), Japanese, Khmer, Korean, Lao, and Thai write words with no space between them, so those seven modules use a with stop words, which cuts each run of script into overlapping two-character n-grams instead of splitting on whitespace.
The other sixty-three tokenise and filter stop words with no stemmer, because Snowball publishes no algorithm for any of them: Albanian, Amharic, Azerbaijani, Bambara, Belarusian, Bengali, Bosnian, Breton, Croatian, Dagbani, Ewe, Faroese, Fijian, Ga, Galician, Georgian, Guarani, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Icelandic, Igbo, Kannada, Kazakh, Kinyarwanda, Kirundi, Kurmanji, Kyrgyz, Latin, Latvian, Lingala, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Oromo, Punjabi, Samoan, Scottish Gaelic, Shona, Sinhala, Slovak, Sorani, Tagalog, Tatar, Telugu, Tibetan, Tigrinya, Tongan, Twi (Akan), Urdu, Vietnamese, Welsh, Wolof, Xhosa, Yoruba, and Zulu. Eleven of those sixty-three carry a normaliser that folds the spellings their orthography leaves optional.
Every module names the source of its stop word list at the top of its file, and a list with no published source records what it was curated from. Load any of the 107 the way you loaded French and Swahili above: import the module from its own entry point, then pass it to registerLanguage before you create an index that names it.
Override the stop words for one index
A language module sets the defaults for its name, but you can override them on a single index without touching the module. The stopWords field takes the default set and returns a new Set, and it goes in the index config alongside language. Add a Swahili term that carries no signal in this corpus to the articles-sw index you created above.
import { french } from '@delali/narsil/languages/french'
import { swahili } from '@delali/narsil/languages/swahili'
import { createNarsil, registerLanguage } from '@delali/narsil'
const narsil = await createNarsil()
registerLanguage(french)
registerLanguage(swahili)
await narsil.createIndex('articles', {
schema: { title: 'string', body: 'string' },
language: 'french',
})
await narsil.insert('articles', {
id: 'recherche-vectorielle',
title: 'La recherche vectorielle expliquée',
body: 'Les vecteurs denses capturent le sens au-delà des mots exacts.',
})
await narsil.insert('articles', {
id: 'partitions',
title: 'Partitionner un index',
body: 'Le partitionnement répartit un index sur plusieurs processus.',
})
await narsil.createIndex('articles-sw', {
schema: { title: 'string', body: 'string' },
language: 'swahili',
stopWords: (defaults) => new Set([...defaults, 'faharasa']),
})
await narsil.insert('articles-sw', {
id: 'utafutaji-wa-vekta',
title: 'Utafutaji wa vekta umefafanuliwa',
body: 'Vekta hunasa maana zaidi ya maneno kamili.',
})The override applies to this index alone, so other Swahili indexes keep the module defaults. The index config also takes a tokenizer, an object with a tokenize(text) method that returns { token, position } entries, for a field whose splitting rules the language module does not provide. A custom tokeniser replaces the whole analysis pipeline for that index: the engine indexes the tokens it returns exactly as they come back, with no stemming, no diacritic folding, and no stop word removal. Give it every rule the field needs, because nothing else runs afterwards.
Name a tokeniser or a stop word set
The overrides above are code, a function in one case and an object in the other, and code cannot leave the script you write it in. A checkpoint stores data, so it cannot hold a tokeniser. A worker thread receives its index config as a copy, so it cannot receive one either. A name solves both cases, because a name is data: register the implementation once, and every index config refers to it by that name.
registerTokenizer(name, tokenizer) and registerStopWords(name, override) fill the two registries. This is a separate file, catalogue.ts, that registers one of each and creates an index for each. The products index keeps a product code whole because its tokeniser splits on whitespace alone, and the listings index drops the word refurbished from its stop word set.
import { createNarsil, registerStopWords, registerTokenizer } from '@delali/narsil'
registerTokenizer('product-codes', {
tokenize: (text) =>
text
.toLowerCase()
.split(/\s+/)
.map((token, position) => ({ token, position })),
})
registerStopWords('catalogue-noise', (defaults) => new Set([...defaults, 'refurbished']))
const narsil = await createNarsil()
await narsil.createIndex('products', {
schema: { title: 'string' },
tokenizer: 'product-codes',
})
await narsil.createIndex('listings', {
schema: { title: 'string' },
stopWords: 'catalogue-noise',
})
await narsil.insert('products', { id: 'kb-042', title: 'Split Keyboard SKU-4821' })
await narsil.insert('listings', { id: 'kb-042', title: 'Refurbished Split Keyboard' })
const wholeCode = await narsil.query('products', { term: 'sku-4821' })
const codeFragment = await narsil.query('products', { term: '4821' })
const droppedWord = await narsil.query('listings', { term: 'refurbished' })
console.log(
JSON.stringify(
{ wholeCode: wholeCode.count, codeFragment: codeFragment.count, droppedWord: droppedWord.count },
null,
2,
),
)Naming a tokeniser or stop word set you have not registered fails with CONFIG_INVALID, and the error's details list the names that are registered. hasTokenizer(name) and hasStopWords(name) report whether a name is taken, and getTokenizer(name) and getStopWords(name) return the implementation behind it.
Durability and workers accept the named forms alone. A durable engine writes the name into the index metadata and rebinds the implementation from the registry during recovery, so register your names before you call createNarsil. An index config that hands a durable engine a tokeniser instance or a stop word function fails with CONFIG_INVALID at createIndex.
A worker thread resolves names from its own registry, which workers.bootstrapModule fills at startup, so an index carrying an inline tokeniser or stop word function keeps answering on the main thread while the rest promote. Persistence and durability covers recovery, and partitions and workers covers the bootstrap module.
Bring your own language
Call registerLanguage(module) to add your own language. A module provides six fields: a name, a revision string that identifies its analysis, a stemmer function or null where no algorithm exists, a stopWords set, an optional normalizer that folds a token's spelling before the stemmer runs, which helps a writing system where one word has several accepted spellings, and an optional tokenizer that adjusts the splitting defaults. name, revision, stemmer, and stopWords are required, so a module that omits revision fails to compile against the LanguageModule type. Any bundled module serves as a reference implementation.
The shortest module to write extends a bundled one. This is a separate file, legal.ts, that keeps English analysis, drops three words a contract corpus repeats in every document, and names its tokeniser config through the exported TokenizerConfig type. Once registered, the name goes in the language option exactly as a bundled module's would.
import type { TokenizerConfig } from '@delali/narsil'
import { english } from '@delali/narsil/languages/english'
import { createNarsil, registerLanguage } from '@delali/narsil'
const contractTokenizer: TokenizerConfig = { minTokenLength: 2, stripPossessive: true }
registerLanguage({
...english,
name: 'english-legal',
revision: '1',
stopWords: new Set([...english.stopWords, 'hereinafter', 'whereto', 'whereas']),
tokenizer: contractTokenizer,
})
const narsil = await createNarsil()
await narsil.createIndex('contracts', {
schema: { title: 'string', body: 'string' },
language: 'english-legal',
})
await narsil.insert('contracts', {
id: 'supply-agreement',
title: 'Supply Agreement',
body: 'Whereas the supplier delivers the goods hereinafter described to the buyer.',
})
const droppedTerm = await narsil.query('contracts', { term: 'hereinafter' })
const keptTerm = await narsil.query('contracts', { term: 'supplier' })
console.log(JSON.stringify({ droppedTerm: droppedTerm.count, keptTerm: keptTerm.count }, null, 2))Give a new language the revision '1', and change that string whenever you change the stemmer, the normaliser, the stop words, or the tokeniser config. The next section covers what the engine does with that change.
Keep an index current when a language changes
A change to a language module changes how it analyses text, so the terms an index already holds stop matching the terms a query produces. The revision string is what the engine compares. Every index records the analysis revision it was built with, and on recovery the engine checks that stored value against the one the language module now carries. A difference marks the index stale.
Upgrading @delali/narsil is the usual way an index reaches that state, because a release that corrects a stemmer or a stop word list bumps that module's revision.
A stale index keeps answering. Every query, preflight, and suggest result for it carries analysisStale: true, and listIndexes() reports the same flag, which marks results built from terms the current analysis no longer produces.
This is a separate file, revisions.ts, that makes the whole cycle visible in one run. It registers a language of its own at revision '1', indexes a Swahili document, and shuts the engine down. It then reopens the same directory with the revision changed to '2', which is what a package upgrade does to a bundled module. rebuild: 'manual' holds the automatic rebuild back so that the stale state stays observable.
import { swahili } from '@delali/narsil/languages/swahili'
import { createNarsil, registerLanguage } from '@delali/narsil'
registerLanguage({ ...swahili, name: 'swahili-news', revision: '1' })
const first = await createNarsil({ durability: { directory: './narsil-revisions' } })
await first.createIndex('habari', {
schema: { title: 'string', body: 'string' },
language: 'swahili-news',
})
await first.insert('habari', {
id: 'utafutaji-wa-vekta',
title: 'Utafutaji wa vekta umefafanuliwa',
body: 'Vekta hunasa maana zaidi ya maneno kamili.',
})
await first.checkpoint('habari')
await first.shutdown()
registerLanguage({ ...swahili, name: 'swahili-news', revision: '2' })
const second = await createNarsil({
durability: { directory: './narsil-revisions' },
analysis: { rebuild: 'manual' },
})
const stale = await second.query('habari', { term: 'vekta' })
console.log(JSON.stringify({ indexes: second.listIndexes(), stale: stale.analysisStale }, null, 2))
await second.shutdown()Rebuild the terms
By default the engine repairs a stale index on its own. It rebuilds the terms in the background from the documents the index already stores, one partition at a time, and it rebuilds one index at a time. A rebuild re-analyses the text of every document in the index, so budget for it on a large one. It leaves vectors and embeddings untouched, because the revision covers text analysis alone. When it finishes, a durable engine writes the new revision into the index metadata and takes a checkpoint, so the index is already current after a later restart.
Control the rebuild yourself when the timing matters. Under rebuild: 'manual' every stale index stays as it is, listIndexes() reports which ones they are, and rebuildAnalysis(indexName) resolves once every partition of one index carries current terms. The analysisRebuild event reports progress with a status of started, completed, or failed. Add the listener and the rebuild loop to revisions.ts, and query the same term on both sides of the rebuild.
import { swahili } from '@delali/narsil/languages/swahili'
import { createNarsil, registerLanguage } from '@delali/narsil'
registerLanguage({ ...swahili, name: 'swahili-news', revision: '1' })
const first = await createNarsil({ durability: { directory: './narsil-revisions' } })
await first.createIndex('habari', {
schema: { title: 'string', body: 'string' },
language: 'swahili-news',
})
await first.insert('habari', {
id: 'utafutaji-wa-vekta',
title: 'Utafutaji wa vekta umefafanuliwa',
body: 'Vekta hunasa maana zaidi ya maneno kamili.',
})
await first.checkpoint('habari')
await first.shutdown()
registerLanguage({ ...swahili, name: 'swahili-news', revision: '2' })
const second = await createNarsil({
durability: { directory: './narsil-revisions' },
analysis: { rebuild: 'manual' },
})
const progress: string[] = []
second.on('analysisRebuild', (event) => {
progress.push(`${event.status} ${event.partitionsRebuilt}/${event.partitionCount}`)
})
const stale = await second.query('habari', { term: 'vekta' })
for (const index of second.listIndexes()) {
if (index.analysisStale) await second.rebuildAnalysis(index.name)
}
const current = await second.query('habari', { term: 'vekta' })
console.log(JSON.stringify({ indexes: second.listIndexes(), stale: stale.analysisStale }, null, 2))
console.log(
JSON.stringify(
{
beforeRebuild: { count: stale.count, analysisStale: stale.analysisStale },
progress,
afterRebuild: { count: current.count, analysisStale: current.analysisStale },
},
null,
2,
),
)
await second.shutdown()The flag is absent from afterRebuild rather than false, because the engine sets analysisStale only while an index holds terms an earlier analysis wrote. Calling rebuildAnalysis on an index that is already current does nothing. A failed rebuild emits analysisRebuild with status: 'failed' and the error, and that index stays stale and answers from its old terms until you try again.
onStaleAnalysis covers the case the loop above cannot: it runs once for each stale index at start-up, before an automatic rebuild begins, and it is the only place the stored and the current revision appear together. A third revision of the same language shows it. stale-watch.ts reads the directory revisions.ts wrote, so run revisions.ts first.
import { createNarsil, registerLanguage } from '@delali/narsil'
import { swahili } from '@delali/narsil/languages/swahili'
registerLanguage({ ...swahili, name: 'swahili-news', revision: '3' })
const narsil = await createNarsil({
durability: { directory: './narsil-revisions' },
analysis: {
onStaleAnalysis: (index) => {
console.warn(`${index.indexName} was built with ${index.storedRevision}, ${index.language} now reports ${index.currentRevision}`)
},
},
})
await narsil.shutdown()habari was built with 2, swahili-news now reports 3Its second argument starts the rebuild for that index. Awaiting that argument inside the callback holds createNarsil open until the index finishes, which suits a deployment that must never answer from stale terms. An index built by an engine older than the revision field records nothing about its analysis, so the engine treats it as stale and rebuilds it once.
A server deployment sets analysis on the engine it hands to createServer. The REST API reports analysisStale on search responses and on GET /indexes, and it exposes no endpoint that starts a rebuild, so call rebuildAnalysis from your launcher. Persistence and durability covers the recovery pass that runs this comparison.