> Complete content index: https://narsil.sondelali.com/llms.txt # Language support > Tokenise, stem, and filter stop words in 107 languages through per-language entry points, and rebuild an index when a language module changes. URL: https://narsil.sondelali.com/docs/language-support Section: Core concepts Version: 0.2 (@delali/narsil@0.2.2, latest) Narsil includes language modules for tokenisation, Snowball stemming, and stop word removal. Each module is a separate entry point, so your bundle includes only the languages you use. This page builds one file, `articles.ts`, step by step. You load a language, create an index that names it, add a second index in another language, and override the stop words for one index. A second file then registers a tokeniser and a stop word set under names that indexes reference. ## Load a language A language module exports a language object you pass to `registerLanguage`. Register it once and its name becomes valid for any index. English is always available, so an English index requires no import or registration. Start `articles.ts` with a French module and an index that names French. ```ts title="articles.ts" const narsil = await createNarsil() registerLanguage(french) await narsil.createIndex('articles', { schema: { title: 'string', body: 'string' }, language: 'french', }) ``` Naming a language that was never registered fails with `LANGUAGE_NOT_SUPPORTED`. The `language` option selects the analyser Narsil uses to tokenise and stem every text field on the index, so a French index stems `chercher`, `cherche`, and `cherchons` to the same root and drops French stop words such as `le` and `des`. ## Index documents in the chosen language With the module loaded and the index created, add two French articles. Narsil tokenises and stems each text field on the way in, using the French analyser the index selected. ```ts title="articles.ts" const narsil = await createNarsil() registerLanguage(french) await narsil.createIndex('articles', { schema: { title: 'string', body: 'string' }, language: 'french', }) await narsil.insert('articles', { // [!code ++:5] id: 'recherche-vectorielle', title: 'La recherche vectorielle expliquée', body: 'Les vecteurs denses capturent le sens au-delà des mots exacts.', }) await narsil.insert('articles', { // [!code ++:5] id: 'partitions', title: 'Partitionner un index', body: 'Le partitionnement répartit un index sur plusieurs processus.', }) ``` A French query for `vecteurs` matches the singular `vecteur` and the verb form `vectorielle`, because the stemmer reduces them to the same root before ranking. ## Add an index in another language Each index names its own analyser, so a Swahili index works alongside the French one. A created index keeps the language it was made with, so to add Swahili you register the module and create a second index that names it. Register the Swahili module, create `articles-sw`, and insert a Swahili document. ```ts title="articles.ts" import { swahili } from '@delali/narsil/languages/swahili' // [!code ++] const narsil = await createNarsil() registerLanguage(french) registerLanguage(swahili) // [!code ++] await narsil.createIndex('articles', { schema: { title: 'string', body: 'string' }, language: 'french', }) await narsil.insert('articles', { id: 'recherche-vectorielle', title: 'La recherche vectorielle expliquée', body: 'Les vecteurs denses capturent le sens au-delà des mots exacts.', }) await narsil.insert('articles', { id: 'partitions', title: 'Partitionner un index', body: 'Le partitionnement répartit un index sur plusieurs processus.', }) await narsil.createIndex('articles-sw', { // [!code ++:4] schema: { title: 'string', body: 'string' }, language: 'swahili', }) await narsil.insert('articles-sw', { // [!code ++:5] id: 'utafutaji-wa-vekta', title: 'Utafutaji wa vekta umefafanuliwa', body: 'Vekta hunasa maana zaidi ya maneno kamili.', }) ``` Swahili has a full stemmer, so the Swahili index stems and filters stop words the same way the French one does. Each index keeps its own analyser, and the schema, inserts, and queries share the same shape whichever language a field uses. ## Coverage Narsil includes 107 language modules. Thirty-seven of them tokenise, stem, and filter stop words: Arabic, Armenian, Basque, Bulgarian, Catalan, Czech, Danish, Dutch, English, Esperanto, Estonian, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Lithuanian, Nepali, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Sanskrit, Serbian, Slovenian, Spanish, Swahili, Swedish, Tamil, Turkish, and Ukrainian. Thirty-two of those stemmers come from the Snowball reference sources, and each one reproduces Snowball's own output on all 11.4 million word pairs Snowball publishes. The Bulgarian, Sanskrit, Slovenian, Swahili, and Ukrainian stemmers were written for Narsil, because Snowball covers none of those five. Burmese, Chinese (Mandarin), Japanese, Khmer, Korean, Lao, and Thai write words with no space between them, so those seven modules use a character n-gram tokeniser with stop words, which cuts each run of script into overlapping two-character n-grams instead of splitting on whitespace. The other sixty-three tokenise and filter stop words with no stemmer, because Snowball publishes no algorithm for any of them: Albanian, Amharic, Azerbaijani, Bambara, Belarusian, Bengali, Bosnian, Breton, Croatian, Dagbani, Ewe, Faroese, Fijian, Ga, Galician, Georgian, Guarani, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Icelandic, Igbo, Kannada, Kazakh, Kinyarwanda, Kirundi, Kurmanji, Kyrgyz, Latin, Latvian, Lingala, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Oromo, Punjabi, Samoan, Scottish Gaelic, Shona, Sinhala, Slovak, Sorani, Tagalog, Tatar, Telugu, Tibetan, Tigrinya, Tongan, Twi (Akan), Urdu, Vietnamese, Welsh, Wolof, Xhosa, Yoruba, and Zulu. Eleven of those sixty-three carry a normaliser that folds the spellings their orthography leaves optional. Every module names the source of its stop word list at the top of its file, and a list with no published source records what it was curated from. Load any of the 107 the way you loaded French and Swahili above: import the module from its own entry point, then pass it to `registerLanguage` before you create an index that names it. ## Override the stop words for one index A language module sets the defaults for its name, but you can override them on a single index without touching the module. The `stopWords` field takes the default set and returns a new `Set`, and it goes in the index config alongside `language`. Add a Swahili term that carries no signal in this corpus to the `articles-sw` index you created above. ```ts title="articles.ts" const narsil = await createNarsil() registerLanguage(french) registerLanguage(swahili) await narsil.createIndex('articles', { schema: { title: 'string', body: 'string' }, language: 'french', }) await narsil.insert('articles', { id: 'recherche-vectorielle', title: 'La recherche vectorielle expliquée', body: 'Les vecteurs denses capturent le sens au-delà des mots exacts.', }) await narsil.insert('articles', { id: 'partitions', title: 'Partitionner un index', body: 'Le partitionnement répartit un index sur plusieurs processus.', }) await narsil.createIndex('articles-sw', { schema: { title: 'string', body: 'string' }, language: 'swahili', stopWords: (defaults) => new Set([...defaults, 'faharasa']), // [!code ++] }) await narsil.insert('articles-sw', { id: 'utafutaji-wa-vekta', title: 'Utafutaji wa vekta umefafanuliwa', body: 'Vekta hunasa maana zaidi ya maneno kamili.', }) ``` The override applies to this index alone, so other Swahili indexes keep the module defaults. The index config also takes a `tokenizer`, an object with a `tokenize(text)` method that returns `{ token, position }` entries, for a field whose splitting rules the language module does not provide. A custom tokeniser replaces the whole analysis pipeline for that index: the engine indexes the tokens it returns exactly as they come back, with no stemming, no diacritic folding, and no stop word removal. Give it every rule the field needs, because nothing else runs afterwards. ## Name a tokeniser or a stop word set The overrides above are code, a function in one case and an object in the other, and code cannot leave the script you write it in. A checkpoint stores data, so it cannot hold a tokeniser. A worker thread receives its index config as a copy, so it cannot receive one either. A name solves both cases, because a name is data: register the implementation once, and every index config refers to it by that name. `registerTokenizer(name, tokenizer)` and `registerStopWords(name, override)` fill the two registries. This is a separate file, `catalogue.ts`, that registers one of each and creates an index for each. The `products` index keeps a product code whole because its tokeniser splits on whitespace alone, and the `listings` index drops the word `refurbished` from its stop word set. ```ts title="catalogue.ts" registerTokenizer('product-codes', { tokenize: (text) => text .toLowerCase() .split(/\s+/) .map((token, position) => ({ token, position })), }) registerStopWords('catalogue-noise', (defaults) => new Set([...defaults, 'refurbished'])) const narsil = await createNarsil() await narsil.createIndex('products', { schema: { title: 'string' }, tokenizer: 'product-codes', }) await narsil.createIndex('listings', { schema: { title: 'string' }, stopWords: 'catalogue-noise', }) await narsil.insert('products', { id: 'kb-042', title: 'Split Keyboard SKU-4821' }) await narsil.insert('listings', { id: 'kb-042', title: 'Refurbished Split Keyboard' }) const wholeCode = await narsil.query('products', { term: 'sku-4821' }) const codeFragment = await narsil.query('products', { term: '4821' }) const droppedWord = await narsil.query('listings', { term: 'refurbished' }) console.log( JSON.stringify( { wholeCode: wholeCode.count, codeFragment: codeFragment.count, droppedWord: droppedWord.count }, null, 2, ), ) ``` ```json result { "wholeCode": 1, "codeFragment": 0, "droppedWord": 0 } ``` Naming a tokeniser or stop word set you have not registered fails with `CONFIG_INVALID`, and the error's `details` list the names that are registered. `hasTokenizer(name)` and `hasStopWords(name)` report whether a name is taken, and `getTokenizer(name)` and `getStopWords(name)` return the implementation behind it. Durability and workers accept the named forms alone. A durable engine writes the name into the index metadata and rebinds the implementation from the registry during recovery, so register your names before you call `createNarsil`. An index config that hands a durable engine a tokeniser instance or a stop word function fails with `CONFIG_INVALID` at `createIndex`. A worker thread resolves names from its own registry, which `workers.bootstrapModule` fills at startup, so an index carrying an inline tokeniser or stop word function keeps answering on the main thread while the rest promote. [Persistence and durability](/docs/persistence-and-durability) covers recovery, and [partitions and workers](/docs/partitions-and-workers) covers the bootstrap module. ## Bring your own language Call `registerLanguage(module)` to add your own language. A module provides six fields: a `name`, a `revision` string that identifies its analysis, a `stemmer` function or `null` where no algorithm exists, a `stopWords` set, an optional `normalizer` that folds a token's spelling before the stemmer runs, which helps a writing system where one word has several accepted spellings, and an optional `tokenizer` that adjusts the splitting defaults. `name`, `revision`, `stemmer`, and `stopWords` are required, so a module that omits `revision` fails to compile against the `LanguageModule` type. Any bundled module serves as a reference implementation. The shortest module to write extends a bundled one. This is a separate file, `legal.ts`, that keeps English analysis, drops three words a contract corpus repeats in every document, and names its tokeniser config through the exported `TokenizerConfig` type. Once registered, the name goes in the `language` option exactly as a bundled module's would. ```ts title="legal.ts" const contractTokenizer: TokenizerConfig = { minTokenLength: 2, stripPossessive: true } registerLanguage({ ...english, name: 'english-legal', revision: '1', stopWords: new Set([...english.stopWords, 'hereinafter', 'whereto', 'whereas']), tokenizer: contractTokenizer, }) const narsil = await createNarsil() await narsil.createIndex('contracts', { schema: { title: 'string', body: 'string' }, language: 'english-legal', }) await narsil.insert('contracts', { id: 'supply-agreement', title: 'Supply Agreement', body: 'Whereas the supplier delivers the goods hereinafter described to the buyer.', }) const droppedTerm = await narsil.query('contracts', { term: 'hereinafter' }) const keptTerm = await narsil.query('contracts', { term: 'supplier' }) console.log(JSON.stringify({ droppedTerm: droppedTerm.count, keptTerm: keptTerm.count }, null, 2)) ``` ```json result { "droppedTerm": 0, "keptTerm": 1 } ``` Give a new language the revision `'1'`, and change that string whenever you change the stemmer, the normaliser, the stop words, or the tokeniser config. The next section covers what the engine does with that change. ## Keep an index current when a language changes A change to a language module changes how it analyses text, so the terms an index already holds stop matching the terms a query produces. The `revision` string is what the engine compares. Every index records the **analysis revision** it was built with, and on recovery the engine checks that stored value against the one the language module now carries. A difference marks the index stale. Upgrading `@delali/narsil` is the usual way an index reaches that state, because a release that corrects a stemmer or a stop word list bumps that module's revision. A stale index keeps answering. Every `query`, `preflight`, and `suggest` result for it carries `analysisStale: true`, and `listIndexes()` reports the same flag, which marks results built from terms the current analysis no longer produces. This is a separate file, `revisions.ts`, that makes the whole cycle visible in one run. It registers a language of its own at revision `'1'`, indexes a Swahili document, and shuts the engine down. It then reopens the same directory with the revision changed to `'2'`, which is what a package upgrade does to a bundled module. `rebuild: 'manual'` holds the automatic rebuild back so that the stale state stays observable. ```ts title="revisions.ts" registerLanguage({ ...swahili, name: 'swahili-news', revision: '1' }) const first = await createNarsil({ durability: { directory: './narsil-revisions' } }) await first.createIndex('habari', { schema: { title: 'string', body: 'string' }, language: 'swahili-news', }) await first.insert('habari', { id: 'utafutaji-wa-vekta', title: 'Utafutaji wa vekta umefafanuliwa', body: 'Vekta hunasa maana zaidi ya maneno kamili.', }) await first.checkpoint('habari') await first.shutdown() registerLanguage({ ...swahili, name: 'swahili-news', revision: '2' }) const second = await createNarsil({ durability: { directory: './narsil-revisions' }, analysis: { rebuild: 'manual' }, }) const stale = await second.query('habari', { term: 'vekta' }) console.log(JSON.stringify({ indexes: second.listIndexes(), stale: stale.analysisStale }, null, 2)) await second.shutdown() ``` ```json result { "indexes": [ { "name": "habari", "documentCount": 1, "partitionCount": 1, "language": "swahili-news", "analysisStale": true } ], "stale": true } ``` ## Rebuild the terms By default the engine repairs a stale index on its own. It rebuilds the terms in the background from the documents the index already stores, one partition at a time, and it rebuilds one index at a time. A rebuild re-analyses the text of every document in the index, so budget for it on a large one. It leaves vectors and embeddings untouched, because the revision covers text analysis alone. When it finishes, a durable engine writes the new revision into the index metadata and takes a checkpoint, so the index is already current after a later restart. Control the rebuild yourself when the timing matters. Under `rebuild: 'manual'` every stale index stays as it is, `listIndexes()` reports which ones they are, and `rebuildAnalysis(indexName)` resolves once every partition of one index carries current terms. The `analysisRebuild` event reports progress with a `status` of `started`, `completed`, or `failed`. Add the listener and the rebuild loop to `revisions.ts`, and query the same term on both sides of the rebuild. ```ts title="revisions.ts" registerLanguage({ ...swahili, name: 'swahili-news', revision: '1' }) const first = await createNarsil({ durability: { directory: './narsil-revisions' } }) await first.createIndex('habari', { schema: { title: 'string', body: 'string' }, language: 'swahili-news', }) await first.insert('habari', { id: 'utafutaji-wa-vekta', title: 'Utafutaji wa vekta umefafanuliwa', body: 'Vekta hunasa maana zaidi ya maneno kamili.', }) await first.checkpoint('habari') await first.shutdown() registerLanguage({ ...swahili, name: 'swahili-news', revision: '2' }) const second = await createNarsil({ durability: { directory: './narsil-revisions' }, analysis: { rebuild: 'manual' }, }) const progress: string[] = [] // [!code ++:5] second.on('analysisRebuild', (event) => { progress.push(`${event.status} ${event.partitionsRebuilt}/${event.partitionCount}`) }) const stale = await second.query('habari', { term: 'vekta' }) for (const index of second.listIndexes()) { // [!code ++:5] if (index.analysisStale) await second.rebuildAnalysis(index.name) } const current = await second.query('habari', { term: 'vekta' }) console.log(JSON.stringify({ indexes: second.listIndexes(), stale: stale.analysisStale }, null, 2)) // [!code --] console.log( // [!code ++:11] JSON.stringify( { beforeRebuild: { count: stale.count, analysisStale: stale.analysisStale }, progress, afterRebuild: { count: current.count, analysisStale: current.analysisStale }, }, null, 2, ), ) await second.shutdown() ``` ```json result { "beforeRebuild": { "count": 1, "analysisStale": true }, "progress": [ "started 0/1", "completed 1/1" ], "afterRebuild": { "count": 1 } } ``` The flag is absent from `afterRebuild` rather than `false`, because the engine sets `analysisStale` only while an index holds terms an earlier analysis wrote. Calling `rebuildAnalysis` on an index that is already current does nothing. A failed rebuild emits `analysisRebuild` with `status: 'failed'` and the error, and that index stays stale and answers from its old terms until you try again. `onStaleAnalysis` covers the case the loop above cannot: it runs once for each stale index at start-up, before an automatic rebuild begins, and it is the only place the stored and the current revision appear together. A third revision of the same language shows it. `stale-watch.ts` reads the directory `revisions.ts` wrote, so run `revisions.ts` first. ```ts title="stale-watch.ts" registerLanguage({ ...swahili, name: 'swahili-news', revision: '3' }) const narsil = await createNarsil({ durability: { directory: './narsil-revisions' }, analysis: { onStaleAnalysis: (index) => { console.warn(`${index.indexName} was built with ${index.storedRevision}, ${index.language} now reports ${index.currentRevision}`) }, }, }) await narsil.shutdown() ``` ```txt result open habari was built with 2, swahili-news now reports 3 ``` Its second argument starts the rebuild for that index. Awaiting that argument inside the callback holds `createNarsil` open until the index finishes, which suits a deployment that must never answer from stale terms. An index built by an engine older than the revision field records nothing about its analysis, so the engine treats it as stale and rebuilds it once. A [server deployment](/docs/http-server) sets `analysis` on the engine it hands to `createServer`. The REST API reports `analysisStale` on search responses and on `GET /indexes`, and it exposes no endpoint that starts a rebuild, so call `rebuildAnalysis` from your launcher. [Persistence and durability](/docs/persistence-and-durability) covers the recovery pass that runs this comparison.