Imagine a grower who asks a seed supplier's AI agent what to use against downy mildew on courgette leaves, and who gets advice about watering at the root and thinning the crowded leaves. That advice is for powdery mildew, which is a different disease with a different remedy. The growing notes contain a note on powdery mildew on courgettes and none on downy mildew, while the question has three words in common with the powdery mildew note.
The authors of a 2026 benchmark measure what a model does when a tool returns a value that the model cannot use. Under a deployment-style system prompt, 14.10% of the responses in their benchmark are dishonest, which the authors define as asserting a value that has no support in the payload, or declining while citing a fabricated limit. The figure rises to 45.3% when the tool reports success and returns an empty, stale, or truncated value, and it falls to 0.87% when the authors add a single instruction to the prompt, asking the model to state a retrieval status before it answers (arXiv 2609.14758).
Because the model here is a scripted mock, you get the same answers below every time you execute the files. Swap in a model string when you want a real model to write the wording. Each code block below holds one whole file.
A note with some of the same words
termMatch defaults to any, under which a document matches when it contains any one of the query terms. The analyser drops the stop words from the grower's question, which leaves four terms, and the courgette note contains three of them.
import { createNarsil } from '@delali/narsil'
import { ToolLoopAgent, tool } from 'ai'
import { z } from 'zod'
import { type GrowingNote, growingModel, QUESTION } from './mock-model.ts'
const narsil = await createNarsil()
await narsil.createIndex('growing_notes', {
schema: { crop: 'enum', section: 'string', advice: 'string' },
language: 'english',
})
await narsil.insertBatch('growing_notes', [
{
id: 'courgette-powdery-mildew',
crop: 'courgette',
section: 'Powdery mildew on courgettes',
advice:
'Powdery mildew shows as a white dust on courgette leaves from midsummer. Water at the root in the morning, thin the crowded leaves for airflow, and pick the fruit young so the plant keeps growing.',
},
{
id: 'tomato-blight',
crop: 'tomato',
section: 'Late blight on tomatoes',
advice:
'Late blight browns the tomato leaf from the tip and blackens the stem within days. Lift and burn the affected plants, keep the remaining foliage dry, and sow a blight-resistant variety next season.',
},
{
id: 'broad-bean-rust',
crop: 'broad bean',
section: 'Chocolate spot on broad beans',
advice:
'Chocolate spot marks broad bean leaves with brown flecks in a wet spring. Space the rows wider, stake the plants upright, and harvest the lower pods first.',
},
{
id: 'carrot-fly',
crop: 'carrot',
section: 'Carrot fly',
advice:
'Carrot fly larvae tunnel into the root and leave rusty scars. Sow after the first flight in June, cover the bed with fine mesh, and sow thinly so that no thinning releases the scent.',
},
])
const growingTools = {
searchNotes: tool({
description: 'Search the growing notes by what their text says.',
inputSchema: z.object({ text: z.string() }),
execute: async ({ text }: { text: string }): Promise<GrowingNote[]> => {
const found = await narsil.query<GrowingNote>('growing_notes', {
term: text,
document: { include: ['section', 'advice'] },
limit: 1,
})
return found.hits.map((hit) => hit.document)
},
}),
}
const askGrowingAgent = async (): Promise<string> => {
const agent = new ToolLoopAgent({
model: growingModel(),
instructions: 'Answer the grower from the growing notes.',
tools: growingTools,
})
const reply = await agent.generate({ prompt: QUESTION })
return reply.text
}
console.log(JSON.stringify({ question: QUESTION, growerHeard: await askGrowingAgent() }, null, 2))
await narsil.shutdown(){
"question": "What do I use for downy mildew on courgette leaves?",
"growerHeard": "Powdery mildew shows as a white dust on courgette leaves from midsummer. Water at the root in the morning, thin the crowded leaves for airflow, and pick the fruit young so the plant keeps growing. That comes from the 'Powdery mildew on courgettes' note."
}The grower receives advice from the supplier's own notes, in the supplier's own wording, for the wrong disease. The answer looks like a full match, because the tool drops the score before it returns the note.
WhatdoIusefordownymildewoncourgetteleaves?
Require every term, and watch what the model does with an empty result
Under termMatch: 'all', a document matches only when it contains every query term. The word downy is absent from every note, so the strict query matches nothing. suggest with the prefix 'down' also returns an empty list, because no term in the dictionary starts with those letters.
The tool now returns an empty array, and the model answers from its own training about mildew.
import { createNarsil } from '@delali/narsil'
import { ToolLoopAgent, tool } from 'ai'
import { z } from 'zod'
import { type GrowingNote, growingModel, QUESTION } from './mock-model.ts'
const narsil = await createNarsil()
await narsil.createIndex('growing_notes', {
schema: { crop: 'enum', section: 'string', advice: 'string' },
language: 'english',
})
await narsil.insertBatch('growing_notes', [
{
id: 'courgette-powdery-mildew',
crop: 'courgette',
section: 'Powdery mildew on courgettes',
advice:
'Powdery mildew shows as a white dust on courgette leaves from midsummer. Water at the root in the morning, thin the crowded leaves for airflow, and pick the fruit young so the plant keeps growing.',
},
{
id: 'tomato-blight',
crop: 'tomato',
section: 'Late blight on tomatoes',
advice:
'Late blight browns the tomato leaf from the tip and blackens the stem within days. Lift and burn the affected plants, keep the remaining foliage dry, and sow a blight-resistant variety next season.',
},
{
id: 'broad-bean-rust',
crop: 'broad bean',
section: 'Chocolate spot on broad beans',
advice:
'Chocolate spot marks broad bean leaves with brown flecks in a wet spring. Space the rows wider, stake the plants upright, and harvest the lower pods first.',
},
{
id: 'carrot-fly',
crop: 'carrot',
section: 'Carrot fly',
advice:
'Carrot fly larvae tunnel into the root and leave rusty scars. Sow after the first flight in June, cover the bed with fine mesh, and sow thinly so that no thinning releases the scent.',
},
])
const growingTools = {
searchNotes: tool({
description: 'Search the growing notes by what their text says.',
inputSchema: z.object({ text: z.string() }),
execute: async ({ text }: { text: string }): Promise<GrowingNote[]> => {
const found = await narsil.query<GrowingNote>('growing_notes', {
term: text,
termMatch: 'all',
document: { include: ['section', 'advice'] },
limit: 1,
})
return found.hits.map((hit) => hit.document)
},
}),
}
const askGrowingAgent = async (): Promise<string> => {
const agent = new ToolLoopAgent({
model: growingModel(),
instructions: 'Answer the grower from the growing notes.',
tools: growingTools,
})
const reply = await agent.generate({ prompt: QUESTION })
return reply.text
}
console.log(JSON.stringify({ question: QUESTION, growerHeard: await askGrowingAgent() }, null, 2))
await narsil.shutdown(){
"question": "What do I use for downy mildew on courgette leaves?",
"growerHeard": "Spray a copper-based fungicide every ten days and remove the worst leaves as you go."
}The answer no longer contains the wrong note, and it is worse. A copper fungicide is plausible advice for a mildew; however, the supplier publishes no such advice. If the grower follows it, they are following a recommendation that appears in none of the supplier's notes. The tool reports success and returns nothing usable, which is the case where the authors above measure 45.3%.
Return a status, and the nearest note with it
The model's new instruction is to open its answer with retrievalStatus, the same one-sentence change with which the authors lower dishonesty from 14.10% to 0.87%. The tool sets that field from the result of the strict query, which leaves the model no say in its value.
preflight counts the documents that a loose query would match without building any hits, and the tool then fetches the nearest note only when that count is above zero. When the count is zero, the tool returns the status alone, and the model tells the grower that the notes contain nothing on that disease.
import { createNarsil } from '@delali/narsil'
import { ToolLoopAgent, tool } from 'ai'
import { z } from 'zod'
import { type GrowingNote, growingModel, QUESTION } from './mock-model.ts'
import { growingModel, type NoteSearch, QUESTION } from './mock-model.ts'
const narsil = await createNarsil()
await narsil.createIndex('growing_notes', {
schema: { crop: 'enum', section: 'string', advice: 'string' },
language: 'english',
})
await narsil.insertBatch('growing_notes', [
{
id: 'courgette-powdery-mildew',
crop: 'courgette',
section: 'Powdery mildew on courgettes',
advice:
'Powdery mildew shows as a white dust on courgette leaves from midsummer. Water at the root in the morning, thin the crowded leaves for airflow, and pick the fruit young so the plant keeps growing.',
},
{
id: 'tomato-blight',
crop: 'tomato',
section: 'Late blight on tomatoes',
advice:
'Late blight browns the tomato leaf from the tip and blackens the stem within days. Lift and burn the affected plants, keep the remaining foliage dry, and sow a blight-resistant variety next season.',
},
{
id: 'broad-bean-rust',
crop: 'broad bean',
section: 'Chocolate spot on broad beans',
advice:
'Chocolate spot marks broad bean leaves with brown flecks in a wet spring. Space the rows wider, stake the plants upright, and harvest the lower pods first.',
},
{
id: 'carrot-fly',
crop: 'carrot',
section: 'Carrot fly',
advice:
'Carrot fly larvae tunnel into the root and leave rusty scars. Sow after the first flight in June, cover the bed with fine mesh, and sow thinly so that no thinning releases the scent.',
},
])
const growingTools = {
searchNotes: tool({
description: 'Search the growing notes by what their text says.',
inputSchema: z.object({ text: z.string() }),
execute: async ({ text }: { text: string }): Promise<GrowingNote[]> => {
const found = await narsil.query<GrowingNote>('growing_notes', {
execute: async ({ text }: { text: string }): Promise<NoteSearch> => {
const matched = await narsil.query<{ section: string; advice: string }>('growing_notes', {
term: text,
termMatch: 'all',
document: { include: ['section', 'advice'] },
limit: 1,
})
return found.hits.map((hit) => hit.document)
if (matched.hits.length > 0) {
return { retrievalStatus: 'matched', notes: matched.hits.map((hit) => hit.document) }
}
const loose = await narsil.preflight('growing_notes', { term: text })
if (loose.count === 0) return { retrievalStatus: 'nothing matched', notes: [] }
const nearest = await narsil.query<{ section: string }>('growing_notes', {
term: text,
document: { include: ['section'] },
limit: 1,
})
return {
retrievalStatus: 'nothing matched',
notes: [],
nearest: { section: nearest.hits[0].document.section, looseMatches: loose.count },
}
},
}),
}
const askGrowingAgent = async (): Promise<string> => {
const agent = new ToolLoopAgent({
model: growingModel(),
instructions: 'Answer the grower from the growing notes.',
instructions:
'Answer the grower from the growing notes, and open the answer with the retrievalStatus that the tool returned.',
tools: growingTools,
})
const reply = await agent.generate({ prompt: QUESTION })
return reply.text
}
console.log(JSON.stringify({ question: QUESTION, growerHeard: await askGrowingAgent() }, null, 2))
await narsil.shutdown(){
"question": "What do I use for downy mildew on courgette leaves?",
"growerHeard": "retrievalStatus: nothing matched. No note covers downy mildew on courgettes, and the nearest of 3 partial matches is 'Powdery mildew on courgettes', which is a different disease."
}The grower now learns three facts, which are that the notes contain nothing on their disease, that one note has some of the same words, and that this note is about a different disease. REMEMBERED_TREATMENT is still in the mock; however, the answer never contains it, because after 'nothing matched' the model has no way to present that treatment as the supplier's advice.
You can also set minScore, although a floor on the score is harder to choose. The engine ranks with BM25, whose scores have no fixed upper bound, so a floor of 2.0 chosen for this index and this query could reject every hit on a longer corpus or a shorter question. Measure the scores on your own index before you set a floor, and use termMatch when the words of the question are enough to decide the match.
Where to go next
- Read Full-text search for
termMatch,minScore, the typo tolerance, andsuggest. - For a question whose words appear in no note at all, where keyword search returns nothing under any term policy, see Search by wording and by meaning.
- To attach a reference to each hit that the tool does return, follow Cite the passage behind an answer.