Skip to content

Send a search to the search tool, never to the browser - #841

Closed
guidovizoso wants to merge 6 commits into
mainfrom
guido/search-routing
Closed

guidovizoso wants to merge 6 commits into
mainfrom
guido/search-routing

Conversation

@guidovizoso

Copy link
Copy Markdown
Collaborator

What this changes

Asked to "search for" something in a demo, a Bot drove its browser to a search engine instead of calling the search tool it held. The prompt invited it: the computer guidance said to navigate whenever asked to "look at, check or read" anything and to "never claim you cannot browse", and the one sentence saying a page is not a search read as a hedge beside them. A model given both picks the emphatic one.

Now a Bot is told one thing, chosen by what it holds:

  • With a search tool, finding information goes to it, and the computer is for what only a browser can do: signing in, clicking through a site, filling in a form, or opening a page the person asked to see.
  • Without one, the browser never goes to Google, Bing or DuckDuckGo to look something up. The Bot answers from what it knows, says it has not checked it, and calls show_app_access once with one of Parallel Search, Exa Search or Firecrawl, so the person gets a card to ask an administrator for one. The wording is "if none of your tools searches the web", not "you hold no search tool", because a search tool can arrive unrecognised (a Composio toolkit, a server added by URL) and the Bot should not be told something false about it.
  • "Never claim you cannot browse" is gone. Opening a page the person names is still something the Bot does.
  • computer_navigate's description now says it opens a particular site and is not a web search, so the tool list agrees with the guidance beside it.

How each kind of Bot gets the right wording. COMPUTER_GUIDANCE (a constant) becomes computerGuidance({ search }). Built-in Bots are worded in builtInAgentConfiguration from the tools they are built with (holdsWebSearch, from grant refs), so a narrowed run follows what it was offered. agent-bot and agent-langgraph work it out from the tool names each run offers them (researchRoleOfToolName, from mcp__server__tool), which are the only view they have of their grants. RESEARCH_APP_NAMES lists the catalogue's search apps by the title show_app_access resolves; a test fails if one stops matching the catalogue.

Where it runs

  • New state that outlives a request? None. The guidance is built per run from the tools that run is offered.
  • What happens on the second replica? The same wording: it is a pure function of the run's tools, which every process resolves from the same grants.
  • Anything serialised? No.
  • Anything fanned out to a browser? No.
  • New listener, port, or schedule? No.

Boundary and audit

  • Every acting call still goes through the gateway. Only prompt text and a tool description change; a Bot that does open a page still goes through the computer gateway as before.
  • New refusals and new failures each write a row. None are added.
  • Nothing new is trusted from the client that the server can resolve itself. Built-in Bots are worded from the server's own grant refs. agent-bot and agent-langgraph read the tool names in their run input, which the server builds from the grants; a wrong guess there changes only which sentence the Bot reads, not what it may call.

Changelog

  • "A Bot searches with its search tool, never its browser" under Unreleased.

Merge notes

This removes the COMPUTER_GUIDANCE export in favour of computerGuidance({ search }). The calm-computer-start PR adds a test in shared/bot-prompt.test.ts that references COMPUTER_GUIDANCE (and a line to the guidance's list of acting rules, which here lives in ACTING_LINES). Whichever of the two merges second has to switch that test to computerGuidance, checking both { search: true } and { search: false }.

How it was checked

  • Tests: shared/bot-prompt.test.ts (each variant's wording, the unchecked answer and the search-app offer), server/tests/copilot.test.ts (a built-in Bot is worded from the tools it is built with, and a deployment with no computer is told about neither), server/tests/web-research.test.ts (any provider's search tool counts, by ref or by the name a model sees; the offered names are the catalogue's titles), and agent-bot/tests/history.test.ts and agent-langgraph/tests/history.test.ts (the system message follows the run's tools).
  • In a running app on a combined branch of this and five sibling PRs (local stack, 2026-10-26), with no search app on the Bot: asked it to search for something. It did not open its computer, and drew a Parallel Search card. As an admin, "Turn on and give it to General Assistant" (from the connect-card-resume PR) turned the card ready, "Parallel Search is ready now. Please carry on." was sent, and the Bot searched the web and cited its sources.
  • The full server suite on the combined branch: 5,821 pass. The only failures were the assertion-shape guard, since fixed on each branch, and one Mastra test that needs a package not installed locally. The app suite: 1,675 pass.

holdsWebSearch reads the grant refs a built-in Bot is built with, and
researchRoleOfToolName reads the mcp__server__tool names a Bot running its
own loop is offered. Either counts a search tool from any research
provider, without needing its reader alongside.

RESEARCH_APP_NAMES lists the catalogue's search apps by title, the name
show_app_access resolves. There is no app called "search", so a Bot with
none has to name one, and listing every provider keeps the choice with
the administrator. A test holds the list to the catalogue's titles.
Asked to "search for" something in a demo, a Bot drove its browser to a
search engine. The computer guidance told it to navigate whenever asked
to look at, check or read anything, and never to claim it cannot browse;
the one line saying a page is not a search read as a hedge beside them.

computerGuidance now takes what the run holds and says one thing:

- With a search tool, finding information goes to it, and the computer
  is for signing in, clicking through, filling in forms, or opening a
  page the person asked to see.
- Without one, the browser never goes to Google, Bing or DuckDuckGo to
  look something up. The Bot answers from what it knows, says it has not
  checked it, and calls show_app_access once with one of Parallel
  Search, Exa Search or Firecrawl so the person can ask for it.

"Never claim you cannot browse" is gone; opening a page the person names
is still something the Bot does.

Every path gets the right wording. Built-in Bots are worded in
builtInAgentConfiguration from the tools they are built with, so a
narrowed run follows what it was offered. agent-bot and agent-langgraph
work it out from the tool names each run offers them, which are the
only view they have of their grants.
The tool description repeated the old prompt's "look at, visit, open or
check a website", which reads "check the latest on X" as a page to open.
It now says to open a particular site and never a search engine, so the
tool list agrees with the guidance beside it.
…holds no search tool

Search tools the catalogue roles do not recognise (a Composio toolkit, a server added by URL, Exa's advanced search granted alone) would otherwise be contradicted by the prompt.
@guidovizoso

Copy link
Copy Markdown
Collaborator Author

Landed in #847.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant