Repository navigation
feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools - #614
Open
Shabonasar wants to merge 9 commits into
Open
Shabonasar wants to merge 9 commits into
Shabonasar wants to merge 9 commits into
Conversation
…Position Adds conversation_instructions: context that is stable within a conversation. It stays in the system message right after the static instructions and is never moved: its own content block with a cache breakpoint for Anthropic and OpenRouter, a block followed by a cache point for Bedrock, appended text for other providers. additional_instructions keeps its per-turn behaviour. A host that passes conversation_instructions gets the compaction summary, both when created inside a run and when restored as initialSummary, as one stable user message right after the system message, with a breakpoint of its own while the request has one left. Token accounting counts it as instruction overhead. Both paths render the same carrier text, so the requests after a compaction form one continuous prefix. Hosts that do not pass the field keep the earlier placement and accounting. Adds a prompt cache harness (src/specs/prompt-cache) that runs declarative scenarios through Run and each provider class, captures the serialized request before the network, and asserts the cache invariants per provider.
Providers serialize `tools` ahead of the system prompt, so binding a tool mid-conversation resets the prompt cache from the first byte. With the new opt-in `AgentInputs.toolDispatch`, a fixed `call_tool` dispatcher is bound from the first turn and deferred registry tools are reached through it, so `tools` stays byte-identical while the host announces new tools in the history. - ToolNode canonicalizes every batch before hooks, approval, HITL, and execution: a dispatcher call becomes its target (same call id, arguments coerced and validated against the target schema), so PreToolUse, approval policies, `action_requests`, and ON_TOOL_EXECUTE see the real tool. The same `resolveDispatchedToolCall` is exported for permission layers outside ToolNode. - Results and completion events keep the model form: the ToolMessage answers the dispatcher call under its name, and the completed tool call carries the real tool in `dispatched`, so stored history names only declared tools. - A direct call to a dispatchable tool runs the same way and is rewritten into a dispatcher call in the assistant message, since the model's own message would otherwise put an undeclared tool name into the history. - Deferred tools are never bound while the dispatcher is enabled, even after discovery. tool_search results carry full names and parameter schemas and point the model to the dispatcher. - `LCTool.disabled` keeps a tool bound but refuses its calls with a tool error before any hook, without changing `tools`. - A schema the validator cannot apply, or any error while resolving a call, becomes a tool error message instead of an exception. `ToolDispatchConfig.canDispatch` limits dispatchable tools to those the host declared. - Dispatcher calls, dispatchable tools, and disabled tools are excluded from eager prestart. Tests: unit coverage in toolDispatch.test.ts; the prompt-cache harness gains a dispatched-call scenario for every provider case.
This was referenced Oct 15, 2026
With prompt caching on Bedrock, additional_instructions was appended to the system message after its last cachePoint. Bedrock reads tools, system and messages in that order, so a tail that changes between turns sits in front of the whole history and every message checkpoint misses. The tail now moves to a message right before the newest user message, as for Anthropic and OpenRouter, with one cachePoint at the end of the history in front of it and one at the end of the request. The conversation block keeps its own cachePoint only while the four-checkpoint limit leaves room for it (tools, stable block, history, request end come first). Without prompt caching the tail stays in the system message.
…Marks Two Messages The graph can place a second message checkpoint for the latest human message. Beside the system checkpoints and the tool list that overran the limit of four in two places: - the conversation block kept its own cachePoint only on the moved-tail path; the Bedrock system message now drops it whenever the checkpoints already in the messages leave no room, with or without a moved tail; - the opening message of the history in front of a moved tail kept the graph's checkpoint on top of the two the tail path places; it is stripped now.
An opt-in call_tool dispatcher is bound from the first turn; ToolNode unwraps its calls to the deferred target before hooks and execution, and tool_search points to it, so tools can be added mid-conversation without changing the bound tools.
8 tasks done
Author
|
Reduced to simplify dependencies and the review: based on
|
This was referenced Oct 27, 2026
Author
|
Branch updated with a merge commit (no file changes) so that the agents PRs of LibreChat-AI/LibreChat#17102 merge together with a plain |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Providers serialize
toolsahead of the system prompt, so binding a tool in the middle of a conversation (an MCP server switched on, a tool found withtool_search) resets the prompt cache from the first byte. This PR adds an opt-inAgentInputs.toolDispatch. It binds a fixedcall_tooldispatcher from the first turn, and deferred registry tools are reached through it, sotoolsstays byte-identical while the host announces new tools in the history. Without the option nothing changes.Related to LibreChat-AI/LibreChat#17102 (tracking; design discussion LibreChat-AI/LibreChat#17481). Base:
main, no dependency on other PRs.How it works
toolDispatch: true | { canDispatch }call_toolafter the other tools; deferred tools (defer_loading: true) are never bound, even once discovered;canDispatchnarrows the targetsargumentsparsed as a JSON object) before hooks, approval, HITL and execution; result and completion event keep the dispatcher form with the target indispatchedcall_toolcall in the assistant message, including provider-native copies (content blocks, raw calls, Responses output items)LCTool.disabledtool_searchwithtoolDispatchcall_toolTOOL_DISPATCHER_NAMEis exported.canDispatch, or has arguments that are not a JSON object is answered with a tool error and does not run.Type of change
Testing
Tested environments/configuration:
/api/agents/chat) with the LibreChat series from Tracking: stable prompt prefix cache across turns, conversations and providers LibreChat#17102 and all five SDK PRs (fix: Keep Per-Conversation Context and the Summary at a Fixed System Position #613, feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools #614, fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615, feat: Replay Signed Reasoning in Restored History #619, fix(tools): persist tool output reference labels and keep keys unique across runs #629), cache TTL 1h. G2 (web search and three MCP servers toggled mid-chat and called): 85.5% of prompt tokens read from cache, 0 misses, no API errors. Tools with thinking: 87.0%, 0 misses.Automated tests:
anthropic,google,vertexaillm.spec.ts).ToolNode.dispatchTiming.test.ts: target runs on the host and answers under the dispatcher call, non-deferred target refused, direct call stored as a dispatcher call,canDispatchrefusal, disabled tool, failed dispatched call completed under the dispatcher call, calls arriving throughSend.AgentContext.test.ts(binding, discovered deferred tool stays unbound, no dispatcher without opt-in, schema token count),ToolSearch.test.ts,aggregator.test.ts(dispatchedkept on the completed part),stream.eagerEventExecution.test.ts(no prestart).durability-checkpoint.integration.test.tsandpersistent-handoffs.test.ts, which need MongoDB and fail onmaintoo.packages/apisrc/agents6913 passed (1 failure inhandlers.shutdown.spec.ts, a shutdown timing test);apiserver/controllers/agents1371 passed.Risk / compatibility
toolDispatch, the only new behaviour isLCTool.disabled, which a host has to set itself.call_tool. fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615 aligns stored OpenAI Responses items with that stored form when both PRs are used.Checklist