Repository navigation
fix: Restore Tool Rounds Exactly as the Live Request Sent Them - #615
Open
Shabonasar wants to merge 9 commits into
Open
Shabonasar wants to merge 9 commits into
Shabonasar wants to merge 9 commits into
Conversation
…Position Adds conversation_instructions: context that is stable within a conversation. It stays in the system message right after the static instructions and is never moved: its own content block with a cache breakpoint for Anthropic and OpenRouter, a block followed by a cache point for Bedrock, appended text for other providers. additional_instructions keeps its per-turn behaviour. A host that passes conversation_instructions gets the compaction summary, both when created inside a run and when restored as initialSummary, as one stable user message right after the system message, with a breakpoint of its own while the request has one left. Token accounting counts it as instruction overhead. Both paths render the same carrier text, so the requests after a compaction form one continuous prefix. Hosts that do not pass the field keep the earlier placement and accounting. Adds a prompt cache harness (src/specs/prompt-cache) that runs declarative scenarios through Run and each provider class, captures the serialized request before the network, and asserts the cache invariants per provider.
A restored turn has to serialize like the live requests of that turn, or the next turn stops being a prefix continuation at the first difference. Four things drifted: - Sequential tool rounds of one turn were merged into one parallel batch when restored. Rounds now carry a round id and restore as separate assistant messages; text emitted before a call stays with that call. - A Responses tool round restored from stored parts lost the provider items the live request carried (function_call ids and status, reasoning items, message ids) and gained an empty assistant message. After a response with tool calls the graph slices the output per call and hands the slices to the new ON_PROVIDER_OUTPUT_ITEMS event; the content aggregator stores them on the tool_call parts and formatAgentMessages puts them back. They are replayed only to the model and endpoint that produced them, never under zdrEnabled, only when every item is a reasoning, function_call, or message item under a size cap, and reasoning references older than the retention of stored responses are left out. The replay gate takes the current time as an argument. Hosts must route the event to aggregateContent and persist the parts unchanged. - An in-process tool result made of content blocks was stored as JSON and restored as one string. The completion now marks it with outputFormat 'blocks' and restore parses it back into the same blocks. - Anthropic sent a tool_result as a text block only while the tail cache marker sat on it, and OpenRouter with prompt caching did the same for tool messages. A single plain text tool_result is now always sent as its string (Anthropic), and every non-empty tool message as one text block (OpenRouter). Tests: a generic live-versus-restored suite runs real turns against scripted streams in each wire protocol and requires the first request of turn N+1 to start byte for byte with the last request of turn N; Responses specs cover the reasoning variants and the gate branches.
This was referenced Oct 15, 2026
With prompt caching on Bedrock, additional_instructions was appended to the system message after its last cachePoint. Bedrock reads tools, system and messages in that order, so a tail that changes between turns sits in front of the whole history and every message checkpoint misses. The tail now moves to a message right before the newest user message, as for Anthropic and OpenRouter, with one cachePoint at the end of the history in front of it and one at the end of the request. The conversation block keeps its own cachePoint only while the four-checkpoint limit leaves room for it (tools, stable block, history, request end come first). Without prompt caching the tail stays in the system message.
…ponses-tool-round-restore
…Marks Two Messages The graph can place a second message checkpoint for the latest human message. Beside the system checkpoints and the tool list that overran the limit of four in two places: - the conversation block kept its own cachePoint only on the moved-tail path; the Bedrock system message now drops it whenever the checkpoints already in the messages leave no room, with or without a moved tail; - the opening message of the history in front of a moved tail kept the graph's checkpoint on top of the two the tail path places; it is stripped now.
…ponses-tool-round-restore
Sequential tool rounds, stored Responses output items and block tool results are restored as the live request carried them, so the next turn keeps the prefix.
This was referenced Oct 27, 2026
Author
|
Reduced to simplify dependencies and the review: based on
|
Shabonasar
added a commit
to Shabonasar/LibreChat
that referenced
this pull request
Oct 28, 2026
… tool rounds OpenAI Responses tool rounds carry ids, status and encrypted reasoning that the next turn has to send back unchanged. They are stored beside the message (server-side only, capped per message) and put back on the restored tool calls, so a restored round equals the request sent live. Active when the installed agents SDK reports them (LibreChat-AI/agents#615); otherwise requests are built as before.
Author
|
Branch updated with a merge commit (no file changes) so that the agents PRs of LibreChat-AI/LibreChat#17102 merge together with a plain |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A turn restored from stored content parts has to serialize like the live requests of that turn; otherwise the next turn stops being a prefix continuation at the first difference, and the provider cache writes everything after it again. Three things drift today. Sequential tool rounds of one turn are merged into one parallel batch. An OpenAI Responses tool round loses the provider items the live request carried (ids, status, reasoning items) and gains an empty assistant message. A tool result made of content blocks is restored as one string. This PR stores what is needed on the tool call part and restores it, so the restored request equals the live one in these cases.
Related to LibreChat-AI/LibreChat#17102 (tracking; design discussion LibreChat-AI/LibreChat#17481). Base:
main, no dependency on other PRs. Hosts route the newON_PROVIDER_OUTPUT_ITEMSevent toaggregateContentand persist the parts unchanged.How it works
tool_callroundIdproviderItems{ origin, created_at, items }outputFormat: 'blocks'outputholds the JSON of the content blocks and is parsed back into the same blocksproviderItems: only to the model and endpoint that produced them (origindigest), never underzdrEnabled, onlyreasoning,function_callandmessageitems, under a size cap per call. A reasoning item withoutencrypted_contentis left out once it is older than the retention of stored responses. A round restored without items falls back to the message fields, without an empty assistant message.alignProviderItemsWithCall: when the stored call differs from the item the model produced, the item'sfunction_callis rewritten to match the stored call, which is what the live request sent. This covers the combination with feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools #614, where a direct call to a deferred tool is stored ascall_tool.Type of change
Testing
Tested environments/configuration:
/api/agents/chat) with the LibreChat series from Tracking: stable prompt prefix cache across turns, conversations and providers LibreChat#17102 and all five SDK PRs (fix: Keep Per-Conversation Context and the Summary at a Fixed System Position #613, feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools #614, fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615, feat: Replay Signed Reasoning in Restored History #619, fix(tools): persist tool output reference labels and keep keys unique across runs #629), cache TTL 1h: tools with thinking 87.0% of prompt tokens read from cache, 0 misses; G2 (tools toggled) 85.5%, 0 misses; G3 (compaction) 44.7%, with every miss explained by the scenario; no API errors.Automated tests:
anthropic,google,vertexaillm.spec.ts).formatAgentMessages.tools.test.ts: sequential rounds kept apart, stored Responses items put back, the stored call form replayed, a stored call kept when its args did not survive storage, nothing restored above the size cap, content blocks parsed back; replay gate for origin, ZDR and reasoning older than 29 days.Graph.reasoning.test.ts: each call gets its slice of the stamped output and one round id per model response; the output is stamped with the model and endpoint.custom-chat-models.smoke.test.ts: origin stamp on the Responses client.durability-checkpoint.integration.test.tsandpersistent-handoffs.test.ts, which need MongoDB and fail onmaintoo.packages/apisrc/agents6913 passed (1 failure inhandlers.shutdown.spec.ts, a shutdown timing test);apiserver/controllers/agents1371 passed.Risk / compatibility
ON_PROVIDER_OUTPUT_ITEMSget noproviderItemsand restore Responses rounds from the message fields as before, minus the empty assistant message.roundIdandoutputFormatride on the existing completion events.providerItemscan hold encrypted reasoning. The host stores it like the other parts, and it is replayed only to its origin, never under ZDR.Checklist