feat: Replay Signed Reasoning in Restored History by Shabonasar · Pull Request #619 · LibreChat-AI/agents

Shabonasar · 2026-10-16T17:21:42Z

Summary

Claude models with thinking keep the thinking blocks of earlier turns in the prompt. A history restored from stored content parts has no signed blocks, so the next turn differs from the live request at the first thinking block of the previous turn, and the provider cache writes everything after it again. This is what rmesser measured in LibreChat-AI/LibreChat#17102. The miss happens on every turn where the model thought before calling a tool. This PR captures the signed blocks while streaming and, when the host asks for it, puts them back into the restored history at their original position. Turn N+1 then starts with the bytes of turn N.

Builds on the work of upman in #371 (persisting provider reasoning across runs, revalidation across model boundaries); his authorship is kept as co-author. The gpt-5.6 match in shouldIncludeEncryptedReasoning (Bedrock-hosted openai.gpt-5.6-* ids) is his change, carried over from #371.

Related to LibreChat-AI/LibreChat#17102 (tracking; design discussion LibreChat-AI/LibreChat#17481). Builds on #615 only. Opt-in: without providerReasoningReplay / replayProviderReasoning nothing changes.

How it works

live turn N:  [...] assistant[thinking(sig), tool_use] tool_result assistant[thinking(sig), text]
stream     -> think part + provider_replay { anthropic | bedrock: { model, blocks } } -> host stores it
restore:      formatAgentMessages(parts, { replayProviderReasoning: { provider, model } })
           =  the same assistant messages with the same signed blocks -> prefix of turn N kept

piece	behaviour
`AgentInputs.providerReasoningReplay`	the aggregator stores signed Anthropic `thinking` / `redacted_thinking` and Bedrock `reasoningContent` on the `think` part, with the model that actually served the call (fallback included); reasoning that resumes after text keeps its own part, so blocks stay in order
`formatAgentMessages(..., { replayProviderReasoning })`	puts the blocks back before their tool call or text, only for the same provider and model; unsigned blocks are dropped; token counts of messages with replayed blocks are recomputed
replay start	a signed block is valid only if everything before it is unchanged, so replay starts after the last assistant message whose reasoning cannot be replayed (not stored, edited away, another model); a summary starts a new chain
fallback model	restored signed reasoning is sent to a fallback only when it has the primary's provider and model
Anthropic	requests with replay and enabled thinking carry `thinking.block_binding.prefix_mismatch_behavior: "drop_block"` and its beta header `thinking-binding-controls-2026-08-01`, so a block that no longer matches is dropped by the API instead of failing the call
Bedrock	when Bedrock rejects the replayed reasoning and nothing has been streamed yet, the request is retried once without signed reasoning (Converse stream, non-streaming and local-converter paths)

If compaction, truncation or masking rewrote an earlier message inside a run, the next turn can replay blocks the live request no longer carried. Anthropic drops them and Bedrock retries without them, so the cost is one cache miss, not a failed request.

Type of change

Feature
Bug fix

Testing

Tested environments/configuration:

Claude Haiku 5.5 on the Anthropic API through LibreChat (/api/agents/chat) with the LibreChat series from Tracking: stable prompt prefix cache across turns, conversations and providers LibreChat#17102 and all five SDK PRs (fix: Keep Per-Conversation Context and the Summary at a Fixed System Position #613, feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools #614, fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615, feat: Replay Signed Reasoning in Restored History #619, fix(tools): persist tool output reference labels and keep keys unique across runs #629), cache TTL 1h, MCP tool calls with thinking.

scenario	`dev` (earlier run, LibreChat-AI/LibreChat#17102)	all five PRs
tools with thinking	80.2%, 3 misses	87.0%, 0 misses
G2: tools toggled mid-chat		85.5%, 0 misses

Signed thinking was replayed with its signatures. Requests had at most 4 cache_control markers and there were no API errors.
Bedrock: covered by the tests below only, not run against a live Bedrock endpoint.
The prompt-cache harness tests from the earlier version of this PR are kept outside the PR, in the measurement kit: https://github.com/Shabonasar/LibreChat/tree/tools/prompt-cache-measurements-delete_in_2027_q3/prompt-cache-measurements

Automated tests:

npx jest src/messages src/llm src/graphs src/tools src/agents src/summarization
npx tsc --noEmit

This PR with fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615: 4819 passed. 3 suites fail because they need API keys (anthropic, google, vertexai llm.spec.ts).
formatAgentMessages.test.ts: Anthropic and Bedrock blocks back before their tool call per round, only to the provider and model that signed them, unsigned blocks dropped, nothing replayed before a turn whose reasoning cannot be replayed, a new chain after a summary.
stream.eagerEventExecution.test.ts, stream.test.ts, aggregator.test.ts: capture only with the opt-in, Anthropic signature-only and redacted parts, Bedrock reasoning, modelName alias, attribution to the fallback that served the call, consecutive blocks kept apart.
AgentContext.test.ts (client options per provider), Graph.contextOverflow.test.ts (fallback), Graph.reasoning.test.ts (reasoning after text in its own part), llm/anthropic/llm.spec.ts (drop_block and beta only with enabled thinking), llm/bedrock/prefill.test.ts (one retry, only for a reasoning rejection).
All five SDK PRs together: 6987 passed. Failing: the 3 suites above, plus durability-checkpoint.integration.test.ts and persistent-handoffs.test.ts, which need MongoDB and fail on main too.
LibreChat agent suites against all five together: packages/api src/agents 6913 passed (1 failure in handlers.shutdown.spec.ts, a shutdown timing test); api server/controllers/agents 1371 passed.

Risk / compatibility

Opt-in on both sides (capture and restore). Without it, requests and stored parts are unchanged.
Anthropic: the block_binding field is sent only together with its beta header, and only with enabled thinking. A gateway between LibreChat and Anthropic has to pass both through.
Stored provider_replay holds signed reasoning. It goes back only to the provider and model that produced it.

Checklist

I reviewed my own changes
Relevant tests have been added or updated
Existing relevant tests pass
The change does not introduce new warnings or errors
User-facing or complex behavior is documented where necessary
Required dependency changes have been merged/published: fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615

…Position Adds conversation_instructions: context that is stable within a conversation. It stays in the system message right after the static instructions and is never moved: its own content block with a cache breakpoint for Anthropic and OpenRouter, a block followed by a cache point for Bedrock, appended text for other providers. additional_instructions keeps its per-turn behaviour. A host that passes conversation_instructions gets the compaction summary, both when created inside a run and when restored as initialSummary, as one stable user message right after the system message, with a breakpoint of its own while the request has one left. Token accounting counts it as instruction overhead. Both paths render the same carrier text, so the requests after a compaction form one continuous prefix. Hosts that do not pass the field keep the earlier placement and accounting. Adds a prompt cache harness (src/specs/prompt-cache) that runs declarative scenarios through Run and each provider class, captures the serialized request before the network, and asserts the cache invariants per provider.

A restored turn has to serialize like the live requests of that turn, or the next turn stops being a prefix continuation at the first difference. Four things drifted: - Sequential tool rounds of one turn were merged into one parallel batch when restored. Rounds now carry a round id and restore as separate assistant messages; text emitted before a call stays with that call. - A Responses tool round restored from stored parts lost the provider items the live request carried (function_call ids and status, reasoning items, message ids) and gained an empty assistant message. After a response with tool calls the graph slices the output per call and hands the slices to the new ON_PROVIDER_OUTPUT_ITEMS event; the content aggregator stores them on the tool_call parts and formatAgentMessages puts them back. They are replayed only to the model and endpoint that produced them, never under zdrEnabled, only when every item is a reasoning, function_call, or message item under a size cap, and reasoning references older than the retention of stored responses are left out. The replay gate takes the current time as an argument. Hosts must route the event to aggregateContent and persist the parts unchanged. - An in-process tool result made of content blocks was stored as JSON and restored as one string. The completion now marks it with outputFormat 'blocks' and restore parses it back into the same blocks. - Anthropic sent a tool_result as a text block only while the tail cache marker sat on it, and OpenRouter with prompt caching did the same for tool messages. A single plain text tool_result is now always sent as its string (Anthropic), and every non-empty tool message as one text block (OpenRouter). Tests: a generic live-versus-restored suite runs real turns against scripted streams in each wire protocol and requires the first request of turn N+1 to start byte for byte with the last request of turn N; Responses specs cover the reasoning variants and the gate branches.

Rebased onto main and LibreChat-AI#615. The OpenAI Responses reasoning carrier (provider_replay.openai_responses) is not ported: on this base Responses reasoning is carried by providerItems (LibreChat-AI#615). The gpt-5.6 prefixed-model regex fix is kept. Co-authored-by: shars <32410570+Shabonasar@users.noreply.github.com>

Rebased onto main and LibreChat-AI#615. OpenAI Responses parts (snapshot dedup, Responses filtering) are not ported: Responses reasoning travels as providerItems on this base. Co-authored-by: shars <32410570+Shabonasar@users.noreply.github.com>

…tion Adapts the persisted provider reasoning from the previous two commits to the tool-round restore of LibreChat-AI#615. - Opt-in on both sides: AgentInputs.providerReasoningReplay turns capture on, formatAgentMessages({ replayProviderReasoning: { provider, model } }) turns restore on. Without them provider_replay is neither produced nor read. - Restore puts the signed blocks at the slot of their THINK part inside the round they belong to (before the text, after the text, between rounds) instead of at the start or end of the message; reasoning that resumes after text gets its own step so the aggregator keeps that position. - Capture records the model that served the call: the attempt stamp of the primary or fallback, then the model the graph resolved for the call (override model, modelName alias), then client options. - Recounts the stored token weight of messages that carry replay when the option is set, since the blocks are not part of it. - Anthropic requests with replay active carry the thinking binding beta and block_binding.prefix_mismatch_behavior=drop_block, and surface the top-level input_transformations of the response in response_metadata; nothing is sent otherwise. - Harness: live and restored requests keep earlier turns' thinking byte for byte (signed, redacted, empty text, interleaved, Anthropic and Bedrock), the replayed history is append-only, and the same scenarios without replay break the turn boundary. Model change and fallback scenarios are covered. - Only the contiguous tail after the last hole is replayed: a turn whose blocks are missing ends the chain, since the provider binds each block to the whole prefix before it. A summary resets the chain; the message that carries the summary keeps the blocks it produced after it. - Blocks whose prefix was rewritten inside a run (compaction, the retained tail after a summary, fading of old tool results) are dropped from the request and withdrawn from the host, so the next turn restores what was actually sent. The summarizer never receives signed reasoning. - Without the opt-in nothing changes: no thinking is injected, and the fallback and summarization filters stay off. Co-authored-by: Ravi Kumar L <ravi.lazar@clickhouse.com>

…e path

Shabonasar · 2026-10-18T09:49:16Z

Merged current main (aea8711). One conflict, with #622 in src/llm/bedrock/index.ts: the new non-streaming path through generateWithLocalConverter (OpenAI models on Bedrock, Claude 4.6+ without prefill) now also retries once without the signed reasoning when the API rejects it, the same as the Converse path already did; a new test in prefill.test.ts covers it (fails without the retry). Nothing else in the PR changed.

Checked: tsc --noEmit; npx jest src/specs/prompt-cache src/messages src/llm src/summarization src/graphs 3162 passed. Three suites that call live APIs (anthropic, google, vertexai llm.spec.ts) fail the same way on main without credentials. Short re-run on the real Anthropic API with LibreChat-AI/LibreChat#17189 on current dev, Claude Haiku 5.5, effort medium: tools scenario 87.4% read from cache, 0 misses (was 87.7%); G2 85.3%, 0 misses; no API errors.

…ly signatures stream With the thinking display omitted a block streams no text, so the aggregator saw a later block only as a signature and appended it to the previous block's signature. The merged block was stored and replayed, the provider could not verify it and dropped it, which also rewrote the cached prefix. A signature closes its block and arrives whole, so a signature that follows a signed block now starts a new one, for Anthropic and Bedrock alike.

With thinking binding controls the provider reports in `input_transformations` the blocks it dropped from the history because the prefix they were signed under no longer matches. The field reached the response metadata only; a run could lose its signed reasoning without any trace. Emit a warning agent log with the transformations when a response carries any.

Shabonasar · 2026-10-20T04:13:30Z

Two changes after @SwDevOp72's review in LibreChat-AI/LibreChat#17102, found by re-running the series on the real Anthropic API.

Consecutive signed thinking blocks stay apart. With the thinking display omitted a later block arrived as a bare signature and was appended to the previous block's signature; the provider dropped the merged block and the cached prefix was rewritten. A signature that follows a signed block now starts a new one (Anthropic and Bedrock). Tests in aggregator.test.ts fail without the change. Rare (1 request in 86 on Sonnet 5.5, 0 in 120 on Haiku 5.5), so not reproduced live after the fix.
input_transformations is logged: a response that carries any emits a warning agent log with the transformations. Test in Graph.reasoning.test.ts.

With prompt caching on Bedrock, additional_instructions was appended to the system message after its last cachePoint. Bedrock reads tools, system and messages in that order, so a tail that changes between turns sits in front of the whole history and every message checkpoint misses. The tail now moves to a message right before the newest user message, as for Anthropic and OpenRouter, with one cachePoint at the end of the history in front of it and one at the end of the request. The conversation block keeps its own cachePoint only while the four-checkpoint limit leaves room for it (tools, stable block, history, request end come first). Without prompt caching the tail stays in the system message.

…ponses-tool-round-restore

…eplay

…Marks Two Messages The graph can place a second message checkpoint for the latest human message. Beside the system checkpoints and the tool list that overran the limit of four in two places: - the conversation block kept its own cachePoint only on the moved-tail path; the Bedrock system message now drops it whenever the checkpoints already in the messages leave no room, with or without a moved tail; - the opening message of the history in front of a moved tail kept the graph's checkpoint on top of the two the tail path places; it is stripped now.

…ponses-tool-round-restore

…eplay

Sequential tool rounds, stored Responses output items and block tool results are restored as the live request carried them, so the next turn keeps the prefix.

Streamed Anthropic and Bedrock reasoning keeps its signed blocks on the think part (`provider_replay`), and `formatAgentMessages` puts them back for the model that signed them. Anthropic asks the API to drop blocks that no longer match; Bedrock retries once without reasoning. Co-authored-by: Ravi Kumar L <ravi.lazar@clickhouse.com>

Shabonasar · 2026-10-27T17:10:22Z

Reduced to simplify dependencies and the review: builds on #615 only, +7864 -> +1375 lines on top of #615 (+2371 with #615 until it merges).

Moved out: the harness tests (measurement kit), docs (now in the description). Dropped: withdrawal of blocks after an in-run prefix rewrite (the Anthropic drop_block binding and the Bedrock retry cover it at the cost of one miss), logging of input_transformations.
Kept: capture with the serving model, replay for the same provider and model, replay start after the last hole and after a summary, fallback guard, Anthropic drop_block with its beta, one Bedrock retry; the work of upman from feat(messages): persist provider reasoning across runs #371 stays as co-author; regression tests fail without the fix.
Checked: SDK suites; LibreChat agent suites against all five SDK PRs together; live Claude Haiku 5.5: tools with thinking 87.0% cached, 0 misses (dev 80.2%, 3), signed thinking replayed with signatures, no API errors. Bedrock: tests only, no live run.

No file changes (tree of main c56fc5e). Joins an earlier snapshot that the branches of the series share (83eea26), so any two of them have a single merge base and merge cleanly.

…ges)

…Context With endpoints.agents.promptCache.stableContext, the signed thinking blocks of Anthropic Messages API responses are stored beside the message and sent back in the restored history, to the model that signed them only, so the next turn starts with the bytes of the live request. Also through a reverse proxy with a custom base URL (it must forward the anthropic-beta header); not used through Vertex AI (a custom client) or Bedrock. Needs LibreChat-AI/agents#619. Based on LibreChat-AI/agents#371 by upman

Shabonasar · 2026-10-28T05:39:26Z

Branch updated with a merge commit (no file changes) so that the agents PRs of LibreChat-AI/LibreChat#17102 merge together with a plain git merge; the PR diff is unchanged. All five together: integration/prompt-cache.

Shabonasar and others added 5 commits October 15, 2026 10:22

This was referenced Oct 16, 2026

feat: Store and Replay Signed Reasoning for Claude Agents with Stable Context LibreChat-AI/LibreChat#17189

Open

Tracking: stable prompt prefix cache across turns, conversations and providers LibreChat-AI/LibreChat#17102

Open

Shabonasar added 2 commits October 18, 2026 12:00

Merge remote-tracking branch 'upstream/main' into feat/reasoning-replay

6deb6f2

test(bedrock): cover the reasoning retry on the local-converter invok…

e149069

…e path

Shabonasar added 2 commits October 20, 2026 00:03

Shabonasar mentioned this pull request Oct 20, 2026

fix(tools): persist tool output reference labels and keep keys unique across runs #629

Open

6 tasks done

This was referenced Oct 21, 2026

Bedrock prompt cache misses on the next turn: reasoning is not replayed and dynamic instructions sit after the system cache point #617

Open

🔖 fix: Add a Second Bedrock Message Cache Point So Rebuilt Turns Reach the Previous Turn's Cache #631

Open

Shabonasar added 8 commits October 26, 2026 09:59

Merge branch 'fix/prompt-cache-stable-dynamic-placement' into fix/res…

fd82513

…ponses-tool-round-restore

Merge branch 'fix/responses-tool-round-restore' into feat/reasoning-r…

97b91c0

…eplay

Merge branch 'fix/prompt-cache-stable-dynamic-placement' into fix/res…

e5f2ebc

…ponses-tool-round-restore

Merge branch 'fix/responses-tool-round-restore' into feat/reasoning-r…

982c374

…eplay

fix: restore tool rounds and Responses items as sent live

7b4e033

Sequential tool rounds, stored Responses output items and block tool results are restored as the live request carried them, so the next turn keeps the prefix.

This was referenced Oct 27, 2026

fix: Keep Per-Conversation Context and the Summary at a Fixed System Position #613

Open

feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools #614

Open

fix: Restore Tool Rounds Exactly as the Live Request Sent Them #615

Open

Shabonasar added 2 commits October 28, 2026 07:30

chore: join the shared history of the prompt-cache PRs

ff20595

No file changes (tree of main c56fc5e). Joins an earlier snapshot that the branches of the series share (83eea26), so any two of them have a single merge base and merge cleanly.

chore: merge the shared history of the prompt-cache PRs (no file chan…

1b7ec67

…ges)

chore: merge fix/responses-tool-round-restore (no file changes)

81d9c90

Navigation Menu

Repository navigation

feat: Replay Signed Reasoning in Restored History - #619

feat: Replay Signed Reasoning in Restored History#619
Shabonasar wants to merge 20 commits into
LibreChat-AI:mainfrom
Shabonasar:feat/reasoning-replay

Shabonasar commented Oct 16, 2026 •

edited

Loading

Uh oh!

Shabonasar commented Oct 18, 2026

Uh oh!

Shabonasar commented Oct 20, 2026

Uh oh!

Shabonasar commented Oct 27, 2026

Uh oh!

Shabonasar commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Shabonasar commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Type of change

Testing

Risk / compatibility

Checklist

Uh oh!

Shabonasar commented Oct 18, 2026

Uh oh!

Shabonasar commented Oct 20, 2026

Uh oh!

Shabonasar commented Oct 27, 2026

Uh oh!

Shabonasar commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Shabonasar commented Oct 16, 2026 •

edited

Loading