Repository navigation
feat: convert Anthropic tool result media like user files - #73
ethanndickson wants to merge 3 commits into
Conversation
2a6d7dc to
e7fe298
Compare
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
ToolResultOutputContentMedia and ToolResponse gain an optional Filename, and the agent copies it into the media tool result. FilePart() decodes the base64 data so providers can convert tool-result media with the same rules they use for user files.
Tool-result media always became an image block, so a PDF from a tool got a 400 from the API. Tool media now goes through the same file converter as user files: a PDF becomes a document block titled with the filename, text becomes a text document, and images stay images. Blocks inside tool_result get no cache_control; the breakpoint stays on tool_result. Media that cannot be converted becomes a text note plus upstream's warning, "tool result media type %s not supported, sending text placeholder only". The note uses "file" when the media type is empty.
e7fe298 to
574b1ff
Compare
Tool-result media data is already base64, but FilePart() decoded it and the Anthropic converter encoded it again. For a 4 MiB image this costs about 4 ms and 15 MB of extra allocation per request. providers/internal/filepart.File now keeps the content in the encoding of its source: raw bytes for user file parts, base64 for tool-result media. Images and PDFs send the base64 unchanged. Only text documents decode it. FilePart() is removed. Invalid base64 in an image or PDF now goes to the API unchanged, the same as upstream. Invalid base64 in text media still becomes a text notice and a warning.
|
@codex review |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Tool result media had no filename, and each provider converted it with its own code rather than the code it already uses for user file attachments. Those copies drifted apart. On Anthropic, tool media always became an
imageblock, so a PDF from an MCP tool got a 400, which is why coder/coder currently swaps PDF tool results for a note before they reach the provider.This adds an optional
FilenametoToolResultOutputContentMediaandToolResponse, and an internalfilepart.Filetype that holds either a user file's bytes or tool media's base64, so a provider can run tool media through its user file converter without decoding it first. Stored results without a filename still decode as before.On Anthropic, we'll now do just that. A PDF becomes a
documentblock inside thetool_result, titled with the filename,text/*becomes a text document, and images stay images. The nested blocks don't getcache_control, as the breakpoint stays on thetool_resultand Anthropic only allows four. Anything we can't convert becomes a short note saying the content can't be displayed, plus upstream's warning, rather than an invalid block.I sent a PDF tool result to Claude Haiku through our AI gateway with this stack, and it read the PDF fine. I've since run it through chatd on a local coderd, where PDF and PNG results from an MCP server (and from a workspace agent's MCP server) reached Claude as
documentandimageblocks. That included a turn where Claude thought before calling the tools, so the signed thinking block was replayed next to the PDFtool_result, and Anthropic accepted it. #74 does the same for OpenAI.Part of CODAGT-1430