Repository navigation
fix(agent): flatten content tool outputs to text for chatd - #405
Conversation
chatd stores a client tool result as opaque JSON and gives it to the model as text. A `content` output (from a tool's `toModelOutput`) reached the model as a JSON array with the file bytes as base64 characters. Keep the text parts, replace each file part with an "[<mediaType> <filename> omitted]" note, and submit one string. Closes #404 Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_ Change-Id: Ic94476e1bfc5933b2a18442dc3a3dcf49a6b1c07
|
@codex review |
|
@codex security review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
An inline text file (`{ type: "text" }` data) now passes through as text,
and a URL file keeps its URL in the note. Both reached the model before
this branch as part of the raw JSON array.
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
Change-Id: I034d3bf0abec212700176931e1516e31edba5c8f
Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Closes #404
What changes
A client tool that returns a
contentoutput (AI SDKtoModelOutput, or effect/aitoModelOutputthroughCoderAgentModelsince #403) now submits one text string to chatd:{ type: "text" }data) passes through as its text.[image/png chart.png at https://…].[image/png swatch.png omitted](the format of effect'sPrompt.toolModelOutputToText).custompart becomes[custom content omitted].Before,
toolResultOutputToChatdforwarded the raw array. The model then read the file bytes as base64 characters, and never saw an image. AUint8Arrayfile payload went over the wire as a JSON object of byte indexes.The README "Custom tools" section states the limit: the model does not see images in client tool results. To show the model a file, put it in a user message as a
filepart.Why chatd cannot show the image (verified against source and live)
Source: coder/coder
8d643af(the commit dogfood runs, v2.38.4-devel+8d643afee4).codersdk.ToolResulthas onlytool_call_id,output(json.RawMessage), andis_error(codersdk/chats.go:871-876). A client cannot setis_media.CompleteRequiresActionstores each result as atool-resultpart withResult: r.Outputand noIsMedia(coderd/x/chatd/chatstate/transitions.go:1247-1253). Validation only checksjson.Valid(chatstate/toolresults.go:13).toolResultPartToMessagePartsendsstring(part.Result)to the LLM asToolResultOutputContentText(coderd/x/chatd/chatprompt/chatprompt.go:1379-1385, 1464-1472). OnlyIsMediaresults (the computer-use screenshot shape{"data","mime_type","text"}) become media (chatprompt.go:1414-1430).Live probe on dogfood (
sonnet, client tool returns a text part plus a 4x4 solid red PNG):outputresult8d689ca)RECEIVED: TEXT,COLOR: UNKNOWN"Swatch label: PROBE-404-LABEL\n[image/png swatch.png omitted]"RECEIVED: NOTE,COLOR: UNKNOWNBoth probe chats are archived.
Showing images from client tool results to the model needs a chatd change (for example an
is_media-style field onToolResult). That is out of scope here.Validation
pnpm install --frozen-lockfile,pnpm -r build,pnpm check,pnpm -r test(agent 379, sandbox 193, provider 22, release-please-ai 13, effect 101 unit),pnpm publint,pnpm attw: all pass.mainand passes with the fix.3b56489) dropped inline text files and file URLs, which reached the model before.ac8b577keeps them. Gates and live e2e 7/7 rerun onac8b577.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high