Repository navigation
agent: verify chatd handling of content tool result outputs (toModelOutput) #404
Description
@coder/ai-sdk-agent forwards a content tool result output to chatd unchanged. Nobody has checked what chatd does with it.
Where
packages/agent/src/model/prompt.tstoolResultOutputToChatd:case "content"returnsoutput.value(an array oftext/fileitems, file data as{ type: "data", data }).- The resume path then submits that array as the client tool result
output.
Who hits it
- AI SDK v7 users whose client tools set
toModelOutput(AI SDK core emitscontentoutputs). @coder/ai-sdk-effectusers ofCoderAgentModelwhose effect/ai tools settoModelOutput(effect >= 4.0.7). Since chore: bump effect to 4.0.7 #403, the effect bridge maps effect/aimodelOutputonto the samecontentoutput.
What to check
- Run one live turn with a client tool whose result is a
contentoutput with a text part and a small PNG file part. - See whether chatd gives the model the image, or serializes the array as JSON text (base64 inside the text).
- If chatd cannot take images in tool results, map file items to text placeholders (as effect's
Prompt.toolModelOutputToTextdoes for OpenAI-compatible APIs) or upload them, and document the limit.
Found by the advisor pass on #403. CoderLanguageModel (Gateway) is not affected: the AI SDK providers handle content outputs natively.
Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high
Activity
ThomasK33 commentedon Oct 29, 2026
Result: chatd does not show the image to the model. It stores the
contentarray as opaque JSON and gives it to the model as text, with the PNG as base64 characters.Source (coder/coder
8d643af, the commit dogfood runs):codersdk.ToolResulthas onlytool_call_id,output(json.RawMessage), andis_error(codersdk/chats.go:871-876). A client cannot mark a result as media.Result: r.Outputand noIsMedia(coderd/x/chatd/chatstate/transitions.go:1247-1253).string(part.Result)to the LLM as a text tool result (coderd/x/chatd/chatprompt/chatprompt.go:1379-1385, 1464-1472). Only internalIsMediaresults (computer-use screenshots) become media (chatprompt.go:1414-1430).Live probe on dogfood (
sonnet, client tool returns a text part and a 4x4 solid red PNG, both chats archived):resultwas the raw array with the base64 PNG. The model answeredRECEIVED: TEXT,COLOR: UNKNOWN.resultis"Swatch label: PROBE-404-LABEL\n[image/png swatch.png omitted]". The model answeredRECEIVED: NOTE,COLOR: UNKNOWN.#405 flattens
contentoutputs to text in@coder/ai-sdk-agentand documents the limit. Images in client tool results need a chatd change (anis_media-style field onToolResult).Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high