Skip to content

feat(app): a step that read a picture shows the picture, not "[image]" - #212

Draft
Stener1 wants to merge 1 commit into
cosscom:mainfrom
Stener1:tool-result-images
Draft

Stener1 wants to merge 1 commit into
cosscom:mainfrom
Stener1:tool-result-images

Conversation

@Stener1

@Stener1 Stener1 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Opening a tool call that returned a picture (a Read of a .png, an MCP screenshot tool) showed only [image]. It now shows the picture: a thumbnail (up to 320 px tall) where [image] stood, which opens at full size on a click.

The picture comes from the conversation's own record (the tool_result's base64 image block), not from the file. So it shows for files outside the worktree (/tmp/…), and for files changed or deleted since.

 GET …/sessions/{name}/transcript/tool/{id}
-  { output: "…\n[image]", … }
+  { output: "…\n[image]", images: [{ n: 0, line: 1, mediaType: "image/png" }], … }   # never the bytes
+GET …/sessions/{name}/transcript/tool/{id}/image/{n}            # the bytes, Content-Type image/*
+GET …/sessions/{name}/subagents/{id}/tool/{tool}/image/{n}      # the same for a helper's call
 <ToolRow>                      (conversation-view.tsx)
   <ToolDetailView image={…}>
-    <pre>{output}</pre>
+    outputParts(output, images)   (lib/tool-images.ts: each picture at its line)
+      <pre>{text}</pre>
+      <ToolImage n load name>     (tool-image.tsx: boxBlob → blob URL, fetched once the row opens)
+        <Dialog> full size
  • Box (internal/transcript/detail.go): Detail lists the result's pictures in the same pass that reads its text, and skips their data. line is where each picture stands in output: its own [image] line, or the … more lines … line when a long output's middle is cut. So a text block that itself reads [image] stays text. Image decodes only picture n and stops reading at the result. It serves only PNG, JPEG, GIF and WebP (nosniff, a sandbox CSP, Cache-Control: private, immutable).
  • App: fetches the pictures only once a row is opened, through boxBlob, because a plain <img src> can't carry the token. Anything the box can't give back (the record rotated past its 32 MB window, another type) shows [image] as before. It shows no toast for this.
  • The chat's 2 s polling and the 32 KB output cap never carry image bytes.
  • Codex sessions are unchanged, since their results hold no image blocks.

Evidence

The real app is driven against a dev box of this branch (.claude/skills/verify-berthd, new features/chat-tool-calls.md, scenario chat-tool-image-live). A real Claude Code reads a synthetic /tmp/acme-shot.png (480x300) from the chat.

  • Before (main's berthd + main's app, same box): the Read opens to [image].

    before

  • After (terminal driver): the Read opens to the picture. Its naturalWidth is 480, and the row shows no [image] text.

    after

  • After: a click opens it at full size, and Escape closes it.

    full size

  • After: /tmp/acme-shot.png is deleted, then the row is closed and opened again. The picture still shows, read from the record (screenshot).

  • After (structured driver, where the chat is berthd's own host log): the same scenario passes (screenshot).

  • Second view: the scenario fetches …/transcript/tool/{id}/image/0 through the agent. Its body is byte-for-byte the PNG Claude read, with Content-Type: image/png. The console shows no errors.

Tests:

internal/transcript  TestDetailNamesTheImagesAResultHoldsButNotTheirBytes
                       text + png + tiff + gif → images [{0,png},{2,gif}], output unchanged, no base64 in the JSON
                       Image(r1, 0|2) → the exact bytes; 1 (tiff), 3, -1, unknown id → ErrNoImage
                     TestAToolsImagesStandAtTheirOwnLines
                       a text block reading "[image]" before a real picture → the picture's line is its own
                       a 112 KB output with 3 pictures → head, "… more lines …", tail lines
internal/box         TestAToolsImagesAreServedFromTheTranscript
                       session and helper routes: image 0, 1 → bytes + Content-Type; 2, x, unknown call → 404
app/src/lib          tool-images.test.ts (placement, a literal [image] stays text, a cut output)
app/e2e/chat.spec.ts "a call that returned a picture shows it once opened, and opens it at full size"
                       not requested before the row opens; thumbnail and full size naturalWidth 640

Checks run locally (fork PRs wait for approval before CI runs):

  • Passed: go vet ./..., go test -race ./..., pnpm check:*, pnpm test (632), pnpm build, npx tsc -p e2e, the touched e2e specs, and docs-site build.
  • Failed, and fails on main too: TestSleepDurable (internal/box/runs) failed once under load and passed 5/5 when rerun. keyboard.spec.ts "⌘⇧E puts the keyboard in the Files panel" fails on origin/main as well.
  • Not run: cargo check. This box lacks the GTK dev libraries, and no Rust changed.

Merge Danger

Door: two-way

A new GET route and an additive JSON field. Nothing is written anywhere, so a revert removes the change cleanly. An older app ignores images, and an older box sends none, so the app shows [image] as before.

Blast Radius: chat

  • Only an opened tool row changes.
  • The new route reads the same transcript window as the tool detail, and only for paired peers.
  • The mock's release chat now has a third Read (checkout-result.png). "a tool call opens to what it did" looks for "Read 3 files".

🤖 Generated with Claude Code

View guided diff

Opening a tool call whose result holds images (a Read of a .png, a
screenshot tool) showed only "[image]". The detail now names the result's
pictures (images: [{n, line, mediaType}], never their bytes; line is where
each stands in the output, also once its middle is cut), and the box serves
each one from the conversation's record at GET
…/transcript/tool/{id}/image/{n} (and a helper's …/subagents/{id}/tool/
{tool}/image/{n}): PNG, JPEG, GIF and WebP only. The app fetches them once
the row is open, through boxBlob, as thumbnails that open at full size; a
picture it can't get stays "[image]".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 30, 2026

Copy link
Copy Markdown

@Stener1 is attempting to deploy a commit to the seanbrydonio's projects Team on Vercel.

A member of the Team first needs to authorize it.

@Stener1

Stener1 commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Opened as a draft because the author's ready-PR limit is full; it is marked ready automatically when a slot frees. I can't add the run-ci label from a fork: a maintainer please add it. The checks listed under Evidence were run locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant