diff --git a/.claude/skills/verify-berthd/SKILL.md b/.claude/skills/verify-berthd/SKILL.md index d484f15e8..1f2714927 100644 --- a/.claude/skills/verify-berthd/SKILL.md +++ b/.claude/skills/verify-berthd/SKILL.md @@ -37,7 +37,7 @@ Read only. It checks the port answers with the app, that the PID in `/tmp/verify node .claude/skills/verify-berthd/scripts/drive.mjs --port $PORT [--out DIR] ``` -`` is a file in `scripts/scenarios/` (by name, without `.mjs`) or a path. With `--agent http://127.0.0.1:9378` and `VERIFY_BERTH_TOKEN` in the environment it opens the app on that laptop agent instead of the mock (a scenario that needs it exports `live = true`), and gives the scenario `call(path, init)`, the agent's API (a box's own under `/v1/boxes//api/`), to read a change's effect on the box. The driver: +`` is a file in `scripts/scenarios/` (by name, without `.mjs`) or a path. With `--agent http://127.0.0.1:9378` and `VERIFY_BERTH_TOKEN` in the environment it opens the app on that laptop agent instead of the mock (a scenario that needs it exports `live = true`), and gives the scenario `call(path, init)`, the agent's API (a box's own under `/v1/boxes//api/`), to read a change's effect on the box (JSON; with `init.raw`, `{ type, bytes }` for a body that isn't). The driver: - opens `http://localhost:$PORT/?mock=1` at 1440×900 in dark mode, and waits until the sidebar's `nav-home` is there; - refuses every request to the laptop agent's ports (1377-1379), as the e2e fixtures do; diff --git a/.claude/skills/verify-berthd/features/README.md b/.claude/skills/verify-berthd/features/README.md index 11a07f637..66ed97972 100644 --- a/.claude/skills/verify-berthd/features/README.md +++ b/.claude/skills/verify-berthd/features/README.md @@ -19,6 +19,7 @@ The maintained list of what a person can do in the app, and how to prove each on | Plugins: built-in and your own, what they add (screens, sidebar items, widgets, panels) | [plugins.md](plugins.md) | `plugin-nav-badges` (sidebar items); `plugin-attach-live` (`api.attach`, a real box, `--agent`) | | Home's widget grid | [home-widgets.md](home-widgets.md) | none yet; `app/e2e/home-widgets.spec.ts` asserts it | | Questions in chat: an agent's form of questions as a card, answered on a real box; kept with Retry and Copy when it can't be | [chat-questions.md](chat-questions.md) | `chat-questions-live` (a real box, `--agent`); `app/e2e/chat.spec.ts` on the mock | +| Tool calls in chat: a turn's steps folded and grouped, a call opened to its detail, the pictures a call returned (from the record, at full size on a click) | [chat-tool-calls.md](chat-tool-calls.md) | `chat-tool-image-live` (a real box, `--agent`); `app/e2e/chat.spec.ts` on the mock | | Chat's foot and reply box: a live turn's spinner, timer, Stop and Queue; a turn that ended while its work runs on shows an hourglass and Send | [chat-turn-state.md](chat-turn-state.md) | `chat-turn-state`; `chat-turn-state-live` (a real box, `--agent`); `app/e2e/chat.spec.ts` asserts it | | Chat and board: no question card or answers but the options the agent's screen shows; the mode menu while turn check waits; a structured agent's form while its helpers report back, and a reply box that answers a card with nothing to pick | [chat-ask-card.md](chat-ask-card.md) | `chat-ask-card`, `chat-structured-ask-live` (a real box, `--agent`); `app/e2e/ask-card.spec.ts` on a stand-in box | | The board's cards for agents in chat: what they ask with its answers, their reply's last lines, the prompt they started with; never the host's frames or "stdio" | [chat-board-cards.md](chat-board-cards.md) | `chat-board-cards`; `app/e2e/dashboard-chat-cards.spec.ts` asserts it | diff --git a/.claude/skills/verify-berthd/features/chat-tool-calls.md b/.claude/skills/verify-berthd/features/chat-tool-calls.md new file mode 100644 index 000000000..3dc560c2b --- /dev/null +++ b/.claude/skills/verify-berthd/features/chat-tool-calls.md @@ -0,0 +1,40 @@ +# Chat: tool calls, opened + +An agent's steps fold into the chat. A turn's steps sit under one line (**Working · read 2 files**, then **Worked …** once it ends); under it, calls of a kind group (**Read 2 files**, **Searched 3 times**, **Ran 2 commands**), and each call is a row: its verb, its target, and a chevron. Opening a row asks the box for that call's detail (GET `…/sessions/{name}/transcript/tool/{id}`; a helper's `…/subagents/{id}/tool/{tool}`) and draws it as the terminal does: a command and its output, a search and its matches, an edit's change, a written file (`app/src/components/conversation/conversation-view.tsx` `Tools`, `ToolRow`, `ToolDetailView`; `internal/transcript/detail.go`). + +A call that returned pictures (a Read of a .png or .jpg, a screenshot tool) shows each picture where the terminal says `[image]`, from the conversation's own record, not the file: it shows for files outside the worktree, changed or deleted since. The detail only names them (`images: [{n, line, mediaType}]`, `line` being where each stands in the output: its own `[image]` line, or the `… more lines …` line when the middle of a long output is cut, so a text block that itself reads `[image]` stays text); the app fetches each one's bytes once the row is open (GET `…/tool/{id}/image/{n}`, through the agent with its token, as a blob URL), as a thumbnail up to 320 px tall that opens at full size (`tool-image.tsx`, `lib/tool-images.ts`). Only PNG, JPEG, GIF and WebP are served; anything else, or a picture the box can't read back (the record rotated past the box's 32 MB window), stays the text `[image]`. + +## Sub-features + +- `tool-groups`: a turn's steps fold under one line; calls group by kind with a count; a group still running opens, and folds to its summary a moment after it finishes in view. +- `tool-detail`: a row opens to its detail, read when opened; a call that is gone from the record says so. +- `tool-images`: a call's pictures in place of its `[image]` lines, fetched only once opened; a click opens one at full size, Escape closes it; the picture is the record's, so it outlives the file. + +## How to get to it (user POV) + +- Open an agent in chat, open **Worked …** and a group such as **Read 3 files**, then a row. +- On the mock: `gpu/shop`'s chat, **Read 3 files**: `webhook.ts` opens to its lines, `checkout-result.png` to a drawn checkout page. +- On a real box: ask the agent to read a picture, such as a screenshot in `/tmp`. + +## Driving it with drive.mjs + +- Mock: `app/e2e/chat.spec.ts` asserts it ("a tool call opens to what it did", "a call that returned a picture shows it once opened, …": the picture isn't asked for before the row opens, `window.__berthToolImages`). +- Real box: `scenarios/chat-tool-image-live.mjs` (`live = true`), on a dev box of this checkout, in either driver: + +```sh +VERIFY_BOX_DIR=/tmp/vb-$$ .claude/skills/verify-berthd/scripts/dev-box.sh up # or VERIFY_DRIVER=claude … up +eval "$(VERIFY_BOX_DIR=/tmp/vb-$$ .claude/skills/verify-berthd/scripts/dev-box.sh env)" +VERIFY_WORKTREE=devq/acme-shop node .claude/skills/verify-berthd/scripts/drive.mjs chat-tool-image-live --port $PORT --agent http://127.0.0.1:9378 +# The build before: [image] as text. Main's berthd and main's app, same box: +VERIFY_BOX_DIR=/tmp/vb-$$ .claude/skills/verify-berthd/scripts/dev-box.sh restart
/bin/berthd +VERIFY_EXPECT=text VERIFY_WORKTREE=devq/acme-shop node .claude/skills/verify-berthd/scripts/drive.mjs chat-tool-image-live --port --agent http://127.0.0.1:9378 +``` + +It draws `/tmp/acme-shot.png` (a synthetic acme checkout page, 480x300), has Claude Code read it from the chat, and waits until that Read's detail names one PNG. Second view: the box's image route, through the agent (`call(…, { raw: true })`), gives back the very bytes of the file. Then it opens the row (the thumbnail's `naturalWidth` is 480, no `[image]` text), opens it full size and closes it, deletes the file, and opens the row again: the picture still shows. + +- Go: `TestDetailNamesTheImagesAResultHoldsButNotTheirBytes` and `TestAToolsImagesStandAtTheirOwnLines` (`internal/transcript`), `TestAToolsImagesAreServedFromTheTranscript` (`internal/box`: the session's and a helper's route, a bad index and an unknown call are 404). + +## Gotchas + +- A terminal agent's folder that Claude Code already trusts gets dev-box.sh's Up+Enter as a prompt, which re-sends the last one: on a second `up` in the same `VERIFY_BOX_DIR` the chat has an extra Read. The scenario waits for a Read after its own prompt, so it still holds; for clean screenshots use a fresh `VERIFY_BOX_DIR`, or the structured driver. +- The pictures are never in the detail itself, so a chat's 2 s polling and the 32 KB output cap never carry image bytes. The box marks them cacheable (`private, immutable`): a row opened again may not ask the box at all. diff --git a/.claude/skills/verify-berthd/scripts/drive.mjs b/.claude/skills/verify-berthd/scripts/drive.mjs index 4917aae96..633c73df1 100755 --- a/.claude/skills/verify-berthd/scripts/drive.mjs +++ b/.claude/skills/verify-berthd/scripts/drive.mjs @@ -87,10 +87,12 @@ const shot = async (label) => { // call(path, init) is the laptop agent's API as the app calls it (a box's // own under /v1/boxes//api/), for a live scenario to read a change's -// effect on the box. -const call = async (path, init = {}) => { +// effect on the box. init.raw answers with the body's bytes and type +// instead of its JSON. +const call = async (path, { raw, ...init } = {}) => { const res = await fetch(`${agent}${path}`, { ...init, headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json", ...init.headers } }); if (!res.ok) throw new Error(redact(`${init.method ?? "GET"} ${path}: ${res.status} ${await res.text()}`)); + if (raw) return { type: res.headers.get("content-type"), bytes: Buffer.from(await res.arrayBuffer()) }; return res.json(); }; diff --git a/.claude/skills/verify-berthd/scripts/scenarios/chat-tool-image-live.mjs b/.claude/skills/verify-berthd/scripts/scenarios/chat-tool-image-live.mjs new file mode 100644 index 000000000..bfde2f318 --- /dev/null +++ b/.claude/skills/verify-berthd/scripts/scenarios/chat-tool-image-live.mjs @@ -0,0 +1,153 @@ +// chat-tool-image-live: a real Claude Code on a dev box (../dev-box.sh up) +// reads a picture outside its worktree, /tmp/acme-shot.png, drawn here. +// Its Read opens to the picture itself, read back from the conversation's +// record by the box (GET …/transcript/tool/{id}/image/0), and the picture +// opens at full size. Then the file is deleted and the call opened again: +// the picture still shows, so it never came from the file. +// See ../../features/chat-tool-calls.md. +// +// VERIFY_WORKTREE the worktree row's box/name (devq/acme-shop). +// VERIFY_SESSION its agent's session (ask). +// VERIFY_EXPECT image (the default) or text: the build before, which +// shows the call's output as the text "[image]". +import { existsSync, rmSync, writeFileSync } from "node:fs"; + +export const live = true; +export const params = { view: "conversation", sidebar: "list" }; + +const SHOT = "/tmp/acme-shot.png"; +const WIDTH = 480; +const HEIGHT = 300; + +export async function run({ page, shot, step, expect, call }) { + const expectNow = process.env.VERIFY_EXPECT ?? "image"; + const worktree = process.env.VERIFY_WORKTREE; + if (!worktree) throw new Error("set VERIFY_WORKTREE to the worktree row's box/name"); + const [box] = worktree.split("/"); + + step(`draw ${SHOT}: an acme checkout page, ${WIDTH}x${HEIGHT}`); + const png = await page.evaluate( + ([w, h]) => { + const c = document.createElement("canvas"); + c.width = w; + c.height = h; + const g = c.getContext("2d"); + g.fillStyle = "#f8fafc"; + g.fillRect(0, 0, w, h); + g.fillStyle = "#0f172a"; + g.fillRect(0, 0, w, 48); + g.fillStyle = "#fff"; + g.font = "bold 20px sans-serif"; + g.fillText("acme shop", 18, 31); + g.fillStyle = "#16a34a"; + g.beginPath(); + g.arc(w / 2, 130, 36, 0, Math.PI * 2); + g.fill(); + g.fillStyle = "#0f172a"; + g.font = "bold 24px sans-serif"; + g.textAlign = "center"; + g.fillText("Order confirmed", w / 2, 210); + g.font = "16px sans-serif"; + g.fillStyle = "#475569"; + g.fillText("Order #1042 · 3 items · $84.00", w / 2, 240); + return c.toDataURL("image/png").split(",")[1]; + }, + [WIDTH, HEIGHT], + ); + const bytes = Buffer.from(png, "base64"); + writeFileSync(SHOT, bytes); + + step("open the agent's chat"); + await page.locator(`[data-testid=worktree-row][data-worktree="${worktree}"]`).click({ position: { x: 32, y: 12 } }); + const pane = page.locator("[data-testid=pane]:visible").first(); + const chat = pane.getByTestId("chat"); + await expect(chat).toBeVisible({ timeout: 30_000 }); + let reply = pane.getByRole("textbox", { name: "What should Claude Code do?", exact: true }); + try { + await expect(reply).toBeEditable({ timeout: 15_000 }); + } catch { + reply = pane.getByTestId("composer").getByRole("textbox", { name: "Reply" }); + } + const session = process.env.VERIFY_SESSION ?? "ask"; + const api = `/v1/boxes/${box}/api/sessions/${encodeURIComponent(session)}`; + // Its Reads of the picture so far, so this run's is the one after them. + const reads = async () => { + const res = await call(`${api}/transcript?since=0`); + return (res.items ?? []).flatMap((it) => (it.kind === "tools" ? (it.items ?? []) : [])).filter((c) => c.target?.includes("acme-shot.png") && c.id); + }; + const before = (await reads()).length; + step(`ask Claude to read ${SHOT}`); + await reply.fill(`Use the Read tool on ${SHOT}, then say in one short sentence what it shows. Do nothing else.`); + await reply.press("Enter"); + + step("its Read of the picture is in the chat, answered"); + let id; + await expect + .poll( + async () => { + const all = await reads(); + if (all.length <= before) return "no read yet"; + id = all[all.length - 1].id; + const d = await call(`${api}/transcript/tool/${encodeURIComponent(id)}`); + return d.pending ? "pending" : JSON.stringify(d.images ?? []); + }, + { timeout: 240_000, intervals: [3_000] }, + ) + .toBe(expectNow === "image" ? JSON.stringify([{ n: 0, line: 0, mediaType: "image/png" }]) : "[]"); + + if (expectNow === "image") { + step("second view: the box serves the very bytes Claude read, from its record"); + const got = await call(`${api}/transcript/tool/${encodeURIComponent(id)}/image/0`, { raw: true }); + if (got.type !== "image/png" || !got.bytes.equals(bytes)) throw new Error(`image 0 is ${got.type}, ${got.bytes.length} bytes; the file is ${bytes.length}`); + } + + // The steps fold away once the turn ends: open them, then the Read. + const row = chat.getByRole("button", { name: /^Read\b.*acme-shot\.png$/ }).last(); + const openRead = async () => { + const fold = chat.getByRole("button", { name: /^Work(ed|ing)/ }).last(); + if ((await fold.count()) && (await fold.getAttribute("aria-expanded")) === "false") await fold.click(); + const group = chat.getByRole("button", { name: /^Read \d+ files?/ }).last(); + await expect(group).toBeVisible({ timeout: 30_000 }); + if ((await group.getAttribute("aria-expanded")) !== "true") await group.click(); + await expect(row).toBeVisible(); + if ((await row.getAttribute("aria-expanded")) !== "true") await row.click(); + await expect(row).toHaveAttribute("aria-expanded", "true"); + }; + step("open the steps, then the Read"); + await openRead(); + + if (expectNow === "text") { + await expect(chat.getByText("[image]", { exact: true })).toBeVisible({ timeout: 15_000 }); + await expect(chat.getByTestId("tool-image")).toHaveCount(0); + await shot("1-read-shows-text"); + rmSync(SHOT, { force: true }); + return; + } + if (expectNow !== "image") throw new Error("set VERIFY_EXPECT to image or text"); + + const thumb = chat.getByTestId("tool-image").last(); + await expect(thumb).toBeVisible({ timeout: 15_000 }); + await expect.poll(() => thumb.locator("img").evaluate((el) => el.naturalWidth)).toBe(WIDTH); + await expect(row.locator("xpath=..").getByText("[image]", { exact: true })).toHaveCount(0); + await row.scrollIntoViewIfNeeded(); + await shot("1-read-shows-picture"); + + step("the picture opens at full size, and closes with Escape"); + await thumb.click(); + const full = page.getByTestId("tool-image-full"); + await expect(full).toBeVisible(); + await expect.poll(() => full.locator("img").evaluate((el) => el.naturalWidth)).toBe(WIDTH); + await shot("2-full-size"); + await page.keyboard.press("Escape"); + await expect(full).toBeHidden(); + + step(`delete ${SHOT}, close and open the Read again: the picture is the record's`); + rmSync(SHOT, { force: true }); + if (existsSync(SHOT)) throw new Error(`${SHOT} is still there`); + await expect(row).toBeVisible(); + if ((await row.getAttribute("aria-expanded")) === "true") await row.click(); + await expect(row).toHaveAttribute("aria-expanded", "false"); + await openRead(); + await expect.poll(() => chat.getByTestId("tool-image").last().locator("img").evaluate((el) => el.naturalWidth)).toBe(WIDTH); + await shot("3-after-the-file-is-gone"); +} diff --git a/app/e2e/chat.spec.ts b/app/e2e/chat.spec.ts index e9019f0cf..694ef408c 100644 --- a/app/e2e/chat.spec.ts +++ b/app/e2e/chat.spec.ts @@ -130,7 +130,7 @@ test("a tool call opens to what it did", async ({ app }) => { const fold = app.chat.getByRole("button", { name: /^Work(ed|ing)/ }).first(); await fold.click(); await expect(fold).toHaveAttribute("aria-expanded", "true"); - const group = app.chat.getByRole("button", { name: /Read 2 files/ }); + const group = app.chat.getByRole("button", { name: /Read 3 files/ }); await group.click(); const call = app.chat.getByRole("button", { name: /Read\s*ts\s*webhook\.ts/ }); await call.click(); @@ -138,6 +138,31 @@ test("a tool call opens to what it did", async ({ app }) => { await expect(app.chat.getByText("export async function handleWebhook(req: Request)")).toBeVisible(); }); +test("a call that returned a picture shows it once opened, and opens it at full size", async ({ app }) => { + mockOnly(); + await app.openWorktree("gpu/shop"); + const fold = app.chat.getByRole("button", { name: /^Work(ed|ing)/ }).first(); + await fold.click(); + await app.chat.getByRole("button", { name: /Read 3 files/ }).click(); + const call = app.chat.getByRole("button", { name: /^Read\b.*checkout-result\.png$/ }); + await expect(call).toBeVisible(); + // Nothing is fetched for a call that stays closed. + const asked = () => app.page.evaluate(() => (window as unknown as { __berthToolImages?: string[] }).__berthToolImages ?? []); + expect(await asked()).toEqual([]); + await call.click(); + const thumb = app.chat.getByTestId("tool-image"); + await expect(thumb).toBeVisible(); + await expect.poll(() => thumb.locator("img").evaluate((el: HTMLImageElement) => el.naturalWidth)).toBe(640); + expect(await asked()).toEqual(["mock-read-shot/0"]); + await expect(app.chat.getByText("[image]", { exact: true })).toHaveCount(0); + await thumb.click(); + const full = app.page.getByTestId("tool-image-full"); + await expect(full).toBeVisible(); + await expect.poll(() => full.locator("img").evaluate((el: HTMLImageElement) => el.naturalWidth)).toBe(640); + await app.page.keyboard.press("Escape"); + await expect(full).toBeHidden(); +}); + test("a form of questions steps through to a review and submits", async ({ app }) => { mockOnly("answers a question on a box"); await app.openWorktree("gpu/shop"); diff --git a/app/src/components/conversation/conversation-pane.tsx b/app/src/components/conversation/conversation-pane.tsx index 5ab30b90d..0f93ac000 100644 --- a/app/src/components/conversation/conversation-pane.tsx +++ b/app/src/components/conversation/conversation-pane.tsx @@ -244,6 +244,7 @@ export function ConversationPane({ box, session, agent: remembered, visible, onS return { load: (file) => boxApi.diff(client, box, session, file), tool: (id) => (mock ? mockConversation().then((m) => m.mockToolDetail(id)) : boxApi.toolDetail(client, box, session, id)), + toolImage: (id, n) => (mock ? mockConversation().then((m) => m.mockToolImage(id, n)) : boxApi.toolImage(client, box, session, id, n)), comments: (file) => reviewKey ? { diff --git a/app/src/components/conversation/conversation-view.tsx b/app/src/components/conversation/conversation-view.tsx index eff616c5c..be134129f 100644 --- a/app/src/components/conversation/conversation-view.tsx +++ b/app/src/components/conversation/conversation-view.tsx @@ -12,6 +12,8 @@ import { ON_ITS_WORK } from "@/lib/derive"; import { errorMessage } from "@/lib/format"; import type { LineComments } from "@/lib/git/diff-view"; import { type ToolCall, type ToolDetail, toolSummary, type TranscriptItem } from "@/lib/transcript"; +import { outputParts } from "@/lib/tool-images"; +import { ToolImage } from "@/components/conversation/tool-image"; import { cn } from "@/lib/utils"; import { Markdown } from "@/components/conversation/markdown"; import { HARBOUR_WORDS } from "@/lib/screen-status"; @@ -70,6 +72,8 @@ export interface EditActions { load(file: string): Promise; // One tool call opened up: its full command and output, or exact change. tool?(id: string): Promise; + // The n-th picture a call returned. + toolImage?(id: string, n: number): Promise; comments(file: string): LineComments | undefined; review(file: string): void; } @@ -821,6 +825,12 @@ function ToolRow({ c, last, Icon, edits }: { c: ToolCall; last: boolean; Icon: t const [open, setOpen] = useState(false); const [detail, setDetail] = useState<{ state: "loading" } | { state: "ready"; d: ToolDetail } | { state: "error"; message: string }>(); const can = !!(c.id && edits?.tool); + // Its pictures' loader stays the same while the chat's actions change + // (a review comment), so an open picture isn't fetched again. + const toolImage = useRef(edits?.toolImage); + toolImage.current = edits?.toolImage; + const hasImages = !!edits?.toolImage; + const image = useMemo(() => (c.id && hasImages ? (n: number) => toolImage.current!(c.id!, n) : undefined), [c.id, hasImages]); const ext = c.file ? (c.target.split(".").pop() ?? "") : ""; const toggle = () => { if (!can) return; @@ -870,7 +880,7 @@ function ToolRow({ c, last, Icon, edits }: { c: ToolCall; last: boolean; Icon: t )} {detail?.state === "error" &&

Couldn't open this step: {detail.message}

} - {detail?.state === "ready" && } + {detail?.state === "ready" && } )} @@ -878,8 +888,9 @@ function ToolRow({ c, last, Icon, edits }: { c: ToolCall; last: boolean; Icon: t } // ToolDetailView draws a call the way the terminal does: a command and its -// output, a search and its matches, an edit's change, a new file. -export function ToolDetailView({ d }: { d: ToolDetail }) { +// output, a search and its matches, an edit's change, a new file, and the +// pictures it returned, fetched with image. +export function ToolDetailView({ d, image, name }: { d: ToolDetail; image?: (n: number) => Promise; name: string }) { const edit = d.old != null || (d.new != null && !d.command); return (
@@ -891,7 +902,13 @@ export function ToolDetailView({ d }: { d: ToolDetail }) { )} {edit && } {!edit && (d.output ? ( - 
{plain(d.output)}
+ outputParts(plain(d.output), image && d.images).map((p, i) => + p.kind === "image" ? ( + + ) : ( + 
{p.text}
+ ), + ) ) : (

{d.pending ? "Still running…" : "No output."}

))} diff --git a/app/src/components/conversation/subagent-view.tsx b/app/src/components/conversation/subagent-view.tsx index 12f125fda..f2df645eb 100644 --- a/app/src/components/conversation/subagent-view.tsx +++ b/app/src/components/conversation/subagent-view.tsx @@ -263,6 +263,7 @@ export function HelperChat({ box, session, h, wide }: { box: string; session: st ? { load: (file) => boxApi.diff(client, box, session, file), tool: (id) => historyApi.helperTool(client, box, session, h.id, id), + toolImage: (id, n) => historyApi.helperToolImage(client, box, session, h.id, id, n), comments: () => undefined, review: () => useStore.getState().setView({ kind: "review" }), } diff --git a/app/src/components/conversation/tool-image.tsx b/app/src/components/conversation/tool-image.tsx new file mode 100644 index 000000000..9a536ae57 --- /dev/null +++ b/app/src/components/conversation/tool-image.tsx @@ -0,0 +1,63 @@ +import { useEffect, useState } from "react"; + +import { Dialog, DialogPopup, DialogTitle } from "@/components/ui/dialog"; +import { Spinner } from "@/components/ui/spinner"; + +type State = { kind: "loading" } | { kind: "ready"; url: string } | { kind: "unavailable" }; + +// ToolImage is a picture a tool call returned, read from the box's record of +// the conversation (never the file, which may be gone or changed since): +// a thumbnail that opens to its full size. One the box can't give back reads +// as the "[image]" the terminal shows. +export function ToolImage({ n, load, name }: { n: number; load: (n: number) => Promise; name: string }) { + const [state, setState] = useState({ kind: "loading" }); + const [full, setFull] = useState(false); + useEffect(() => { + let live = true; + let url: string | undefined; + setState({ kind: "loading" }); + load(n).then( + (blob) => { + url = URL.createObjectURL(blob); + if (live) setState({ kind: "ready", url }); + else URL.revokeObjectURL(url); + }, + () => { + if (live) setState({ kind: "unavailable" }); + }, + ); + return () => { + live = false; + if (url) URL.revokeObjectURL(url); + }; + }, [load, n]); + + if (state.kind === "unavailable") return 
[image]
; + if (state.kind === "loading") + return ( +
+ + + +
+ ); + return ( +
+ + + + {name} + {name} + + +
+ ); +} diff --git a/app/src/lib/api.ts b/app/src/lib/api.ts index 8a4b74241..4470fb909 100644 --- a/app/src/lib/api.ts +++ b/app/src/lib/api.ts @@ -325,6 +325,8 @@ export const boxApi = { // One file's diff in the session's worktree ("diff" capability). diff: (c: Client, box: string, name: string, file: string) => c.box(box, "GET", `sessions/${encodeURIComponent(name)}/diff?${new URLSearchParams({ file })}`), toolDetail: (c: Client, box: string, name: string, id: string) => c.box(box, "GET", `sessions/${encodeURIComponent(name)}/transcript/tool/${encodeURIComponent(id)}`), + // A picture a call returned, as the transcript holds it. + toolImage: (c: Client, box: string, name: string, id: string, n: number) => c.boxBlob(box, `sessions/${encodeURIComponent(name)}/transcript/tool/${encodeURIComponent(id)}/image/${n}`), // waitTurn long-polls until the turn ends (or waits for someone, with // until "waiting"), or timeout seconds pass. waitTurn: (c: Client, box: string, id: string, timeout: number, until: "end" | "waiting" = "end") => diff --git a/app/src/lib/history.ts b/app/src/lib/history.ts index 5d65ed0d6..8d5ef7368 100644 --- a/app/src/lib/history.ts +++ b/app/src/lib/history.ts @@ -52,6 +52,7 @@ export const historyApi = { helperTranscript: (c: Client, box: string, session: string, id: string, since: number) => c.box(box, "GET", `sessions/${s(session)}/subagents/${s(id)}/transcript?since=${since}`), helperTool: (c: Client, box: string, session: string, id: string, tool: string) => c.box(box, "GET", `sessions/${s(session)}/subagents/${s(id)}/tool/${s(tool)}`), + helperToolImage: (c: Client, box: string, session: string, id: string, tool: string, n: number) => c.boxBlob(box, `sessions/${s(session)}/subagents/${s(id)}/tool/${s(tool)}/image/${n}`), fork: (c: Client, box: string, session: string, req: { at?: string; text?: string; title?: string; open?: "tab" | "split" }) => c.box(box, "POST", `sessions/${s(session)}/fork`, req), rewind: (c: Client, box: string, session: string, req: { text: string; nth?: number; restore?: "conversation" | "both" | "code" }) => c.box<{ restored: "conversation" | "both" | "code"; text: string }>(box, "POST", `sessions/${s(session)}/rewind`, req), diff --git a/app/src/lib/mock-conversation.ts b/app/src/lib/mock-conversation.ts index b496102a2..8c5e1dd4f 100644 --- a/app/src/lib/mock-conversation.ts +++ b/app/src/lib/mock-conversation.ts @@ -206,7 +206,7 @@ function releaseChat(): TranscriptItem[] { const testReport = new URLSearchParams(location.search).get("scene") === "chat-tests"; return [ { kind: "user", id: "rq-u1", text: "Plan the release of the checkout fix." }, - { kind: "tools", id: "rq-t1", verb: "Read", done: true, items: [{ verb: "Read", target: "webhook.ts", file: true, id: "mock-read-1" }, { verb: "Read", target: "createOrder.ts", file: true, id: "mock-read-2" }] }, + { kind: "tools", id: "rq-t1", verb: "Read", done: true, items: [{ verb: "Read", target: "webhook.ts", file: true, id: "mock-read-1" }, { verb: "Read", target: "createOrder.ts", file: true, id: "mock-read-2" }, { verb: "Read", target: "checkout-result.png", file: true, id: "mock-read-shot" }] }, { kind: "text", id: "rq-x1", text: RELEASE_PLAN + picture }, ...(tables ? [{ kind: "text" as const, id: "rq-x2", text: TABLE_REPORT }] : []), ...(testReport ? [{ kind: "text" as const, id: "rq-x2", text: TEST_REPORT }] : []), @@ -396,6 +396,8 @@ const DETAILS: Record = { "mock-read-1": { id: "mock-read-1", name: "Read", file: "apps/web/lib/payments/webhook.ts", output: " 1\timport { createOrder } from \"../checkout/createOrder\";\n 2\timport { verify } from \"./signature\";\n 3\t\n 4\texport async function handleWebhook(req: Request) {\n 5\t const event = await verify(req);\n 6\t return createOrder(event.data);\n 7\t}" }, "mock-read-2": { id: "mock-read-2", name: "Read", file: "apps/web/lib/checkout/createOrder.ts", output: " 1\texport async function createOrder(data: OrderInput) {\n 2\t const charge = await payments.charge(data.amount);\n 3\t return db.order.create({ data: { ...data, chargeId: charge.id } });\n 4\t}" }, "mock-read-3": { id: "mock-read-3", name: "Read", file: "apps/web/lib/payments/payments.test.ts", output: " 1\tdescribe(\"webhook\", () => {\n 2\t test(\"creates an order\", async () => { … });\n 3\t});" }, + // A screenshot outside the worktree: its picture comes from the record. + "mock-read-shot": { id: "mock-read-shot", name: "Read", file: "/tmp/acme/checkout-result.png", output: "[image]", images: [{ n: 0, line: 0, mediaType: "image/png" }] }, "mock-search-1": { id: "mock-search-1", name: "Grep", pattern: "idempotencyKey", output: "apps/web/lib/payments/stripe.ts:41: idempotencyKey: event.id,\napps/web/lib/payments/types.ts:12: idempotencyKey?: string;" }, "mock-search-2": { id: "mock-search-2", name: "Grep", pattern: "retryWebhook", output: "No matches found" }, "mock-run-1": { id: "mock-run-1", name: "Bash", command: "pnpm test payments", output: " ✓ webhook › creates an order (12 ms)\n ✓ webhook › returns the same order for a repeated event (9 ms)\n ✓ webhook › stops after five retries (4 ms)\n\n Test Files 1 passed (1)\n Tests 3 passed (3)\n Duration 1.21s" }, @@ -531,6 +533,41 @@ const DETAILS: Record = { // The demo's change to a file, for its uncommitted diff to agree with it. export const mockToolDetailSync = (file: string): ToolDetail | undefined => Object.values(DETAILS).find((d) => d.file === file && d.hunks); +// The pictures the demo's calls returned: a checkout page, drawn. Each +// fetch is noted in window.__berthToolImages, so a test can tell one wasn't +// made before its call was opened. +export const mockToolImage = async (id: string, n: number): Promise => { + const asked = window as unknown as { __berthToolImages?: string[] }; + (asked.__berthToolImages ??= []).push(`${id}/${n}`); + await wait(150); + if (!DETAILS[id]?.images?.some((i) => i.n === n)) throw new Error("That picture isn't in the demo."); + const canvas = document.createElement("canvas"); + canvas.width = 640; + canvas.height = 400; + const g = canvas.getContext("2d")!; + g.fillStyle = "#f8fafc"; + g.fillRect(0, 0, 640, 400); + g.fillStyle = "#0f172a"; + g.fillRect(0, 0, 640, 56); + g.fillStyle = "#ffffff"; + g.font = "bold 22px sans-serif"; + g.fillText("acme shop", 24, 36); + g.fillStyle = "#16a34a"; + g.beginPath(); + g.arc(320, 170, 44, 0, Math.PI * 2); + g.fill(); + g.fillStyle = "#0f172a"; + g.font = "bold 26px sans-serif"; + g.textAlign = "center"; + g.fillText("Order confirmed", 320, 260); + g.font = "18px sans-serif"; + g.fillStyle = "#475569"; + g.fillText("Order #1042 · 3 items · $84.00", 320, 292); + const blob = await new Promise((done) => canvas.toBlob(done, "image/png")); + if (!blob) throw new Error("The demo couldn't draw its picture."); + return blob; +}; + export const mockToolDetail = async (id: string): Promise => { await wait(250); const d = DETAILS[id] ?? benchToolDetail(id); diff --git a/app/src/lib/tool-images.test.ts b/app/src/lib/tool-images.test.ts new file mode 100644 index 000000000..784fe8af8 --- /dev/null +++ b/app/src/lib/tool-images.test.ts @@ -0,0 +1,37 @@ +// node --experimental-strip-types --test src/lib/tool-images.test.ts (pnpm test) +import assert from "node:assert/strict"; +import { test } from "node:test"; + +import { outputParts } from "./tool-images.ts"; + +const png = (n: number, line: number) => ({ n, line, mediaType: "image/png" }); + +test("each served picture stands in place of its own [image] line", () => { + assert.deepEqual(outputParts("two shots\n[image]\nbetween\n[image]", [png(0, 1), png(1, 3)]), [ + { kind: "text", text: "two shots" }, + { kind: "image", image: png(0, 1) }, + { kind: "text", text: "between" }, + { kind: "image", image: png(1, 3) }, + ]); +}); + +test("text that only reads [image], and a picture the box doesn't serve, stay text", () => { + assert.deepEqual(outputParts("written:\n[image]\n[image]\n[image]", [png(2, 3)]), [ + { kind: "text", text: "written:\n[image]\n[image]" }, + { kind: "image", image: png(2, 3) }, + ]); +}); + +test("a picture cut out of a long output comes after the cut", () => { + assert.deepEqual(outputParts("head\n… 9 more lines …\ntail", [png(0, 1), png(1, 1)]), [ + { kind: "text", text: "head\n… 9 more lines …" }, + { kind: "image", image: png(0, 1) }, + { kind: "image", image: png(1, 1) }, + { kind: "text", text: "tail" }, + ]); +}); + +test("output without pictures is its text", () => { + assert.deepEqual(outputParts("ok", undefined), [{ kind: "text", text: "ok" }]); + assert.deepEqual(outputParts("", []), []); +}); diff --git a/app/src/lib/tool-images.ts b/app/src/lib/tool-images.ts new file mode 100644 index 000000000..e0131dc23 --- /dev/null +++ b/app/src/lib/tool-images.ts @@ -0,0 +1,27 @@ +import type { ToolImageRef } from "@/lib/transcript"; + +// A call's output in pieces: its text, and the pictures it returned where +// they stand. +export type OutputPart = { kind: "text"; text: string } | { kind: "image"; image: ToolImageRef }; + +// outputParts puts each picture at its line of the output: in place of its +// "[image]" line, or after the line where the output was cut. A picture the +// box doesn't serve stays its "[image]" line. +export function outputParts(output: string, images: ToolImageRef[] | undefined): OutputPart[] { + if (!images?.length) return output ? [{ kind: "text", text: output }] : []; + const parts: OutputPart[] = []; + let text: string[] = []; + const flush = () => { + if (text.length) parts.push({ kind: "text", text: text.join("\n") }); + text = []; + }; + output.split("\n").forEach((line, i) => { + const here = images.filter((img) => img.line === i); + if (!here.length || line !== "[image]") text.push(line); + if (!here.length) return; + flush(); + for (const image of here) parts.push({ kind: "image", image }); + }); + flush(); + return parts; +} diff --git a/app/src/lib/transcript.ts b/app/src/lib/transcript.ts index 0605957b0..2f67514b4 100644 --- a/app/src/lib/transcript.ts +++ b/app/src/lib/transcript.ts @@ -199,4 +199,16 @@ export interface ToolDetail { pending?: boolean; // A background shell whose output is still growing. live?: boolean; + // The pictures the call returned that the box serves, fetched one by one + // once the call is opened. Output keeps an "[image]" line for each. + images?: ToolImageRef[]; +} + +// ToolImageRef is a picture in a call's result: n counts the result's +// images, the ones not served too; line is where it stands in the output +// (its "[image]" line, or the "… more lines …" line where it was cut). +export interface ToolImageRef { + n: number; + line: number; + mediaType: string; } diff --git a/docs/changelog.mdx b/docs/changelog.mdx index cc4ef4e5a..3a95042db 100644 --- a/docs/changelog.mdx +++ b/docs/changelog.mdx @@ -10,6 +10,11 @@ upgrades with `berth upgrade BOX`, or by running the again. ## Unreleased +- **See the picture a step read.** Opening a step that read an image (a + screenshot, a .png) showed only `[image]`. It now shows the picture + itself, as the agent saw it, and a click opens it at full size. It comes + from the conversation's record, so it shows even when the file is + outside the worktree or gone since. - **Quiet projects are grayed out.** On the rail and in the two-column sidebar, a project with no agent or shell open in it is grayed out, so the ones you're using stand out. Switching to one of those opens New diff --git a/internal/box/api.go b/internal/box/api.go index 2efa67ec8..2ef2c1ba7 100644 --- a/internal/box/api.go +++ b/internal/box/api.go @@ -223,6 +223,7 @@ func (b *Box) Mount(s *wire.Server) { route("GET /v1/sessions/{name}/draft", b.draft) route("GET /v1/sessions/{name}/transcript", b.transcript) route("GET /v1/sessions/{name}/transcript/tool/{id}", b.toolDetail) + route("GET /v1/sessions/{name}/transcript/tool/{id}/image/{n}", b.toolImage) route("POST /v1/sessions/{name}/attachments", b.sessionAttachment) route("POST /v1/sessions/{name}/send", b.sendToSession) route("POST /v1/sessions/{name}/keys", b.sessionKeys) diff --git a/internal/box/history.go b/internal/box/history.go index 638b8583a..775b50817 100644 --- a/internal/box/history.go +++ b/internal/box/history.go @@ -24,6 +24,7 @@ func (b *Box) mountHistory(route func(string, func(http.ResponseWriter, *http.Re route("GET /v1/sessions/{name}/subagents", b.listHelpers) route("GET /v1/sessions/{name}/subagents/{id}/transcript", b.helperTranscript) route("GET /v1/sessions/{name}/subagents/{id}/tool/{tool}", b.helperTool) + route("GET /v1/sessions/{name}/subagents/{id}/tool/{tool}/image/{n}", b.helperToolImage) route("POST /v1/sessions/{name}/fork", b.forkSession) route("POST /v1/sessions/{name}/rewind", b.rewindSession) } @@ -144,6 +145,16 @@ func (b *Box) helperTool(w http.ResponseWriter, r *http.Request) error { return nil } +// helperToolImage answers GET …/subagents/{id}/tool/{tool}/image/{n}: a +// picture one of a helper's calls returned. +func (b *Box) helperToolImage(w http.ResponseWriter, r *http.Request) error { + _, hp, err := b.helperFile(r) + if err != nil { + return err + } + return serveToolImage(w, "claude", hp, r.PathValue("tool"), r.PathValue("n")) +} + // ForkRequest starts a new session that goes on from a point in this one. type ForkRequest struct { // At is the entry to go on from: a prompt's parent (the transcript diff --git a/internal/box/toolimage_test.go b/internal/box/toolimage_test.go new file mode 100644 index 000000000..52e5cc78c --- /dev/null +++ b/internal/box/toolimage_test.go @@ -0,0 +1,95 @@ +package box + +import ( + "bytes" + "context" + "encoding/base64" + "encoding/json" + "io" + "net/http" + "net/url" + "os" + "path/filepath" + "testing" + "time" + + "github.com/cosscom/shipyard/internal/transcript" +) + +// A picture a call returned is served from the transcript, by its place in +// the result: the file it was read from is gone, and the detail names the +// pictures without carrying them. +func TestAToolsImagesAreServedFromTheTranscript(t *testing.T) { + bin := t.TempDir() + os.WriteFile(filepath.Join(bin, "claude"), []byte("#!/bin/sh\nexec sleep 600\n"), 0o755) + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + account := t.TempDir() + t.Setenv("CLAUDE_CONFIG_DIR", account) + c, _ := servedBox(t) + call(t, c, "POST", "/v1/locations", "", map[string]string{"name": "shop", "path": gitRepo(t)}, nil) + var sess Session + if status := call(t, c, "POST", "/v1/sessions", "", SessionRequest{Location: "shop", Command: "claude"}, &sess); status != 200 { + t.Fatalf("session: %d", status) + } + png, jpeg := []byte("\x89PNG\r\n\x1a\nacme-shot"), []byte("\xff\xd8\xff\xe0acme-photo") + img := func(typ string, data []byte) map[string]any { + return map[string]any{"type": "image", "source": map[string]any{"type": "base64", "media_type": typ, "data": base64.StdEncoding.EncodeToString(data)}} + } + at := time.Now().UTC().Add(time.Second).Format(time.RFC3339Nano) + lines := []map[string]any{ + {"type": "assistant", "timestamp": at, "message": map[string]any{"role": "assistant", "content": []any{ + map[string]any{"type": "tool_use", "id": "toolu_shot", "name": "Read", "input": map[string]any{"file_path": "/tmp/acme-shot.png"}}, + }}}, + {"type": "user", "timestamp": at, "message": map[string]any{"role": "user", "content": []any{ + map[string]any{"type": "tool_result", "tool_use_id": "toolu_shot", "content": []any{img("image/png", png), img("image/jpeg", jpeg)}}, + }}}, + } + var rec bytes.Buffer + for _, l := range lines { + b, _ := json.Marshal(l) + rec.Write(append(b, '\n')) + } + proj := transcript.ClaudeDirIn(account, sess.Dir) + os.MkdirAll(proj, 0o755) + conv := filepath.Join(proj, "conv.jsonl") + os.WriteFile(conv, rec.Bytes(), 0o600) + // A helper's record holds the same call. + os.MkdirAll(transcript.HelpersDir(conv), 0o755) + os.WriteFile(filepath.Join(transcript.HelpersDir(conv), "agent-a1.jsonl"), rec.Bytes(), 0o600) + + base := "/v1/sessions/" + url.PathEscape(sess.Name) + var d transcript.ToolDetail + if status := call(t, c, "GET", base+"/transcript/tool/toolu_shot", "", nil, &d); status != 200 || len(d.Images) != 2 || d.Images[1].MediaType != "image/jpeg" { + t.Fatalf("detail: %d %+v", status, d) + } + get := func(path string) (int, string, []byte) { + t.Helper() + resp, err := c.Do(context.Background(), "GET", path, nil) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + b, _ := io.ReadAll(resp.Body) + return resp.StatusCode, resp.Header.Get("Content-Type"), b + } + for _, prefix := range []string{base + "/transcript/tool/toolu_shot", base + "/subagents/a1/tool/toolu_shot"} { + for i, want := range []struct { + typ string + data []byte + }{{"image/png", png}, {"image/jpeg", jpeg}} { + status, typ, body := get(prefix + "/image/" + string(rune('0'+i))) + if status != 200 || typ != want.typ || !bytes.Equal(body, want.data) { + t.Fatalf("%s image %d: %d %q %q", prefix, i, status, typ, body) + } + } + if status, _, _ := get(prefix + "/image/2"); status != http.StatusNotFound { + t.Fatalf("%s: a third image: %d", prefix, status) + } + if status, _, _ := get(prefix + "/image/x"); status != http.StatusNotFound { + t.Fatalf("%s: a bad index: %d", prefix, status) + } + } + if status, _, _ := get(base + "/transcript/tool/toolu_none/image/0"); status != http.StatusNotFound { + t.Fatalf("an unknown call: %d", status) + } +} diff --git a/internal/box/transcriptapi.go b/internal/box/transcriptapi.go index 4154525ae..8b2225605 100644 --- a/internal/box/transcriptapi.go +++ b/internal/box/transcriptapi.go @@ -127,3 +127,36 @@ func (b *Box) toolDetail(w http.ResponseWriter, r *http.Request) error { writeJSON(w, d) return nil } + +// toolImage answers GET /v1/sessions/{name}/transcript/tool/{id}/image/{n}: +// the n-th picture a call returned, as the transcript holds it (not the +// file it was read from, which may be gone or changed since). +func (b *Box) toolImage(w http.ResponseWriter, r *http.Request) error { + sess, all, err := b.Sessions.getWithAll(r.Context(), r.PathValue("name")) + if err != nil { + return err + } + source, path := b.driverFor(sess).Record(r.Context(), sess, all) + if path == "" { + return httpError{http.StatusNotFound, "this session's conversation can't be read"} + } + return serveToolImage(w, source, path, r.PathValue("id"), r.PathValue("n")) +} + +// serveToolImage sends a call's picture, decoded from the transcript at +// path. A call's pictures never change, so the app may keep them. +func serveToolImage(w http.ResponseWriter, source, path, id, n string) error { + i, err := strconv.Atoi(n) + if err != nil { + i = -1 + } + typ, data, err := transcript.Image(source, path, id, i) + if errors.Is(err, transcript.ErrNoImage) { + return httpError{http.StatusNotFound, "that step returned no such image"} + } + if err != nil { + return err + } + w.Header().Set("Cache-Control", "private, max-age=86400, immutable") + return writePicture(w, typ, data) +} diff --git a/internal/box/worktreefiles.go b/internal/box/worktreefiles.go index 711e94e53..b3a833016 100644 --- a/internal/box/worktreefiles.go +++ b/internal/box/worktreefiles.go @@ -317,12 +317,18 @@ func serveImage(w http.ResponseWriter, abs, rel string) error { if err != nil { return err } + w.Header().Set("ETag", `"`+etagOf(data)+`"`) + return writePicture(w, typ, data) +} + +// writePicture sends a picture's bytes as typ, and never as anything a +// browser would run. +func writePicture(w http.ResponseWriter, typ string, data []byte) error { h := w.Header() h.Set("Content-Type", typ) h.Set("X-Content-Type-Options", "nosniff") h.Set("Content-Security-Policy", "default-src 'none'; sandbox") - h.Set("ETag", `"`+etagOf(data)+`"`) - _, err = w.Write(data) + _, err := w.Write(data) return err } diff --git a/internal/transcript/detail.go b/internal/transcript/detail.go index c66552659..cb8fc04af 100644 --- a/internal/transcript/detail.go +++ b/internal/transcript/detail.go @@ -3,6 +3,7 @@ package transcript import ( "bufio" "bytes" + "encoding/base64" "encoding/json" "errors" "io" @@ -39,6 +40,27 @@ type ToolDetail struct { Pending bool `json:"pending,omitempty"` // Live is a background shell whose output is still growing. Live bool `json:"live,omitempty"` + // Images are the pictures the call returned that the app can show, + // fetched one by one (GET …/tool/{id}/image/{n}): never inline, so a + // detail stays small. Output keeps an "[image]" line for every one. + Images []ToolImage `json:"images,omitempty"` +} + +// ToolImage is one picture in a call's result. N counts the result's image +// blocks, the ones not served too; Line is where it stands in Output: its +// "[image]" line, or the "… more lines …" line when the middle was cut. +type ToolImage struct { + N int `json:"n"` + Line int `json:"line"` + MediaType string `json:"mediaType"` +} + +// imageTypes are the pictures a result's image may be served as. +var imageTypes = map[string]bool{"image/png": true, "image/jpeg": true, "image/gif": true, "image/webp": true} + +// servable says an image block's source is one the box serves. +func servable(sourceType, mediaType string) bool { + return sourceType == "base64" && imageTypes[mediaType] } // Hunk is one piece of a change, as a unified diff has it: where it starts @@ -55,6 +77,10 @@ type Hunk struct { // ErrNoTool is a call the transcript doesn't hold (any more). var ErrNoTool = errors.New("no such tool call in this conversation") +// ErrNoImage is a picture a call's result doesn't hold, or not as one of +// imageTypes. +var ErrNoImage = errors.New("no such image in that step") + const ( // detailWindow is how far back a call is looked for: the end of the // file, where the conversation on screen is. @@ -66,53 +92,93 @@ const ( // Detail finds one call in a transcript by its ID. func Detail(source, path, dir, id string) (ToolDetail, error) { - if id == "" || strings.ContainsAny(id, "\"\\\n") { + d := ToolDetail{ID: id} + found, answered := false, false + err := scanFor(path, id, func(line []byte) bool { + var call, result bool + if source == "codex" { + call, result = codexDetail(line, id, dir, &d) + } else if source == SourceCodexHost { + call, result = codexHostDetail(line, id, dir, &d) + } else if source == SourceClaudeHost { + if inner, ok := unwrapHost(line); ok { + call, result = claudeDetail(inner, id, dir, &d) + } + } else { + call, result = claudeDetail(line, id, dir, &d) + } + found = found || call + answered = answered || result + return false + }) + if err != nil { + return ToolDetail{}, err + } + if !found { return ToolDetail{}, ErrNoTool } + d.Pending = !answered + liveOutput(&d) + return d, nil +} + +// Image is the n-th picture in call id's result, decoded: its type and +// bytes. Only Claude Code's results hold pictures. +func Image(source, path, id string, n int) (string, []byte, error) { + if n < 0 || (source != "claude" && source != SourceClaudeHost) { + return "", nil, ErrNoImage + } + var typ string + var data []byte + err := scanFor(path, id, func(line []byte) bool { + if source == SourceClaudeHost { + var ok bool + if line, ok = unwrapHost(line); !ok { + return false + } + } + results := claudeResults(line, id) + if len(results) == 0 { + return false + } + typ, data = resultImage(results[0], n) + return true + }) + if errors.Is(err, ErrNoTool) || (err == nil && data == nil) { + return "", nil, ErrNoImage + } + return typ, data, err +} + +// scanFor calls fn with each line of the transcript's last detailWindow +// that mentions id, until fn says it has what it wanted. +func scanFor(path, id string, fn func(line []byte) (done bool)) error { + if id == "" || strings.ContainsAny(id, "\"\\\n") { + return ErrNoTool + } f, err := os.Open(path) if err != nil { - return ToolDetail{}, err + return err } defer f.Close() if st, err := f.Stat(); err == nil && st.Size() > detailWindow { if _, err := f.Seek(st.Size()-detailWindow, io.SeekStart); err != nil { - return ToolDetail{}, err + return err } } r := bufio.NewReaderSize(f, 256<<10) needle := []byte(`"` + id + `"`) - d := ToolDetail{ID: id} - found, answered := false, false var line []byte for { var err error line, err = readLineInto(r, line) - if len(line) > 0 && bytes.Contains(line, needle) { - var call, result bool - if source == "codex" { - call, result = codexDetail(line, id, dir, &d) - } else if source == SourceCodexHost { - call, result = codexHostDetail(line, id, dir, &d) - } else if source == SourceClaudeHost { - if inner, ok := unwrapHost(line); ok { - call, result = claudeDetail(inner, id, dir, &d) - } - } else { - call, result = claudeDetail(line, id, dir, &d) - } - found = found || call - answered = answered || result + if len(line) > 0 && bytes.Contains(line, needle) && fn(line) { + return nil } if err != nil { - break + return nil } } - if !found { - return ToolDetail{}, ErrNoTool - } - d.Pending = !answered - liveOutput(&d) - return d, nil } // readLine reads one line, skipping (not keeping) any longer than @@ -208,7 +274,15 @@ func claudeDetail(line []byte, id, dir string, d *ToolDetail) (call, result bool case b.Type == "tool_result" && b.ToolUseID == id: result = true d.Error = b.IsError - d.Output, d.Truncated = capOutput(resultFull(b.Content)) + full, images := resultParts(b.Content) + d.Output, d.Truncated = capOutput(full) + if d.Truncated { + full = plain(full) + for i := range images { + images[i].Line = cappedLine(full, images[i].Line) + } + } + d.Images = images if !b.IsError { d.Hunks = structuredHunks(l.ToolUseResult) } @@ -251,26 +325,104 @@ func structuredHunks(raw json.RawMessage) []Hunk { // resultFull is a tool result's whole text: a string, or all its text blocks. func resultFull(raw json.RawMessage) string { + s, _ := resultParts(raw) + return s +} + +// resultParts is a tool result's whole text, with an "[image]" line for +// each of its pictures, and the pictures the box serves, each at its line. +// Their bytes are skipped, not read. +func resultParts(raw json.RawMessage) (string, []ToolImage) { var s string if json.Unmarshal(raw, &s) == nil { - return s + return s, nil } var parts []struct { - Type string `json:"type"` - Text string `json:"text"` + Type string `json:"type"` + Text string `json:"text"` + Source struct { + Type string `json:"type"` + MediaType string `json:"media_type"` + } `json:"source"` } if json.Unmarshal(raw, &parts) != nil { - return "" + return "", nil } var out []string + var images []ToolImage + line, n := 0, 0 for _, p := range parts { - if p.Type == "text" { + switch p.Type { + case "text": out = append(out, p.Text) - } else if p.Type == "image" { + line += strings.Count(p.Text, "\n") + 1 + case "image": + if servable(p.Source.Type, p.Source.MediaType) { + images = append(images, ToolImage{N: n, Line: line, MediaType: p.Source.MediaType}) + } out = append(out, "[image]") + line, n = line+1, n+1 } } - return strings.Join(out, "\n") + return strings.Join(out, "\n"), images +} + +// resultImage is the n-th image block in a tool's result, decoded, if the +// box serves it. +func resultImage(raw json.RawMessage, n int) (string, []byte) { + if !isArray(raw) { + return "", nil + } + var parts []json.RawMessage + if json.Unmarshal(raw, &parts) != nil { + return "", nil + } + for _, p := range parts { + var head struct { + Type string `json:"type"` + } + if json.Unmarshal(p, &head) != nil || head.Type != "image" { + continue + } + if n--; n >= 0 { + continue + } + var img struct { + Source struct { + Type string `json:"type"` + MediaType string `json:"media_type"` + Data string `json:"data"` + } `json:"source"` + } + if json.Unmarshal(p, &img) != nil || !servable(img.Source.Type, img.Source.MediaType) { + return "", nil + } + data, err := base64.StdEncoding.DecodeString(img.Source.Data) + if err != nil { + return "", nil + } + return img.Source.MediaType, data + } + return "", nil +} + +// claudeResults are the contents of line's tool_result blocks for call id. +func claudeResults(line []byte, id string) []json.RawMessage { + var l struct { + Message struct { + Content []claudeBlock `json:"content"` + } `json:"message"` + } + if json.Unmarshal(line, &l) != nil { + return nil + } + var out []json.RawMessage + for _, b := range l.Message.Content { + if b.Type == "tool_result" && b.ToolUseID == id { + out = append(out, b.Content) + } + } + return out } func codexDetail(line []byte, id, dir string, d *ToolDetail) (call, result bool) { @@ -361,6 +513,21 @@ func plain(s string) string { return ansi.ReplaceAllString(s, "") } +// cappedLine is where line l of s, an output past outputCap, stands once +// capOutput has cut its middle: a line cut away (even in part) is the +// "… more lines …" line. +func cappedLine(s string, l int) int { + head := strings.Count(s[:outputCap/2], "\n") + if l < head { + return l + } + tail := strings.Count(s[:len(s)-outputCap/2], "\n") + if l > tail { + return head + 2 + l - tail + } + return head + 1 +} + // capOutput keeps a long output's head and tail, as the terminal does. func capOutput(s string) (string, bool) { s = plain(s) diff --git a/internal/transcript/detail_test.go b/internal/transcript/detail_test.go index 12fb8fdc9..a172b6915 100644 --- a/internal/transcript/detail_test.go +++ b/internal/transcript/detail_test.go @@ -1,6 +1,10 @@ package transcript import ( + "bytes" + "encoding/base64" + "encoding/json" + "errors" "path/filepath" "reflect" "strings" @@ -80,3 +84,91 @@ func TestDetailDropsHugeHunks(t *testing.T) { t.Fatalf("want none past the cap, got %d", len(d.Hunks)) } } + +// imageBlock is a picture in a tool's result, as Claude Code keeps it. +func imageBlock(mediaType string, data []byte) m { + return m{"type": "image", "source": m{"type": "base64", "media_type": mediaType, "data": base64.StdEncoding.EncodeToString(data)}} +} + +func TestDetailNamesTheImagesAResultHoldsButNotTheirBytes(t *testing.T) { + p := filepath.Join(t.TempDir(), "s.jsonl") + png, gif := []byte("\x89PNG\r\n\x1a\nacme-one"), []byte("GIF89a-acme-two") + write(t, p, + assistant(tool("r1", "Read", m{"file_path": "/tmp/acme-shot.png"})), + user([]m{{"type": "tool_result", "tool_use_id": "r1", "content": []m{ + {"type": "text", "text": "two screenshots"}, + imageBlock("image/png", png), + imageBlock("image/tiff", []byte("II*\x00acme")), + imageBlock("image/gif", gif), + }}}), + ) + d, err := Detail("claude", p, "/w/shop", "r1") + if err != nil { + t.Fatal(err) + } + if want := []ToolImage{{N: 0, Line: 1, MediaType: "image/png"}, {N: 2, Line: 3, MediaType: "image/gif"}}; !reflect.DeepEqual(d.Images, want) { + t.Fatalf("images %+v, want %+v", d.Images, want) + } + if d.Output != "two screenshots\n[image]\n[image]\n[image]" { + t.Fatalf("output %q", d.Output) + } + b, _ := json.Marshal(d) + if strings.Contains(string(b), base64.StdEncoding.EncodeToString(png)) || strings.Contains(string(b), "base64") { + t.Fatalf("the detail carries the image: %s", b) + } + + typ, data, err := Image("claude", p, "r1", 0) + if err != nil || typ != "image/png" || !bytes.Equal(data, png) { + t.Fatalf("image 0: %q %q %v", typ, data, err) + } + if typ, data, err = Image("claude", p, "r1", 2); err != nil || typ != "image/gif" || !bytes.Equal(data, gif) { + t.Fatalf("image 2: %q %q %v", typ, data, err) + } + for _, c := range []struct { + id string + n int + }{{"r1", 1}, {"r1", 3}, {"r1", -1}, {"nope", 0}, {"", 0}} { + if _, _, err := Image("claude", p, c.id, c.n); !errors.Is(err, ErrNoImage) { + t.Fatalf("%s/%d: want ErrNoImage, got %v", c.id, c.n, err) + } + } +} + +// A picture stands at its own line: never at a text block's line that only +// reads "[image]", and at the cut when the middle of a long output is left +// out. +func TestAToolsImagesStandAtTheirOwnLines(t *testing.T) { + p := filepath.Join(t.TempDir(), "s.jsonl") + shot := imageBlock("image/png", []byte("\x89PNG acme")) + long := strings.Repeat("acme log line\n", 4000) + write(t, p, + assistant(tool("m1", "mcp__acme__shot", m{"page": "checkout"})), + user([]m{{"type": "tool_result", "tool_use_id": "m1", "content": []m{ + {"type": "text", "text": "how it is written:\n[image]\nthen"}, + shot, + }}}), + assistant(tool("m2", "mcp__acme__shots", m{"page": "logs"})), + user([]m{{"type": "tool_result", "tool_use_id": "m2", "content": []m{ + shot, {"type": "text", "text": long}, shot, {"type": "text", "text": long}, shot, + }}}), + ) + d, _ := Detail("claude", p, "/w", "m1") + if want := []ToolImage{{N: 0, Line: 3, MediaType: "image/png"}}; !reflect.DeepEqual(d.Images, want) { + t.Fatalf("images %+v, want %+v", d.Images, want) + } + if lines := strings.Split(d.Output, "\n"); lines[1] != "[image]" || lines[3] != "[image]" { + t.Fatalf("output %q", d.Output) + } + + d, _ = Detail("claude", p, "/w", "m2") + if !d.Truncated || len(d.Images) != 3 { + t.Fatalf("truncated %v, images %+v", d.Truncated, d.Images) + } + lines := strings.Split(d.Output, "\n") + for i, want := range []string{"[image]", "… ", "[image]"} { + at := d.Images[i].Line + if at >= len(lines) || !strings.HasPrefix(lines[at], want) { + t.Fatalf("image %d at line %d of %d: %q", i, at, len(lines), lines[min(at, len(lines)-1)]) + } + } +}