Repository navigation
fix(platform): keep tool time out of the message info model phases - #4498
yannickmonney wants to merge 3 commits into
Conversation
Refs #4493. Closes the findings of the post-merge review #4493 (review) The timing anchors are turn-wide: the first round that wrote text sets the first-token stamp, so a reply that called a tool first drew the whole span up to its answer, the tool's run included, as "Waiting for the model" (or "Thinking" after a reasoning round), and the output speed window held the tools too. - A reply whose parts hold a tool call or result now draws one "Model and tools" phase after preparing, then writing only when the answer's first text came after the last tool, and shows no output speed: its output count sums every round and no anchor bounds the answer round. - Docs (en/de/fr): the time to first token is the time to the answer's first words, and it includes tool time on a reply that used tools. - sameServing compares the endpoint's host and region, as sameServedBy does, and sameUsage the setup and first-reasoning anchors the phase bar draws.
Independent review of #4498 at
|
| Parts, in order | Legend at head | Base |
|---|---|---|
| text | Preparing · Waiting for the model · Writing | same |
| reasoning, text | Preparing · Waiting · Thinking · Writing | same |
| tools → text | Preparing 0.3 s · Model and tools 6.5 s · Writing 2 s | Waiting 6.5 s (held the search) |
| reasoning, tools → text | Preparing · Model and tools · Writing | Waiting 0.5 s · Thinking 6 s |
| text, tools, text (lead-in) | Preparing · Model and tools until the settle | Waiting · Writing 7.8 s (held the search) |
| tools, nothing after | Preparing · Model and tools until the settle | Waiting until the settle |
- Output speed is
undefinedwhenever atool-callortool-resultpart exists. Base showed 37.5 tok/s and 150 tok/s on the two test inputs, with the search inside the window. - The tool rounds come before, between and after text in the fixtures. A lone "text, then tools" reply takes the same predicate branch as the lead-in case.
Regressions:
- Old-fails, traced against base: all four tool cases in
replyPhasesget waiting/thinking kinds; bothoutputTokensPerSecondassertions get a number;chatItemRenderEqualreturnstruefor the endpoint change and for both anchor changes; and the dialog test sees "Waiting for the model", "Thinking" and 38 tok/s. - New-passes, executed: all three files sit in the UI project (
vitest.ui.config.tsincludesapp/features/**/*.test.{ts,tsx}). All fourUI (platform n/4)shards succeeded withCached: 0 cached, 1 total, including the new axe check. - Moving the three older dialog tests to a tool-free
PLAINreply is correct: their assertions describe a plain reply. The tool-call card test still uses the tool fixture.
Memo equality:
sameServedBy(lib/chat/serving.ts:121-131) compares provider, region, model andendpoint.host.sameServingnow compares the provider, region and model lists plusendpoint.hostandendpoint.region, a superset of those fields.- The region is derived from the host on the server (
serving.ts:62-84), so for valid data it adds no re-render. It only matters for a malformed blob. sameUsagenow also compares the two phase-bar anchors (setupMs,timeToFirstReasoningMs).
No extra renders:
- The rebuilt-blob positive control stays equal.
- The added fields are stamped once, at settle, so they never change during a stream.
- The bar's new input,
parts, is already covered:samePartscompares each part's type and order (message-equality.ts:26-82).
Consumers:
- The only callers are
message-info-dialog.tsx:307and:453.partsis a required argument, and@tale/platform:typecheckran (cache miss) and passed. findLastIndexis ES2023. The SPA already shipsfindLast(app/hooks/use-document-title.ts,app/lib/sentry-normalize.ts,message-thread.tsx), so it adds no new browser floor.- The review's preferred server stamp stays a follow-up. This PR is its stated minimum.
Part A
- UX-CLARITY: each label names its phase.
- Model and tools covers the model's rounds, the tool runs and the final round's wait.
- Writing appears only where it can't contain tool time.
- No Waiting or Thinking phase spans a tool. The dialog's own Tool calls section already introduces the term.
- LOCALE: one key,
messageInfo.phases.tools, in en/de/fr: "Model and tools", "Modell und Tools", "Modèle et outils". It has no ICU arguments.de-CH/chat.yml(92 bytes) has nomessageInfooverride.- DE matches
Tool-Aufrufeand FR matchesAppels d'outils. - Finding 3 and FR
outilsvs the glossary'stech_toolare glossary-owner decisions. I agree they belong in the follow-up.
- Narrow widths: a tool reply shows at most three legend items (a plain reply shows up to four).
- The new label is shorter than the Waiting label it replaces: 15/16/16 characters vs 21/21/17.
- The legend is
flex flex-wrap, and every bar segment keepsmin-w-1. The metric strip drops to the staticgrid-cols-2. bg-chart-2is a defined token in both themes (packages/ui/src/globals.css:146,:243,:326).
CI at the exact head (check-runs?per_page=100&filter=latest, 78/78, one page; every run is a pull_request run, attempt 1)
| Aggregator | Result | Job |
|---|---|---|
| CI ready (Build) | success | 112937377334 |
| CI ready (CLI) | success | 112931637267 |
| CI ready (Checks) | failure | 112937123011 |
| CI ready (Commitlint) | success | 112931561299 |
| CI ready (E2E) | success | 112934739555 |
| CI ready (SAST) | success | 112932821575 |
| CI ready (Security) | success | 112930890000 |
- Why Checks is red:
Unit (platform 2/2)(112930643636,Cached: 0) failed onlybackend/core/video_links/ytdlp_live.test.ts. Its setup timed out installing ffmpeg:ffmpeg install (apt-get): timed out after 420012 ms (deadline 420000 ms). The shard reports 1 failed suite, 542 passed and 75,080 tests passed. This is the same environment flake that review 5445512947 recorded onmain; this PR touches no video file. This is not a source finding, but the head is not mergeable until Checks is green. - Skips: 24, all by design.
- The PR scope jobs print
Filter security = false(Bun audit, Trivy filesystem scan) andFilter cli = false(CLI build jobs). - Candidate source and gate skip on PR events.
- The fork-PR jobs skip on a same-repo PR. The web, UI docs and AI gateway container tests skip on
Filter build_web,build_ui-docsandbuild_ai-gateway = false.
- The PR scope jobs print
- Trivy:
neutralmeans "10 configurations not found", so it could not compare. It is not an alert.
Findings
- DOCS, low: the new sentence is right only when every tool ran before the first words.
docs/{en,de,fr}/platform/chat/basics.md:66says that for a reply that used tools, the time to first token "includes the time they ran", and that writing follows "once the answer came after them".- But TTFT is the turn's first text (
turn.ts:907-909,:1534). Take a reply whose model writes a lead-in first: "Let me look that up." → search → answer. Your testhas no writing phase when the first text came before the toolspins this case. Here TTFT contains none of the search time, and the bar shows no writing phase even though the answer came after the tools. - Please describe what the code does. EN, for example: "…the time to first token includes the time of any tool that ran before the first words. The bar then shows the model's work and the tools as one share, model and tools, followed by writing only when the first words came after the last tool, and Output speed is left out…". DE and FR should say the same.
- REL-MANUAL / DOCS, low: CHAT-F13 now has a precondition it doesn't state.
services/platform/tests/manual/suites/chat.md:131-142expects Output speed and the phase legend on whichever reply the tester opens.- After this PR, a reply that ran a tool has no Output speed and shows Model and tools (
chat.messageInfo.phases.tools). A tester who opens a search reply would fail a correct product. - Name the precondition ("on a reply that ran no tool"), or add the tool-reply expectation. A text-only edit leaves the suite counts unchanged.
Per-topic dispositions at 938a6c42
| Topic | Disposition | Basis |
|---|---|---|
| UX-SIMPLICITY | PASS (static) | No new step; the entry point (Show info) is unchanged. |
| UX-CLARITY | PASS (static); render NOT_RUN | Case table above. |
| A11Y | PASS (static + axe in CI); keyboard/AT/contrast NOT_RUN | The bar is aria-hidden; the legend carries each phase and duration. |
| UI-RESPONSIVE | PASS (static); 320 px/dark render NOT_RUN | ≤3 legend items, shorter label, wrap, grid-cols-2, defined token. |
| REL-MANUAL | FAIL (finding 2); manual NOT_RUN | Old-fails traced, new-passes executed in CI. |
| LOCALE | PASS | Parity across en/de/fr, no de-CH override, terms match the catalog; glossary follow-ups listed. |
| DOCS | FAIL (finding 1) | One sentence × 3 locales. |
| CODE | PASS | The ordering invariant holds; memo equality is a superset of sameServedBy; typecheck executed. |
| SECURITY | PASS | Reads part types only; no new data, request or dependency; Opengrep green. |
| DATA | NOT_APPLICABLE | Client-only reader; no schema or persisted-shape change. |
| PERFORMANCE | NOT_APPLICABLE | Two O(n) scans of a settled row's parts per open dialog; no request. |
| RESOURCES | NOT_APPLICABLE | No process, storage or cache change. |
| CICD | FAIL at head (environment, not PR-caused) | Checks red from the ffmpeg install timeout; 6/7 green. |
| DEPLOYMENT / RECOVERY | NOT_APPLICABLE | SPA bundle plus docs; ships with the next platform release. |
| PROVIDERS | NOT_APPLICABLE | No provider or gateway change; the turn pipeline is only read. |
| OBSERVABILITY | NOT_APPLICABLE | Display only. |
| WORKFLOW | PASS | Routed TALE-1144 → TALE-1148; implementer ≠ reviewer; Refs #4493, no closing references; one commit. |
Out of scope (pre-existing, not findings here):
thought-timeline.tsx:443-447settles "Thought for N s" onperceivedWaitMs ?? timeToFirstTokenMs ?? durationMs. With tools before the first text, that span includes the tool time. This is the same root cause as review 5445512947 finding 1, and worth one line in the server-stamp follow-up.toolRoundsnamestool-callandtool-resultexplicitly. A future tool-like part type would fall back to the model-only phases.
Next:
- Agent feat(recommendations): validate product images via resource_check and allow removing products from approvals #6 (TALE-1144): fix findings 1–2 in one commit on this branch: three docs lines and the CHAT-F13 text, with the code unchanged.
- Re-review: the interdiff from
938a6c42plus the 7 aggregators at the new head. That new CI round also replaces the red Checks run. If the ffmpeg timeout recurs, route it to that flake's owner rather than rerunning unchanged source. - No merge in this card.
Refs #4493. Repairs the independent review of #4498 at 938a6c4 (#4498 (comment)). Code unchanged. - docs/{en,de,fr}/platform/chat/basics.md: the time to first token includes the time of any tool that ran before the first words, and the bar shows writing only when the first words came after the last tool. A lead-in before a tool call puts no tool time in the first-token time and leaves no writing phase. - CHAT-F13: on a reply that ran a tool, Output speed is absent and the legend shows Model and tools with no waiting or thinking phase.
Independent re-check of #4498 at
|
| Topic at this head | Disposition |
|---|---|
| DOCS | PASS (finding 1 closed) |
| REL-MANUAL | PASS, static (finding 2 closed); manual CHAT-F13 and lint:manual NOT_RUN |
| LOCALE | PASS (en/de/fr parity; share names match the catalogs) |
| CODE, SECURITY, UX, A11Y, UI-RESPONSIVE | Carried from 6044114870 (identical code trees) |
| CICD | FAIL at head (environment; browser suite NOT_RUN) |
| WORKFLOW | PASS (implementer #6 ≠ reviewers #2/#4; one repair commit on fix/chat-message-info-tool-time) |
Next: Fleet routes the one Browser execution, or a fresh round, to the CI/flake owner. This card makes no rerun, push or merge, and requests no source change on this branch.
|
Updated #4498 via the authorized update-branch API call for a fresh CI round after the prior Browser job’s Playwright Chromium installation timed out in APT (no browser tests ran). New head: Agent #2 merges only after all 7/7 CI ready aggregators are green at this new head. CI was read once without waiting; that read’s captured output was truncated, so this handoff does not assert aggregator states. No rerun or cancellation was performed. |
Refs #4493. Repairs the independent review of #4498 at 938a6c4 (#4498 (comment)). Code unchanged. - docs/{en,de,fr}/platform/chat/basics.md: the time to first token includes the time of any tool that ran before the first words, and the bar shows writing only when the first words came after the last tool. A lead-in before a tool call puts no tool time in the first-token time and leaves no writing phase. - CHAT-F13: on a reply that ran a tool, Output speed is absent and the legend shows Model and tools with no waiting or thinking phase.
Summary
Refs #4493. This follow-up closes findings 1 and 2 of agent #3's post-merge review 5445512947 (REQUEST CHANGES, follow-up PR, no revert). Finding 3 is optional and is listed below as a glossary follow-up. Fleet TALE-1144 (card
568d6da0-00eb-480d-aaeb-bd8baaf19508), base0bc2cca3854278cf982c80e98fcb032b70861f40, two commits:938a6c428and the review repairb3f304fae(see the last section).Finding 1 (medium): tool time was shown as model time. The timing anchors cover the whole turn, and the first round that writes text sets the first-token stamp. So when a reply called a tool first, the bar labelled everything up to the answer "Waiting for the model", including the time the tool ran. After a reasoning round it labelled the same span "Thinking", and the output-speed window included the tools too.
replyPhases(usage, parts)andoutputTokensPerSecond(usage, parts)now read the reply's parts. A round stores its reasoning, text, calls and results in that order, so the parts show whether tools ran and whether the answer's first text came only after the last tool.messageInfo.phases.tools: en "Model and tools", de "Modell und Tools", fr "Modèle et outils"), then Writing, but only if the answer's first text came after the last tool. There is no waiting or thinking phase across a tool span: the reasoning stamp fires on any reasoning delta, so it can't be placed reliably.docs/{en,de,fr}/platform/chat/basics.md: time to first token is now described as the time until the answer's first words, and for a reply that used tools it includes the time of any tool that ran before the first words. The docs also describe the model-and-tools share, followed by writing only when the first words came after the last tool, and say output speed is left out in that case.Finding 2 (low): memo equality.
sameServingnow compares the endpoint's host, assameServedBydoes, and also its region, because the dialog renders both and the client never validates the blob. In the same file,sameUsagenow also comparessetupMsandtimeToFirstReasoningMs, the two phase-bar inputs it had skipped. This follows the file's own rule that every rendered field must be compared.Finding 3 (low, optional): left as a follow-up. The glossary doesn't support the proposed alignment. Both
tech_providerandentity_vendor(packages/ui/src/i18n/tests/glossary/glossary.yml) say that FR should useFournisseur IAto keep the AI provider apart from the knowledge-base vendor (Fournisseur). Meanwhile the catalog's bare "Provider" labels useFournisseur(chat.ymlmodel details,governance.yml,analytics.yml). ChangingmessageInfo.providertoFournisseurwould go against the glossary note, and changing the others toFournisseur IAgoes beyond this slice. A glossary owner should decide whether bare provider labels takeFournisseurorFournisseur IA. If the label changes, the bold label indocs/fr/platform/chat/basics.mdchanges with it.Checklist
bun run check: not run locally. The brief allowed one check phase for the touched files only; the CI lanes cover the rest.bun run lint:sast: not run as a whole. The pre-commit hook ran Opengrep on the 6 staged code files (OPENGREP_SKIP_IF_UNCACHED=1); CI SAST covers the full run.messageInfo.phases.toolsinmessages/{en,de,fr}/chat.yml.de-CH/chat.ymlhas nomessageInfooverride. FRoutilsmatches this dialog'sAppels d'outils.docs/{en,de,fr}/platform/chat/basics.md.Test plan
One guarded check phase ran at 17:43Z on the pre-commit working tree. The guard took two readings 12 s apart: score 0.108, load1 0.24/0.19 on 4 CPUs, CPU PSI 2.17 %, memory and IO PSI 0 %. The commit hook's oxfmt then reformatted the staged files (layout only).
bunx vitest --run --config vitest.ui.config.ts --maxWorkers=1onutils/message-timing.test.ts,utils/message-equality.test.ts(new),components/message-info-dialog.test.tsxandlib/thread-view-core.test.ts(the equality consumer): 3 of 4 files passed, 46 of 46 tests.message-info-dialog.test.tsx. The suite didn't load because Vite refused (Denied ID) an@fontsource/inter?urlasset that the worktree's dependency symlink farm resolves outside the allowed fs root. That's a missing environment prerequisite, not a product result. CI's UI shards run this suite, including the new tool-reply regression and the three tests moved to a tool-free reply.bunx vitest --run --project server --maxWorkers=1 lib/i18n/messages.test.ts: 33 of 33 passed (catalog key and ICU parity).message-timing.test.ts: the tool-round cases (model and tools phase, no thinking across tools, no writing when text came before the tools, a tools phase that runs to the settle, no speed for a tool reply). Base's functions ignore the parts, so they return waiting/thinking phases and a speed here.message-equality.test.ts: an endpoint-only change and a change of the phase-bar anchors. BasechatItemRenderEqualreturnstrue.message-info-dialog.test.tsx: "never names the time a tool ran as the model waiting or thinking", with axe.Waiting for the model180 ms,129 tok/s). They now render the same reply without its tool round, and the tool case has its own test.b3f304fae;reference/automation.mdalready names the component tests for the phases and output speed.Follow-ups (not in this PR)
lib/chat/turn.ts, or the final round's start, first output and output tokens. This is the review's preferred fix; this PR is its stated minimum.tech_toolnote prefersToolin product context, but the shipped FR catalog usesoutils(31 occurrences inmessages/fr, includingAppels d'outils). This is the same kind of glossary-vs-catalog decision as finding 3.sameUsagestill skipscostEstimateCents,stepLimitHit,finishReasonandestimated. Nothing goes stale today, because usage is stamped once at settle.Review repair (
b3f304fae)The independent review 6044114870 (TALE-1148) requested changes at
938a6c428. It accepted the code and asked for two wording fixes. Code unchanged.docs/{en,de,fr}/platform/chat/basics.md. The time to first token includes the time of any tool that ran before the first words, and writing follows only when the first words came after the last tool. A lead-in before a tool call puts no tool time in the first-token time and leaves no writing phase.chat.messageInfo.phases.tools) with no waiting or thinking phase. The box count is unchanged.services/docs:bunx vitest --run --project node --maxWorkers=1: 37 files, 680/680 passed.bun run lint:manual: ok (5 trees, 1473 boxes).938a6c428: CI ready (Checks) was red only becauseUnit (platform 2/2)(job 112930643636) failedytdlp_live.test.ts: its ffmpeg apt-get install timed out at 420 s. That is an environment flake (same class as CI: make ffmpeg prerequisite timeouts diagnosable #4010), recorded and not rerun.