Skip to content

feat: remove the knowledge store - #60

Merged
cryptotavares merged 3 commits into
cryptotavares/mmcli-p1-03-no-prior-knowledgefrom
cryptotavares/mmcli-p1-04-remove-knowledge-store
Oct 29, 2026
Merged

cryptotavares merged 3 commits into
cryptotavares/mmcli-p1-03-no-prior-knowledgefrom
cryptotavares/mmcli-p1-04-remove-knowledge-store

Conversation

@cryptotavares

@cryptotavares cryptotavares commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary: Remove the knowledge store and its per-tool step recording. Keep session metadata as an explicit consumer write, separate from tool execution and observation collection.

Stack context: mm CLI v2, Phase 1, P1-04, decision D6. Base: #59; next: #61. Review this delta against #59.

Problem

The shared HTTP executor awaits a knowledge-store write after each tool execution with a session ID. It also records execution failures. The store persists tool inputs, outcomes, observations, and artifact references under test-artifacts/llm-knowledge/<sessionId>/steps/.

Four knowledge_* tools read that history. Decision D6 removes this subsystem from the core. Session metadata and HTTP observations still need independent homes after the store disappears.

Solution

Delete the store, its recording hooks, retrieval tools, schemas, CLI commands, and public exports. Remove knowledgeStore from ServerConfig and ToolContext, so tool execution no longer depends on a history service.

Keep two boundaries explicit:

  • The daemon collects observations and builds responses through the existing execution path.
  • Consumers call the exported writeSessionMetadata(artifactsDir, metadata) when they persist launch metadata.
flowchart TD
    A[Tool request] --> B[Validate and run tool through queue]
    B --> C[Collect observations under existing policy]
    C --> D[Build timed HTTP response]
    E[Consumer launch metadata] --> F[Validate session ID in writeSessionMetadata]
    F -->|valid| G[Write session.json beneath artifactsDir]
    G --> I[Return absolute file path]
    F -->|invalid| H[Reject; no metadata write]

    classDef new fill:#fff3cd,stroke:#d39e00
    classDef stop fill:#f8d7da,stroke:#c82333
    class F,G,I new
    class H stop
Loading

Yellow boxes are new. Red boxes stop the metadata write.

The metadata writer creates missing directories, overwrites session.json, and writes formatted JSON with a trailing newline. Relative artifact directories resolve against the process working directory. Invalid session IDs reject with Invalid session id: "<id>" before directory creation.

Move StepRecordObservation and createDefaultObservation to src/tools/types/observation.ts. Move SessionMetadata to src/tools/types/session.ts, without goal or flowTags. Keep free-form tags.

What changes

Condition or boundary Before After
Tool execution with a session ID Awaits a step-record write after execution No knowledge-store recording on success or execution failure
Tool registry and schemas Include four knowledge_* tools Remove those tools; registry has 43 entries
mm knowledge-* Routes to history tools Unknown command; exit 1
Launch --goal or --flow-tags Adds knowledge metadata Warns about the unknown flag; omits its field
Persist session metadata Store writes under llm-knowledge/<id>/ Consumer calls the writer for <artifactsDir>/<id>/session.json
Observation collection and response policy Collects beyond tools that return observations Same policy; later stack work limits collection

The CLI uses its existing unknown-command message:

Error: unknown command 'knowledge-last'. Run 'mm --help' for usage.

Removed launch flags use the existing warning path, such as Warning: unknown launch flag '--goal'. The warning alone does not reject launch. The protected sections below list all breaking exports and consumer migration details.

Decisions for reviewers

Decision Reason
Remove the subsystem under D6 / P1-04 A compatibility wrapper would retain the history dependency and its public surface. Consumers migrate explicitly.
Let consumers persist session metadata The writer accepts their artifact directory without constructing a store or running inside each tool request.
Keep observation behavior in this PR Read-only and discovery observations still feed the next response's diff. Changing collection belongs to later stack work.
Keep StepRecordObservation and createDefaultObservation names Their implementations remain useful outside persistence. Moving them avoids an additional public rename.

Review guide

Read the files in this order:

# File What to check
1 src/server/create-server.ts, src/types/http.ts Recording disappears from success and failure paths; observations and timing remain.
2 src/server/session-metadata.ts, src/tools/types/session.ts Consumer-owned persistence, path validation, and retained metadata fields.
3 src/tools/types/observation.ts, src/index.ts, src/tools/types/index.ts Moved observation contracts and public exports.
4 src/tools/registry.ts, src/validation/schemas.ts, src/cli/mm.ts Removed tools, launch fields, commands, and help text.
5 src/server/session-metadata.test.ts, src/tools/types/observation.test.ts, CLI/server/registry tests Metadata writes and rejection cases, moved defaults, command removal, and retained observation policy.
6 README.md, SKILL.md Updated architecture, metadata path, command inventory, and launch options.
Testing
  • Fresh check at remote head 2f10dd5ac47198f9259f24de061ee2b0bd0209a8: CI=true yarn vitest run --maxWorkers=2 --minWorkers=1 --coverage.enabled=false passes: 46 files, 1,397 tests.
  • yarn lint:eslint, yarn prettier README.md SKILL.md --check, and yarn tsc --noEmit --project tsconfig.build.json pass.
  • Standalone yarn tsc --noEmit --project tsconfig.test.json fails on existing mobile-driver, network-mock, and screenshot test typing issues. Vitest reports no type errors, so its result does not replace this standalone check.
  • Metadata tests cover the destination path, missing directories, overwrite, and unsafe session IDs. CLI tests cover removed commands and launch flags.
  • The Proof and Unit tests sections below retain the original historical evidence. These fresh checks do not repeat live consumer validation.

Breaking changes

  • Removed public exports:
    • Store: KnowledgeStore, KnowledgeStoreConfig, createKnowledgeStore, setKnowledgeStore, getKnowledgeStore, hasKnowledgeStore, knowledgeStore
    • Tokenization: tokenize, tokenizeIdentifier, expandWithSynonyms
    • Tools and schemas: the knowledge*Tool functions, knowledge*InputSchema, knowledgeScopeSchema, knowledgeFiltersSchema, Knowledge*Z
    • Constants and types: FLOW_TAGS, FlowTag, STEP_LABELS, StepLabel, the StepRecord* types except StepRecordObservation, the PriorKnowledge*, Knowledge*, RecipeStep and SessionSummary types
    • Other: ErrorCodes.MM_KNOWLEDGE_ERROR, extractTargetFromInput, extractScreenshotInfo, extractToolOutcome
  • ToolName no longer includes knowledge_*.
  • ServerConfig.knowledgeStore and ToolContext.knowledgeStore are removed.
  • Launch input no longer accepts goal/flowTags, and SessionMetadata no longer has them. mm launch --goal now gives the unknown-flag warning.
  • mm knowledge-* commands are removed (unknown command, exit 1).
  • Path change: session metadata moves from test-artifacts/llm-knowledge/<id>/session.json to <artifactsDir>/<id>/session.json (test-artifacts/<id>/session.json with the consumers' artifactsDir: 'test-artifacts'). Consumers call writeSessionMetadata() themselves.

Consumer impact

Consumer Change needed Done on branch
Extension test/e2e/playwright/llm-workflow/ Drop KnowledgeStore/setKnowledgeStore from daemon.ts; metamask-provider.ts calls writeSessionMetadata(artifactsDir, …); no goal/flowTags; tests; README, AGENTS.md cryptotavares/mm-cli-v2-extension 45c821d168a
Mobile tests/llm-workflow/ Same (write stays non-blocking) cryptotavares/mm-cli-v2-mobile 4e2661f209f

Proof

Tested with a yalc build of this branch (0.12.0+f2caa574) in the Extension (e2e test build 13.56.0, headless) and Mobile (dev client 8.19.0, iOS sim + Android emulator), with the consumer commits above.

Check Before (#59) After
Files written under test-artifacts/llm-knowledge/ during a session one step file per tool call none (ext 116 → 116 files, mobile 260 → 260)
session.json per launch llm-knowledge/<id>/session.json test-artifacts/<id>/session.json (ext ×2, iOS, Android)
mm knowledge-last step list Error: unknown command 'knowledge-last', exit 1
mm launch --goal x accepted Warning: unknown launch flag '--goal', launch ok
Smoke (ext / iOS / Android) pass pass / pass / pass (Android list-testids 30 s timeout = known B11)
rg -i knowledge src many only the tests asserting the commands are gone
Coverage (stmts / branch / funcs / lines) 96.38 / 91.7 / 94.63 / 96.62 96.92 / 93.38 / 94.84 / 97.1
Transcripts
$ yarn mm knowledge-last
Error: unknown command 'knowledge-last'
exit=1

$ yarn mm launch --context e2e --state default --goal x
Warning: unknown launch flag '--goal'
... (launch ok, exit 0)

$ find test-artifacts -newer <marker> -type f      # Extension
test-artifacts/mm-mvl3vmad-zd909k/session.json
test-artifacts/mm-mvl3whgy-3sm2ye/session.json
test-artifacts/screenshots/p1-04-ext-home-1792768182733.png

$ cat test-artifacts/mm-mvl3vmad-zd909k/session.json
{ "schemaVersion": 1, "sessionId": "mm-mvl3vmad-zd909k", "createdAt": "...", "tags": [], "launch": { "stateMode": "default", ... } }
Unit tests
core:       Test Files 46 passed (46) · Tests 1397 passed (1397) · Type Errors no errors
            All files | 96.92 | 93.38 | 94.84 | 97.1
Extension:  yarn jest test/e2e/playwright/llm-workflow → 12 suites, 144 tests passed
Mobile:     yarn test:unit:llm-workflow → 18 suites, 231 tests passed

Every tool call wrote a step record to test-artifacts/llm-knowledge/ and the
knowledge_* tools read them back. Nothing uses this at run time, so it goes:

- delete src/knowledge-store (store, tokenization, redaction),
  src/tools/knowledge.ts and the knowledge/step-record types
- remove the knowledge_last/search/summarize/sessions tools, schemas and
  `mm knowledge-*` commands, and ErrorCodes.MM_KNOWLEDGE_ERROR
- remove launch `goal`/`flowTags` (`--goal`, `--flow-tags`)
- remove `knowledgeStore` from ToolContext and ServerConfig, and step
  recording from the server

Kept and moved:
- StepRecordObservation and createDefaultObservation -> tools/types/observation.ts
- SessionMetadata -> tools/types/session.ts (without goal/flowTags)
- new writeSessionMetadata(artifactsDir, metadata) writes
  <artifactsDir>/<sessionId>/session.json

Observation collection is unchanged.

BREAKING CHANGE: Removed public exports: KnowledgeStore, KnowledgeStoreConfig,
createKnowledgeStore, setKnowledgeStore, getKnowledgeStore, hasKnowledgeStore,
knowledgeStore, tokenize, tokenizeIdentifier, expandWithSynonyms, the
knowledge*Tool functions, knowledge*InputSchema and Knowledge*Z, FLOW_TAGS,
STEP_LABELS, StepRecord* types (except StepRecordObservation), PriorKnowledge*
and Knowledge* types, ErrorCodes.MM_KNOWLEDGE_ERROR, extractTargetFromInput,
extractScreenshotInfo, extractToolOutcome. ServerConfig and ToolContext no
longer have knowledgeStore. Launch input no longer accepts goal/flowTags and
SessionMetadata no longer has them. `mm knowledge-*` commands are gone.
Session metadata moves from test-artifacts/llm-knowledge/<id>/session.json to
<artifactsDir>/<id>/session.json, written by writeSessionMetadata().
@cryptotavares
cryptotavares force-pushed the cryptotavares/mmcli-p1-04-remove-knowledge-store branch from a10dd91 to 05760ef Compare October 28, 2026 11:28
@cryptotavares cryptotavares changed the title feat!: remove the knowledge store feat: remove the knowledge store Oct 28, 2026

@seaona seaona left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, same question as previous comment: #59 (review)

@cryptotavares
cryptotavares merged commit 6dd0378 into main Oct 29, 2026
27 checks passed
@cryptotavares
cryptotavares deleted the cryptotavares/mmcli-p1-04-remove-knowledge-store branch October 29, 2026 13:52
cryptotavares added a commit that referenced this pull request Oct 29, 2026
> **Summary:** Remove `priorKnowledge` from `describe_screen` and its
public observation types. Screen discovery no longer reads historical
sessions to generate suggestions on each call.

**Stack context:** mm CLI v2, Phase 1, **P1-03**. Base:
`cryptotavares/mmcli-p1-02-device-mcp-0-5`; review
[#58](#58) first.
[#60](#60) removes the
remaining knowledge store.

## Problem

Each `describe_screen` call gathers the current screen, then asks the
knowledge store for historical context. That lookup reads past step
files and appends related sessions, similar steps, and suggested actions
to the result.

This couples current-screen discovery to history size on disk. A
separate change must isolate that lookup's cost before the remaining
knowledge store is removed.

## Solution

Remove the history-generation call at the shared discovery-tool
boundary. Browser, iOS, and Android still collect current-screen data
through their platform driver.

```mermaid
flowchart TD
    A[describe_screen] --> B[Collect state, test IDs, and accessibility tree]
    B --> C[Refresh accessibility refs and resolve active tab]
    C --> D[Capture screenshot when requested]
    D --> E[Return current-screen data without priorKnowledge]

    classDef new fill:#fff3cd,stroke:#d39e00
    class E new
```

Yellow boxes are new.

Remove the same field from the result and step-observation types,
observation diffing, and prior-knowledge redaction.
`createDefaultObservation` now returns its three-field observation
directly.

## What changes

| Condition or boundary | Before | After |
|---|---|---|
| `describe_screen` history access | Calls `generatePriorKnowledge`
after screen collection | Returns without that call |
| Screen result and step-observation contracts | Optional
`priorKnowledge` field | Field removed; current-screen data remains |
| `createDefaultObservation` | Optional fourth argument | Three
arguments: state, test IDs, accessibility nodes |
| Observation diffing and step redaction | Carry or scrub prior
knowledge | Remove prior-knowledge-specific handling |
| Remaining knowledge-store features | Step recording, explicit
knowledge tools, and history-generation API | Retained for the next PR |

## Decisions for reviewers

| Decision | Reason |
|---|---|
| Split **P1-03** from the full store removal | Isolates the history
lookup and payload cost. The protected proof below records the original
measurements. |
| Remove the field from both public types | Keeps discovery results and
recorded observations aligned with the producer. This is a breaking
contract change. |
| Retain the knowledge store in this PR | Keeps step recording and
explicit knowledge queries available until
[#60](#60). |

## Review guide

Read the files in this order:

| # | File | What to check |
|---|---|---|
| 1 | `src/tools/discovery-tools.ts` | `describeScreenTool` returns
current-screen data without consulting the knowledge store. |
| 2 | `src/tools/types/tool-outputs.ts`,
`src/tools/types/step-record.ts` | The two public contracts no longer
expose `priorKnowledge`. |
| 3 | `src/knowledge-store/knowledge-store.ts` |
`createDefaultObservation` accepts three arguments; the remaining store
stays available. |
| 4 | `src/server/observation-compaction.ts`,
`src/knowledge-store/utils/redaction.ts` | Remove only prior-knowledge
propagation and redaction. |
| 5 | Changed tests and `src/tools/test-utils/mock-factories.ts` |
Discovery, driver delegation, recording, redaction, and compaction match
the reduced contract. |
| 6 | `README.md`, `SKILL.md` | Remove historical-suggestion claims from
the discovery instructions. |

<details>
<summary>Testing</summary>

- Fresh runtime check at remote head
`6a20fa8c3cc3af828aedd6612db36590667c4952`: 7 files, 369 tests pass:

  ```sh
CI=true yarn vitest run src/tools/discovery-tools.test.ts
src/tools/driver-delegation.test.ts
src/knowledge-store/knowledge-store.test.ts
src/knowledge-store/utils/redaction.test.ts
src/server/observation-compaction.test.ts
src/server/create-server.test.ts
src/tools/test-utils/mock-factories.test.ts --coverage.enabled=false
--typecheck.enabled=false --maxWorkers=2 --minWorkers=1
  ```

- `yarn tsc --noEmit -p tsconfig.build.json` passes. ESLint passes for
all 14 changed TypeScript files.
- `yarn prettier README.md SKILL.md --check` passes.
- `yarn tsc --noEmit -p tsconfig.test.json` fails with inherited
`TS1309` at `src/platform/mobile-platform-driver.hermes.test.ts:25`.
That file is identical to the immediate parent.
- The `Proof` and `Unit tests` sections below retain the original
historical evidence.

</details>

## Breaking changes

- `describe_screen` results (`mm describe-screen`) no longer contain
`priorKnowledge`.
- The `DescribeScreenResult` and `StepRecordObservation` types no longer
have `priorKnowledge`.
- `createDefaultObservation(state, testIds, nodes)` no longer takes a
4th `priorKnowledge` argument.

## Consumer impact

| Consumer | Change needed | Done on branch |
|---|---|---|
| Extension `test/e2e/playwright/llm-workflow/` | None (`rg
priorKnowledge`: 0 hits) | n/a |
| Mobile `tests/llm-workflow/` | None (`rg priorKnowledge`: 0 hits) |
n/a |

## Proof

Tested with yalc builds of the parent (`c87dbc3`) and of this branch in
the Extension (e2e test build 13.56.0, headless) and Mobile (dev client
8.19.0, iOS sim + Android emulator), on wallet home after unlock.

| Check | Before | After |
|---|---|---|
| `priorKnowledge` in `describe-screen` (ext / iOS / Android) | present
(1/1/1) | absent (0/0/0) |
| Size of the removed block (pretty-printed, ext / iOS / Android) | 3881
/ 5060 / 4856 B | 0 |
| `describe-screen` bytes p50, Extension | 14019 | 9816 |
| `describe-screen` `toolMs` p50, Extension (5 runs) | 712 | 614 |
| `describe-screen` `toolMs` p50, Android (3 runs) | 6916 | 4034 |
| `generatePriorKnowledge` alone, Mobile store (18 sessions, 31 MB) |
p50 294 ms per call | 0 |
| `generatePriorKnowledge` alone, Extension store (9 sessions, 1 MB) |
p50 30 ms per call | 0 |

The iOS device bench was too noisy on this host (load average 11–19) to
show the ~0.3 s gain: `observationMs`, which this change does not touch,
moved by the same amount as `toolMs`. The isolated timing of the removed
call is the reliable number. Its cost grows with the history on disk.

<details>
<summary>Transcripts</summary>

```
describe-screen bench p50 (bytes / wall / toolMs / observationMs / totalMs)
ext      before 14019 / 2177 / 712 / 634 / 1348     after 9816 / 2182 / 614 / 642 / 1242
android  before 1010607 / 10957 / 6916 / 2928 / 9849   after 1014571 / 8457 / 4034 / 2845 / 7355
(Android a11y tree on wallet home varies by ~7 kB between runs, which hides the 4.9 kB removal)

$ grep -c priorKnowledge {ext,ios,android}/describe-screen.txt
0 0 0
Top-level keys after: state, activeTab (ext), testIds, a11y, screenshot, meta
```

</details>

<details>
<summary>Unit tests</summary>

```
 Test Files  49 passed (49)
      Tests  1636 passed (1636)
Type Errors  no errors
All files          |   96.38 |     91.7 |   94.63 |   96.62 |
```

</details>
cryptotavares added a commit that referenced this pull request Oct 29, 2026
> **Summary:** Skip automatic observation collection for read-only,
discovery, and raw CDP tools. Keep observations for mutating tools and
eligible batches, and reset the diff baseline after accessibility
snapshots.

**Stack context:** mm CLI v2, Phase 1, **P1-05**. Base:
[#60](#60)
(`cryptotavares/mmcli-p1-04-remove-knowledge-store`); followed by
[#62](#62). Review #60
first.

## Problem

The daemon collects observations after read-only and discovery calls,
although their responses omit those observations. That work also
refreshes refs and advances the diff baseline. The knowledge store that
consumed it is gone after #60.

Raw `cdp` calls use the `mutating` category. A small result, such as
`Runtime.evaluate 1+1`, therefore triggers and returns an additional
observation. On mobile, observation collection queries app state and
requests test IDs and an accessibility tree.

## Solution

Keep the policy at the shared HTTP executor. Tool handlers and platform
drivers retain their existing responsibilities.

```mermaid
flowchart TD
    A[Queued tool finishes] --> B{Category and batch policy}
    B -->|readonly, discovery, cdp, or batch none| C[Skip observation collection]
    B -->|mutating or eligible batch| D{Active session?}
    D -->|yes| E[Collect observation through platform driver]
    D -->|no| C
    E --> F[Apply response inclusion policy]
    F -->|included| G[Compact against last observation]
    F -->|omitted| H[Return result and timing]
    C --> H
    G --> H

    classDef new fill:#fff3cd,stroke:#d39e00
    class B,C,F new
```

Yellow boxes are new.

`shouldIncludeObservationsInResponse` calls `shouldCollectObservations`
first. A shared batch-policy helper supplies the default, so collection
and response inclusion use the same rule.

The registry gives `cdp` its own public category. CDP method execution
stays in the platform driver. A state-changing CDP call needs a
subsequent `mm describe-screen` to refresh refs. Per-call `observe`
opt-in remains a **P1-06** follow-up.

`accessibility_snapshot` now clears the diff baseline, like
`describe_screen`, `launch`, and `cleanup`. The next collected mutation
observation is full and compacted. Other read calls leave the baseline
unchanged.

## What changes

| Condition or boundary | Before | After |
|---|---|---|
| `readonly` or `discovery` call | Collects an observation but omits it
from the response | Skips automatic collection; `observationMs` is zero
|
| Raw `cdp` call | Uses `mutating`; collects and returns an observation
when available | Uses `cdp`; skips automatic collection and returns no
`observations` |
| Mutating call with an active session | Collects and returns an
observation when available | Same collection, compaction, and non-fatal
failure handling |
| `run_steps` policy | `all` by default; `none` skips collection;
`failures` filters the response | Same batch-level policy; `failures`
returns observations only for a tool error or failed summary |
| Ref refresh | Automatic observations refresh the session refMap |
Skipped observations do not refresh refs; tree-producing discovery
handlers still update refs themselves |
| `accessibility_snapshot` followed by a mutation | Retains an older
diff baseline | Clears the baseline; the next mutation returns a full
compact observation |

## Decisions for reviewers

| Decision | Reason |
|---|---|
| Stop collection at the executor boundary | Removes redundant browser
and mobile work without changing each read-tool handler. |
| Add `cdp` to public `ToolCategory` | Makes the default explicit. CDP
methods can change state, so callers must request fresh discovery
afterward. |
| Keep collection separate from response inclusion | A successful batch
with `failures` still collects an observation and advances the baseline,
but omits it from the response. |
| Clear the baseline after accessibility snapshots | The caller receives
a tree directly; the next mutation must not diff against an older
observation. |

## Review guide

Read the files in this order:

| # | File | What to check |
|---|---|---|
| 1 | `src/tools/registry.ts` | Public `cdp` category, tool assignment,
and unchanged safe default for unknown tool names. |
| 2 | `src/server/create-server.ts` | Collection predicate, shared batch
policy, response inclusion, and accessibility baseline reset. |
| 3 | `src/server/create-server.test.ts` | Per-category timing, skipped
collection, batch failures, and baseline behavior. |
| 4 | `src/tools/registry.test.ts` | `cdp` classification and
registry/category coverage. |
| 5 | `README.md`, `SKILL.md` | Observation rules, CDP ref-refresh
instructions, and baseline reset documentation. |

<details>
<summary>Testing</summary>

- Fresh check at remote head `74b9819d26e4e1ff63b5a9be675a4b3cefd94e68`:
`CI=true yarn vitest run src/server/create-server.test.ts
src/tools/registry.test.ts src/tools/batch.test.ts src/tools/cdp.test.ts
src/tools/discovery-tools.test.ts --coverage.enabled=false
--maxWorkers=2 --minWorkers=1` passes: 5 files, 181 tests.
- `yarn lint` passes. Production type checking passes with `yarn tsc
--project tsconfig.build.json --noEmit`.
- `yarn tsc --project tsconfig.test.json --noEmit` fails at the
unchanged remote head with nine diagnostics. They affect the new
timing-table callback and existing mobile-driver, network-mock, and
screenshot tests. Vitest reports no type errors, but its result does not
establish a clean explicit test-project type check.
- The `Proof` and `Unit tests` sections below retain the original
historical evidence. These fresh checks do not repeat live-device
validation.

</details>

## Breaking changes

- `readonly`, `discovery` and `cdp` tools no longer collect
observations. As a result:
- their responses no longer include `observations` (`cdp` used to
include them);
- they no longer refresh the session refMap through an observation
(discovery tools still set it themselves);
  - they no longer move the diff baseline for the next mutating call.
- A `cdp` call that changes state (for example `Runtime.evaluate` with a
`.click()`) must now be followed by `describe-screen` to get fresh refs.
- The public `ToolCategory` type gains `'cdp'`. Exhaustive switches on
it need a new case.

## Consumer impact

| Consumer | Change needed | Done on branch |
|---|---|---|
| Extension `test/e2e/playwright/llm-workflow/` | None. No code or test
reads `observations`. | n/a |
| Mobile `tests/llm-workflow/` | None. Same check. | n/a |

## Proof

Tested with a yalc build of this branch in the Extension (e2e build,
headless) and in Mobile (dev client on iOS sim and Android emulator), on
wallet home after unlock. Before = #60 tip (`0.12.0+f2caa574`), after =
`0.12.0+ce3fd9f0`. Ext was rechecked on the final commit
(`0.12.0+8a7091bf`). Bench: `scripts/bench.mjs`, 5 runs (Android 3),
p50.

| Check | Before | After |
|---|---|---|
| `cdp Runtime.evaluate 1+1` bytes, ext / iOS / Android | 7408 / 59228 /
1054409 (smoke) | **267 / 240 / 244** |
| `get-state` totalMs, ext / iOS / Android | 695 / 5736 / 6345 | **9 /
652 / 129** |
| `cdp` totalMs, ext / iOS / Android | 963 / 3334 / 6882 | **2 / 6 /
497** |
| `list-testids` totalMs, ext / iOS / Android | 1340 / 8901 / 13665 |
632 / 3835 / 11639 |
| `observationMs` for `get-state`, `get-text`, `hermes-targets`,
`list-testids`, `describe-screen`, `screenshot`,
`accessibility-snapshot`, `cdp` | 53–8001 | **0** (no `observations`
key), all platforms |
| `click --testid unlock-submit` observations, ext / iOS / Android | yes
| yes (71 / 4638 / 2355 ms) |

<details>
<summary>Notes from the proof run</summary>

- Host load was 7.7–18.4 during the runs, so mobile toolMs varies a lot
from run to run.
- Android discovery `toolMs` went up in the after run (`list-testids`
p50 5975 → 11639 ms). This change does not touch discovery tools. The
likely cause is variation in the ~1 MB Android a11y snapshot (the same
area as the existing `list-testids` timeout). Totals still fell because
observation time is now 0.
- On Android, some mutating calls (`wait-for`, `dismiss-keyboard`) still
hit the 8 s observation timeout and return no `observations`. This
already happened before the change (P1-04 build: iOS `click-unlock` 8003
ms).

</details>

<details>
<summary>Transcripts (ext, final commit)</summary>

```
$ yarn mm cdp Runtime.evaluate '{"expression":"1+1","returnByValue":true}'
{
  "method": "Runtime.evaluate",
  "result": {
    "result": {
      "type": "number",
      "value": 2,
      "description": "2"
    }
  },
  "meta": { "timing": { "queueMs": 0, "toolMs": 2, "observationMs": 0, "totalMs": 3 } }
}

click-unlock     obs=yes observationMs=16
describe-screen  obs=no  observationMs=0
get-state        obs=no  observationMs=0
list-testids     obs=no  observationMs=0
cdp-eval         obs=no  observationMs=0  bytes=266
screenshot       obs=no  observationMs=0
```

</details>

<details>
<summary>Unit tests</summary>

```
Test Files  46 passed (46)
     Tests  1406 passed (1406)
Type Errors no errors
All files | 96.92 | 93.37 | 94.85 | 97.1
yarn lint → exit 0
```

New or changed: per-category tests for `shouldCollectObservations` and
`shouldIncludeObservationsInResponse`; endpoint tests for
`observationMs` (0 for readonly, discovery and cdp; > 0 for mutating);
read tools don't move the diff baseline; `accessibility_snapshot` resets
it; the cdp category in the registry.

</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants