Skip to content

feat: drop priorKnowledge from describe_screen - #59

Merged
cryptotavares merged 3 commits into
cryptotavares/mmcli-p1-02-device-mcp-0-5from
cryptotavares/mmcli-p1-03-no-prior-knowledge
Oct 29, 2026
Merged

cryptotavares merged 3 commits into
cryptotavares/mmcli-p1-02-device-mcp-0-5from
cryptotavares/mmcli-p1-03-no-prior-knowledge

Conversation

@cryptotavares

@cryptotavares cryptotavares commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary: Remove priorKnowledge from describe_screen and its public observation types. Screen discovery no longer reads historical sessions to generate suggestions on each call.

Stack context: mm CLI v2, Phase 1, P1-03. Base: cryptotavares/mmcli-p1-02-device-mcp-0-5; review #58 first. #60 removes the remaining knowledge store.

Problem

Each describe_screen call gathers the current screen, then asks the knowledge store for historical context. That lookup reads past step files and appends related sessions, similar steps, and suggested actions to the result.

This couples current-screen discovery to history size on disk. A separate change must isolate that lookup's cost before the remaining knowledge store is removed.

Solution

Remove the history-generation call at the shared discovery-tool boundary. Browser, iOS, and Android still collect current-screen data through their platform driver.

flowchart TD
    A[describe_screen] --> B[Collect state, test IDs, and accessibility tree]
    B --> C[Refresh accessibility refs and resolve active tab]
    C --> D[Capture screenshot when requested]
    D --> E[Return current-screen data without priorKnowledge]

    classDef new fill:#fff3cd,stroke:#d39e00
    class E new
Loading

Yellow boxes are new.

Remove the same field from the result and step-observation types, observation diffing, and prior-knowledge redaction. createDefaultObservation now returns its three-field observation directly.

What changes

Condition or boundary Before After
describe_screen history access Calls generatePriorKnowledge after screen collection Returns without that call
Screen result and step-observation contracts Optional priorKnowledge field Field removed; current-screen data remains
createDefaultObservation Optional fourth argument Three arguments: state, test IDs, accessibility nodes
Observation diffing and step redaction Carry or scrub prior knowledge Remove prior-knowledge-specific handling
Remaining knowledge-store features Step recording, explicit knowledge tools, and history-generation API Retained for the next PR

Decisions for reviewers

Decision Reason
Split P1-03 from the full store removal Isolates the history lookup and payload cost. The protected proof below records the original measurements.
Remove the field from both public types Keeps discovery results and recorded observations aligned with the producer. This is a breaking contract change.
Retain the knowledge store in this PR Keeps step recording and explicit knowledge queries available until #60.

Review guide

Read the files in this order:

# File What to check
1 src/tools/discovery-tools.ts describeScreenTool returns current-screen data without consulting the knowledge store.
2 src/tools/types/tool-outputs.ts, src/tools/types/step-record.ts The two public contracts no longer expose priorKnowledge.
3 src/knowledge-store/knowledge-store.ts createDefaultObservation accepts three arguments; the remaining store stays available.
4 src/server/observation-compaction.ts, src/knowledge-store/utils/redaction.ts Remove only prior-knowledge propagation and redaction.
5 Changed tests and src/tools/test-utils/mock-factories.ts Discovery, driver delegation, recording, redaction, and compaction match the reduced contract.
6 README.md, SKILL.md Remove historical-suggestion claims from the discovery instructions.
Testing
  • Fresh runtime check at remote head 6a20fa8c3cc3af828aedd6612db36590667c4952: 7 files, 369 tests pass:

    CI=true yarn vitest run src/tools/discovery-tools.test.ts src/tools/driver-delegation.test.ts src/knowledge-store/knowledge-store.test.ts src/knowledge-store/utils/redaction.test.ts src/server/observation-compaction.test.ts src/server/create-server.test.ts src/tools/test-utils/mock-factories.test.ts --coverage.enabled=false --typecheck.enabled=false --maxWorkers=2 --minWorkers=1
  • yarn tsc --noEmit -p tsconfig.build.json passes. ESLint passes for all 14 changed TypeScript files.

  • yarn prettier README.md SKILL.md --check passes.

  • yarn tsc --noEmit -p tsconfig.test.json fails with inherited TS1309 at src/platform/mobile-platform-driver.hermes.test.ts:25. That file is identical to the immediate parent.

  • The Proof and Unit tests sections below retain the original historical evidence.

Breaking changes

  • describe_screen results (mm describe-screen) no longer contain priorKnowledge.
  • The DescribeScreenResult and StepRecordObservation types no longer have priorKnowledge.
  • createDefaultObservation(state, testIds, nodes) no longer takes a 4th priorKnowledge argument.

Consumer impact

Consumer Change needed Done on branch
Extension test/e2e/playwright/llm-workflow/ None (rg priorKnowledge: 0 hits) n/a
Mobile tests/llm-workflow/ None (rg priorKnowledge: 0 hits) n/a

Proof

Tested with yalc builds of the parent (c87dbc3) and of this branch in the Extension (e2e test build 13.56.0, headless) and Mobile (dev client 8.19.0, iOS sim + Android emulator), on wallet home after unlock.

Check Before After
priorKnowledge in describe-screen (ext / iOS / Android) present (1/1/1) absent (0/0/0)
Size of the removed block (pretty-printed, ext / iOS / Android) 3881 / 5060 / 4856 B 0
describe-screen bytes p50, Extension 14019 9816
describe-screen toolMs p50, Extension (5 runs) 712 614
describe-screen toolMs p50, Android (3 runs) 6916 4034
generatePriorKnowledge alone, Mobile store (18 sessions, 31 MB) p50 294 ms per call 0
generatePriorKnowledge alone, Extension store (9 sessions, 1 MB) p50 30 ms per call 0

The iOS device bench was too noisy on this host (load average 11–19) to show the ~0.3 s gain: observationMs, which this change does not touch, moved by the same amount as toolMs. The isolated timing of the removed call is the reliable number. Its cost grows with the history on disk.

Transcripts
describe-screen bench p50 (bytes / wall / toolMs / observationMs / totalMs)
ext      before 14019 / 2177 / 712 / 634 / 1348     after 9816 / 2182 / 614 / 642 / 1242
android  before 1010607 / 10957 / 6916 / 2928 / 9849   after 1014571 / 8457 / 4034 / 2845 / 7355
(Android a11y tree on wallet home varies by ~7 kB between runs, which hides the 4.9 kB removal)

$ grep -c priorKnowledge {ext,ios,android}/describe-screen.txt
0 0 0
Top-level keys after: state, activeTab (ext), testIds, a11y, screenshot, meta
Unit tests
 Test Files  49 passed (49)
      Tests  1636 passed (1636)
Type Errors  no errors
All files          |   96.38 |     91.7 |   94.63 |   96.62 |

@cryptotavares
cryptotavares added this pull request to stack #63 October 23, 2026 20:59
@cryptotavares
cryptotavares marked this pull request as ready for review October 26, 2026 23:42
@cryptotavares
cryptotavares requested a review from a team as a code owner October 26, 2026 23:42
describe_screen no longer calls KnowledgeStore.generatePriorKnowledge, which
read past step files from disk on every call, and its result no longer has a
`priorKnowledge` field. The field is also gone from StepRecordObservation,
createDefaultObservation and the observation compaction/redaction code.
The rest of the knowledge store stays until it is removed separately.

BREAKING CHANGE: `describe_screen` results (and `DescribeScreenResult`,
`StepRecordObservation`) no longer contain `priorKnowledge`.
`createDefaultObservation` no longer takes a fourth `priorKnowledge`
argument.
@cryptotavares
cryptotavares force-pushed the cryptotavares/mmcli-p1-03-no-prior-knowledge branch from 82cbc27 to fccafb3 Compare October 28, 2026 08:58
@cryptotavares cryptotavares changed the title feat!: drop priorKnowledge from describe_screen feat: drop priorKnowledge from describe_screen Oct 28, 2026

@seaona seaona left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey there 👋 the removal code looks good.
Just a question to further develop, are there any plans to replace this with a different (more performant) approach in the future?
So from what I undestand, now we are transitioning towards a stateless tool. However, we could move towards agent memory or teh saved recipies/scripts we talked about before.
Is the path forward documented somewhere / decided?

@cryptotavares

Copy link
Copy Markdown
Collaborator Author

hey there 👋 the removal code looks good. Just a question to further develop, are there any plans to replace this with a different (more performant) approach in the future? So from what I undestand, now we are transitioning towards a stateless tool. However, we could move towards agent memory or teh saved recipies/scripts we talked about before. Is the path forward documented somewhere / decided?

I believe that a knowledge store or memory should not be tied to the tool itself. The tool should be a stateless tool. Agents knowledge/memory, should be tackled separately. In MetaMask scope, maybe up to date feature maps is all we need

@cryptotavares
cryptotavares merged commit 16cc12c into main Oct 29, 2026
46 of 49 checks passed
@cryptotavares
cryptotavares deleted the cryptotavares/mmcli-p1-03-no-prior-knowledge branch October 29, 2026 13:52
cryptotavares added a commit that referenced this pull request Oct 29, 2026
> **Summary:** Remove the knowledge store and its per-tool step
recording. Keep session metadata as an explicit consumer write, separate
from tool execution and observation collection.

**Stack context:** mm CLI v2, Phase 1, **P1-04**, decision **D6**. Base:
[#59](#59); next:
[#61](#61). Review this
delta against #59.

## Problem

The shared HTTP executor awaits a knowledge-store write after each tool
execution with a session ID. It also records execution failures. The
store persists tool inputs, outcomes, observations, and artifact
references under `test-artifacts/llm-knowledge/<sessionId>/steps/`.

Four `knowledge_*` tools read that history. Decision **D6** removes this
subsystem from the core. Session metadata and HTTP observations still
need independent homes after the store disappears.

## Solution

Delete the store, its recording hooks, retrieval tools, schemas, CLI
commands, and public exports. Remove `knowledgeStore` from
`ServerConfig` and `ToolContext`, so tool execution no longer depends on
a history service.

Keep two boundaries explicit:

- The daemon collects observations and builds responses through the
existing execution path.
- Consumers call the exported `writeSessionMetadata(artifactsDir,
metadata)` when they persist launch metadata.

```mermaid
flowchart TD
    A[Tool request] --> B[Validate and run tool through queue]
    B --> C[Collect observations under existing policy]
    C --> D[Build timed HTTP response]
    E[Consumer launch metadata] --> F[Validate session ID in writeSessionMetadata]
    F -->|valid| G[Write session.json beneath artifactsDir]
    G --> I[Return absolute file path]
    F -->|invalid| H[Reject; no metadata write]

    classDef new fill:#fff3cd,stroke:#d39e00
    classDef stop fill:#f8d7da,stroke:#c82333
    class F,G,I new
    class H stop
```

Yellow boxes are new. Red boxes stop the metadata write.

The metadata writer creates missing directories, overwrites
`session.json`, and writes formatted JSON with a trailing newline.
Relative artifact directories resolve against the process working
directory. Invalid session IDs reject with `Invalid session id: "<id>"`
before directory creation.

Move `StepRecordObservation` and `createDefaultObservation` to
`src/tools/types/observation.ts`. Move `SessionMetadata` to
`src/tools/types/session.ts`, without `goal` or `flowTags`. Keep
free-form `tags`.

## What changes

| Condition or boundary | Before | After |
|---|---|---|
| Tool execution with a session ID | Awaits a step-record write after
execution | No knowledge-store recording on success or execution failure
|
| Tool registry and schemas | Include four `knowledge_*` tools | Remove
those tools; registry has 43 entries |
| `mm knowledge-*` | Routes to history tools | Unknown command; exit 1 |
| Launch `--goal` or `--flow-tags` | Adds knowledge metadata | Warns
about the unknown flag; omits its field |
| Persist session metadata | Store writes under `llm-knowledge/<id>/` |
Consumer calls the writer for `<artifactsDir>/<id>/session.json` |
| Observation collection and response policy | Collects beyond tools
that return observations | Same policy; later stack work limits
collection |

The CLI uses its existing unknown-command message:

```text
Error: unknown command 'knowledge-last'. Run 'mm --help' for usage.
```

Removed launch flags use the existing warning path, such as `Warning:
unknown launch flag '--goal'`. The warning alone does not reject launch.
The protected sections below list all breaking exports and consumer
migration details.

## Decisions for reviewers

| Decision | Reason |
|---|---|
| Remove the subsystem under **D6 / P1-04** | A compatibility wrapper
would retain the history dependency and its public surface. Consumers
migrate explicitly. |
| Let consumers persist session metadata | The writer accepts their
artifact directory without constructing a store or running inside each
tool request. |
| Keep observation behavior in this PR | Read-only and discovery
observations still feed the next response's diff. Changing collection
belongs to later stack work. |
| Keep `StepRecordObservation` and `createDefaultObservation` names |
Their implementations remain useful outside persistence. Moving them
avoids an additional public rename. |

## Review guide

Read the files in this order:

| # | File | What to check |
|---|---|---|
| 1 | `src/server/create-server.ts`, `src/types/http.ts` | Recording
disappears from success and failure paths; observations and timing
remain. |
| 2 | `src/server/session-metadata.ts`, `src/tools/types/session.ts` |
Consumer-owned persistence, path validation, and retained metadata
fields. |
| 3 | `src/tools/types/observation.ts`, `src/index.ts`,
`src/tools/types/index.ts` | Moved observation contracts and public
exports. |
| 4 | `src/tools/registry.ts`, `src/validation/schemas.ts`,
`src/cli/mm.ts` | Removed tools, launch fields, commands, and help text.
|
| 5 | `src/server/session-metadata.test.ts`,
`src/tools/types/observation.test.ts`, CLI/server/registry tests |
Metadata writes and rejection cases, moved defaults, command removal,
and retained observation policy. |
| 6 | `README.md`, `SKILL.md` | Updated architecture, metadata path,
command inventory, and launch options. |

<details>
<summary>Testing</summary>

- Fresh check at remote head `2f10dd5ac47198f9259f24de061ee2b0bd0209a8`:
`CI=true yarn vitest run --maxWorkers=2 --minWorkers=1
--coverage.enabled=false` passes: 46 files, 1,397 tests.
- `yarn lint:eslint`, `yarn prettier README.md SKILL.md --check`, and
`yarn tsc --noEmit --project tsconfig.build.json` pass.
- Standalone `yarn tsc --noEmit --project tsconfig.test.json` fails on
existing mobile-driver, network-mock, and screenshot test typing issues.
Vitest reports no type errors, so its result does not replace this
standalone check.
- Metadata tests cover the destination path, missing directories,
overwrite, and unsafe session IDs. CLI tests cover removed commands and
launch flags.
- The `Proof` and `Unit tests` sections below retain the original
historical evidence. These fresh checks do not repeat live consumer
validation.

</details>

## Breaking changes

- **Removed public exports:**
- Store: `KnowledgeStore`, `KnowledgeStoreConfig`,
`createKnowledgeStore`, `setKnowledgeStore`, `getKnowledgeStore`,
`hasKnowledgeStore`, `knowledgeStore`
  - Tokenization: `tokenize`, `tokenizeIdentifier`, `expandWithSynonyms`
- Tools and schemas: the `knowledge*Tool` functions,
`knowledge*InputSchema`, `knowledgeScopeSchema`,
`knowledgeFiltersSchema`, `Knowledge*Z`
- Constants and types: `FLOW_TAGS`, `FlowTag`, `STEP_LABELS`,
`StepLabel`, the `StepRecord*` types except `StepRecordObservation`, the
`PriorKnowledge*`, `Knowledge*`, `RecipeStep` and `SessionSummary` types
- Other: `ErrorCodes.MM_KNOWLEDGE_ERROR`, `extractTargetFromInput`,
`extractScreenshotInfo`, `extractToolOutcome`
- `ToolName` no longer includes `knowledge_*`.
- `ServerConfig.knowledgeStore` and `ToolContext.knowledgeStore` are
removed.
- Launch input no longer accepts `goal`/`flowTags`, and
`SessionMetadata` no longer has them. `mm launch --goal` now gives the
unknown-flag warning.
- `mm knowledge-*` commands are removed (unknown command, exit 1).
- **Path change:** session metadata moves from
`test-artifacts/llm-knowledge/<id>/session.json` to
`<artifactsDir>/<id>/session.json` (`test-artifacts/<id>/session.json`
with the consumers' `artifactsDir: 'test-artifacts'`). Consumers call
`writeSessionMetadata()` themselves.

## Consumer impact

| Consumer | Change needed | Done on branch |
|---|---|---|
| Extension `test/e2e/playwright/llm-workflow/` | Drop
`KnowledgeStore`/`setKnowledgeStore` from `daemon.ts`;
`metamask-provider.ts` calls `writeSessionMetadata(artifactsDir, …)`; no
`goal`/`flowTags`; tests; README, AGENTS.md |
`cryptotavares/mm-cli-v2-extension` 45c821d168a |
| Mobile `tests/llm-workflow/` | Same (write stays non-blocking) |
`cryptotavares/mm-cli-v2-mobile` 4e2661f209f |

## Proof

Tested with a yalc build of this branch (`0.12.0+f2caa574`) in the
Extension (e2e test build 13.56.0, headless) and Mobile (dev client
8.19.0, iOS sim + Android emulator), with the consumer commits above.

| Check | Before (#59) | After |
|---|---|---|
| Files written under `test-artifacts/llm-knowledge/` during a session |
one step file per tool call | **none** (ext 116 → 116 files, mobile 260
→ 260) |
| `session.json` per launch | `llm-knowledge/<id>/session.json` |
`test-artifacts/<id>/session.json` (ext ×2, iOS, Android) |
| `mm knowledge-last` | step list | `Error: unknown command
'knowledge-last'`, exit 1 |
| `mm launch --goal x` | accepted | `Warning: unknown launch flag
'--goal'`, launch ok |
| Smoke (ext / iOS / Android) | pass | pass / pass / pass (Android
`list-testids` 30 s timeout = known B11) |
| `rg -i knowledge src` | many | only the tests asserting the commands
are gone |
| Coverage (stmts / branch / funcs / lines) | 96.38 / 91.7 / 94.63 /
96.62 | 96.92 / 93.38 / 94.84 / 97.1 |

<details>
<summary>Transcripts</summary>

```
$ yarn mm knowledge-last
Error: unknown command 'knowledge-last'
exit=1

$ yarn mm launch --context e2e --state default --goal x
Warning: unknown launch flag '--goal'
... (launch ok, exit 0)

$ find test-artifacts -newer <marker> -type f      # Extension
test-artifacts/mm-mvl3vmad-zd909k/session.json
test-artifacts/mm-mvl3whgy-3sm2ye/session.json
test-artifacts/screenshots/p1-04-ext-home-1792768182733.png

$ cat test-artifacts/mm-mvl3vmad-zd909k/session.json
{ "schemaVersion": 1, "sessionId": "mm-mvl3vmad-zd909k", "createdAt": "...", "tags": [], "launch": { "stateMode": "default", ... } }
```

</details>

<details>
<summary>Unit tests</summary>

```
core:       Test Files 46 passed (46) · Tests 1397 passed (1397) · Type Errors no errors
            All files | 96.92 | 93.38 | 94.84 | 97.1
Extension:  yarn jest test/e2e/playwright/llm-workflow → 12 suites, 144 tests passed
Mobile:     yarn test:unit:llm-workflow → 18 suites, 231 tests passed
```

</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants