Repository navigation
feat: drop priorKnowledge from describe_screen - #59
cryptotavares merged 3 commits into
Conversation
describe_screen no longer calls KnowledgeStore.generatePriorKnowledge, which read past step files from disk on every call, and its result no longer has a `priorKnowledge` field. The field is also gone from StepRecordObservation, createDefaultObservation and the observation compaction/redaction code. The rest of the knowledge store stays until it is removed separately. BREAKING CHANGE: `describe_screen` results (and `DescribeScreenResult`, `StepRecordObservation`) no longer contain `priorKnowledge`. `createDefaultObservation` no longer takes a fourth `priorKnowledge` argument.
82cbc27 to
fccafb3
Compare
seaona
left a comment
There was a problem hiding this comment.
hey there 👋 the removal code looks good.
Just a question to further develop, are there any plans to replace this with a different (more performant) approach in the future?
So from what I undestand, now we are transitioning towards a stateless tool. However, we could move towards agent memory or teh saved recipies/scripts we talked about before.
Is the path forward documented somewhere / decided?
I believe that a knowledge store or memory should not be tied to the tool itself. The tool should be a stateless tool. Agents knowledge/memory, should be tackled separately. In MetaMask scope, maybe up to date feature maps is all we need |
> **Summary:** Remove the knowledge store and its per-tool step recording. Keep session metadata as an explicit consumer write, separate from tool execution and observation collection. **Stack context:** mm CLI v2, Phase 1, **P1-04**, decision **D6**. Base: [#59](#59); next: [#61](#61). Review this delta against #59. ## Problem The shared HTTP executor awaits a knowledge-store write after each tool execution with a session ID. It also records execution failures. The store persists tool inputs, outcomes, observations, and artifact references under `test-artifacts/llm-knowledge/<sessionId>/steps/`. Four `knowledge_*` tools read that history. Decision **D6** removes this subsystem from the core. Session metadata and HTTP observations still need independent homes after the store disappears. ## Solution Delete the store, its recording hooks, retrieval tools, schemas, CLI commands, and public exports. Remove `knowledgeStore` from `ServerConfig` and `ToolContext`, so tool execution no longer depends on a history service. Keep two boundaries explicit: - The daemon collects observations and builds responses through the existing execution path. - Consumers call the exported `writeSessionMetadata(artifactsDir, metadata)` when they persist launch metadata. ```mermaid flowchart TD A[Tool request] --> B[Validate and run tool through queue] B --> C[Collect observations under existing policy] C --> D[Build timed HTTP response] E[Consumer launch metadata] --> F[Validate session ID in writeSessionMetadata] F -->|valid| G[Write session.json beneath artifactsDir] G --> I[Return absolute file path] F -->|invalid| H[Reject; no metadata write] classDef new fill:#fff3cd,stroke:#d39e00 classDef stop fill:#f8d7da,stroke:#c82333 class F,G,I new class H stop ``` Yellow boxes are new. Red boxes stop the metadata write. The metadata writer creates missing directories, overwrites `session.json`, and writes formatted JSON with a trailing newline. Relative artifact directories resolve against the process working directory. Invalid session IDs reject with `Invalid session id: "<id>"` before directory creation. Move `StepRecordObservation` and `createDefaultObservation` to `src/tools/types/observation.ts`. Move `SessionMetadata` to `src/tools/types/session.ts`, without `goal` or `flowTags`. Keep free-form `tags`. ## What changes | Condition or boundary | Before | After | |---|---|---| | Tool execution with a session ID | Awaits a step-record write after execution | No knowledge-store recording on success or execution failure | | Tool registry and schemas | Include four `knowledge_*` tools | Remove those tools; registry has 43 entries | | `mm knowledge-*` | Routes to history tools | Unknown command; exit 1 | | Launch `--goal` or `--flow-tags` | Adds knowledge metadata | Warns about the unknown flag; omits its field | | Persist session metadata | Store writes under `llm-knowledge/<id>/` | Consumer calls the writer for `<artifactsDir>/<id>/session.json` | | Observation collection and response policy | Collects beyond tools that return observations | Same policy; later stack work limits collection | The CLI uses its existing unknown-command message: ```text Error: unknown command 'knowledge-last'. Run 'mm --help' for usage. ``` Removed launch flags use the existing warning path, such as `Warning: unknown launch flag '--goal'`. The warning alone does not reject launch. The protected sections below list all breaking exports and consumer migration details. ## Decisions for reviewers | Decision | Reason | |---|---| | Remove the subsystem under **D6 / P1-04** | A compatibility wrapper would retain the history dependency and its public surface. Consumers migrate explicitly. | | Let consumers persist session metadata | The writer accepts their artifact directory without constructing a store or running inside each tool request. | | Keep observation behavior in this PR | Read-only and discovery observations still feed the next response's diff. Changing collection belongs to later stack work. | | Keep `StepRecordObservation` and `createDefaultObservation` names | Their implementations remain useful outside persistence. Moving them avoids an additional public rename. | ## Review guide Read the files in this order: | # | File | What to check | |---|---|---| | 1 | `src/server/create-server.ts`, `src/types/http.ts` | Recording disappears from success and failure paths; observations and timing remain. | | 2 | `src/server/session-metadata.ts`, `src/tools/types/session.ts` | Consumer-owned persistence, path validation, and retained metadata fields. | | 3 | `src/tools/types/observation.ts`, `src/index.ts`, `src/tools/types/index.ts` | Moved observation contracts and public exports. | | 4 | `src/tools/registry.ts`, `src/validation/schemas.ts`, `src/cli/mm.ts` | Removed tools, launch fields, commands, and help text. | | 5 | `src/server/session-metadata.test.ts`, `src/tools/types/observation.test.ts`, CLI/server/registry tests | Metadata writes and rejection cases, moved defaults, command removal, and retained observation policy. | | 6 | `README.md`, `SKILL.md` | Updated architecture, metadata path, command inventory, and launch options. | <details> <summary>Testing</summary> - Fresh check at remote head `2f10dd5ac47198f9259f24de061ee2b0bd0209a8`: `CI=true yarn vitest run --maxWorkers=2 --minWorkers=1 --coverage.enabled=false` passes: 46 files, 1,397 tests. - `yarn lint:eslint`, `yarn prettier README.md SKILL.md --check`, and `yarn tsc --noEmit --project tsconfig.build.json` pass. - Standalone `yarn tsc --noEmit --project tsconfig.test.json` fails on existing mobile-driver, network-mock, and screenshot test typing issues. Vitest reports no type errors, so its result does not replace this standalone check. - Metadata tests cover the destination path, missing directories, overwrite, and unsafe session IDs. CLI tests cover removed commands and launch flags. - The `Proof` and `Unit tests` sections below retain the original historical evidence. These fresh checks do not repeat live consumer validation. </details> ## Breaking changes - **Removed public exports:** - Store: `KnowledgeStore`, `KnowledgeStoreConfig`, `createKnowledgeStore`, `setKnowledgeStore`, `getKnowledgeStore`, `hasKnowledgeStore`, `knowledgeStore` - Tokenization: `tokenize`, `tokenizeIdentifier`, `expandWithSynonyms` - Tools and schemas: the `knowledge*Tool` functions, `knowledge*InputSchema`, `knowledgeScopeSchema`, `knowledgeFiltersSchema`, `Knowledge*Z` - Constants and types: `FLOW_TAGS`, `FlowTag`, `STEP_LABELS`, `StepLabel`, the `StepRecord*` types except `StepRecordObservation`, the `PriorKnowledge*`, `Knowledge*`, `RecipeStep` and `SessionSummary` types - Other: `ErrorCodes.MM_KNOWLEDGE_ERROR`, `extractTargetFromInput`, `extractScreenshotInfo`, `extractToolOutcome` - `ToolName` no longer includes `knowledge_*`. - `ServerConfig.knowledgeStore` and `ToolContext.knowledgeStore` are removed. - Launch input no longer accepts `goal`/`flowTags`, and `SessionMetadata` no longer has them. `mm launch --goal` now gives the unknown-flag warning. - `mm knowledge-*` commands are removed (unknown command, exit 1). - **Path change:** session metadata moves from `test-artifacts/llm-knowledge/<id>/session.json` to `<artifactsDir>/<id>/session.json` (`test-artifacts/<id>/session.json` with the consumers' `artifactsDir: 'test-artifacts'`). Consumers call `writeSessionMetadata()` themselves. ## Consumer impact | Consumer | Change needed | Done on branch | |---|---|---| | Extension `test/e2e/playwright/llm-workflow/` | Drop `KnowledgeStore`/`setKnowledgeStore` from `daemon.ts`; `metamask-provider.ts` calls `writeSessionMetadata(artifactsDir, …)`; no `goal`/`flowTags`; tests; README, AGENTS.md | `cryptotavares/mm-cli-v2-extension` 45c821d168a | | Mobile `tests/llm-workflow/` | Same (write stays non-blocking) | `cryptotavares/mm-cli-v2-mobile` 4e2661f209f | ## Proof Tested with a yalc build of this branch (`0.12.0+f2caa574`) in the Extension (e2e test build 13.56.0, headless) and Mobile (dev client 8.19.0, iOS sim + Android emulator), with the consumer commits above. | Check | Before (#59) | After | |---|---|---| | Files written under `test-artifacts/llm-knowledge/` during a session | one step file per tool call | **none** (ext 116 → 116 files, mobile 260 → 260) | | `session.json` per launch | `llm-knowledge/<id>/session.json` | `test-artifacts/<id>/session.json` (ext ×2, iOS, Android) | | `mm knowledge-last` | step list | `Error: unknown command 'knowledge-last'`, exit 1 | | `mm launch --goal x` | accepted | `Warning: unknown launch flag '--goal'`, launch ok | | Smoke (ext / iOS / Android) | pass | pass / pass / pass (Android `list-testids` 30 s timeout = known B11) | | `rg -i knowledge src` | many | only the tests asserting the commands are gone | | Coverage (stmts / branch / funcs / lines) | 96.38 / 91.7 / 94.63 / 96.62 | 96.92 / 93.38 / 94.84 / 97.1 | <details> <summary>Transcripts</summary> ``` $ yarn mm knowledge-last Error: unknown command 'knowledge-last' exit=1 $ yarn mm launch --context e2e --state default --goal x Warning: unknown launch flag '--goal' ... (launch ok, exit 0) $ find test-artifacts -newer <marker> -type f # Extension test-artifacts/mm-mvl3vmad-zd909k/session.json test-artifacts/mm-mvl3whgy-3sm2ye/session.json test-artifacts/screenshots/p1-04-ext-home-1792768182733.png $ cat test-artifacts/mm-mvl3vmad-zd909k/session.json { "schemaVersion": 1, "sessionId": "mm-mvl3vmad-zd909k", "createdAt": "...", "tags": [], "launch": { "stateMode": "default", ... } } ``` </details> <details> <summary>Unit tests</summary> ``` core: Test Files 46 passed (46) · Tests 1397 passed (1397) · Type Errors no errors All files | 96.92 | 93.38 | 94.84 | 97.1 Extension: yarn jest test/e2e/playwright/llm-workflow → 12 suites, 144 tests passed Mobile: yarn test:unit:llm-workflow → 18 suites, 231 tests passed ``` </details>
Stack context: mm CLI v2, Phase 1, P1-03. Base:
cryptotavares/mmcli-p1-02-device-mcp-0-5; review #58 first. #60 removes the remaining knowledge store.Problem
Each
describe_screencall gathers the current screen, then asks the knowledge store for historical context. That lookup reads past step files and appends related sessions, similar steps, and suggested actions to the result.This couples current-screen discovery to history size on disk. A separate change must isolate that lookup's cost before the remaining knowledge store is removed.
Solution
Remove the history-generation call at the shared discovery-tool boundary. Browser, iOS, and Android still collect current-screen data through their platform driver.
flowchart TD A[describe_screen] --> B[Collect state, test IDs, and accessibility tree] B --> C[Refresh accessibility refs and resolve active tab] C --> D[Capture screenshot when requested] D --> E[Return current-screen data without priorKnowledge] classDef new fill:#fff3cd,stroke:#d39e00 class E newYellow boxes are new.
Remove the same field from the result and step-observation types, observation diffing, and prior-knowledge redaction.
createDefaultObservationnow returns its three-field observation directly.What changes
describe_screenhistory accessgeneratePriorKnowledgeafter screen collectionpriorKnowledgefieldcreateDefaultObservationDecisions for reviewers
Review guide
Read the files in this order:
src/tools/discovery-tools.tsdescribeScreenToolreturns current-screen data without consulting the knowledge store.src/tools/types/tool-outputs.ts,src/tools/types/step-record.tspriorKnowledge.src/knowledge-store/knowledge-store.tscreateDefaultObservationaccepts three arguments; the remaining store stays available.src/server/observation-compaction.ts,src/knowledge-store/utils/redaction.tssrc/tools/test-utils/mock-factories.tsREADME.md,SKILL.mdTesting
Fresh runtime check at remote head
6a20fa8c3cc3af828aedd6612db36590667c4952: 7 files, 369 tests pass:yarn tsc --noEmit -p tsconfig.build.jsonpasses. ESLint passes for all 14 changed TypeScript files.yarn prettier README.md SKILL.md --checkpasses.yarn tsc --noEmit -p tsconfig.test.jsonfails with inheritedTS1309atsrc/platform/mobile-platform-driver.hermes.test.ts:25. That file is identical to the immediate parent.The
ProofandUnit testssections below retain the original historical evidence.Breaking changes
describe_screenresults (mm describe-screen) no longer containpriorKnowledge.DescribeScreenResultandStepRecordObservationtypes no longer havepriorKnowledge.createDefaultObservation(state, testIds, nodes)no longer takes a 4thpriorKnowledgeargument.Consumer impact
test/e2e/playwright/llm-workflow/rg priorKnowledge: 0 hits)tests/llm-workflow/rg priorKnowledge: 0 hits)Proof
Tested with yalc builds of the parent (
c87dbc3) and of this branch in the Extension (e2e test build 13.56.0, headless) and Mobile (dev client 8.19.0, iOS sim + Android emulator), on wallet home after unlock.priorKnowledgeindescribe-screen(ext / iOS / Android)describe-screenbytes p50, Extensiondescribe-screentoolMsp50, Extension (5 runs)describe-screentoolMsp50, Android (3 runs)generatePriorKnowledgealone, Mobile store (18 sessions, 31 MB)generatePriorKnowledgealone, Extension store (9 sessions, 1 MB)The iOS device bench was too noisy on this host (load average 11–19) to show the ~0.3 s gain:
observationMs, which this change does not touch, moved by the same amount astoolMs. The isolated timing of the removed call is the reliable number. Its cost grows with the history on disk.Transcripts
Unit tests