Repository navigation
feat: remove the knowledge store - #60
Merged
cryptotavares merged 3 commits intoOct 29, 2026
Merged
cryptotavares merged 3 commits into
cryptotavares merged 3 commits into
Conversation
Every tool call wrote a step record to test-artifacts/llm-knowledge/ and the knowledge_* tools read them back. Nothing uses this at run time, so it goes: - delete src/knowledge-store (store, tokenization, redaction), src/tools/knowledge.ts and the knowledge/step-record types - remove the knowledge_last/search/summarize/sessions tools, schemas and `mm knowledge-*` commands, and ErrorCodes.MM_KNOWLEDGE_ERROR - remove launch `goal`/`flowTags` (`--goal`, `--flow-tags`) - remove `knowledgeStore` from ToolContext and ServerConfig, and step recording from the server Kept and moved: - StepRecordObservation and createDefaultObservation -> tools/types/observation.ts - SessionMetadata -> tools/types/session.ts (without goal/flowTags) - new writeSessionMetadata(artifactsDir, metadata) writes <artifactsDir>/<sessionId>/session.json Observation collection is unchanged. BREAKING CHANGE: Removed public exports: KnowledgeStore, KnowledgeStoreConfig, createKnowledgeStore, setKnowledgeStore, getKnowledgeStore, hasKnowledgeStore, knowledgeStore, tokenize, tokenizeIdentifier, expandWithSynonyms, the knowledge*Tool functions, knowledge*InputSchema and Knowledge*Z, FLOW_TAGS, STEP_LABELS, StepRecord* types (except StepRecordObservation), PriorKnowledge* and Knowledge* types, ErrorCodes.MM_KNOWLEDGE_ERROR, extractTargetFromInput, extractScreenshotInfo, extractToolOutcome. ServerConfig and ToolContext no longer have knowledgeStore. Launch input no longer accepts goal/flowTags and SessionMetadata no longer has them. `mm knowledge-*` commands are gone. Session metadata moves from test-artifacts/llm-knowledge/<id>/session.json to <artifactsDir>/<id>/session.json, written by writeSessionMetadata().
cryptotavares
force-pushed
the
cryptotavares/mmcli-p1-04-remove-knowledge-store
branch
from
October 28, 2026 11:28
a10dd91 to
05760ef
Compare
itsyoboieltr
approved these changes
Oct 28, 2026
seaona
approved these changes
Oct 29, 2026
seaona
left a comment
Member
There was a problem hiding this comment.
looks good, same question as previous comment: #59 (review)
cryptotavares
deleted the
cryptotavares/mmcli-p1-04-remove-knowledge-store
branch
October 29, 2026 13:52
cryptotavares
added a commit
that referenced
this pull request
Oct 29, 2026
> **Summary:** Remove `priorKnowledge` from `describe_screen` and its public observation types. Screen discovery no longer reads historical sessions to generate suggestions on each call. **Stack context:** mm CLI v2, Phase 1, **P1-03**. Base: `cryptotavares/mmcli-p1-02-device-mcp-0-5`; review [#58](#58) first. [#60](#60) removes the remaining knowledge store. ## Problem Each `describe_screen` call gathers the current screen, then asks the knowledge store for historical context. That lookup reads past step files and appends related sessions, similar steps, and suggested actions to the result. This couples current-screen discovery to history size on disk. A separate change must isolate that lookup's cost before the remaining knowledge store is removed. ## Solution Remove the history-generation call at the shared discovery-tool boundary. Browser, iOS, and Android still collect current-screen data through their platform driver. ```mermaid flowchart TD A[describe_screen] --> B[Collect state, test IDs, and accessibility tree] B --> C[Refresh accessibility refs and resolve active tab] C --> D[Capture screenshot when requested] D --> E[Return current-screen data without priorKnowledge] classDef new fill:#fff3cd,stroke:#d39e00 class E new ``` Yellow boxes are new. Remove the same field from the result and step-observation types, observation diffing, and prior-knowledge redaction. `createDefaultObservation` now returns its three-field observation directly. ## What changes | Condition or boundary | Before | After | |---|---|---| | `describe_screen` history access | Calls `generatePriorKnowledge` after screen collection | Returns without that call | | Screen result and step-observation contracts | Optional `priorKnowledge` field | Field removed; current-screen data remains | | `createDefaultObservation` | Optional fourth argument | Three arguments: state, test IDs, accessibility nodes | | Observation diffing and step redaction | Carry or scrub prior knowledge | Remove prior-knowledge-specific handling | | Remaining knowledge-store features | Step recording, explicit knowledge tools, and history-generation API | Retained for the next PR | ## Decisions for reviewers | Decision | Reason | |---|---| | Split **P1-03** from the full store removal | Isolates the history lookup and payload cost. The protected proof below records the original measurements. | | Remove the field from both public types | Keeps discovery results and recorded observations aligned with the producer. This is a breaking contract change. | | Retain the knowledge store in this PR | Keeps step recording and explicit knowledge queries available until [#60](#60). | ## Review guide Read the files in this order: | # | File | What to check | |---|---|---| | 1 | `src/tools/discovery-tools.ts` | `describeScreenTool` returns current-screen data without consulting the knowledge store. | | 2 | `src/tools/types/tool-outputs.ts`, `src/tools/types/step-record.ts` | The two public contracts no longer expose `priorKnowledge`. | | 3 | `src/knowledge-store/knowledge-store.ts` | `createDefaultObservation` accepts three arguments; the remaining store stays available. | | 4 | `src/server/observation-compaction.ts`, `src/knowledge-store/utils/redaction.ts` | Remove only prior-knowledge propagation and redaction. | | 5 | Changed tests and `src/tools/test-utils/mock-factories.ts` | Discovery, driver delegation, recording, redaction, and compaction match the reduced contract. | | 6 | `README.md`, `SKILL.md` | Remove historical-suggestion claims from the discovery instructions. | <details> <summary>Testing</summary> - Fresh runtime check at remote head `6a20fa8c3cc3af828aedd6612db36590667c4952`: 7 files, 369 tests pass: ```sh CI=true yarn vitest run src/tools/discovery-tools.test.ts src/tools/driver-delegation.test.ts src/knowledge-store/knowledge-store.test.ts src/knowledge-store/utils/redaction.test.ts src/server/observation-compaction.test.ts src/server/create-server.test.ts src/tools/test-utils/mock-factories.test.ts --coverage.enabled=false --typecheck.enabled=false --maxWorkers=2 --minWorkers=1 ``` - `yarn tsc --noEmit -p tsconfig.build.json` passes. ESLint passes for all 14 changed TypeScript files. - `yarn prettier README.md SKILL.md --check` passes. - `yarn tsc --noEmit -p tsconfig.test.json` fails with inherited `TS1309` at `src/platform/mobile-platform-driver.hermes.test.ts:25`. That file is identical to the immediate parent. - The `Proof` and `Unit tests` sections below retain the original historical evidence. </details> ## Breaking changes - `describe_screen` results (`mm describe-screen`) no longer contain `priorKnowledge`. - The `DescribeScreenResult` and `StepRecordObservation` types no longer have `priorKnowledge`. - `createDefaultObservation(state, testIds, nodes)` no longer takes a 4th `priorKnowledge` argument. ## Consumer impact | Consumer | Change needed | Done on branch | |---|---|---| | Extension `test/e2e/playwright/llm-workflow/` | None (`rg priorKnowledge`: 0 hits) | n/a | | Mobile `tests/llm-workflow/` | None (`rg priorKnowledge`: 0 hits) | n/a | ## Proof Tested with yalc builds of the parent (`c87dbc3`) and of this branch in the Extension (e2e test build 13.56.0, headless) and Mobile (dev client 8.19.0, iOS sim + Android emulator), on wallet home after unlock. | Check | Before | After | |---|---|---| | `priorKnowledge` in `describe-screen` (ext / iOS / Android) | present (1/1/1) | absent (0/0/0) | | Size of the removed block (pretty-printed, ext / iOS / Android) | 3881 / 5060 / 4856 B | 0 | | `describe-screen` bytes p50, Extension | 14019 | 9816 | | `describe-screen` `toolMs` p50, Extension (5 runs) | 712 | 614 | | `describe-screen` `toolMs` p50, Android (3 runs) | 6916 | 4034 | | `generatePriorKnowledge` alone, Mobile store (18 sessions, 31 MB) | p50 294 ms per call | 0 | | `generatePriorKnowledge` alone, Extension store (9 sessions, 1 MB) | p50 30 ms per call | 0 | The iOS device bench was too noisy on this host (load average 11–19) to show the ~0.3 s gain: `observationMs`, which this change does not touch, moved by the same amount as `toolMs`. The isolated timing of the removed call is the reliable number. Its cost grows with the history on disk. <details> <summary>Transcripts</summary> ``` describe-screen bench p50 (bytes / wall / toolMs / observationMs / totalMs) ext before 14019 / 2177 / 712 / 634 / 1348 after 9816 / 2182 / 614 / 642 / 1242 android before 1010607 / 10957 / 6916 / 2928 / 9849 after 1014571 / 8457 / 4034 / 2845 / 7355 (Android a11y tree on wallet home varies by ~7 kB between runs, which hides the 4.9 kB removal) $ grep -c priorKnowledge {ext,ios,android}/describe-screen.txt 0 0 0 Top-level keys after: state, activeTab (ext), testIds, a11y, screenshot, meta ``` </details> <details> <summary>Unit tests</summary> ``` Test Files 49 passed (49) Tests 1636 passed (1636) Type Errors no errors All files | 96.38 | 91.7 | 94.63 | 96.62 | ``` </details>
cryptotavares
added a commit
that referenced
this pull request
Oct 29, 2026
> **Summary:** Skip automatic observation collection for read-only, discovery, and raw CDP tools. Keep observations for mutating tools and eligible batches, and reset the diff baseline after accessibility snapshots. **Stack context:** mm CLI v2, Phase 1, **P1-05**. Base: [#60](#60) (`cryptotavares/mmcli-p1-04-remove-knowledge-store`); followed by [#62](#62). Review #60 first. ## Problem The daemon collects observations after read-only and discovery calls, although their responses omit those observations. That work also refreshes refs and advances the diff baseline. The knowledge store that consumed it is gone after #60. Raw `cdp` calls use the `mutating` category. A small result, such as `Runtime.evaluate 1+1`, therefore triggers and returns an additional observation. On mobile, observation collection queries app state and requests test IDs and an accessibility tree. ## Solution Keep the policy at the shared HTTP executor. Tool handlers and platform drivers retain their existing responsibilities. ```mermaid flowchart TD A[Queued tool finishes] --> B{Category and batch policy} B -->|readonly, discovery, cdp, or batch none| C[Skip observation collection] B -->|mutating or eligible batch| D{Active session?} D -->|yes| E[Collect observation through platform driver] D -->|no| C E --> F[Apply response inclusion policy] F -->|included| G[Compact against last observation] F -->|omitted| H[Return result and timing] C --> H G --> H classDef new fill:#fff3cd,stroke:#d39e00 class B,C,F new ``` Yellow boxes are new. `shouldIncludeObservationsInResponse` calls `shouldCollectObservations` first. A shared batch-policy helper supplies the default, so collection and response inclusion use the same rule. The registry gives `cdp` its own public category. CDP method execution stays in the platform driver. A state-changing CDP call needs a subsequent `mm describe-screen` to refresh refs. Per-call `observe` opt-in remains a **P1-06** follow-up. `accessibility_snapshot` now clears the diff baseline, like `describe_screen`, `launch`, and `cleanup`. The next collected mutation observation is full and compacted. Other read calls leave the baseline unchanged. ## What changes | Condition or boundary | Before | After | |---|---|---| | `readonly` or `discovery` call | Collects an observation but omits it from the response | Skips automatic collection; `observationMs` is zero | | Raw `cdp` call | Uses `mutating`; collects and returns an observation when available | Uses `cdp`; skips automatic collection and returns no `observations` | | Mutating call with an active session | Collects and returns an observation when available | Same collection, compaction, and non-fatal failure handling | | `run_steps` policy | `all` by default; `none` skips collection; `failures` filters the response | Same batch-level policy; `failures` returns observations only for a tool error or failed summary | | Ref refresh | Automatic observations refresh the session refMap | Skipped observations do not refresh refs; tree-producing discovery handlers still update refs themselves | | `accessibility_snapshot` followed by a mutation | Retains an older diff baseline | Clears the baseline; the next mutation returns a full compact observation | ## Decisions for reviewers | Decision | Reason | |---|---| | Stop collection at the executor boundary | Removes redundant browser and mobile work without changing each read-tool handler. | | Add `cdp` to public `ToolCategory` | Makes the default explicit. CDP methods can change state, so callers must request fresh discovery afterward. | | Keep collection separate from response inclusion | A successful batch with `failures` still collects an observation and advances the baseline, but omits it from the response. | | Clear the baseline after accessibility snapshots | The caller receives a tree directly; the next mutation must not diff against an older observation. | ## Review guide Read the files in this order: | # | File | What to check | |---|---|---| | 1 | `src/tools/registry.ts` | Public `cdp` category, tool assignment, and unchanged safe default for unknown tool names. | | 2 | `src/server/create-server.ts` | Collection predicate, shared batch policy, response inclusion, and accessibility baseline reset. | | 3 | `src/server/create-server.test.ts` | Per-category timing, skipped collection, batch failures, and baseline behavior. | | 4 | `src/tools/registry.test.ts` | `cdp` classification and registry/category coverage. | | 5 | `README.md`, `SKILL.md` | Observation rules, CDP ref-refresh instructions, and baseline reset documentation. | <details> <summary>Testing</summary> - Fresh check at remote head `74b9819d26e4e1ff63b5a9be675a4b3cefd94e68`: `CI=true yarn vitest run src/server/create-server.test.ts src/tools/registry.test.ts src/tools/batch.test.ts src/tools/cdp.test.ts src/tools/discovery-tools.test.ts --coverage.enabled=false --maxWorkers=2 --minWorkers=1` passes: 5 files, 181 tests. - `yarn lint` passes. Production type checking passes with `yarn tsc --project tsconfig.build.json --noEmit`. - `yarn tsc --project tsconfig.test.json --noEmit` fails at the unchanged remote head with nine diagnostics. They affect the new timing-table callback and existing mobile-driver, network-mock, and screenshot tests. Vitest reports no type errors, but its result does not establish a clean explicit test-project type check. - The `Proof` and `Unit tests` sections below retain the original historical evidence. These fresh checks do not repeat live-device validation. </details> ## Breaking changes - `readonly`, `discovery` and `cdp` tools no longer collect observations. As a result: - their responses no longer include `observations` (`cdp` used to include them); - they no longer refresh the session refMap through an observation (discovery tools still set it themselves); - they no longer move the diff baseline for the next mutating call. - A `cdp` call that changes state (for example `Runtime.evaluate` with a `.click()`) must now be followed by `describe-screen` to get fresh refs. - The public `ToolCategory` type gains `'cdp'`. Exhaustive switches on it need a new case. ## Consumer impact | Consumer | Change needed | Done on branch | |---|---|---| | Extension `test/e2e/playwright/llm-workflow/` | None. No code or test reads `observations`. | n/a | | Mobile `tests/llm-workflow/` | None. Same check. | n/a | ## Proof Tested with a yalc build of this branch in the Extension (e2e build, headless) and in Mobile (dev client on iOS sim and Android emulator), on wallet home after unlock. Before = #60 tip (`0.12.0+f2caa574`), after = `0.12.0+ce3fd9f0`. Ext was rechecked on the final commit (`0.12.0+8a7091bf`). Bench: `scripts/bench.mjs`, 5 runs (Android 3), p50. | Check | Before | After | |---|---|---| | `cdp Runtime.evaluate 1+1` bytes, ext / iOS / Android | 7408 / 59228 / 1054409 (smoke) | **267 / 240 / 244** | | `get-state` totalMs, ext / iOS / Android | 695 / 5736 / 6345 | **9 / 652 / 129** | | `cdp` totalMs, ext / iOS / Android | 963 / 3334 / 6882 | **2 / 6 / 497** | | `list-testids` totalMs, ext / iOS / Android | 1340 / 8901 / 13665 | 632 / 3835 / 11639 | | `observationMs` for `get-state`, `get-text`, `hermes-targets`, `list-testids`, `describe-screen`, `screenshot`, `accessibility-snapshot`, `cdp` | 53–8001 | **0** (no `observations` key), all platforms | | `click --testid unlock-submit` observations, ext / iOS / Android | yes | yes (71 / 4638 / 2355 ms) | <details> <summary>Notes from the proof run</summary> - Host load was 7.7–18.4 during the runs, so mobile toolMs varies a lot from run to run. - Android discovery `toolMs` went up in the after run (`list-testids` p50 5975 → 11639 ms). This change does not touch discovery tools. The likely cause is variation in the ~1 MB Android a11y snapshot (the same area as the existing `list-testids` timeout). Totals still fell because observation time is now 0. - On Android, some mutating calls (`wait-for`, `dismiss-keyboard`) still hit the 8 s observation timeout and return no `observations`. This already happened before the change (P1-04 build: iOS `click-unlock` 8003 ms). </details> <details> <summary>Transcripts (ext, final commit)</summary> ``` $ yarn mm cdp Runtime.evaluate '{"expression":"1+1","returnByValue":true}' { "method": "Runtime.evaluate", "result": { "result": { "type": "number", "value": 2, "description": "2" } }, "meta": { "timing": { "queueMs": 0, "toolMs": 2, "observationMs": 0, "totalMs": 3 } } } click-unlock obs=yes observationMs=16 describe-screen obs=no observationMs=0 get-state obs=no observationMs=0 list-testids obs=no observationMs=0 cdp-eval obs=no observationMs=0 bytes=266 screenshot obs=no observationMs=0 ``` </details> <details> <summary>Unit tests</summary> ``` Test Files 46 passed (46) Tests 1406 passed (1406) Type Errors no errors All files | 96.92 | 93.37 | 94.85 | 97.1 yarn lint → exit 0 ``` New or changed: per-category tests for `shouldCollectObservations` and `shouldIncludeObservationsInResponse`; endpoint tests for `observationMs` (0 for readonly, discovery and cdp; > 0 for mutating); read tools don't move the diff baseline; `accessibility_snapshot` resets it; the cdp category in the registry. </details>
4 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack context: mm CLI v2, Phase 1, P1-04, decision D6. Base: #59; next: #61. Review this delta against #59.
Problem
The shared HTTP executor awaits a knowledge-store write after each tool execution with a session ID. It also records execution failures. The store persists tool inputs, outcomes, observations, and artifact references under
test-artifacts/llm-knowledge/<sessionId>/steps/.Four
knowledge_*tools read that history. Decision D6 removes this subsystem from the core. Session metadata and HTTP observations still need independent homes after the store disappears.Solution
Delete the store, its recording hooks, retrieval tools, schemas, CLI commands, and public exports. Remove
knowledgeStorefromServerConfigandToolContext, so tool execution no longer depends on a history service.Keep two boundaries explicit:
writeSessionMetadata(artifactsDir, metadata)when they persist launch metadata.flowchart TD A[Tool request] --> B[Validate and run tool through queue] B --> C[Collect observations under existing policy] C --> D[Build timed HTTP response] E[Consumer launch metadata] --> F[Validate session ID in writeSessionMetadata] F -->|valid| G[Write session.json beneath artifactsDir] G --> I[Return absolute file path] F -->|invalid| H[Reject; no metadata write] classDef new fill:#fff3cd,stroke:#d39e00 classDef stop fill:#f8d7da,stroke:#c82333 class F,G,I new class H stopYellow boxes are new. Red boxes stop the metadata write.
The metadata writer creates missing directories, overwrites
session.json, and writes formatted JSON with a trailing newline. Relative artifact directories resolve against the process working directory. Invalid session IDs reject withInvalid session id: "<id>"before directory creation.Move
StepRecordObservationandcreateDefaultObservationtosrc/tools/types/observation.ts. MoveSessionMetadatatosrc/tools/types/session.ts, withoutgoalorflowTags. Keep free-formtags.What changes
knowledge_*toolsmm knowledge-*--goalor--flow-tagsllm-knowledge/<id>/<artifactsDir>/<id>/session.jsonThe CLI uses its existing unknown-command message:
Removed launch flags use the existing warning path, such as
Warning: unknown launch flag '--goal'. The warning alone does not reject launch. The protected sections below list all breaking exports and consumer migration details.Decisions for reviewers
StepRecordObservationandcreateDefaultObservationnamesReview guide
Read the files in this order:
src/server/create-server.ts,src/types/http.tssrc/server/session-metadata.ts,src/tools/types/session.tssrc/tools/types/observation.ts,src/index.ts,src/tools/types/index.tssrc/tools/registry.ts,src/validation/schemas.ts,src/cli/mm.tssrc/server/session-metadata.test.ts,src/tools/types/observation.test.ts, CLI/server/registry testsREADME.md,SKILL.mdTesting
2f10dd5ac47198f9259f24de061ee2b0bd0209a8:CI=true yarn vitest run --maxWorkers=2 --minWorkers=1 --coverage.enabled=falsepasses: 46 files, 1,397 tests.yarn lint:eslint,yarn prettier README.md SKILL.md --check, andyarn tsc --noEmit --project tsconfig.build.jsonpass.yarn tsc --noEmit --project tsconfig.test.jsonfails on existing mobile-driver, network-mock, and screenshot test typing issues. Vitest reports no type errors, so its result does not replace this standalone check.ProofandUnit testssections below retain the original historical evidence. These fresh checks do not repeat live consumer validation.Breaking changes
KnowledgeStore,KnowledgeStoreConfig,createKnowledgeStore,setKnowledgeStore,getKnowledgeStore,hasKnowledgeStore,knowledgeStoretokenize,tokenizeIdentifier,expandWithSynonymsknowledge*Toolfunctions,knowledge*InputSchema,knowledgeScopeSchema,knowledgeFiltersSchema,Knowledge*ZFLOW_TAGS,FlowTag,STEP_LABELS,StepLabel, theStepRecord*types exceptStepRecordObservation, thePriorKnowledge*,Knowledge*,RecipeStepandSessionSummarytypesErrorCodes.MM_KNOWLEDGE_ERROR,extractTargetFromInput,extractScreenshotInfo,extractToolOutcomeToolNameno longer includesknowledge_*.ServerConfig.knowledgeStoreandToolContext.knowledgeStoreare removed.goal/flowTags, andSessionMetadatano longer has them.mm launch --goalnow gives the unknown-flag warning.mm knowledge-*commands are removed (unknown command, exit 1).test-artifacts/llm-knowledge/<id>/session.jsonto<artifactsDir>/<id>/session.json(test-artifacts/<id>/session.jsonwith the consumers'artifactsDir: 'test-artifacts'). Consumers callwriteSessionMetadata()themselves.Consumer impact
test/e2e/playwright/llm-workflow/KnowledgeStore/setKnowledgeStorefromdaemon.ts;metamask-provider.tscallswriteSessionMetadata(artifactsDir, …); nogoal/flowTags; tests; README, AGENTS.mdcryptotavares/mm-cli-v2-extension45c821d168atests/llm-workflow/cryptotavares/mm-cli-v2-mobile4e2661f209fProof
Tested with a yalc build of this branch (
0.12.0+f2caa574) in the Extension (e2e test build 13.56.0, headless) and Mobile (dev client 8.19.0, iOS sim + Android emulator), with the consumer commits above.test-artifacts/llm-knowledge/during a sessionsession.jsonper launchllm-knowledge/<id>/session.jsontest-artifacts/<id>/session.json(ext ×2, iOS, Android)mm knowledge-lastError: unknown command 'knowledge-last', exit 1mm launch --goal xWarning: unknown launch flag '--goal', launch oklist-testids30 s timeout = known B11)rg -i knowledge srcTranscripts
Unit tests