Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Update the `testing/visual-testing` skill for `@metamask/client-mcp-core` 0.13.1: remove the knowledge store commands and the mandatory knowledge-reuse step, and document the global `--observe` and `--pretty` flags, one-line JSON output, observations without app state, `run-steps` `observe` (replaces `includeObservations`), timed-out calls being cancelled, and one element per ref. Replace the removed `yarn build:test:webpack` script with `yarn build:test`.
- Update the MetaMask Mobile performance overlay and animation references for React Native 0.86.x, Expo SDK 57, and Reanimated 4.x with `react-native-worklets`. From a worklet, call `scheduleOnRN` (`runOnJS` is a deprecated alias). Static-ESM tree-shaking stays off. ([#203](https://github.com/MetaMask/skills/pull/203))
- Require every revert yes/no to be one line that names the pull request or commit, the files, and exactly one yes/no question, whether or not security or privacy is involved. A shorter message is not that step.
- Keep a new public method or other contract change on the brief and spec path when the objection is that another package should own the edit. Put that objection in the brief.
Expand Down
51 changes: 36 additions & 15 deletions domains/testing/skills/visual-testing/references/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,12 @@ Complete command reference for the `mm` CLI. For core workflow and getting start
## Contents

- [Syntax Rules](#syntax-rules)
- [Global Options](#global-options)
- [Lifecycle](#lifecycle)
- [Interaction](#interaction)
- [Navigation & Tabs](#navigation--tabs)
- [Context](#context)
- [State, Knowledge & Seeding](#state-knowledge--seeding)
- [State & Seeding](#state--seeding)
- [Advanced](#advanced)

## Syntax Rules
Expand Down Expand Up @@ -62,6 +63,26 @@ mm click --selector "#connectButton" # CSS selector (flag)
| Type by testId | `mm type --testid X "text"` | `mm type --testId X "text"` |
| Type by a11y ref | `mm type e2 "text"` | `mm type --ref e2 "text"` |

## Global Options

These flags work with every command and can appear anywhere in the command line.

| Flag | Description |
|---|---|
| `--project <path>` | Target a specific project directory. Overrides `MM_PROJECT` and git-based discovery |
| `--observe <none\|diff\|full>` | Observation returned after the call. Default: `diff` for mutating commands and `run-steps`, `none` for reads, discovery and `cdp` |
| `--pretty` | Indent JSON output. Default: one line per result when stdout is not a terminal, indented on a terminal. `MM_PRETTY=1` / `0` sets the default |

| `--observe` level | Response |
|---|---|
| `none` | No `observations`. Fastest. Refs are **not** refreshed: run `mm describe-screen` before you use refs again |
| `diff` | Only what changed since the previous observation |
| `full` | The whole compacted screen (`a11y` + `testIds`). Becomes the new diff baseline |

Observations contain `testIds` and `a11y` only, no app state. Use `mm get-state` for state.

Every tool, `launch` and `cleanup` response includes `meta.timing` (`queueMs`, `toolMs`, `observationMs`, `totalMs`).

## Lifecycle

| Command | Description |
Expand Down Expand Up @@ -111,20 +132,18 @@ Tab roles: `extension`, `notification`, `dapp`, `other`.

Two contexts: `e2e` (default — local Anvil, fixtures, seeding) and `prod` (no fixtures, no local chain). Cannot switch during active session — run `mm cleanup` first.

## State, Knowledge & Seeding

| Command | Description |
| ----------------------------- | ----------------------------------------- |
| `mm get-state` | Get current extension state |
| `mm knowledge-search <query>` | Search steps across sessions |
| `mm knowledge-last` | Get last N step records from this session |
| `mm knowledge-sessions` | List recent sessions with metadata |
| `mm knowledge-summarize` | Generate session recipe |
| `mm run-steps <json>` | Execute multiple tools in sequence |
| `mm seed-contract <type>` | Deploy a test contract |
| `mm seed-contracts` | Deploy multiple test contracts |
| `mm get-contract-address` | Get deployed contract address |
| `mm list-contracts` | List all deployed contracts |
## State & Seeding

| Command | Description |
| ------------------------- | ----------------------------------------------------------- |
| `mm get-state` | Get current extension state (the only source of app state) |
| `mm run-steps <json>` | Execute multiple tools in sequence |
| `mm seed-contract <type>` | Deploy a test contract |
| `mm seed-contracts` | Deploy multiple test contracts |
| `mm get-contract-address` | Get deployed contract address |
| `mm list-contracts` | List all deployed contracts |

The knowledge store commands (`mm knowledge-search`, `mm knowledge-last`, `mm knowledge-sessions`, `mm knowledge-summarize`) and the `mm launch --goal` / `--flow-tags` flags were removed in `@metamask/client-mcp-core` 0.13.1 (the first published 0.13 version). Session metadata is written to `test-artifacts/<sessionId>/session.json`.

## Advanced

Expand All @@ -136,5 +155,7 @@ Two contexts: `e2e` (default — local Anvil, fixtures, seeding) and `prod` (no
| `mm mock-network requests [--limit <n>]` | Show recorded matched and missed mocked-origin requests |
| `mm cdp <method> [params-json] [--timeout <ms>]` | Send raw Chrome DevTools Protocol command against active page|

`mm cdp` returns no observation by default. If the call changed page state, add `--observe full` or run `mm describe-screen`.

For mock network details, see [mock-network.md](mock-network.md).
For CDP state manipulation, see [state-manipulation.md](state-manipulation.md).
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,7 @@
- `home` → continue, but check for modals or blockers
- `onboarding-*` → complete onboarding
- `unknown` → take a screenshot and investigate
3. Query prior runs:
```bash
mm knowledge-search "send"
mm knowledge-sessions
mm knowledge-last
```
4. Capture `mm screenshot --name "debug"` for diagnosis
3. Capture `mm screenshot --name "debug"` for diagnosis

## Error Codes

Expand Down Expand Up @@ -53,7 +47,7 @@
| ------------------------------------------- | ---------------------------- | ------------------------------------------------------------- |
| `MM_SESSION_ALREADY_RUNNING` | Previous session not cleaned | `mm cleanup` |
| `MM_NO_ACTIVE_SESSION` | No browser running | `mm launch` |
| Extension not loading | Extension not built | `yarn build:test:webpack` then `mm launch` |
| Extension not loading | Extension not built | `yarn build:test` then `mm launch` |
| `EADDRINUSE` port error | Orphan processes | Check `.mm-server` for ports, kill orphaned process |
| `MM_TARGET_NOT_FOUND` | Element not visible | `mm describe-screen` to check state |
| `MM_WAIT_TIMEOUT` | Slow environment | Increase `--timeout`, inspect screenshot |
Expand All @@ -65,3 +59,7 @@
| `MM_CONTEXT_SWITCH_BLOCKED` | Active session | `mm cleanup` before `mm set-context` |
| Fixtures not available | Running in prod context | `mm set-context e2e` |
| Stale a11yRefs after navigate | Refs not refreshed | `mm describe-screen` for fresh refs |
| Ref clicks the wrong element | Ref used after `--observe none` | `mm describe-screen`, or target by `--testid` |
| Strict mode violation on click | Target matches more than one element | Target by `--testid` or `--selector` |
| `Tool execution timed out` | Call cancelled after the request limit | `mm describe-screen` before retry; the action can have happened |
| `Error: unknown command 'knowledge-...'` | Knowledge store removed | No replacement. Discover with `mm describe-screen` |
48 changes: 27 additions & 21 deletions domains/testing/skills/visual-testing/repos/metamask-extension.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,12 @@ For architecture details, see `test/e2e/playwright/llm-workflow/README.md`.

## Gotchas

- `a11yRef`s (`e1`, `e2`, ...) are **ephemeral**. After `mm describe-screen`, `mm accessibility-snapshot`, or major navigation, re-describe before reusing refs.
- `a11yRef`s (`e1`, `e2`, ...) are **ephemeral**. Always take refs from the latest response (`mm describe-screen` or the `observations` of your last action).
- After a call with `--observe none`, refs are **not** refreshed. Target by `--testid`, or run `mm describe-screen` before you use refs again: a stale ref built from role and name can click a different element.
- Each ref resolves to exactly one element. If a click fails with a strict mode violation, target the element by `--testid` or `--selector`.
- Output is one line of JSON when stdout is not a terminal. Parse it as JSON. Add `--pretty` (or set `MM_PRETTY=1`) only when a human reads the output.
- Observations carry no app state. Use `mm get-state` for the screen, account, network, or balance.
- When a call times out (`Tool execution timed out`), it is cancelled. An action that already started can still have happened, so run `mm describe-screen` before you retry.
- `mm type` uses Playwright `fill()` and **clears the field first**.
- After confirm or reject in sidepanel mode, the page does **not** close. It stays open and navigates back to the home route.
- `mm wait-for-notification` waits for the sidepanel confirmation route, not a legacy popup window.
Expand Down Expand Up @@ -51,7 +56,7 @@ yarn mm <command> # or: npx mm <command>
- `mm launch` validates the build and returns an actionable error if missing.

```bash
yarn install && yarn build:test:webpack
yarn install && yarn build:test # fresh worktree: run `yarn webpack:tsc` first
```

**Port conflicts** — if ports are stuck from a previous run, check `.mm-server` for active ports:
Expand All @@ -75,26 +80,26 @@ Available presets: `withMultipleAccounts`, `withERC20Tokens`, `withConnectedDapp

Two contexts: **e2e** (default — local Anvil, fixtures, seeding) and **prod** (no fixtures, no local chain). Use `mm get-context` / `mm set-context` to switch. Cannot switch during active session.

### 2. Reuse Knowledge (Mandatory)
### 2. Describe Screen

```bash
mm knowledge-search "<flow name>"
mm knowledge-sessions
mm describe-screen
```

Reuse discovered sequences when they exist. If none exist, proceed with discovery and let this session record the new steps.
Returns current screen, active tab, visible testIds, and accessibility tree with refs.

**Observation efficiency:** Mutating actions (`click`, `type`, `navigate`) and `mm run-steps` return compact diff-based observations after the first observation. Use these for quick next-step targeting. Reads, discovery commands and `mm cdp` return no observation. Call `mm describe-screen` when you need the full a11y tree, screenshots, or to reset the baseline.

### 3. Describe Screen
Use the global `--observe <none|diff|full>` flag to change the observation of one call:

```bash
mm describe-screen
mm click --testid unlock-submit --observe none # chain known actions: skip the observation
mm navigate-home --observe full # after navigation: get the whole screen
```

Returns current screen, active tab, visible testIds, and accessibility tree with refs.

**Observation efficiency:** Mutating actions (`click`, `type`, `navigate`) return compact diff-based observations after the first mutation. Use these for quick next-step targeting. Call `mm describe-screen` when you need the full a11y tree, screenshots, or to reset the baseline.
Only mutating and batch calls wait for animations before they observe. To get a settled screen after an action, let the action observe (`--observe full`) instead of running a read afterwards.

### 4. Interact
### 3. Interact

Use exactly **one targeting method** per call. Priority: `testId` (stable) > `a11yRef` (discovery) > `selector` (fallback).

Expand Down Expand Up @@ -149,15 +154,15 @@ Phase-specific error codes on timeout:

For the full error code table, see [references/error-recovery.md](references/error-recovery.md).

### 5. Verify-Fix Loop
### 4. Verify-Fix Loop

After any interaction sequence:

1. `mm describe-screen` to verify expected state
2. If wrong: `mm screenshot --name "debug-<action>"`, check `mm knowledge-search "<flow>"`, retry
2. If wrong: `mm screenshot --name "debug-<action>"`, then retry
3. Continue only when expected state is confirmed

### 5b. Edge Case: Component Not Responding to State Changes
### 4b. Edge Case: Component Not Responding to State Changes

If you change Redux state via CDP but the target component doesn't update:

Expand All @@ -167,7 +172,7 @@ If you change Redux state via CDP but the target component doesn't update:
4. Disconnect the live data feed first (prevents overwrite), then push your test data.
5. Re-verify with `mm describe-screen`.

### 6. Confirmations (Dapp Flows)
### 5. Confirmations (Dapp Flows)

```bash
mm navigate https://test-dapp.io # navigate to dapp
Expand All @@ -181,23 +186,23 @@ mm switch-to-tab dapp # back to dapp

Tab roles: `extension`, `notification`, `dapp`, `other`.

### 7. Navigate
### 6. Navigate

```bash
mm navigate-home
mm navigate-settings
mm navigate https://test-dapp.io
```

### 8. Screenshots
### 7. Screenshots

```bash
mm screenshot --name "after-unlock"
```

Capture before and after meaningful state changes for visual validation evidence.

### 9. Cleanup (Always Required)
### 8. Cleanup (Always Required)

```bash
mm cleanup # stop browser and services
Expand Down Expand Up @@ -229,10 +234,11 @@ mm run-steps '{"steps":[
Details:
- Prefer `testId`, `a11yRef`, or `selector` in args (not `ref`)
- Use `within` in args to scope targeting within a parent element
- Add `batchTimeoutMs` for overall timeout
- Add `batchTimeoutMs` (1000–30000 ms) for an overall deadline. The whole call stops after 30 s, so use a lower value (for example 20000) to get partial results, and split longer flows into several batches
- Add a top-level `observe` (`none` / `diff` / `full`, default `diff`) to set the observation collected after the batch. `includeObservations` was removed; check `summary.ok` to react to failures
- Tool aliases like `navigate_home` and `navigate-home` are supported

Pattern: discover with `mm describe-screen` → reuse from `mm knowledge-search` → batch with `mm run-steps` → re-verify with `mm describe-screen`.
Pattern: discover with `mm describe-screen` → batch known steps with `mm run-steps` → re-verify with `mm describe-screen`.

## Default Credentials

Expand Down
Loading