Skip to content

fix!: scroll_to_element reports swipes and fails unless in the viewport #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
cryptotavares wants to merge 1 commit into from
Draft

fix!: scroll_to_element reports swipes and fails unless in the viewport #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed

- Start the CLI faster: find the project root without running `git` and do not load zod. Load `@metamask/device-mcp` only when a mobile session uses it, so browser daemons start without it.
- **BREAKING:** `scroll_to_element` `direction` is now the content direction (`down` reveals content below; it was the finger direction) and defaults to `auto`. The result adds `swipes`, `inViewport` and `bounds`; a target that is found but not brought into view fails with the new `MM_TARGET_NOT_VISIBLE`, and one never found with `MM_TARGET_NOT_FOUND`. The viewport check needs a device-mcp release with MetaMask/device-mcp#50; with device-mcp 0.7.0 only `bounds` is returned.
- Mobile `click` and `type` tap the element that the target wait found, so a tap takes one device snapshot instead of two (needs a device-mcp release with `tapElement(query, { element })`; older releases ignore the option). `describe_screen` reads the app state and the screen at the same time on mobile.

### Fixed
Expand Down
38 changes: 19 additions & 19 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,7 +432,7 @@ The daemon routes `POST /tool/:name` requests through the registry, applies Zod
| **Hermes (mobile only)** | |
| `hermes_targets` | Lists and diagnoses the debuggable Hermes targets Metro exposes (iOS and Android), reporting which target would be chosen or why selection is ambiguous. Use to confirm Metro is running and the app is registered. Pass `all` to bypass the appId filter and discover the real appId. Mobile only — there is no browser equivalent. |
| **Mobile (iOS/Android only)** | |
| `scroll_to_element` | Scrolls the screen until an element (a11y ref, test ID, or selector) becomes visible. Supports `direction` (`up`/`down`) and `maxAttempts`. Mobile only. |
| `scroll_to_element` | Swipes until the target (a11y ref or test ID) is inside the viewport, not only in the UI tree, and returns `swipes`, `inViewport` and `bounds`. `direction` is the content direction (`down` reveals content below); default `auto` picks it from the element's position. Fails with `MM_TARGET_NOT_VISIBLE` (found, not in view) or `MM_TARGET_NOT_FOUND`. `maxAttempts` = maximum swipes (default 10). Mobile only. |
| `device_swipe` | Swipes the screen in a `direction` (`up`/`down`/`left`/`right`), with optional `startX`/`startY`/`distance`. Mobile only. |
| `long_press` | Long-presses an element (a11y ref, test ID, or selector) for an optional `durationMs`. Mobile only. |
| `tap_coordinates` | Taps raw `x`/`y` screen coordinates. Mobile only. |
Expand Down Expand Up @@ -754,26 +754,26 @@ reach the agent intact.
These commands require a mobile session (`mm launch --platform ios|android`). Run on a
browser session, they return `MM_TOOL_NOT_SUPPORTED_ON_PLATFORM`.

| Command | Description |
| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `mm scroll-to-element <ref> [--direction up\|down] [--maxAttempts <n>] [--selector <css>] [--testid <id>]` | Scrolls until the target element is visible. |
| `mm device-swipe --direction <up\|down\|left\|right> [--startX <n>] [--startY <n>] [--distance <n>]` | Swipes the screen in a direction. |
| `mm long-press <ref> [--duration <ms>] [--selector <css>] [--testid <id>]` | Long-presses the target element. |
| `mm tap-coordinates <x> <y>` | Taps raw screen coordinates. |
| `mm dismiss-keyboard` | Dismisses the on-screen keyboard. |
| `mm dismiss-alert [--accept]` | Dismisses a native alert; `--accept` accepts it. |
| `mm get-alert-text` | Prints the text of a visible native alert. |
| `mm open-app <bundleId>` | Launches or foregrounds an app. |
| `mm close-app <bundleId>` | Terminates an app. |
| `mm press-button <home\|back\|enter\|lock>` | Presses a hardware/system button. |
| `mm device-context list` / `mm device-context switch <name>` | Lists or switches native/webview contexts. |
| `mm device-clipboard read` / `mm device-clipboard write <text>` | Reads or writes the device clipboard. |
| `mm screen-recording start [--output <path>]` / `mm screen-recording stop` | Starts or stops a screen recording; `--output` is sandboxed to `artifactsDir`. |
| `mm device-logs [--duration <seconds>] [--filter <text>]` | Fetches device logs. |
| `mm generate-locators` | Lists ranked selector suggestions for on-screen elements. |
| Command | Description |
| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `mm scroll-to-element <ref> [--direction auto\|down\|up] [--maxAttempts <n>] [--selector <css>] [--testid <id>]` | Swipes until the target element is inside the viewport. |
| `mm device-swipe --direction <up\|down\|left\|right> [--startX <n>] [--startY <n>] [--distance <n>]` | Swipes the screen in a direction. |
| `mm long-press <ref> [--duration <ms>] [--selector <css>] [--testid <id>]` | Long-presses the target element. |
| `mm tap-coordinates <x> <y>` | Taps raw screen coordinates. |
| `mm dismiss-keyboard` | Dismisses the on-screen keyboard. |
| `mm dismiss-alert [--accept]` | Dismisses a native alert; `--accept` accepts it. |
| `mm get-alert-text` | Prints the text of a visible native alert. |
| `mm open-app <bundleId>` | Launches or foregrounds an app. |
| `mm close-app <bundleId>` | Terminates an app. |
| `mm press-button <home\|back\|enter\|lock>` | Presses a hardware/system button. |
| `mm device-context list` / `mm device-context switch <name>` | Lists or switches native/webview contexts. |
| `mm device-clipboard read` / `mm device-clipboard write <text>` | Reads or writes the device clipboard. |
| `mm screen-recording start [--output <path>]` / `mm screen-recording stop` | Starts or stops a screen recording; `--output` is sandboxed to `artifactsDir`. |
| `mm device-logs [--duration <seconds>] [--filter <text>]` | Fetches device logs. |
| `mm generate-locators` | Lists ranked selector suggestions for on-screen elements. |

```bash
mm scroll-to-element e12 --direction down
mm scroll-to-element e12 # auto: scrolls toward the element
mm device-swipe --direction up --distance 400
mm open-app io.metamask
mm device-context switch WEBVIEW_1
Expand Down
12 changes: 9 additions & 3 deletions SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -591,9 +591,15 @@ These commands require a mobile session (`mm launch --platform ios|android`). On
session they return `MM_TOOL_NOT_SUPPORTED_ON_PLATFORM`. Element-targeting commands accept
the same `<ref>` / `--testid` / `--selector` targeting as `mm click`.

#### `mm scroll-to-element <ref> [--direction up|down] [--maxAttempts <n>]`
#### `mm scroll-to-element <ref> [--direction auto|down|up] [--maxAttempts <n>]`

Scrolls the screen until the target element becomes visible.
Swipes until the target is inside the viewport (being in the UI tree is not
enough; React Native keeps off-screen rows in the tree), so a following `click`
can tap it. Returns `{scrolled, swipes, inViewport, bounds}`. `--direction` is
the content direction: `down` reveals content below. The default `auto` picks
it from where the element is. Errors: `MM_TARGET_NOT_VISIBLE` (found but still
outside the viewport after `--maxAttempts` swipes or at the end of the list;
the message has its last frame), `MM_TARGET_NOT_FOUND` (never in the UI tree).

#### `mm device-swipe --direction <up|down|left|right> [--startX <n>] [--startY <n>] [--distance <n>]`

Expand Down Expand Up @@ -652,7 +658,7 @@ for every interactive element on the current screen. Read-only — useful when a
mobile automation to pick stable selectors.

```bash
mm scroll-to-element e12 --direction down
mm scroll-to-element e12
mm device-swipe --direction up --distance 400
mm open-app io.metamask
mm device-context switch WEBVIEW_1
Expand Down
2 changes: 1 addition & 1 deletion src/cli/mm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -917,7 +917,7 @@ export async function routeCommand(
!args.includes('--testid')
) {
process.stderr.write(
'Usage: mm scroll-to-element <ref> [--selector <css>] [--testid <id>] [--direction up|down] [--maxAttempts <n>]\n',
'Usage: mm scroll-to-element <ref> [--selector <css>] [--testid <id>] [--direction auto|down|up] [--maxAttempts <n>]\n',
);
process.exit(1);
}
Expand Down
Loading
Loading