Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,9 @@ PR opened ──▶ restore baseline (screenmaps branch) ──▶ static parse
You need all four of these:

- An Expo or React Native app using expo-router or a react-navigation route map, with `expo-dev-client` installed and deep-linkable routes.
- An EAS build profile that produces a simulator build, or a simulator build of your own.
- An EAS build profile that produces a simulator build (iOS) or an APK (Android — set `"android": { "buildType": "apk" }`, since an `.aab` cannot be installed on an emulator), or a build of your own.
- A GitHub repo you can add secrets to.
- macOS runner minutes. screenmap runs on macOS only, which bills at ten times the Linux rate. A JavaScript-only pull request takes about 12 minutes end to end.
- Runner minutes. iOS needs macOS, which bills at ten times the Linux rate; Android runs on `ubuntu-latest`. A JavaScript-only pull request takes about 12 minutes end to end.

### Install

Expand Down Expand Up @@ -121,10 +121,15 @@ You need all four of these:
| --- | --- | --- |
| `effort` | `balanced` | The preset from step 3. The `effort` input and `SCREENMAP_EFFORT` set the same thing |
| `scheme` | from the parsed app config | URL scheme the deep links use |
| `device` | `iPhone 16 Pro` | Simulator to boot. In the Action, the `simulator` input boots the device |
| `platforms` | `["ios"]` | Which platforms a run captures — `["ios"]`, `["android"]`, or both. In the Action each platform is its own job (the `platform` input), folded together afterwards by `screenmap-ci merge` |
| `ios.device` | `iPhone 16 Pro` | Simulator to boot. In the Action, the `simulator` input boots the device |
| `ios.appPath` | discovered under `ios/build` | A prebuilt simulator `.app`. The `app_path` input wins over it |
| `ios.appId` | read from the `.app` | Bundle id. Override when discovery picks the wrong one |
| `android.device` | any attached device, else the first AVD | AVD name, or the model of an attached device. In the Action, the `avd` input |
| `android.appPath` | discovered under `android/app/build/outputs/apk` | A prebuilt `.apk`. The `app_path` input wins over it |
| `android.appId` | read from the APK with `aapt2` | Package name. Also accepted as `android.packageName` |
| `appName` | the project directory name | Name recorded in the bundle |
| `bundleId` | read from the `.app` | Override when discovery picks the wrong one |
| `appPath` | discovered under `ios/build` | A prebuilt simulator `.app`. The `app_path` input wins over it |
| `device`, `bundleId`, `appPath` | — | Pre-multi-platform spellings of the `ios.*` keys above; still honoured |
| `metroPort` | `8081` | Port Metro starts on |
| `params` | `{}` | Real values for route parameters, see below |
| `suspects.depth` | from `effort` | Import hops followed out from a changed file |
Expand Down Expand Up @@ -283,7 +288,7 @@ What you get:
- **Runtime states as first-class screens.** Bottom-sheet snap points, modals and drawers are captured as variants of the screen they belong to.
- **The path to every screen, saved.** Each screen carries the exact tap sequence that reaches it, ready to replay headlessly. Commit those flows and CI replays them instead of paying an agent to rediscover them.

Output lands in `<project>/.screenmap/out/`, so add that to your `.gitignore`. You need a macOS host with the iOS simulator.
Output lands in `<project>/.screenmap/out/`, so add that to your `.gitignore`. You need a macOS host with the iOS simulator, or an Android emulator (`--platform android`); `--platform both` captures each screen on both and puts them in one map behind a platform switcher.

## The map viewer

Expand Down Expand Up @@ -322,7 +327,7 @@ Drop a `.scrmap` bundle on the landing page. The demo bundle ships in `public/de
```

1. **Static parse** (no dependencies). Reads expo-router file conventions and react-navigation route maps (the kind Bluesky keeps in `src/routes.ts`), so the screen list is complete rather than whatever a crawler happened to find. It produces the route list, navigation edges from `Link` and `navigate()` calls, and state hints saying which screens use a bottom-sheet or dialog system.
2. **Agent exploration** in the iOS simulator. A deep-link sweep captures every screen and classifies each capture (real, empty state, not found, error boundary, auth wall). For the screens a deep link cannot reach, an agent drives the app and records the tap path as an [argent](https://argent.swmansion.com) flow in YAML, replayable later with `argent flow run`. Runtime states get captured too: open drawers, bottom-sheet snap points, dialogs. If a sticky error boundary blocks the app, the agent recovers and carries on.
2. **Agent exploration** in the iOS simulator or Android emulator. A deep-link sweep captures every screen and classifies each capture (real, empty state, not found, error boundary, auth wall). For the screens a deep link cannot reach, an agent drives the app and records the tap path as an [argent](https://argent.swmansion.com) flow in YAML, replayable later with `argent flow run`. Runtime states get captured too: open drawers, bottom-sheet snap points, dialogs. If a sticky error boundary blocks the app, the agent recovers and carries on.
3. **Pack.** Everything merges into a producer-agnostic `.scrmap` zip. The format contract is in [docs/scrmap-format.md](docs/scrmap-format.md), which is what you need if you want to write your own producer.
4. **Visualise.** The viewer draws a top-down graph with the root screen at the top-center and phone-framed screenshots. Load a second bundle, a `.diff.scrmap`, and it overlays what a pull request changed.

Expand All @@ -342,7 +347,10 @@ The expensive part is step 2, and you only pay it once. Recorded flows get commi

## Known limits

- **iOS only.** Android is not supported yet. The interactive phases need a macOS host with the iOS simulator, and there is a web fallback for capture but not for tap recording.
- **One platform per CI job.** iOS needs a macOS runner and Android is only worth running on Linux, so capturing both means two jobs and a `screenmap-ci merge` step; the workflow templates show the shape. A local run does both in one pass.
- **Android's dev-menu muting is best-effort.** iOS writes the preference through `simctl spawn defaults`; Android has to reach the app's SharedPreferences through `run-as`, which only works for a debuggable build. When it fails, the dev-menu floating button stays in the captures — cosmetic, and the run continues.
- **OCR on Linux is tesseract, not Vision.** The landing checks, deep-link verification and system-alert dismissal all read the screen, and tesseract recovers noticeably fewer words than Apple Vision. Screen-to-screen comparisons hold up (same-text scores are unchanged; different-screen scores only move further apart), but a landmark check is likelier to miss, so a drift warning from a Linux run is less certain than one from macOS. The run summary and the PR comment name the backend when it is not Vision.
- **Flows are per platform.** Coordinates are normalized, but layouts and system chrome are not, so a flow recorded on iOS is not guaranteed to replay on Android. Record and commit them per platform.
- **Your app needs a router screenmap can read.** expo-router file conventions or a react-navigation route map. Screens registered without URLs are invisible to the static parse, and only show up through agent exploration.
- **Edge extraction is regex-based**, so dynamic hrefs resolve to their route pattern.
- **It reports, it does not gate.** There is no pass/fail check, by design. A reviewer decides what the screenshots mean.
Expand Down
133 changes: 133 additions & 0 deletions TODOS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,139 @@ the full write-up of what that turned up is in `site/docs/setup-instruction-fixe

---

## Android: verified on CI, and what the first runs cost

Android support landed on 2026-09-01 and ran green on GitHub Actions on
2026-09-08 against `aleqsio/screenmap-test`: emulator booted, APK installed,
`adb reverse` tunnelled, `brew://` deep links resolved, first JS bundle served,
eight screens captured and published. **The adb driver itself needed no changes
after the first successful boot** — every fix was in `action.yml` or in how the
run reported failure. What it took, in order, because none of it was reachable
from a laptop:

1. **`yes | sdkmanager` failed a step that succeeded.** GitHub runs `shell: bash`
with `-eo pipefail`; `yes` dies of SIGPIPE the moment sdkmanager stops
reading, and pipefail takes that as the pipeline's status. Read
`PIPESTATUS[1]` instead.
2. **`libpulse.so.0` is not on GitHub's ubuntu images**, and the SDK's qemu links
against it, so `emulator` could not start at all. Installed for android runs
along with the X libs, those best-effort since the names drift between
releases.
3. **avdmanager and emulator disagreed on where AVDs live.** Different resolution
chains (`ANDROID_AVD_HOME`, then `$ANDROID_SDK_HOME/.android/avd` for one and
`$HOME/.android/avd` for the other). The AVD was created and invisible.
`ANDROID_AVD_HOME` is now pinned for both.
4. **Every capture came back behind "Pixel Launcher isn't responding".** An
emulator on software rendering trips the ANR watchdog, the dialog is modal,
and it lands in every capture after it. `hide_error_dialogs` stops the system
drawing them; `ALERT_HINTS` learned the wording as a second layer.

Two of those cost far more than they should have because the run reported the
wrong thing, which is the lesson worth keeping: `emulator -version` was `|| true`,
so a broken binary surfaced seventeen minutes and one EAS build later as "no AVD
defined" — the one thing that was not wrong. Both are now hard gates that print
the underlying tool's own complaint.

Incidental finds along the way: EAS generates Android credentials
non-interactively, so no keystore setup is needed; fingerprint reuse works
(a rebuild collapsed to an 8-second download); and the baseline workflow's
`full` input had never been wired to `--full`.

### The PR lane, and the clock

Both settled on 2026-09-08 by
[screenmap-test#17](https://github.com/aleqsio/screenmap-test/pull/17), a
one-file copy change to `/grind`.

The diff lane works: suspects narrowed to exactly one node, only that screen was
captured on the head side, the base side came from the Android baseline, and the
sticky comment rendered with the before/after pair, the right reason
("its own source changed"), the Android device name, and the tesseract line.

**The clock is a non-issue.** Base and head agree: neither shows one. The
status-bar strip in both contains only the app's own eyebrow text, because this
app draws edge-to-edge over that area. The `9:41` seen in the run with the ANR
dialog up was the anomaly — the dialog was changing the window insets. Demo mode
is still worth setting for the icons, but on an edge-to-edge app the clock it
pins may never be visible, and that is fine: what a diff needs is base and head
agreeing, which they do.

### Still unverified

- **`muteDevMenu()` remains a guess.** No dev-menu overlay appeared in any
capture, but this app may simply not show one where iOS would, so the
SharedPreferences filename and keys are still unconfirmed. It stays
best-effort and non-fatal.
- **Flow replay on Android.** Every run so far was flowless — `0 by flow
replay`, everything deep-linked. argent's device tools take an Android serial,
but no committed flow has actually been replayed on one, so `replayFlow()` and
`verifyLanding()`'s landmark check are still untested on this platform.
- **The agent lane on Android.** Deliberately off (no key) for these runs, so the
platform-specific prompt in `agent.mjs` has never been exercised.

## The screenmaps branch has no platform in its paths

Found while planning the first Android CI run, on 2026-09-08. Not fixed.

A baseline publishes to `main/<sha7>.scrmap` and `main/latest.scrmap`, and a PR
run restores `main/<base_sha7>.scrmap` falling back to `main/latest.scrmap`.
Nothing in either path names a platform. So in a repo that already maps iOS, an
Android baseline overwrites the iOS map, and the next iOS PR run restores an
Android baseline: the diff's static verdicts still hold, but every base-side
capture is missing or belongs to the wrong device.

The workaround for a first experiment is the existing `screenmaps_branch` input
— point the Android runs at their own branch and nothing collides. That is fine
for a trial and wrong as an answer, because a repo mapping both platforms wants
one map, not two branches.

Deciding it properly means picking where the merge happens:

1. **Per-platform paths plus a merged map.** Each platform publishes
`main/<platform>/latest.scrmap`; a merge job writes `main/latest.scrmap` from
them. PR runs restore the merged one, so the viewer keeps getting a single
bundle. Costs a third job and makes `latest.scrmap` a derived artifact.
2. **Per-platform paths only**, with the viewer loading two bundles. Cheaper in
CI, but it pushes the merge onto every reader and the PR comment can only
show one.

Whichever wins, existing repos have an unprefixed `main/latest.scrmap` that must
keep resolving, or every repo loses its baseline on upgrade — the same hazard as
the `appmaps` -> `screenmaps` branch rename.

## OCR recall on the Linux lane

Measured on 2026-09-01 against six real captures (downscaled 368x800) plus a
full-resolution simulator capture:

- **The coordinate flip is correct**, which was the one thing that had to be. The
tesseract adapter reports pixels from the top-left and Vision reports normalized
from the bottom-left; across four real captures, 22 of 23 strings both backends
read agree on the resulting tap-y to within 0.006. (The one outlier is a screen
with two "About" labels, where the backends matched different instances.) A wrong
flip would have sent every system-alert dismissal to the mirror image of the
button.
- tesseract recovers ~61% of the words Vision does on app screens, and ~47% on a
sparse springboard capture, where it also missed the frozen "9:41" clock that
Vision read. Chrome-heavy, low-text screens are its worst case. Tuning did not
move it:
`--psm 6/11/12/3/4`, `--oem 1`, and a confidence floor all landed within a
point of each other.
- The decisions that gate a capture transfer intact. Landmark containment — the
strong signal — passed on the right screen and scored 0.00 on the wrong one
under both backends. Same-screen jaccard is 1.00 on both; different-screen
jaccard moves *down* (0.88 -> 0.75, 0.59 -> 0.48), which makes the
`bogus-param` probe more conservative rather than less.
- The weak `deeplink-text` fallback (`j >= 0.35`) is equally blunt on both: this
app's screens share enough chrome that different screens score 0.48-0.88. That
is a pre-existing property, not a tesseract regression, but it means a route
with no landmarks is barely verified on either platform. Committed landmarks
remain the only strong signal — say so in the docs rather than tuning 0.35.

The run summary and the PR comment now name the OCR backend whenever it is not
Vision, so a drift warning from a Linux run can be read with the right amount of
suspicion.

## Drifted flows have no repair path without an agent

`effort=deterministic` (now the automatic choice when no agent key is set) makes
Expand Down
Loading