Capture on Android, and put both platforms in one map

[aleqsio]

screenmap has been iOS-only. This adds Android as a peer platform: a run captures on iOS, on Android, or on both, and both land in a single .scrmap behind a platform switcher in the viewer.

The shape of it

The device layer splits in two. lib/sim.mjs keeps the simctl primitives, lib/android.mjs is its adb counterpart, and lib/device.mjs holds what is genuinely shared — Metro, the dev-client connect loop, the capture helpers — so the two platforms cannot quietly drift apart.

argent needed no changes at all. Its device tools take an Android serial wherever they take an iOS UDID, so committed-flow replay — the thing that makes deterministic runs work — came along for free.

Three things are not one-to-one with simctl:

• Android has no "Open in …?" scheme prompt, so there is nothing to pre-approve and no prompt to tap through.
• The emulator's localhost is not the host's, so the driver opens an adb reverse tunnel before pointing the dev client at Metro. That also covers USB devices, which the 10.0.2.2 alias does not.
• The status bar is frozen through SystemUI demo mode rather than a dedicated override command.

Configuration

platforms: ["ios"] (the default), ["android"], or both, with per-platform ios.* / android.* blocks. The pre-multi-platform top-level device / bundleId / appPath keys still mean iOS, so existing configs need no edits — verified against sample-app's untouched config.

Bundles stay compatible by construction

A single-platform run emits exactly the v2 layout it always did. Only a multi-platform run becomes v3: screens move under screens/<platform>/, each node gains a captures map, and capture keeps mirroring the first platform so a v2 reader renders the map rather than showing every screen as missing. The diff bundle gains the same axis under its existing base/head split.

Capture status is per platform, which is the point — a screen that renders on iOS and crashes on Android is one node with two verdicts, not two maps to compare by eye.

CI

The platforms are separate jobs: iOS needs macOS, and Android is only worth running on ubuntu, which bills at a tenth of the rate. screenmap-ci merge folds their bundles into one map, so the split stays an implementation detail of the pipeline rather than something a reviewer has to hold in their head. The workflow templates carry the two-job shape.

OCR on Linux

Apple Vision is macOS-only, and without a fallback the landing checks, deep-link verification and system-alert dismissal all go dark on a Linux runner. So ocr.mjs gains a tesseract backend behind the unchanged ocr() / words() interface.

I measured it rather than assuming:

• The coordinate flip is correct, which was the one thing that had to be — tesseract reports pixels from the top-left, Vision normalized from the bottom-left, and every caller taps at 1 - (y + h/2). Across four real captures, 22 of 23 strings both backends read agree on the resulting tap-y to within 0.006. A wrong flip would have sent every alert dismissal to the mirror image of the button.
• The checks that gate a capture transfer intact. Landmark containment — the strong signal — passed on the right screen and scored 0.00 on the wrong one under both backends. Different-screen jaccard moves down under tesseract (0.88 → 0.75), making the bogus-param probe more conservative, not less.
• tesseract recovers ~61% of Vision's words on app screens (~47% on a sparse springboard capture). No --psm / --oem / scale tuning moved that.

The run summary and the PR comment now name the backend whenever it is not Vision.

Testing

Run against sample-app and its real captures: its pre-existing config still produces identical v2 output; incremental reuse carries all six real captures forward; the multi-platform bundle and diff round-trip; the viewer's switcher swaps captures and the bezel now takes its aspect ratio from the image rather than a hardcoded iPhone ratio. The refactored iOS driver boots a real simulator and captures from it.

That testing caught one bug worth naming. A single-platform bundle used to answer for any platform asked of it, so a repo turning Android on "reused" its iOS screenshots as Android captures — labelled Android in the viewer, and indistinguishable from a real run. Bundles now declare which platforms they hold, and merge refuses an input whose platform does not match its mapping.

What is not verified

The adb driver has never met a live emulator — this machine has no Android SDK. TODOS.md ranks what to check on the first real run; the riskiest is muteDevMenu(), whose SharedPreferences filename and keys are inferred from the iOS EXDevMenu* defaults rather than read off a device. It is deliberately non-fatal: a wrong guess costs a floating button in the corner of every screenshot, not a failed run. The KVM setup in action.yml is likewise unproven on a real ubuntu runner.

Opening as a draft for that reason.

---

Left alone deliberately: diff-map.mjs still defaults its output to .appmapdiff and reports generator: 'expo-map/2.0'. The CLI always passes --out so it rarely fires, but those look like leftover branding rather than the two deliberate appmap references — worth a separate look rather than folding a rename into this change.

🤖 Generated with Claude Code

[aleqsio]

Verified on CI — and it took six runs to get there

Marking this ready. The Android lane now runs green on GitHub Actions against aleqsio/screenmap-test, on ubuntu-latest, in about 5½ minutes:

booting AVD screenmap
adb reverse tcp:8081 → host
session ready: com.aleqsio.screenmaptest on sdk_gphone64_x86_64 (emulator-5554), Metro :8081
packed Brew-72d147f.scrmap: 8 nodes, 8 shots across android
published 2 file(s) to screenmaps-android

The adb driver needed no changes after the first successful boot. Every fix was in action.yml or in how a failure reported itself — none of them reachable without a real runner:

1. yes | sdkmanager failed a step that had succeeded — GitHub runs shell: bash with -eo pipefail, so yes dying of SIGPIPE became the pipeline's status.
2. libpulse.so.0 isn't on GitHub's ubuntu images and the SDK's qemu links against it, so emulator couldn't start at all.
3. avdmanager and emulator resolve the AVD directory through different env chains, so the AVD was created and invisible.
4. Every capture came back behind "Pixel Launcher isn't responding" — an emulator on software rendering trips the ANR watchdog, and the dialog is modal, so it landed in all eight screens.

Two of those cost far more than they should have because the run reported the wrong thing. emulator -version was || true, so a broken binary surfaced seventeen minutes and one EAS build later as "no AVD defined" — the one thing that wasn't wrong. Both are now hard gates that print the underlying tool's own complaint.

Corrections to the original description

• No keystore setup is needed. I claimed eas build --non-interactive couldn't generate Android credentials. It did, first try.
• Fingerprint reuse works on Android: a 17-minute rebuild collapsed to an 8-second download on the next run.

Found in passing, fixed here

The baseline workflow template has offered a full dispatch input described as "Rebuild the whole map instead of incrementally" since it was written, but action.yml never read it and never passed --full. Dispatching with full=true silently did an incremental run — which is exactly how a "successful" run tested none of the above.

Still unproven

• muteDevMenu() is still a guess; no overlay appeared, but this app may not show one where iOS would.
• The status-bar clock renders in one run and not another. Consistent within a run, so the baseline is fine — but across base and head it would mark every screen changed. Worth settling before the Android PR lane is trusted.
• The PR/diff lane itself has not run on Android. Only the baseline has.
• screenmaps paths carry no platform, so a repo mapping both needs screenmaps_branch to keep them apart. Written up in TODOS.md with the two ways it could resolve.

🤖 Generated with Claude Code

[aleqsio]

The PR lane works on Android too

screenmap-test#17 — a one-file copy change to /grind — exercised the diff lane end to end on ubuntu-latest:

status comment (pending) updated on aleqsio/screenmap-test#17
suspects: 0 added, 0 removed, 1 modified → capture 1 screens
session ready: com.aleqsio.screenmaptest on sdk_gphone64_x86_64 (emulator-5554)
published 3 file(s) to screenmaps-android
comment updated on aleqsio/screenmap-test#17

Suspects narrowed to exactly the one changed screen, only that screen was captured on the head side, the base side came from the Android baseline, and the sticky comment rendered with the before/after pair, the right reason ("its own source changed (grind.tsx)"), the Android device name, and the new OCR line:

Captured on sdk_gphone64_x86_64: 0 by flow replay (argent 0.21.0), 1 by deep link, 0 by agent.
Screen text read with tesseract (lower recall than Vision — verification warnings here are less certain).

The status-bar clock was a red herring

I flagged this as needing settling before the PR lane could be trusted. It doesn't. Base and head agree — neither shows a clock. The status-bar strip in both contains only the app's own eyebrow text, because this app draws edge-to-edge over that area. The 9:41 in the earlier run was the anomaly: the ANR dialog was changing the window insets.

Demo mode is still worth setting for the icons, but on an edge-to-edge app the clock it pins may never be visible — and that's fine, because what a diff needs is the two sides agreeing, which they do. The captured pair differs only where the copy changed (text jaccard 0.53, distinct words on each side).

Still untested on Android

Narrower than before, and neither blocks this:

• Flow replay. Every run so far was flowless — 0 by flow replay, everything deep-linked. argent's device tools take an Android serial, but no committed flow has actually been replayed on one, so replayFlow() and verifyLanding()'s landmark check are unexercised on this platform.
• The agent lane. Deliberately off (no key) throughout, so the platform-specific prompt in agent.mjs has never run.

🤖 Generated with Claude Code

[capy-ai]

Capy found no issues.

[capy-ai]

Capy found no issues.