Skip to content

feat(backends): win32 WindowClient for remote-display window replay#159

Merged
abrichr merged 4 commits into
mainfrom
feat/win32-window-client
Jul 19, 2026
Merged

feat(backends): win32 WindowClient for remote-display window replay#159
abrichr merged 4 commits into
mainfrom
feat/win32-window-client

Conversation

@abrichr

@abrichr abrichr commented Jul 19, 2026

Copy link
Copy Markdown
Member

What

Implements the win32 WindowClient for RemoteDisplayBackend — the largest single flow gap in the multi-substrate design: window-scoped remote-display replay was macOS-host-only, so a Windows endpoint (the wedge clinic's actual front-desk machine) could record a Citrix/RDP client window (capture 0.6.0 win32) but not replay into it. This PR closes the replay half on Windows hosts with the same client seam, same contract, same fail-loud discipline as the macOS client.

New module: openadapt_flow/backends/win32_window_client.py

  • SelectionEnumWindows with capture-0.6.0-parity owner matching (owning process executable basename, e.g. wfica32.exe for Citrix Workspace; exact case-insensitive, .exe-tolerant, never substring), exact case-insensitive title, optional exact window-class filter (expected_class, for dynamic-title targets), cloaked/untitled/invisible windows skipped. Zero or multiple matches surface to the backend's existing ambiguity halt — the client never "picks the largest".
  • Capture in the window's own pixel space — client-area rect (GetClientRect + ClientToScreen) as WindowInfo.bounds; PrintWindow(PW_CLIENTONLY | PW_RENDERFULLCONTENT) first, BitBlt of the client DC as fallback; both failing, window-gone, and window-minimized are typed errors (CaptureFailedError / WindowGoneError / WindowMinimizedError, all subclasses of RemoteDisplayError → existing halt semantics).
  • DPI: per-monitor v2 — awareness is established before any window call (SetProcessDpiAwarenessContext(PER_MONITOR_AWARE_V2), with shcore / SetProcessDPIAware fallbacks). Under per-monitor awareness, client rect, screen coordinates, and SendInput all speak physical pixels, so the backend's derived scale is 1.0 by construction. If only system/unaware can be achieved, the client refuses to capture or inject (DpiAwarenessError) — virtualized coordinates can mis-target a click.
  • Injection with identity disciplineSendInput (absolute virtual-desktop normalization with the 65535/(w−1) mapping), hardware-like VK+scancode key events (extended-key flags for the nav cluster), layout-resolved typing via VkKeyScanW with a documented Unicode fallback for AltGr/unmapped chars, real modifier transitions wrapped around chord keys (released in reverse order; the backend's finally guarantees no latched modifier). key_window_id requires the exact target HWND to hold the foreground (GetForegroundWindow root) — the backend's before/after focus proofs therefore halt on any wrong-window state, and window_at_point (WindowFromPointGA_ROOT) backs the occlusion refusal.
  • UIPI honesty — Windows silently discards synthetic input aimed at a higher-integrity process (the exact "dropped click that looks like success" this project refuses). The client evaluates target-process elevation against the last resolution: elevated or unknown elevation ⇒ input_trusted() == False and every input method independently refuses (InputDeliveryError). SendInput's injected-event count is additionally verified on the native path.
  • Import guard — nothing at module scope touches ctypes.windll; only NativeWin32Api() construction does (and raises a typed error off-Windows). Stdlib ctypes only — no pywin32 (the repo has no win32 dependency and keeps it that way).

Backend integration (remote_display.py, minimal and behavior-preserving on macOS):

  • Default client is now host-dispatched (darwinMacWindowClient, win32Win32WindowClient, others fail loud at construction instead of failing later inside Quartz imports).
  • Key-chord resolution moved behind the client seam: new WindowClient.resolve_key(token) -> (keycode, needs_shift) | None. MacWindowClient resolves via the existing mac tables (resolve_mac_key, byte-identical semantics — the existing remote-display suite still asserts the same mac keycodes); the win32 client resolves Windows VKs + the active layout. The backend no longer owns a platform keycode table.
  • The input-trust refusal message is now platform-neutral (macOS Accessibility / Windows UIPI); same match surface for existing tests.
  • factory.py: docstring only (window_client default is the host's native client).

Tests (mock-only, honestly labeled)

tests/test_win32_window_client.py — 30 contract tests, the entire Win32 layer faked behind the Win32Api seam (runs identically on any platform; also wired into the post-merge windows-mock job so it runs on a real Windows interpreter, still mocked):

  • selection matching: exe-basename owner (± .exe, substring refused), exact CI title, class filter, cloaked/untitled/invisible skipping, duplicate-title ambiguity halt through the backend;
  • coordinate mapping math: scale-1.0 client-area mapping end-to-end through RemoteDisplayBackend.click (origin offset asserted), normalize_to_virtual_desktop incl. the (w−1) off-by-one, negative-origin multi-monitor, and out-of-desktop refusal; DPI virtualization (system/unaware) refused;
  • focus-verification halt paths: foreground held by another window ⇒ refuse before any input; same-pid wrong-window; occlusion at point; focus stolen during the readiness probe ⇒ refuse; activate restores minimized then foregrounds; ambiguous pid ⇒ activation no-op;
  • capture fallback ordering: PrintWindow → BitBlt → typed error; gone/minimized typed errors; size-disagreement refusal;
  • UIPI: elevated target halts via the backend and directly at the client for every input method; unknown elevation fails closed; elevated driver may drive an elevated target;
  • input synthesis: shift-wrapped layout typing, Unicode fallback, chord modifier ordering, wheel sign convention (Backend parity), resolve_key named-table-then-layout (macOS chord parity), press() end-to-end through the backend emitting Windows VKs;
  • platform guard: NativeWin32Api() refuses non-Windows hosts; the module imports everywhere.

Full non-E2E suite + ruff (lint + format) + mypy green; claims gate green; docs/VERIFICATION.md/.json regenerated.

Mocked vs real — what this PR does and does not prove

Proven (CI): the win32 client conforms to the WindowClient seam contract the macOS client satisfies — selection/ambiguity, mapping math, refusal paths, fallback ordering — under a scripted Win32 API.

NOT proven: anything about a real Windows host. Real PrintWindow rendering quirks, real DPI virtualization, real foreground arbitration (SetForegroundWindow restrictions), real UIPI filtering, and real SendInput delivery are exercised by zero tests here. The new claims entry win32-window-replay-roadmap is deliberately tier: roadmap with a caveat saying exactly this; no supported/validating language was added anywhere. A real-Windows counted qualification is explicitly out of scope for this PR.

Validation procedure (for the lane owner — real Windows, later)

Environment: the Parallels "Windows 11" VM (snapshot first via prlctl snapshot; never delete the user's VM), or the internal cloud Windows runner image. Python 3.10–3.12 inside the guest.

  1. Interpreter leg (already automated): the post-merge/nightly windows-mock CI job now runs this suite on windows-latest — confirms the mocked contract on a real Windows interpreter.
  2. Live smoke (unscored): in the guest, pip install -e .[dev]; open Notepad; drive RemoteDisplayBackend at it (run --backend rdp with rdp_window: notepad in the deployment YAML, or construct directly with owner_substr="notepad"). Verify: screenshot() returns the client-area PNG with _scale == 1.0; ensure_foreground() proof passes; click/type_text land in Notepad; minimizing ⇒ WindowMinimizedError; overlapping another window over the click point ⇒ occlusion refusal; targeting an elevated app (e.g. admin Task Manager) ⇒ UIPI refusal, not a silent no-op.
  3. DPI matrix: repeat the smoke at 100%/150%/200% display scaling and (if available) mixed-DPI dual monitors; assert the scale stays 1.0 and clicks land — the highest-risk real-world divergence from the mocks.
  4. Counted analog batch (claims rung 2): use mstsc.exe connected to a second Windows session as the remote-display client window. Record window-scoped with capture 0.6.0 (RecordingConfig(window_owner="mstsc", window_title=…)), convert_capture → compile → run --backend rdp (rdp_window: mstsc) with an external effect verifier (--effects-kind file or document-hash), N≥20 runs; count success/halt/wrong-action. Publish under the label "remote-display client window (RDP analog) on a Windows host" — never as Citrix evidence (the guard tests forbid the substitution).
  5. Only after step 4 may win32-window-replay-roadmap move roadmap → validating (analog label); Citrix-labeled evidence remains design-partner-gated.

Files

  • openadapt_flow/backends/win32_window_client.py — new (client + Win32Api seam + NativeWin32Api ctypes bindings)
  • openadapt_flow/backends/remote_display.py — host-dispatched default client; resolve_key seam; platform-neutral trust message
  • openadapt_flow/backends/factory.py — docstring
  • tests/test_win32_window_client.py — new (30 mock-contract tests)
  • tests/test_remote_display_backend.py, tests/test_backend_factory.py — fakes gain resolve_key
  • .github/workflows/ci.yml — windows-mock job runs the new suite + the remote-display suite
  • claims.yaml + docs/VERIFICATION.md + docs/verification.json — roadmap-tier claim, regenerated report
  • docs/desktop/CITRIX_PIXEL.md — notes the win32 client (mock-tested, unqualified)

🤖 Generated with Claude Code

abrichr and others added 4 commits July 19, 2026 05:19
Windows hosts can now replay into a specific client window's own pixel
space (the Citrix/RDP client-window wedge for Windows-endpoint clinics):

- new openadapt_flow/backends/win32_window_client.py: EnumWindows
  selection (process-executable owner matching per capture 0.6.0, exact
  case-insensitive title, optional class filter; zero/multiple matches
  surface to the backend's ambiguity halt), client-area capture in the
  window's own pixel space (PrintWindow PW_CLIENTONLY|RENDERFULLCONTENT
  first, BitBlt fallback), per-monitor-v2 DPI awareness (refuses DPI
  virtualization), SendInput injection with foreground verification and
  a UIPI elevation guard, and typed failure modes that map onto the
  existing RemoteDisplayError halt semantics. Stdlib ctypes only behind
  a mockable Win32Api seam; import-safe on every platform.
- remote_display: default client is host-dispatched (darwin/win32);
  key-chord resolution moved behind the client seam (resolve_key, mac
  semantics byte-identical); input-trust refusal message made
  platform-neutral.
- 30 mock-contract tests (selection/ambiguity, coordinate + DPI math,
  focus-verification halts, capture fallback ordering, UIPI refusal);
  wired into the post-merge windows-mock CI job.
- claims: win32-window-replay-roadmap at roadmap tier (mock-tested
  only, real-Windows qualification explicitly out of scope);
  VERIFICATION regenerated; CITRIX_PIXEL.md notes the win32 client.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abrichr
abrichr force-pushed the feat/win32-window-client branch from abea7d5 to 7b05ebb Compare July 19, 2026 12:46
@abrichr
abrichr merged commit 79f4dda into main Jul 19, 2026
14 checks passed
@abrichr
abrichr deleted the feat/win32-window-client branch July 19, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant