feat(backends): win32 WindowClient for remote-display window replay#159
Merged
Conversation
Windows hosts can now replay into a specific client window's own pixel space (the Citrix/RDP client-window wedge for Windows-endpoint clinics): - new openadapt_flow/backends/win32_window_client.py: EnumWindows selection (process-executable owner matching per capture 0.6.0, exact case-insensitive title, optional class filter; zero/multiple matches surface to the backend's ambiguity halt), client-area capture in the window's own pixel space (PrintWindow PW_CLIENTONLY|RENDERFULLCONTENT first, BitBlt fallback), per-monitor-v2 DPI awareness (refuses DPI virtualization), SendInput injection with foreground verification and a UIPI elevation guard, and typed failure modes that map onto the existing RemoteDisplayError halt semantics. Stdlib ctypes only behind a mockable Win32Api seam; import-safe on every platform. - remote_display: default client is host-dispatched (darwin/win32); key-chord resolution moved behind the client seam (resolve_key, mac semantics byte-identical); input-trust refusal message made platform-neutral. - 30 mock-contract tests (selection/ambiguity, coordinate + DPI math, focus-verification halts, capture fallback ordering, UIPI refusal); wired into the post-merge windows-mock CI job. - claims: win32-window-replay-roadmap at roadmap tier (mock-tested only, real-Windows qualification explicitly out of scope); VERIFICATION regenerated; CITRIX_PIXEL.md notes the win32 client. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
abrichr
force-pushed
the
feat/win32-window-client
branch
from
July 19, 2026 12:46
abea7d5 to
7b05ebb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements the win32
WindowClientforRemoteDisplayBackend— the largest single flow gap in the multi-substrate design: window-scoped remote-display replay was macOS-host-only, so a Windows endpoint (the wedge clinic's actual front-desk machine) could record a Citrix/RDP client window (capture 0.6.0 win32) but not replay into it. This PR closes the replay half on Windows hosts with the same client seam, same contract, same fail-loud discipline as the macOS client.New module:
openadapt_flow/backends/win32_window_client.pyEnumWindowswith capture-0.6.0-parity owner matching (owning process executable basename, e.g.wfica32.exefor Citrix Workspace; exact case-insensitive,.exe-tolerant, never substring), exact case-insensitive title, optional exact window-class filter (expected_class, for dynamic-title targets), cloaked/untitled/invisible windows skipped. Zero or multiple matches surface to the backend's existing ambiguity halt — the client never "picks the largest".GetClientRect+ClientToScreen) asWindowInfo.bounds;PrintWindow(PW_CLIENTONLY | PW_RENDERFULLCONTENT)first,BitBltof the client DC as fallback; both failing, window-gone, and window-minimized are typed errors (CaptureFailedError/WindowGoneError/WindowMinimizedError, all subclasses ofRemoteDisplayError→ existing halt semantics).SetProcessDpiAwarenessContext(PER_MONITOR_AWARE_V2), with shcore /SetProcessDPIAwarefallbacks). Under per-monitor awareness, client rect, screen coordinates, andSendInputall speak physical pixels, so the backend's derived scale is 1.0 by construction. If only system/unaware can be achieved, the client refuses to capture or inject (DpiAwarenessError) — virtualized coordinates can mis-target a click.SendInput(absolute virtual-desktop normalization with the 65535/(w−1) mapping), hardware-like VK+scancode key events (extended-key flags for the nav cluster), layout-resolved typing viaVkKeyScanWwith a documented Unicode fallback for AltGr/unmapped chars, real modifier transitions wrapped around chord keys (released in reverse order; the backend'sfinallyguarantees no latched modifier).key_window_idrequires the exact target HWND to hold the foreground (GetForegroundWindowroot) — the backend's before/after focus proofs therefore halt on any wrong-window state, andwindow_at_point(WindowFromPoint→GA_ROOT) backs the occlusion refusal.input_trusted() == Falseand every input method independently refuses (InputDeliveryError).SendInput's injected-event count is additionally verified on the native path.ctypes.windll; onlyNativeWin32Api()construction does (and raises a typed error off-Windows). Stdlibctypesonly — no pywin32 (the repo has no win32 dependency and keeps it that way).Backend integration (
remote_display.py, minimal and behavior-preserving on macOS):darwin→MacWindowClient,win32→Win32WindowClient, others fail loud at construction instead of failing later inside Quartz imports).WindowClient.resolve_key(token) -> (keycode, needs_shift) | None.MacWindowClientresolves via the existing mac tables (resolve_mac_key, byte-identical semantics — the existing remote-display suite still asserts the same mac keycodes); the win32 client resolves Windows VKs + the active layout. The backend no longer owns a platform keycode table.factory.py: docstring only (window_clientdefault is the host's native client).Tests (mock-only, honestly labeled)
tests/test_win32_window_client.py— 30 contract tests, the entire Win32 layer faked behind theWin32Apiseam (runs identically on any platform; also wired into the post-mergewindows-mockjob so it runs on a real Windows interpreter, still mocked):.exe, substring refused), exact CI title, class filter, cloaked/untitled/invisible skipping, duplicate-title ambiguity halt through the backend;RemoteDisplayBackend.click(origin offset asserted),normalize_to_virtual_desktopincl. the (w−1) off-by-one, negative-origin multi-monitor, and out-of-desktop refusal; DPI virtualization (system/unaware) refused;resolve_keynamed-table-then-layout (macOS chord parity),press()end-to-end through the backend emitting Windows VKs;NativeWin32Api()refuses non-Windows hosts; the module imports everywhere.Full non-E2E suite + ruff (lint + format) + mypy green; claims gate green;
docs/VERIFICATION.md/.jsonregenerated.Mocked vs real — what this PR does and does not prove
Proven (CI): the win32 client conforms to the
WindowClientseam contract the macOS client satisfies — selection/ambiguity, mapping math, refusal paths, fallback ordering — under a scripted Win32 API.NOT proven: anything about a real Windows host. Real
PrintWindowrendering quirks, real DPI virtualization, real foreground arbitration (SetForegroundWindowrestrictions), real UIPI filtering, and realSendInputdelivery are exercised by zero tests here. The new claims entrywin32-window-replay-roadmapis deliberately tier: roadmap with a caveat saying exactly this; no supported/validating language was added anywhere. A real-Windows counted qualification is explicitly out of scope for this PR.Validation procedure (for the lane owner — real Windows, later)
Environment: the Parallels "Windows 11" VM (snapshot first via
prlctl snapshot; never delete the user's VM), or the internal cloud Windows runner image. Python 3.10–3.12 inside the guest.windows-mockCI job now runs this suite onwindows-latest— confirms the mocked contract on a real Windows interpreter.pip install -e .[dev]; open Notepad; driveRemoteDisplayBackendat it (run --backend rdpwithrdp_window: notepadin the deployment YAML, or construct directly withowner_substr="notepad"). Verify:screenshot()returns the client-area PNG with_scale == 1.0;ensure_foreground()proof passes;click/type_textland in Notepad; minimizing ⇒WindowMinimizedError; overlapping another window over the click point ⇒ occlusion refusal; targeting an elevated app (e.g. admin Task Manager) ⇒ UIPI refusal, not a silent no-op.mstsc.execonnected to a second Windows session as the remote-display client window. Record window-scoped with capture 0.6.0 (RecordingConfig(window_owner="mstsc", window_title=…)),convert_capture→ compile →run --backend rdp(rdp_window: mstsc) with an external effect verifier (--effects-kind fileordocument-hash), N≥20 runs; count success/halt/wrong-action. Publish under the label "remote-display client window (RDP analog) on a Windows host" — never as Citrix evidence (the guard tests forbid the substitution).win32-window-replay-roadmapmove roadmap → validating (analog label); Citrix-labeled evidence remains design-partner-gated.Files
openadapt_flow/backends/win32_window_client.py— new (client +Win32Apiseam +NativeWin32Apictypes bindings)openadapt_flow/backends/remote_display.py— host-dispatched default client;resolve_keyseam; platform-neutral trust messageopenadapt_flow/backends/factory.py— docstringtests/test_win32_window_client.py— new (30 mock-contract tests)tests/test_remote_display_backend.py,tests/test_backend_factory.py— fakes gainresolve_key.github/workflows/ci.yml— windows-mock job runs the new suite + the remote-display suiteclaims.yaml+docs/VERIFICATION.md+docs/verification.json— roadmap-tier claim, regenerated reportdocs/desktop/CITRIX_PIXEL.md— notes the win32 client (mock-tested, unqualified)🤖 Generated with Claude Code