feat: PyVNCServer 3.3 — native DXGI metadata and encoding interoperability

.github/workflows/ci.yml

@@ -55,6 +55,31 @@ jobs:
       - name: Run test suite
         run: python -m pytest -q
 
+  dxgi-contract:
+    name: DXGI contract · Windows · Python 3.13
+    runs-on: windows-latest
+    timeout-minutes: 15
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v7
+
+      - name: Set up Python
+        uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+          cache: pip
+          cache-dependency-path: pyproject.toml
+
+      - name: Upgrade packaging tools
+        run: python -m pip install --upgrade pip setuptools wheel
+
+      - name: Install performance and test dependencies
+        run: python -m pip install -e ".[dev,performance]"
+
+      - name: Verify DXCam private integration contract
+        run: python -m pytest -q tests/test_dxgi_dxcam_contract.py tests/test_dxgi_metadata.py
+
   coverage:
     name: Coverage
     runs-on: ubuntu-latest
@@ -91,7 +116,7 @@ jobs:
 
   build:
     name: Build package
-    needs: test
+    needs: [test, dxgi-contract]
     runs-on: ubuntu-latest
     timeout-minutes: 15
 

benchmarks/benchmark_dxgi_metadata.py

@@ -0,0 +1,107 @@
+"""Inspect DXGI dirty/move metadata quality on a real Windows desktop.
+
+Run on Windows with the performance extra installed:
+
+    python -m pip install -e ".[performance]"
+    python benchmarks/benchmark_dxgi_metadata.py --frames 300 --fps 60
+
+The benchmark is diagnostic rather than a synthetic score. It reports how
+often native Desktop Duplication metadata was available, rectangle counts and
+the approximate fraction of the framebuffer affected by each frame.
+"""
+
+from __future__ import annotations
+
+import argparse
+import statistics
+import time
+
+from vnc_lib.screen_capture import ScreenCapture
+
+
+NATIVE_BGR0 = {
+    "bits_per_pixel": 32,
+    "depth": 24,
+    "big_endian_flag": 0,
+    "true_colour_flag": 1,
+    "red_max": 255,
+    "green_max": 255,
+    "blue_max": 255,
+    "red_shift": 16,
+    "green_shift": 8,
+    "blue_shift": 0,
+}
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Benchmark DXGI dirty/move metadata")
+    parser.add_argument("--frames", type=int, default=300)
+    parser.add_argument("--fps", type=float, default=60.0)
+    args = parser.parse_args()
+
+    frames = max(1, args.frames)
+    fps = max(1.0, args.fps)
+    interval = 1.0 / fps
+    capture = ScreenCapture(backend_preference="dxcam")
+    if capture.get_backend_name() != "dxcam":
+        raise SystemExit(
+            "DXCam backend is unavailable; install the performance extra on Windows"
+        )
+
+    capture.set_cache_frame_rate(fps)
+    native_frames = 0
+    fallback_frames = 0
+    dirty_counts: list[int] = []
+    move_counts: list[int] = []
+    changed_ratios: list[float] = []
+    capture_ms: list[float] = []
+
+    next_tick = time.perf_counter()
+    try:
+        for _ in range(frames):
+            frame = capture.capture_frame(NATIVE_BGR0)
+            result = frame.result
+            metadata = frame.metadata
+            if result.pixel_data is None:
+                continue
+
+            capture_ms.append(result.capture_time * 1000.0)
+            if metadata.dirty_regions is None:
+                fallback_frames += 1
+            else:
+                native_frames += 1
+                dirty_counts.append(len(metadata.dirty_regions))
+                move_counts.append(len(metadata.move_rects))
+                changed_area = sum(w * h for _, _, w, h in metadata.dirty_regions)
+                changed_ratios.append(
+                    min(1.0, changed_area / max(1, result.width * result.height))
+                )
+
+            next_tick += interval
+            delay = next_tick - time.perf_counter()
+            if delay > 0:
+                time.sleep(delay)
+            else:
+                next_tick = time.perf_counter()
+    finally:
+        capture.close_current_thread_sessions()
+
+    total = native_frames + fallback_frames
+    print(f"backend:             {capture.get_backend_name()}")
+    print(f"frames sampled:      {total}")
+    print(f"native metadata:     {native_frames} ({native_frames / max(1, total):.1%})")
+    print(f"metadata fallbacks:  {fallback_frames}")
+    if capture_ms:
+        ordered = sorted(capture_ms)
+        p95 = ordered[int((len(ordered) - 1) * 0.95)]
+        print(f"capture avg:         {statistics.mean(capture_ms):.2f} ms")
+        print(f"capture p95:         {p95:.2f} ms")
+    if dirty_counts:
+        print(f"dirty rects avg:     {statistics.mean(dirty_counts):.2f}")
+        print(f"move rects avg:      {statistics.mean(move_counts):.2f}")
+        print(f"changed area avg:    {statistics.mean(changed_ratios):.2%}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

docs/performance.md

@@ -8,7 +8,7 @@ PyVNCServer optimizes the path from desktop capture to encoded rectangle rather
 
 | Backend | Platform | Role |
 | --- | --- | --- |
-| DXCam / DXGI | Windows | optional fast Desktop Duplication capture path |
+| DXCam / DXGI | Windows | optional fast Desktop Duplication capture path with native dirty/move metadata in v3.3 |
 | MSS | cross-platform | portable primary fallback |
 | Pillow ImageGrab | platform dependent | fallback capture path |
 
@@ -18,16 +18,29 @@ Install the performance extras:
 python -m pip install -e ".[performance]"
 ```
 
+PyVNCServer 3.3 requires DXCam `0.3.0+` for the native metadata integration.
+
 ## Shared capture producer
 
 A server-wide producer captures once and distributes framebuffer generations to sessions. This avoids N clients causing N independent desktop captures.
 
-## Changed regions
+## Native DXGI changed regions
+
+On an unscaled, unrotated full-output DXCam capture, v3.3 reads Desktop Duplication metadata directly from the active `IDXGIOutputDuplication` object:
+
+- `GetFrameDirtyRects` identifies framebuffer regions whose pixel contents changed;
+- `GetFrameMoveRects` identifies regions copied from another framebuffer location and exposes them as CopyRect hints;
+- a DXGI timeout/no-new-frame is treated as an authoritative empty update;
+- any metadata read/validation failure returns to the existing software change detector instead of assuming the screen is unchanged.
 
-Incremental requests benefit from limiting work to changed regions. When a backend cannot provide native dirty metadata, PyVNCServer performs change detection above the backend.
+Native metadata is deliberately disabled for scaled, rotated or cropped DXCam captures until coordinate translation is implemented. MSS and Pillow also continue to use software change detection.
 
-!!! note
-    Native DXGI dirty/move rectangle harvesting is not yet implemented in the current DXCam integration. CopyRect/dirty-region opportunities therefore depend on metadata available to the higher layers.
+!!! important
+    Native metadata is an optimization, never a correctness requirement. An ambiguous native result becomes `dirty_regions = None`, which explicitly activates the software differ.
+
+### Move-rectangle safety in v3.3
+
+Move destinations are currently included in the dirty pixel list even when a CopyRect hint is emitted. This is intentionally conservative: clients without CopyRect support and clients that skip capture generations still converge to the correct framebuffer. Once the real-client interoperability matrix has broader coverage, the redundant pixel update can be removed for clients that are exactly one generation behind and advertise CopyRect.
 
 ## Request coalescing
 
@@ -59,16 +72,34 @@ Relevant LAN settings include zlib/ZRLE compression levels, raw thresholds and J
 
 ## Benchmarks
 
+Encoder/capture benchmarks:
+
 ```bash
 PYTHONPATH=src python benchmarks/benchmark_encoders.py
 PYTHONPATH=src python benchmarks/benchmark_screen_capture.py
 PYTHONPATH=src python benchmarks/benchmark_screen_capture_methods.py
 PYTHONPATH=src python benchmarks/benchmark_lan_latency.py
 ```
 
+### DXGI metadata benchmark
+
+On a real Windows desktop with the performance extra installed:
+
+```powershell
+python benchmarks/benchmark_dxgi_metadata.py --frames 300 --fps 60
+```
+
+The diagnostic reports:
+
+- native metadata hit rate vs software-diff fallback rate;
+- average and p95 capture time;
+- average dirty/move rectangle count;
+- approximate changed framebuffer area.
+
 For meaningful numbers:
 
 - benchmark on the target OS/GPU/display setup;
+- test idle desktop, text editing, window movement, scrolling and video separately;
 - separate capture time from encode time;
 - test full-screen and small-region updates;
 - include the actual viewer over the intended network path;

docs/ultravnc.md

@@ -51,6 +51,56 @@ These are useful control encodings when isolating a compatibility problem:
 - **Zlib**: compressed full rectangles;
 - **ZRLE**: tiled zlib/RLE, usually a good general-purpose option.
 
+## v3.3 interoperability regression suite
+
+Version 3.3 adds a real TCP/RFB regression test that keeps one connection open while changing the advertised encoding set. The test parses the actual rectangle payload rather than only checking that the socket remained connected.
+
+The automated sequence is:
+
+```text
+Raw → RRE → Hextile → Zlib → ZRLE → Tight → Raw
+```
+
+For each step it verifies the rectangle header encoding ID and the corresponding wire format. This specifically guards against failures such as advertising RRE while sending Raw bytes, stale zlib state after an encoding switch, or a connection reset caused by stream desynchronization.
+
+This test is not a substitute for running the UltraVNC binary itself: client-specific decoder behavior still needs validation before a release. It does, however, make the protocol-level failure modes reproducible in Linux and Windows CI.
+
+## Real UltraVNC Viewer smoke test
+
+UltraVNC Viewer exposes a command-line `-encoding` option, so v3.3 also ships a PowerShell smoke-test for the actual `vncviewer.exe` binary.
+
+Start PyVNCServer in one terminal:
+
+```powershell
+pyvncserver serve --log-level DEBUG
+```
+
+Then run in another PowerShell window:
+
+```powershell
+.\scripts\ultravnc_smoke.ps1 `
+  -ViewerPath 'C:\Program Files\uvnc bvba\UltraVNC\vncviewer.exe'
+```
+
+The default sequence is:
+
+```text
+raw → rre → hextile → zlib → tight
+```
+
+For every encoding the script launches a fresh view-only UltraVNC process, keeps it connected for a short observation period and treats an early viewer exit as failure. Viewer logs are written under `%TEMP%\pyvncserver-ultravnc`.
+
+A longer run is useful when validating a release candidate:
+
+```powershell
+.\scripts\ultravnc_smoke.ps1 `
+  -SecondsPerEncoding 10 `
+  -Encodings raw,rre,hextile,zlib,tight
+```
+
+!!! note
+    The script proves that the real UltraVNC process can negotiate and keep the session alive. Visual correctness should still be checked while moving windows, scrolling and changing screen content, especially for Tight and RRE.
+
 ## What to log
 
 Run:
@@ -68,9 +118,9 @@ Useful lines include:
 - selected encoding per update/region;
 - socket reset/timeout messages.
 
-## Suggested compatibility matrix
+## Suggested manual compatibility matrix
 
-After a code change to an encoder, test a single connection while switching in this order:
+After a code change to an encoder, test a single UltraVNC connection while switching in this order:
 
 ```text
 Raw → Hextile → Zlib → ZRLE → Tight → RRE → Auto

pyproject.toml

@@ -18,7 +18,7 @@ dependencies = [
 [project.optional-dependencies]
 performance = [
     "numpy>=1.26",
-    "dxcam>=0.0.5; platform_system == 'Windows'",
+    "dxcam>=0.3.0; platform_system == 'Windows'",
 ]
 h264 = ["av>=12.0"]
 dev = ["pytest>=8.0"]