E53: with camera frames and a CNN, SB3 still ends on byte-identical weights through the bridge

experiments/e53-image-observations/FINDINGS.md

@@ -0,0 +1,64 @@
+# E53: with camera frames and a CNN, SB3 still ends on byte-identical weights through the bridge; 16 frames a step cost 2.6-3.0 ms
+
+2026-10-02 · guangzhao (bridges-venv: torch 2.14.1+cpu, SB3 2.9.0, gymnasium
+1.3.0, ale-py 0.12.1, opencv-python-headless 5.0.0.93; plugrl-bridges#3,
+803141d) · protocol: [`PROTOCOL.md`](PROTOCOL.md) (`32af99b`, after the pilot
+in `results/pilot.txt`, before the registered runs) · no amendment
+
+---
+
+## The result
+
+SB3's PPO with CnnPolicy trained 16 `AtariWrapper(ALE/Pong-v5)` environments
+for 40,960 steps, with 84x84 grayscale frames, stacked 4 deep. The two arms:
+- **inprocess:** in SB3's DummyVecEnv;
+- **bridge:** in another process, behind
+  `PlugRLVecEnv(image_key="frame")`, which receives each uint8 frame as
+  rendered.
+
+| seed | final weights (SHA-256) | episodes | in-process | bridge | per vector step |
+| --- | --- | --- | --- | --- | --- |
+| 0 | `82f80c02...` both | 183, identical | 67.4 s | 75.0 s (+11%) | +2.99 ms |
+| 1 | `60acba42...` both | 176, identical | 69.6 s | 76.2 s (+10%) | +2.60 ms |
+| 2 | `6bf04095...` both | 183, identical | 68.1 s | 75.8 s (+11%) | +2.98 ms |
+
+Every check and prediction holds (`results/verdicts.txt`):
+- **V1:** all 6 runs are complete. Each bridge run served one connection, and
+  its observation space was `Box(0, 255, (84, 84, 4), uint8)`, as in process.
+- **P1:** 3 of 3 seeds end on the same weights.
+- **P2:** both arms logged the same episodes, and the bridge's own episode
+  log agrees. (AtariWrapper ends an episode at each lost life.)
+- **P3:** the bridge adds 2.60-2.99 ms per vector step, inside 1.0-5.0.
+
+---
+
+## What it means
+
+- **The boundary stays transparent with images.** E50 showed this for state
+  vectors. Here the observations are camera frames: a uint8 image space, a
+  CNN, SB3's own transposition and scaling, and frame stacking on the
+  trainer's side. Nothing in that path changes when the frames cross a
+  process boundary.
+- **Images cost more to cross, in proportion to their bytes.** Each step
+  moves 16 frames of 7 KiB twice: in the feedback, and again in the next
+  infer, since this client does not use `reuse-feedback-obs`. That is about
+  2.8 ms on the loopback, against 0.4-0.6 ms for E50's states. It is still
+  10-11% of a short CNN run on CPU, and a far smaller share beside a VLA's
+  forward pass (E10).
+
+---
+
+## What E53 does not show
+
+* **Learning.** 40,960 steps is far too short to learn Pong. E53 compares
+  weights. E49 shows learning through the bridge, with state observations.
+* **A GPU.** The CNN ran on CPU with four threads, in both arms. On a GPU,
+  bit-identical comparisons need deterministic kernels.
+* **Two machines, or reuse.** E43 and E46 measured what frames cost across a
+  link, and how `reuse-feedback-obs` halves it.
+
+Kept here:
+- each run's `result.json`, `monitor.csv`, the bridge runs' `episodes.csv`,
+  and the logs;
+- `registered.out`, `load.txt`, `bridges-src.sha256` and `verdicts.txt`
+  (`summarise.py`).

experiments/e53-image-observations/PROTOCOL.md

@@ -0,0 +1,97 @@
+# E53 measurement protocol (pre-registered)
+
+**Written 2026-10-02, after the pilot in `results/pilot.txt`, before the
+registered runs.**
+
+This file must not be edited after the first registered data point. Anything
+learned afterwards goes in `AMENDMENT.md`, dated.
+
+---
+
+## The question
+
+E50 found the boundary transparent on one machine: SB3 ends on byte-identical
+weights with its environments in its own process or behind the bridge. But
+only state vectors crossed. Image policies are where the boundary matters
+most (VLAs; E43's and E46's costs), and a camera frame takes a different
+path: uint8 arrays, an image observation space, a CNN, frame stacking. So:
+
+**With image observations and a CNN policy, does SB3 still end on the same
+weights through the bridge? And what does crossing cost per step when every
+step carries 16 frames?**
+
+---
+
+## Declared in advance: what was already known
+
+1. **The code.**
+   - `train.py` follows E50's design. SB3 2.9.0 PPO runs CnnPolicy at
+     rl-baselines3-zoo's Atari settings, held constant: n_steps 128,
+     4 epochs, minibatch 256, lr 2.5e-4, clip 0.1, entropy 0.01, value 0.5.
+   - There are 16 `AtariWrapper(gym.make("ALE/Pong-v5"))` environments,
+     with VecMonitor and VecFrameStack(4) on both arms.
+   - The run is 40,960 steps on CPU, four torch threads.
+   - The bridge arm uses plugrl-bridges#3 (`803141d`):
+     `PlugRLVecEnv(image_key="frame")`.
+   - `atari_client.py` sends each 84x84x1 uint8 frame as rendered. It
+     applies the float action that crosses as an integer, and it seeds and
+     resets as DummyVecEnv does.
+2. **The versions.** ale-py 0.12.1 and opencv-python-headless 5.0.0.93 were
+   installed into bridges-venv for E53.
+3. **The pilot, seed 9** (`results/pilot.txt`).
+   - Both arms ended on `76f991b8...`, with the observation space
+     `Box(0, 255, (84, 84, 4), uint8)` in both.
+   - The arms took 68.3 s in process and 74.5 s through the bridge. That is
+     2.44 ms more per vector step (2,560 of them).
+   - An earlier attempt failed for want of OpenCV. Its stuck bridge process
+     then took a later attempt's client, and both were discarded
+     (`pilot.txt`).
+4. **This is far too short to learn Pong.** E53 compares weights, not
+   returns.
+
+---
+
+## Design
+
+On guangzhao, one run at a time, port 8830: seeds 0, 1 and 2, each with arms
+inprocess and bridge. That is 6 runs (`run.sh`). The port is checked free
+before each run.
+
+---
+
+## Checks
+
+* **V1 - complete.** Every run has 40,960 steps. Every bridge run's client
+  exited 0, its log has one `plugrl-bridges: slots 0-15` line, and its
+  observation space is uint8.
+
+---
+
+## Predictions, and what falsifies each
+
+**P1 - the boundary leaves the weights alone, with images too.** For each
+seed, the two arms' weights are equal: 3 of 3.
+
+> Grounds: the pilot, and E50. Nothing in the data path differs. The frames
+> are the same uint8 arrays, and SB3 applies its own image preprocessing
+> (transpose, /255) in both arms alike.
+
+**P2 - and the episodes.** For each seed, the two `monitor.csv` files list
+the same episodes, and the bridge's episode log agrees with the bridge
+arm's.
+
+**P3 - the price of 16 frames a step, on one machine.** The bridge arm takes
+1.0 to 5.0 ms more per vector step than its in-process pair, on every seed.
+
+> Grounds: the pilot's 2.44 ms. E50's state-only steps cost 0.37-0.58 ms.
+> Each step now moves 16 x 7 KiB of frames twice, in the feedback and again
+> in the next infer, since this client does not reuse observations.
+
+**Reported, not predicted:** each run's learn time and steps per second, and
+the overhead as a fraction of the in-process time.
+
+---
+
+## Reading order
+
+V1, P1, P2, P3, then the reported figures.

experiments/e53-image-observations/atari_client.py

@@ -0,0 +1,128 @@
+"""E53's env client: N Atari environments behind SB3's AtariWrapper, on one connection.
+
+    python atari_client.py --port 8830 --num-envs 16 --seed 0 [--env-id ALE/Pong-v5]
+
+E50's gym_client.py, with three changes:
+- Each env is `AtariWrapper(gym.make(env_id))`, as SB3 preprocesses Atari:
+  no-op starts, frame skip 4, 84x84 grayscale, a life lost ends the episode,
+  and rewards clipped to their sign.
+- The observation it sends is that frame, a uint8 [n, 84, 84, 1] image
+  named `frame`, with no states.
+- The action arrives as one float per env (the protocol's actions are
+  float), and is applied as the integer it is.
+
+Env i is reset with seed + i first and with no seed afterwards, as SB3's
+DummyVecEnv does after `VecEnv.seed(seed)`. An env whose episode ended is
+reset before the next infer, as DummyVecEnv resets it inside the step.
+"""
+
+from __future__ import annotations
+
+import argparse
+import time
+
+import ale_py
+import gymnasium as gym
+import numpy as np
+import websockets.exceptions
+import websockets.sync.client
+from stable_baselines3.common.atari_wrappers import AtariWrapper
+
+from plugrl_protocol import msgpack_numpy
+
+gym.register_envs(ale_py)
+STOP_REASON = "plugrl-server-stop"
+
+
+def observation(frames: np.ndarray) -> dict:
+    return {"images": {"frame": frames}, "states": {}, "text": "atari"}
+
+
+def connect(uri: str, deadline: float):
+    while True:
+        try:
+            return websockets.sync.client.connect(
+                uri, compression=None, max_size=None, open_timeout=10
+            )
+        except (OSError, websockets.exceptions.InvalidHandshake):
+            if time.monotonic() > deadline:
+                raise
+            time.sleep(0.1)
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument("--host", default="127.0.0.1")
+    p.add_argument("--port", type=int, default=8830)
+    p.add_argument("--num-envs", type=int, default=16)
+    p.add_argument("--seed", type=int, default=0)
+    p.add_argument("--env-id", default="ALE/Pong-v5")
+    a = p.parse_args()
+
+    n = a.num_envs
+    envs = [AtariWrapper(gym.make(a.env_id)) for _ in range(n)]
+    obs = np.stack([env.reset(seed=a.seed + i)[0] for i, env in enumerate(envs)])
+    indices = np.arange(n, dtype=np.int64)
+    step_ids = np.zeros(n, dtype=np.int64)
+    packer = msgpack_numpy.Packer()
+    steps = 0
+
+    ws = connect(f"ws://{a.host}:{a.port}", time.monotonic() + 120)
+    with ws:
+        msgpack_numpy.unpackb(ws.recv())  # the metadata; nothing in it is needed
+        try:
+            while True:
+                ws.send(
+                    packer.pack(
+                        {
+                            "message_type": "infer",
+                            "data": observation(obs),
+                            "env_indices": indices,
+                            "step_ids": step_ids.copy(),
+                        }
+                    )
+                )
+                reply = msgpack_numpy.unpackb(ws.recv())
+                actions = np.asarray(reply["data"]["action"])[0]  # [H, n, 1] -> [n, 1]
+                ended = np.empty_like(obs)
+                after = np.empty_like(obs)
+                rewards = np.zeros(n, dtype=np.float32)
+                terminated = np.zeros(n, dtype=np.bool_)
+                truncated = np.zeros(n, dtype=np.bool_)
+                sent_ids = step_ids.copy()
+                for i, env in enumerate(envs):
+                    o, r, te, tr, _ = env.step(int(round(float(actions[i][0]))))
+                    ended[i] = o
+                    rewards[i], terminated[i], truncated[i] = r, te, tr
+                    if te or tr:
+                        after[i] = env.reset()[0]
+                        step_ids[i] = 0
+                    else:
+                        after[i] = o
+                        step_ids[i] += 1
+                ws.send(
+                    packer.pack(
+                        {
+                            "message_type": "feedback",
+                            "env_indices": indices,
+                            "step_ids": sent_ids,
+                            "data": {
+                                "obs": observation(ended),
+                                "rewards": rewards,
+                                "terminated": terminated,
+                                "truncated": truncated,
+                                "info": {},
+                            },
+                        }
+                    )
+                )
+                obs = after
+                steps += 1
+        except websockets.exceptions.ConnectionClosed as closed:
+            reason = closed.rcvd.reason if closed.rcvd is not None else ""
+            print(f"closed after {steps} steps: {reason!r}", flush=True)
+            return 0 if reason == STOP_REASON else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

experiments/e53-image-observations/results/bridges-src.sha256

@@ -0,0 +1,6 @@
+766ad3d83be9696dff26543ce2fb35c3c31cb845fc42cb148d411b263e8bf9b5  src/plugrl_bridges/gym.py
+e89ebb2c4ed69ed5788cd945bc4d36ecc023949627433818b98d9ea43d1fa7d5  src/plugrl_bridges/__init__.py
+4001b6400d4031504525949fa721ad425fa0cb16a91cf30537dfd6d5859d8336  src/plugrl_bridges/observations.py
+1c22fca6e0f01e8ec730ee893568cc301d2d21c3e16cc2102c22959ced00ff08  src/plugrl_bridges/rlinf.py
+f02264e551925eed72ba2e4dfb5159f7e174e4ac7231aa122689ef2172484646  src/plugrl_bridges/sb3.py
+0e56388fea17a73d4defcb437d3a4b6c8392c609dfca07efe92343e6cac72656  src/plugrl_bridges/server.py

experiments/e53-image-observations/results/load.txt

@@ -0,0 +1,2 @@
+ 09:56:03 up 27 days, 22:57,  1 user,  load average: 0.96, 1.31, 0.93
+ 10:03:32 up 27 days, 23:05,  1 user,  load average: 3.27, 3.00, 1.94

experiments/e53-image-observations/results/pilot.txt

@@ -0,0 +1,23 @@
+E53 pilot, 2026-10-02, seed 9 (not a registered seed), guangzhao, bridges-venv
+(torch 2.14.1+cpu, SB3 2.9.0, gymnasium 1.3.0, ale-py 0.12.1, then
+opencv-python-headless 5.0.0.93), plugrl-bridges#3 (803141d).
+
+First attempt: both arms failed. SB3's AtariWrapper needs OpenCV to resize
+frames, and bridges-venv had none; the client died with "OpenCV is not
+installed". OpenCV was installed (opencv-python-headless 5.0.0.93).
+
+Second attempt:
+  inprocess  76f991b80e896bd88068d6e1507bf05845710580b40a05cb375b5dcf622edddc  68.284 s  599.8 steps/s
+  bridge     failed to bind port 8830: the first attempt's bridge arm was still
+             running, waiting (timeout 600 s) for the client that had died. That
+             stuck process then accepted the second attempt's client. Both
+             processes were killed and their output deleted, unread.
+
+Third attempt, bridge arm only, port 8830 checked free first:
+  bridge     76f991b80e896bd88068d6e1507bf05845710580b40a05cb375b5dcf622edddc  74.524 s  549.6 steps/s  client rc 0
+  observation space in both: Box(0, 255, (84, 84, 4), uint8)
+
+Same weights. (74.524 - 68.284) / 2560 = 2.44 ms more per vector step.
+
+Before the registered runs: run.sh checks that port 8830 is free before every
+run. Nothing else was changed.

experiments/e53-image-observations/results/pong-bridge-seed0.log

@@ -0,0 +1,4 @@
+A.L.E: Arcade Learning Environment (version 0.12.1+8a8fafb)
+[Powered by Stella]
+plugrl-bridges: slots 0-15 <- ('127.0.0.1', 34864)
+{"env": "ALE/Pong-v5", "arm": "bridge", "seed": 0, "weights_sha256": "82f80c02616bc24c4f2baa984af8162e0f94532e1f4c3a3ff49fb977ae1b489f", "learn_s": 75.04, "steps": 40960, "steps_per_s": 545.8, "observation_space": "Box(0, 255, (84, 84, 4), uint8)", "torch": "2.14.1+cpu", "client_rc": 0}

experiments/e53-image-observations/results/pong-bridge-seed0/client.log

@@ -0,0 +1,3 @@
+A.L.E: Arcade Learning Environment (version 0.12.1+8a8fafb)
+[Powered by Stella]
+closed after 2560 steps: 'plugrl-server-stop'