E48: RLinf, through the bridge, trains environments that PlugRL env clients run outside its cluster

experiments/e48-rlinf-bridge/AMENDMENT.md

@@ -0,0 +1,52 @@
+# E48 amendment
+
+**2026-10-02 00:55, during the cross arm. Cross seed 0 had finished on
+guangzhao; its returns had not been looked at.**
+
+## What happened
+
+`run_e48.sh` hands `cross_clients.py` the run's directory on guangzhao,
+`/home/guangzhao/zuogou/plugrl/e48/cross-seed<s>`. Git Bash converts an
+argument that looks like a POSIX path when it starts a Windows program, so
+the script received `D:/Program Files/Git/home/guangzhao/...`. Its poll then
+ran `grep -q RUN_DONE` on a path that does not exist on guangzhao, every 10 s,
+and never saw the run end.
+
+Meanwhile RLinf's run of cross seed 0 ended at 00:26:24 with rc 0. RLinf
+does not close its environments when it ends: the bridge's connection dropped
+with no close frame, so the bridge never sent `plugrl-server-stop`. Following
+SPEC.md, the laptop's 16 clients took the drop for a lost connection and
+kept reconnecting, 3,190 attempts by 00:51. Seed 1 could not start, since
+`run_e48.sh` waits for `cross_clients.py`.
+
+## What was done
+
+`tmp/e48_stop_watcher.sh` (copied to `results/stop-watcher.sh`) does what the
+poll was meant to do. For each cross seed it waits for RUN_DONE in that run's
+`run.txt` on guangzhao, then stops that seed's client tree on the laptop.
+`cross_clients.py` then returns as designed, and `run_e48.sh` carries on.
+Its log is `results/stop-watcher.txt`. Seed 0's clients were stopped at
+00:53:14, and seed 1 started at 00:53:18.
+
+`run_e48.sh` and `cross_clients.py` were not edited while they ran.
+
+## What it changes
+
+- **Nothing that is judged.** Each RLinf run had finished before its clients
+  were stopped. The watcher only ends clients that are idle and reconnecting.
+  V1, V2, P1 and P2 read RLinf's log and TensorBoard events, which are
+  complete by then.
+- **The wall clock between seeds.** It includes seed 0's 27 idle minutes,
+  which are reported as such.
+- **`clients exit` in `e48.txt` is 1 for watcher-stopped seeds**, because
+  `taskkill /F` ends the tree. The clients' own log shows the reconnecting
+  that preceded it.
+
+## Found, for later
+
+1. `cross_clients.py` should get the remote path in a form Git Bash leaves
+   alone: `MSYS_NO_PATHCONV=1`, or a path it does not convert.
+2. A trainer that ends without closing its environments leaves remote clients
+   reconnecting forever. The bridge could send stop from an exit handler,
+   though that does not help when the process is killed. Alternatively the
+   env client could give up after a bounded time with no server.

experiments/e48-rlinf-bridge/FINDINGS.md

@@ -0,0 +1,105 @@
+# E48: RLinf, through the bridge, trains environments that PlugRL env clients run outside its cluster - on its own machine and on a laptop with no torch, Ray or RLinf
+
+2026-10-01/02 · RLinf on guangzhao (one RTX 5090); env clients on guangzhao
+(local) or on the E43 laptop over Tailscale (cross) · protocol:
+[`PROTOCOL.md`](PROTOCOL.md) (`c64ad7c`, after the pilot in
+`results/pilot.txt`, before the registered runs) · amendment:
+[`AMENDMENT.md`](AMENDMENT.md) (an orchestration fault in the cross arm; no
+judged figure is affected)
+
+---
+
+## The result
+
+RLinf's PPO-MLP, through plugrl-rlinf's bridge (fca69db), trained
+HalfCheetah-v5 run by 16 plugrl-env-client processes: 100 epochs of 4,096
+frames, 409,600 steps. Mean `env/return` of the first episode batch against
+the mean of the last five:
+
+| seed | local | cross |
+| --- | --- | --- |
+| 0 | -284 -> 449 (**+733**) | -274 -> 450 (**+724**) |
+| 1 | -290 -> 485 (**+775**) | -274 -> 463 (**+737**) |
+| 2 | -268 -> 481 (**+749**) | -244 -> 496 (**+740**) |
+
+Every check and prediction holds (`results/verdicts.txt`):
+- **V1:** each run's log has 16 `plugrl-rlinf: slots` lines. They came from
+  127.0.0.1 in the local arm, and from the laptop's Tailscale address,
+  100.65.224.34, in the cross arm.
+- **V2:** every run logged 100 epochs, with no Traceback, ClientLost or
+  TimeoutError.
+- **P1:** the local arm learns on 3 of 3 seeds, against a bar of +200.
+- **P2:** the cross arm learns on 3 of 3 seeds.
+
+The laptop's side was a 276 MB venv with mujoco and plugrl-env-client, and no
+torch, Ray or RLinf. RLinf ran on guangzhao as it always does. Its
+environment worker reached the 16 environments only through the PlugRL
+protocol.
+
+---
+
+## What it cost
+
+| | local | cross |
+| --- | --- | --- |
+| median s per epoch | 3.71, 3.69, 3.69 | 23.68, 24.62, 22.42 |
+| run, start to RLinf's exit | 6.5, 6.4, 6.4 min | 41.1, 50.3, 55.9 min |
+| Tailscale ping before / after | | 46/59, 44/48, 50/42 ms |
+
+Crossing made an epoch 6.4x slower. The step itself is unchanged; the cost is
+the exchange. Each epoch is 256 lockstep steps, and each step waits for all 16
+clients across a 42-59 ms link. That is about 90 ms per step, against 14 ms
+locally. The run times spread more than the medians do: some epochs were much
+slower than the median. This is the same shared Wi-Fi link that E43 found
+does not hold a constant cost.
+
+The bridge is lockstep. A step finishes when its slowest client has fed back.
+That is why the crossing costs more here than in E43, where plugrl-server
+answers each client as soon as it can.
+
+---
+
+## What it means
+
+The bridge does what it was built to do. A trainer whose environments must
+normally run inside its own Ray cluster, in its own Python, trained
+environments that ran on a machine that has none of that. The learning curves
+of the two arms overlap seed for seed. Where the environment is stepped
+changes the time a step takes, not what it produces.
+
+---
+
+## What the run found, beyond its predictions
+
+1. **RLinf never closes its environments when a run ends**
+   (`rlinf/workers/env/env_worker.py` calls no `close()`). In its own cluster
+   that costs nothing. Clients outside it see the connection drop with no
+   close frame, never get `plugrl-server-stop`, and reconnect for good, as
+   SPEC.md tells them to: 3,190 attempts in 25 minutes after cross seed 0
+   ended. The local arm never showed this, because `run_guangzhao.sh` kills
+   its clients when RLinf exits.
+2. **Git Bash rewrites POSIX paths passed to Windows programs.** The cross
+   arm's poll looked for `D:/Program Files/Git/home/...` on guangzhao and
+   never saw a run end. `AMENDMENT.md` records how the watcher stood in for
+   it. plugrl-bridges' `cross_clients.py` now refuses such a path.
+
+---
+
+## What E48 does not show
+
+* **Any trainer but RLinf, or any environment but HalfCheetah.** E49 runs
+  SB3 and CleanRL, and the C++ Pendulum client.
+* **Images through the bridge.** The observations are 17 floats.
+* **Identical trajectories across machines.** The two machines' MuJoCo
+  builds differ (E43), so the arms are compared by the status rule.
+* **A remote client that survives the run's end.** See item 1 above. The fix
+  belongs in the bridge, as a stop on exit, or in the client, as a bounded
+  wait for a server.
+
+Checkpoints stay on guangzhao, in `~/zuogou/plugrl/e48/<run>/rlinf-logs/`.
+Kept here:
+- each run's `run.txt`, `rlinf.log` and TensorBoard events;
+- the clients' logs;
+- `e48.txt` and `run_e48.out`;
+- `stop-watcher.sh` and `stop-watcher.txt`;
+- `pilot.txt` and `verdicts.txt` (`summarise.py`).

experiments/e48-rlinf-bridge/PROTOCOL.md

@@ -0,0 +1,119 @@
+# E48 measurement protocol (pre-registered)
+
+**Written 2026-10-01, after the pilot in `results/pilot.txt`, before the
+registered runs.**
+
+This file must not be edited after the first registered data point. Anything
+learned afterwards goes in `AMENDMENT.md`, dated.
+
+---
+
+## The question
+
+Every PlugRL result so far was trained by plugrl-server. The project's
+claim is about the protocol: an environment that speaks it can be trained
+by any trainer that does. RLinf is a second trainer, independent of this
+codebase. Its environments normally run inside its own Ray cluster, in its
+own Python, with its own torch, on every node. So:
+
+**Does RLinf, through the bridge, train environments that PlugRL env clients
+run outside its cluster: as other processes on its machine, and on another
+machine that has no torch, no Ray and no RLinf?**
+
+---
+
+## Declared in advance: what was already known
+
+1. **The code** (`results/pilot.txt`).
+   - **The bridge:** PlugRL/plugrl-rlinf at fca69db. `PlugRLEnv` is an RLinf
+     environment, and `BridgeServer` is a lockstep PlugRL server inside
+     RLinf's environment worker.
+   - **RLinf:** 034579c plus the bridge's two-hunk patch, installed by its
+     own `install.sh`, with torch 2.11.0+cu130 and ray 2.58.0.
+   - **The env clients:** plugrl-env-client main, 16 processes of one
+     HalfCheetah-v5 env each.
+     - On guangzhao: mujoco 3.14.0, gymnasium 1.3.0.
+     - On the laptop: mujoco 3.6.0, gymnasium 1.2.3, in a venv with no
+       torch, ray or rlinf, 276 MB in all.
+   - **The config:** RLinf's own CartPole PPO-MLP config with the env swapped,
+     16 envs × 256 steps = 4,096 frames per epoch, 100 epochs, 409,600 steps
+     (E43's budget), one RTX 5090.
+2. **The pilot.**
+   - Locally, the return of the first five episode batches rose from -267
+     to -110 in 20 epochs, at 3.6-4.6 s per epoch.
+   - Across machines, 5 epochs took 22.9-24.3 s each over a 49-54 ms link,
+     and the per-step reward rose from -0.34 to -0.22.
+   - Four defects were found and fixed before this file (pilot.txt). The
+     worst was a reconnect storm: 2 million connections from leftover
+     client processes.
+3. **The physics differ across the two machines.** The mujoco and gymnasium
+   versions are not the same, so the two arms are compared by the status
+   rule, not step by step.
+4. **The link varies.** It measured 3 ms and 54 ms on the same day. The
+   cross arm's time per epoch is reported, not predicted.
+
+---
+
+## Design
+
+Six runs, one at a time, all with RLinf on guangzhao. `run_e48.sh` runs them
+from the laptop:
+- **local, seeds 0, 1, 2:** RLinf and the 16 env clients both on guangzhao
+  (`run_guangzhao.sh OUT SEED local`);
+- **cross, seeds 0, 1, 2:** RLinf on guangzhao, the 16 env clients on the
+  laptop over Tailscale (`run_guangzhao.sh OUT SEED remote` and
+  `cross_clients.py`).
+
+The seed seeds RLinf's actor and the env clients. Episodes are 1,000 steps
+(HalfCheetah's time limit), so all 16 envs end an episode together about
+every 4 epochs, and RLinf logs `env/return` once per batch: 25 batches in
+100 epochs.
+
+---
+
+## Checks
+
+* **V1 - the envs are where they should be.** Each run's log has 16
+  `plugrl-rlinf: slots` lines. They come from 127.0.0.1 in the local arm,
+  and from the laptop's Tailscale address, 100.65.224.34, in the cross arm.
+* **V2 - the run is complete.** RLinf logs 100 epochs, and its log has no
+  `Traceback`, `ClientLost` or `TimeoutError`.
+
+---
+
+## Predictions, and what falsifies each
+
+The status rule is E43's: a run **learns** if the mean `env/return` of its
+last five episode batches is at least **200** above its first batch's.
+
+**P1 - RLinf learns through the bridge, with the clients on its machine.**
+The local arm learns on at least 2 of 3 seeds.
+
+> Grounds: known item 2. The pilot rose 157 in four batches.
+
+**P2 - and with the clients on another machine.** The cross arm learns on at
+least 2 of 3 seeds.
+
+> Grounds: as P1. Where the environment is stepped changes the timing, not
+> the data a step produces (E43). The physics versions differ (known item
+> 3), which is why this is a status rule and not a comparison of curves.
+
+**Reported, not predicted:**
+- every run's `env/return` curve;
+- the median time per epoch in each arm;
+- the link's ping before and after each cross run;
+- the wall clock;
+- the slot lines.
+
+---
+
+## Declared deviations allowed in advance
+
+1. A cross run that dies for a reason outside the experiment, such as the
+   laptop's Wi-Fi dropping, is rerun once. This is recorded in `AMENDMENT.md`.
+
+---
+
+## Reading order
+
+V1, V2, P1, P2, then the reported figures.