diff --git a/README.md b/README.md index b7280db..4b3aad4 100644 --- a/README.md +++ b/README.md @@ -28,10 +28,17 @@ has been measured: | [E43](experiments/e43-cross-machine-training/) | Does training still work with the env clients on another physical machine? | **Yes** - the quickstart pair learns with its env clients on a Windows laptop over campus Wi-Fi | | [E46](experiments/e46-reuse-feedback-obs/) | What does crossing cost? | About **3 ms plus the observation's bytes over the link** per exchange: 11.6 ms for 184 KiB at 23.7 MB/s. Each observation crosses once with `reuse-feedback-obs`, which the server offers and the env client uses by default; [E43](experiments/e43-cross-machine-training/) measured twice the bytes before it, and the weights are byte-identical either way | | [E10](experiments/e10-vla-forward-cost/) | Is that cheap beside a VLA forward pass? | **On a fast link.** On that link a 184 KiB observation is 12% of pi0.5's 100 ms forward, not E10's 1.3-3.6% | -| [E49](experiments/e49-multi-trainer/) | Do other trainers train through it? | **Yes** - RLinf, Stable-Baselines3 and CleanRL train PlugRL env clients through `plugrl-bridges` (not public yet), 3 of 3 seeds in every pairing; RLinf also with the env clients on a laptop with no torch, Ray or RLinf ([E48](experiments/e48-rlinf-bridge/)) | +| [E49](experiments/e49-multi-trainer/) | Do other trainers train through it? | **Yes** - RLinf, Stable-Baselines3 and CleanRL train PlugRL env clients through [`plugrl-bridges`](https://github.com/PlugRL/plugrl-bridges), 3 of 3 seeds in every pairing; RLinf also with the env clients on a laptop with no torch, Ray or RLinf ([E48](experiments/e48-rlinf-bridge/)) | | [E50](experiments/e50-boundary-transparency/) | Does training see the boundary? | **No** - SB3 with its 16 environments behind the protocol ends on byte-identical weights to SB3 with them in its own process, also on Atari frames ([E53](experiments/e53-image-observations/)) and with 25 ms each way between the two ([E52](experiments/e52-latency-sweep/)). It costs 0.4-0.6 ms per vector step on one machine | +| [E54](experiments/e54-boundary-faults/) | Would a fault at the boundary show? | **Not in the learning curve** - unless every value crossing is wrong, a fault passes a learning-curve threshold in at least nine runs of ten, and a clean evaluation of the policy nearly always ([E57](experiments/e57-clean-eval/)). A step ledger kept on both sides caught and located every such fault wherever it was kept, on Pendulum, Hopper and Pusher ([E55](experiments/e55-step-ledger/), [E56](experiments/e56-hopper-faults/), [E58](experiments/e58-pusher-faults/)), for a median 3% of training time on Pendulum | | [E11](experiments/e11-vla-rl-libero/) | Does a full-size VLA go through it? | **Yes** - pi0.5 on LIBERO scores what openpi publishes, and the server's episode and step counts match the clients' exactly | +The boundary checks are the paper *Detecting Silent Faults at the +Environment-Trainer Boundary in Reinforcement Learning*, accepted to the CoRL +2026 Workshop on Everything Beneath the Policy +([PDF](https://plugrl.github.io/assets/silent-faults-webp2026.pdf), +[summary and video](https://plugrl.github.io/#when-the-boundary-is-wrong-nothing-says-so)). + Fine-tuning pi0.5 with reinforcement learning through it has not made the policy better yet; that record is on [its own page](https://plugrl.github.io/vla/). diff --git a/experiments/README.md b/experiments/README.md index ad6cd43..23bfeb8 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -3,12 +3,12 @@ PlugRL's claim is that training should not care where its environments run: in another process, on another machine, on a machine with no GPU, or in a program that is not Python, and whatever trainer is on the other side. These -fifty-two experiments test that claim, measure what it costs, and record the -defects found on the way. Fifty-one have run. E3 has a protocol and no data. +fifty-seven experiments test that claim, measure what it costs, and record the +defects found on the way. Fifty-six have run. E3 has a protocol and no data. Each directory has a `FINDINGS.md`: what was asked, what came back, and what the result does **not** support. The logs behind it are under `results*/`. -Forty-five of the experiments that ran have a `PROTOCOL.md` with their +Fifty of the experiments that ran have a `PROTOCOL.md` with their predictions. Most were written before the data existed; the exceptions (E15 was written with its runs in flight) say so. Anything changed after a protocol is dated in an `AMENDMENT.md`, not folded into it. @@ -53,6 +53,22 @@ adapters for RLinf, gymnasium's `VectorEnv` and Stable-Baselines3's | [`e52-latency-sweep`](e52-latency-sweep/) | Does latency change what is learned? | **No, only the time.** With 0 to 25 ms added each way between client and bridge, every run ends on the in-process weights. Each step costs two one-way delays plus up to 3.1 ms, a little over the predicted band at 25 ms | | [`e53-image-observations`](e53-image-observations/) | And with camera frames and a CNN? | **Still the same weights.** SB3's CnnPolicy on Atari frames ends byte-identical in process and through the bridge; 16 frames a step cost 2.6-3.0 ms | +## When the boundary is wrong + +Faults injected into what crosses the bridge, and the checks that notice +them. These five are the paper *Detecting Silent Faults at the +Environment-Trainer Boundary in Reinforcement Learning*, accepted to the CoRL +2026 Workshop on Everything Beneath the Policy +([PDF](https://plugrl.github.io/assets/silent-faults-webp2026.pdf)). + +| | Question | Answer | +|---|---|---| +| [`e54-boundary-faults`](e54-boundary-faults/) | Would a fault in what crosses show in the learning curve? | **Rarely.** 21 client faults and 2 log faults on Pendulum and HalfCheetah, from one value to every value: below every value, the gain threshold caught at most one run in ten. Byte-identical weights or the env-side episode record caught every one | +| [`e55-step-ledger`](e55-step-ledger/) | Can a check find the fault with no fault-free run to compare? | **Yes.** A step ledger on both sides caught and located all 189 client-fault runs on Pendulum at their first changed value, with no false alarm in 27, and all 21 across two machines, for a median 3% of training time | +| [`e56-hopper-faults`](e56-hopper-faults/) | And on Hopper, which ends episodes when it falls? | **The same.** The ledger caught and located all 207 client-fault runs, and the curves missed most. Byte identity missed 12 final-observation faults that SB3 never reads | +| [`e57-clean-eval`](e57-clean-eval/) | Are the policies behind the faults the curves miss any worse? | **Almost never.** On a clean environment, 5 of 255 Pendulum runs that passed both curve checks scored below the fault-free band, none on HalfCheetah or Hopper. Every policy the faults wrecked, the curve checks had already caught | +| [`e58-pusher-faults`](e58-pusher-faults/) | And on a 7-joint arm? | **E54-E57 hold on Pusher.** The ledger caught and located all 189 client-fault runs, the curves missed most, and no fault the curves missed left a worse policy | + ## What trains through it These are the sixteen cells [on the project page](https://plugrl.github.io/#what-runs-on-it),