Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,17 @@ has been measured:
| [E43](experiments/e43-cross-machine-training/) | Does training still work with the env clients on another physical machine? | **Yes** - the quickstart pair learns with its env clients on a Windows laptop over campus Wi-Fi |
| [E46](experiments/e46-reuse-feedback-obs/) | What does crossing cost? | About **3 ms plus the observation's bytes over the link** per exchange: 11.6 ms for 184 KiB at 23.7 MB/s. Each observation crosses once with `reuse-feedback-obs`, which the server offers and the env client uses by default; [E43](experiments/e43-cross-machine-training/) measured twice the bytes before it, and the weights are byte-identical either way |
| [E10](experiments/e10-vla-forward-cost/) | Is that cheap beside a VLA forward pass? | **On a fast link.** On that link a 184 KiB observation is 12% of pi0.5's 100 ms forward, not E10's 1.3-3.6% |
| [E49](experiments/e49-multi-trainer/) | Do other trainers train through it? | **Yes** - RLinf, Stable-Baselines3 and CleanRL train PlugRL env clients through `plugrl-bridges` (not public yet), 3 of 3 seeds in every pairing; RLinf also with the env clients on a laptop with no torch, Ray or RLinf ([E48](experiments/e48-rlinf-bridge/)) |
| [E49](experiments/e49-multi-trainer/) | Do other trainers train through it? | **Yes** - RLinf, Stable-Baselines3 and CleanRL train PlugRL env clients through [`plugrl-bridges`](https://github.com/PlugRL/plugrl-bridges), 3 of 3 seeds in every pairing; RLinf also with the env clients on a laptop with no torch, Ray or RLinf ([E48](experiments/e48-rlinf-bridge/)) |
| [E50](experiments/e50-boundary-transparency/) | Does training see the boundary? | **No** - SB3 with its 16 environments behind the protocol ends on byte-identical weights to SB3 with them in its own process, also on Atari frames ([E53](experiments/e53-image-observations/)) and with 25 ms each way between the two ([E52](experiments/e52-latency-sweep/)). It costs 0.4-0.6 ms per vector step on one machine |
| [E54](experiments/e54-boundary-faults/) | Would a fault at the boundary show? | **Not in the learning curve** - unless every value crossing is wrong, a fault passes a learning-curve threshold in at least nine runs of ten, and a clean evaluation of the policy nearly always ([E57](experiments/e57-clean-eval/)). A step ledger kept on both sides caught and located every such fault wherever it was kept, on Pendulum, Hopper and Pusher ([E55](experiments/e55-step-ledger/), [E56](experiments/e56-hopper-faults/), [E58](experiments/e58-pusher-faults/)), for a median 3% of training time on Pendulum |
| [E11](experiments/e11-vla-rl-libero/) | Does a full-size VLA go through it? | **Yes** - pi0.5 on LIBERO scores what openpi publishes, and the server's episode and step counts match the clients' exactly |

The boundary checks are the paper *Detecting Silent Faults at the
Environment-Trainer Boundary in Reinforcement Learning*, accepted to the CoRL
2026 Workshop on Everything Beneath the Policy
([PDF](https://plugrl.github.io/assets/silent-faults-webp2026.pdf),
[summary and video](https://plugrl.github.io/#when-the-boundary-is-wrong-nothing-says-so)).

Fine-tuning pi0.5 with reinforcement learning through it has not made the
policy better yet; that record is on
[its own page](https://plugrl.github.io/vla/).
Expand Down
22 changes: 19 additions & 3 deletions experiments/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
PlugRL's claim is that training should not care where its environments run:
in another process, on another machine, on a machine with no GPU, or in a
program that is not Python, and whatever trainer is on the other side. These
fifty-two experiments test that claim, measure what it costs, and record the
defects found on the way. Fifty-one have run. E3 has a protocol and no data.
fifty-seven experiments test that claim, measure what it costs, and record the
defects found on the way. Fifty-six have run. E3 has a protocol and no data.

Each directory has a `FINDINGS.md`: what was asked, what came back, and what
the result does **not** support. The logs behind it are under `results*/`.
Forty-five of the experiments that ran have a `PROTOCOL.md` with their
Fifty of the experiments that ran have a `PROTOCOL.md` with their
predictions. Most were written before the data existed; the exceptions
(E15 was written with its runs in flight) say so. Anything changed after a
protocol is dated in an `AMENDMENT.md`, not folded into it.
Expand Down Expand Up @@ -53,6 +53,22 @@ adapters for RLinf, gymnasium's `VectorEnv` and Stable-Baselines3's
| [`e52-latency-sweep`](e52-latency-sweep/) | Does latency change what is learned? | **No, only the time.** With 0 to 25 ms added each way between client and bridge, every run ends on the in-process weights. Each step costs two one-way delays plus up to 3.1 ms, a little over the predicted band at 25 ms |
| [`e53-image-observations`](e53-image-observations/) | And with camera frames and a CNN? | **Still the same weights.** SB3's CnnPolicy on Atari frames ends byte-identical in process and through the bridge; 16 frames a step cost 2.6-3.0 ms |

## When the boundary is wrong

Faults injected into what crosses the bridge, and the checks that notice
them. These five are the paper *Detecting Silent Faults at the
Environment-Trainer Boundary in Reinforcement Learning*, accepted to the CoRL
2026 Workshop on Everything Beneath the Policy
([PDF](https://plugrl.github.io/assets/silent-faults-webp2026.pdf)).

| | Question | Answer |
|---|---|---|
| [`e54-boundary-faults`](e54-boundary-faults/) | Would a fault in what crosses show in the learning curve? | **Rarely.** 21 client faults and 2 log faults on Pendulum and HalfCheetah, from one value to every value: below every value, the gain threshold caught at most one run in ten. Byte-identical weights or the env-side episode record caught every one |
| [`e55-step-ledger`](e55-step-ledger/) | Can a check find the fault with no fault-free run to compare? | **Yes.** A step ledger on both sides caught and located all 189 client-fault runs on Pendulum at their first changed value, with no false alarm in 27, and all 21 across two machines, for a median 3% of training time |
| [`e56-hopper-faults`](e56-hopper-faults/) | And on Hopper, which ends episodes when it falls? | **The same.** The ledger caught and located all 207 client-fault runs, and the curves missed most. Byte identity missed 12 final-observation faults that SB3 never reads |
| [`e57-clean-eval`](e57-clean-eval/) | Are the policies behind the faults the curves miss any worse? | **Almost never.** On a clean environment, 5 of 255 Pendulum runs that passed both curve checks scored below the fault-free band, none on HalfCheetah or Hopper. Every policy the faults wrecked, the curve checks had already caught |
| [`e58-pusher-faults`](e58-pusher-faults/) | And on a 7-joint arm? | **E54-E57 hold on Pusher.** The ledger caught and located all 189 client-fault runs, the curves missed most, and no fault the curves missed left a worse policy |

## What trains through it

These are the sixteen cells [on the project page](https://plugrl.github.io/#what-runs-on-it),
Expand Down
Loading