diff --git a/docs/assets/silent-faults-webp2026.pdf b/docs/assets/silent-faults-webp2026.pdf new file mode 100644 index 0000000..f610e94 Binary files /dev/null and b/docs/assets/silent-faults-webp2026.pdf differ diff --git a/docs/index.md b/docs/index.md index 7b6c162..c010a2b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -83,8 +83,8 @@ Pendulum and trains a policy through it, as well as the Python env client does ## Other trainers train through it The protocol is not tied to PlugRL's own server. Small adapters present it as -the environment interface a trainer already has (`plugrl-bridges`, not public -yet). Through them, three trainers that were not written for PlugRL train +the environment interface a trainer already has +([`plugrl-bridges`](https://github.com/PlugRL/plugrl-bridges)). Through them, three trainers that were not written for PlugRL train PlugRL env clients, and every seed of every pairing learns: - RLinf's PPO, on the C++ Pendulum client; - Stable-Baselines3's PPO, on the C++ Pendulum client and on HalfCheetah; @@ -114,6 +114,50 @@ Only the time changes: 0.4-0.6 ms per step of 16 environments on one machine, 2.6-3.0 ms when each step carries 16 frames, and about two one-way delays per step behind a slow link. +## When the boundary is wrong, nothing says so + + + +*70 seconds, no sound: one value changed on its way to the trainer, a learning +curve that rises anyway, and the step ledger that finds the change.* + +The same bridge let us ask how a broken boundary shows itself. While SB3's +PPO trained Pendulum, HalfCheetah, Hopper and Pusher, a 7-joint arm, the env +client changed one field of what crossed, in one way: an observation, action, +reward or final observation zeroed, stale, swapped with the next +environment's or rounded to float16, an episode end invented or sent as the +wrong kind, a step skipped. Nothing raised an error. Unless every value was +wrong, the learning-curve threshold we had relied on caught at most one run +in ten, and a comparison with ten fault-free seeds at most 21% +([E54](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e54-boundary-faults), +[E56](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e56-hopper-faults), +[E58](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e58-pusher-faults)). +A clean evaluation of every final policy agreed with the curves: of the 728 +boundary-fault runs that passed both curve checks, 723 scored like fault-free +seeds ([E57](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e57-clean-eval)). +The runs were not the same runs, though. Nearly all ended on other weights, +and reward faults changed the returns the trainer logged. + +What saw every fault was a check of the run itself. A step ledger keeps a +record on each side of the boundary, what each environment took and gave and +what the trainer sent and received, and compares the two step by step. On +one machine it caught and located all 585 boundary-fault runs on Pendulum, +Hopper and Pusher, and across two machines all 21 on Pendulum, with no +fault-free run to compare against, for a median 3% of training time on +Pendulum ([E55](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e55-step-ledger); +Hopper and Pusher in E56 and E58). +An episode record kept at the environment caught the faults planted in the +trainer's own log. Such faults happen outside experiments too: under +`SAME_STEP` autoreset, Gymnasium 1.3's vector episode statistics dropped the +first reward of every episode after an environment's first, until 1.4 +([E51](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e51-episode-accounting)). + +This is the paper *Detecting Silent Faults at the Environment-Trainer +Boundary in Reinforcement Learning*, accepted to the CoRL 2026 Workshop on +Everything Beneath the Policy ([PDF](/assets/silent-faults-webp2026.pdf); +[citing it](#citing)). + ## What the split costs On one machine, almost nothing: an exchange takes 0.1 ms with states only and @@ -208,3 +252,16 @@ side that uses them. - [Environments](env/index.md) - [Policies](policy/index.md) - [Contributing](contributing/index.md) + +## Citing + +If the boundary checks are useful to you, cite the paper: + +```bibtex +@inproceedings{gou2026silent, + title = {Detecting Silent Faults at the Environment-Trainer Boundary in Reinforcement Learning}, + author = {Gou, Zuo and Lu, Chenhao}, + booktitle = {CoRL 2026 Workshop on Everything Beneath the Policy}, + year = {2026} +} +``` diff --git a/docs/index.zh.md b/docs/index.zh.md index 491ee15..a1d0338 100644 --- a/docs/index.zh.md +++ b/docs/index.zh.md @@ -62,7 +62,7 @@ Pendulum,通过 PlugRL 训练出了策略,学得和 Python 环境端一样 ## 别的训练框架也能通过它训练 协议不绑定 PlugRL 自己的训练端。几个小适配器把它包装成训练框架本来就认的环境接口 -(`plugrl-bridges`,暂未公开)。通过它们,三个不是为 PlugRL 写的训练框架训练 PlugRL +([`plugrl-bridges`](https://github.com/PlugRL/plugrl-bridges))。通过它们,三个不是为 PlugRL 写的训练框架训练 PlugRL 的环境端,每种组合的每个种子都学会了: - RLinf 的 PPO,训练 C++ Pendulum 客户端; - Stable-Baselines3 的 PPO,训练 C++ Pendulum 客户端和 HalfCheetah; @@ -89,6 +89,42 @@ RLinf 还训练了跑在一台笔记本上的 16 个 HalfCheetah 环境端,那 变的只有时间:同一台机器上 16 个环境每步多 0.4 到 0.6 毫秒,每步带 16 帧画面时多 2.6 到 3.0 毫秒,链路慢时每步大约多两个单程延迟。 +## 边界出错时,什么都不会报错 + + + +*70 秒,无声:一个值在送往训练端的路上被改掉,学习曲线照样上升,步账本找到了这处改动。* + +借同一个 bridge,我们还能看边界出错时会有什么表现。Stable-Baselines3 的 PPO 训练 +Pendulum、HalfCheetah、Hopper 和 7 关节机械臂 Pusher 时,环境端把发出去的某一个字段按 +某一种方式改掉:观测、动作、奖励或回合最后一帧观测被置零、换成上一步的、换成下一个环境的, +或者经过 float16 舍入;凭空多出一个回合结束,或者把回合结束的类型发错;跳过一步。没有任何 +报错。除非每个值都错了,我们原来依赖的学习曲线阈值最多只抓到十分之一的运行,和十个无错 +种子比较的检查最多抓到 21% +([E54](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e54-boundary-faults)、 +[E56](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e56-hopper-faults)、 +[E58](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e58-pusher-faults))。 +在干净的环境里评估每个最终策略,结论和曲线一样:两项曲线检查都通过的 728 个边界出错运行中, +723 个和无错种子表现相当 +([E57](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e57-clean-eval))。 +但这些运行已经不是原来那次运行了:几乎全都停在了不同的权重上,奖励出错时训练端记下的回报 +也变了。 + +能看到每一处错误的,是检查运行本身。步账本在边界两侧各记一份:每个环境收到了什么、给出了 +什么,训练端发出了什么、收到了什么,然后逐步比对。单机上,它在 Pendulum、Hopper 和 Pusher +上抓到并定位了全部 585 个边界出错的运行;跨两台机器时,抓到了 Pendulum 上全部 21 个。全程 +不需要无错运行做参照,在 Pendulum 上训练时间的中位数只多 3% +([E55](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e55-step-ledger); +Hopper 和 Pusher 见 E56、E58)。在环境端记的回合记录,抓到了故意埋进训练端日志里的错误。 +这类错误在实验之外也会发生:Gymnasium 1.3 的向量回合统计在 `SAME_STEP` 自动重置下,每个 +环境从第二个回合起都会丢掉回合的第一个奖励,直到 1.4 才修好 +([E51](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e51-episode-accounting))。 + +这部分内容是论文 *Detecting Silent Faults at the Environment-Trainer Boundary in +Reinforcement Learning*,已被 CoRL 2026 Workshop on Everything Beneath the Policy 录用 +([PDF](/assets/silent-faults-webp2026.pdf);[引用方式](#citing))。 + ## 拆开的代价 同一台机器上几乎没有代价:只传状态时一次交换 0.1 毫秒,观测有 588 KiB 也不到 1 毫秒。 @@ -171,3 +207,16 @@ episode 回报从 -300 附近起步,三个随机种子到 50 万步达到 **19 - [环境](env/index.zh.md) - [策略](policy/index.zh.md) - [贡献指南](contributing/index.zh.md) + +## 引用 {#citing} + +如果边界检查对你有用,请引用这篇论文: + +```bibtex +@inproceedings{gou2026silent, + title = {Detecting Silent Faults at the Environment-Trainer Boundary in Reinforcement Learning}, + author = {Gou, Zuo and Lu, Chenhao}, + booktitle = {CoRL 2026 Workshop on Everything Beneath the Policy}, + year = {2026} +} +``` diff --git a/docs/media/boundary-faults.mp4 b/docs/media/boundary-faults.mp4 new file mode 100644 index 0000000..00e9c78 Binary files /dev/null and b/docs/media/boundary-faults.mp4 differ