Repository navigation
E48: RLinf, through the bridge, trains environments that PlugRL env clients run outside its cluster - #115
Merged
Conversation
…ridge, train environments PlugRL env clients run outside its cluster PROTOCOL.md, written after the pilot in results/pilot.txt. Six runs of RLinf's PPO-MLP on HalfCheetah through PlugRL/plugrl-rlinf (fca69db): three seeds with the 16 env clients on guangzhao beside RLinf, three with them on the laptop over Tailscale, in a venv with no torch, Ray or RLinf. The status rule is E43's.
…lugRL env clients run outside its cluster, on its machine and on a laptop with no torch, Ray or RLinf RLinf PPO-MLP x HalfCheetah, 409,600 steps, 16 env clients, three seeds per arm. Local -284/-290/-268 -> 449/485/481; cross (laptop over Tailscale) -274/-274/-244 -> 450/463/496. V1, V2, P1, P2 all hold. Crossing made an epoch 6.4x slower (median 3.7 s against 22-25 s): the bridge is lockstep, so each step waits for all 16 clients across a 42-59 ms link. AMENDMENT.md: Git Bash rewrote the remote run directory passed to cross_clients.py, so its RUN_DONE poll never fired; and RLinf never closes its environments, so the remote clients got no stop and kept reconnecting. A watcher stopped each seed's clients once its run had ended. No judged figure is affected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The question
Every PlugRL result so far was trained by plugrl-server. Does RLinf, a trainer independent of this codebase, train environments that PlugRL env clients run outside its cluster? It does so through plugrl-rlinf's bridge (PlugRL/plugrl-bridges#1, fca69db), and the clients run either as other processes on its machine or on another machine with no torch, Ray or RLinf.
The answer
Yes, on every seed, with both placements. RLinf PPO-MLP × HalfCheetah-v5, 409,600 steps, 16 env clients:
V1, V2, P1 and P2 all hold (
results/verdicts.txt). Crossing machines made each epoch 6.4× slower: a median of 3.7 s locally against 22–25 s across. The bridge is lockstep, so each step waits for all 16 clients across a 42–59 ms link.Found on the way (
AMENDMENT.md)plugrl-server-stop, and clients outside the cluster reconnect indefinitely: 3,190 attempts in 25 minutes.cross_clients.py, so its end-of-run poll never fired. A watcher stopped each seed's clients after its run had ended. No judged figure is affected. plugrl-bridges' copy of the script now refuses such a path (plugrl-bridges: gymnasium and SB3 adapters beside RLinf's, and a per-episode log plugrl-bridges#2).Not in this PR
The experiments README's index and counts. #112 changes the same lines, so one PR will update them for E48–E51.