Repository navigation
Conversation
…, and a per-episode log The lockstep BridgeServer moves out of the RLinf adapter into plugrl_bridges.server, and three adapters sit on it: - plugrl_bridges.rlinf.PlugRLEnv: RLinf's env contract (was plugrl_rlinf.env); - plugrl_bridges.gym.PlugRLVectorEnv: a gymnasium VectorEnv under SAME_STEP autoreset, with episode statistics counted by the bridge, and a NormalizeObservation that also normalizes final_obs (gymnasium 1.3's vector observation wrappers refuse SAME_STEP); - plugrl_bridges.sb3.PlugRLVecEnv: an SB3 VecEnv with terminal_observation, TimeLimit.truncated and episode. BridgeServer(episode_log=PATH), or PLUGRL_BRIDGES_EPISODE_LOG, writes one CSV line per episode (step, slot, episode, return, length), summed from the rewards the clients sent, whichever trainer steps the bridge. Examples: CleanRL's ppo_continuous_action.py with only its env creation and episode logging changed (MIT, LICENSE-CLEANRL), SB3's PPO with rl-zoo's Pendulum settings as a preset, RLinf PPO-MLP configs for the C++ Pendulum client, and run_bridged.sh for one run of any trainer with 16 clients. cross_clients.py now refuses a --run-dir that Git Bash has rewritten. CI: ruff 0.12.12 (check, format) and pytest; torch from the CPU index.
…ort check finds RLinf E49's pilot found the Pendulum config claiming rl-zoo's PPO settings while keeping the global_batch_size 1,024 it inherited from RLinf's CartPole template: 40 updates per 4,096-frame rollout against rl-zoo's 640 (rl-zoo leaves batch_size at SB3's default, 64). Now 64. install_guangzhao.sh checked the env class without RLinf on PYTHONPATH (RLinf is not installed as a package), so the check failed after the install had worked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1, which E48 runs at fca69db; that branch is left as it is.
What changes
The RLinf bridge becomes plugrl-bridges: one lockstep
BridgeServer, and three adapters on it.plugrl_bridges.rlinf.PlugRLEnvenv_type: plugrl(wasplugrl_rlinf.env)plugrl_bridges.gym.PlugRLVectorEnvgymnasium.vector.VectorEnv, SAME_STEP autoresetplugrl_bridges.sb3.PlugRLVecEnvVecEnvRecordEpisodeStatisticsdrops each episode's first reward under SAME_STEP (plugrl-server E51), and its vector observation wrappers refuse SAME_STEP.plugrl_bridges.gym.NormalizeObservationis provided instead.BridgeServer(episode_log=PATH)writesstep,slot,episode,return,lengthfor every episode. So doesPLUGRL_BRIDGES_EPISODE_LOG, and RLinf'senv.train.episode_log. It is the same record whichever trainer steps the bridge, and E49 judges every trainer on it.ppo_continuous_action.py, with only its env creation and episode logging changed (MIT,LICENSE-CLEANRL).run_bridged.sh TRAINER ENV OUT SEEDfor one run of any of them with 16 clients.cross_clients.pyrefuses a--run-dirthat Git Bash has rewritten. That rewrite left E48's clients reconnecting after their run (E48AMENDMENT.md).Tested
monitor.csv, and each C++ client's own count.PlugRLVecEnv(E50): byte-identical final weights, on Pendulum and on HalfCheetah, on every seed.The repository is renamed to plugrl-bridges once E48 has finished.