Repository navigation
Merged
Conversation
…through the PlugRL protocol - src/plugrl_rlinf/server.py: BridgeServer, a PlugRL protocol server that RLinf steps in lockstep from its environment worker. It holds every infer until RLinf asks for a step, answers all of them with that step's actions, and returns once each client has fed back and asked again. It offers reuse-feedback-obs. A client that breaks lockstep is closed for a resync, a client too many is told to stop, and a client lost mid-run stops the run with ClientLost: it never fills in a step. - src/plugrl_rlinf/env.py: PlugRLEnv, RLinf's environment contract (reset, chunk_step under auto_reset) on top of it, with RLinf's final_observation / final_info conventions. - patches/rlinf-plugrl-env-type.patch: RLinf (034579c) learns env_type plugrl, two hunks in rlinf/envs/__init__.py. - tests/test_bridge.py: PlugRLEnv driven as RLinf's env worker drives an env, against fake clients speaking the real protocol (10 tests). - examples/drive_probe.py: two real plugrl-env-client processes running the probe environment, every step of every slot checked. - examples/rlinf/: an RLinf PPO-MLP config for HalfCheetah through the bridge, and the scripts that run it on one machine or with the clients on another.
…, and a per-episode log The lockstep BridgeServer moves out of the RLinf adapter into plugrl_bridges.server, and three adapters sit on it: - plugrl_bridges.rlinf.PlugRLEnv: RLinf's env contract (was plugrl_rlinf.env); - plugrl_bridges.gym.PlugRLVectorEnv: a gymnasium VectorEnv under SAME_STEP autoreset, with episode statistics counted by the bridge, and a NormalizeObservation that also normalizes final_obs (gymnasium 1.3's vector observation wrappers refuse SAME_STEP); - plugrl_bridges.sb3.PlugRLVecEnv: an SB3 VecEnv with terminal_observation, TimeLimit.truncated and episode. BridgeServer(episode_log=PATH), or PLUGRL_BRIDGES_EPISODE_LOG, writes one CSV line per episode (step, slot, episode, return, length), summed from the rewards the clients sent, whichever trainer steps the bridge. Examples: CleanRL's ppo_continuous_action.py with only its env creation and episode logging changed (MIT, LICENSE-CLEANRL), SB3's PPO with rl-zoo's Pendulum settings as a preset, RLinf PPO-MLP configs for the C++ Pendulum client, and run_bridged.sh for one run of any trainer with 16 clients. cross_clients.py now refuses a --run-dir that Git Bash has rewritten. CI: ruff 0.12.12 (check, format) and pytest; torch from the CPU index.
…ort check finds RLinf E49's pilot found the Pendulum config claiming rl-zoo's PPO settings while keeping the global_batch_size 1,024 it inherited from RLinf's CartPole template: 40 updates per 4,096-frame rollout against rl-zoo's 640 (rl-zoo leaves batch_size at SB3's default, 64). Now 64. install_guangzhao.sh checked the env class without RLinf on PYTHONPATH (RLinf is not installed as a package), so the check failed after the install had worked.
plugrl-bridges: gymnasium and SB3 adapters beside RLinf's, and a per-episode log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first version of the RLinf bridge: RLinf trains environments that PlugRL env clients run, through the PlugRL protocol.
How it works.
PlugRLEnvis an RLinf environment type,plugrl. It implements RLinf's environment contract,reset()andchunk_step()underauto_reset.BridgeServeris a PlugRL server that runs in a thread of RLinf's environment worker:chunk_stepsends every client its actions, and returns once every client has fed back and asked again;reuse-feedback-obs.patches/).It is a lockstep server, because RLinf steps all its environments together. Every infer and feedback on a connection must name all of that connection's envs, which a client running one-step chunks always does. It never fills in a step it did not receive:
ClientLost.Checks so far:
tests/test_bridge.py, 10 tests: fake clients over the real protocol, driven as RLinf's env worker drives an env.examples/drive_probe.py: two realplugrl-env-clientprocesses on the probe environment, 5 slots for 40 steps, with every action, step count, episode end, terminal observation and statistic checked. No failures.Private for now. Whether it goes public is your call.