Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

☑️Accepted by ``NeurIPS 2026'' (CCF A)

Training Speed Demo

1.mp4

If the player above does not show up (e.g. on GitHub), click here to watch: https://github.com/user-attachments/assets/21b040dc-b5aa-4ea3-a41d-e8ff84333976

Installation

cd FastRL-main
pip install -e .

FastRL Training

bash examples/scr/qwen2_5_vl_7b_fastrl.sh

Baseline Training

bash examples/scr/qwen2_5_vl_7b_cppo.sh
bash examples/scr/qwen2_5_vl_7b_grpo.sh
bash examples/scr/qwen2_5_vl_7b_dapo.sh
bash examples/scr/qwen2_5_vl_7b_gspo.sh

Merge Checkpoint in Hugging Face Format

python3 scripts/model_merger.py --local_dir checkpoints/easy_r1/exp_name/global_step_1/actor

Please cite this paper as follows (BibTeX):

@inproceedings{yang2026fastrl,
  title={Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO},
  author={Yang, Jiahua and Yang, Zhiwei and Zhang, Xianpeng and Chen, Dongyu and Chen, Xing and Su, Tianhuang and Lu, Haonan and Guan, Quanlong and Tang, Kai and Wang, Chuangchuang},
  booktitle={Proceedings of the 40th Annual Conference on Neural Information Processing Systems (NeurIPS)},
  pages={xx--xx},
  month={December},
  year={2026}
}

The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS) will take place in Sydney (Main) Dec. 6-12, 2026.