diff --git a/README.md b/README.md index 23afbbe5300..ae6cf35dd1c 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@ Trinity-RFT provides functionalities for users with different backgrounds and ob ## 🚀 News +* [2026-09] Introducing [CoD ("Connect the Dots")](https://arxiv.org/pdf/2606.20002): training LLM agents by RL for cross-task adaptation and continual learning at deployment. +* [2026-08] 🎉 [R3L](https://github.com/shiweijiezero/R3L) gets accepted by EMNLP 2026. * [2026-06] [[Release Notes]](https://github.com/agentscope-ai/Trinity-RFT/releases/tag/v0.6.0) Trinity-RFT v0.6.0 released: SGLang support, optimized fully async weights synchronization and scheduling to reduce bubbles, improved MoE training stability, and verl 0.8.0 upgrade. * [2026-06] 🎉 Two papers accepted by ICML 2026: [Learn-to-Ask](https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/learn_to_ask), and [LLM-RL entropy dynamics](https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/entropy). * [2026-04] [[Release Notes]](https://github.com/agentscope-ai/Trinity-RFT/releases/tag/v0.5.2) Trinity-RFT v0.5.2 released: support for Qwen3.5 series, bug fixes and optimizations. diff --git a/README_zh.md b/README_zh.md index f9f08a1b509..1442f841c51 100644 --- a/README_zh.md +++ b/README_zh.md @@ -41,6 +41,8 @@ Trinity-RFT 面向不同背景和目标的用户提供相应功能： ## 🚀 新闻 +* [2026-09] 推出 [CoD ("Connect the Dots")](https://arxiv.org/pdf/2606.20002)：用强化学习教会 LLM 智能体如何在部署阶段跨任务持续学习。 +* [2026-08] 🎉 [R3L](https://github.com/shiweijiezero/R3L) 论文被 EMNLP 2026 接收。 * [2026-06] [[发布说明]](https://github.com/agentscope-ai/Trinity-RFT/releases/tag/v0.6.0) Trinity-RFT v0.6.0 发布：支持 SGLang，优化全异步参数同步以及调度策略以减少空泡，优化 MoE 模型训练稳定性，升级到 verl 0.8.0。 * [2026-06] 🎉 两篇论文被 ICML 2026 接收：[Learn-to-Ask](https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/learn_to_ask) 和 [LLM-RL 熵动力学](https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/entropy)。 * [2026-04] [[发布说明]](https://github.com/agentscope-ai/Trinity-RFT/releases/tag/v0.5.2) Trinity-RFT v0.5.2 发布：支持 Qwen3.5 系列，修复 Bug 并进行多项优化。 diff --git a/examples/research_cod/README.md b/examples/research_cod/README.md index 2f4219a4eea..8bbed078757 100644 --- a/examples/research_cod/README.md +++ b/examples/research_cod/README.md @@ -10,11 +10,11 @@ CoD groups related tasks into a pack and generates a long rollout trajectory int The whole pack is trained end-to-end with RL, with fine-grained credit assignment rewarding context updates that make future tasks easier. A trained model's reward rises across pack positions, which is the signature of the elicited CoD meta-capability. -> **The full implementation lives on the [`research/cod`](https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod) branch.** Check out that branch to view or run the full code. +> **The full implementation lives on the [`research/cod_v2`](https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod_v2/examples/research_cod) branch.** Check out that branch to view or run the full code.

CoD overview -
Figure 1: a visualization of CoD-Deploy and CoD-Train (compared with standard task-by-task RL). Environments A and B are used for training; M is a new environment for deployment or evaluation. Each block is one rollout episode, for solving a task x_i (which may itself be a long-horizon multi-turn task) or for updating the agent's context z_i about the current environment. +
Figure: a visualization of CoD-Deploy and CoD-Train (compared with standard task-by-task RL). Environments A and B are used for training; M is a new environment for deployment or evaluation. Each block is one rollout episode, for solving a task x_i (which may itself be a long-horizon multi-turn task) or for updating the agent's context z_i about the current environment.

--- @@ -40,7 +40,12 @@ Key metric: `reward_iterative_hint_e2e_taskset_{ts}_pos_{pos}`, the mean reward

reward rising across pack positions -
Figure 2: the CoD effect. Reward rises across pack positions during training and at OOD evaluation, both in-domain (harder FrozenLake) and cross-domain (Alchemy, Terminal). +
Figure: the CoD effect. Reward rises across pack positions during training and at OOD evaluation, both in-domain (harder FrozenLake) and cross-domain (Alchemy, Terminal). Qwen3-8B is used as the initial model. +

+ +

+ reward rising across pack positions +
Figure: scaling CoD to larger models and harder environments.

--- @@ -56,14 +61,14 @@ In each environment, the tasks in a pack share something reusable: sometimes a h | Terminal | `cod_terminal_workflow` | How commands and paths work and their pitfalls, and roughly where files live | | Learn2Ask | `cod_learn2ask_workflow` | When to keep asking vs. when to stop and give a diagnosis | -The workflow implementations live under [`trinity/common/workflows/connect_the_dots/`](https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/trinity/common/workflows/connect_the_dots) on the `research/cod` branch. +The workflow implementations live under [`trinity/common/workflows/connect_the_dots/`](https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod_v2/trinity/common/workflows/connect_the_dots) on the `research/cod_v2` branch. --- ## Run it ```bash -git clone -b research/cod https://github.com/agentscope-ai/Trinity-RFT.git +git clone -b research/cod_v2 https://github.com/agentscope-ai/Trinity-RFT.git cd Trinity-RFT conda create -n trinity python=3.12 && conda activate trinity pip install -e ".[vllm,flash_attn]" diff --git a/examples/research_cod/README_zh.md b/examples/research_cod/README_zh.md index 5743ba76fe7..28e9c4e104c 100644 --- a/examples/research_cod/README_zh.md +++ b/examples/research_cod/README_zh.md @@ -10,11 +10,11 @@ CoD 把相关任务打成一个 pack（任务包），作为一条交替进行** 整个 pack 用 RL 端到端训练，细粒度的信用分配让“使后续任务更好解的 context 更新”获得奖励。 训练好的模型 reward 随 pack 位置递增，这正是 CoD 元能力被激发出来的标志。 -> **完整实现在 [`research/cod`](https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod) 分支**，请切到该分支查看或运行完整代码。 +> **完整实现在 [`research/cod_v2`](https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod_v2/examples/research_cod) 分支**，请切到该分支查看或运行完整代码。

CoD 方法总览 -
图 1：CoD-Deploy 与 CoD-Train 的示意（与标准的逐任务 RL 对比）。环境 A、B 用于训练，M 是部署或评测的新环境。每个 block 是一个 rollout 回合：求解任务 x_i（其本身可能是长程多轮任务），或更新 agent 对当前环境的 context z_i。 +
图：CoD-Deploy 与 CoD-Train 的示意（与标准的逐任务 RL 对比）。环境 A、B 用于训练，M 是部署或评测的新环境。每个 block 是一个 rollout 回合：求解任务 x_i（其本身可能是长程多轮任务），或更新 agent 对当前环境的 context z_i。

--- @@ -40,7 +40,12 @@ pack = [ task0, task1, task2, task3 ] # 同一 taskset 的一包相关任

reward 随 pack 位置递增 -
图 2：CoD 效应。reward 随 pack 位置上升，训练时如此，OOD 评测时也如此，包括 in-domain（更难的 FrozenLake）与 cross-domain（Alchemy、Terminal）。 +
图：CoD 效应。reward 随 pack 位置上升，训练时如此，OOD 评测时也如此，包括 in-domain（更难的 FrozenLake）与 cross-domain（Alchemy、Terminal）。使用 Qwen3-8B 作为初始模型。 +

+ +

+ reward rising across pack positions +
图：将 CoD 扩展到更大的模型与更困难的环境。

--- @@ -56,14 +61,14 @@ pack = [ task0, task1, task2, task3 ] # 同一 taskset 的一包相关任 | Terminal | `cod_terminal_workflow` | 命令、路径的用法与易踩的坑，以及文件大致在哪 | | Learn2Ask | `cod_learn2ask_workflow` | 何时继续追问、何时停下来给诊断 | -各环境的 workflow 实现位于 `research/cod` 分支的 [`trinity/common/workflows/connect_the_dots/`](https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/trinity/common/workflows/connect_the_dots)。 +各环境的 workflow 实现位于 `research/cod_v2` 分支的 [`trinity/common/workflows/connect_the_dots/`](https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod_v2/trinity/common/workflows/connect_the_dots)。 --- ## 运行 ```bash -git clone -b research/cod https://github.com/agentscope-ai/Trinity-RFT.git +git clone -b research/cod_v2 https://github.com/agentscope-ai/Trinity-RFT.git cd Trinity-RFT conda create -n trinity python=3.12 && conda activate trinity pip install -e ".[vllm,flash_attn]" diff --git a/examples/research_cod/assets/cod_qwen27b.png b/examples/research_cod/assets/cod_qwen27b.png new file mode 100644 index 00000000000..9126e752a4e Binary files /dev/null and b/examples/research_cod/assets/cod_qwen27b.png differ