feat(rl): add SAO synchronous training baseline with skip-observation GAE, DIS policy loss and value critic ```

[xxyyrr598]

Description

Summary

This PR adds a complete implementation of the SAO (Skip-Observation Advantage) synchronous correctness baseline to twinkle, including all core algorithmic components plus a GSM8K training example. SAO is an asynchronous RL algorithm designed for Agent / multi-turn reasoning (paper parameters are documented in cookbook/rl/sao/README.md §11). This PR first lands the full algorithm formulas and a synchronous rollout baseline to validate correctness, providing a clean foundation for a future asynchronous pipeline.

Implemented components: Single-Rollout, Direct Double-Sided Importance Sampling (DIS) policy loss, strict double-sided trust region, an independent value critic, Faster Value Update (K=2), a Frozen-Attention critic, Skip-Observation token-level GAE, and length-adaptive policy lambda.

Motivation

The latest main already defines the old_logps interface, ragged logp alignment, token-mean aggregation, value model, and GAE semantics. This PR reimplements SAO on top of that main following these principles:

• Rollout log-probs are passed to the common loss interface via old_logps; no redundant parameters are introduced.
• No PPO training code is modified; PPOLoss and the default GAEAdvantage behavior remain unchanged.
• SAO advantage and value loss are separate classes and do not replace the default PPO components.

Major Changes

New algorithm components

File	Description
src/twinkle/advantage/sao_gae.py	SAOGAEAdvantage: skip-observation token-level GAE. Positions with action_masks=True form the Bellman chain, skipping observation/prompt/padding tokens; supports terminal/truncated semantics (truncated requires a bootstrap value); length-adaptive λ = 1 − 1/(α·l)
src/twinkle/loss/sao.py	SAOLoss: DIS policy loss. Tokens whose ratio falls outside the strict double-sided trust region (ε_low=0.3 / ε_high=5.0) get zero weight (no gradient); importance weight is detached by default; token-mean aggregation
src/twinkle/loss/value.py	SAOValueLoss: masked MSE critic loss, reusing the PPO/GRPO common alignment code

Extensions & registration

• src/twinkle/model/transformers/value_model.py: adds freeze_attention_for_value_training (freezes attention, trains only MLP + value head) and trainable_parameter_summary (parameter accounting).
• src/twinkle/metric/grpo.py: adds SAOMetric — trust-rejection statistics that are unconditional on advantage sign, with closed interval boundaries matching the strict trust region of SAOLoss.
• src/twinkle/advantage/__init__.py / loss/__init__.py / metric/__init__.py: register SAOGAEAdvantage, SAOLoss, SAOValueLoss, SAOMetric.
• src/twinkle/cli/cli.py: adds SAO args (epsilon_low, detach_importance_weight, critic_updates_per_actor_update, sao_alpha, sao_policy_lambda_adaptive, sao_critic_lambda, freeze_critic_attention).

Example & documentation

• cookbook/rl/sao/sao_sync.py: GSM8K synchronous training loop (single rollout → save vLLM logps → critic computes fixed returns → K critic updates → recompute advantage → one actor update).
• cookbook/rl/sao/sao_sync.sh: default launch script (4 policy + 4 critic + 4 sampler GPUs).
• cookbook/rl/sao/README.md: ~1300-line delivery doc (paper comparison, algorithm principles, line-by-line code walkthrough, log interpretation, experiment protocol, known limitations).

Tests (4 new files, +140 lines)

• tests/advantage/test_sao_gae.py: observation skipping, zero bootstrap on terminal, bootstrap required on truncated, no cross-batch linking, length-adaptive λ.
• tests/loss/test_sao.py: strict trust-region boundaries, zero gradient outside the region, detached ratio gradient, ragged alignment and token-mean denominator.
• tests/model/test_value_model.py: MLP/value head remain trainable after attention freeze.
• tests/cli/test_cli.py: new field parsing.

Correctness Constraints (DIS prerequisites)

SAO's importance ratio requires the rollout and learner log-probs to describe the same distribution, so the training script enforces: --num-generations 1, --temperature 1.0, --top-p 1.0, --top-k -1, --repetition-penalty 1.0. It also validates at runtime that token/logprob/action-label counts are aligned, and raises otherwise.

Known Limitations / Future Work

• This PR is a synchronous-barrier correctness baseline; it does not include the paper's asynchronous rollout/learner parallelism, train-on-arrival, policy lag, or throughput gains.
• The GSM8K single-turn example has no real observations; Skip-Observation GAE is covered by unit tests, but the cookbook has no multi-turn tool trajectories yet.
• Non-terminal truncated trajectories are rejected; stop_reason=length responses are treated as terminal (an engineering simplification — no truncated bootstrap).
• Paper-scale value pretraining is not included.
• Future work: connect real TIR/Agent environments → value-pretraining checkpoints → build the asynchronous actor-learner pipeline on top of main.

Training Results

How to Test

# Run the new unit tests
pytest tests/advantage/test_sao_gae.py tests/loss/test_sao.py \
       tests/model/test_value_model.py tests/cli/test_cli.py

[tpx818]

这份AI生成的readme文档是否需要？

[xxyyrr598]

不需要，不小心提交了多余的文件

[tpx818]

这部分代码能否兼容qwen3.5系列的混合注意力模型

[xxyyrr598]

原实现仅识别 self_attn/attn，会遗漏 Qwen3.5 混合注意力架构中的 linear_attn（GatedDeltaNet）层，现已将其加入冻结范围并补充测试，确保 full attention 和 linear attention 均被冻结，而 MLP/FFN 与 value head 保持可训练。

[tpx818]

DIS计算的部分是否可以拆成一个可复用的插件，让其与GRPO可组合

[xxyyrr598]

已将 SAO Loss 中的 DIS 计算拆分为独立的内部 PolicyObjective 组件，并由 SAOLoss 组合调用；本次重构未改变原有计算逻辑和数值结果。该接口也可以扩展实现 PPO Clip 等策略目标，从而与 GRPO 等优势估计方式组合使用，但本次 PR 暂未修改 PPO/GRPO，后续如有需要可统一接入。