From da8252d93d8a6a7a285442a12cb58428282816ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9B=BE=E7=AB=8B=E5=AE=8F?= <曾立宏@buaa.edu.cn> Date: Thu, 20 Aug 2026 00:01:16 +0800 Subject: [PATCH 1/6] feat(eval): true end-to-end choice-protocol evaluator + unit tests; classification freeze doc (phase 13.5) --- docs/design/framework-freeze.md | 74 +++++ script/verl/sft/evaluate_true_e2e.py | 309 +++++++++++++++++++++ tests/evaluation/test_evaluate_true_e2e.py | 222 +++++++++++++++ 3 files changed, 605 insertions(+) create mode 100644 docs/design/framework-freeze.md create mode 100644 script/verl/sft/evaluate_true_e2e.py create mode 100644 tests/evaluation/test_evaluate_true_e2e.py diff --git a/docs/design/framework-freeze.md b/docs/design/framework-freeze.md new file mode 100644 index 0000000..1ba7558 --- /dev/null +++ b/docs/design/framework-freeze.md @@ -0,0 +1,74 @@ +# Framework Freeze（阶段 13.5） + +本文件把两阶段叶分类工程的关键接口声明为 **frozen**。冻结后，后续算法实验 +（GRPO / RLOO / ReMax / PPO-DAPO 等）除真正 bug 外**不得修改**这些层； +算法差异只允许落在 launcher / Hydra 配置 / 算法层（见 §完整流程清单）。 + +冻结日期：Phase 13.5（2026-08-19，Phase 11/12/13 smoke 与 7B SFT baseline 均已 PASS）。 + +## 1. Frozen contracts（冻结接口清单） + +| # | 层 | 内容 / 入口 | 不得修改的点 | +|---|---|---|---| +| 1 | canonical category_id | `agent.task.contracts.LeafRegistry` / `SampleTarget`；`data//canonical/all.json` | 唯一身份：`target.category_id`；`classification.level_*` 仅 provenance，禁止 reintroduce level_4 fallback | +| 2 | PromptChoice protocol | `agent.task.prompt_choices.PromptChoiceRegistry`（global "1".."N" / stage2 local "1".."5"） | choice id 映射、display name 规则；canonical id 永不进 prompt/不作为模型动作 | +| 3 | candidate policy | `agent.task.prompts.build_stage1_prompt`（全 registry）/ `build_stage2_prompt`（5 candidate bundle，按 category_id 查 corpus） | 现行为准：Stage1 全目录无 description；Stage2 bundle=5；hard-negative 策略不抽象（沿用 GT+registry 前四 fixture 策略） | +| 4 | parquet schema | SFT：`messages/stage/source_id/metadata/ground_truth/candidates`；RL：`data_source/prompt/ability/reward_model/extra_info` | 列名与语义；label 唯一来源 `target.category_id`；assistant supervision = choice-id JSON | +| 5 | parser / decode | `agent.task.parser.check_stage1_choices / check_stage2_choices / check_stage{1,2}_output` | 严格 JSON + 精确 schema + 无 fuzzy fallback；decode 到 canonical 后才进入正确性逻辑 | +| 6 | reward contract | `agent.training.rl.reward`（RewardConfig 0/0.3/1.0 + stage2_partial(0.5 provisional)） | 数值表与 `reward_stage{1,2}_choices` / `reward_for_choice_result`（decode 先行，表不变） | +| 7 | VeRL reward adapter | `agent.training.rl.verl_adapter.compute_score` | choice-aware 路由（`/stage<1|2>`），不实现 parser/reward 表 | +| 8 | evaluation protocol | `agent.evaluation.classification`（`evaluate_stage{1,2}_choices`）+ `script/verl/sft/evaluate_baseline.py`（factorized/proxy）+ `script/verl/sft/evaluate_true_e2e.py`（true E2E） | 指标定义：stage1 format/contract/Recall@5；stage2 conditional / format/contract；true E2E（Stage2 用 Stage1 预测 Top-5） | + +## 2. 模型起点与 RL 初始化（冻结） + +- 所有正式实验统一使用 **Qwen/Qwen2.5-7B-Instruct**（小规模功能链路可用 + Qwen2.5-0.5B-Instruct 作为 config 开关打通，不改变语义）。 +- RL 共同初始化默认 **SFT final = `global_step_140`（merged HF `merged_step140`）** + （Phase 13）；该选择仅作起点，后续若完成更完整的 SFT 训练，再统一重选一个 + SFT checkpoint，选定后同样冻结。 +- **不得用 test 指标选择 checkpoint**（Phase 13.5 起为硬约束）。 + +## 3. 完整流程清单（数据处理 → 训练 → 评估 → 其他 RL 算法） + +```text +学长预处理/corpus + └─> data//canonical/all.json + split JSON（split 按 id 隔离） + ├─> script/verl/sft/export （canonical → SFT messages parquet，choice protocol） + ├─> script/verl/rl/export （canonical → RL 五字段 parquet） + └─> validate / check_token_budget （契约 + token 预算 gate） + +SFT 训练：script/verl/sft/run_baseline.sh → verl.trainer.sft_trainer（LoRA/FSDP/bf16） + └─> checkpoints（global_step_N，verl LoRA FSDP，含全量 base 权重 ~15GB/个 → 必须配 SAVE_FREQ+MAX_CKPT_KEEP） + └─> script/verl/sft/merge_lora_checkpoint → 合并 HF 目录（可给 eval / RL 加载） + +评估（统一 greedy / prompt / parser / seed=42）： + ├─> script/verl/sft/evaluate_baseline.py — factorized/proxy 指标（Stage2 用 parquet 预构造 bundle） + └─> script/verl/sft/evaluate_true_e2e.py — true E2E（Stage2 由 Stage1 预测 Top-5 动态构建） + +RL（在冻结层之上只改算法层）： + script/verl/rl/grpo_smoke.sh -> verl.trainer.main_ppo + ├─ reward.custom_reward_function.path=pkg://agent.training.rl.verl_adapter（冻结） + └─ algorithm.* / actor.* / rollout.*（launcher 层，可换） +``` + +**实现其他 RL 算法（RLOO / ReMax / PPO-DAPO）的改动面**（各 4B/12 报告约定）： +- 只改 `script/verl/rl/grpo_smoke.sh` 的算法/采样配置：`algorithm.adv_estimator` + （grpo→dapo/rloo），RLHF-style 需加 critic/ref/`use_kl_loss` 相关配置。 +- rollout 后端 / 显存旋钮在 launcher env（`ENFORCE_EAGER` / `PARAM_OFFLOAD` / + `GPU_MEM_UTIL` / `MAX_MODEL_LEN` 等）。 +- 若必须新增 reward 形态：只在 `agent.training.rl.verl_adapter`（薄路由）后加 + compute_score 分支，parser/reward 表不动。 +- **不需要改**：canonical/parquet/parser/prompt/reward 表/evaluator/VeRL 源码 + （零 vendored patch，兼容只靠 pip 依赖 + 配置）。 + +## 4. True-E2E evaluator（阶段 13.5 新增） + +`script/verl/sft/evaluate_true_e2e.py`：raw test → `build_stage1_prompt` → +greedy → `check_stage1_choices` decode 真实 canonical Top-5 → 按这 5 个 +category_id 查 corpus → `build_stage2_prompt` 动态构建 → greedy → +`check_stage2_choices` local decode → final canonical → 与 GT 比较。 +输出：stage1 format/contract/Recall@5；stage2 conditional acc（仅 GT∈预测 +Top-5）+ format/contract 失败率；true E2E acc。metric 命名： +**Proxy E2E**（`evaluate_baseline` 的 `proxy_e2e`）与 **True E2E**（本 evaluator）必须区分。 +单测：`tests/evaluation/test_evaluate_true_e2e.py`（stage1 miss / hit+correct / +hit+wrong / malformed 不 crash / candidates==预测 Top-5）。 diff --git a/script/verl/sft/evaluate_true_e2e.py b/script/verl/sft/evaluate_true_e2e.py new file mode 100644 index 0000000..1f03ead --- /dev/null +++ b/script/verl/sft/evaluate_true_e2e.py @@ -0,0 +1,309 @@ +"""TRUE end-to-end evaluator for the two-stage choice-protocol task. + +Unlike the factorized/proxy evaluator (``evaluate_baseline.py`` — Stage 2 is +scored against the pre-constructed gold-containing bundle from the parquet), +this evaluator chains the REAL task pipeline on a raw test sample: + + raw test sample (stage1) + -> build Stage1 prompt (reuses agent.task.prompts.build_stage1_prompt) + -> model greedy generate + -> shared choice parser decode -> REAL canonical top-5 + -> fetch description/descriptions/examples for those 5 category ids + from the canonical corpus (reuses build_stage2_prompt) + -> dynamic Stage2 prompt from the PREDICTED top-5 + -> model greedy generate + -> local bundle-id decode -> final canonical category_id + -> compare against ground truth + +Every identity/decode/prompt step is the FROZEN shared layer +(agent.task.prompts / PromptChoiceRegistry / check_stage{1,2}_choices / +LeafRegistry + canonical corpus). Nothing is copied, no fuzzy fallback. + +Metrics: +- stage1: format-valid rate, contract-valid rate, Recall@5 +- stage2: conditional accuracy over sources with GT in the PREDICTED top-5, + format/contract failure rates (over attempted stage2) +- true end-to-end accuracy (stage1 recall AND stage2 correct, per source) + +Generation is injected (``generate(messages) -> str``) so unit tests drive the +full pipeline on CPU with canned model output; the CLI wraps a fixed greedy +transformers decoder (identical decoding settings for base vs SFT). + +Usage (server, SFT venv): + python -m script.verl.sft.evaluate_true_e2e \ + --model-path \ + --data \ + --registry cfg/task/registry/pers_info.registry.json \ + --corpus cfg/task/corpus/pers_info.corpus.json \ + --metadata-fields field_name field_description \ + --max-new-tokens 128 --seed 42 --report +""" + +from __future__ import annotations + +import argparse +import json +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable, Mapping +import sys + +from agent.evaluation.classification import ( + evaluate_stage1_choices, + evaluate_stage2_choices, +) +from agent.task.contracts import CorpusCategory, LeafRegistry, TaskConfig +from agent.task.prompts import ( + Prompt, + build_stage1_prompt, + build_stage2_prompt, +) + +Split = tuple[str, str, str] # (system, user, -) placeholder for typing simplicity + +GenerateFn = Callable[[list[dict[str, str]]], str] + + +@dataclass(frozen=True) +class E2EOutcome: + """One source's TRUE end-to-end result (choice decode already applied).""" + + source_id: str + ground_truth: str + # stage 1 + stage1_format_valid: bool + stage1_contract_valid: bool + recalled: bool # GT in the predicted top-5 + predicted_top5: tuple[str, ...] | None + stage1_completion: str + # stage 2 (None when stage1 was contract-invalid -> never prompted) + stage2_attempted: bool + stage2_prompt_candidates: tuple[str, ...] | None # == predicted top-5 + stage2_format_valid: bool + stage2_contract_valid: bool + stage2_correct: bool + final_decision: str | None + stage2_completion: str | None + failures: tuple[str, ...] = field(default_factory=tuple) + + @property + def e2e_correct(self) -> bool: + return self.recalled and self.stage2_correct + + +def run_one( + source: dict[str, Any], + *, + registry: LeafRegistry, + config: TaskConfig, + corpus: Mapping[str, CorpusCategory] | None, + seed: int, + generate: GenerateFn, +) -> E2EOutcome: + """Run the true pipeline for one source (a stage1 row from the parquet).""" + metadata = source["metadata"] + ground_truth = source["ground_truth"] + stage1_prompt = build_stage1_prompt(metadata, registry, config) + stage1_completion = generate( + [ + {"role": "system", "content": stage1_prompt.system}, + {"role": "user", "content": stage1_prompt.user}, + ] + ) + stage1_eval = evaluate_stage1_choices( + stage1_completion, ground_truth=ground_truth, registry=registry + ) + + base: dict[str, Any] = { + "source_id": source["source_id"], + "ground_truth": ground_truth, + "stage1_format_valid": stage1_eval.format_valid, + "stage1_contract_valid": stage1_eval.contract_valid, + "recalled": stage1_eval.ground_truth_recalled and stage1_eval.contract_valid, + "predicted_top5": stage1_eval.prediction, + "stage1_completion": stage1_completion, + "stage2_attempted": False, + "stage2_prompt_candidates": None, + "stage2_format_valid": False, + "stage2_contract_valid": False, + "stage2_correct": False, + "final_decision": None, + "stage2_completion": None, + "failures": stage1_eval.errors, + } + if not stage1_eval.contract_valid: + return E2EOutcome(**base) + + predicted = tuple(stage1_eval.prediction) # non-None because contract-valid + try: + stage2_prompt: Prompt = build_stage2_prompt( + metadata, predicted, registry, config, corpus=corpus + ) + except ValueError as exc: # e.g. predicted id absent from canonical corpus + base["failures"] = base["failures"] + (f"stage2 build: {exc}",) + return E2EOutcome(**base) + + stage2_completion = generate( + [ + {"role": "system", "content": stage2_prompt.system}, + {"role": "user", "content": stage2_prompt.user}, + ] + ) + stage2_eval = evaluate_stage2_choices( + stage2_completion, ground_truth=ground_truth, candidates=predicted, registry=registry + ) + base.update( + { + "stage2_attempted": True, + "stage2_prompt_candidates": predicted, + "stage2_format_valid": stage2_eval.format_valid, + "stage2_contract_valid": stage2_eval.contract_valid, + "stage2_correct": stage2_eval.correct, + "final_decision": stage2_eval.prediction, + "stage2_completion": stage2_completion, + "failures": stage2_eval.errors, + } + ) + return E2EOutcome(**base) + + +def aggregate_true_e2e(outcomes: list[E2EOutcome]) -> dict[str, Any]: + """Aggregate per-source E2E outcomes into the published metric table.""" + n = len(outcomes) + stage1_fmt = sum(o.stage1_format_valid for o in outcomes) / n if n else 0.0 + stage1_contract = sum(o.stage1_contract_valid for o in outcomes) / n if n else 0.0 + recalled = sum(o.recalled for o in outcomes) + attempted = sum(o.stage2_attempted for o in outcomes) + s2_fmt_fail = ( + sum(1 for o in outcomes if o.stage2_attempted and not o.stage2_format_valid) / attempted + if attempted + else 0.0 + ) + s2_contract_fail = ( + sum(1 for o in outcomes if o.stage2_attempted and not o.stage2_contract_valid) / attempted + if attempted + else 0.0 + ) + s2_correct = sum(o.stage2_correct for o in outcomes) + conditional_acc = s2_correct / recalled if recalled else 0.0 + e2e_correct = sum(o.e2e_correct for o in outcomes) + return { + "sources": n, + "stage1_format_valid": stage1_fmt, + "stage1_contract_valid": stage1_contract, + "stage1_recall_at_5": recalled / n if n else 0.0, + "stage1_recalled_count": recalled, + "stage2_attempted": attempted, + "stage2_format_failure_rate": s2_fmt_fail, + "stage2_contract_failure_rate": s2_contract_fail, + "stage2_conditional_accuracy": conditional_acc, + "true_e2e_accuracy": e2e_correct / n if n else 0.0, + "true_e2e_correct": e2e_correct, + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", required=True, help="HF model dir (base or merged)") + parser.add_argument("--data", required=True, help="phase-8 SFT parquet (test split)") + parser.add_argument("--registry", required=True, help="leaf registry JSON") + parser.add_argument("--corpus", required=True, help="canonical corpus JSON") + parser.add_argument("--metadata-fields", nargs="+", default=["field_name", "field_description"]) + parser.add_argument("--max-new-tokens", type=int, default=128) + parser.add_argument("--seed", type=int, default=42) + parser.add_argument("--report", required=True) + args = parser.parse_args(argv) + + import pyarrow.parquet as pq + + registry = LeafRegistry.from_path(args.registry) + config = TaskConfig(metadata_fields=tuple(args.metadata_fields)) + from agent.task.canonical_dataset import load_corpus_categories + + corpus = { + category.category_id: category + for category in load_corpus_categories(args.corpus) + } + rows = pq.read_table(args.data).to_pylist() + stage1_rows = [r for r in rows if r["stage"] == "stage1"] + + # greedy, fixed decoding settings (identical for base and SFT), model loaded once + import torch + from transformers import AutoModelForCausalLM, AutoTokenizer + + torch.manual_seed(args.seed) + tokenizer = AutoTokenizer.from_pretrained(args.model_path) + model = AutoModelForCausalLM.from_pretrained( + args.model_path, torch_dtype=torch.bfloat16, device_map="auto" + ) + if tokenizer.pad_token_id is None: + tokenizer.pad_token_id = tokenizer.eos_token_id + model.eval() + + def hf_generate(messages: list[dict[str, str]]) -> str: + text = tokenizer.apply_chat_template( + messages, tokenize=False, add_generation_prompt=True + ) + inputs = tokenizer(text, return_tensors="pt").to(model.device) + with torch.inference_mode(): + output = model.generate( + **inputs, + do_sample=False, + num_beams=1, + max_new_tokens=args.max_new_tokens, + pad_token_id=tokenizer.eos_token_id, + ) + return tokenizer.decode( + output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True + ).strip() + + outcomes = [ + run_one(row, registry=registry, config=config, corpus=corpus, + seed=args.seed, generate=hf_generate) + for row in stage1_rows + ] + metrics = aggregate_true_e2e(outcomes) + report = { + "metrics": metrics, + "generation": { + "model_path": args.model_path, + "do_sample": False, + "num_beams": 1, + "max_new_tokens": args.max_new_tokens, + "seed": args.seed, + }, + "per_source": [ + { + "source_id": o.source_id, + "ground_truth": o.ground_truth, + "stage1_format_valid": o.stage1_format_valid, + "stage1_contract_valid": o.stage1_contract_valid, + "recalled": o.recalled, + "predicted_top5": o.predicted_top5, + "stage1_completion": o.stage1_completion, + "stage2_attempted": o.stage2_attempted, + "stage2_prompt_candidates": o.stage2_prompt_candidates, + "stage2_format_valid": o.stage2_format_valid, + "stage2_contract_valid": o.stage2_contract_valid, + "stage2_correct": o.stage2_correct, + "final_decision": o.final_decision, + "stage2_completion": o.stage2_completion, + "e2e_correct": o.e2e_correct, + "failures": list(o.failures), + } + for o in outcomes + ], + "registry": str(args.registry), + "corpus": str(args.corpus), + "data": str(args.data), + } + rendered = json.dumps(report, ensure_ascii=False, indent=2) + path = Path(args.report) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(rendered, encoding="utf-8") + print(json.dumps(metrics, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/evaluation/test_evaluate_true_e2e.py b/tests/evaluation/test_evaluate_true_e2e.py new file mode 100644 index 0000000..81d31c7 --- /dev/null +++ b/tests/evaluation/test_evaluate_true_e2e.py @@ -0,0 +1,222 @@ +"""TRUE end-to-end evaluator unit tests (Phase 13.5). + +Drives ``evaluate_true_e2e.run_one`` / ``aggregate_true_e2e`` on CPU with an +injected ``generate`` function (canned model output keyed on whether the +prompt is Stage 1 catalog or Stage 2 candidate bundle). Covers: + +- Stage1 miss -> true E2E fails (GT is not among the PREDICTED top-5) +- Stage1 hit + Stage2 correct -> true E2E success +- Stage1 hit + Stage2 wrong -> true E2E fail (conditional denominator counts) +- malformed Stage1 / Stage2 -> never raises, maps to zero +- Stage2 candidates must be EXACTLY the Stage1 predicted top-5 + +Identity/decode/prompt logic is the frozen shared layer (PromptChoiceRegistry, +build_stage1/2_prompt, check_stage1/2_choices); nothing is re-implemented here. +""" + +from __future__ import annotations + +import json + +import pytest + +from agent.task.contracts import CorpusCategory, LeafRegistry, TaskConfig +from script.verl.sft.evaluate_true_e2e import aggregate_true_e2e, run_one + +# ---- synthetic registry/corpus: 8 categories -> choice ids "1".."8" ---- +REG_IDS = [f"reg:{i}" for i in range(1, 9)] + + +def make_registry() -> LeafRegistry: + return LeafRegistry.from_mapping(REG_IDS) + + +def make_corpus() -> dict[str, CorpusCategory]: + return { + category_id: CorpusCategory( + category_id=category_id, + name=f"name-{category_id}", + description=f"description-{category_id}", + descriptions=(f"extra-{category_id}",), + examples=(f"example-{category_id}",), + ) + for category_id in REG_IDS + } + + +def make_config() -> TaskConfig: + return TaskConfig(metadata_fields=("field_name", "field_description")) + + +def source_row(gt: str = "reg:3", source_id: str = "s-1") -> dict: + return { + "stage": "stage1", + "source_id": source_id, + "metadata": {"field_name": "fn", "field_description": "fd"}, + "ground_truth": gt, + } + + +def make_generate(stage1_out: str, stage2_out: str): + """Injected generate: dispatch on the prompt type (Stage 1 catalog vs bundle).""" + + def generate(messages): + user_content = messages[1]["content"] + if "Candidate bundle:" in user_content: + return stage2_out + return stage1_out + + return generate + + +@pytest.fixture +def ctx(): + return { + "registry": make_registry(), + "config": make_config(), + "corpus": make_corpus(), + } + + +def test_stage1_hit_and_stage2_correct_true_e2e(ctx): + row = source_row(gt="reg:3") + gen = make_generate( + stage1_out=json.dumps({"candidates": ["3", "1", "2", "4", "5"]}), + stage2_out=json.dumps({"answer": "1"}), # predicted[0] == reg:3 == GT + ) + out = run_one(row, seed=1, generate=gen, **ctx) + assert out.stage1_format_valid and out.stage1_contract_valid + assert out.recalled is True + assert out.predicted_top5 == ("reg:3", "reg:1", "reg:2", "reg:4", "reg:5") + assert out.stage2_attempted + assert out.stage2_prompt_candidates == out.predicted_top5 + assert out.stage2_correct and out.final_decision == "reg:3" + assert out.e2e_correct is True + metrics = aggregate_true_e2e([out]) + assert metrics["stage1_recall_at_5"] == 1.0 + assert metrics["stage2_conditional_accuracy"] == 1.0 + assert metrics["true_e2e_accuracy"] == 1.0 + + +def test_stage1_hit_and_stage2_wrong_true_e2e_fails(ctx): + row = source_row(gt="reg:3") + gen = make_generate( + stage1_out=json.dumps({"candidates": ["3", "1", "2", "4", "5"]}), + stage2_out=json.dumps({"answer": "2"}), # predicted[1] == reg:1 != GT + ) + out = run_one(row, seed=1, generate=gen, **ctx) + assert out.recalled is True + assert out.stage2_attempted and not out.stage2_correct + assert out.e2e_correct is False + metrics = aggregate_true_e2e([out]) + assert metrics["stage1_recall_at_5"] == 1.0 + # conditional accuracy: denominator = recalled (1), numerator = correct (0) + assert metrics["stage2_conditional_accuracy"] == 0.0 + assert metrics["true_e2e_accuracy"] == 0.0 + + +def test_stage1_miss_stage2_still_uses_predicted_top5_and_fails(ctx): + row = source_row(gt="reg:3") + # contract-valid top-5 that does NOT contain the GT (reg:3) + gen = make_generate( + stage1_out=json.dumps({"candidates": ["1", "2", "4", "5", "6"]}), + stage2_out=json.dumps({"answer": "1"}), + ) + out = run_one(row, seed=1, generate=gen, **ctx) + assert out.stage1_contract_valid + assert out.recalled is False # GT not among predicted + # Stage 2 is still built from the PREDICTED top-5 (never from the GT bundle) + assert out.stage2_attempted + assert out.stage2_prompt_candidates == ("reg:1", "reg:2", "reg:4", "reg:5", "reg:6") + assert out.predicted_top5 == ("reg:1", "reg:2", "reg:4", "reg:5", "reg:6") + assert out.e2e_correct is False + metrics = aggregate_true_e2e([out]) + assert metrics["stage1_recall_at_5"] == 0.0 + # conditional accuracy excludes (denominator) the recalled==False source + assert metrics["stage2_conditional_accuracy"] == 0.0 + assert metrics["true_e2e_accuracy"] == 0.0 + + +def test_malformed_stage1_never_raises(ctx): + row = source_row(gt="reg:3") + gen = make_generate(stage1_out="not json at all", stage2_out="unused") + out = run_one(row, seed=1, generate=gen, **ctx) + assert not out.stage1_format_valid + assert not out.stage1_contract_valid + assert not out.recalled + assert not out.stage2_attempted # never prompts Stage 2 without a valid top-5 + assert out.e2e_correct is False + assert out.failures # structured error, not an exception + + +def test_malformed_stage2_never_raises(ctx): + row = source_row(gt="reg:3") + gen = make_generate( + stage1_out=json.dumps({"candidates": ["3", "1", "2", "4", "5"]}), + stage2_out="also not json", + ) + out = run_one(row, seed=1, generate=gen, **ctx) + assert out.recalled and out.stage2_attempted + assert not out.stage2_format_valid and not out.stage2_contract_valid + assert out.e2e_correct is False + assert out.failures + metrics = aggregate_true_e2e([out]) + assert metrics["stage2_format_failure_rate"] == 1.0 + assert metrics["stage2_contract_failure_rate"] == 1.0 + + +def test_stage2_candidates_exactly_equal_predicted_top5(ctx): + """The dynamic Stage-2 prompt must use the Stage-1 PREDICTED top-5, not the + pre-constructed gold bundle (this is the definitional difference from the + factorized/proxy evaluator).""" + row = source_row(gt="reg:3") + predicted = ["3", "1", "2", "4", "5"] + captured = {} + + def generate(messages): + user_content = messages[1]["content"] + if "Candidate bundle:" in user_content: + captured["stage2_user"] = user_content + return json.dumps({"answer": "1"}) + return json.dumps({"candidates": predicted}) + + out = run_one(row, seed=1, generate=generate, **ctx) + assert out.stage2_prompt_candidates == out.predicted_top5 == ( + "reg:3", "reg:1", "reg:2", "reg:4", "reg:5", + ) + bundle = json.loads(captured["stage2_user"].split("Candidate bundle:\n", 1)[1].split("\nField metadata:")[0]) + entry_ids = [entry["id"] for entry in bundle] + # bundle ids are the LOCAL positions over the PREDICTED top-5 + assert entry_ids == ["1", "2", "3", "4", "5"] + # names are the registry DISPLAY names (like the exported parquet), + # descriptions/examples come from the canonical corpus by category_id + names = [entry["name"] for entry in bundle] + assert names == list(out.predicted_top5) + assert bundle[0]["description"] == "description-reg:3" + assert "extra-reg:3" in bundle[0]["descriptions"] + assert "example-reg:3" in bundle[0]["examples"] + + +def test_aggregate_separates_conditional_and_true_e2e(ctx): + gen_hit_correct = make_generate( + stage1_out=json.dumps({"candidates": ["3", "1", "2", "4", "5"]}), + stage2_out=json.dumps({"answer": "1"}), + ) + gen_hit_wrong = make_generate( + stage1_out=json.dumps({"candidates": ["3", "1", "2", "4", "5"]}), + stage2_out=json.dumps({"answer": "2"}), + ) + gen_miss = make_generate( + stage1_out=json.dumps({"candidates": ["1", "2", "4", "5", "6"]}), + stage2_out=json.dumps({"answer": "1"}), + ) + outcomes = [ + run_one(source_row(gt="reg:3", source_id="a"), seed=1, generate=gen_hit_correct, **ctx), + run_one(source_row(gt="reg:3", source_id="b"), seed=1, generate=gen_hit_wrong, **ctx), + run_one(source_row(gt="reg:3", source_id="c"), seed=1, generate=gen_miss, **ctx), + ] + metrics = aggregate_true_e2e(outcomes) + assert metrics["sources"] == 3 + assert metrics["stage1_recalled_count"] == 2 + assert metrics["stage2_conditional_accuracy"] == pytest.approx(1 / 2) # 1 of 2 recalled correct + assert metrics["true_e2e_accuracy"] == pytest.approx(1 / 3) # 1 of 3 sources e2e correct From 7ed073d4fbc6769ac6d0adf2879f3b264e2805af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9B=BE=E7=AB=8B=E5=AE=8F?= <曾立宏@buaa.edu.cn> Date: Wed, 19 Aug 2026 23:11:22 +0800 Subject: [PATCH 2/6] docs(freeze): add real per-stage I/O examples and true-E2E success/fail traces (pers_info) --- docs/design/framework-freeze.md | 117 ++++++++++++++++++++++++++++++-- 1 file changed, 110 insertions(+), 7 deletions(-) diff --git a/docs/design/framework-freeze.md b/docs/design/framework-freeze.md index 1ba7558..8460ddc 100644 --- a/docs/design/framework-freeze.md +++ b/docs/design/framework-freeze.md @@ -23,15 +23,11 @@ - 所有正式实验统一使用 **Qwen/Qwen2.5-7B-Instruct**（小规模功能链路可用 Qwen2.5-0.5B-Instruct 作为 config 开关打通，不改变语义）。 -- RL 共同初始化默认 **SFT final = `global_step_140`（merged HF `merged_step140`）** - （Phase 13）；该选择仅作起点，后续若完成更完整的 SFT 训练，再统一重选一个 - SFT checkpoint，选定后同样冻结。 -- **不得用 test 指标选择 checkpoint**（Phase 13.5 起为硬约束）。 ## 3. 完整流程清单（数据处理 → 训练 → 评估 → 其他 RL 算法） ```text -学长预处理/corpus +预处理/corpus └─> data//canonical/all.json + split JSON（split 按 id 隔离） ├─> script/verl/sft/export （canonical → SFT messages parquet，choice protocol） ├─> script/verl/rl/export （canonical → RL 五字段 parquet） @@ -51,7 +47,90 @@ RL（在冻结层之上只改算法层）： └─ algorithm.* / actor.* / rollout.*（launcher 层，可换） ``` -**实现其他 RL 算法（RLOO / ReMax / PPO-DAPO）的改动面**（各 4B/12 报告约定）： +### 3.1 各阶段输入/输出示例（真实数据 · pers_info） + +**(1) 输入 · 数据处理前（raw canonical record）** + +```json +{ + "id": "f374612b-7a1b-52c4-97b0-fb0851603dd6", + "key": "dpname", + "label_status": "labeled", + "metadata": { "database_name": "CCENSE", "table_name": "M_BASE_CUSTDEPT", + "field_name": "DPNAME", "field_description": "部门" }, + "classification": { "level_1": "", "level_4": "学校概况基本信息" }, + "resolution_status": "resolved", + "target": { "leaf_level": "level_4", "leaf_name": "学校概况基本信息", + "category_id": "pers_info:学校概况基本信息" } // 唯一 label 来源 +} +``` + +**(2) 处理后 · SFT parquet 行（`data/sft/pers_info/test.parquet`, stage1）** + +```text +stage=stage1 source_id=016e8ce6-… ground_truth=pers_info:教职工个人基本信息 +messages[0] system = You are a leaf-category candidate retriever. …（见 (3)） +messages[1] user = （见 (3) 的确切 user prompt） +messages[2] assistant (gold supervision, choice-id JSON) = {"candidates":["2","11","4","3","1"]} +``` + +**(3) Stage 1 prompt + output** + +system（逐字）： +``` +You are a leaf-category candidate retriever. Return exactly one JSON object with key "candidates". The value must contain exactly five unique choice ids from the catalog. Do not output Markdown, commentary, canonical category ids, or any other keys. +``` +user（逐字，目录节选：完整 18 项为 `[choice_id, display_name]`，无 description/无 canonical id）： +```json +Retrieve five candidate leaf categories from this catalog: +[["1", "人力资源数据"], ["2", "任课信息"], ["3", "基本信息年级信息和班级信息"], …(共 18 项)…, ["18", "课程信息"]] +Field metadata: +{"field_name":"eid","field_description":"指导老师工号"} +``` +模型输出（SFT final 实际, hit case）→ choice decode → canonical Top-5： +```json +{"candidates":["4","1","2","3","11"]} + "4" → pers_info:学历学位信息 "1" → pers_info:人力资源数据 + "2" → pers_info:任课信息 "3" → pers_info:基本信息年级信息和班级信息 + "11" → pers_info:教职工个人基本信息 ← GT ∈ Top-5 ✓ +``` +malformed 输出（不 crash，判 format/contract fail → reward 0）： +```json +{"candidates":[{"id":17,"name":"职称信息"}, …]} // 旧 id/name 对象形状，非 choice-id 字符串数组 +``` + +**(4) Stage 2 prompt（由 Stage1 预测 Top-5 动态构建）+ output** + +system（逐字）： +``` +You are a leaf-category reranker. Return exactly one JSON object with key "answer". Its value must be one of the five candidate ids "1" through "5". Do not output Markdown, commentary, or any other keys. +``` +user（逐字 — bundle 的 5 个 candidate **= 上面预测的 Top-5**，local id 1..5，description/examples 取自 canonical corpus）： +```json +Candidate bundle: +[{"id":"1","name":"学历学位信息","description":"","descriptions":[],"examples":[]},{"id":"2","name":"人力资源数据",…},{"id":"3","name":"任课信息",…},{"id":"4","name":"基本信息年级信息和班级信息",…},{"id":"5","name":"教职工个人基本信息",…}] +Field metadata: +{"field_name":"eid","field_description":"指导老师工号"} +``` +输出 → local bundle-id decode： +```json +correct : {"answer":"5"} → 预测 Top-5[4] = pers_info:教职工个人基本信息 == GT → e2e TRUE +wrong : {"answer":"1"} → 预测 Top-5[0] = 学历学位信息 != GT → e2e FALSE +malformed : "1"（裸 id，base 常见）→ format fail → 0，不 crash +``` + +**(5) 处理后 · RL parquet 行（`data/rl/pers_info/train.parquet`，五字段，brief）** + +```text +data_source = pers_info/stage1 | pers_info/stage2 +prompt = [system, user]（与 SFT 相同 choice-protocol 文本，无 assistant gold） +ability = data_classification +reward_model = {"ground_truth": "pers_info:…", "style": "rule"} +extra_info = {"candidates": null | [5 个 canonical id], "dataset": "pers_info", + "metadata": {"field_name":…, "field_description":…}, "source_id": …} +``` + +**实现其他 RL 算法的改动面**（各 4B/12 报告约定）： - 只改 `script/verl/rl/grpo_smoke.sh` 的算法/采样配置：`algorithm.adv_estimator` （grpo→dapo/rloo），RLHF-style 需加 critic/ref/`use_kl_loss` 相关配置。 - rollout 后端 / 显存旋钮在 launcher env（`ENFORCE_EAGER` / `PARAM_OFFLOAD` / @@ -61,7 +140,7 @@ RL（在冻结层之上只改算法层）： - **不需要改**：canonical/parquet/parser/prompt/reward 表/evaluator/VeRL 源码 （零 vendored patch，兼容只靠 pip 依赖 + 配置）。 -## 4. True-E2E evaluator（阶段 13.5 新增） +## 4. True-E2E evaluator `script/verl/sft/evaluate_true_e2e.py`：raw test → `build_stage1_prompt` → greedy → `check_stage1_choices` decode 真实 canonical Top-5 → 按这 5 个 @@ -72,3 +151,27 @@ Top-5）+ format/contract 失败率；true E2E acc。metric 命名： **Proxy E2E**（`evaluate_baseline` 的 `proxy_e2e`）与 **True E2E**（本 evaluator）必须区分。 单测：`tests/evaluation/test_evaluate_true_e2e.py`（stage1 miss / hit+correct / hit+wrong / malformed 不 crash / candidates==预测 Top-5）。 + +### 4.1 true-E2E 样例（SFT final · pers_info test · greedy/seed=42） + +**【success】source=fb23c995-… gt=pers_info:教职工个人基本信息** +```text +stage1 输出 : {"candidates":["4","1","2","3","11"]} + → decode Top-5: [学历学位信息, 人力资源数据, 任课信息, 基本信息年级信息和班级信息, 教职工个人基本信息] ← 含 GT ✓ +动态 stage2 : 用该 Top-5 构建 bundle（见 §3.1(4)，bundle id = 该 Top-5 的位置） +stage2 输出 : {"answer":"5"} → 断言 stage2 candidates == 预测 Top-5 ✓ +final : pers_info:教职工个人基本信息 == GT → TRUE E2E ✓ +``` + +**【fail：Stage1 miss】source=016e8ce6-… gt=pers_info:教职工个人基本信息** +```text +stage1 输出 : {"candidates":["3","4","2","1","17"]} + → decode Top-5: [基本信息年级信息和班级信息, 学历学位信息, 任课信息, 人力资源数据, 职称信息] ← 不含 GT ✗ +stage2 输出 : {"answer":"4"} → 人力资源数据（预测 bundle 第 4 项） +final : pers_info:人力资源数据 ≠ GT （GT 不在候选，Stage2 无法命中）→ TRUE E2E ✗ +``` + +**【fail：malformed stage2】（stage1 命中但 stage2 非 JSON）** +```text +stage1 输出含 GT，stage2 输出裸 "1"（base 常见）→ format/contract fail → reward 0 → TRUE E2E ✗（不 crash，结构化失败） +``` From 2f653f22f328663198974e4e6a37a8f9336d87c3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9B=BE=E7=AB=8B=E5=AE=8F?= <曾立宏@buaa.edu.cn> Date: Wed, 19 Aug 2026 23:19:16 +0800 Subject: [PATCH 3/6] docs(freeze): label the field metadata as the data-to-classify + raw->prompt visibility mapping --- docs/design/framework-freeze.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/design/framework-freeze.md b/docs/design/framework-freeze.md index 8460ddc..8f5d577 100644 --- a/docs/design/framework-freeze.md +++ b/docs/design/framework-freeze.md @@ -51,6 +51,10 @@ RL（在冻结层之上只改算法层）： **(1) 输入 · 数据处理前（raw canonical record）** +> **待分类数据是什么**：一个数据库**字段（column）**。原始记录 metadata 存全量字段信息 +> （库/表名、字段名、字段描述、类型、值），经 `TaskConfig.metadata_fields` 显式裁剪后 +> 才进入 prompt（见 (3)，pers_info/finance/infra 均只暴露 `field_name, field_description`）。 + ```json { "id": "f374612b-7a1b-52c4-97b0-fb0851603dd6", @@ -87,6 +91,13 @@ Retrieve five candidate leaf categories from this catalog: Field metadata: {"field_name":"eid","field_description":"指导老师工号"} ``` + +> **待分类数据（在 input 中的位置）＝ user prompt 末尾 `Field metadata` 块**： +> 它就是上面那条 **字段**。原始记录 metadata 为 +> `{"database_name":"USR_DATAI","table_name":"T_JW_BKSCXCY","field_name":"eid", +> "field_description":"指导老师工号","field_type":"STRING", …}`， +> 经 `TaskConfig.metadata_fields` 只暴露前两个（其余如库/表名、字段类型、值都不进 prompt）。 +> 模型的任务=**仅凭这个字段（名+描述）从目录里选出 5 个候选叶类**。 模型输出（SFT final 实际, hit case）→ choice decode → canonical Top-5： ```json {"candidates":["4","1","2","3","11"]} @@ -130,7 +141,7 @@ extra_info = {"candidates": null | [5 个 canonical id], "dataset": "pers_info "metadata": {"field_name":…, "field_description":…}, "source_id": …} ``` -**实现其他 RL 算法的改动面**（各 4B/12 报告约定）： +**实现其他 RL 算法的改动面**： - 只改 `script/verl/rl/grpo_smoke.sh` 的算法/采样配置：`algorithm.adv_estimator` （grpo→dapo/rloo），RLHF-style 需加 critic/ref/`use_kl_loss` 相关配置。 - rollout 后端 / 显存旋钮在 launcher env（`ENFORCE_EAGER` / `PARAM_OFFLOAD` / From 6b8714ce265e399870f0ee60e8b5b836f8fc40e5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9B=BE=E7=AB=8B=E5=AE=8F?= <曾立宏@buaa.edu.cn> Date: Wed, 19 Aug 2026 23:58:10 +0800 Subject: [PATCH 4/6] docs(freeze): narrow freeze scope to classification pipeline; data_level/grading declared not-frozen (pending clarification) --- docs/design/framework-freeze.md | 37 ++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/docs/design/framework-freeze.md b/docs/design/framework-freeze.md index 8f5d577..5188568 100644 --- a/docs/design/framework-freeze.md +++ b/docs/design/framework-freeze.md @@ -1,12 +1,15 @@ -# Framework Freeze（阶段 13.5） +# Classification Framework Freeze（阶段 13.5） -本文件把两阶段叶分类工程的关键接口声明为 **frozen**。冻结后，后续算法实验 -（GRPO / RLOO / ReMax / PPO-DAPO 等）除真正 bug 外**不得修改**这些层； -算法差异只允许落在 launcher / Hydra 配置 / 算法层（见 §完整流程清单）。 +本文件把**分类（classification）管道**的关键接口声明为 **frozen**（classification +pipeline / classification contract **frozen for experiments**）。冻结范围**只覆盖 +classification**（§1 清单）；**data_level / grading 不在冻结范围内**（见 §1.1，语义 +待确认）。冻结后，后续算法实验（GRPO / RLOO / ReMax / PPO-DAPO 等）除真正 bug 外 +**不得修改**这些层；算法差异只允许落在 launcher / Hydra 配置 / 算法层（见 +§完整流程清单）。 冻结日期：Phase 13.5（2026-08-19，Phase 11/12/13 smoke 与 7B SFT baseline 均已 PASS）。 -## 1. Frozen contracts（冻结接口清单） +## 1. Frozen contracts（分类管道冻结接口清单） | # | 层 | 内容 / 入口 | 不得修改的点 | |---|---|---|---| @@ -19,7 +22,25 @@ | 7 | VeRL reward adapter | `agent.training.rl.verl_adapter.compute_score` | choice-aware 路由（`/stage<1|2>`），不实现 parser/reward 表 | | 8 | evaluation protocol | `agent.evaluation.classification`（`evaluate_stage{1,2}_choices`）+ `script/verl/sft/evaluate_baseline.py`（factorized/proxy）+ `script/verl/sft/evaluate_true_e2e.py`（true E2E） | 指标定义：stage1 format/contract/Recall@5；stage2 conditional / format/contract；true E2E（Stage2 用 Stage1 预测 Top-5） | -## 2. 模型起点与 RL 初始化（冻结） +### 1.1 Not frozen / pending clarification（data_level / grading） + +以下内容**当前不冻结**，仓库内语义 / 契约未定： + +- `data_level` 语义（L1–L4 业务定义 / 分级规则 / 标准文档）——仓库无正式定义，UNKNOWN +- grading target schema（分级目标的数据结构） +- grading prompt / output contract（分级提示词与输出格式） +- grading parser / evaluator（分级解析与评估） +- grading reward（分级奖励） +- classification + grading 联合指标（分类与分级联合评估） + +> `data_level` is currently preserved as source provenance only. Its promotion to a +> supervised grading target is pending confirmation of the L1-L4 business +> definition and task protocol. + +未来若加入 grading，是在 **frozen classification contract 之上新增一个 grading +contract**，而不是重新设计 classification pipeline；classification 层保持冻结不动。 + +## 2. 模型起点与 RL 初始化（分类实验约束） - 所有正式实验统一使用 **Qwen/Qwen2.5-7B-Instruct**（小规模功能链路可用 Qwen2.5-0.5B-Instruct 作为 config 开关打通，不改变语义）。 @@ -41,9 +62,9 @@ SFT 训练：script/verl/sft/run_baseline.sh → verl.trainer.sft_trainer（LoRA ├─> script/verl/sft/evaluate_baseline.py — factorized/proxy 指标（Stage2 用 parquet 预构造 bundle） └─> script/verl/sft/evaluate_true_e2e.py — true E2E（Stage2 由 Stage1 预测 Top-5 动态构建） -RL（在冻结层之上只改算法层）： +RL（在冻结的分类层之上只改算法层）： script/verl/rl/grpo_smoke.sh -> verl.trainer.main_ppo - ├─ reward.custom_reward_function.path=pkg://agent.training.rl.verl_adapter（冻结） + ├─ reward.custom_reward_function.path=pkg://agent.training.rl.verl_adapter（分类路径，frozen） └─ algorithm.* / actor.* / rollout.*（launcher 层，可换） ``` From 6a1156116f81e2edd823787779322211bfedcb1a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9B=BE=E7=AB=8B=E5=AE=8F?= <曾立宏@buaa.edu.cn> Date: Thu, 20 Aug 2026 00:49:58 +0800 Subject: [PATCH 5/6] docs(freeze): PR review - unfreeze synthetic candidate fixture & provisional reward coefficient; grading additive-extension; RL-init wording (no specific checkpoint frozen) --- docs/design/framework-freeze.md | 36 +++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/docs/design/framework-freeze.md b/docs/design/framework-freeze.md index 5188568..b2302eb 100644 --- a/docs/design/framework-freeze.md +++ b/docs/design/framework-freeze.md @@ -15,10 +15,10 @@ classification**（§1 清单）；**data_level / grading 不在冻结范围内* |---|---|---|---| | 1 | canonical category_id | `agent.task.contracts.LeafRegistry` / `SampleTarget`；`data//canonical/all.json` | 唯一身份：`target.category_id`；`classification.level_*` 仅 provenance，禁止 reintroduce level_4 fallback | | 2 | PromptChoice protocol | `agent.task.prompt_choices.PromptChoiceRegistry`（global "1".."N" / stage2 local "1".."5"） | choice id 映射、display name 规则；canonical id 永不进 prompt/不作为模型动作 | -| 3 | candidate policy | `agent.task.prompts.build_stage1_prompt`（全 registry）/ `build_stage2_prompt`（5 candidate bundle，按 category_id 查 corpus） | 现行为准：Stage1 全目录无 description；Stage2 bundle=5；hard-negative 策略不抽象（沿用 GT+registry 前四 fixture 策略） | +| 3 | candidate / Stage 输入输出契约 | `agent.task.prompts.build_stage1_prompt` / `build_stage2_prompt` | 冻结的是**契约**：Stage1 恰好输出 Top-5 candidate；decode 后为 canonical category_ids；Stage2 恰好接收 5 个 predicted candidate category_ids；Stage2 候选顺序决定 local choice id 1..5。**不冻结**：离线合成候选构造（build_candidates 为 fixture 而非生产检索策略）、负采样 / hard-negative 策略、候选采样改进（见 §1.2） | | 4 | parquet schema | SFT：`messages/stage/source_id/metadata/ground_truth/candidates`；RL：`data_source/prompt/ability/reward_model/extra_info` | 列名与语义；label 唯一来源 `target.category_id`；assistant supervision = choice-id JSON | | 5 | parser / decode | `agent.task.parser.check_stage1_choices / check_stage2_choices / check_stage{1,2}_output` | 严格 JSON + 精确 schema + 无 fuzzy fallback；decode 到 canonical 后才进入正确性逻辑 | -| 6 | reward contract | `agent.training.rl.reward`（RewardConfig 0/0.3/1.0 + stage2_partial(0.5 provisional)） | 数值表与 `reward_stage{1,2}_choices` / `reward_for_choice_result`（decode 先行，表不变） | +| 6 | reward contract（框架） | `agent.training.rl.reward` + `reward_for_choice_result` | 冻结：choice 输出→严格 parser→choice decode→canonical category_id→reward adapter 路由→malformed/constraint-invalid 处理框架。**不冻结**：仍标 provisional 的 reward 系数（如 stage2_partial=0.5 数值）与后续由任务定义确认的 reward weighting（见 §1.3）；当前 reward.py 数值不动 | | 7 | VeRL reward adapter | `agent.training.rl.verl_adapter.compute_score` | choice-aware 路由（`/stage<1|2>`），不实现 parser/reward 表 | | 8 | evaluation protocol | `agent.evaluation.classification`（`evaluate_stage{1,2}_choices`）+ `script/verl/sft/evaluate_baseline.py`（factorized/proxy）+ `script/verl/sft/evaluate_true_e2e.py`（true E2E） | 指标定义：stage1 format/contract/Recall@5；stage2 conditional / format/contract；true E2E（Stage2 用 Stage1 预测 Top-5） | @@ -40,10 +40,39 @@ classification**（§1 清单）；**data_level / grading 不在冻结范围内* 未来若加入 grading，是在 **frozen classification contract 之上新增一个 grading contract**，而不是重新设计 classification pipeline；classification 层保持冻结不动。 +**grading 允许 additive 扩展**：若后续确认 Stage2 需同时输出 classification + grading， +允许对 Stage2 output schema 做 **additive** 扩展，例如 +`{"answer":"3","data_level":"L2"}`。冻结的是「`answer` choice → canonical category_id」 +映射与分类正确性语义，**不是**「Stage2 JSON 只能有 answer 一个字段」。classification +identity / PromptChoice 分类映射不变；grading 契约后续作为新增契约引入。 + +### 1.2 Not frozen：synthetic candidate construction（fixture，非生产策略） + +当前 SFT/RL 数据流用 `build_candidates()`（GT + registry 前四个 negatives + +deterministic permutation）为样本合成候选；源码明确它属 **fixture，不是生产 Stage1 +检索策略**，因此**不冻结**，`build_candidates()` 本身也不改： + +- 离线 SFT/RL 合成候选构造 +- 负采样策略 / hard-negative 策略 +- 未来候选采样 / 召回改进（正式检索需按需求另行设计） + +候选相关冻结的只有 §1 表 row 3 的输入输出**契约**。 + +### 1.3 Not frozen：provisional reward coefficient + +- 冻结：reward **框架**（choice 输出→严格 parser→canonical decode→reward adapter + 路由→malformed/constraint-invalid 的 0/partial 映射框架）。 +- **不冻结**：仍标 provisional 的 reward 数值（例如 Stage-2 `stage2_partial` 当前默认 + 0.5）；最终权重待任务定义确认后由配置决定。当前 `reward.py` 数值不动。 + ## 2. 模型起点与 RL 初始化（分类实验约束） - 所有正式实验统一使用 **Qwen/Qwen2.5-7B-Instruct**（小规模功能链路可用 Qwen2.5-0.5B-Instruct 作为 config 开关打通，不改变语义）。 +- Phase 13 的 SFT checkpoint（如 `global_step_140`）是 reproducible SFT baseline / + candidate downstream-initialization artifact，**不冻结**任何具体 checkpoint；RL + 初始化选择属于实验配置（run config），不在本冻结契约内。 +- **不得用 test 指标选择 checkpoint**（Phase 13.5 起为硬约束）。 ## 3. 完整流程清单（数据处理 → 训练 → 评估 → 其他 RL 算法） @@ -68,6 +97,9 @@ RL（在冻结的分类层之上只改算法层）： └─ algorithm.* / actor.* / rollout.*（launcher 层，可换） ``` +> 候选构造（`build_candidates`：GT+registry 前四 negatives+固定 permutation）目前是 +> **合成 fixture**，非生产检索策略；正式候选采样 / 召回需按需求另行设计（见 §1.2）。 + ### 3.1 各阶段输入/输出示例（真实数据 · pers_info） **(1) 输入 · 数据处理前（raw canonical record）** From c8e38c9c4d80f970e1c134800b2d1c0af953f205 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9B=BE=E7=AB=8B=E5=AE=8F?= <曾立宏@buaa.edu.cn> Date: Thu, 20 Aug 2026 01:30:12 +0800 Subject: [PATCH 6/6] docs(freeze): check_stage2_choices stays frozen classification-only strict parser; grading resolved by a new joint parser reusing answer-choice decode --- docs/design/framework-freeze.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/design/framework-freeze.md b/docs/design/framework-freeze.md index b2302eb..8aac602 100644 --- a/docs/design/framework-freeze.md +++ b/docs/design/framework-freeze.md @@ -46,6 +46,12 @@ contract**，而不是重新设计 classification pipeline；classification 层 映射与分类正确性语义，**不是**「Stage2 JSON 只能有 answer 一个字段」。classification identity / PromptChoice 分类映射不变；grading 契约后续作为新增契约引入。 +**parser 不共用 / 不修改**：现有 `check_stage2_choices`（以及 `check_stage1_choices`） +是 frozen 的 **classification-only 严格 parser**，语义只覆盖 `answer choice → canonical +category_id` 分类解码。未来若加入 grading，**不修改**它；而是**新增**一个 +joint/grading-aware parser 来解析含 grading 字段的新输出 schema，并**复用**上述 +classification 语义（`answer` 字段仍走同一 choice → canonical 解码）。 + ### 1.2 Not frozen：synthetic candidate construction（fixture，非生产策略） 当前 SFT/RL 数据流用 `build_candidates()`（GT + registry 前四个 negatives +