Skip to content

feat(eval): Phase 13.5 classification framework freeze + true end-to-end evaluator - #12

Merged
xianyu9n merged 6 commits into
masterfrom
phase13-5-e2e-freeze
Aug 19, 2026
Merged

feat(eval): Phase 13.5 classification framework freeze + true end-to-end evaluator#12
xianyu9n merged 6 commits into
masterfrom
phase13-5-e2e-freeze

Conversation

@xianyu9n

@xianyu9n xianyu9n commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Phase 13.5 — Classification Framework Freeze + true end-to-end evaluator

已 rebase 到最新 master(PR #11 已合入)。本 PR 相对 master 只含 3 个 Phase-13.5 文件
docs/design/framework-freeze.mdscript/verl/sft/evaluate_true_e2e.py
tests/evaluation/test_evaluate_true_e2e.py。不重复包含 Phase 13 的 launcher/evaluator/merge tool。

Commits(rebase 后)

  • da8252d feat(eval): true end-to-end choice-protocol evaluator + unit tests + classification freeze doc
  • 7ed073d docs(freeze): 各阶段真实 I/O 示例 + true-E2E 成功/失败 trace
  • 2f653f2 docs(freeze): Field metadata 标注为待分类数据 + raw→prompt 可见性映射
  • 6b8714c docs(freeze): 冻结范围收窄为 classification pipeline;data_level/grading not-frozen
  • 6a11561 docs(freeze): PR review —— 合成候选 fixture / provisional reward 系数移出 frozen;grading additive 扩展;RL-init 措辞
  • c8e38c9 docs(freeze): PR review —— check_stage2_choices 保持 frozen classification-only 严格 parser;grading 由新增 joint/grading-aware parser(复用 answer-choice decode)处理

Changed files(相对 master)

  • script/verl/sft/evaluate_true_e2e.py — true E2E(Stage2 由 Stage1 预测 Top-5 动态构建;复用 build_stage1/2_prompt + check_stage{1,2}_choices + corpus)
  • tests/evaluation/test_evaluate_true_e2e.py — Stage1 miss / hit+correct / hit+wrong / malformed 不 crash / candidates==预测 Top-5
  • docs/design/framework-freeze.mdClassification Framework Freeze

Freeze 文档口径(本 PR 修改摘要)

  • Frozen(classification contract):canonical category_id · PromptChoice protocol · Stage1/2 分类契约 · candidate 输入输出契约(Stage1 输出 Top-5、decode 为 canonical;Stage2 接收 5 个 predicted category_id、顺序定 local id)· parser/decode · reward 框架 · SFT/RL parquet 分类契约 · VeRL adapter 分类路径 · true-E2E 分类评估协议
  • parser 口径(c8e38c9):现有 check_stage2_choices(及 check_stage1_choices)是 frozen 的 classification-only 严格 parser;未来加入 grading 不修改它,而是新增 joint/grading-aware parser,复用 answer choice → canonical category_id 分类语义
  • Not frozen(§1.2):synthetic candidate construction(build_candidates 为 fixture 非生产检索策略)、负采样 / hard-negative 策略、未来候选采样改进 —— build_candidates() 本身不改
  • Not frozen(§1.3):provisional reward 系数(Stage-2 stage2_partial=0.5 数值)、后续任务定义确认的 reward weighting —— reward.py 数值不动
  • Not frozen(§1.1):data_level 语义(L1–L4 UNKNOWN)、grading target/prompt/parser/evaluator/reward、分类+分级联合指标;data_level remains provenance only
  • grading additive 扩展:允许 Stage2 输出 schema 追加字段(如 {"answer":"3","data_level":"L2"});冻结的是「answer choice → canonical category_id」,不是「只能有 answer 一个字段」
  • RL 初始化:仅固定正式基模 Qwen/Qwen2.5-7B-Instruct;Phase 13 checkpoint = reproducible baseline / candidate downstream-init artifact,不冻结任何具体 checkpoint;不得用 test 选 checkpoint

验证

  • true-E2E(test n=18,greedy/seed 42):base 0.000 → SFT final 0.444 (8/18);Stage2 conditional acc 8/8=1.0;瓶颈=Stage1 Recall@5 0.444;proxy_e2e(PR11)与 true_e2e 明确区分
  • 测试口径:
  • 未改 prompt/reward/parser/data/parquet schema;data_level 未纳入 supervised target

曾立宏 added 5 commits August 20, 2026 00:47
…vel/grading declared not-frozen (pending clarification)
…isional reward coefficient; grading additive-extension; RL-init wording (no specific checkpoint frozen)
@xianyu9n
xianyu9n force-pushed the phase13-5-e2e-freeze branch from 6303b65 to 6a11561 Compare August 19, 2026 16:50
…trict parser; grading resolved by a new joint parser reusing answer-choice decode
@xianyu9n
xianyu9n merged commit 073240c into master Aug 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant