Skip to content

Document benchmark metrics and publish checkpoint PSR reconstruction - #8

Merged
zomux merged 1 commit into
developfrom
codex/benchmark-metrics
Oct 17, 2026
Merged

zomux merged 1 commit into
developfrom
codex/benchmark-metrics

Conversation

@zomux

@zomux zomux commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

The PSR reconstruction used for the additional-model experiments existed only in the local checkout, leaving readers without its formula or an executable implementation. Add scripts/score_psr.py and docs/benchmark/metrics.md, link the guide from the README and scoring documentation, and explicitly retain the historical-reconstruction caveat.

The script preserves fractional checkpoint credit and equal task weighting. It supports arbitrary model names and relocated result folders, checks exact task coverage, rejects duplicate/invalid rows, and invalidates cached inputs when traces change. The guide also explains SR, CCE, Rounds/Chats, verifier provenance, and split/judge comparability.

Validation:

  • Eight offline PSR tests cover the worked example, clipping, booleans, round limits, weighting, complete/partial coverage, invalid inputs, relocated traces, and cache invalidation.
  • Recollected all 900 locally archived experiment trajectories: checkpoint credits and outcomes match exactly; task PSR differences are at most 5.55e-17, and all 12 split summaries match within 1e-12.
  • All 29 benchmark tests pass with runner/judge dependencies installed; all four scoring/runner CLI help checks pass.
  • Documentation file links and git diff --check pass.

No game/model calls were made, and historical result files were not modified. This does not certify equivalence to the unavailable historical PSR program or change the newer adjacent-task SR verifier protocol.

@zomux
zomux merged commit 23a07d5 into develop Oct 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant