You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
eval reference: stamp a digest of the questions file so headroom catches an edit at the same path #356
eval reference writes reference_scoring into an arms file: the scorer kind and the questions file it
was graded against. eval headroom (#279) compares the recorded scorer and questions path across inputs,
so two reference-scored files that name the same path are pooled as one measurement even when the file was
edited in between. The file's grading content is not in the record, so the edit is not seen.
Example: two runs scored against questions.json before and after an edit that changes the exact target of
one question. Both carry the same reference_scoring, the pool averages 100 and 0 and can hide a ceiling.
Proposal
Have eval reference stamp a digest of the questions file (the grading content, canonicalized) into reference_scoring in the arms file (an arms.v1 field), and have eval headroom compare that digest across
inputs and refuse on a mismatch (exit 2), as it does for the scorer kind now.
Changes: eval reference writes questions_sha256; eval headroom refuses a mismatch in the pooling guard;
old files without the digest keep the path check only.
Context: PR #279, thread PRRT_kwDOVBg7g86yYDIs (the open decision).
Activity