Skip to content

eval reference: stamp a digest of the questions file so headroom catches an edit at the same path #356

Open

Description

@napetrov

Problem

eval reference writes reference_scoring into an arms file: the scorer kind and the questions file it
was graded against. eval headroom (#279) compares the recorded scorer and questions path across inputs,
so two reference-scored files that name the same path are pooled as one measurement even when the file was
edited in between. The file's grading content is not in the record, so the edit is not seen.

Example: two runs scored against questions.json before and after an edit that changes the exact target of
one question. Both carry the same reference_scoring, the pool averages 100 and 0 and can hide a ceiling.

Proposal

Have eval reference stamp a digest of the questions file (the grading content, canonicalized) into
reference_scoring in the arms file (an arms.v1 field), and have eval headroom compare that digest across
inputs and refuse on a mismatch (exit 2), as it does for the scorer kind now.

Changes: eval reference writes questions_sha256; eval headroom refuses a mismatch in the pooling guard;
old files without the digest keep the path check only.

Context: PR #279, thread PRRT_kwDOVBg7g86yYDIs (the open decision).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions