[antonwolfy]

Summary

Settles #291 (issue, decisions): dokimi:review_replay credits a human thread to the nearest finding on its file within window lines, by location alone, so a finding that only sits near a thread gets the credit.

On main there is no way to rescore a credited pair: repo report --hits lists the pairs with comment and message cut to 200 characters, and python cli.py repo hits judge ... exits 2 with invalid choice: 'hits'. In the test batch of this PR (two arms, one cell each, three gold threads), the grader credits, among others, a thread asking to add a TODO naming the follow-up issue (a 396-character comment) to a finding saying the TODO is repeated five times:

cell	credited pairs	recall (grader)	labels (scripted judge)	recall_judged
guidance:none r1	t_todo ↔ "The TODO is repeated five times", t_op ↔ "mul adds"	0.667	unrelated, same point	0.333
guidance:raw r1	t_todo ↔ "Add a TODO naming the issue", 7 ↔ "read() has no docstring"	0.667	same point, related	0.333

The labels in that table come from the tests' fake judge, not a real model; no real model was called for this PR.

The second commit (86c37d3) settles the review of this PR: the judge check missed the model that actually answered a cell, a labels file with no jointly labelled pair hid the uncalibrated warning, and a sheet entry could label a finding other than the credited one. On the same test batch, at 05ebb9a and at 86c37d3:

input	at 05ebb9a	at 86c37d3
a guidance:raw cell asked of claude-sonnet-5-5 with model_resolved: "claude-opus-5-5", --judge anthropic/claude-opus-5-5	exit 0, 4 judge calls, the cell judged by the model that answered it	exit 2, no call: ✗ judge anthropic/claude-opus-5-5 runs the same model as the answer model 'claude-opus-5-5' of this batch; ...
--labels with the untouched repo hits sheet file (every label: null)	exit 0, an empty calibration against the labels (person vs judge): heading, no uncalibrated line; JSON calibration: []	recall_judged is uncalibrated (no pair carries both a person's and the judge's label): ...; JSON calibration: [], uncalibrated_arms: ["guidance:none", "guidance:raw"]
a filled sheet with entry 1's finding changed to 99	exit 0, 4 judge calls; the label is compared against finding 0 as if the person had labelled it	exit 2, no call: ✗ <sheet> entry 1: the sheet labels finding 99 of thread 't_todo', but the batch credits finding 0 for it in cell ...; write a new sheet with `repo hits sheet`

The third commit (af4c4a4) settles a later review comment: a hits_judged.json that cannot be written, or an earlier one that cannot be removed when --max-usd stops its cell, escaped repo hits judge as a traceback with exit 1, while repo hits sheet reports an unwritable output with exit 2. On the same test batch with the fake judge at $0.40 a call, at 86c37d3 and at af4c4a4:

input	at 86c37d3	at af4c4a4
a directory where guidance:raw r1's hits_judged.json goes	traceback, exit 1: IsADirectoryError: [Errno 21] Is a directory: '<cell>/hits_judged.json.<pid>.<tid>.tmp' -> '<cell>/hits_judged.json'	exit 2: ✗ cannot write <cell>/hits_judged.json: Is a directory; stopped there after 4 judge call(s), $1.6000; the 1 cell(s) written before keep their hits_judged.json
guidance:raw r1's cell directory read-only (chmod 555)	traceback, exit 1: PermissionError: [Errno 13] Permission denied: '<cell>/hits_judged.json.<pid>.<tid>.tmp'	exit 2: ✗ cannot write <cell>/hits_judged.json: Permission denied; stopped there after 4 judge call(s), ...
--max-usd 1 with a directory in place of the stale file of the cell the budget stops	traceback, exit 1: IsADirectoryError: [Errno 21] Is a directory: '<cell>/hits_judged.json'	exit 2: ✗ cannot remove <cell>/hits_judged.json: Is a directory; stopped there after 3 judge call(s), $1.2000; the 1 cell(s) written before keep their hits_judged.json

The fourth commit (38e1da9) settles a later review comment: repo hits judge visits only the cells whose grade.json has matches, so a cell judged once and then regraded into a state load_hits skips kept its old hits_judged.json, which runs list / runs show still list. On the same test batch, after a first repo hits judge, with guidance:raw r1 regraded and the command run again, at af4c4a4 and at 38e1da9:

regraded into	at af4c4a4	at 38e1da9
a grader failure (result.json grade.status: grader_failure)	exit 0; the cell's earlier hits_judged.json stays; no stale_removed key, no line	exit 0; the file is removed before the first call; removed the earlier hits_judged.json of 1 cell(s) not judged now (a grader failure, a grade without hit details, another grader): review_replay_pr3098__guidance-raw__claude-sonnet-5-5__r1; JSON stale_removed: ["review_replay_pr3098__guidance-raw__claude-sonnet-5-5__r1"]
a grade with no matches (a parse error)	the same: the file stays	the same as above: the file is removed and named
a grader failure, with a directory in place of that file	exit 0, 2 judge calls (the other cell), the directory stays	exit 2, no call: ✗ cannot remove <cell>/hits_judged.json: Is a directory; stopped there after 0 judge call(s), $0.0000; the 0 cell(s) written before keep their hits_judged.json

The fifth commit (8e05118) settles napetrov's review, after a merge of main (6c52d10, the artifacts.py docstring list conflict resolved by keeping both kinds):

• --max-usd with an unpriced judge (comment): a reply with no known price counted as $0, so the cap never fired. Under --max-usd, such a reply now stops the run after that call, as novelty's cap does: a cap stop (exit 1) naming the judge model and saying --max-usd cannot be enforced; --json adds budget_stop. Without --max-usd the spend line says how many calls had no price (--json: unpriced_calls).
• The judge refusal (comment) now states the same-model rule the check enforces, not a family rule.
• Two different labels for one pair in --labels (comment) exit 2, naming both entries; the same label twice is taken.
• repo hits sheet --out (comment) refuses, exit 2 before writing, to replace a file holding a set label; an unlabelled sheet is replaced as before.

The sixth commit (790c0c1) settles a later review comment (comment): --labels checked an entry's cell, thread and finding index, not the texts the person labelled. When the credited finding's text in answer.txt changed with the batch still crediting the same index, or the thread's comment in the gold file changed, the old label was still compared with the judge's label of the new texts, and the arm read as calibrated. On the same test batch, with every pair of a fresh sheet labelled and then guidance:none r1 changed, repo hits judge --labels <sheet> --json at 8e05118 and at 790c0c1:

change after the sheet was labelled	at 8e05118	at 790c0c1
the message of the finding credited for t_op rewritten from "mul adds" to "mul adds, see #12" at the same line, and the cell regraded (still finding 1)	exit 0, 4 judge calls; calibration gives guidance:none 2 labelled pairs, the same as for the unchanged sheet	exit 2, no call: ✗ <sheet> entry 0: the finding's text of thread 't_op' of cell review_replay_pr3098__guidance-none__claude-sonnet-5-5__r1 is not the text the sheet labelled (a sheet older than a change to answer.txt or the gold file, or a hand-edit); write a new sheet with `repo hits sheet`
t_op's comment in the task's gold.json changed	exit 0, 4 judge calls, the same calibration	exit 2, no call: the same line, with the thread's comment
none (the sheet as labelled)	exit 0, 4 judge calls	the same: exit 0, 4 judge calls, the same calibration

Changes

• repo hits judge <batch> --judge PROVIDER/MODEL [--labels FILE] [--max-usd X] [--dry-run] [--json] (dokimi/repos/hits_judge.py, dokimi/commands/repo_hits.py): pairs come from repos/report.py::load_hits. For each pair it reads the thread's full comment from the task's gold file (the cell's task.json gold, default gold.json, in the task's directory under <batch>/_eval/content, as the built-in grader resolves it) and the finding's full message, rule and evidence from the cell's answer.txt via graders/review.py::findings_of at index finding. A pair whose text is not found (no single task directory, no such thread, no finding at the index, or a file other than the one matches names) is reported and left unjudged. One call per pair goes through dokimi.judged.judged_reply (redraws, response cache) and asks for {"label": "same point" | "related" | "unrelated", "reason": ...}. A reply that never parses is unjudged.
• Judge independence: before any call, a judge that runs the model of any answer of the batch is refused with exit 2. The answer models are meta.json and result.json model (the model asked for) and model_resolved (the models the trace says answered, a string or a list, read through report.resolved_models; 86c37d3), so a harness alias that mapped a cell to another model cannot hide it. The check reuses novelty.client.same_client (built on llm_cli.same_model), with each answer model read as a CLI ref, so claude-cli/sonnet is refused for a claude-sonnet-5-5 batch. repos and novelty are both in the tracks layer, and novelty imports nothing from repos, so this adds no upward import and no cycle.
• <cell>/hits_judged.json, a new artifact repo_hits_judged.v1 (dokimi/schemas/repo_hits_judged.v1.json, registered in artifacts.SCHEMA_VERSIONS, with its runs list/runs show fields in runs/kinds.py and a fixture). It holds the judge, each pair's thread, finding, label and reason (error when unjudged), the counts of same point, related, unrelated and unjudged, n_threads (from grade.json metrics), the grader's recall, and recall_judged = same point / n_threads. A rerun rewrites it, and the response cache replays the calls.
• --dry-run counts the pairs and prices one call each with dokimi.llm_pricing.list_cost_usd (prompt characters / 4, 100 reply tokens), with no call. --max-usd X starts no new live call once X is spent, or once a live call came back with no known price (8e05118); a cell not fully judged gets no file (an earlier one is removed), it is listed, and the command exits 1, as repo run does after --max-batch-usd.
• Output writes (af4c4a4): judge_cells stops at the first hits_judged.json it cannot write, or earlier one it cannot remove when the budget stops a cell, before any further judge call, and raises HitsWriteError (dokimi/repos/hits_judge.py) with the path, the OSError and what it had done, so a Python caller gets a typed error. cmd_repo_hits_judge turns it into one ✗ line on stderr naming the path and the error, the judge calls made and their cost, and that the cells written before keep their files, and exits 2 (EXIT_USAGE), as repo hits sheet does through commands/_output.exit_on_unwritable. The command keeps all its exit codes.
• Earlier files of cells no longer judged (38e1da9): before its first judge call, judge_cells removes the hits_judged.json of every finished cell of the batch (report.load_cells) that load_hits does not read: a cell regraded since into a grader failure, a grade without matches, or another grader. A removal that fails raises HitsWriteError, so the command exits 2 before it spends anything. --dry-run returns before judge_cells and removes nothing. The text summary names the cells (the first ten, then and N more, as repo report lists missing cells), and --json lists them in stale_removed ([] when none).
• Output: per (arm, model), the cells written, pairs, label counts, and mean recall and recall_judged. --json prints the same summary, plus uncalibrated_arms (86c37d3): the arms of rows with no pair labelled by both the person and the judge, which is every arm without --labels.
• Calibration (dokimi/repos/hits_calibration.py): repo hits sheet <batch> --out FILE [--per-arm N] [--seed N] makes no call. It writes a JSON list of pairs with their full texts and label: null, N per arm (default 25; all of an arm's pairs when it has fewer), drawn with random.Random(seed). An --out file that holds a set label is refused (exit 2), not replaced (8e05118). --labels FILE reads that file back. These entries exit 2 naming the entry, before any call: one that names a cell or thread of no credited pair, one with a label outside the three, two that give one pair different labels (8e05118), (86c37d3) one with no finding or with a finding other than the one the batch credits for its thread (a hand-edit, or a sheet older than a regrade), and (790c0c1) a labelled entry whose comment or finding_text is missing or is not, exactly (no normalisation), the pair's full text now, or whose pair's texts are no longer found (the message then names why, rather than a changed text). hits_calibration._check_texts does the check; an entry with label: null is not checked. The output then gives agreement and Cohen's kappa per arm (a few lines, no new dependency). Without --labels it prints recall_judged is uncalibrated: label a sample of pairs (repo hits sheet) .... So does a labels file where no pair carries both labels (an unfilled sheet, [], or labels only on pairs the judge left unjudged), with that reason added and no empty calibration heading. When only some arms are covered, a line names the others (recall_judged is uncalibrated in guidance:raw: ...); 86c37d3.
• load_hits also returns grader_failures and cells (each replay cell whose matches was read, with or without a pair, plus its n_threads and recall). Without cells, a cell with no credited pair would drop out of the means.
• CLI: --labels and --per-arm go on tests/data/cli_flags_allowlist.txt with a reason (no registry concept for them yet). --judge, --max-usd, --dry-run, --json, --out and --seed are canonical. Route table: repo hits judge is paid and repo hits sheet is free. The commands are registered from commands/repo_run.py::register (+4 lines), which passes in its _usd parser. docs/reference/cli-interface.json is regenerated.
• Docs: contract §4.5 (what judged recall is and is not, and that a figure needs labels before it is quoted; 86c37d3: no jointly labelled pair is no calibration) and §9 (both commands, their flags, files and exit codes, and the new --hits --json keys; 86c37d3: model_resolved in the judge check, the finding check of --labels, the uncalibrated line and uncalibrated_arms; af4c4a4: exit 2 when a cell's file cannot be written or removed; 38e1da9: the removal of earlier files of cells no longer judged, and stale_removed; 790c0c1: the text check of --labels); docs/architecture.md module entry.
• Changelog fragment (Added).

Not in this PR, as decided on the issue: reading recall_judged in repo compare (repos/compare.py is at its line cap; after #332), and judging near-misses outside the window.

Risk

Grading is unchanged: the grader, grade.json and repo report's table are as before. repo report --hits --json gains two keys (grader_failures, cells); the existing keys are unchanged. runs list --kind gains the choice repo_hits_judged. The new commands write only <cell>/hits_judged.json and the --out sheet. A judge call that raises (a provider or credential error) is not caught: the command stops with the error, and the cells already judged keep their files. That is how the CLI's top level treats a provider error for every command: it turns only input errors into an exit 2. The --labels file must carry each entry's finding, as repo hits sheet writes it. A hand-written file without it is refused, and so is a sheet written before a regrade moved a thread's credit. A labelled entry must also carry comment and finding_text equal to the pair's current texts (790c0c1): a sheet labelled before answer.txt or the gold file changed, or with a text edited by hand (whitespace included), is refused, and a new sheet must be written and labelled again. A hits_judged.json that cannot be written or removed now exits 2 with one line, not 1 with a traceback (af4c4a4). The run stops there as before, and the calls already paid for are in the response cache when it is on. A run that judges now also deletes the hits_judged.json of a cell whose current grade it does not read (38e1da9); a person who wants to keep such a file must copy it before rerunning the command. The --json summary gains stale_removed.

Testing

On a Linux review host, at 790c0c1 (on main 41b3d8d):

• make ci-local: 30 passed, 0 failed, 7 skipped. Skipped: external-catalog-smoke (needs docker and network), test-latest, docs.yml:deploy and pip-audit.yml:report (job conditions false on a PR), cli-surface / Package the result (an expression ci-local cannot evaluate), conda-forge-deps (no micromamba), and semgrep secrets (not installed). 12596 tests passed, 5 skipped and 6 xfailed; changed-line coverage against main was 94% (511 lines measured; every changed line of 790c0c1 is covered). The PR merges main twice: a1b8822 (at ad734f5; its three conflicts were both sides adding an artifact kind to artifacts.py, runs/kinds.py and tests/test_runs_scan.py) and 6c52d10 (at 41b3d8d; the artifacts.py docstring list), each resolved by keeping both.
• The tables above: each "at 05ebb9a", "at 86c37d3", "at af4c4a4" or "at 8e05118" row is what that commit prints on the test batch with the fake judge, and each row of the next commit is what that commit prints. All 14 new or changed cases of round two fail against 05ebb9a's code, the 2 new cases of af4c4a4 fail against 86c37d3's code, 3 of the 4 new cases of 38e1da9 fail against af4c4a4's code (the fourth, --dry-run removes nothing, holds there too, since af4c4a4 removes nothing), and the 3 new cases of 790c0c1 fail against 8e05118's code (each gets exit 0 and judge calls where it expects exit 2 and none).
• tests/test_repo_hits_judge.py (49 cases, fake LLM, offline): the prompt carries a 396-character comment in full and the finding's rule/evidence; recall_judged and label counts on known labels; a cell with no credited pair is written and counted in the means; an unparseable reply is unjudged after its 3 draws; a judge that is an answer model (same id, claude-cli/sonnet, and a claude-cli/haiku against a meta.json model) is refused with exit 2 and no call, with the same-model wording, while a haiku judge on a sonnet batch is taken; so is a judge that a cell's model_resolved names, as a string or a list, in result.json or in meta.json (4 cases); --dry-run makes no call and prices each pair; --max-usd 1 at $0.40 a call makes 3 calls, leaves one cell without a file (removing a stale one) and exits 1; unpriced replies under --max-usd 0.50 stop after 1 call with the reason, 5 calls with no cap, and a priced control stops at the cap; a missing text is reported and not judged; grader-failure and pre-matches cells are counted, not judged; kappa 0.531 and agreement 0.7 on a known 10-pair table; --labels per arm, plus refusals of an unknown thread, an unknown cell, a non-object entry, a fourth label and two different labels for one pair, each with no judge call; an all-null sheet, [], and labels only on unjudged pairs print the uncalibrated line and give uncalibrated_arms of both arms (text and JSON); labels on one arm name the other as uncalibrated (text and JSON); an entry whose finding is 99, or that has none, exits 2 with no judge call; new in 790c0c1: a labelled sheet that calibrates is refused, exit 2 and no judge call, once the credited finding's text changes at the same index and once the gold comment changes, with the entry, thread, cell and changed text in the message (2 cases); a labelled entry with no finding_text is refused, an entry with label: null and an edited comment is taken, and a labelled entry whose pair's answer.txt holds no findings any more is refused with that reason; the sheet is stratified, capped per arm, seeded, carries the full texts, refuses to replace a labelled sheet byte for byte and replaces an unlabelled one; the schema validates a written file and rejects a fourth label; a rerun with the response cache makes no call and writes the same file; a directory where a cell's hits_judged.json goes, and an earlier file whose removal raises PermissionError when --max-usd 1 stops its cell, each exit 2 with the path in the message, make no judge call after the failure (4 and 3 calls), and leave the cell written before in place; a judged cell regraded to a grader failure, and in another case to a grade with no matches, loses its hits_judged.json on the next run while the other cell keeps its own, with the removal named in the text and in stale_removed, and a third run reports stale_removed: [] (2 cases); a directory in place of that file exits 2 with the path, no judge call and the other cell's file kept; --dry-run removes nothing.
• Guard and touched tests at 790c0c1: test_repo_hits_judge, test_repo_report_hits: 61 passed; test_docs_code_refs, test_growth_ratchet, test_audit_test_quality, test_import_architecture: 292 passed. ruff check is clean and mypy passes on the whole package (317 files).
• Mutants of 790c0c1: 5 breaks in a copy of the tree, each failing a test: the text comparison dropped (both changed-text cases fail), the call of the check dropped (those two and the missing-key case fail), the not-found branch dropped and the missing-key branch dropped (the missing-key case fails for each), and the check run on null-labelled entries too (that case and the fourth-label case fail). The mutants of 8e05118, 38e1da9, af4c4a4, 86c37d3 and 05ebb9a were run at those commits.
• Bad inputs run by hand (at 05ebb9a): a missing batch directory and a Windows path (C:\Users\x\batch) exit 2 (✗ not a directory); --judge without a provider exits 2 listing the providers; a labels file that is not a JSON list, or missing, exits 2 naming the file; an empty batch dir gives 0 of 0 credited pair(s) for --dry-run and an empty sheet.
• Not run: a real judge model (no paid call from this host).
• Merged together at 790c0c1 with the branches for novelty: spread the atom cap across the artifact instead of its opening #293 (novelty cap spread, 82d576a), repo compare: flag replay tasks a guidance revision may have seen #290's recording half (f6a9cb9) onto main 41b3d8d: the full suite (12645 passed, 5 skipped, 6 xfailed), mypy (318 files) and ruff check pass.

Revert

Revert the PR's six commits (05ebb9a, 86c37d3, af4c4a4, 38e1da9, 8e05118, 790c0c1); the two merges of main need no revert. What they leave behind: hits_judged.json files in batch cells, which nothing else reads, and sheet files people wrote. A plain revert drops the repo_hits_judged.v1 schema with no superseded entry, so remove those files if anything is to validate them later.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi repo hits`
    + option `--help/-h`
+ added command `dokimi repo hits judge`
    + argument `batch_dir`
    + option `--dry-run`
    + option `--help/-h`
    + option `--json`
    + option `--judge`
    + option `--labels`
    + option `--max-usd`
+ added command `dokimi repo hits sheet`
    + argument `batch_dir`
    + option `--help/-h`
    + option `--out`
    + option `--per-arm`
    + option `--seed`
~ `dokimi runs list` parameters changed:
    ~ changed option `--kind`: choices: ["'answers'", "'arms'", "'cli_routes'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'guidance_reach'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'plan'", "'plugin_delta'", "'questions'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_readiness'", "'result'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"] -> ["'answers'", "'arms'", "'cli_routes'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'guidance_reach'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'plan'", "'plugin_delta'", "'questions'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_readiness'", "'result'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"]

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[Copilot]

🟡 Changes recommended

The judge exclusion check ignores resolved models, allowing self-judging, while calibration output can suppress warnings despite having no jointly labelled pairs.

2 open issues

• Exclude resolved judge models to prevent self-judging · New
• Warn when no jointly labeled pairs provide calibration evidence · New

What changed in this PR

2 new findings.

This PR adds optional judged recall to replay evaluation, checking whether location-credited findings actually match human review threads without changing deterministic grading.

Changes:

• Adds repo hits judge with full-text pairing, budget controls, cost estimates, and per-cell artifacts.
• Adds sampled human-label sheets and agreement statistics for calibration.
• Registers the artifact and CLI commands, with offline tests and documentation; repo compare integration remains deferred.

File	Description
dokimi/​tests/​test_runs_show.py	Adds judged-hit artifact display expectations.
dokimi/​tests/​test_runs_scan.py	Adds artifact discovery expectations.
dokimi/​tests/​test_repo_hits_judge.py	Tests judging, calibration, budgets, caching, and input validation.
dokimi/​tests/​test_cli_routes.py	Checks the paid judging route.
dokimi/​tests/​fixtures/​artifacts/​repo_hits_judged.v1.json	Provides a judged-hit artifact fixture.
dokimi/​tests/​data/​cli_flags_allowlist.txt	Allows calibration-specific flags.
dokimi/​dokimi/​schemas/​repo_hits_judged.v1.json	Defines the judged-hit artifact format.
dokimi/​dokimi/​runs/​kinds.py	Registers artifact display fields.
dokimi/​dokimi/​repos/​report.py	Exposes replay cells and grader-failure counts.
dokimi/​dokimi/​repos/​hits_judge.py	Implements text recovery, judging, artifacts, and summaries.
dokimi/​dokimi/​repos/​hits_calibration.py	Implements sampling and agreement statistics.
dokimi/​dokimi/​commands/​repo_run.py	Registers the hits subcommands.
dokimi/​dokimi/​commands/​repo_hits.py	Adds judge and sheet command handlers.
dokimi/​dokimi/​cli_routes.py	Classifies judging as paid and sampling as free.
dokimi/​dokimi/​artifacts.py	Registers the new artifact schema.
dokimi/​docs/​repo-evaluation-contract.md	Documents judged recall, calibration, and command behavior.
dokimi/​docs/​reference/​cli-interface.json	Updates the generated CLI reference.
dokimi/​docs/​architecture.md	Documents the new replay-judging modules.
dokimi/​changelog.d/​20261101_120000_antonwolfy_replay-judged-recall.md	Announces judged recall and calibration commands.

_{Review effort: Balanced}

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

Review details

• Commit reviewed: 790c0c1 against merge base 8e05118
• Coverage: 4 files in 1 groups
• Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
• Usage: 63k tokens (16k from cache), 12s
• Tool calls: 5 (5 file_read)
• Replay judged recall calibration, evaluation contract, and tests: 4 files

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 91.19374% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.25%. Comparing base (41b3d8d) to head (790c0c1).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/hits_judge.py	90.06%	19 Missing and 12 partials ⚠️
dokimi/dokimi/commands/repo_hits.py	91.57%	6 Missing and 2 partials ⚠️
dokimi/dokimi/repos/hits_calibration.py	93.54%	5 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #357      +/-   ##
==========================================
- Coverage   94.29%   94.25%   -0.05%     
==========================================
  Files         308      311       +3     
  Lines       39322    39830     +508     
  Branches     6651     6718      +67     
==========================================
+ Hits        37079    37540     +461     
- Misses       1430     1462      +32     
- Partials      813      828      +15     

Files with missing lines	Coverage Δ
dokimi/dokimi/artifacts.py	95.12% <ø> (ø)
dokimi/dokimi/cli_routes.py	100.00% <ø> (ø)
dokimi/dokimi/commands/repo_run.py	94.41% <100.00%> (+0.03%)	⬆️
dokimi/dokimi/repos/report.py	97.81% <100.00%> (+0.04%)	⬆️
dokimi/dokimi/runs/kinds.py	100.00% <ø> (ø)
dokimi/dokimi/repos/hits_calibration.py	93.54% <93.54%> (ø)
dokimi/dokimi/commands/repo_hits.py	91.57% <91.57%> (ø)
dokimi/dokimi/repos/hits_judge.py	90.06% <90.06%> (ø)

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review rounds at 38e1da9, one commit per round, each thread answered and resolved:

• 86c37d3: the judge check reads model_resolved too, so a model that answered a cell under another requested name cannot judge it. A labels file where no pair carries both labels (an untouched sheet, [], labels only on unjudged pairs) keeps the uncalibrated line, and --json adds uncalibrated_arms. A sheet entry whose finding is missing or is not the credited one exits 2 before any call.
• af4c4a4: a cell's hits_judged.json that cannot be written or removed stops the run before any further call and exits 2, naming the path and what was spent.
• 38e1da9: before the first call, the hits_judged.json of a cell the run no longer reads (a grader failure, a grade without matches) is removed and reported (stale_removed).

make ci-local is green at each commit. With #358 and #359 at their heads, merged together onto main ff646be, the full suite (12203 passed), mypy and ruff pass. The description is updated.

[antonwolfy]

a1b8822 merges main at ad734f5 to resolve the conflict with #340. Both sides added an artifact kind at the end of the same lists: the artifacts.py docstring and SCHEMA_VERSIONS, runs/kinds.py FIELDS, and tests/test_runs_scan.py EXPECTED. Each list keeps main's need_graph and eval_coverage entries, then repo_hits_judged. The auto-merged docs/reference/cli-interface.json is what scripts/cli_surface.py dump writes for the merged tree. Against main, the merged branch changes the same 19 files by the same line counts as the PR did against its base.

make ci-local is green at a1b8822 (30 passed, 0 failed, 12208 tests). With #358 and #359 at their heads, merged together onto main d3e2a26, the full suite (12331 passed), mypy and ruff pass. The description's Testing section is updated.

[napetrov]

Verdict: comment only, no blocking finding. 0 blocking, 2 questions, 2 nits, all introduced by this PR (code new to it, so probes ran on head only). Head a1b8822, merge base ad734f5, lane L4 (a new persisted format repo_hits_judged.v1 and new CLI leaves).

CI: 0 failing Actions jobs on head; the merge base is behind the base tip 41b3d8d, which is not a failure.

Raised before: the Copilot and exai threads T1–T7 are fixed on head (checked at each line). The four findings here are not among them.

Claims checked: C1–C8 implemented, no contradiction in the PR body. The one contradicted claim is the refusal text, not a PR-body claim: see the nit on hits_judge.py.

Unverified: the removal of hits_judged.json for a cell with no grade.json (not probed); the live judge (not run, no paid call); the context traces line of cli-interface.json (not checked against base).

Coverage: checked the 19 files of the diff on head, the grader pairing in graders/review.py, and probes V1–V4 on head. Not looked at: a live judge, the full test suite (the edge pass ran test_repo_hits_judge.py: 38 passed on head).

[antonwolfy]

Review rounds at 790c0c1, after a merge of main at 41b3d8d (6c52d10: the artifacts.py docstring list conflicted, both kinds kept), each thread answered and resolved:

• 8e05118: under --max-usd, a judge reply with no known price stops the run after that call (as novelty's cap does; --json budget_stop), and without a cap the spend line says how many calls had no price (unpriced_calls). The judge refusal states the same-model rule the check enforces. Two different labels for one pair, and repo hits sheet --out over a sheet that holds labels, each exit 2.
• 790c0c1: a labelled sheet entry whose comment or finding_text is not, exactly, the pair's current text (a change to answer.txt or the gold file at the same finding index) exits 2 before any judge call, as does one missing either text or whose pair's texts are gone.

make ci-local is green at each commit, and CI is green on 790c0c1. With #358 and #359 at their heads, merged together onto main 41b3d8d, the full suite (12645 passed), mypy and ruff pass. The description is updated.