[antonwolfy]

Summary

Follow-up to an audit of a written-up repository review that replayed past PR reviews as
dokimi:review_replay tasks (guidance:none against a guidance:rev:<sha> arm, 26 tasks, 3 reps).
The write-up over-read the batch in seven ways that the docs did not warn about. It credited a
nested-file rule when only the root AGENTS.md was in the start context. It called a date split
out of sample when later guidance commits named replayed PRs. It read an interval of
[-0.066, +0.100] as "no measurable gain". It rested a suggestion on a descriptive p of 0.049. It
cited location-credited hits that made a different point. It added up totals by hand and got them
wrong. And its fact-check proposed replacement lines that were themselves wrong.

The tooling for each check is already on main: agent.first_prompt_tokens and its compare row,
repo compare --task, the inconclusive note with mde, the interval-vs-p note, the
descriptive-p line, details.matches and repo report --hits. What was missing is a page that
tells the person writing up a replay batch to use them. docs/reviewing.md's Step 4 note said
only "without tasks there is no outcome measurement".

The docs also named the wrong arm for a replay. docs/reviewing.md (step 4 row, "Past review
comments") and the contract's §9 replay example compared guidance:none with guidance:raw. A
review_replay task's base is its PR's merge base (contract §4.6), and workspace preparation
applies the arm to that base's tree (§3, steps 1 and 5). guidance:raw keeps the files as they
are there (workspace.py _apply_arm returns before touching them), so each task gets the
guidance as it was at that PR's base. That is a different set per task, and none for a PR older
than the guidance. The arm that runs the guidance under review over old bases is
guidance:rev:<sha> (§3), which is what the audited replay used.

The fourth commit answers the review comments on the subsection; each change below was checked
against the code. The first_prompt_tokens delta is a mean over tasks and cells of a total
(compare.py), so it points at files but does not name them. Guidance is read with symlinks
followed (contract §8), so a leak check that logs only the link misses a commit to its target.
Leakage can mask a loss as well as inflate a gain. Where the interval includes 0 but p_holm is
below 0.05, compare prints the disagreement note in place of the inconclusive one, so there is no
mde to quote. repo report --hits cuts each text to 100 characters (format_hits), and a
stored match to 200 (CUT in graders/review.py). A finding's rule is the agent's own label
(the replay prompt asks for a "short rule name").

Changes

• docs/reviewing.md: a new "Reading a replay batch" subsection under "A repository's agent
  setup", linked from the Step 4 reading note. It asks the reader to:
  • Delivery first: treat the first_prompt_tokens delta as evidence, not proof. Confirm
    what loaded with reach under the cells' harness and version and, where a credit turns on it,
    with the cells' traces. Do not read process:guidance_read 0 from a Bash-only agent as
    "never read". Name what was delivered as what was tested.
  • A leakage cut: list the revision's guidance commits over each guidance file, every link
    on the way and the file it resolves to. Flag the tasks a commit names or whose threads
    predate a rule change, and read a gain, a null or a loss without them via
    repo compare --task.
  • Inconclusive, with the mde: an interval that includes 0 is inconclusive. Quote the
    interval and mde, say what the data rules out, and size the next run by
    n_tasks * (mde / target)^2. Where p_holm is below 0.05 anyway, read the disagreement
    note and p_holm instead.
  • No suggestion on a descriptive p: cost, turns, first_prompt_tokens, --metric and
    process rows are descriptive. A guidance suggestion needs a gate or score row at p_holm,
    and a section or rule suggestion needs an arm that edits that section; without one, say it
    comes from reading.
  • Read the hits: credit is by location within window. Find the pairs with
    repo report --hits, then read each cited pair in full (the thread's comment in
    gold.json or at its url, the finding in answer.txt), and say how many held. A hit
    attributed to a guidance rule also needs that rule in a file the cell started with or read.
  • Totals from command output: copy them from repo report --json, repo compare --json
    and grade.json metrics; a total no command prints is summed by a script.
• docs/reviewing.md: a sibling of the "A rule the review proposes" bullet. A replacement line
  proposed for a stale claim is checked like the claim, in every place it speaks for.
• docs/repo-evaluation-contract.md §9: one sentence after the repo report --hits text that
  points at the new subsection. The contract's rules are unchanged.
• docs/reviewing.md and contract §9, second commit: a replay compares guidance:none with
  guidance:rev:<sha> of the guidance under review (the PR's head, or main). The "Past review
  comments" bullet says so and why not guidance:raw. The step 4 row keeps its
  guidance:none,guidance:raw command for the repository's own tasks and adds one clause for a
  replay batch. The §9 replay example runs --arm guidance:none,guidance:rev:<sha>, with a
  one-line comment.
• docs/preparing-a-repository.md §6 ("Measure what the guidance would have caught"), third
  commit: the same pairing, fixed the same way.
• docs/reviewing.md, fourth commit (review comments): in "Reading a replay batch", the
  first_prompt_tokens delta is evidence to confirm, not proof. The leakage git log covers
  symlinks and their targets, and leakage is read as two-sided. The inconclusive bullet covers
  the case where the disagreement note takes the inconclusive note's place. A cited hit is read
  in full past --hits' cut. The delivered-file check applies only when a hit is attributed to
  a guidance rule. The subsection stays at about 60 lines.
• No changelog fragment: dokimi/docs/ is outside the paths scripts/check_changelog_fragment.py
  gates (dokimi/dokimi/, dokimi/data/, the install tables).

Two open PRs would give two of these checks a field each: #359 (recording public_at
and guidance_commits for leak checks) for the leakage cut, and #357 (judged recall
over credited hits) for the hit check. The subsection will name those fields once the PRs merge.
Until then it describes the manual check.

Risk

None: docs only. No command, field or output changes.

Testing

On a Linux review host, at cbe759d (on main 93ff3a4):

• make ci-local: 30 passed, 0 failed, 7 skipped. Three skips are on the PR side: test-latest,
  docs.yml:deploy and pip-audit.yml:report have a job condition that is false on a PR. Four are
  what the host lacks: external-catalog-smoke needs Docker and network, conda-forge-deps needs
  micromamba, semgrep secrets needs semgrep and the network, and cli-surface / Package the result
  uses an expression the local runner does not evaluate. 12542 tests passed (5 skipped, 6 xfailed).
  Changed-line coverage: no measurable lines (docs only).
• tests/test_docs_code_refs.py, tests/test_growth_ratchet.py, tests/test_audit_test_quality.py
  and tests/test_import_architecture.py: 292 passed. The commands and flags the subsection names
  (repo report --hits, repo compare --task, --json) are checked against the CLI. No test
  reads the §9 replay example.
• mkdocs build --strict: builds; the #reading-a-replay-batch anchor and the linked contract
  anchors (#45-built-in-graders, #9-running) exist in the built site.
• Mutants: none. No code or test changes, so there is no test to break.
• Not yet run merged together with the other follow-up branches.

Revert

git revert of the four commits. Nothing is stored or versioned.

Checklist

• Tests added/updated and pytest passes (no tests change; the suite passes)
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml) (no new files)

[Copilot]

🟡 Changes recommended

The guidance needs full-text hit validation, history checks covering symlink chains and targets, and leakage-filtered comparisons for null results to prevent misleading review conclusions.

3 open issues

• History audit misses changes to symlink targets · New
• Pooled null results can hide regressions in unexposed tasks · New
• Truncated matches can make distinct findings appear equivalent · New

What changed in this PR

3 new findings.

This docs-only PR adds guidance for interpreting repository review-replay batches before writing conclusions, using existing reporting and comparison tools.

Changes:

• Adds checks for guidance delivery, leakage, statistical interpretation, credited hits, and totals.
• Requires proposed replacement claims to be checked across all relevant files.
• Links the evaluation contract to the new guidance without changing runtime behavior or contract rules.

File	Description
dokimi/​docs/​reviewing.md	Adds replay-batch interpretation and replacement-claim verification guidance.
dokimi/​docs/​repo-evaluation-contract.md	Links hit-reporting documentation to the replay-batch guidance.

_{Review effort: Balanced}

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Review details

• Commit reviewed: cbe759d against merge base 786465c
• Coverage: 1 files in 1 groups
• Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
• Usage: 61k tokens (45k from cache), 12s
• Tool calls: 13 (7 code_search, 4 file_read, 2 file_find)
• dokimi/docs/reviewing.md: 1 file

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.29%. Comparing base (93ff3a4) to head (cbe759d).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #365   +/-   ##
=======================================
  Coverage   94.29%   94.29%           
=======================================
  Files         308      308           
  Lines       39322    39322           
  Branches     6651     6651           
=======================================
  Hits        37077    37077           
  Misses       1432     1432           
  Partials      813      813           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review round on the bots' comments (Copilot and exai-bot, 8 threads, all on d840bc8), answered in cbe759d:

• Delivery: the first_prompt_tokens delta is evidence, not proof (a mean of a total, naming no file); confirm with reach and the cells' traces.
• Leakage: the git log covers symlinks and their targets; leakage can inflate a gain or mask a loss, so a gain, a null and a loss are all read without the flagged tasks.
• Inconclusive: where the interval includes 0 but p_holm < 0.05, compare prints the disagreement note with no mde; read p_holm there.
• Hits: --hits cuts text to 100 characters (200 with --json); read the full comment in gold.json and the full finding in answer.txt before citing a pair. The delivered-file check applies only when a hit is attributed to a guidance rule.

Earlier commits since opening: be55ddd and 786465c (a replay compares guidance:none with guidance:rev:<sha>, not guidance:raw, in reviewing.md, contract §9 and preparing-a-repository.md). Every thread is replied to and resolved; CI is green on cbe759d.