docs(repo): how to read a replay batch before writing it up

dokimi/docs/preparing-a-repository.md

@@ -167,7 +167,8 @@ Recurring review comments are review knowledge a repository already has.
 - **Measure what the guidance would have caught.** `dokimi repo tasks replay`
   turns the review threads of merged PRs into `dokimi:review_replay` tasks
   ([contract §4.6](repo-evaluation-contract.md#46-replay-tasks-from-review-threads)).
-  Run them with `guidance:none` and `guidance:raw` and compare
+  Run them with `guidance:none` and `guidance:rev:<sha>` of the guidance under
+  review (not `guidance:raw`: each task starts from its PR's base) and compare
   `recall_addressed`, instead of counting by hand which comments the guidance
   covers.
 - **A maintainer curates the list.** In one run, one mined rule contradicted the

dokimi/docs/repo-evaluation-contract.md

@@ -1504,7 +1504,8 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo
 ```bash
 # review tasks from the threads of merged PRs (§4.6): no LLM, no docker, no GitHub call
 python cli.py repo tasks replay ~/dpnp --threads threads.json --out ~/dpnp/.agent-eval/tasks
-python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm guidance:none,guidance:raw \
+# a task starts from its PR's base: guidance:rev:<sha> runs the guidance under review there
+python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm guidance:none,guidance:rev:<sha> \
     --model claude-sonnet-5-5 --batch replay --agent-network host
 python cli.py repo compare repo-runs/replay --baseline guidance:none --metric recall_addressed
 ```
@@ -1576,7 +1577,8 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re
   record in `hits` per pair (`task`, `arm`, `model`, `rep`, `cell` and the
   match's fields; the first five come from the cell's `result.json`, and a
   match key of the same name does not replace them). A cell graded before `matches` was recorded is counted
-  (`N cell(s) graded without hit details`), not listed.
+  (`N cell(s) graded without hit details`), not listed. How to read a replay
+  batch before writing it up: [reviewing.md](reviewing.md#reading-a-replay-batch).
 - `repo compare` reduces cells to one value per (task, arm, model) first:
   repetitions share a task's difficulty and are not paired across arms, so the
   task is the unit. Per treatment arm, model and measure (gate share, score,

dokimi/docs/reviewing.md

@@ -104,7 +104,7 @@ The full path with reading notes: [skill-novelty-walkthrough.md](skill-novelty-w
 | 1. Do the paths, links and commands named in the guidance exist? | `dokimi repo scan <repo>` | free |
 | 2. Which guidance files does the harness actually load? | `dokimi repo reach <repo> --model <id> --dry-run`, then a few `--only <scenario>` with `--budget` | capped by `--budget` (default $0.10) per scenario; needs the `claude` binary and `ANTHROPIC_API_KEY`, or `AWS_BEARER_TOKEN_BEDROCK` with `AWS_REGION` and `CLAUDE_CODE_USE_BEDROCK` |
 | 3. What does the root guidance add? | `dokimi novelty analyze --artifact text:<repo>/AGENTS.md --tier triage …` | ~100 calls |
-| 4. Does the guidance change task outcomes? | `dokimi repo run <repo> --manifest-file … --model <id> --batch <name> --arm guidance:none,guidance:raw --agent-network host --budget <usd> --max-batch-usd <usd>` | needs the repository's own tasks, Docker, `claude` binary on PATH (or `--bin`); `--budget` defaults to $2.00 per agent run with no batch cap unless you pass `--max-batch-usd` |
+| 4. Does the guidance change task outcomes? | `dokimi repo run <repo> --manifest-file … --model <id> --batch <name> --arm guidance:none,guidance:raw --agent-network host --budget <usd> --max-batch-usd <usd>`; a replay batch passes `guidance:rev:<sha>` instead of `guidance:raw` (see Past review comments below) | needs the repository's own tasks, Docker, `claude` binary on PATH (or `--bin`); `--budget` defaults to $2.00 per agent run with no batch cap unless you pass `--max-batch-usd` |
 
 With no `repo-eval.yaml` in the repository, steps 1 and 2 check the default
 guidance globs (`<repo>/**/AGENTS.md`, `<repo>/**/CLAUDE.md`,
@@ -147,7 +147,8 @@ How to read the results:
   the docs is a finding. `reach` probes the tree as it is: with no
   `CLAUDE.md` in the tree, it does not show what one would turn off.
 - **Step 4:** without tasks there is no outcome measurement. The review
-  should say so rather than infer one from steps 1–3.
+  should say so rather than infer one from steps 1–3. A batch of replayed
+  reviews is read as [Reading a replay batch](#reading-a-replay-batch) says.
 
 A review also says which of its points come from reading rather than from a command:
 
@@ -161,11 +162,80 @@ A review also says which of its points come from reading rather than from a comm
   ([preparing-a-repository.md](preparing-a-repository.md#6-review-rules-learned-from-past-reviews)),
   while the count asks what the guidance would have caught. To measure it
   instead, replay the merged pull requests as `dokimi:review_replay` tasks
-  and compare `guidance:none` with `guidance:raw` in step 4.
+  and compare `guidance:none` with `guidance:rev:<sha>` of the guidance under
+  review (the PR's head, or `main`) in step 4. Not `guidance:raw`: each task
+  starts from its PR's base, so `raw` gives it the guidance of that time,
+  another set per task, or none before the guidance was added.
 - **A rule the review proposes:** read every file it names; that the name
   exists is not enough (a workflow that calls another one holds no matrix of
   its own). Check it against the existing guidance: a nested `AGENTS.md` may
   keep old code on purpose. Put what applies to one directory in that
   directory's file.
+- **A replacement line the review proposes** for a stale claim: check it the
+  way you checked the claim, every file and name it states, in every place it
+  speaks for (all the workflows, not the first one found).
+
+### Reading a replay batch
+
+A step 4 batch of `dokimi:review_replay` tasks is easy to over-read. Check
+these, in order, before writing what it showed (the fields:
+[§4.5](repo-evaluation-contract.md#45-built-in-graders) and
+[§9](repo-evaluation-contract.md#9-running) of the contract).
+
+- **Delivery first.** Say what the cells started with before crediting any
+  file. The `first_prompt_tokens` delta of `repo compare` is evidence, not
+  proof: a mean over tasks and cells of a total, it names no file. A delta the
+  size of the root file points at that file alone; confirm with reach (step 2)
+  under the cells' harness and version, and where a credit turns on it with
+  the cells' traces. A nested `AGENTS.md` loads later if at all (reach's
+  `cwd=<dir>`, `read=<dir>` show what loads it). `process:guidance_read`
+  counts Read-tool reads only: 0 from an agent that used Bash is not "never
+  read"; look for a `cat` or `grep` of a guidance file in `events.jsonl` or
+  `agent.stdout`, or add a `process_metrics` counter. Name what was delivered
+  as what was tested ("the root `AGENTS.md`", not "the guidance").
+- **A leakage cut.** A task is out of sample for a guidance revision only when
+  no guidance commit in it could have been written from the task's review; a
+  date split is not that cut. List the revision's guidance commits
+  (`git log -p --format='%h %cI %B' <rev> -- <paths>`). Guidance is read with
+  symlinks followed, so the paths are each guidance file, every link on the
+  way and the file it resolves to (`AGENTS.md -> docs/GUIDE.md`: both), or a
+  commit that changed only the target is missed. Flag the tasks whose PR a
+  commit names (`task.json` `source_pr`) or whose threads (`createdAt` in the
+  threads file) predate a commit that changed the rule they raise. Flagged
+  tasks can inflate a gain or mask a loss on the others, so read a gain, a
+  null and a loss alike without them (`repo compare --task` on the rest).
+- **Inconclusive, with the `mde`.** A gate or score row whose interval
+  includes 0 is inconclusive, not "no effect". Quote the interval and the
+  note's `mde`, and say what the data rules out: a gain above the interval's
+  upper end. A smaller `target` needs about `n_tasks * (mde / target)^2`
+  tasks. Where the note says no effect can reach p < 0.05 here, say that.
+  Where the interval includes 0 but `p_holm` is below 0.05, compare prints
+  the disagreement note instead, with no `mde`: read `p_holm` there.
+- **No suggestion rests on a descriptive p.** Cost, turns,
+  `first_prompt_tokens`, every `--metric` row (`n_findings`, `n_severe`,
+  `recall_addressed`) and the `process` rows are descriptive: their p is
+  uncorrected and outside Holm's family (the line under the comparison names
+  those below 0.05). Report them as descriptions. A suggestion to change the
+  guidance needs a gate or score row read at `p_holm`, also where its interval
+  disagrees; one about a section or a rule needs an arm that removes or
+  changes it (a `guidance:rev:` of that edit). Without one, say the suggestion
+  comes from reading.
+- **Read the hits before citing one.** Replay credits a finding by place, not
+  content: the same file, within `window` lines of a thread, with no judge.
+  `dokimi repo report <batch> --hits` finds the pairs but cuts each comment
+  and finding to 100 characters (200 with `--json`): read each pair you cite
+  in full, the thread's `comment` in the task's `gold.json` (or at its `url`),
+  then the finding in the cell's `answer.txt`. Cite only pairs that make the
+  same point; say how many credited pairs you read and how many held. To
+  attribute a hit to a guidance rule, check that the rule is in a file the
+  cell started with or read (above): a finding's `rule` is the agent's own
+  label. A hit cited without attribution (a `guidance:none` one) needs none.
+- **Totals from command output.** Copy counts, costs and turns from
+  `repo report --json` (`total_cost_usd`, a row's `n`, `cost_usd`, `turns`),
+  `repo compare --json` (`n_tasks`, `delta`, `mde`) and `grade.json` `metrics`
+  (`n_threads`, `n_findings`, `n_severe`). A total no command prints, such as
+  the gold threads (`n_threads`, once per task), is summed by a script over
+  those files, not by hand. A range is the lowest and highest per-cell value
+  (`result.json` `agent.num_turns`), not of row means.
 
 The contract a repository codes against: [repo-evaluation-contract.md](repo-evaluation-contract.md).