From d840bc8c7b3fc657906da673a80becf3038a61d1 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 13:06:38 +0000 Subject: [PATCH 1/9] docs(repo): how to read a replay batch before writing it up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A write-up of a review replay batch can over-read it in ways the docs did not warn about: crediting "the guidance" when only the root file was in the start context, calling a date split out of sample, reading an interval that includes 0 as no effect, resting a suggestion on a descriptive p, quoting location-credited hits as content matches, and adding totals by hand. reviewing.md gets a "Reading a replay batch" subsection, linked from the Step 4 reading note. It gives one rule per mistake, each naming the field or command that answers it: first_prompt_tokens and reach for delivery, a guidance-commit listing and repo compare --task for leakage, the inconclusive note's mde, p_holm versus descriptive rows, repo report --hits, and the --json outputs for totals. A sibling of the "A rule the review proposes" bullet asks that a replacement line proposed for a stale claim be checked like the claim, in every place it speaks for. Contract §9 points at the subsection from the repo report --hits bullet; the contract's own rules are unchanged. Docs only: no changelog fragment, since dokimi/docs/ is outside the paths the fragment step gates. Signed-off-by: Anton Volkov --- dokimi/docs/repo-evaluation-contract.md | 3 +- dokimi/docs/reviewing.md | 60 ++++++++++++++++++++++++- 2 files changed, 61 insertions(+), 2 deletions(-) diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 6377283ab..feadf29ef 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -1576,7 +1576,8 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re record in `hits` per pair (`task`, `arm`, `model`, `rep`, `cell` and the match's fields; the first five come from the cell's `result.json`, and a match key of the same name does not replace them). A cell graded before `matches` was recorded is counted - (`N cell(s) graded without hit details`), not listed. + (`N cell(s) graded without hit details`), not listed. How to read a replay + batch before writing it up: [reviewing.md](reviewing.md#reading-a-replay-batch). - `repo compare` reduces cells to one value per (task, arm, model) first: repetitions share a task's difficulty and are not paired across arms, so the task is the unit. Per treatment arm, model and measure (gate share, score, diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 6fa1a3ff1..3b82b9010 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -147,7 +147,8 @@ How to read the results: the docs is a finding. `reach` probes the tree as it is: with no `CLAUDE.md` in the tree, it does not show what one would turn off. - **Step 4:** without tasks there is no outcome measurement. The review - should say so rather than infer one from steps 1–3. + should say so rather than infer one from steps 1–3. A batch of replayed + reviews is read as [Reading a replay batch](#reading-a-replay-batch) says. A review also says which of its points come from reading rather than from a command: @@ -167,5 +168,62 @@ A review also says which of its points come from reading rather than from a comm its own). Check it against the existing guidance: a nested `AGENTS.md` may keep old code on purpose. Put what applies to one directory in that directory's file. +- **A replacement line the review proposes** for a stale claim: check it the + way you checked the claim, every file and name it states, in every place it + speaks for (all the workflows, not the first one found). + +### Reading a replay batch + +A step 4 batch of `dokimi:review_replay` tasks is easy to over-read. Check +these, in order, before writing what it showed (the fields: +[§4.5](repo-evaluation-contract.md#45-built-in-graders) and +[§9](repo-evaluation-contract.md#9-running) of the contract). + +- **Delivery first.** Say what the cells started with before crediting any + file. Compare the `first_prompt_tokens` row of `repo compare` with each + guidance file's size in tokens: a delta the size of the root file means only + the root file was in the start context. A nested `AGENTS.md` loads later if + at all; reach (step 2) shows what loads it (`cwd=`, `read=`). + `process:guidance_read` counts Read-tool reads only, so 0 from an agent that + used Bash is not "never read": look for a `cat` or `grep` of a guidance file + in `events.jsonl` or `agent.stdout`, or add a `process_metrics` counter. + Name what was delivered as what was tested ("the root `AGENTS.md`", not "the + guidance"), and credit no hit to a rule in a file the cell neither started + with nor read. +- **A leakage cut.** A task is out of sample for a guidance revision only when + no guidance commit in it could have been written from the task's review; a + date split is not that cut. List the revision's guidance commits + (`git log -p --format='%h %cI %B' -- `) and flag the + tasks whose PR a commit names (`task.json` `source_pr`) or whose threads + (`createdAt` in the threads file) predate a commit that changed the rule + they raise. Read the gain without them (`repo compare --task` on the rest). + Leakage can only favour the guidance arm: it weakens a gain, not a null. +- **Inconclusive, with the `mde`.** A gate or score row whose interval + includes 0 is inconclusive, not "no effect". Quote the interval and the + note's `mde`, and say what the data rules out: a gain above the interval's + upper end. A smaller `target` needs about `n_tasks * (mde / target)^2` + tasks. Where the note says no effect can reach p < 0.05 here, say that. +- **No suggestion rests on a descriptive p.** Cost, turns, + `first_prompt_tokens`, every `--metric` row (`n_findings`, `n_severe`, + `recall_addressed`) and the `process` rows are descriptive: their p is + uncorrected and outside Holm's family (the line under the comparison names + those below 0.05). Report them as descriptions. A suggestion to change the + guidance needs a gate or score row read at `p_holm`, also where its interval + disagrees; one about a section or a rule needs an arm that removes or + changes it (a `guidance:rev:` of that edit). Without one, say the suggestion + comes from reading. +- **Read the hits before citing one.** Replay credits a finding by place, not + content: the same file, within `window` lines of a thread, with no judge. + Run `dokimi repo report --hits` and read each pair you cite, the + thread's comment and then the finding; cite only pairs that make the same + point, and say how many credited pairs you read and how many held. Check + that the finding's `rule` (in the cell's `answer.txt`) is in a delivered file. +- **Totals from command output.** Copy counts, costs and turns from + `repo report --json` (`total_cost_usd`, a row's `n`, `cost_usd`, `turns`), + `repo compare --json` (`n_tasks`, `delta`, `mde`) and `grade.json` `metrics` + (`n_threads`, `n_findings`, `n_severe`). A total no command prints, such as + the gold threads (`n_threads`, once per task), is summed by a script over + those files, not by hand. A range is the lowest and highest per-cell value + (`result.json` `agent.num_turns`), not of row means. The contract a repository codes against: [repo-evaluation-contract.md](repo-evaluation-contract.md). From be55ddd2dcbdab563d72292211137774cb61ce59 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 13:37:56 +0000 Subject: [PATCH 2/9] docs(repo): a replay compares guidance:none with guidance:rev: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The audit of a replay write-up (item5/replay-reading) used guidance:rev: against guidance:none, as the new "Reading a replay batch" subsection assumes, but reviewing.md and the contract's §9 replay example named guidance:raw. For a replay that is wrong: a review_replay task's base is its PR's merge base, and workspace preparation applies the arm to that base's tree, so guidance:raw keeps the guidance as it was at each PR's base, a different set per task and none before the guidance was added, not the guidance under review. - reviewing.md "Past review comments": compare guidance:none with guidance:rev: of the guidance under review, and say why not raw. - reviewing.md step 4 row: the raw command stays for the repository's own tasks; a clause says a replay batch passes guidance:rev:. - contract §9: the replay example runs guidance:rev:, with a one-line comment on why. Signed-off-by: Anton Volkov --- dokimi/docs/repo-evaluation-contract.md | 3 ++- dokimi/docs/reviewing.md | 7 +++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index feadf29ef..b6f99556f 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -1504,7 +1504,8 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo ```bash # review tasks from the threads of merged PRs (§4.6): no LLM, no docker, no GitHub call python cli.py repo tasks replay ~/dpnp --threads threads.json --out ~/dpnp/.agent-eval/tasks -python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm guidance:none,guidance:raw \ +# a task starts from its PR's base: guidance:rev: runs the guidance under review there +python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm guidance:none,guidance:rev: \ --model claude-sonnet-5-5 --batch replay --agent-network host python cli.py repo compare repo-runs/replay --baseline guidance:none --metric recall_addressed ``` diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 3b82b9010..8e1cef127 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -104,7 +104,7 @@ The full path with reading notes: [skill-novelty-walkthrough.md](skill-novelty-w | 1. Do the paths, links and commands named in the guidance exist? | `dokimi repo scan ` | free | | 2. Which guidance files does the harness actually load? | `dokimi repo reach --model --dry-run`, then a few `--only ` with `--budget` | capped by `--budget` (default $0.10) per scenario; needs the `claude` binary and `ANTHROPIC_API_KEY`, or `AWS_BEARER_TOKEN_BEDROCK` with `AWS_REGION` and `CLAUDE_CODE_USE_BEDROCK` | | 3. What does the root guidance add? | `dokimi novelty analyze --artifact text:/AGENTS.md --tier triage …` | ~100 calls | -| 4. Does the guidance change task outcomes? | `dokimi repo run --manifest-file … --model --batch --arm guidance:none,guidance:raw --agent-network host --budget --max-batch-usd ` | needs the repository's own tasks, Docker, `claude` binary on PATH (or `--bin`); `--budget` defaults to $2.00 per agent run with no batch cap unless you pass `--max-batch-usd` | +| 4. Does the guidance change task outcomes? | `dokimi repo run --manifest-file … --model --batch --arm guidance:none,guidance:raw --agent-network host --budget --max-batch-usd `; a replay batch passes `guidance:rev:` instead of `guidance:raw` (see Past review comments below) | needs the repository's own tasks, Docker, `claude` binary on PATH (or `--bin`); `--budget` defaults to $2.00 per agent run with no batch cap unless you pass `--max-batch-usd` | With no `repo-eval.yaml` in the repository, steps 1 and 2 check the default guidance globs (`/**/AGENTS.md`, `/**/CLAUDE.md`, @@ -162,7 +162,10 @@ A review also says which of its points come from reading rather than from a comm ([preparing-a-repository.md](preparing-a-repository.md#6-review-rules-learned-from-past-reviews)), while the count asks what the guidance would have caught. To measure it instead, replay the merged pull requests as `dokimi:review_replay` tasks - and compare `guidance:none` with `guidance:raw` in step 4. + and compare `guidance:none` with `guidance:rev:` of the guidance under + review (the PR's head, or `main`) in step 4. Not `guidance:raw`: each task + starts from its PR's base, so `raw` gives it the guidance of that time, + another set per task, or none before the guidance was added. - **A rule the review proposes:** read every file it names; that the name exists is not enough (a workflow that calls another one holds no matrix of its own). Check it against the existing guidance: a nested `AGENTS.md` may From 786465c056ef87532da6e1c02d96ec2897008ffa Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 14:43:46 +0100 Subject: [PATCH 3/9] docs(repo): preparing-a-repository runs a replay with guidance:rev: The same guidance:raw pairing as the reviewing.md bullet the previous commit fixed: section 6 told a reader to run review_replay tasks with guidance:none and guidance:raw. Each task starts from its PR's base, so raw gives it the guidance of that time; the arm to compare is guidance:rev: of the guidance under review. Signed-off-by: Anton Volkov --- dokimi/docs/preparing-a-repository.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/dokimi/docs/preparing-a-repository.md b/dokimi/docs/preparing-a-repository.md index a2c1c842d..fc23a2311 100644 --- a/dokimi/docs/preparing-a-repository.md +++ b/dokimi/docs/preparing-a-repository.md @@ -167,7 +167,8 @@ Recurring review comments are review knowledge a repository already has. - **Measure what the guidance would have caught.** `dokimi repo tasks replay` turns the review threads of merged PRs into `dokimi:review_replay` tasks ([contract §4.6](repo-evaluation-contract.md#46-replay-tasks-from-review-threads)). - Run them with `guidance:none` and `guidance:raw` and compare + Run them with `guidance:none` and `guidance:rev:` of the guidance under + review (not `guidance:raw`: each task starts from its PR's base) and compare `recall_addressed`, instead of counting by hand which comments the guidance covers. - **A maintainer curates the list.** In one run, one mined rule contradicted the From cbe759d42aca8f349dfd6e42778883df5b59dc07 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 14:01:41 +0000 Subject: [PATCH 4/9] docs(repo): replay reading checks delivery, leakage and hits in full Review comments on the "Reading a replay batch" subsection of reviewing.md, each checked against the code: - Delivery: the first_prompt_tokens delta is a mean over tasks and cells of a total and names no file. It is evidence, confirmed with reach under the cells' harness and version and, where a credit turns on it, with the cells' traces. - Leakage: guidance is read with symlinks followed (contract section 8), so the git log covers every link on the way and the file it resolves to; a commit that changed only the target is otherwise missed. - Leakage is not one-sided: flagged tasks can inflate a gain or mask a loss, so a gain, a null and a loss are all read without them. The sentence saying leakage weakens a gain, not a null, is gone. - Inconclusive: where the interval includes 0 but p_holm is below 0.05, repo compare prints the disagreement note in place of the inconclusive one, with no mde; p_holm is the one to read there. - Hits: repo report --hits cuts each comment and finding to 100 characters, and a match keeps 200 (graders/review.py CUT). A cited pair is read in full, the thread's comment in gold.json or at its url and the finding in answer.txt. - The delivered-file check is for attributing a hit to a guidance rule only: a finding's rule is the agent's own label, and a guidance:none hit has no guidance to check. Signed-off-by: Anton Volkov --- dokimi/docs/reviewing.md | 47 ++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 8e1cef127..61e3f6993 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -183,29 +183,34 @@ these, in order, before writing what it showed (the fields: [§9](repo-evaluation-contract.md#9-running) of the contract). - **Delivery first.** Say what the cells started with before crediting any - file. Compare the `first_prompt_tokens` row of `repo compare` with each - guidance file's size in tokens: a delta the size of the root file means only - the root file was in the start context. A nested `AGENTS.md` loads later if - at all; reach (step 2) shows what loads it (`cwd=`, `read=`). - `process:guidance_read` counts Read-tool reads only, so 0 from an agent that - used Bash is not "never read": look for a `cat` or `grep` of a guidance file - in `events.jsonl` or `agent.stdout`, or add a `process_metrics` counter. - Name what was delivered as what was tested ("the root `AGENTS.md`", not "the - guidance"), and credit no hit to a rule in a file the cell neither started - with nor read. + file. The `first_prompt_tokens` delta of `repo compare` is evidence, not + proof: a mean over tasks and cells of a total, it names no file. A delta the + size of the root file points at that file alone; confirm with reach (step 2) + under the cells' harness and version, and where a credit turns on it with + the cells' traces. A nested `AGENTS.md` loads later if at all (reach's + `cwd=`, `read=` show what loads it). `process:guidance_read` + counts Read-tool reads only: 0 from an agent that used Bash is not "never + read"; look for a `cat` or `grep` of a guidance file in `events.jsonl` or + `agent.stdout`, or add a `process_metrics` counter. Name what was delivered + as what was tested ("the root `AGENTS.md`", not "the guidance"). - **A leakage cut.** A task is out of sample for a guidance revision only when no guidance commit in it could have been written from the task's review; a date split is not that cut. List the revision's guidance commits - (`git log -p --format='%h %cI %B' -- `) and flag the - tasks whose PR a commit names (`task.json` `source_pr`) or whose threads - (`createdAt` in the threads file) predate a commit that changed the rule - they raise. Read the gain without them (`repo compare --task` on the rest). - Leakage can only favour the guidance arm: it weakens a gain, not a null. + (`git log -p --format='%h %cI %B' -- `). Guidance is read with + symlinks followed, so the paths are each guidance file, every link on the + way and the file it resolves to (`AGENTS.md -> docs/GUIDE.md`: both), or a + commit that changed only the target is missed. Flag the tasks whose PR a + commit names (`task.json` `source_pr`) or whose threads (`createdAt` in the + threads file) predate a commit that changed the rule they raise. Flagged + tasks can inflate a gain or mask a loss on the others, so read a gain, a + null and a loss alike without them (`repo compare --task` on the rest). - **Inconclusive, with the `mde`.** A gate or score row whose interval includes 0 is inconclusive, not "no effect". Quote the interval and the note's `mde`, and say what the data rules out: a gain above the interval's upper end. A smaller `target` needs about `n_tasks * (mde / target)^2` tasks. Where the note says no effect can reach p < 0.05 here, say that. + Where the interval includes 0 but `p_holm` is below 0.05, compare prints + the disagreement note instead, with no `mde`: read `p_holm` there. - **No suggestion rests on a descriptive p.** Cost, turns, `first_prompt_tokens`, every `--metric` row (`n_findings`, `n_severe`, `recall_addressed`) and the `process` rows are descriptive: their p is @@ -217,10 +222,14 @@ these, in order, before writing what it showed (the fields: comes from reading. - **Read the hits before citing one.** Replay credits a finding by place, not content: the same file, within `window` lines of a thread, with no judge. - Run `dokimi repo report --hits` and read each pair you cite, the - thread's comment and then the finding; cite only pairs that make the same - point, and say how many credited pairs you read and how many held. Check - that the finding's `rule` (in the cell's `answer.txt`) is in a delivered file. + `dokimi repo report --hits` finds the pairs but cuts each comment + and finding to 100 characters (200 with `--json`): read each pair you cite + in full, the thread's `comment` in the task's `gold.json` (or at its `url`), + then the finding in the cell's `answer.txt`. Cite only pairs that make the + same point; say how many credited pairs you read and how many held. To + attribute a hit to a guidance rule, check that the rule is in a file the + cell started with or read (above): a finding's `rule` is the agent's own + label. A hit cited without attribution (a `guidance:none` one) needs none. - **Totals from command output.** Copy counts, costs and turns from `repo report --json` (`total_cost_usd`, a row's `n`, `cost_usd`, `turns`), `repo compare --json` (`n_tasks`, `delta`, `mde`) and `grade.json` `metrics` From f334ff97ef2e323cd97b4c6ae8bc8ba045f6a211 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 19:23:12 +0000 Subject: [PATCH 5/9] docs(repo): a replay is decided on the score row at p_holm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Settles review comment 4446226568 on #365: the edited passages told the reader to compare recall_addressed, a --metric row, without naming the row that decides a replay. A review_replay task's score is recall over all its threads (replay/build.py writes score.metric "recall"), and repo compare corrects only the gate and score rows under Holm; a metric: row keeps its raw p, outside the family. - preparing-a-repository.md §6: decide on repo compare's score row, read at p_holm; --metric recall_addressed adds a descriptive row for the addressed threads, to read the result, not to decide it. - contract §9: the replay example says the same in a comment above its repo compare line. - reviewing.md, "Reading a replay batch": the descriptive-p bullet names the score row as the replay's decision. Signed-off-by: Anton Volkov --- dokimi/docs/preparing-a-repository.md | 9 ++++++--- dokimi/docs/repo-evaluation-contract.md | 1 + dokimi/docs/reviewing.md | 15 ++++++++------- 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/dokimi/docs/preparing-a-repository.md b/dokimi/docs/preparing-a-repository.md index fc23a2311..7831f32fd 100644 --- a/dokimi/docs/preparing-a-repository.md +++ b/dokimi/docs/preparing-a-repository.md @@ -168,9 +168,12 @@ Recurring review comments are review knowledge a repository already has. turns the review threads of merged PRs into `dokimi:review_replay` tasks ([contract §4.6](repo-evaluation-contract.md#46-replay-tasks-from-review-threads)). Run them with `guidance:none` and `guidance:rev:` of the guidance under - review (not `guidance:raw`: each task starts from its PR's base) and compare - `recall_addressed`, instead of counting by hand which comments the guidance - covers. + review (not `guidance:raw`: each task starts from its PR's base) and decide + on `repo compare`'s `score` row (`recall` over all the threads, read at + `p_holm`), instead of counting by hand which comments the guidance covers. + `--metric recall_addressed` adds the threads the author addressed as a + descriptive row (uncorrected p, outside Holm), to read the result, not to + decide it. - **A maintainer curates the list.** In one run, one mined rule contradicted the repository's own guidance and had to be dropped. - **Place them in the prompt for now.** E6 measured rules in the prompt; the diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index b6f99556f..5d700c228 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -1507,6 +1507,7 @@ python cli.py repo tasks replay ~/dpnp --threads threads.json --out ~/dpnp/.agen # a task starts from its PR's base: guidance:rev: runs the guidance under review there python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm guidance:none,guidance:rev: \ --model claude-sonnet-5-5 --batch replay --agent-network host +# decide on the score row (recall, p_holm); metric:recall_addressed is descriptive python cli.py repo compare repo-runs/replay --baseline guidance:none --metric recall_addressed ``` diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 61e3f6993..d26ca0749 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -211,13 +211,14 @@ these, in order, before writing what it showed (the fields: tasks. Where the note says no effect can reach p < 0.05 here, say that. Where the interval includes 0 but `p_holm` is below 0.05, compare prints the disagreement note instead, with no `mde`: read `p_holm` there. -- **No suggestion rests on a descriptive p.** Cost, turns, - `first_prompt_tokens`, every `--metric` row (`n_findings`, `n_severe`, - `recall_addressed`) and the `process` rows are descriptive: their p is - uncorrected and outside Holm's family (the line under the comparison names - those below 0.05). Report them as descriptions. A suggestion to change the - guidance needs a gate or score row read at `p_holm`, also where its interval - disagrees; one about a section or a rule needs an arm that removes or +- **No suggestion rests on a descriptive p.** A replay is decided on the + `score` row, read at `p_holm`: the task's score is `recall`, over all its + threads. Cost, turns, `first_prompt_tokens`, every `--metric` row + (`n_findings`, `n_severe`, `recall_addressed`) and the `process` rows are + descriptive: their p is uncorrected and outside Holm's family (the line + under the comparison names those below 0.05). Report them as descriptions. + A suggestion to change the guidance needs the score row at `p_holm`, also + where its interval disagrees; one about a section or a rule needs an arm that removes or changes it (a `guidance:rev:` of that edit). Without one, say the suggestion comes from reading. - **Read the hits before citing one.** Replay credits a finding by place, not From a016e1f67a325aff996b22d38a2cac07f4eb075b Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 19:23:20 +0000 Subject: [PATCH 6/9] docs(repo): an inconclusive replay rules out both ends of its interval Settles a nit from the review of #365: "what the data rules out" named only a gain above the interval's upper end. An interval that covers 0 also rules out a loss below its lower end; the "Inconclusive, with the mde" bullet of reviewing.md's "Reading a replay batch" now says both. Signed-off-by: Anton Volkov --- dokimi/docs/reviewing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index d26ca0749..af1137970 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -206,8 +206,8 @@ these, in order, before writing what it showed (the fields: null and a loss alike without them (`repo compare --task` on the rest). - **Inconclusive, with the `mde`.** A gate or score row whose interval includes 0 is inconclusive, not "no effect". Quote the interval and the - note's `mde`, and say what the data rules out: a gain above the interval's - upper end. A smaller `target` needs about `n_tasks * (mde / target)^2` + note's `mde`, and say what the data rules out at both ends: a gain above the + interval's upper end and a loss below its lower end. A smaller `target` needs about `n_tasks * (mde / target)^2` tasks. Where the note says no effect can reach p < 0.05 here, say that. Where the interval includes 0 but `p_holm` is below 0.05, compare prints the disagreement note instead, with no `mde`: read `p_holm` there. From 1c10e954d90c79e9ac8eb546d73a0dee5e6d6217 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 20:25:17 +0000 Subject: [PATCH 7/9] docs(repo): a leakage cut logs each guidance file through its renames The leakage bullet of "Reading a replay batch" listed guidance commits with one path-limited `git log -- `. That log does not follow renames: with a rule added to docs/OLD.md, the file then renamed to docs/GUIDE.md and AGENTS.md linking to it, logging AGENTS.md and docs/GUIDE.md shows the rename and the first commit but not the one that added the rule, so a task whose PR that commit names stays in the out-of-sample set. Log each guidance file on its own with `git log --follow -p` (it takes one path), the links and the file they resolve to as before, and each file a link pointed to earlier, which `git log -p -- ` shows. Settles review comment 4447271326 on #365. Signed-off-by: Anton Volkov --- dokimi/docs/reviewing.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index af1137970..97b586b8e 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -195,11 +195,13 @@ these, in order, before writing what it showed (the fields: as what was tested ("the root `AGENTS.md`", not "the guidance"). - **A leakage cut.** A task is out of sample for a guidance revision only when no guidance commit in it could have been written from the task's review; a - date split is not that cut. List the revision's guidance commits - (`git log -p --format='%h %cI %B' -- `). Guidance is read with - symlinks followed, so the paths are each guidance file, every link on the - way and the file it resolves to (`AGENTS.md -> docs/GUIDE.md`: both), or a - commit that changed only the target is missed. Flag the tasks whose PR a + date split is not that cut. List the commits of each guidance file on its + own, through its renames (`--follow` takes one path; without it a commit + made before a rename is missed): + `git log --follow -p --format='%h %cI %B' -- `. Guidance is read + with symlinks followed, so log every link on the way, the file it resolves + to (`AGENTS.md -> docs/GUIDE.md`: both) and each file a link pointed to + before (`git log -p -- ` shows them). Flag the tasks whose PR a commit names (`task.json` `source_pr`) or whose threads (`createdAt` in the threads file) predate a commit that changed the rule they raise. Flagged tasks can inflate a gain or mask a loss on the others, so read a gain, a From 00f6b534f9ebf5f864df12edab6ebbe44b2ce108 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 20:43:20 +0000 Subject: [PATCH 8/9] docs(repo): quote the replay example's guidance:rev: arm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The contract's §9 replay example passed --arm guidance:none,guidance:rev: unquoted. Copied into bash, is an input redirect: the line fails with "sha: No such file or directory" before the replay runs, or, where a file named sha exists, writes the command's output to a file named --model. Quote the argument that holds the placeholder and add a comment line saying to replace . The example keeps the placeholder because it is about pinning the revision under review; guidance:rev:main would also run, since pin_arm resolves a branch to its commit. This is the only shell block this PR adds or edits. Settles the review comment on the §9 replay example (shell-unsafe placeholder). Signed-off-by: Anton Volkov --- dokimi/docs/repo-evaluation-contract.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 5d700c228..9b8e41bd0 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -1505,7 +1505,8 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo # review tasks from the threads of merged PRs (§4.6): no LLM, no docker, no GitHub call python cli.py repo tasks replay ~/dpnp --threads threads.json --out ~/dpnp/.agent-eval/tasks # a task starts from its PR's base: guidance:rev: runs the guidance under review there -python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm guidance:none,guidance:rev: \ +# (replace with that revision; the quotes keep the shell from reading as a redirect) +python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm 'guidance:none,guidance:rev:' \ --model claude-sonnet-5-5 --batch replay --agent-network host # decide on the score row (recall, p_holm); metric:recall_addressed is descriptive python cli.py repo compare repo-runs/replay --baseline guidance:none --metric recall_addressed From c412ae241db3fc88a5f0c1e4950ba25d260c7d23 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Mon, 2 Nov 2026 21:00:11 +0000 Subject: [PATCH 9/9] docs(repo): a replay's delivery is probed on a cell's start commit The delivery bullet of "Reading a replay batch" said to confirm the first_prompt_tokens delta with reach under the cells' harness and version. Reach probes the tree it is given, so on the current checkout it can report a file the cells never had (a CLAUDE.md added after the guidance revision), and a guidance:rev: arm writes each guidance link as a plain file, so even @ differs from a cell. No new command is needed: a cell keeps workspace/ with its history, and meta.json start_sha is the commit the agent started from, the task's base with its arm applied. The bullet now gives the probe on that commit, with the run's manifest (the cell has no eval_dir), the cells' harness and a binary of their harness_version, and says that reach on the checkout or on @ checks that tree, not the cells'. Settles review comment 4447598182 on #365. Signed-off-by: Anton Volkov --- dokimi/docs/reviewing.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 97b586b8e..b0c2cd50a 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -185,9 +185,15 @@ these, in order, before writing what it showed (the fields: - **Delivery first.** Say what the cells started with before crediting any file. The `first_prompt_tokens` delta of `repo compare` is evidence, not proof: a mean over tasks and cells of a total, it names no file. A delta the - size of the root file points at that file alone; confirm with reach (step 2) - under the cells' harness and version, and where a credit turns on it with - the cells' traces. A nested `AGENTS.md` loads later if at all (reach's + size of the root file points at that file alone. Confirm with reach (step 2) + on a cell's start commit (`meta.json` `start_sha`: the task's base with its + arm applied; `guidance:rev:` writes each link as a plain file), with the + run's manifest (the cell has no `eval_dir`), the cells' `--harness` and a + `--bin` of their `harness_version`: + `dokimi repo reach /workspace@ --manifest-file --model `. + Reach on the checkout or on `@` (links kept, no base) checks + that tree, not the cells'. Where a credit turns on it, read the cells' + traces too. A nested `AGENTS.md` loads later if at all (reach's `cwd=`, `read=` show what loads it). `process:guidance_read` counts Read-tool reads only: 0 from an agent that used Bash is not "never read"; look for a `cat` or `grep` of a guidance file in `events.jsonl` or