diff --git a/dokimi/docs/preparing-a-repository.md b/dokimi/docs/preparing-a-repository.md index a2c1c842..7831f32f 100644 --- a/dokimi/docs/preparing-a-repository.md +++ b/dokimi/docs/preparing-a-repository.md @@ -167,9 +167,13 @@ Recurring review comments are review knowledge a repository already has. - **Measure what the guidance would have caught.** `dokimi repo tasks replay` turns the review threads of merged PRs into `dokimi:review_replay` tasks ([contract §4.6](repo-evaluation-contract.md#46-replay-tasks-from-review-threads)). - Run them with `guidance:none` and `guidance:raw` and compare - `recall_addressed`, instead of counting by hand which comments the guidance - covers. + Run them with `guidance:none` and `guidance:rev:` of the guidance under + review (not `guidance:raw`: each task starts from its PR's base) and decide + on `repo compare`'s `score` row (`recall` over all the threads, read at + `p_holm`), instead of counting by hand which comments the guidance covers. + `--metric recall_addressed` adds the threads the author addressed as a + descriptive row (uncorrected p, outside Holm), to read the result, not to + decide it. - **A maintainer curates the list.** In one run, one mined rule contradicted the repository's own guidance and had to be dropped. - **Place them in the prompt for now.** E6 measured rules in the prompt; the diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 6377283a..9b8e41bd 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -1504,8 +1504,11 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo ```bash # review tasks from the threads of merged PRs (§4.6): no LLM, no docker, no GitHub call python cli.py repo tasks replay ~/dpnp --threads threads.json --out ~/dpnp/.agent-eval/tasks -python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm guidance:none,guidance:raw \ +# a task starts from its PR's base: guidance:rev: runs the guidance under review there +# (replace with that revision; the quotes keep the shell from reading as a redirect) +python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm 'guidance:none,guidance:rev:' \ --model claude-sonnet-5-5 --batch replay --agent-network host +# decide on the score row (recall, p_holm); metric:recall_addressed is descriptive python cli.py repo compare repo-runs/replay --baseline guidance:none --metric recall_addressed ``` @@ -1576,7 +1579,8 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re record in `hits` per pair (`task`, `arm`, `model`, `rep`, `cell` and the match's fields; the first five come from the cell's `result.json`, and a match key of the same name does not replace them). A cell graded before `matches` was recorded is counted - (`N cell(s) graded without hit details`), not listed. + (`N cell(s) graded without hit details`), not listed. How to read a replay + batch before writing it up: [reviewing.md](reviewing.md#reading-a-replay-batch). - `repo compare` reduces cells to one value per (task, arm, model) first: repetitions share a task's difficulty and are not paired across arms, so the task is the unit. Per treatment arm, model and measure (gate share, score, diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 6fa1a3ff..b0c2cd50 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -104,7 +104,7 @@ The full path with reading notes: [skill-novelty-walkthrough.md](skill-novelty-w | 1. Do the paths, links and commands named in the guidance exist? | `dokimi repo scan ` | free | | 2. Which guidance files does the harness actually load? | `dokimi repo reach --model --dry-run`, then a few `--only ` with `--budget` | capped by `--budget` (default $0.10) per scenario; needs the `claude` binary and `ANTHROPIC_API_KEY`, or `AWS_BEARER_TOKEN_BEDROCK` with `AWS_REGION` and `CLAUDE_CODE_USE_BEDROCK` | | 3. What does the root guidance add? | `dokimi novelty analyze --artifact text:/AGENTS.md --tier triage …` | ~100 calls | -| 4. Does the guidance change task outcomes? | `dokimi repo run --manifest-file … --model --batch --arm guidance:none,guidance:raw --agent-network host --budget --max-batch-usd ` | needs the repository's own tasks, Docker, `claude` binary on PATH (or `--bin`); `--budget` defaults to $2.00 per agent run with no batch cap unless you pass `--max-batch-usd` | +| 4. Does the guidance change task outcomes? | `dokimi repo run --manifest-file … --model --batch --arm guidance:none,guidance:raw --agent-network host --budget --max-batch-usd `; a replay batch passes `guidance:rev:` instead of `guidance:raw` (see Past review comments below) | needs the repository's own tasks, Docker, `claude` binary on PATH (or `--bin`); `--budget` defaults to $2.00 per agent run with no batch cap unless you pass `--max-batch-usd` | With no `repo-eval.yaml` in the repository, steps 1 and 2 check the default guidance globs (`/**/AGENTS.md`, `/**/CLAUDE.md`, @@ -147,7 +147,8 @@ How to read the results: the docs is a finding. `reach` probes the tree as it is: with no `CLAUDE.md` in the tree, it does not show what one would turn off. - **Step 4:** without tasks there is no outcome measurement. The review - should say so rather than infer one from steps 1–3. + should say so rather than infer one from steps 1–3. A batch of replayed + reviews is read as [Reading a replay batch](#reading-a-replay-batch) says. A review also says which of its points come from reading rather than from a command: @@ -161,11 +162,89 @@ A review also says which of its points come from reading rather than from a comm ([preparing-a-repository.md](preparing-a-repository.md#6-review-rules-learned-from-past-reviews)), while the count asks what the guidance would have caught. To measure it instead, replay the merged pull requests as `dokimi:review_replay` tasks - and compare `guidance:none` with `guidance:raw` in step 4. + and compare `guidance:none` with `guidance:rev:` of the guidance under + review (the PR's head, or `main`) in step 4. Not `guidance:raw`: each task + starts from its PR's base, so `raw` gives it the guidance of that time, + another set per task, or none before the guidance was added. - **A rule the review proposes:** read every file it names; that the name exists is not enough (a workflow that calls another one holds no matrix of its own). Check it against the existing guidance: a nested `AGENTS.md` may keep old code on purpose. Put what applies to one directory in that directory's file. +- **A replacement line the review proposes** for a stale claim: check it the + way you checked the claim, every file and name it states, in every place it + speaks for (all the workflows, not the first one found). + +### Reading a replay batch + +A step 4 batch of `dokimi:review_replay` tasks is easy to over-read. Check +these, in order, before writing what it showed (the fields: +[§4.5](repo-evaluation-contract.md#45-built-in-graders) and +[§9](repo-evaluation-contract.md#9-running) of the contract). + +- **Delivery first.** Say what the cells started with before crediting any + file. The `first_prompt_tokens` delta of `repo compare` is evidence, not + proof: a mean over tasks and cells of a total, it names no file. A delta the + size of the root file points at that file alone. Confirm with reach (step 2) + on a cell's start commit (`meta.json` `start_sha`: the task's base with its + arm applied; `guidance:rev:` writes each link as a plain file), with the + run's manifest (the cell has no `eval_dir`), the cells' `--harness` and a + `--bin` of their `harness_version`: + `dokimi repo reach /workspace@ --manifest-file --model `. + Reach on the checkout or on `@` (links kept, no base) checks + that tree, not the cells'. Where a credit turns on it, read the cells' + traces too. A nested `AGENTS.md` loads later if at all (reach's + `cwd=`, `read=` show what loads it). `process:guidance_read` + counts Read-tool reads only: 0 from an agent that used Bash is not "never + read"; look for a `cat` or `grep` of a guidance file in `events.jsonl` or + `agent.stdout`, or add a `process_metrics` counter. Name what was delivered + as what was tested ("the root `AGENTS.md`", not "the guidance"). +- **A leakage cut.** A task is out of sample for a guidance revision only when + no guidance commit in it could have been written from the task's review; a + date split is not that cut. List the commits of each guidance file on its + own, through its renames (`--follow` takes one path; without it a commit + made before a rename is missed): + `git log --follow -p --format='%h %cI %B' -- `. Guidance is read + with symlinks followed, so log every link on the way, the file it resolves + to (`AGENTS.md -> docs/GUIDE.md`: both) and each file a link pointed to + before (`git log -p -- ` shows them). Flag the tasks whose PR a + commit names (`task.json` `source_pr`) or whose threads (`createdAt` in the + threads file) predate a commit that changed the rule they raise. Flagged + tasks can inflate a gain or mask a loss on the others, so read a gain, a + null and a loss alike without them (`repo compare --task` on the rest). +- **Inconclusive, with the `mde`.** A gate or score row whose interval + includes 0 is inconclusive, not "no effect". Quote the interval and the + note's `mde`, and say what the data rules out at both ends: a gain above the + interval's upper end and a loss below its lower end. A smaller `target` needs about `n_tasks * (mde / target)^2` + tasks. Where the note says no effect can reach p < 0.05 here, say that. + Where the interval includes 0 but `p_holm` is below 0.05, compare prints + the disagreement note instead, with no `mde`: read `p_holm` there. +- **No suggestion rests on a descriptive p.** A replay is decided on the + `score` row, read at `p_holm`: the task's score is `recall`, over all its + threads. Cost, turns, `first_prompt_tokens`, every `--metric` row + (`n_findings`, `n_severe`, `recall_addressed`) and the `process` rows are + descriptive: their p is uncorrected and outside Holm's family (the line + under the comparison names those below 0.05). Report them as descriptions. + A suggestion to change the guidance needs the score row at `p_holm`, also + where its interval disagrees; one about a section or a rule needs an arm that removes or + changes it (a `guidance:rev:` of that edit). Without one, say the suggestion + comes from reading. +- **Read the hits before citing one.** Replay credits a finding by place, not + content: the same file, within `window` lines of a thread, with no judge. + `dokimi repo report --hits` finds the pairs but cuts each comment + and finding to 100 characters (200 with `--json`): read each pair you cite + in full, the thread's `comment` in the task's `gold.json` (or at its `url`), + then the finding in the cell's `answer.txt`. Cite only pairs that make the + same point; say how many credited pairs you read and how many held. To + attribute a hit to a guidance rule, check that the rule is in a file the + cell started with or read (above): a finding's `rule` is the agent's own + label. A hit cited without attribution (a `guidance:none` one) needs none. +- **Totals from command output.** Copy counts, costs and turns from + `repo report --json` (`total_cost_usd`, a row's `n`, `cost_usd`, `turns`), + `repo compare --json` (`n_tasks`, `delta`, `mde`) and `grade.json` `metrics` + (`n_threads`, `n_findings`, `n_severe`). A total no command prints, such as + the gold threads (`n_threads`, once per task), is summed by a script over + those files, not by hand. A range is the lowest and highest per-cell value + (`result.json` `agent.num_turns`), not of row means. The contract a repository codes against: [repo-evaluation-contract.md](repo-evaluation-contract.md).