Skip to content

feat(repo): judged recall over the credited replay hits #357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
antonwolfy wants to merge 8 commits into
base: main
Choose a base branch
from
+1,981 −10
Open

feat(repo): judged recall over the credited replay hits #357

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
05ebb9a
feat(repo): judged recall over the credited replay hits
antonwolfy Nov 1, 2026
86c37d3
fix(repo): judge refusal, calibration and sheet checks of hits judge
antonwolfy Nov 1, 2026
af4c4a4
fix(repo): exit 2 when hits judge cannot write a cell's file
antonwolfy Nov 1, 2026
38e1da9
fix(repo): remove hits_judged.json of cells hits judge no longer reads
antonwolfy Nov 1, 2026
a1b8822
Merge main into feat/replay-judged-recall
antonwolfy Nov 1, 2026
6c52d10
Merge main into feat/replay-judged-recall
antonwolfy Nov 2, 2026
8e05118
fix(repo): hits judge cap on unpriced calls, label and sheet checks
antonwolfy Nov 2, 2026
790c0c1
fix(repo): hits judge refuses a label on texts that changed
antonwolfy Nov 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions dokimi/changelog.d/20261101_120000_antonwolfy_replay-judged-recall.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Added

- **Judged recall for replay tasks** — `dokimi repo hits judge <batch> --judge PROVIDER/MODEL` asks a judge model whether each credited (thread, finding) pair of the batch's `dokimi:review_replay` cells makes the same point (`same point`, `related` or `unrelated`), reading the thread's full comment from the task's gold file and the finding's full text from `answer.txt` rather than the 200-character `matches`. Each fully judged cell gets `hits_judged.json` (`repo_hits_judged.v1`) with the labels, their counts and `recall_judged`, which is at most `recall` since only credited pairs are rescored; the grade is not changed. A judge that runs any answer model of the batch (a CLI alias counting as its family's full id) and a `--labels` entry naming no credited pair, labelling a pair whose thread comment or finding text is no longer the one the sheet holds (compared exactly), or giving one pair two different labels, are refused with exit 2 before any call; `--dry-run` prices the run without a call, and `--max-usd` starts no new call once spent (a cell left gets no file, exit 1), and stops after a judge reply with no known price, since the cap cannot be enforced on it. `dokimi repo hits sheet <batch> --out FILE` samples the pairs for a person to label, `--per-arm` per arm with a fixed `--seed`, and refuses to replace a sheet that has labels; with that file, `--labels` reports agreement and Cohen's kappa per arm, and without it the output says `recall_judged` is uncalibrated. `repo report --hits --json` also gives `grader_failures` and `cells`, the replay cells whose `matches` was read.
5 changes: 5 additions & 0 deletions dokimi/docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,11 @@ next release.
`repo run` (one agent per task × arm × model × rep cell, then `grade`, one
`result.json` per cell) and `repo report` (gated success with a Wilson
interval, grader failures, missing cells, cost).
- `dokimi/repos/hits_judge.py` + `dokimi/repos/hits_calibration.py` —
`repo hits judge`: a judge model labels each credited pair of the replay
cells with their full texts, and each cell gets `hits_judged.json`
(`repo_hits_judged.v1`, `recall_judged`); `repo hits sheet` samples pairs
for a person, and the labels give agreement and Cohen's kappa per arm.
- `dokimi/repos/compare.py` — `repo compare`: treatment arms against a
baseline arm over one or more batches, one value per task and arm (gate
share, score, cost, turns, chosen `grade.json` metrics, `result.json`
Expand Down
234 changes: 234 additions & 0 deletions dokimi/docs/reference/cli-interface.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10772,6 +10772,239 @@
"subcommand_required": null,
"subcommands": {}
},
"hits": {
"allow_abbrev": true,
"params": [
{
"action": "_HelpAction",
"choices": null,
"const": null,
"default": "==SUPPRESS==",
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": true,
"key": "--help/-h",
"kind": "option",
"nargs": 0,
"position": null,
"required": false,
"type": null
}
],
"path": "dokimi repo hits",
"subcommand_required": true,
"subcommands": {
"judge": {
"allow_abbrev": true,
"params": [
{
"action": "_StoreAction",
"choices": null,
"const": null,
"default": null,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": false,
"key": "batch_dir",
"kind": "argument",
"nargs": null,
"position": 0,
"required": true,
"type": null
},
{
"action": "_StoreTrueAction",
"choices": null,
"const": true,
"default": false,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": true,
"key": "--dry-run",
"kind": "option",
"nargs": 0,
"position": null,
"required": false,
"type": null
},
{
"action": "_HelpAction",
"choices": null,
"const": null,
"default": "==SUPPRESS==",
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": true,
"key": "--help/-h",
"kind": "option",
"nargs": 0,
"position": null,
"required": false,
"type": null
},
{
"action": "_StoreTrueAction",
"choices": null,
"const": true,
"default": false,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": true,
"key": "--json",
"kind": "option",
"nargs": 0,
"position": null,
"required": false,
"type": null
},
{
"action": "_StoreAction",
"choices": null,
"const": null,
"default": null,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": false,
"key": "--judge",
"kind": "option",
"nargs": null,
"position": null,
"required": true,
"type": "dokimi.commands.repo_hits._judge_ref#f8f456f0565f"
},
{
"action": "_StoreAction",
"choices": null,
"const": null,
"default": null,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": false,
"key": "--labels",
"kind": "option",
"nargs": null,
"position": null,
"required": false,
"type": null
},
{
"action": "_StoreAction",
"choices": null,
"const": null,
"default": null,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": false,
"key": "--max-usd",
"kind": "option",
"nargs": null,
"position": null,
"required": false,
"type": "dokimi.commands.repo_run._usd#591d6e2f0474"
}
],
"path": "dokimi repo hits judge",
"subcommand_required": null,
"subcommands": {}
},
"sheet": {
"allow_abbrev": true,
"params": [
{
"action": "_StoreAction",
"choices": null,
"const": null,
"default": null,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": false,
"key": "batch_dir",
"kind": "argument",
"nargs": null,
"position": 0,
"required": true,
"type": null
},
{
"action": "_HelpAction",
"choices": null,
"const": null,
"default": "==SUPPRESS==",
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": true,
"key": "--help/-h",
"kind": "option",
"nargs": 0,
"position": null,
"required": false,
"type": null
},
{
"action": "_StoreAction",
"choices": null,
"const": null,
"default": null,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": false,
"key": "--out",
"kind": "option",
"nargs": null,
"position": null,
"required": true,
"type": null
},
{
"action": "_StoreAction",
"choices": null,
"const": null,
"default": 25,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": false,
"key": "--per-arm",
"kind": "option",
"nargs": null,
"position": null,
"required": false,
"type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"
},
{
"action": "_StoreAction",
"choices": null,
"const": null,
"default": 0,
"group_key": null,
"group_required": null,
"hidden": false,
"is_flag": false,
"key": "--seed",
"kind": "option",
"nargs": null,
"position": null,
"required": false,
"type": "int"
}
],
"path": "dokimi repo hits sheet",
"subcommand_required": null,
"subcommands": {}
}
}
},
"inspect": {
"allow_abbrev": true,
"params": [
Expand Down Expand Up @@ -13157,6 +13390,7 @@
"'redteam'",
"'repo_budget'",
"'repo_environment'",
"'repo_hits_judged'",
"'repo_readiness'",
"'result'",
"'skill_evidence'",
Expand Down
96 changes: 93 additions & 3 deletions dokimi/docs/repo-evaluation-contract.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -680,6 +680,21 @@ review thread of a merged PR, anchored on a line of the reviewed revision.
(repository-relative), `line` (as read for matching), `severity` and
`message`, and their `distance` in lines; `comment`, `severity` and
`message` are cut to 200 characters. `repo report --hits` lists them (§9).
- Judged recall (`repo hits judge`, §9) is an optional, paid step after
grading; the grader stays deterministic and `grade.json` is not rewritten.
A judge model labels each credited pair `same point`, `related` or
`unrelated`, reading the thread's full `comment` from the gold file and the
finding's full `message` (with its `rule` and `evidence`) from the cell's
`answer.txt` at index `finding`; `matches` gives the pairing only.
`recall_judged` is the `same point` pairs over `n_threads`. Only credited
pairs are rescored: a thread no finding was credited with stays missed, and
a finding outside the window is not judged, so `recall_judged` is at most
`recall`. It is a judge's reading, not a verdict: quote a figure only next to
a person's labels for a sample of the pairs (`repo hits sheet`, then
`--labels`), whose agreement with the judge and Cohen's kappa per arm the
command reports; without them, or when no pair carries both a person's
and the judge's label, it says the figure is uncalibrated, and it names
the arms that no labelled pair covers.
- `recall_addressed` counts only threads with `addressed: true` (the code
changed in response), `recall_by_kind` groups by `kind`, and
`precision_proxy` is the share of findings within the window of any
Expand Down Expand Up @@ -1507,6 +1522,9 @@ python cli.py repo tasks replay ~/dpnp --threads threads.json --out ~/dpnp/.agen
python cli.py repo run ~/dpnp --task review_replay_pr3107 --arm guidance:none,guidance:raw \
--model claude-sonnet-5-5 --batch replay --agent-network host
python cli.py repo compare repo-runs/replay --baseline guidance:none --metric recall_addressed
# judged recall (§4.5): a sample for a person to label (no LLM), then the judge (paid)
python cli.py repo hits sheet repo-runs/replay --out labels.json
python cli.py repo hits judge repo-runs/replay --judge openai/gpt-5.1 --labels labels.json --max-usd 5
```

- Each cell holds the §4 inputs plus `agent.stdout` (the raw trace) and
Expand Down Expand Up @@ -1572,11 +1590,83 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re
`parse_errors` on every row. `repo report --hits` prints, instead of the table, each credited pair of
the `dokimi:review_replay` cells (`details.matches`, §4.5) by task, arm,
model and rep: the thread and its comment, then the finding that hit it.
`--hits --json` gives `{batch, replay_cells, without_matches, hits}`, one
`--hits --json` gives `{batch, replay_cells, without_matches, grader_failures, cells, hits}`, one
record in `hits` per pair (`task`, `arm`, `model`, `rep`, `cell` and the
match's fields; the first five come from the cell's `result.json`, and a
match key of the same name does not replace them). A cell graded before `matches` was recorded is counted
(`N cell(s) graded without hit details`), not listed.
match key of the same name does not replace them), and one in `cells` per
cell whose `matches` was read, with or without a pair (those five, and
`n_threads` and `recall` from `grade.json`). A cell graded before `matches` was recorded is counted
(`N cell(s) graded without hit details`), not listed, and so is a cell whose
`result.json` says `grader_failure` (`grader_failures`).
- `repo hits judge <batch> --judge PROVIDER/MODEL` (§4.5) labels each credited
pair of the batch's `dokimi:review_replay` cells, one judge call per pair
(up to 3 draws when a reply does not parse; the response cache replays
them), and writes `<cell>/hits_judged.json` (`repo_hits_judged.v1`): the
judge, each pair's `thread`, `finding`, `label` and `reason`, the counts of
`same point`, `related`, `unrelated` and `unjudged`, `n_threads` (from
`grade.json` `metrics`), the grader's `recall`, and `recall_judged`. A rerun
rewrites it. Before its first call, a run that judges (not `--dry-run`)
removes the `hits_judged.json` an earlier run left in every other finished
cell of the batch, one regraded since into a grader failure, a grade
without `matches` or another grader, and a line names those cells
(`--json`: `stale_removed`). The command prints, per (arm, model), the cells written, their
pairs and label counts, and the mean `recall` and `recall_judged` of those
cells (`--json`: the same as JSON). Before any call it refuses, with exit 2,
a judge that runs the model of any answer of the batch (`meta.json` or
`result.json` `model`, the model asked for, and `model_resolved`, the
model or models the trace says answered; a CLI alias counts as its family's
full id, the rule of `novelty`), and a `--labels` entry that names no
credited pair of the batch, has no `finding` or names another finding than
the one the batch credits for its thread (a hand-edit, or a sheet older
than a regrade: write a new one), or has a label other than the three, a
labelled entry whose `comment` or `finding_text` is missing or is not the
pair's text now (the thread's full comment in the gold file, the finding's
full text in `answer.txt`, compared exactly, with no normalisation), or whose
pair's texts are not found now (write a new sheet), and
two entries that give one pair two different labels (the same label twice,
or a `null` next to a label, is taken). A
cell graded without `matches`
and a grader-failure cell are counted, not judged. A pair whose full text is
not found (no single task directory or no such thread under
`<batch>/_eval/content`, no finding at its index in `answer.txt`, or a file
other than the one `matches` names) and a pair whose reply never parses are
`unjudged`, with `error` saying why, and never count as `same point`.
`--dry-run` counts the pairs and prints their cost at list price (one call
each; prompt characters over 4, 100 reply tokens) and makes no call.
`--max-usd X` starts no new call once X is spent: a cell not fully judged
gets no `hits_judged.json` (one an earlier run wrote is removed), and the
command exits 1. A judge reply with no known price stops such a run after
that call the same way, as `novelty`'s spend cap does: the cap cannot be
enforced on spend it cannot see, and the line naming the cells left says
so (`--json`: `budget_stop`; drop `--max-usd` to run unmetered). Without
`--max-usd` an unpriced call counts as 0, and the spend line says how many
calls had no price and that the spend is not known (`--json`:
`unpriced_calls`). A reply replayed from the response cache is free. A
`hits_judged.json` that cannot be written, or an earlier one that cannot
be removed (a read-only batch, a directory in its place), stops
the run there with no further call: a `✗` line names the path and the
error, the judge calls made and their cost, and says the cells written
before keep their files; the command exits 2. With `--labels FILE` it
reports, per arm, the pairs both the person and the judge labelled, the share with equal labels, and Cohen's
kappa over the three labels. Without it, or when no pair carries both the
person's and the judge's label (a sheet left `null`, `[]`, labels only on
pairs the judge left `unjudged`), a line says `recall_judged` is
uncalibrated; when only some arms have such pairs, a line names the others.
`--json` lists the arms with no such pair in `uncalibrated_arms` (every
arm without `--labels`); `calibration` is `null` without `--labels`.
- `repo hits sheet <batch> --out FILE` makes no call. It writes a JSON list of
credited pairs for a person to label: each with its `cell`, `task`, `arm`,
`model`, `rep`, `thread`, `finding`, their files and lines, the full
`comment` and `finding_text`, and `label: null`. It takes `--per-arm N`
pairs of each arm (default 25; all of an arm's pairs when it has fewer),
drawn with `random.Random(seed)` for `--seed N` (default 0), so a seed gives
one sheet; a pair whose full text is not found is left out. Set each
`label` to one of the three (`null` leaves the pair unlabelled) and pass the
file to `repo hits judge --labels`. When `--out` names a file that holds a
JSON list with an entry whose `label` is set, the command refuses with
exit 2 before writing, naming the path and the number of labelled entries
(move it aside or pass another `--out`); a file with no label set, or that
is not a JSON list, is replaced.
- `repo compare` reduces cells to one value per (task, arm, model) first:
repetitions share a task's difficulty and are not paired across arms, so the
task is the unit. Per treatment arm, model and measure (gate share, score,
Expand Down
Loading
Loading