feat(repo): rule ledger: repo compare --ledger with per-rule keep/review/insufficient-evidence verdicts

dokimi/benchmarks/spec.v1.yaml

@@ -76,7 +76,7 @@ future:
 # that matched almost nothing, max_docs one that escaped; raise it with each new doc.
 golden_manifest:
   min_docs: 10
-  max_docs: 37
+  max_docs: 38
   include:
     - docs/**/*.md
   exclude:

dokimi/changelog.d/20261031_120000_napetrov_repo-rule-ledger.md

@@ -0,0 +1,3 @@
+### Added
+
+- **`repo compare --ledger FILE`** — gives each guidance rule of a `repo_ledger.v1` file a verdict: its arm against its baseline on the rule's linked cases (task ids), with the paired statistics of `repo compare`, Holm correction across the rules, and `keep`, `review` or `insufficient-evidence` with the reason. A rule whose `reviewed` date is older than `review_within_days` (default 180) gets a warning; its verdict is computed as usual. The ledger only computes verdicts; cutting a rule is left to a person. The design, the ledger format and the incident-to-case path are in `docs/rule-ledger.md`. `--json` adds a `ledger` key to the comparison.

dokimi/docs/README.md

@@ -368,6 +368,10 @@ code.
 - [preparing-a-repository-evidence.md](preparing-a-repository-evidence.md) —
   the runs behind each figure on that page: task set, arms, model, cell count
   and raw count.
+- [rule-ledger.md](rule-ledger.md) — one row per guidance rule: its
+  hypothesis, the replay or executable cases of the failures it prevents, and
+  the review date; `repo compare --ledger` gives each rule a keep, review or
+  insufficient-evidence verdict. The cut decision stays with a person.
 
 ## Extending Dokimi
 

dokimi/docs/architecture.md

@@ -347,6 +347,10 @@ next release.
   what its `--task` selection keeps; `dokimi/repos/compare_reach.py` how small
   an effect a row can find (`mde`, an uncorrected t-based estimate, and the
   lowest `p_holm` Holm's family leaves it).
+- `dokimi/repos/ledger.py` — `repo compare --ledger`: loads a `repo_ledger.v1`
+  file (rules with their hypothesis, arm, linked cases and review date) and gives
+  each rule a verdict from `repo compare` over its own cases, Holm-corrected
+  across the rules. [rule-ledger.md](rule-ledger.md).
 - `dokimi/repos/selfcheck.py` — `repo tasks check`: on each task
   `grade` must fail the untouched workspace and pass it after `oracle`, and an
   isolation probe must not see the tasks or the source from the agent's

dokimi/docs/reference/cli-interface.json

@@ -10615,6 +10615,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--ledger",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": null
+            },
             {
               "action": "_AppendAction",
               "choices": null,
@@ -13121,6 +13137,7 @@
                 "'redteam'",
                 "'repo_budget'",
                 "'repo_environment'",
+                "'repo_ledger'",
                 "'repo_readiness'",
                 "'result'",
                 "'skill_evidence'",

dokimi/docs/rule-ledger.md

@@ -0,0 +1,152 @@
+<!--
+Copyright (C) 2026 Intel Corporation
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Rule ledger: past failures as replay cases
+
+A guidance rule is a bet: the author expects an agent that reads it to make a
+specific mistake less often. The ledger keeps one row per rule with that bet
+written down, the cases that stand for the failures the rule was written to
+prevent, and the date a person last confirmed it still earns its place.
+`repo compare --ledger` then checks the bet on those cases only and gives a
+verdict. It does not cut anything. This follows the held-out ablation proposal
+in [agentsmd/agents.md#213](https://github.com/agentsmd/agents.md/issues/213).
+
+## What exists
+
+| Piece | Where | What it does |
+|---|---|---|
+| Replay tasks from review threads | `dokimi/repos/replay/`, `repo tasks replay` ([contract §4.6](repo-evaluation-contract.md#46-replay-tasks-from-review-threads)) | One `dokimi:review_replay` task per merged PR, with the change as reviewed and the reviewers' threads as gold. No GitHub call. |
+| Deterministic review grader | `dokimi/repos/graders/review.py` | Matches the agent's findings JSON to gold threads by file and a line window (`window`, default 5). No LLM. |
+| Executable tasks and self-check | `repo tasks check` (`dokimi/repos/selfcheck.py`, [contract §4.3](repo-evaluation-contract.md#43-self-check-tasks-check-no-llm)) | Checks that a task fails on the untouched workspace and passes after its oracle. Run this check and resolve broken cases before starting paid runs; `repo run` does not enforce its results. |
+| Arms | `dokimi/repos/arms.py` | `guidance:none`, `guidance:raw`, `guidance:rev:`, `patch:` and the rest. |
+| Repetitions | `repo run --reps` | Several reps per (task, arm, model). |
+| Paired statistics | `repo compare` (`dokimi/repos/compare.py`) | Per-task deltas, sign-flip p, Holm over the gate and score rows of one invocation. |
+| Rule vocabulary | [repo-evaluation-agent-modes.md §4](repo-evaluation-agent-modes.md#4-team-review-rules) | Mined `review_rules.json` entries (`R13`, `cut_date`). Method only: no code reads that file yet. |
+
+Before this change nothing tied a rule to the cases it was written from, and
+`repo compare` had no notion of a rule.
+
+## From an incident to a held-out case
+
+An incident is a failure a rule should prevent. Three sources are in use or
+planned:
+
+1. **A review comment on a merged PR.** Run `repo tasks replay` on the PR's
+   threads. The case is `review_replay_pr<N>` and already exists. Its grader is
+   recall of the gold threads, so the check is deterministic.
+2. **A failed agent run, or a bug a fix PR corrected.** Write an executable task
+   in the content repository: a test that fails on the base commit and passes
+   with the oracle. `repo tasks check` enforces both halves. No scaffold command
+   exists yet; the task is written by hand.
+3. **A transcript slice.** A transcript shows what the agent did, not what
+   should have happened. It is evidence for case 1 or 2, not a case: a case
+   needs a check that does not read the transcript.
+
+Then, per rule:
+
+1. Record the incident as the rule's `source` (a file and anchor, or the
+   comment URL) and state the `hypothesis`: what the rule should change in
+   an answer on its cases.
+2. Put the case ids in `cases`. Use cases the rule was not written from: a
+   rule checked on the failure it was mined from only shows it can repeat
+   itself.
+3. Run `repo tasks check` on the cases, then the rule's arm and its baseline
+   with `--reps 3` to `5`.
+4. Compare with `repo compare --ledger`.
+
+## Ledger format (`repo_ledger.v1`)
+
+`dokimi/schemas/repo_ledger.v1.json` is the schema. `rules` must not be
+empty, and ids must be unique in the file (the loader checks this, since JSON
+Schema cannot).
+
+```json
+{
+  "schema_version": "repo_ledger.v1",
+  "policy": {"alpha": 0.05, "review_within_days": 180},
+  "rules": [
+    {
+      "id": "R13",
+      "rule": "Revert edits to files the change does not need.",
+      "source": "AGENTS.md#commits-and-pull-requests",
+      "hypothesis": "With the rule, the agent's review findings land on more of the reviewers' threads on the replayed PRs.",
+      "arm": "guidance:raw",
+      "baseline": "guidance:none",
+      "cases": ["review_replay_pr3580", "review_replay_pr3659"],
+      "reviewed": "2026-10-01"
+    }
+  ]
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `id` | Stable across edits, `[A-Za-z0-9][A-Za-z0-9._-]{0,63}` (at most 64 characters). |
+| `rule`, `source` | Optional text and anchor, printed for the reader. |
+| `hypothesis` | Required. The claim the verdict tests, in words. |
+| `arm` | The treatment arm that carries the rule, `ARM` or `BATCH/ARM`. |
+| `baseline` | Optional. Overrides the `--baseline` of the compare for this rule. |
+| `cases` | Task ids, at least one, unique. Ids with no finished cell are listed as missing. |
+| `reviewed` | `YYYY-MM-DD`, the last date a person confirmed the rule. |
+| `policy.alpha` | Default 0.05; a number strictly between 0 and 1. |
+| `policy.review_within_days` | Default 180. An older `reviewed` gets a warning. |
+
+## Verdicts
+
+For each rule and model, `repo compare --ledger` runs the ordinary paired
+comparison of the rule's arm against its baseline over the rule's linked cases
+(`repo compare` itself, so the pairing, headroom and provenance warnings are
+the same). The p values of the rule results are Holm-corrected together. A result whose
+linked cases mix gate and score tasks has no single test, so it takes no slot in
+that family and is `insufficient-evidence`. The rules' family is separate from the
+family of the comparison's own rows.
+
+| Verdict | When | Meaning |
+|---|---|---|
+| `keep` | Mean delta above zero and Holm p below alpha | The arm beats its baseline on the linked cases. |
+| `review` | Mean delta below zero and Holm p below alpha | The baseline beats the arm there. A person decides whether to fix or cut the rule. |
+| `insufficient-evidence` | Anything else | Not measured, mixed kinds, every case tied, no significant difference, or too few changed cases for any outcome to reach alpha. The reason says which. |
+
+A review older than `review_within_days` adds a warning and leaves the verdict
+alone. "No significant difference" is not a verdict to cut: the rule is not
+shown to help, and whether that is enough to cut it is a decision for a person.
+
+## CLI
+
+```bash
+python cli.py repo compare repo-runs/replay --baseline guidance:none --ledger repo-ledger.json
+python cli.py repo compare repo-runs/replay --baseline guidance:none --ledger repo-ledger.json --json
+```
+
+The text output keeps the ordinary comparison and adds a `rule ledger` section.
+A ledger that does not validate exits 2 with a `✗` line naming the file (and the rule,
+where the error is the rule's). A `--baseline` the batches do not have exits 2 with the
+ordinary comparison's `✗` line, which does not name the ledger: the comparison runs first.
+A rule whose own `baseline` or `arm` has no cell among its cases is `not measured` for
+that rule, and the command still exits 0. The ordinary comparison runs first, so a
+`--task` or `--arm` it rejects (no cell matches) exits 2 before any ledger is evaluated.
+Once it succeeds, its `--arm`, `--task` and `--metric` filters do not change the ledger,
+which uses the rules' own cases and arms. `--model` does apply: it limits the ledger to
+those models, which changes its Holm family and so its verdicts.
+
+## Limits
+
+- A rule's arm carries every rule that is in it. A verdict on one rule compares
+  the whole arm with its baseline, so it confounds the rule with its neighbours.
+  A per-rule claim needs an arm that differs by that rule alone (a `patch:`
+  arm that removes one rule, say). The ledger does not build those arms.
+- Few cases give few verdicts. With fewer than six cases whose delta is not zero,
+  no outcome reaches p below 0.05, and the reason says so.
+- Gate and score cases do not mix in one rule.
+
+## Not built
+
+- A scaffold that turns a failed run directory, or a description and a failing
+  check, into a task skeleton. The executable-task format needs an oracle and a
+  verifier, so a skeleton would still need hand work.
+- A check that a rule's cases are held out from the PRs it was mined from. The
+  [mining leak guard](repo-evaluation-agent-modes.md#4-team-review-rules)
+  (`cut_date`) is the method; nothing enforces it for a ledger yet.
+- Reading the mined `review_rules.json` into the ledger.

dokimi/dokimi/artifacts.py

@@ -61,6 +61,7 @@
     "task_inspect": "task_inspect.v1",
     "redteam": "redteam.v1",
     "cli_routes": "cli_routes.v1",
+    "repo_ledger": "repo_ledger.v1",
     "need_graph": "need_graph.v1",
     "eval_coverage": "eval_coverage.v1",
 }

dokimi/dokimi/cli_flags.py

@@ -242,6 +242,7 @@ class Concept:
             Flag("--answers", "FILE", "path"),
             Flag("--registry", "FILE", "path"),
             Flag("--from", "RUN", "text"),
+            Flag("--ledger", "FILE", "path"),
         ),
         banned=("--product", "--library", "--libraries", "--target", "--root", "--repo-root",
                 "--tasks-root", "--subjects-root", "--questions-root", "--questions-from",

dokimi/dokimi/commands/repo_run.py

@@ -352,8 +352,10 @@ def cmd_repo_report(args: argparse.Namespace) -> None:
 
 
 def cmd_repo_compare(args: argparse.Namespace) -> None:
-    """Per (treatment, model, measure): per-task deltas against the baseline, 95% CI, sign-flip p."""
+    """Per (treatment, model, measure): per-task deltas against the baseline, 95% CI, sign-flip p; ``--ledger``
+    adds one verdict per guidance rule."""
     from dokimi.repos.compare import CompareError, compare, format_compare
+    from dokimi.repos.ledger import LedgerError, evaluate, format_ledger, load_ledger
     from dokimi.repos.report import CellFileError
 
     dirs = [Path(d) for d in args.batch_dirs]
@@ -365,10 +367,16 @@ def cmd_repo_compare(args: argparse.Namespace) -> None:
     try:
         summary = compare(dirs, args.baseline, arms, args.model, tuple(args.metric or ()),
                           pass_k=args.pass_k, path_stability=args.path_stability, tasks=args.task)
-    except (CompareError, CellFileError) as exc:  # an unusable baseline, arm, metric or cell file: exit 2
+        if args.ledger is not None:
+            summary["ledger"] = evaluate(load_ledger(Path(args.ledger)), dirs, args.baseline, args.model,
+                                         ledger_name=args.ledger)
+    except (CompareError, CellFileError, LedgerError) as exc:  # an unusable baseline, arm, metric, cell or ledger
         print(f"✗ {exc}", file=sys.stderr)
         sys.exit(EXIT_USAGE)
-    print(json.dumps(summary, indent=2) if args.json else format_compare(summary))
+    text = format_compare(summary)
+    if "ledger" in summary:
+        text += "\n\n" + format_ledger(summary["ledger"])
+    print(json.dumps(summary, indent=2) if args.json else text)
 
 
 def _usd(text: str) -> float:
@@ -520,5 +528,8 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None:
                            help="Also compare pass^K: the chance K cells of a task all pass its gate")
     compare_p.add_argument("--path-stability", action="store_true", dest="path_stability",
                            help="Also compare path_distance: how much the tool paths of a task's cells differ")
+    compare_p.add_argument("--ledger", default=None, metavar="FILE",
+                           help="Also give each guidance rule of this repo_ledger.v1 file a verdict (keep, review or "
+                           "insufficient-evidence) from its linked cases, against its baseline (dokimi/docs/rule-ledger.md)")
     compare_p.add_argument("--json", action="store_true", help="Print the comparison as JSON")
     compare_p.set_defaults(func=cmd_repo_compare)