feat(eval): add eval headroom to flag items with no headroom

dokimi/changelog.d/20261028_120000_napetrov_feat-eval-headroom.md

@@ -0,0 +1,3 @@
+### Added
+
+- **`eval headroom` flags arms items with no headroom** — reads one or more `arms.v1` files of the same question set and flags an item as a `ceiling` when the no-skill arm's mean is at least `--ceiling` (default 0.9, on 0-1), or as a `floor` when the no-skill arm was scored, every arm that a run scored has a mean at most `--floor` (default 0), and at least one arm other than the no-skill arm scored. An item with fewer than `--min-reps` runs (default 3) scoring its no-skill arm is `provisional`. The human table goes to stdout; `--out` writes a `headroom.v1` artifact with every item. The flags change no score. Inputs that are not repetitions of one configuration (model, judge, run config other than the seed, harness, arms or library differ) exit 2. Pooling also refuses a file scored by `eval reference` next to a judged one, an arm whose profile prompt digest changed, and a question judged grounded in one file and ungrounded in another.

dokimi/docs/evaluating-treatments.md

@@ -724,6 +724,52 @@ Two gates are deliberately statistical rather than absolute:
   those questions. It pools questions; a mean of per-cell means differs slightly
   when the sets have different sizes — say which one a report uses.
 
+## Headroom: items a comparison cannot learn from
+
+A skill cannot raise an item the no-skill arm already answers, and an item no arm
+answers points at the question or its grader, not at the skill. `eval headroom` flags
+both, from one or more arms files of the same question set (one file per repetition):
+
+```bash
+python cli.py eval headroom run1/arms.json run2/arms.json run3/arms.json \
+    --baseline-arm baseline --ceiling 0.9 --floor 0 --min-reps 3 --out headroom.json
+```
+
+- **ceiling**: the no-skill arm's mean over the runs is at least `--ceiling` (0-1; the
+  0-100 aggregate divided by 100). Default 0.9.
+- **floor**: the no-skill arm was scored, every arm that a run scored has a mean at most
+  `--floor`, and at least one arm other than the no-skill arm scored. Default 0. An arm no run scored is
+  left out of this test and listed in the item's `unscored_arms`; the no-skill arm alone at zero
+  is not a floor.
+- **provisional**: fewer than `--min-reps` runs scored the item with its no-skill arm (the
+  reps are those runs). Default 3. A file gives each item zero or one rep (zero when its
+  no-skill arm was not scored in that file), so one file alone makes every item provisional at the
+  default of 3; the reps of several files add up.
+
+A run is one file. A file repeated on the command line, or holding the same JSON content as
+an earlier one (whitespace and key order do not count), is skipped with a note, so it does
+not add a rep. Numbers are compared as numbers: `95.0` and `95` are the same content. A question id repeated within one
+file is an error (exit 2): it would count as two runs.
+
+The table always goes to stdout: the flagged and provisional items, with the no-skill arm
+in its own column, a note naming any arm no run scored, and the counts. The JSON goes to
+`--out` only, as a `headroom.v1` artifact with every item; if that file cannot be written the
+command prints `Error: cannot write ...` on stderr and exits 2, after the table. Scores are compared unrounded
+and shown rounded down to up to nine decimal places, so 0.89996 prints as 0.89996 and a value just under a threshold never
+prints as the threshold. Without `--baseline-arm`, the runs must
+declare the same baseline; the default is that baseline. The runs must also be repetitions of one
+configuration: a file whose model, judge, run config (the seed aside), harness, arms or library
+differs from the first file's exits 2. So does a file scored by `eval reference` next to a judged one, an arm whose
+recorded profile prompt digest differs between files (an edited profile keeps its name), and a question
+id judged grounded in one file and ungrounded in another (a file with no `grounded_questions` count is not
+compared). With `--baseline-arm`, the arm must
+be one the runs list (exit 2 otherwise, naming the arms that exist), and the declared
+baselines are not checked. A threshold out of range exits 2: `--ceiling` in [1e-6, 1],
+`--floor` 0 or in [1e-6, ceiling), `--min-reps` at least 1. A positive threshold may be finer than one score step. It also exits 2 when no run scored the
+no-skill arm, or when `--out` is one of the input files (or its `.rejected` sibling), or names a directory or a path ending in a separator. Scores are read as the 0-100
+aggregate divided by 100 and are not range-checked. Only arms runs are read. The flags are descriptive: they do not
+change any score or any gate.
+
 ## Scope and caveats
 
 - **Three levels of fidelity for skills/MCP.** (1) `skill:` / `mcp:` inject

dokimi/docs/reference/cli-interface.json

@@ -4489,6 +4489,126 @@
           "subcommand_required": null,
           "subcommands": {}
         },
+        "headroom": {
+          "allow_abbrev": true,
+          "params": [
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "arms",
+              "kind": "argument",
+              "nargs": "+",
+              "position": 0,
+              "required": true,
+              "type": null
+            },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--baseline-arm",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": null
+            },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": 0.9,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--ceiling",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": "float"
+            },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": 0.0,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--floor",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": "float"
+            },
+            {
+              "action": "_HelpAction",
+              "choices": null,
+              "const": null,
+              "default": "==SUPPRESS==",
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": true,
+              "key": "--help/-h",
+              "kind": "option",
+              "nargs": 0,
+              "position": null,
+              "required": false,
+              "type": null
+            },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": 3,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--min-reps",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"
+            },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--out",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": null
+            }
+          ],
+          "path": "dokimi eval headroom",
+          "subcommand_required": null,
+          "subcommands": {}
+        },
         "judge-validity": {
           "allow_abbrev": true,
           "params": [
@@ -13148,6 +13268,7 @@
                 "'eval_coverage'",
                 "'explore_runs'",
                 "'guidance_reach'",
+                "'headroom'",
                 "'matrix_rollup'",
                 "'need_graph'",
                 "'novelty'",

dokimi/dokimi/artifacts.py

@@ -61,6 +61,7 @@
     "repo_budget": "repo_budget.v1",
     "effective_instructions": "effective_instructions.v1",
     "task_inspect": "task_inspect.v1",
+    "headroom": "headroom.v1",
     "redteam": "redteam.v1",
     "cli_routes": "cli_routes.v1",
     "need_graph": "need_graph.v1",

dokimi/dokimi/cli_routes.py

@@ -151,6 +151,7 @@ class Tombstone:
     Leaf(("eval", "pairwise"), PAID),
     Leaf(("eval", "reference"), PAID),
     Leaf(("eval", "native-import"), FREE),
+    Leaf(("eval", "headroom"), FREE),
     Leaf(("eval", "redteam"), PAID),
     Leaf(("eval", "panel-score"), PAID),
     Leaf(("dataset", "export"), FREE),

dokimi/dokimi/commands/evaluate.py

@@ -580,6 +580,10 @@ def register(sub, positive_int) -> None:
 
     answer_formats.register_eval(eval_sub, positive_int)  # eval reference, eval native-import
 
+    from dokimi.commands import headroom
+
+    headroom.register_eval(eval_sub, positive_int)  # eval headroom
+
     from dokimi.commands import redteam
 
     redteam.register_eval(eval_sub)  # eval redteam