feat(repo): time to first useful result, with a rate in repo compare

dokimi/changelog.d/20261031_120000_napetrov_repo-first-useful.md

@@ -0,0 +1,21 @@
+### Added
+
+- **Time to first useful result, per repo-eval run** — a `task.json` can declare
+  `"useful": {"command": "<regex>"}` for a task graded by its target tests, or
+  `"useful": {"edit": "<glob>"}` (or a list of globs) for one graded by an edit to a target
+  file. The first non-denied event with `rc` 0 that hits it (a command whose text matches, or an
+  edit whose path matches) is recorded in `result.json` `first_useful` with its `issued_s` (the
+  event's issue time) and `steps` (events up to and including it). A run with no hit says
+  why (`never`, `unobserved`, or `timeout`: a `command` pattern, searched with the `regex` module, ran past
+  2 s on one event, so the run has no verdict); a task with no declaration is `not_declared`. A hit in a trace
+  without timestamps is `untimed`: it keeps its `steps` and counts as observed, with no seconds.
+  `repo report` shows a row's observed count, medians and reasons under the row, and
+  `repo compare` pairs the per-task medians as `first_useful_s` and `first_useful_steps`, and
+  the share of the cells with a verdict that observed a hit as `first_useful_rate` (`unobserved` and `timeout`
+  cells are unmeasured: neither a hit nor a miss); its header counts observed of measured cells per arm and
+  how many are unmeasured.
+  A malformed `useful` is refused by `repo validate`, `repo tasks check` and `repo run`.
+  Batches run before this change have no `first_useful` and no such rows.
+- A Codex `file_change` record whose `changes` has a malformed entry (not an object, or no path), or is a
+  non-list value other than `null`, is one dropped line, as the other malformed records are. The whole record is dropped: its valid
+  entries are not read either, so no edit from it counts.

dokimi/docs/repo-evaluation-contract.md

@@ -307,6 +307,7 @@ belongs to the repository's grader and is passed through unread, except `grader`
 | `host` | no | framework | `{"min_cpus": <int>, "cpu_vendors": [<str>], "cpu_flags": [<str>]}`, every key optional: what the hidden test needs to show the bug (a thread race that needs 3+ CPUs, an ISA path). Vendor and flag names are `/proc/cpuinfo`'s (`GenuineIntel`, `avx512f`); both match in any case, and a vendor other than `GenuineIntel`, `AuthenticAMD`, `HygonGenuine`, `CentaurHauls`, `Shanghai` is refused (`repo validate` too). A probe that read neither vendor nor flags from `/proc/cpuinfo` is a probe failure, not a miss They are checked against a probe container started like the grading one: CPUs are its affinity mask capped by its cgroup CPU quota (`docker --cpus`). On a host that misses them the task is unsupported (§4.3), and the probed facts go to `meta.json` `host` of a cell that runs, or to its `unsupported.json` `host` when it does not |
 | `egress` | no | framework | extra hosts (DNS names, optional `:port`; `*` stands for one whole label and the last two labels are literal, so `*.pythonhosted.org` passes and `*` or `*.org` does not) this task's agent and entry points reach through the proxy (§2), for example `["pypi.org", "files.pythonhosted.org"]` |
 | `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. `result.json` `skills` records whether the agent used it and whether it used it first |
+| `useful` | no | framework | the definition of the run's first useful result, reported as `first_useful` in `result.json` (§4.1): `{"command": "<regex>"}` for a task whose grade runs the target tests, or `{"edit": "<glob>"}` (or a list of globs) for one that grades an edit to a target file. Omit it when no event is the useful result. A malformed value is refused by `repo validate`, `tasks check` and `repo run` |
 | anything else | — | repository | for example `hidden_files`, `targets`, `regression_targets`, `manifest` |
 
 `setup_patch` (a single string) is accepted as a synonym for a one-element
@@ -471,6 +472,36 @@ parent's and subagents' together, so a grader need not:
 
 `process` is `null` when the trace could not be read, the agent left none (no event and no result record), or computing the metrics failed (the error is in `agent.trace_errors`).
 
+`result.json` `first_useful` is the run's first useful result: the first event that
+hits the task's `useful` key (§3). The definition depends on the grader:
+
+| Grader | Declare | Hit |
+|---|---|---|
+| test: the grade runs the target tests | `{"command": "<regex>"}` | the first `command` event whose text matches the regex (`regex.search`) and whose `rc` is `0`. `rc` 0 says that command passed, not that every target test did, so the regex should name the target tests |
+| edit or diff: the grade reads a target file | `{"edit": "<glob>"}` or a list | the first `edit` event whose path matches a glob (`match_glob`, as the guidance globs) |
+| review (`dokimi:review_*`) | none | none, so `not_declared`: the answer is the product, so no event is the useful result. A declared key is measured all the same |
+| any other repository grader | either form, or none | none declared, `not_declared` |
+
+A refused event (`denied`) never hits, nor does an `edit` without a success result (`rc` 0): an error result (`rc` 1) means the file was not changed, and no result at all (a killed run) proves nothing. `steps` counts every event up to and
+including the hit, the parent's and subagents' (as `process.tool_calls` does).
+`issued_s` is the hit's `t`: the time the call was issued, since a call's result
+is not timed. `first_useful` is `{"kind": "command"|"edit"|null, "reason": ...,
+"issued_s": number|null, "steps": int|null}`. `reason` is `null` when the hit was
+found and timed; otherwise it says why `issued_s` (and `steps`, with no hit) is
+null: `not_declared` (no definition), `unobserved` (the trace was not read: the
+harness never started, it did not parse, or a line of the trace was dropped: an unreadable record or block, such as a tool call without an id, a name or an object of input, a content block without a string type, a Codex file change without a path, or Copilot shell, read or edit arguments that are not an object), `never` (no event hit), `timeout` (a `command` pattern ran past
+2 s on one event: no verdict, and the rest of the trace is not searched) or `untimed`
+(the trace format has no timestamps, so `steps` is set and `issued_s` is null).
+A cell of an older batch has no `first_useful`. Each cell that declares a definition
+is counted in `repo report`: the row says how many observed a hit, and the medians
+cover only those. `repo compare` pairs `first_useful_rate` per task (the share of its
+declared cells with a verdict that observed a hit, so a task where most runs never hit it is not read
+through its medians alone; a cell `unobserved` or `timeout` has no verdict, so it is neither a hit nor
+a miss and is left out of the share) and its header counts observed of measured cells per arm, with
+the unmeasured ones. An edit made by a shell command (`sed -i`) is not an `edit` event, so it is not seen. A
+compared task has one kind, but the mean delta pools the tasks of both kinds: a per-task
+row shows which.
+
 A cell of an arm that installs skills also has a `skills` object in
 `result.json`: `measured`, `skills_installed` (the installed skill names),
 `skills_used` (the installed skills the agent called with the `Skill` tool or
@@ -1482,13 +1513,15 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re
 - `repo compare` reduces cells to one value per (task, arm, model) first:
   repetitions share a task's difficulty and are not paired across arms, so the
   task is the unit. Per treatment arm, model and measure (gate share, score,
-  cost, turns, `first_prompt_tokens`, each `--metric` from `grade.json`, and each `process` count a
+  cost, turns, `first_prompt_tokens`, `first_useful_s` and `first_useful_steps` (the median over a
+  task's cells that observed the first useful result, §4.1), `first_useful_rate` (the share of a task's
+  declared cells with a verdict that observed one), each `--metric` from `grade.json`, and each `process` count a
   compared cell reports as `process:<key>` or `counter:<name>`) it gives the mean per-task
   delta against the baseline with a 95% interval over tasks (a bootstrap for
   gate and score, a t interval for the descriptive measures, `-` for a single
   task), the tasks up, down and tied, and a two-sided sign-flip permutation p (exact up to 16
   changed tasks; at one cell per task and arm it equals McNemar's exact test).
-  A `process` count or `first_prompt_tokens` only one arm reports for the shared tasks of a
+  A `process` count, `first_prompt_tokens` or `first_useful_*` only one arm reports for the shared tasks of a
   (treatment, model) is not compared; a warning names it. When neither arm
   reports it (a batch run before `repo run` recorded it), the row is absent.
   Holm's correction runs over the gate and score comparisons of one call; the

dokimi/dokimi/repos/cli_traces.py

@@ -531,13 +531,16 @@ def codex_json(raw: str, workdir: str | None, cwd: str = "") -> Transcript:
         try:
             rec = json.loads(line)
         except json.JSONDecodeError:
+            out.dropped_lines += 1
             out.errors.append(f"line {n}: not JSON")
             continue
         if not isinstance(rec, dict):
+            out.dropped_lines += 1
             continue
         try:
             _codex_apply(out, rec, messages, workdir, by_id, cwd, event_reads)
         except (AttributeError, TypeError, ValueError) as exc:
+            out.dropped_lines += 1
             out.errors.append(f"line {n}: unexpected shape ({exc})")
     out.answer = messages[-1].strip() if messages else ""
     out.agent_messages = messages
@@ -617,9 +620,18 @@ def _codex_item_completed(out: Transcript, item: dict[str, Any], messages: list[
         else:
             _codex_command_unstarted(out, item, script, workdir, cwd)
     elif itype == "file_change":
-        for change in item.get("changes") or []:
-            if isinstance(change, dict) and change.get("path"):
-                out.events.append({"t": None, "kind": "edit", "path": relative_path(str(change["path"]), workdir)})
+        status = item.get("status")  # a failed patch changed nothing
+        rc = {"completed": 0, "failed": 1}.get(status) if isinstance(status, str) else None
+        changes = [] if item.get("changes") is None else item["changes"]
+        # an entry that is not a change makes the record malformed: it is dropped whole (counted), not read in part
+        if not isinstance(changes, list) or not all(
+                isinstance(c, dict) and isinstance(c.get("path"), str) and c["path"] for c in changes):
+            raise ValueError("file_change with a malformed changes entry")
+        for change in changes:
+            event: dict[str, Any] = {"t": None, "kind": "edit", "path": relative_path(str(change["path"]), workdir)}
+            if rc is not None:
+                event["rc"] = rc
+            out.events.append(event)
     elif itype in ("mcp_tool_call", "web_search"):
         name = f"{item.get('server')}.{item.get('tool')}" if itype == "mcp_tool_call" else "web_search"
         out.events.append({"t": None, "kind": "tool", "name": name})
@@ -712,14 +724,17 @@ def copilot_json(raw: str, workdir: str | None, cwd: str = "") -> Transcript:
         try:
             rec = json.loads(line)
         except json.JSONDecodeError:
+            out.dropped_lines += 1
             out.errors.append(f"line {n}: not JSON")
             continue
         if not isinstance(rec, dict):
+            out.dropped_lines += 1
             continue
         t = run.offset(rec)
         try:
             run.apply(out, rec, t, workdir, cwd)
         except (AttributeError, TypeError, ValueError) as exc:
+            out.dropped_lines += 1
             out.errors.append(f"line {n}: unexpected shape ({exc})")
     out.answer = (run.final or run.last).strip()
     out.num_turns = run.turns or None
@@ -730,14 +745,17 @@ def copilot_json(raw: str, workdir: str | None, cwd: str = "") -> Transcript:
 def _copilot_tool(data: dict[str, Any], t: float | None, workdir: str | None, cwd: str = "") -> list[dict[str, Any]]:
     name = str(data.get("toolName") or "")
     args = data.get("arguments")
+    if name in _COPILOT_SHELL | _COPILOT_READ | _COPILOT_EDIT and not isinstance(args, dict):
+        raise TypeError(f"{name} arguments are not an object")  # counted as a dropped line by the caller
+    obj: dict[str, Any] = args if isinstance(args, dict) else {}
     if name in _COPILOT_SHELL:
-        command = str((args or {}).get("command", "")) if isinstance(args, dict) else ""
+        command = str(obj.get("command", ""))
         # only bash is a POSIX script; a PowerShell command is not parsed for the files it reads
         scripts = shell_reads(command, cwd) if name == "bash" else []
         reads = [{"t": t, "kind": "read", "path": relative_path(p, workdir)} for p in scripts]
         return [*reads, {"t": t, "kind": "command", "text": command, "rc": None}]
-    if name in _COPILOT_READ | _COPILOT_EDIT and isinstance(args, dict):
-        path = str(args.get("path") or args.get("file_path") or "")
+    if name in _COPILOT_READ | _COPILOT_EDIT:
+        path = str(obj.get("path") or obj.get("file_path") or "")
         if not path:  # a call without a path names no file
             return []
         return [{"t": t, "kind": "read" if name in _COPILOT_READ else "edit", "path": relative_path(path, workdir)}]
@@ -752,6 +770,9 @@ def _copilot_result(out: Transcript, events: list[dict[str, Any]], data: dict[st
     reads = [e for e in events if e["kind"] == "read"]
     failed = data.get("success") is False
     for event in events:
+        if event["kind"] == "edit" and data.get("success") is not None:
+            # a failed edit changed nothing: rc 1, so it is no useful result
+            event["rc"] = 0 if data.get("success") else 1
         if event["kind"] != "command":
             continue
         shell = data.get("shellExecution") or {}