[antonwolfy]

Summary

Follow-up to the reviews of #116 and #280, which left three points open.

• feat(repo): Codex and Copilot CLI harness profiles, native review mode #116 (review, "Findings that do not anchor inline", item 1): section 7 of docs/preparing-a-repository.md said "Other harnesses (Codex, Copilot and others) are not listed because no loading behavior was measured for them." The built-in profiles say otherwise: dokimi/repos/profiles/codex.yaml records "Measured on 0.160 with gpt-6-luna (oneDAL review task): root AGENTS.md applied, …", and copilot.yaml records "Measured on Copilot CLI 1.0.91 (BYOK, …)". The checklist half of that item was fixed in 25dd17d; this sentence was not.
• fix(eval): infra errors and empty replies are counted as errors, never scored #280 (review, "What I ran"): the mutation run left two outputs that no test pins. On main each of these changes leaves the suite green:

Output	Code on main	Mutant that passes on main
A check whose judge call failed carries "error" in its checks entry	CheckResult.as_dict, dokimi/eval/reference_checks.py:91-92	delete the two error lines
The baseline arm's per_arm[<baseline>]["delta_vs_baseline"] is present and null	ArmRunner._summarize, dokimi/eval/arm_runner.py:769-771	else None → else 0.0

Changes

• docs/preparing-a-repository.md (section 7): the sentence under the table now says Codex (codex-cli 0.160) and Copilot CLI (1.0.91) were measured, and points at the guidance.notes of the built-in codex and copilot profiles. A harness without a built-in profile still needs its own harness_profile.v1 and a dokimi repo reach measurement, and the text-trace sentence stays. The table still has only the Claude Code row: the paragraph below it already lists the five built-in profiles and says repo reach refuses codex-review.
• tests/test_reference_checks.py: new test_the_failed_judge_check_names_its_error_in_the_cell_and_the_graded_check_has_none. It runs eval reference's score() with a judge that raises TimeoutError on a question with a match check and a judge check. The judge check's entry in the cell's checks must carry "error": "judge: TimeoutError", and the match entry must be exactly {"type": "match", "passed": true, "score": 1.0}, with no error key.
• tests/test_arm_cell_errors.py: test_delta_vs_baseline_is_paired_over_the_questions_both_arms_scored now also asserts that the baseline arm has the delta_vs_baseline key and that it is None. In that test the baseline has scored questions, so a self-paired delta would come out as 0.0.
• No changelog fragment: nothing under dokimi/dokimi/, dokimi/data/ or the install metadata changes, and CI's Fragment for package changes step agrees.

Risk

None: no package code changes. The doc change is one paragraph of section 7.

Testing

On a Linux review host, at 989ef44 (on main 27cc760):

• make ci-local: 30 passed, 0 failed, 7 skipped. Skipped: agent-quality.yml:external-catalog-smoke (needs docker and network), conda-forge.yml:conda-forge-deps (micromamba not on PATH), semgrep secrets (offline; runs on the PR), cli-interface.yml:cli-surface / Package the result (an output expression ci-local cannot evaluate), and ci.yml:test-latest, docs.yml:deploy and pip-audit.yml:report (their job conditions are false on a PR).
  12001 tests passed, 59 skipped, 6 xfailed. Changed-line coverage: no measurable lines, because only tests and docs change.
• The touched test files plus test_docs_code_refs.py, test_growth_ratchet.py, test_audit_test_quality.py and test_import_architecture.py: 381 passed. ruff check is clean on both test files.
• Mutants, each run on a copy of the committed tree. Every one fails the test that pins it, and the unmutated copy passes both tests:
  • Delete if self.error is not None: out["error"] = self.error in CheckResult.as_dict: the new reference-checks test fails (KeyError: 'error').
  • In _summarize, else None → else 0.0: the paired-delta test fails (assert 0.0 is None).
  • In _summarize, drop the baseline condition, so the baseline is paired with itself: fails (assert 0.0 is None).
  • In _summarize, set delta_vs_baseline only for non-baseline arms: fails ('delta_vs_baseline' in {'avg_aggregate': 63.3, 'n': 3, 'errors': 0}).

Revert

A plain revert. Nothing is stored and no schema changes.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[Copilot]

🟢 Approval recommended

No active review issues block the added regression assertions and corrected harness-loading documentation.

No open issues 🎉

What changed in this PR

0 new findings; nothing to fix.

This follow-up to #116 and #280 adds regression assertions for evaluation output and corrects harness-loading documentation. No package code changes.

Changes:

• Verify that failed judge checks include an error while successful match checks omit it.
• Pin the baseline arm’s delta_vs_baseline field as present and null.
• Clarify that Codex and Copilot CLI loading behavior was measured, with references to their built-in profiles.

File	Description
dokimi/​tests/​test_reference_checks.py	Adds assertions for per-check error serialization.
dokimi/​tests/​test_arm_cell_errors.py	Verifies the baseline delta field exists and is None.
dokimi/​docs/​preparing-a-repository.md	Corrects measurement claims for Codex and Copilot CLI.

_{Review effort: Balanced}

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s).

Review details

• Commit reviewed: 989ef44 against merge base 27cc760
• Coverage: 3 files in 1 groups
• Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
• Usage: 49k tokens (29k from cache), 7s
• Tool calls: 9 (7 file_read, 2 file_find)
• small change set: 3 files

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.06%. Comparing base (27cc760) to head (989ef44).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #346      +/-   ##
==========================================
+ Coverage   94.05%   94.06%   +0.01%     
==========================================
  Files         297      297              
  Lines       38018    38018              
  Branches     6424     6424              
==========================================
+ Hits        35758    35762       +4     
+ Misses       1452     1449       -3     
+ Partials      808      807       -1     

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.