(standardize-repo): run task audit:ruleset in the fleet audit and report drift as advisory #858
Description
Problem
harmon-init now ships task audit:ruleset (evanharmon1/harmon-init#1185, issue #1181): a read-only comparison of the live GitHub "Protect Main" ruleset against the checked-in ruleset file, exit 0 identical / 1 drift / 2 unavailable. Generated repos inherit it. The fleet-wide standardize-repo audit does not run it, so ruleset drift in a consumer repo is still invisible until someone looks.
The invariant: the standardize-repo audit reports live-vs-file ruleset drift for every repo that ships the task, as an advisory finding.
Acceptance criteria
- [CI] The
standardize-repoaudit runstask audit:rulesetwhen the target defines it, reports drift as an advisory finding with the diff, treats exit 2 (noghpermission / absent ruleset) as "not checked" rather than drift, and skips repos without the task - [CI] The audit's fixture tests cover identical, drift, unavailable, and task-absent cases
- [HUMAN] The maintainer ticks (ci): task audit:ruleset — detect drift between the live ruleset and the checked-in file, both layers harmon-init#1181's remaining
[HUMAN]criterion once this lands
Provenance
Follow-on from evanharmon1/harmon-init#1181 (2026-09-06).
Metadata
Metadata
Assignees
Labels
The standardize-repo skill and its standards catalogClaimed by GPTComplexity: moderate; several components and some design choicesKeeping repos current with conventions: standardize-repo, drift audits, adoptionImpact: clear benefit to a meaningful share of users or goalsBusiness rules, handlers, calculationPriority (AI): worth doing, not blockingRisk: a mistake is contained, caught quickly, and cheap to undoIssue is in an orchestrator slate plan (membership only, not a state)General work: maintenance, chores, cleanupModel tier: reliable general-purpose coding model first
Activity
evanharmon1 commentedon Nov 4, 2026
Claimed by
codex-cli(gpt-6.1-sol) under orchestrator sessionfleet-m6-agy(slatem6-fleet-foundations, lanelane-858, branchfeat/858-audit-ruleset-advisory).evanharmon1 commentedon Nov 4, 2026
Slate membership:
evanharmon1 commentedon Nov 4, 2026
Dev flow v2 evidence: challenge round 1 for
run-858-audit-ruleset-advisoryat911558e(findercodex-adversarial).{ "run_id": "run-858-audit-ruleset-advisory", "stage": "challenge", "round": 1, "reviewed_head": "911558e621872e9ad88a1442a04ad4a30fa4b0de", "passes": [ { "schema": 2, "role": "challenger", "status": "completed", "head": "911558e621872e9ad88a1442a04ad4a30fa4b0de", "produced_at": "2026-11-04T21:35:12Z", "producer": { "harness": "codex-cli", "model": "gpt-6-astra", "tier": "apex" }, "run": { "run_id": "run-858-audit-ruleset-advisory", "initiated_by": "human" }, "payload": { "stage": "challenge", "round": 1, "reviewed_head": "911558e621872e9ad88a1442a04ad4a30fa4b0de", "finder": "codex-adversarial", "slot": "codex-adversarial", "findings": [ { "id": "challenge-r1-codex-adversarial-1", "path": "ai/skills/repo/standardize-repo/assets/verify-applied.sh", "line": 186, "class": "correctness", "provenance": "original", "fingerprint": "new", "priority": "P2", "recommended_disposition": "fix", "evidence": "Codex [P2] 'Distinguish comparison drift from operational exit 1': an operational failure returning 1 is reported as confirmed drift. Reproducible with the existing scripts/audit-ruleset.sh: set TMPDIR to a nonexistent directory and its mktemp fails before any comparison; Task passes exit 1 through and this branch announces that the live ruleset differs. Codex asks to require evidence that the comparison completed (for example the helper's drift marker), otherwise report not checked, and to add an operational exit-1 fixture, since the current unexpected-failure fixture only tests exit 7. Own check: `git show 911558e6:ai/skills/repo/standardize-repo/assets/verify-applied.sh` line 186 is the case arm `1) echo \"WARN: ruleset audit drift (advisory) — live ruleset differs from the checked-in file\" >&2 ;;`, keyed on the exit status alone. `git show 911558e6:scripts/audit-ruleset.sh` has `set -euo pipefail` (line 3) and an unguarded `tmp_dir=\"$(mktemp -d -t harmon-init-ruleset-XXXXXX)\"` (line 47), so any unguarded command failure exits 1 like the deliberate drift exit (line 84), which is preceded by the `RULESET AUDIT DRIFT:` marker. Reproduced with go-task 3.53.1 against that script: `TMPDIR=<missing> RULESET_AUDIT_REPO=owner/repo task --color=false --exit-code audit:ruleset` printed the mktemp error and exited 1 with no RULESET AUDIT DRIFT line. The diagnostic is still printed after the WARN, so the failure is a misleading classification, not lost output." } ], "counts": { "P0": 0, "P1": 0, "P2": 1, "P3": 0 }, "attack_scenarios": [ { "id": "as-1", "description": "Ran the production 3r section against scripts/audit-ruleset.sh with TMPDIR pointing at a nonexistent directory, so the helper fails before any comparison and exits 1. A separate probe, a task running `test -f missing-file`, also passed exit 1 through Task.", "outcome": "surfaced-finding", "finding_id": "challenge-r1-codex-adversarial-1" }, { "id": "as-2", "description": "Probed a failed task precondition under `task --exit-code`: exit 201, which falls in the verifier's not-checked arm. Codex reported no finding from this probe.", "outcome": "held", "finding_id": null }, { "id": "as-3", "description": "Probed a task whose command is a missing binary: exit 127, which falls in the not-checked arm. Codex reported no finding from this probe.", "outcome": "held", "finding_id": null }, { "id": "as-4", "description": "Probed a signal-shaped exit with `sh -c \"exit 143\"`: exit 143 passes through and falls in the not-checked arm. Codex reported no finding from this probe.", "outcome": "held", "finding_id": null }, { "id": "as-5", "description": "Probed a `silent: true` task exiting 2: the code passes through and the task stays listed. Codex reported no finding from this probe.", "outcome": "held", "finding_id": null }, { "id": "as-6", "description": "Probed an `internal: true` audit:ruleset task for the capability gate: --list-all printed 'No tasks available' and a direct run exits 202. Codex reported no finding from this probe.", "outcome": "held", "finding_id": null }, { "id": "as-7", "description": "Probed a task with a passing `status:` check: --exit-code returned 0 with 'Task \"audit:ruleset\" is up to date' and the command never ran. Codex reported no finding from this probe.", "outcome": "held", "finding_id": null }, { "id": "as-8", "description": "Ran the changed fixture suite from a temporary copy so nothing landed in the review checkout: all 48 tests passed under go-task. Codex reported no finding from this run.", "outcome": "held", "finding_id": null } ] } } ], "adjudication": { "schema": 2, "run_id": "run-858-audit-ruleset-advisory", "stage": "challenge", "round": 1, "reviewed_head": "911558e621872e9ad88a1442a04ad4a30fa4b0de", "adjudications": [ { "finding_id": "challenge-r1-codex-adversarial-1", "reviewer_priority": "P2", "adjudicated_priority": "P2", "disposition": "fix", "reason": "Confirmed and reproduced with go-task 3.53.1: a missing TMPDIR makes scripts/audit-ruleset.sh exit 1 with no drift marker, and verify-applied.sh reports it as drift because the case arm reads only the status. The helper prints RULESET AUDIT CLEAN, DRIFT or UNAVAILABLE, so classify by status and matching marker together; anything else is not checked, naming the exit. This also closes a status:-up-to-date false OK (exit 0 with no CLEAN marker) and the same weakness at exit 2.", "evidence": "verify-applied.sh@911558e6:186; harmon-init scripts/audit-ruleset.sh:13,81,84", "override": null } ] }, "exit": { "stage": "challenge", "rounds_counted": 1, "next_round": 2, "substitutions": [], "split_candidate": { "detected": false, "round": 1, "mechanism": null, "concentration": 0, "provenance_share": 0, "introduced_by_rounds": [], "finding_ids": [], "consecutive_rounds": [ 1 ], "reason": "no_gating_findings" }, "outcome": "continue", "reason": "below_threshold", "action": "dispatch", "corrections": [], "diagnostics": [], "verified_findings": [ { "id": "challenge-r1-codex-adversarial-1", "provenance_status": "verified", "verified_provenance": "original", "fingerprint_status": "verified", "verified_fingerprint": "new" } ] } }Round
challenge round 1 — reviewed 911558e
Exit: continue — below_threshold (rounds counted: 1; next round: 2)
evanharmon1 commentedon Nov 4, 2026
Dev flow v2 evidence: challenge round 2 for
run-858-audit-ruleset-advisoryatfc9d92c(findercodex-adversarial).{ "run_id": "run-858-audit-ruleset-advisory", "stage": "challenge", "round": 2, "reviewed_head": "fc9d92c0bf5e6f539834526ddc35300cb9b6e47c", "passes": [ { "schema": 2, "role": "challenger", "status": "completed", "head": "fc9d92c0bf5e6f539834526ddc35300cb9b6e47c", "produced_at": "2026-11-04T22:38:01Z", "producer": { "harness": "codex-cli", "model": "gpt-6-astra", "tier": "apex" }, "run": { "run_id": "run-858-audit-ruleset-advisory", "initiated_by": "human" }, "payload": { "stage": "challenge", "round": 2, "reviewed_head": "fc9d92c0bf5e6f539834526ddc35300cb9b6e47c", "finder": "codex-adversarial", "slot": "codex-adversarial", "findings": [ { "id": "challenge-r2-codex-adversarial-1", "path": "ai/skills/repo/standardize-repo/assets/verify-applied.sh", "line": 187, "class": "correctness", "provenance": "original", "fingerprint": "new", "priority": "P2", "recommended_disposition": "fix", "evidence": "Codex [P2] 'Include task aliases in capability discovery' (cited lines 186-187): when audit:ruleset is an alias, `task --list-all` shows it beside the canonical name, for example `* compare-ruleset: (aliases: audit:ruleset)`. The expression misses that and reports the audit as absent although `task audit:ruleset` runs. Reproduced by Codex with real go-task: the direct invocation reported drift while this section skipped. Codex asks to discover aliases as well as canonical names and to add an alias-only fixture. Own check: `git show fc9d92c0:ai/skills/repo/standardize-repo/assets/verify-applied.sh` line 187 is `if grep -qE '^[* ]*audit:ruleset:([[:space:]]|$)' <<<\"$ruleset_tasks\"; then`, which anchors the name at the start of the line. Running the production section under go-task 3.53.1 against a Taskfile with task `ruleset-audit` and `aliases: [audit:ruleset]` listed `* ruleset-audit: audit (aliases: audit:ruleset)`; `task --color=false --exit-code audit:ruleset` printed RULESET AUDIT DRIFT and exited 1, yet the section printed `SKIP: ruleset audit — target has no audit:ruleset task`. test-verify-applied.sh at fc9d92c0 contains no alias fixture. Provenance is original: the same grep is at line 181 of 911558e6, and round 1's fix diff carries it only as context. The shipped harmon-init template defines audit:ruleset as a canonical task (template/Taskfile.yml.jinja line 865), so the gap needs a target that defines it only as an alias." }, { "id": "challenge-r2-codex-adversarial-2", "path": "ai/skills/repo/standardize-repo/assets/verify-applied.sh", "line": 189, "class": "correctness", "provenance": "round:1", "fingerprint": "new", "priority": "P2", "recommended_disposition": "fix", "evidence": "Codex [P2] 'Pin Task's output mode before matching result markers' (cited lines 189-192): for a target using `output: prefixed`, Task renders the marker as `[audit:ruleset] RULESET AUDIT DRIFT: ...`, so the anchored checks classify a genuine drift result as not checked despite exit status 1. Codex reproduced this with real go-task and says it also affects clean and unavailable results. Codex asks to override the output mode on this invocation so classification does not depend on the target's presentation settings, rather than weakening the anchors. Own check: `git show fc9d92c0:ai/skills/repo/standardize-repo/assets/verify-applied.sh` line 189 runs `task --color=false --exit-code audit:ruleset 2>&1` with no output override, and lines 190-194 require `^RULESET AUDIT ...:`. Running the production section under go-task 3.53.1 against an `output: prefixed` target exiting 1 printed `INFO: ruleset audit not checked — task exited 1 without its result marker`, followed by the preserved `[audit:ruleset] RULESET AUDIT DRIFT: x` line. On the same target, `task --color=false --output=interleaved --exit-code audit:ruleset` printed the marker unprefixed and still exited 1. The harmon-init template sets `output: group` (template/Taskfile.yml.jinja line 5), and the production section classified a group-mode drift correctly, so generated repos keep working and only a target with a prefixed output setting is affected. The degrade is toward not checked and the diagnostic output is retained. Provenance is round:1: the marker greps were added in fc9d92c0 and are absent from 911558e6, whose status-only case arm would have reported this result as drift. Fingerprint is new: challenge-r1-codex-adversarial-1 was an operational failure promoted to drift, and this is the opposite failure, a genuine result demoted to not checked." } ], "counts": { "P0": 0, "P1": 0, "P2": 2, "P3": 0 }, "attack_scenarios": [ { "id": "as-1", "description": "Built an alias-only Taskfile (`compare-ruleset` with `aliases: [audit:ruleset]`) and ran `task --list-all`, the direct task, and the production 3r section: the direct run reported drift and the section skipped.", "outcome": "surfaced-finding", "finding_id": "challenge-r2-codex-adversarial-1" }, { "id": "as-2", "description": "Built an `output: prefixed` Taskfile whose audit:ruleset task prints the drift marker and exits 1, and ran the production 3r section: it printed not checked, naming exit 1.", "outcome": "surfaced-finding", "finding_id": "challenge-r2-codex-adversarial-2" }, { "id": "as-3", "description": "Ran `task test:verify-applied` in the review checkout: all 51 tests passed. Codex reported no finding from this run.", "outcome": "held", "finding_id": null }, { "id": "as-4", "description": "Read the harmon-init template audit-ruleset.sh and grepped the template Taskfile for output, silent and prefix settings and ruleset tasks: it showed `output: group` and a canonical audit:ruleset task. Codex reported no finding for the template's own shape.", "outcome": "held", "finding_id": null } ] } } ], "adjudication": { "schema": 2, "run_id": "run-858-audit-ruleset-advisory", "stage": "challenge", "round": 2, "reviewed_head": "fc9d92c0bf5e6f539834526ddc35300cb9b6e47c", "adjudications": [ { "finding_id": "challenge-r2-codex-adversarial-1", "reviewer_priority": "P2", "adjudicated_priority": "P2", "disposition": "restructure", "reason": "Confirmed and reproduced: an alias-only audit:ruleset misses the --list-all grep, so the section prints SKIP while the task runs and reports drift. Round-2 checkpoint: this capability gate predates round 1 and is original code. Stop parsing list text: ask Task directly with task --summary audit:ruleset, which resolves aliases, and detect a Taskfile by every name go-task accepts.", "evidence": "verify-applied.sh@fc9d92c0:187", "override": null }, { "finding_id": "challenge-r2-codex-adversarial-2", "reviewer_priority": "P2", "adjudicated_priority": "P2", "disposition": "restructure", "reason": "Confirmed and reproduced: with output: prefixed the marker no longer starts the line, so a genuine result reads as not checked. Round-2 checkpoint: the subject exists because round 1 added marker matching. It stays in scope because status alone was the round-1 defect; the remedy is to remove the variable: run with --output=interleaved so the helper line is unprefixed whatever the Taskfile sets. That also fixes output: group with error_only. It degrades honestly today (not checked, output kept) and affects only targets that differ from the template, so P2.", "evidence": "verify-applied.sh@fc9d92c0:189", "override": null } ] }, "exit": { "stage": "challenge", "rounds_counted": 2, "next_round": null, "substitutions": [], "split_candidate": { "detected": false, "round": 2, "mechanism": null, "concentration": 0, "provenance_share": 0, "introduced_by_rounds": [], "finding_ids": [], "consecutive_rounds": [ 2 ], "reason": "no_gating_findings" }, "outcome": "converged", "reason": "predicates_satisfied", "action": "advance", "corrections": [], "diagnostics": [], "verified_findings": [ { "id": "challenge-r1-codex-adversarial-1", "provenance_status": "verified", "verified_provenance": "original", "fingerprint_status": "verified", "verified_fingerprint": "new" }, { "id": "challenge-r2-codex-adversarial-1", "provenance_status": "unverified", "verified_provenance": "original", "fingerprint_status": "verified", "verified_fingerprint": "new" }, { "id": "challenge-r2-codex-adversarial-2", "provenance_status": "unverified", "verified_provenance": "round:1", "fingerprint_status": "verified", "verified_fingerprint": "new" } ] } }Round
challenge round 2 — reviewed fc9d92c
Exit: converged — predicates_satisfied (rounds counted: 2)
evanharmon1 commentedon Nov 4, 2026
Dev flow v2 evidence: review round 1 for
run-858-audit-ruleset-advisoryatad6af0b(findercodex-verification).{ "run_id": "run-858-audit-ruleset-advisory", "stage": "review", "round": 1, "reviewed_head": "ad6af0babcd006dd577eb1e865de7ce14e08bb08", "passes": [ { "schema": 2, "role": "reviewer", "status": "completed", "head": "ad6af0babcd006dd577eb1e865de7ce14e08bb08", "produced_at": "2026-11-04T23:21:31Z", "producer": { "harness": "codex-cli", "model": "gpt-6-astra", "tier": "frontier" }, "run": { "run_id": "run-858-audit-ruleset-advisory", "initiated_by": "human" }, "payload": { "stage": "review", "round": 1, "reviewed_head": "ad6af0babcd006dd577eb1e865de7ce14e08bb08", "finder": "codex-verification", "slot": "codex-verification", "findings": [], "counts": { "P0": 0, "P1": 0, "P2": 0, "P3": 0 } } } ], "adjudication": { "schema": 2, "run_id": "run-858-audit-ruleset-advisory", "stage": "review", "round": 1, "reviewed_head": "ad6af0babcd006dd577eb1e865de7ce14e08bb08", "adjudications": [] }, "exit": { "stage": "review", "rounds_counted": 1, "next_round": null, "substitutions": [], "split_candidate": { "detected": false, "round": 1, "mechanism": null, "concentration": 0, "provenance_share": 0, "introduced_by_rounds": [], "finding_ids": [], "consecutive_rounds": [ 1 ], "reason": "no_gating_findings" }, "outcome": "converged", "reason": "empty_round", "action": "advance", "corrections": [], "diagnostics": [], "verified_findings": [] } }Round
review round 1 — reviewed ad6af0b
Exit: converged — empty_round (rounds counted: 1)
evanharmon1 commentedon Nov 4, 2026
Claim released — handoff: draft PR #1854 passed its lane stages and is parked in the merge queue. The claim returns if a code-change round is needed. (Supersedes the claim record above.)
evanharmon1 commentedon Nov 5, 2026
Dev flow v2 evidence: integration round 1 for
run-858-audit-ruleset-advisoryat3de55a2(finderintegrator).{ "run_id": "run-858-audit-ruleset-advisory", "stage": "integration", "round": 1, "reviewed_head": "3de55a205c036efb63b0283eeef6f08c532d3faa", "passes": [ { "schema": 2, "role": "integrator", "status": "completed", "head": "3de55a205c036efb63b0283eeef6f08c532d3faa", "produced_at": "2026-11-05T02:38:11Z", "producer": { "harness": "claude-code", "model": "claude-haiku-5-5", "tier": "economy" }, "run": { "run_id": "run-858-audit-ruleset-advisory", "initiated_by": "human" }, "payload": { "checks": [ { "name": "verify", "bucket": "pass", "run_id": "127006550216", "required": true }, { "name": "devcontainer-permissions", "bucket": "pass", "run_id": "127000219231", "required": false }, { "name": "security", "bucket": "pass", "run_id": "126999532611", "required": true }, { "name": "repo-guards", "bucket": "pass", "run_id": "126999532585", "required": false }, { "name": "test-helpers", "bucket": "pass", "run_id": "126999532598", "required": false }, { "name": "test-skills", "bucket": "pass", "run_id": "126999532572", "required": false }, { "name": "check", "bucket": "pass", "run_id": "126999532449", "required": false }, { "name": "changes", "bucket": "pass", "run_id": "126999532630", "required": false }, { "name": "devcontainer-verify", "bucket": "pass", "run_id": "126999660594", "required": true }, { "name": "guard", "bucket": "pass", "run_id": "126999533089", "required": false }, { "name": "closing-keywords", "bucket": "pass", "run_id": "126999531676", "required": true }, { "name": "guard", "bucket": "pass", "run_id": "126999533372", "required": false }, { "name": "devcontainer-assert-agent", "bucket": "skipping", "run_id": "126999661730", "required": false }, { "name": "devcontainer-assert-bot", "bucket": "skipping", "run_id": "126999661955", "required": false }, { "name": "build", "bucket": "skipping", "run_id": "126999661987", "required": false }, { "name": "build-merge-group", "bucket": "skipping", "run_id": "126999661764", "required": false }, { "name": "test-contracts", "bucket": "pass", "run_id": "126999532481", "required": false }, { "name": "devcontainer-changes", "bucket": "pass", "run_id": "126999533310", "required": false }, { "name": "CodeRabbit", "bucket": "skipping", "run_id": "126999616751", "required": false } ], "codex_cycle": { "head": "3de55a205c036efb63b0283eeef6f08c532d3faa", "cycle": 1, "charged": 1, "exempt": 0, "attempt": 1, "trigger_comment_id": "6648038729", "exit_code": 0, "accepted": { "surface": "comment", "id": "6648060981", "reviewed_commit": "3de55a205c036efb63b0283eeef6f08c532d3faa" } }, "finder_cycles": [ { "finder": "codex-cloud", "head": "3de55a205c036efb63b0283eeef6f08c532d3faa", "cycle": 1, "charged": 1, "exempt": 0, "attempt": 1, "trigger_comment_id": "6648038729", "exit_code": 0, "accepted": { "surface": "comment", "id": "6648060981", "reviewed_commit": "3de55a205c036efb63b0283eeef6f08c532d3faa" } } ], "integration_round": 1, "findings": [ { "id": "integration-r1-human-1", "body": "CodeRabbit inline comment (Minor, Functional Correctness) on ai/skills/repo/standardize-repo/assets/verify-applied.sh line 200, updated 2026-11-04T23:55:02Z: the audit:ruleset summary lookup in the ruleset drift check does not capture its exit status. The comment asks that SKIP be emitted for a missing task only when the status is 200, and that other summary failures, including task-name conflicts, be reported as not checked, keeping the existing audit path for successful summaries.", "source_id": "4474867454" } ], "unanswered_thread_roots": [ "4474867454" ], "settled_at": "2026-11-05T02:38:11Z", "verdict": "findings" } } ], "adjudication": { "schema": 2, "run_id": "run-858-audit-ruleset-advisory", "stage": "integration", "round": 1, "reviewed_head": "3de55a205c036efb63b0283eeef6f08c532d3faa", "adjudications": [ { "finding_id": "integration-r1-human-1", "reviewer_priority": null, "adjudicated_priority": "P2", "disposition": "fix", "reason": "Confirmed on go-task 3.53.1: a missing task exits 200 from --summary and a duplicate alias exits 203, so a configuration error reads as no audit:ruleset task. Capture the status, SKIP only on 200, report anything else as not checked. Advisory only, so P2.", "evidence": "verify-applied.sh@3de55a20:197; orchestrator probe: dup_summary=203 missing_summary=200; inline 4474867454", "override": null } ] }, "exit": { "stage": "review", "rounds_counted": 1, "next_round": null, "substitutions": [], "split_candidate": { "detected": false, "round": 1, "mechanism": null, "concentration": 0, "provenance_share": 0, "introduced_by_rounds": [], "finding_ids": [], "consecutive_rounds": [ 1 ], "reason": "no_gating_findings" }, "outcome": "converged", "reason": "empty_round", "action": "advance", "corrections": [], "diagnostics": [], "verified_findings": [] } }Round
integration round 1 — reviewed 3de55a2
Exit: converged — empty_round (rounds counted: 1)