Skip to content

repo compare: flag replay tasks a guidance revision may have seen #290

Open

Description

@antonwolfy

Problem

A guidance:rev:<sha> arm runs a revision of a repository's guidance against review_replay tasks built from PRs merged earlier. When the guidance was written or edited after a task's review, it can encode that review, and the arm then "catches" the thread because a rule was written from it. Nothing in a batch shows this.

A replay of IntelPython/dpnp#3099 (eight AGENTS.md files; 26 tasks from dpnp PRs merged 2026-09-01..10-28; guidance:none vs guidance:rev:d8fb283) ran into it:

  • The cut looked clean. The guidance's first commit (7781481, 14 Oct 18:45Z) was mined from reviews before it. So a split by review date looked like an out-of-sample test: 9 tasks reviewed after the cut, recall +0.111 against −0.025 for the 17 before.
  • It was not clean. The evaluated revision has later guidance commits that name three of those 9 tasks' PRs:
    • c9b533d "Update test guidance for the fixes in #3103";
    • a9f79ed "Update CMake guidance for the extension helper in #3111";
    • 441690c, with a rule taken from #3098.
  • And 441690c came after every one of them. It was committed after the first review comment of all 9.
  • The batch can't show it. task.json has no review time, and meta.json records the arm's SHA but not its guidance history. The split, and finding the leak, were done by hand.

Proposal

  • Review time: repo tasks replay writes reviewed_at into task.json: the createdAt of the earliest gold thread's first comment. The threads file already holds it.
  • Guidance history: repo run, for a guidance:rev:<rev> arm, records the commits reachable from <rev> that touch the manifest's guidance globs, in meta.json arm_rev.guidance_commits ({sha, date, subject}, newest first, capped).
  • Report it: repo compare flags the tasks such an arm may have seen.
    • By date: some guidance commit is dated after the task's reviewed_at.
    • By name: a guidance commit's message names the task's source_pr (#<n>).
    • The output says how many tasks each flag covers and names the named ones. The rest can be compared on their own (with a repo compare --task filter, or an option for it).
  • Arms with no history: a patch: arm has none, so it reads "unknown". guidance:raw is the base's own guidance and is never flagged.

Open questions

  • The date rule is strict. Here it flags all 26 tasks for d8fb283: the last guidance commit came after every review. Is a per-file rule better, counting only commits to a guidance file the task's diff makes relevant?
  • Which date? Author date or committer date, given that a rebase moves the committer date.
  • Content: should a commit that names no PR, but adds text matching a gold comment, be found too? That is a text-similarity question, likely out of scope.

Done when

The fixture tests cover two cases: a guidance commit that names a task's PR, and one dated after a task's review. repo compare flags both, and leaves the other tasks unflagged. The docs (contract §4.6 and §9) say what the flags mean.

Activity

  1. antonwolfy commented on Nov 1, 2026

    @antonwolfy
    ContributorAuthor

    Proposed answers to the open questions, so a PR can start from them.

    • Keep the date rule strict, scoped by guidance file. A task is out of sample for a revision only if no guidance text it could have shaped entered the revision after its review. So "a guidance commit dated after the review" is the right test, not an over-strict one. What can be narrowed is the scope: a commit counts against a task only when it touches a guidance file that applies to a file in the task's patch. A root file applies everywhere; dir/AGENTS.md applies to dir/ and below. Whether that clears any task for d8fb283 depends on which files the late commits touched. If the root file is among them, all 26 stay flagged, and that is the honest answer: a clean replay of that revision needs PRs reviewed after its last guidance commit.
    • Committer date, not author date. Text a review shaped can enter under an older author date, because git commit --amend and a rebase keep the author date. The committer date is set when the commit is written, so flagging by it can only flag more tasks, never fewer. A rebase moving it later errs in the safe direction. guidance_commits records both (authored, committed); the flag reads committed.
    • Leave content similarity out. It could only narrow the date flag, never widen it: a commit made before the review cannot have used it, and every commit after the review is already flagged by date. The name flag is the narrowing that can be checked mechanically.
    • Name the field public_at, not reviewed_at. It holds the earliest time the task's answer was public: here the createdAt of the earliest gold thread's first comment (repos/replay/threads.py already reads it). Authoring tasks (repo tasks: authoring replay tasks from merged PRs #294) leak the same way and can carry the same field (the PR's createdAt there), so repo compare flags both kinds by one rule.
    • The cap. The date flag needs only the latest-dated commit of each guidance file: a task is flagged when an applicable file's latest commit is after its public_at. So arm_rev keeps that commit per file, uncapped. The capped newest-first guidance_commits list serves the name flag, with truncated: true when the cap cut it.
    • The clean subset. --json gives each task's flags, and repo compare --task (feat(repo): compare records the start context, filters tasks, and flags inconclusive rows #296) already compares a subset. A dedicated option can follow if passing the ids by hand proves clumsy.

    Order: two PRs.

    1. Recording, which can start now: public_at in task.json from repo tasks replay, and guidance_commits in meta.json arm_rev (repos/workspace.py). No compare change.
    2. The flags in repo compare, after feat(repo): time to first useful result, with a rate in repo compare #332 and feat(repo): rule ledger: repo compare --ledger with per-rule keep/review/insufficient-evidence verdicts #336 merge. repos/compare.py is at its 800-line cap, and feat(repo): time to first useful result, with a rate in repo compare #332 moves the per-task values into a new compare_values.py.
  2. self-assigned this
    on Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions