Skip to content

repo tasks: authoring replay tasks from merged PRs #294

New issue
New issue
Open
Open
repo tasks: authoring replay tasks from merged PRs#294
Assignees
antonwolfy
Labels
enhancementNew feature or request

Description

@antonwolfy
antonwolfy
opened on Oct 29, 2026

Problem

repo tasks replay builds dokimi:review_replay tasks, which measure what guidance does to an agent reviewing code. Most guidance is read while writing code. Of the eight AGENTS.md files in IntelPython/dpnp#3099, most lines say where a new function's kernel and dispatch helpers go, how tests are laid out, and what a changelog entry looks like. The review replay of that PR exercised little of it.

dokimi can already grade such work. A task.json with family: feature or fix and the repository's own grade entry point does it (contract §3). But those tasks are written by hand, so this measurement is skipped too, just as review counting was before repo tasks replay.

Proposal

Build authoring tasks from merged PRs, with the same input and the same rules for what to leave out as review replay: repo tasks replay --kind authoring, or a sibling subcommand. Per merged PR:

  • Base: the merge base of the PR with mergeCommit^1, as for review replay.
  • Prompt: the PR's title and description, with links to its commits and diff and anything quoting the change removed. --prompt-file replaces the default.
  • Hidden tests: the test files the PR added or changed. They are held out of the workspace and applied only at grade time.
  • Grade: the repository's test command from the manifest, on the hidden tests; the gate is that they pass. oracle is the PR's own diff, so repo tasks check can show that the base fails and the oracle passes.
  • Left out: guidance files and eval_dir, as review replay leaves them out.

Open questions

  • Which PRs qualify: with tests, under a size limit, not only refactors.
  • Leaked tests: a description that names the test functions.
  • Build cost: dpnp needs its SYCL extensions built for every cell. How much can the manifest's caches and a prebuilt base image save?
  • Partial credit: some hidden tests passing. Should it count, as a score rather than a gate?
  • Flaky tests: run the oracle twice in tasks check?

Done when

On a window of merged PRs of a real repository, the command writes tasks that repo tasks check accepts (the untouched base fails, the oracle passes). The tests use a fixture repository with no network.

Activity

  1. napetrov
    added
    enhancementNew feature or request
    on Nov 1, 2026

  2. napetrov commented on Nov 1, 2026

    @napetrov
    napetrov
    on Nov 1, 2026
    Contributor

    A second source for replay tasks (#342): agent failures, not only review threads. A ledger entry is commit + prompt + check, and it becomes a dokimi:failure_replay task. jimy-r's incident ledger is a possible format. #336's repo_ledger.v1 already links rules to cases and could reference these.

  3. antonwolfy commented on Nov 1, 2026

    @antonwolfy
    antonwolfy
    on Nov 1, 2026
    ContributorAuthor

    Proposed answers to the open questions, and the scope after napetrov's comment above.

    • Scope. Keep this issue to merged PRs. A failure-ledger entry (Configuration Effectiveness (agents.md#213): receipt, reach matrix, rule ledger, arms #342) differs in where the prompt and the check come from, not in the task it becomes. So the PR builds one task writer (base, prompt, hidden check, oracle, left-out files) with the PR source on top of it. A failure source then plugs into the same writer in its own issue under Configuration Effectiveness (agents.md#213): receipt, reach matrix, rule ledger, arms #342.
    • Which PRs qualify: repo tasks check decides. A task whose base already passes its hidden tests fails it (§4.3: the untouched workspace must fail the gate), so a refactor drops out there without a heuristic. The build's own filter only needs to be cheap: the PR adds or changes a test file, and its non-test diff is under --max-lines (default 400).
    • Leaked tests. The prompt drops the lines that name a hidden test file or test function (the names come from the PR's test diff), and task.json records how many it dropped. A PR with more than a few such lines (say 5) is skipped as one whose description is its tests.
    • Build cost: validate on a pure-Python repository first. The done-when needs a window of a real repository. dokimi's own merged PRs suit it: nearly every one has tests, and there is no build step. dpnp comes second, with the manifest's caches. A prebuilt image per merge base is a later optimisation, measured on dpnp then.
    • Partial credit: both. The gate is that all hidden tests pass. grade.json also carries the passing share as a metric, so repo compare --metric can read it.
    • Flaky tests. For these tasks, repo tasks check runs the base and the oracle twice each. A task whose two runs differ is reported flaky and not counted as ok.
    • Leakage. Authoring tasks leak the same way review tasks do: guidance written after a PR was public can encode its change. They carry the field repo compare: flag replay tasks a guidance revision may have seen #290 adds (proposed there as public_at; here the PR's createdAt), so repo compare flags both kinds by one rule.

    Order. This is the last of #290–#294. It comes after #290's recording PR, and after #325, which moves repo tasks check to eval check and edits the repo tasks parser this issue adds to.

  4. antonwolfy
    self-assigned this
    on Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions