Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added .agents/triage/.gitkeep
Empty file.
1 change: 1 addition & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Closes #
- [ ] `./scripts/verify.sh` passed
- [ ] Tests added/updated where appropriate
- [ ] Independent review completed when required
- [ ] Approved triage published on the feature Issue (link the comment):
- [ ] Architecture/docs/ADR updated when required

## Agent involvement
Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,9 @@ venv/
*.key
service-account*.json
credentials*.json
# Review and triage results are working files; their reports are published
# on the feature Issue instead of being committed.
.agents/reviews/*
!.agents/reviews/.gitkeep
.agents/triage/*
!.agents/triage/.gitkeep
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ If a material conflict remains, stop and report it.
- Verification evidence is recorded in the active plan or PR.
- Required independent review is complete and no Critical or Major findings
remain unresolved.
- Required review findings have an approved triage artifact.
- Required review findings have an approved triage, published on the feature
Issue by `triage-review.sh`.
- Deferred findings have a linked follow-up Issue; accepted findings have an
explicit rationale.

Expand Down
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ All non-trivial changes should originate from a GitHub Issue and use a feature b
Everything committed to the template repository is inherited by every project
created from it. When developing the template:

- Do not commit the plans, reviews, triage artifacts, or handoffs of that
work under `.agents/`. Keep the design in the GitHub Issue and the review
outcome in the Pull Request.
- Do not commit the plans or handoffs of that work under `.agents/`. Keep the
design in the GitHub Issue. Review and triage results are ignored by Git and
published on the Issue, as in any project.
- Record design decisions about the template in the Issue and in the workflow
documentation under `docs/`. `docs/decisions/` is reserved for the ADRs of
projects that use the template.
Expand Down
7 changes: 5 additions & 2 deletions docs/agentic-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,12 @@ See `docs/development.md` for the concrete commands.
- `.agents/plans/`: active implementation state for complex work.
- `.agents/handoffs/`: compressed continuation context.
- `.agents/reviews/`: independent-review results as validated JSON, each with a
generated Markdown report.
generated Markdown report. Working files, ignored by Git.
- `.agents/triage/`: approved finding decisions and deferred-Issue traceability
as validated JSON, each with a generated Markdown report.
as validated JSON, each with a generated Markdown report. Working files,
ignored by Git.
- Feature Issue comments: the published review and triage reports of each
round.
- `.agents/schemas/`: the schemas of those results.
- `.agents/lessons/`: recurring failure lessons awaiting/promoting durable rules.
- Git history: what actually changed.
Expand Down
12 changes: 11 additions & 1 deletion docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,13 @@ write a persistent, uniquely named artifact such as:
```

That artifact maps the source review findings to their decisions and any
created Issue numbers. Declining the proposal creates neither an artifact nor
created Issue numbers. After storing it, the script publishes the review and
triage reports as one comment on the source Issue, so the outcome is visible in
the Issue and, through `Closes #12`, from the pull request. The reports are
rendered from the JSON. When they exceed the size of one GitHub comment, they
are published in numbered parts rather than shortened. A failed publication
keeps the stored triage and prints the command to retry each unpublished
part. Declining the proposal creates neither an artifact nor
Issues. The source review remains unchanged. The artifact is stored only after
every follow-up Issue exists; if creating one fails, nothing is stored and a
new triage reuses the Issues created so far, which it finds by their trace
Expand Down Expand Up @@ -316,6 +322,10 @@ each triage is stored as two files with the same name:
- `.md` is a report generated from the JSON for reading. It is never parsed;
editing it has no effect.

Both are working files for the scripts during a feature and are ignored by
Git, so `git add .` does not commit them. The record that stays is the comment
on the Issue and the follow-up Issues, each of which names its source finding.

The agent's result must match a schema in `.agents/schemas/`
(`review.schema.json`, `triage.schema.json`). The schema is passed to the
provider CLI and the script checks the result again with `jq`, including rules
Expand Down
68 changes: 68 additions & 0 deletions scripts/triage-review.sh
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ deferred_count="$(jq '[.[] | select(.decision == "DEFER")] | length' "$proposal_
if [[ "$deferred_count" -gt 0 ]]; then
echo "Approval will create $deferred_count follow-up GitHub issue(s)."
fi
echo "Approval will publish the review and triage reports as a comment on Issue #$source_issue."

printf "Proceed with this triage? [y/N] "
approval=""
Expand Down Expand Up @@ -361,3 +362,70 @@ echo " $artifact_relative.md (generated report)"
echo
jq '.decisions' "$artifact.json" >"$tmp_work/final.json"
render_proposal "$tmp_work/final.json"

# The reports are published on the Issue instead of being committed, rendered
# from the validated JSON. A comment holds at most 65,536 characters: when both
# reports do not fit in one comment, they are published in numbered parts, so
# the record is never shortened.
round="$(jq -r '.round' "$review_path")"
comment_base="$artifact-comment"
rm -f "$comment_base"*.md
heading="## Independent review and triage — round $round"
{
echo "Reviewer verdict: $(jq -r '.verdict | gsub("_"; " ")' "$review_path")"
echo
echo "<details>"
echo "<summary>Review report</summary>"
echo
review_render_markdown "$review_path" "$review_stem.json" | sed '1{/^<!-- Generated/d;}'
echo
echo "</details>"
echo
sed '1{/^<!-- Generated/d;}' "$artifact.md"
} >"$tmp_work/comment-body.md"

comment_limit=60000
if [[ "$(wc -c <"$tmp_work/comment-body.md")" -le "$comment_limit" ]]; then
{ echo "$heading"; echo; cat "$tmp_work/comment-body.md"; } >"$comment_base.md"
else
# Split on line boundaries; a part never ends inside a line.
awk -v limit="$comment_limit" -v base="$tmp_work/part-" '
BEGIN { part = 1; size = 0 }
{
line_size = length($0) + 1
if (size > 0 && size + line_size > limit) { part++; size = 0 }
print > (base part ".md")
size += line_size
}
' "$tmp_work/comment-body.md"
parts="$(find "$tmp_work" -name 'part-*.md' | wc -l | tr -d ' ')"
for ((part = 1; part <= parts; part++)); do
{
echo "$heading (part $part of $parts)"
echo
cat "$tmp_work/part-$part.md"
} >"$comment_base-$part.md"
done
fi

comment_files=()
if [[ -f "$comment_base.md" ]]; then
comment_files=("$comment_base.md")
else
for ((part = 1; part <= parts; part++)); do
comment_files+=("$comment_base-$part.md")
done
fi

echo
for index in "${!comment_files[@]}"; do
if ! gh issue comment "$source_issue" --body-file "${comment_files[$index]}" </dev/null >/dev/null; then
echo "Error: publishing the reports on Issue #$source_issue failed. The triage is stored." >&2
echo "Retry the unpublished part(s) in this order with:" >&2
for ((retry = index; retry < ${#comment_files[@]}; retry++)); do
echo " gh issue comment $source_issue --body-file ${comment_files[$retry]#"$root"/}" >&2
done
exit 1
fi
done
echo "Published the review and triage reports on Issue #$source_issue (${#comment_files[@]} comment(s))."
62 changes: 61 additions & 1 deletion tests/triage-review-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,26 @@ case "${1:-} ${2:-}" in
Test acceptance criteria.}"
exit 0
;;
"issue comment")
if [[ -n "${MOCK_GH_COMMENT_EXIT:-}" ]]; then
echo "simulated comment failure" >&2
exit "$MOCK_GH_COMMENT_EXIT"
fi
shift 2
issue="$1"
shift
while [[ $# -gt 0 ]]; do
if [[ "$1" == "--body-file" ]]; then
{
echo "COMMENT ON #$issue"
cat "$2"
echo "END COMMENT"
} >>"$MOCK_GH_LOG.comments"
fi
shift
done
exit 0
;;
"issue list")
if [[ -n "${MOCK_EXISTING_ISSUE:-}" ]]; then
echo "$MOCK_EXISTING_ISSUE"
Expand Down Expand Up @@ -166,6 +186,15 @@ grep -Fq -- "--strict-mcp-config --permission-mode dontAsk --tools Read,Glob,Gre
grep -Fq -- "--json-schema" "$repo.log" || fail "triage agent was not given the schema"
grep -Fq "Correctness regression" "$repo.log.stdin" || fail "findings were not supplied to the triage agent"

# One comment with both reports is published on the source Issue.
[[ "$(grep -c '^COMMENT ON #5$' "$repo.gh.comments")" -eq 1 ]] || fail "exactly one comment on Issue #5 was expected"
grep -Fq "C1. Correctness regression" "$repo.gh.comments" || fail "the comment lacks the review report"
grep -Fq "## Fix now" "$repo.gh.comments" || fail "the comment lacks the triage report"
grep -Fq "#123" "$repo.gh.comments" || fail "the comment lacks the follow-up Issue link"
if grep -Fq "Generated from" "$repo.gh.comments"; then
fail "the comment contains the generated-file marker"
fi

# The complete follow-up proposal is shown before the approval question.
proposal="$(sed '/Proceed with this triage/,$d' "$repo.out")"
for shown in "Add focused tests." "The boundary is covered by a test."; do
Expand All @@ -191,6 +220,37 @@ MOCK_EXISTING_ISSUE="https://github.com/example/project/issues/77" run_triage "$
fail "an existing remote follow-up Issue was not reused"
[[ ! -e "$repo.gh" ]] || fail "a duplicate of an existing remote follow-up Issue was created"

# The published review is rendered from the JSON, not from an edited report,
# and a record too long for one comment is split rather than shortened.
repo="$(setup_repo long-report)"
printf 'Edited report that must not be published.\n' >"$repo/.agents/reviews/feature-5-test-review-01.md"
long_evidence="$(printf 'evidence line %s\\n' $(seq 1 4000))"
jq --arg evidence "$long_evidence" '.findings[1].evidence = $evidence' \
"$repo/$review" >"$repo/$review.tmp"
mv "$repo/$review.tmp" "$repo/$review"
run_triage "$repo" y "$review" || {
cat "$repo.out" >&2
fail "triage with a long review failed"
}
if grep -Fq "Edited report that must not be published." "$repo.gh.comments"; then
fail "an edited Markdown report was published instead of the JSON"
fi
parts="$(grep -c '^COMMENT ON #5$' "$repo.gh.comments")"
[[ "$parts" -ge 2 ]] || fail "a record too long for one comment was not split"
grep -Fq "evidence line 4000" "$repo.gh.comments" || fail "the long review was shortened"
grep -Fq "## Fix now" "$repo.gh.comments" || fail "the split record lacks the triage report"

# A failed publication keeps the stored triage and tells how to retry.
repo="$(setup_repo comment-fails)"
if MOCK_GH_COMMENT_EXIT=1 run_triage "$repo" y "$review"; then
fail "a failed publication returned success"
fi
[[ -f "$repo/.agents/triage/feature-5-test-review-01-triage.json" ]] ||
fail "a failed publication discarded the stored triage"
grep -Fq "gh issue comment 5 --body-file .agents/triage/feature-5-test-review-01-triage-comment.md" "$repo.out" ||
fail "a failed publication did not print the retry command"
[[ -f "$repo/.agents/triage/feature-5-test-review-01-triage-comment.md" ]] || fail "the comment to retry was not kept"

# When Issue creation fails, no triage artifact is stored.
repo="$(setup_repo create-fails)"
if MOCK_GH_CREATE_EXIT=1 run_triage "$repo" y "$review"; then
Expand All @@ -212,7 +272,7 @@ grep -Fqx "TITLE: [#5][R07][MIN1] Add boundary-condition coverage" "$repo.gh" ||
# Declining has no side effects.
repo="$(setup_repo decline)"
run_triage "$repo" n "$review" --agent codex --model model-c || fail "declined triage returned an error"
if compgen -G "$repo/.agents/triage/*" >/dev/null || [[ -e "$repo.gh" ]]; then
if compgen -G "$repo/.agents/triage/*" >/dev/null || [[ -e "$repo.gh" || -e "$repo.gh.comments" ]]; then
fail "declined triage had side effects"
fi
grep -Fq -- "--sandbox read-only" "$repo.log" || fail "Codex triage agent was not sandboxed read-only"
Expand Down
Loading