Skip to content

Broaden Step 2 to find the PR template under every accepted spelling - #108

Open
dmccoystephenson wants to merge 3 commits into
mainfrom
fix/step-2-pr-template-spellings
Open

Broaden Step 2 to find the PR template under every accepted spelling#108
dmccoystephenson wants to merge 3 commits into
mainfrom
fix/step-2-pr-template-spellings

Conversation

@dmccoystephenson

Copy link
Copy Markdown
Member

Summary

  • Step 2's "Issue and PR patterns" bullet named only .github/pull_request_template.md. Because Step 2's instruction is to "read the following files if they exist (skip silently if absent)", a target repo using any other accepted spelling or location had its PR template missed with no trace left behind. This repository is itself such a case — its template lives at .github/PULL_REQUEST_TEMPLATE.md.
  • The bullet has been broadened to name every spelling and location GitHub accepts: PULL_REQUEST_TEMPLATE.md or pull_request_template.md, in .github/, the repo root, or docs/, plus a .github/PULL_REQUEST_TEMPLATE/ directory when multiple templates are offered.
  • README.md's comparison table repeated the same single lowercase path and has been made spelling-agnostic so the described behavior continues to match what Step 2 actually does.

Closes #105

Research grounding

RESEARCH.md §3 — "SWE-bench Verified is inflated; localization is the real bottleneck" applies. The finding's implication is that the dominant agent failure mode is locating the right surface rather than patching it, and its remedy is to make localization explicit rather than assumed. Step 2 is where a generated skill's localization of its target repo is established; a file list that silently under-matches is a localization failure of exactly the kind the finding warns about, and it is invisible downstream because the miss is defined to be silent. An Implementations entry has been recorded under that finding in this PR.

Doc sync check

  • README.md's "What it does" Step list still matches create-dev-loop.md's Steps 1:1 — no Step was added or removed
  • Every {{placeholder}} added or changed has a corresponding Step 4 substitution-table row — no placeholder was added or changed
  • RESEARCH.md updated — an Implementations entry was added under §3

Test plan

  • python3 scripts/check_docs.py — passes ("Doc consistency check passed.")
  • python3 -m pytest tests/ — 13 passed
  • Step 2's claim verified against this repository's own tree: .github/PULL_REQUEST_TEMPLATE.md exists and the previous bullet's lowercase path does not, confirming the miss the issue describes was real here
  • UNVERIFIED — behavioral validation not run. CLAUDE.md's "Testing changes" anchor requires running /create-dev-loop against a real target repo and inspecting the generated skill. This session is a headless Gardener dispatch whose sandbox is scoped to this repository's own checkout, so no second repository was reachable to generate against, and Step 6 of that procedure creates a GitHub repository — an action outside what this dispatch is authorized to take unattended. Per the generated skill's Phase 4 UNVERIFIED gate, template surface (create-dev-loop.md) is modified here, so the anchor is required and this PR is not eligible for autonomous merge. Human validation against a fixture repo is requested before merge.

Deferred issues (skip reasons)

Recorded here for auditability rather than as comments on the issues themselves:

Merge readiness

Not merged by this session. The modified path create-dev-loop.md is on the generated skill's do-not-auto-merge list — every change to it propagates to all future generated skills — and the behavioral anchor above could not be run. Both conditions independently call for human review.

This PR description was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).

dmccoystephenson and others added 3 commits August 7, 2026 01:01
GitHub accepts PULL_REQUEST_TEMPLATE.md and pull_request_template.md, in
.github/, the repo root, or docs/, plus a .github/PULL_REQUEST_TEMPLATE/
directory. Step 2 named only the lowercase .github/ path, so on a
case-sensitive filesystem a repo using any other spelling had its template
skipped silently under the "skip silently if absent" instruction.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other bullet in Step 2's file lists opens with a backticked path
followed by an em-dash description. Restore that shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dmccoystephenson

Copy link
Copy Markdown
Member Author

Self-review rubric

Scored adversarially against the diff at 469649b and against command output, not judgment. One item was found and fixed during scoring; two are scored FAIL and are the reason this PR is being handed off rather than merged.

Universal

  • Scope: PASS — three files, each load-bearing. create-dev-loop.md carries the fix; README.md:15 repeated the identical stale lowercase path and would have become a false claim the moment Step 2 changed; RESEARCH.md is required by CLAUDE.md ("If your PR implements a finding from RESEARCH.md, also add … an Implementations entry"). No formatting churn, no renames, no unrelated edits.
  • Tests-new: FAIL — the widened bullet has no validation step exercising it. The only step that would is a /create-dev-loop run against a repo whose template uses a non-lowercase spelling, and that anchor could not be run here (see Manual validation). What was verified is the premise rather than the fix: .github/PULL_REQUEST_TEMPLATE.md exists in this repository and .github/pull_request_template.md does not, so the pre-fix bullet demonstrably matched nothing here.
  • Tests-fix: FAIL (measured, not reasoned) — the stash-and-revalidate step was performed rather than asserted. With the create-dev-loop.md change reverted to origin/main and scripts/check_docs.py re-run, the output was Doc consistency check passed. — the defect does not surface. This is a true false-negative, and an expected one: check_docs.py verifies placeholder/substitution-table parity, README-to-Step parity, and relative-link resolution. Step 2's file list is none of those, so no automated surface in this repo can distinguish this fix from a no-op. The green CI run below is therefore evidence of no regression, and must not be read as evidence the fix works.
  • Sibling structure: PASS, after a fix during this review. The first draft of the bullet opened with prose ("The PR template — …") while every other bullet in Step 2's six file lists opens with a backticked path followed by an em-dash description. That deviation was corrected in 469649b before scoring; the bullet now leads with .github/PULL_REQUEST_TEMPLATE.md. Scored PASS on the current diff, but noted here because an all-PASS first draft would have been the suspicious outcome.
  • Sibling renames: PASS — no identifier in a parallel pair or series was renamed. The Step-numbered headings, the "Steps at a glance" links, and the Step 4 substitution table are all untouched.
  • Docs: PASS — every row of the Phase 7 sources-of-truth table checked against the implementation. create-dev-loop.md internally consistent (no placeholder added, Step count unchanged at seven). README.md "What it does" still 1:1 with the Steps, mechanically confirmed by check_docs.py. RESEARCH.md §3 carries the new Implementations entry. SECURITY.md:23 names "the PR template" generically and is now more accurate, not less. CLAUDE.md's doc-source row already used the uppercase spelling and needed no change. A repository-wide grep confirms no fourth reference to the old path survives.
  • Issue resolution: PASSStep 2 names only the lowercase .github/pull_request_template.md, missing the uppercase spelling #105 names exactly one surface, the Step 2 bullet, and that surface is changed. Nothing in the issue is left partially addressed; its own "Suggested fix" is implemented, plus the README.md mirror the issue did not mention.

Repo-specific

  • Placeholder-consistency: PASS — no {{placeholder}} was added, removed, or renamed. check_docs.py enforces the parity mechanically and reports clean.
  • Escaping: PASS — no fenced code block was added inside the template. The diff adds prose to an existing bullet and one RESEARCH.md list item; no triple-backtick appears anywhere in it, so the leading-backslash escape convention is not engaged.
  • Research-grounding: PASSRESEARCH.md §3 is cited in the PR description with the reasoning for why it applies rather than a bare pointer, and the reciprocal Implementations entry (PR number, ship date, observed-effect placeholder) was added under that finding in 282849c.
  • Manual validation: FAIL — UNVERIFIED. CLAUDE.md's "Testing changes" anchor requires generating a skill against a real repository and inspecting the output. This dispatch's sandbox is scoped to this repository's own checkout, so no second repository was reachable to generate against, and Step 6 of that procedure creates a GitHub repository — outside what an unattended run should do. Per the Phase 4 UNVERIFIED gate, the scope test decides the consequence: template surface is modified here, so the anchor is required and autonomous merge is not available.

What did run

  • python3 scripts/check_docs.pyDoc consistency check passed.
  • python3 -m pytest tests/ — 13 passed
  • CI doc-consistency on the PR head — pass (8s), run 31156193053

Recommendation

Hold for human review. Two independent conditions call for it: create-dev-loop.md is on the do-not-auto-merge list because every change to it propagates into all future generated skills, and the behavioral anchor could not run. The targeted validation worth running before merge is a single /create-dev-loop invocation against a repository whose PR template uses the uppercase .github/PULL_REQUEST_TEMPLATE.md spelling — this repository itself qualifies — confirming the generated skill's Phase 4 "PR body must include" guidance now reflects the template's actual contents. A run against a lowercase-spelled repository would pass on the pre-fix template too and would prove nothing.

This review was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Step 2 names only the lowercase .github/pull_request_template.md, missing the uppercase spelling

1 participant