Skip to content

feat(skill): ✨ add the adopt-release-tools agent skill - #29

Merged
robertodr merged 14 commits into
mainfrom
feat/adopt-skill
Nov 5, 2026
Merged

robertodr merged 14 commits into
mainfrom
feat/adopt-skill

Conversation

@robertodr

@robertodr robertodr commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

Stacked on #28 (docs/adoption-baseline); review only its own commits. Rebase onto main once the PRs below it are merged.

Summary

An agent skill for repository admins adopting release-tools: skills/adopt-release-tools/. It inspects the repository and GitHub read-only, plans with the admin, writes the adoption PR (stubs, publish workflow, just changelog, PR template, CHANGELOG.md with the earlier releases, Dependabot), and prepares every command that changes GitHub for the admin to run: no heredocs or placeholders in them, JSON payloads as files, the bot credentials extended one repository at a time. It reads the docs and stubs from a clone of the latest release (or a tag the admin names, e.g. a release candidate for a pilot), so it never carries its own copy. Installed with npx skills add Algorithmiq/release-tools --skill adopt-release-tools; documented in docs/adopting.md. tests/test_skill.py checks its frontmatter, the major version it is written for, that every release-tools path and heading it reads exists, and that it stops for the admin and hands over only literal commands. Tested with dry runs on internal repositories (not named here), with and without the skill.

Adversarial review across model tiers

The skill was stress-tested for executors weaker than the one that wrote it: Claude Haiku 5.5 and Sonnet 5.5 and GPT-5.6 (Luna, Terra, Sol), from no thinking to high. Each took a 22-question quiz on its decision points (graded against a hidden key), ran it on paper against a hostile CodeArtifact fixture, or reviewed it for clarity and consistency with the docs. The core safety rules held on every model; the failures were where the text left a choice open, and the GPT models answered those with full confidence instead of asking.

What changed in the skill:

  • Authority: an explicit list of what the agent runs; it never opens issues or PRs on release-tools (both GPT quiz takers would have); never creates a tag, local or pushed.
  • Waits: a fixed phase-2 message ending in "reply Plan approved", and Stop points in phases 2, 4, 5 and 6; a command that needs an unanswered value is not handed over but listed under Waiting for; one worked example of a handed-over command.
  • CodeArtifact: phase 1 reads the organisation variables' values; an inlined value that differs is a question before phase 3, since the packages would move. With the variables missing, the skill asks an owner which values the organisation uses, then hands over step 5's commands with the answers.
  • Workflows: jobs that ship something for the tag move into the publish workflow; the rest keep their other triggers, with push: tags in place of release; a build that also runs on PRs is copied; the admin decides whether moved workflows run for pre-release tags.
  • Gaps: stub names already taken, integration_id of carried-over checks, querying only the index the repository publishes to, keeping hand-written changelog notes, local release.py/cliff.toml, re-runs on an open adoption PR.

Docs fixed alongside (the skill no longer works around them):

  • docs/repository-setup.md, step 1: adds the repository to the bot's variable and secret by id, not with --repos, which replaces each list.
  • Step 8: finds every ruleset of main through rules/branches/main, whatever its name, and adds the PR checks, a required_status_checks rule and the bot's bypass only where missing, so a second run changes nothing. Checked against the API on a scratch repository; tests/test_repository_setup.py runs its jq.
  • docs/adopting.md: a VERSION file kept in step with the tags next to setuptools-scm goes in version-files.

Edits were re-checked by rerunning the quiz and the CodeArtifact dry run on GPT-5.6 Luna and Claude Haiku 5.5 at low thinking until they converged.

Checklist

  • Bug fix for the current release: add the backport label (or ask a maintainer to)
  • just lint and just test pass
  • docs/adopting.md, docs/organisation-setup.md, and docs/repository-setup.md updated if the stubs, inputs or setup changed
  • Breaking change or deprecation: the Migration section says what consumers must change

Walks a repository admin through the adoption: inspect, plan, the adoption PR, the admin's commands, the first release. The agent reads and writes files; the admin runs every command that changes GitHub.

Assisted-by: Pi:claude-opus-5-5
…dentials one by one

Assisted-by: Pi:claude-opus-5-5
…mmands precise

Assisted-by: Pi:claude-opus-5-5
…al payload paths

Assisted-by: Pi:claude-opus-5-5
…e skill, and tighten done checks

Assisted-by: Pi:claude-opus-5-5
… the rulesets of main by rule

Step 1 adds the repository to the variable and the secret one at a time: --repos replaces each list. Step 8 finds every ruleset that applies to main, whatever its name, through rules/branches/main, and its jq adds the PR checks, a required_status_checks rule when there is none, and the bot's bypass without duplicating them, so it can run again. adopting.md says that a VERSION file kept in step with the tags next to setuptools-scm belongs in version-files.

Assisted-by: Pi:claude-opus-5-5
… tiers

Quizzes, paper dry runs and reviews by Claude 5.5 and GPT-5.6 models at several thinking levels showed where executors diverged or would act wrongly. The skill now: forbids opening issues upstream; lists what the agent may run; shows one handed-over command in full; holds back a command that needs an unanswered value; stops for the admin after the plan and in phases 4 to 6; reads the organisation variables' values and asks when an inlined one differs; places jobs of workflows that also run on PRs or main; asks whether moved workflows run for pre-releases; and covers stub name collisions, check integration ids, the index to query, hand-written changelog notes, local release.py and cliff.toml, re-runs, and step 8 on any ruleset of main. Tests pin the upstream rule, the stops, literal commands and the worked example.

Assisted-by: Pi:claude-opus-5-5
unique_by reordered the required checks, so a second run still sent a changed ruleset. Step 8 now appends each PR check and the bot's bypass only when absent, keeping the order: on release-tools' own main ruleset it is a no-op. Checked against the API on a scratch repository: the PUT of the added required_status_checks rule and of an unchanged ruleset are accepted. tests/test_repository_setup.py runs the docs' jq on sample rulesets.

Assisted-by: Pi:claude-opus-5-5
…r the commands that set them

No CODEARTIFACT_* organisation variable exists yet, so every CodeArtifact adoption stopped at a blocker with no way out. Phase 2 now shows the values phase 1 found inlined, several where workflows disagree, and asks which ones the organisation uses; only then does it hand an owner step 5's gh variable set commands with the answers written in, and ask the admin about every inlined value that differs. Dry runs with gpt-5.6-luna and claude-haiku-5-5 at low thinking first handed over commands before the answer and stated a mismatch instead of asking it; both now wait. A test checks that the six variables the skill maps are the six that organisation-setup.md, step 5, sets.

Assisted-by: Pi:claude-opus-5-5
…tifact

A role that a job also uses for something else, or whose name says it does (an S3 or a deploy role), may hold permissions the CodeArtifact read role lacks, used by code the workflow does not show. It is now a question of phase 2 and stays unless the admin says the CodeArtifact role covers the job; the phase 3 done check allows a role the admin kept.

Assisted-by: Pi:claude-opus-5-5
…d let organisation rulesets get the bypass

Step 8 added the PR checks to every ruleset of main, an organisation ruleset included: that would require them in every repository it targets, where they never report without the stubs, and hold every PR there. Its jq now takes the checks as an argument: a repository ruleset gets them, an organisation ruleset '[]', the bot's bypass only, read and written at orgs/. Without a repository ruleset on main, a new one requires the checks. Tests cover both cases and the new ruleset's payload.

Assisted-by: Pi:claude-opus-5-5
… a dry run on a real repository missed

A dry run on an internal repository found that phase 5 would add the PR checks to an organisation ruleset, holding the PRs of every repository it targets; it now gives such a ruleset the bot's bypass only, read and written at orgs/, and the checks go into a repository ruleset. Also: a script is no way around the file tool; step 6's payload is shown in full; phase 2 asks only what the skill leaves open; the adoption PR makes only the listed changes and keeps the version check as it is; matrix checks on release/* are named in the plan as a risk.

Assisted-by: Pi:claude-opus-5-5
@robertodr
robertodr merged commit 57a3936 into main Nov 5, 2026
5 checks passed
@robertodr
robertodr deleted the feat/adopt-skill branch November 5, 2026 13:50
robertodr added a commit that referenced this pull request Nov 5, 2026
… docs and skill (#36)

🤖 _AI text below_ 🤖

Follow-ups of the reviews of #26–#29.

**Leak checks**
- `scripts/check_internal_names.py`: paths git quotes (spaces, a double
quote, non-ASCII) are reported as named; output of `gh` that is not the
listing skips the check instead of crashing the hook; an empty listing
says "nothing to check" instead of "skipped".
- `tests/test_leaks.py`: each tracked file is read once for both
repository-wide tests, and a tracked file deleted but not staged is left
out.

**`docs/repository-setup.md`, step 2**
- The warning about tag-push workflows covers tagging a published
version, not only `v0.0.0`.
- The history snippet runs in a subshell that removes its temporary
directory and the worktree however it ends.

**`skills/adopt-release-tools/SKILL.md`**
- `gh release list --limit 1000`; inlined CodeArtifact values compared
with their variables in the facts table; the old-process docs row
widened to any manual step; the tag-push row covers tagging a published
version.
- Rulesets with `enforcement` `disabled` or `evaluate` stay out of step
8; an organisation ruleset handed to an owner has a done state.
- `branches-ignore` named for the `release/*` checks; `REPO` set in the
handed-over credentials command; command substitution counts as a
heredoc; every report ends with the commands run that were not reads.

The public-visibility target and the payload-path note were already in
place on `main`.

Closes #31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant