Skip to content

ci: merge the shared infra retries into a step's own retry rules - #38808

Merged
antiguru merged 1 commit into
MaterializeInc:mainfrom
antiguru:retry-merge-shared-rules
Sep 12, 2026
Merged

ci: merge the shared infra retries into a step's own retry rules#38808
antiguru merged 1 commit into
MaterializeInc:mainfrom
antiguru:retry-merge-shared-rules

Conversation

@antiguru

Copy link
Copy Markdown
Member

set_retry_on_agent_lost gives every step four automatic retries for failures that belong to the infrastructure rather than the step: a lost agent, an agent stopped by the OS, exit 128 for a registry or GHCR hiccup, and exit 199 for a Rust ICE. It skipped any step that declared retry.automatic of its own:

retry = step.setdefault("retry", {})
if "automatic" in retry:
    continue

So a step asking for one extra rule lost all four rather than gaining a fifth. Nothing reports that, and the step then hard-fails on precisely the transient classes the shared rules exist for.

The loss is worth more than it looks. ci/plugins/mzcompose/hooks/command deliberately propagates 128 and 199 out of a job rather than collapsing them to 1, with a comment saying that overwriting them "turns a self-healing infra failure into a job that stays red until someone retries it by hand", and mzbuild.py exits 128 when a required image cannot be pulled. A step that overrode the rules went red on the first failed image pull where every other step retried twice — the MinIO Docker Hub outage fixed in #38802 is exactly that shape.

Merge instead of skip, adding only the rules a step does not already declare, and leave a step that retries on every exit status alone since it covers them already.

What regains coverage

Applying the old and new function to every checked-in template, thirteen steps were silently missing rules:

pipeline steps was missing
test devel-docker-tags, console-sql-test all four
test check-merge-with-target agent-lost only, having re-listed the other three by hand
nightly devel-docker-tags, aws-real, aws-glue-schema-registry-real, 4× k8s-node-recovery-*, terraform-aws-upgrade, terraform-azure all four
spec-sheet devel-docker-tags all four

After the change every step in every template carries all four, with no duplicated rules. The ci/deploy steps retry on exit_status: "*" and are left untouched.

Re-listing the shared rules inside a step stays correct and is now redundant rather than required; I left the existing re-listings in place rather than widen the diff.

Found by the post-merge QA review on #38797.

🤖 Generated with Claude Code

`set_retry_on_agent_lost` gives every step four automatic retries for
failures that belong to the infrastructure rather than the step: a lost
agent, an agent stopped by the OS, exit 128 for a registry or GHCR hiccup,
and exit 199 for a Rust ICE. It skipped any step that declared
`retry.automatic` of its own, so a step asking for one extra rule lost all
four instead of gaining a fifth. Nothing reports that, and the step then
hard-fails on precisely the transient classes the shared rules exist for.

The loss is worth more than it looks. The mzcompose plugin deliberately
propagates 128 and 199 out of a job rather than collapsing them to 1, so
that these rules can match, and `mzbuild` exits 128 when a required image
cannot be pulled. A step that overrode the rules therefore went red on the
first failed image pull where every other step retried twice.

Merge instead of skip, adding only the rules a step does not already
declare, and leave a step that retries on every exit status alone since it
covers them already. Thirteen steps across the test, nightly and spec-sheet
pipelines regain rules they were silently missing, among them every
`devel-docker-tags`, the four `k8s-node-recovery` steps, both Terraform
steps and `console-sql-test`. `check-merge-with-target` regains the
agent-lost rule alone, having re-listed the other three by hand.

Re-listing the shared rules in a step stays correct and is now redundant
rather than required.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@bosconi bosconi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 We should ask Claudex to find more of these.

@antiguru
antiguru enabled auto-merge (squash) September 12, 2026 00:34

Copy link
Copy Markdown
Member Author

Thanks for the review!

@antiguru
antiguru merged commit 407c1f9 into MaterializeInc:main Sep 12, 2026
90 checks passed
@antiguru
antiguru deleted the retry-merge-shared-rules branch September 12, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants