ci: make the Console SQL suite survive a degraded agent

[antiguru]

Every test in the console SQL suite carries a wall-clock budget sized against a healthy agent, and one test additionally budgets its subscribe snapshot at a fixed second. Build 134474 landed on an agent that ran 6-12x slow and four tests failed: three on the 45 second per-test timeout, and useClustersDropdown subscribe on an empty snapshot. The dilation was uniform across work Materialize does and work it does not, so the cause was the agent rather than the code under test.

Comparing that build against the previous passing run of the same suite, on the same branch and the same 87 testdrive resets:

measurement	passing run	failing run	ratio
yarn install fetch step	5.3 s	32.1 s	6.0x
yarn install link step	7.8 s	51.3 s	6.6x
testdrive reset, median	0.36 s	4.42 s	12x
testdrive reset, total	32.7 s	403.5 s	12x
whole vitest suite	147.9 s	1281.8 s	8.7x

yarn install never touches Materialize and still slowed 6x, which is what rules out a regression in the code under test.

Make the dropdown test poll the subscribe cursor until the cluster it asserts on arrives, instead of reading one one-second FETCH window and asserting on whatever that returned. Optimizing the subscribe and computing its snapshot is work whose duration the test does not control, and peek optimization alone reached 2.3 seconds in that run, so the old budget could not hold. The poll stops at 30 seconds, below the suite's testTimeout, so a snapshot that genuinely never arrives still fails on the assertion and names the missing cluster.

Retry the step once for the rest. Their budgets are not the problem: a dilated run already spends 25.5 of the step's 30 minutes, so raising them converts test failures into step timeouts, and the suite's largest fixed cost, 403 seconds of catalog resets in that run, is too small to close a gap that size. A retry does mean a genuine console regression is attempted twice before the step goes red.

Modified console/src/platform/shell/useClusterDropdown.test.sql.ts, the test that was failing. No new tests: the change makes an existing test wait on the signal it asserts on rather than on the clock.

🤖 Generated with Claude Code

[leedqin]

LGTM!

[def-]

QA LLM Review (Post Merge)

@antiguru — an automated review of commit c32d888931 found the following potential MEDIUM+ issue(s) after this PR was merged.

1. MEDIUM -- Declaring retry.automatic silently drops this step's infra retries

ci/test/pipeline.template.yml:778

ci/mkpipeline.py:767 skips any step that already declares retry.automatic, so adding the exit_status: 1 rule removes the four pipeline-wide rules this step used to get: agent lost (exit_status: -1/signal_reason: none), agent_stop, exit_status: 128, and exit_status: 199. The step now hard-fails on exactly the infra classes the change is about, including a lost or OS-stopped agent, which is the most likely end state of a 6-12x degraded one.

Details

retry = step.setdefault("retry", {})
if "automatic" in retry:
    continue

exit_status: 1 does not cover those codes. console-sql-test runs through the mzcompose plugin, which deliberately propagates the non-1 codes so the shared rules can match: ci/plugins/mzcompose/hooks/command:419 ("Keep the retryable code (128 for GHCR/DockerHub trouble, 199 for a corrupted cargo target dir) ... Overwriting it with 1 turns a self-healing infra failure into a job that stays red until someone retries it by hand"), and the 128 comes from misc/python/materialize/mzbuild.py:1465 when a required image cannot be pulled. Before this commit that pull failure retried twice; now it goes red on the first attempt.

The convention in this file is to re-list the shared rules when overriding, as the merge-skew step does at ci/test/pipeline.template.yml:130.

         retry:
           automatic:
             - exit_status: 1
               limit: 1
+            - exit_status: -1
+              signal_reason: none
+              limit: 2
+            - signal_reason: agent_stop
+              limit: 2
+            - exit_status: 128
+              limit: 2
+            - exit_status: 199
+              limit: 2

Alternatively, teach set_retry_on_agent_lost to merge its rules into an existing automatic list instead of skipping the step, which fixes this class of mistake for every future override.

[antiguru]

Confirmed, and it is wider than this step. Fix in #38808, taking the second option: set_retry_on_agent_lost now merges the shared rules into whatever a step declares instead of skipping the step, adding only rules the step does not already list and leaving alone any step that retries on exit_status: "*".

Applying the old and the new function to every checked-in template, thirteen steps were silently missing rules, not just this one:

• all four rules: devel-docker-tags in test, nightly and spec-sheet; console-sql-test; aws-real; aws-glue-schema-registry-real; the four k8s-node-recovery-* steps; terraform-aws-upgrade; terraform-azure
• agent-lost only: check-merge-with-target, which re-lists the other three by hand

After the change every step in every template carries all four with no duplicates. Re-listing inside a step stays correct and is now redundant rather than required, so I left the existing re-listings alone.

Posted by Claude Code on behalf of antiguru.