ci: bound every job and harden the unixODBC install

[christophebrun-forest]

What

Two problems, one root cause. The workflow declared no timeout-minutes anywhere, so a job that hangs pins the run for GitHub's 6-hour default — and the thing that hangs is the unixodbc-dev install on the two Snowflake legs.

On #359 those legs hung for 4 h 35 min on sudo apt-get update && sudo apt-get install -y unixodbc-dev, against ~75 s when they behave, and it reproduced on a fresh runner 14 h later. Because test has needs: [lint] and coverage has needs: [test], one stuck leg of a 26-job matrix froze the whole pipeline: no test job ever started, and the PR sat on UNSTABLE with nothing to show.

What the runner actually does

Adding the timeouts was enough to turn the hang into a legible failure, and the logs then answered the three questions the fix depended on.

The Azure mirror is the problem, and it is unreachable in both phases. Every https repo answers in milliseconds; azure.archive.ubuntu.com, which the image's /etc/apt/apt-mirrors.txt points at, does not — during the index refresh and during the package fetch:

08:47:01  Get:6 https://packages.microsoft.com/repos/azure-cli noble InRelease   # 200 ms
08:47:11  Ign:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease           # after 10 s
08:48:34  Need to get 306 kB of archives
08:49:05  Ign:2 http://azure.archive.ubuntu.com/.../libodbccr2 ...               # the .deb fetch

apt fails over on its own. The mirrorlist already carries archive.ubuntu.com, and apt switched to it unaided:

08:47:13  Get:3 https://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]

The package is genuinely absent from the image. The leg that passed reports 0 upgraded, 4 newly installed and unpacks unixodbc-dev itself, so the step cannot simply be dropped.

Changes

Timeouts, where there were none.

Job	Timeout
lint	15 min
test	20 min
coverage	10 min
the unixODBC step, in both lint and test	8 min

Both apt commands bounded from the outside, and nothing else.

sudo timeout -k 30 90 apt-get update -o Acquire::Retries=1 -o Acquire::http::Timeout=10 ||
  echo 'apt-get update incomplete; continuing with the lists on the image'
sudo DEBIAN_FRONTEND=noninteractive timeout -k 30 240 apt-get install -y \
  --no-install-recommends -o Acquire::Retries=1 -o Acquire::http::Timeout=10 unixodbc-dev

Four things this gets right, each of which an earlier revision of this PR got wrong:

• The bound comes from timeout, not from apt. Acquire::http::Timeout caps a single connection, not a whole fetch — with retries on top, one revision spent the entire 8-minute budget looping on Ign: lines and was killed inside its first command.
• Both commands are bounded. An earlier revision wrapped the index refresh only, while the four .deb files come from the same mirror and stall there too, as the log above shows.
• -k 30 makes the bounds real. timeout sends SIGTERM, which apt and dpkg may defer during a transaction; without a kill-after only the step cap would stop anything.
• No hand-rolled mirror fallback. An earlier revision rewrote the mirror host with sed. That duplicates the failover apt already performs, and rewriting the host desynchronises the indices under /var/lib/apt/lists, so the retry could die on Unable to locate package and name the wrong cause.

Refreshing the lists stays best effort: the image carries usable ones and only the install has to succeed. Worst case is 6.5 min, under the 8-minute cap, and there is no control flow left to reason about — the first line cannot fail, the second carries the verdict.

Three deliberate choices

deploy is left unbounded. Cutting semantic-release off in the middle of gem push-ing thirteen gems leaves a half-published release, which is worse than waiting out a hang. The other four jobs are safely interruptible; this one is not.

ci: and not fix:. .releaserc.js uses the angular preset with no path filter, so a fix: commit landing on main publishes all twelve gems. A workflow-only change has no business bumping a version, and ci is in the type list .overcommit.yml accepts.

One risk accepted rather than engineered away. Absorbing the update failure means the install can run against the image's lists and fail with 404 Not Found if the version they name has been superseded in the archive. Real but unlikely, and apt's own message names the cure. Guarding it would mean putting back the conditional chain this PR just removed.

Verification

The timeouts have room against measured history — longest Lint 4 min 21 s against 15, Test 3 min 12 s against 20, Coverage 12 s against 10 — so a slow but healthy run cannot go red on them.

The green run of the final revision is 32240803594, all four Snowflake legs passing. And the bound was exercised for real on an earlier revision, on a leg where the mirror was still down: timeout cut the stalled refresh, the best-effort message fired, and the install went through — 4 min 21 s against 2 min 04 s for the leg served by a healthy mirror, which is the cost of the bound and nothing more.

08:48:31  apt-get update incomplete; continuing with the lists on the image
08:48:34  0 upgraded, 4 newly installed, 0 to remove
08:49:19  Setting up unixodbc-dev:amd64 (2.3.12-1ubuntu0.24.04.1)

Possible follow-up, not done here

strategy.fail-fast is left at its default true, which is why a failing Snowflake leg shows up as cancelled siblings. Setting it to false on lint and test would let every leg report instead of masking some, which helps diagnosis — a separate concern from bounding the pipeline.

The step is still written twice, once per job. At four lines with no logic it is cheap duplication; folding it into a .github/scripts/ shell script or a composite action is a refactor worth its own change, and worth waiting until the mirror behaviour has been observed for a while.

🤖 Generated with Claude Code

Note

Bound CI job timeouts and harden unixODBC install with mirror fallback

• Adds timeout-minutes to all three CI jobs: lint (15 min), test (20 min), and coverage (10 min).
• Replaces the simple apt-get update && apt-get install for unixodbc-dev with a guarded script that sets DEBIAN_FRONTEND=noninteractive, bounds apt-get update with a 90s external timeout, and limits apt Acquire retries.
• On install failure, the script switches apt mirrors from azure.archive.ubuntu.com to archive.ubuntu.com and retries, reducing flaky CI failures on the Snowflake matrix entry.

Changes since #360 opened

• Modified unixodbc-dev installation in CI workflow to use inline DEBIAN_FRONTEND=noninteractive, direct timeout bounds on apt-get update and apt-get install commands, explicit Acquire::Retries and Acquire::http::Timeout options, and removed mirror fallback logic [d5ed2f2]
• Added kill-after timeout flag to apt-get commands in unixODBC installation step [e0c035c]
• Replaced comments in build.yml workflow describing apt mirror behavior and unixodbc-dev availability [c2d8363]

^{Macroscope summarized 238a9d2.}

[qltysh]

Coverage Impact

⬆️ Merging this pull request will increase total coverage on main by 0.3%.

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

[bexchauveto]

Reviewed the workflow change. Three findings, all duplicated verbatim in the test job (L116-139). Verified against the raw log of this PR's own green run (32234283306, job Lint (4.0, forest_admin_datasource_snowflake)).

Also checked and found clean: the bash -e {0} control flow (A && exit 0 failing does not trip errexit, so the fallback is reachable), the sudo timeout ordering (timeout runs as root, SIGTERM reaches apt), || true on the multi-file sed, --no-install-recommends (same 4 packages installed, Snowflake tests pass), and the job budgets (measured jobs run 1-4 min, coverage ~8 s).

[bexchauveto]

The install is unbounded, so the stall this PR fixes still eats the whole step budget and the fallback is never reached.

timeout 90 and Acquire::Retries=1 -o Acquire::http::Timeout=10 are applied to apt-get update only. The install fetches the four .debs from the same mirror, and this PR's own green log stalls exactly there:

08:48:34 Need to get 306 kB
         Ign: ... azure.archive.ubuntu.com   (x N, 38s)
         Fetched 306 kB in 37s (8164 B/s)

That run got lucky - apt's own mirrorlist failover kicked in. When the mirror stalls harder, the install has no bound of its own, burns the remaining ~6.5 min, and GitHub kills the step at timeout-minutes: 8. That fails the job inside the first install, which is precisely the shape the PR body says the earlier version got wrong, so echo 'Install failed; falling back...' never runs.

Suggested change

	sudo apt-get install -y --no-install-recommends unixodbc-dev && exit 0
	sudo timeout 240 apt-get install -o Acquire::Retries=1 -o Acquire::http::Timeout=10 -y --no-install-recommends unixodbc-dev && exit 0

[christophebrun-forest]

Confirmed and fixed in d5ed2f2. Your log reading holds: the Ign: lines at 08:49:05 sit after Need to get 306 kB, so they are inside the .deb fetch, with ~6.5 min of step budget still unspent — the install could indeed have been killed inside its first attempt, which is the exact shape the body claimed to have fixed.

The install now carries its own bound with the same acquisition options you suggested:

sudo DEBIAN_FRONTEND=noninteractive timeout -k 30 240 apt-get install -y \
  --no-install-recommends -o Acquire::Retries=1 -o Acquire::http::Timeout=10 unixodbc-dev

One addition beyond the suggestion, from a later pass: -k 30 on both bounds. timeout sends SIGTERM, which apt and dpkg may defer during a transaction, so without a kill-after the bounds would have stayed decorative and only the 8-minute step cap would have stopped anything. Worst case is now 6.5 min, still under it.

[bexchauveto]

The fallback duplicates failover apt already does, half its sed target list is a no-op, and it can leave things strictly worse.

/etc/apt/sources.list.d/ubuntu.sources on the runner image contains URIs: mirror+file:/etc/apt/apt-mirrors.txt, not the azure host - the substitution never matches there. And /etc/apt/apt-mirrors.txt already lists https://archive.ubuntu.com/ubuntu as a fallback entry; in the same log apt switched to it unaided:

Get:2 https://archive.ubuntu.com/ubuntu noble-updates/main amd64 libodbccr2 ...

So the rewrite mostly just duplicates an entry in the mirrorlist. The downside is real though: the image's indices in /var/lib/apt/lists are keyed to azure.archive.ubuntu.com and match no configured source after the sed, while the following apt_update swallows its own failure by design and 90 s is tight for a full noble/updates/security/backports index fetch from the out-of-datacenter global mirror. Scenario: azure indices unreachable (as observed) + first install fails fast for an unrelated reason (a dpkg lock held by unattended-upgrades is the usual one on hosted runners) -> sed -> second update times out -> final install dies with E: Unable to locate package unixodbc-dev, pointing at the wrong cause.

Suggest dropping the fallback branch entirely and putting the effort into bounding the install. If you keep it, make the second apt_update fatal - unlike the first, its success is a precondition for the retry - or give it a much larger timeout.

[christophebrun-forest]

Took the primary recommendation: the fallback branch is gone (d5ed2f2).

Your central claim is visible in the same log, which settles it — apt failed over on its own, no sed involved:

08:47:13  Get:3 https://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]

So the alternative you offered (making the second apt_update fatal) is moot, and the desynchronised /var/lib/apt/lists risk goes with the branch. I could not inspect the runner filesystem to confirm ubuntu.sources carries mirror+file:, but Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist in the same log corroborates the indirection.

One residual I have chosen to accept rather than engineer away, now stated in the PR body: the update failure is still swallowed, so the install can run against the image's lists and fail with 404 Not Found if the version they name has been superseded in the archive. Guarding it would mean restoring the conditional chain this change just removed, and apt's own message names the cure.

[bexchauveto]

DEBIAN_FRONTEND: noninteractive does not reach apt-get.

It is exported into the step's shell, but every apt invocation goes through sudo, and sudo's default env_reset drops non-env_keep variables. If a package ever did open a debconf prompt, the step would block until the 8-minute cap rather than being protected as intended.

Pass it through the boundary that matters: sudo DEBIAN_FRONTEND=noninteractive apt-get ..., or sudo -E.

[christophebrun-forest]

Correct, and fixed in d5ed2f2. The step-level env: is dropped, since it protected nothing, and the variable now crosses the boundary that matters:

sudo DEBIAN_FRONTEND=noninteractive timeout -k 30 240 apt-get install ...

I used the inline assignment rather than sudo -E to keep it to the single variable that needs to survive. The green run on e0c035c confirms sudoers accepts it: command-line env assignments are normally filtered, but the runner's rule matches ALL, which is the documented exception that permits them.

[forest-bot]

🎉 This PR is included in version 1.39.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀