Skip to content

ALCF CI: report results through a GitLab webhook, re-enable the bridge - #726

Merged
michel2323 merged 1 commit into
mainfrom
alcf-webhook-report
Oct 20, 2026
Merged

michel2323 merged 1 commit into
mainfrom
alcf-webhook-report

Conversation

@michel2323

Copy link
Copy Markdown
Member

Re-enables the ALCF bridge that 55bb6d1 turned off, and brings back alcf-report.yml (removed in bccaac1) together with the GitLab webhook it was waiting for.

Why the pipelines failed "without output"

Over the last few days the GitLab jobs failed in two different ways that had nothing to do with the code under test:

  • stuck_pending_no_matching_runners (38 jobs, e.g. build:support on a0adaa1 and tb/defer-free): no Aurora runner was online on 2026-10-19 from about 13:30 to 19:25 UTC. GitLab marks those jobs as failed without a log.
  • no_updates_running: PBS never started the batch job ("Not enough free nodes available and terminated"), and the runner gave up after 100 min.

The real test failures (script_failure) were separate.

What changes

  • alcf.yml starts the pipeline as before (mirror sync, fork push behind the alcf-ci label, a fresh pipeline on re-run), then sets the commit status ALCF / LTS stack (Aurora) to pending and exits. No runner waits for hours any more. If the commit's pipeline had already finished, it dispatches the report straight away.
  • alcf-report.yml is dispatched by a GitLab pipeline webhook and sets the final status:
    • If the only failures are infrastructure ones (stuck_pending_no_matching_runners, no_updates_running, runner_system_failure, …), it starts one fresh pipeline and leaves the status pending.
    • If that fails the same way, the status becomes an error that names the infrastructure failure, not a test failure.
    • Skipped ([skip ci]) pipelines are left alone.
  • Concurrency moves to the job, as in ALCF CI: keep labeling a PR from cancelling its LTS run #722, which this supersedes.

Verification so far

  • Both scripts were run against real ALCF pipelines, with gh and the trigger call stubbed. Cases covered: a pass (70952), a test failure (70931), an infra failure on a commit that is no longer the branch tip (70764 → error), an infra failure on a branch tip (70792, 70790 → retry), a cancel, a skip, no pipeline yet, and an already finished pipeline (re-dispatched).
  • GitLab can reach api.github.com: a test delivery from a temporary webhook got GitHub's 401, and that webhook has been deleted.

workflow_dispatch only works for workflows on the default branch, so the end-to-end path can only be tested after merging:

  1. Create a fine-grained token (JuliaGPU/oneAPI.jl only, Actions: read and write).
  2. Add the pipeline webhook to GitLab project 238. Its URL, template and header are described at the top of alcf-report.yml.
  3. Report a known pipeline by hand: gh workflow run alcf-report.yml -f pipeline_id=70952.

Until the webhook exists, the ALCF status on this PR stays pending.

alcf.yml no longer keeps a runner polling the Aurora pipeline for hours. It
starts the pipeline as before and sets the commit status
`ALCF / LTS stack (Aurora)` to pending; alcf-report.yml (restored) sets the
result when GitLab's pipeline webhook dispatches it.

Jobs that fail without running their script (no Aurora runner online,
stuck_pending_no_matching_runners; a PBS job the runner lost,
no_updates_running; runner errors) say nothing about the change. Such a
pipeline is retried once with a fresh pipeline, and reported as an error that
names the infrastructure failure if that fails too, instead of as a test
failure.

The concurrency group moves to the job, so that a labeled same-repo PR no
longer cancels its own run (supersedes #722).
@michel2323
michel2323 merged commit 705aeb0 into main Oct 20, 2026
2 of 3 checks passed
@michel2323
michel2323 deleted the alcf-webhook-report branch October 20, 2026 13:48
@codecov

codecov Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.11%. Comparing base (70d831a) to head (071b88d).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #726      +/-   ##
==========================================
- Coverage   84.66%   84.11%   -0.55%     
==========================================
  Files          52       51       -1     
  Lines        4499     4375     -124     
==========================================
- Hits         3809     3680     -129     
- Misses        690      695       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant