Repository navigation
ALCF CI: report results through a GitLab webhook, re-enable the bridge - #726
Merged
Merged
Conversation
alcf.yml no longer keeps a runner polling the Aurora pipeline for hours. It starts the pipeline as before and sets the commit status `ALCF / LTS stack (Aurora)` to pending; alcf-report.yml (restored) sets the result when GitLab's pipeline webhook dispatches it. Jobs that fail without running their script (no Aurora runner online, stuck_pending_no_matching_runners; a PBS job the runner lost, no_updates_running; runner errors) say nothing about the change. Such a pipeline is retried once with a fresh pipeline, and reported as an error that names the infrastructure failure if that fails too, instead of as a test failure. The concurrency group moves to the job, so that a labeled same-repo PR no longer cancels its own run (supersedes #722).
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #726 +/- ##
==========================================
- Coverage 84.66% 84.11% -0.55%
==========================================
Files 52 51 -1
Lines 4499 4375 -124
==========================================
- Hits 3809 3680 -129
- Misses 690 695 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-enables the ALCF bridge that 55bb6d1 turned off, and brings back
alcf-report.yml(removed in bccaac1) together with the GitLab webhook it was waiting for.Why the pipelines failed "without output"
Over the last few days the GitLab jobs failed in two different ways that had nothing to do with the code under test:
stuck_pending_no_matching_runners(38 jobs, e.g.build:supporton a0adaa1 andtb/defer-free): no Aurora runner was online on 2026-10-19 from about 13:30 to 19:25 UTC. GitLab marks those jobs as failed without a log.no_updates_running: PBS never started the batch job ("Not enough free nodes available and terminated"), and the runner gave up after 100 min.The real test failures (
script_failure) were separate.What changes
alcf.ymlstarts the pipeline as before (mirror sync, fork push behind thealcf-cilabel, a fresh pipeline on re-run), then sets the commit statusALCF / LTS stack (Aurora)to pending and exits. No runner waits for hours any more. If the commit's pipeline had already finished, it dispatches the report straight away.alcf-report.ymlis dispatched by a GitLab pipeline webhook and sets the final status:stuck_pending_no_matching_runners,no_updates_running,runner_system_failure, …), it starts one fresh pipeline and leaves the status pending.[skip ci]) pipelines are left alone.Verification so far
ghand the trigger call stubbed. Cases covered: a pass (70952), a test failure (70931), an infra failure on a commit that is no longer the branch tip (70764 → error), an infra failure on a branch tip (70792, 70790 → retry), a cancel, a skip, no pipeline yet, and an already finished pipeline (re-dispatched).api.github.com: a test delivery from a temporary webhook got GitHub's 401, and that webhook has been deleted.workflow_dispatchonly works for workflows on the default branch, so the end-to-end path can only be tested after merging:alcf-report.yml.gh workflow run alcf-report.yml -f pipeline_id=70952.Until the webhook exists, the ALCF status on this PR stays pending.