ci: make unattended workflow failures team-visible

[rwgk]

Description

closes #2204

Adds a default-branch workflow_run monitor for unattended CI, test, and
security workflows. An actionable failure opens a rolling GitHub issue labeled
bug, CI/CD, triage, and ci-workflow-health; later failures are added as
comments, and the next successful run in the same trigger scope records
recovery and closes it. Existing rolling issues gain the workflow-health label
the next time they are processed, without replacing any other labels.

The ci-workflow-health repository label has now been created. It is the
stable query surface for people, saved issue views, and future dashboard
automation; the hidden issue marker remains the stable incident identity.

Scheduled and main-push health are tracked independently. The monitor ignores
PR, manual, and expected push-cancellation results, deduplicates event
deliveries, queues concurrent updates, and rejects stale completions so
out-of-order delivery cannot leave the incident in the wrong state. It uses the
repository GITHUB_TOKEN and does not require a new notification secret.

Validation

• pre-commit run --all-files
• actionlint with ShellCheck enabled
• Eight mocked GitHub API scenarios covering first failure, repeated failure,
  duplicate delivery, recovery, trigger-scope isolation, stale completion,
  ignored cancellation, and duplicate incident recovery

Rollout and live validation

GitHub only dispatches a
workflow_run event
when the listening workflow file exists on the repository's default branch.
The YAML, embedded script, and mocked API behavior can be checked on this
branch, but event delivery and the complete open/update/recover lifecycle can
only be tested easily and fully after this workflow exists on main.

This workflow is a leaf observer: no required check, build, test, package,
release, or downstream workflow depends on its result. Nothing depends on this
monitor yet, so bugs in it cannot cause downstream issues. Their impact would
be confined to the monitor's own reporting, such as a missing or extra
rolling-issue update; they cannot change the result of the monitored workflows
or affect their artifacts. An agile rollout is consequently the most efficient
approach: land the monitor, observe real completion events, and refine its
filters or reporting from that feedback.

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[rwgk]

How does the new workflow work?

This workflow is a small observer that runs after selected CI workflows. It
does not replace those workflows or sit in their critical path:

monitored workflow completes
        -> GitHub emits workflow_run
        -> this monitor classifies the completion
        -> a rolling GitHub issue is opened, updated, or closed

The key GitHub feature is the
workflow_run event.
It lets one workflow react when any of several named workflows completes. The
new monitor subscribes to six workflow names, to the completed activity, and
to runs associated with main. GitHub delivers the event regardless of whether
the triggering workflow succeeded or failed; the event payload contains the
trigger, branch, conclusion, run number, attempt, commit, and run URL that the
monitor needs to make its decision.

There is one particularly important property of workflow_run: GitHub only
dispatches it when the listening workflow file exists on the repository's
default branch. Consequently, pull-request checks can validate the YAML and the
script logic, but the real event delivery and end-to-end issue lifecycle become
easy to exercise only after this file has landed on main.

For each completion, the job first narrows the event to the unattended scopes
we care about:

• scheduled runs; or
• push runs whose head branch is the repository's default branch.

Successful runs are recovery signals. Selected failure conclusions are
unhealthy signals. A cancelled scheduled run is also unhealthy because an
unattended nightly run did not finish, while a cancelled push run is ignored
because superseded main-branch runs are normal.

Each rolling incident is identified by a hidden marker containing the GitHub
workflow ID and a trigger scope (schedule or push-main). This is why CI
can have an independent scheduled incident and main-push incident. A second
hidden marker identifies the exact run and attempt, making redelivered events
idempotent.

The issue lifecycle is then straightforward:

• The first unhealthy completion opens an issue with the run details.
• Another unhealthy completion adds a comment to that same issue.
• The next successful completion in the same scope adds a recovery comment and
  closes the issue.
• Before changing anything, the monitor checks for a newer completed run so an
  out-of-order event cannot restore stale state.

Updates for the same workflow and scope are queued through Actions concurrency,
and duplicate event markers prevent repeated comments. The workflow needs only
actions: read to inspect run history and issues: write to maintain its own
rolling issues; it checks out and executes no code from the triggering run.

A new ci-workflow-health repository label was created already. New rolling issues
receive it at creation time, and an older marker-identified rolling issue gains
it additively the next time it is processed. The hidden marker remains the
stable incident identity, while the label provides a simple, human- and
machine-readable discovery surface for dashboards.

[rwgk]

Sketch for dash-boarding based on the results produced by the new workflow

The rolling issues can remain the source of truth, with progressively richer
views layered on top. The repository label ci-workflow-health already exists
and gives all of those views one stable query contract.

1. Immediate GitHub dashboard

This issue filter shows the currently open rolling incidents:

is:issue state:open label:"ci-workflow-health" sort:updated-desc

Open the filtered view.
It can be bookmarked immediately. A repository maintainer can also turn it
into a shared saved view named, for example, CI workflow health, and pin
that view in the repository sidebar. GitHub supports both
shared repository issue views
and pinning views in the repository sidebar.

The current configuration tracks these scopes:

Workflow	Trigger scope
CI	scheduled
CI	push on main
CI: Coverage	scheduled
CI: Nightly optional-deps	scheduled
CI: pixi run test (source build)	scheduled
Security Suite (Pulse + CodeQL)	push on main
Static Analysis: Bandit Scan	push on main

That is six monitored workflow names but seven possible incident scopes
because CI has independent schedule and push-main scopes. A useful
dashboard should therefore say “7 monitored scopes across 6 workflows” or
show two sub-rows for CI.

2. Daily Slack digest

A scheduled Slack bot or workspace agent could use a read-only GitHub
MCP/connector to run the same query every morning and post one compact message:

CUDA Python CI health - 08:00 PT

2 open incidents across 7 monitored scopes / 6 workflows
- CI: Coverage - scheduled - open 2d - #2740
- CI - push-main - open 5h - #2741

No other open rolling incidents found.
View CI workflow health -> <saved-view URL>

The minimal architecture is:

daily schedule
    -> read-only GitHub issue search
    -> group by workflow and trigger scope
    -> post one message to the team Slack channel

The agent should be restricted to NVIDIA/cuda-python, should not edit issues,
and should treat issue text as data rather than instructions. This can be done
with a native scheduled workspace agent, or with a small scheduled service and
a Slack webhook if deterministic code is preferred.

Two status rules are important:

• An empty query means “No open incidents found,” not necessarily “all
  workflows are healthy.” The monitor may not have observed a workflow yet.
• A GitHub/API failure must produce “Dashboard unavailable,” never a green
  report.

For a stronger green/amber/red dashboard, the agent can additionally verify
that the CI: Track workflow health monitor itself ran recently and inspect
the latest completion for every expected scope. Slack should remain an
informational summary; the labeled GitHub issues remain the authoritative
incident records.

A GitHub Project becomes worthwhile only if we later want charts, owners,
custom fields, historical reporting, or aggregation across repositories. For
the initial seven scopes, the saved issue view plus a morning Slack digest is
the smallest useful dashboard.

[rwgk]

Regarding the change in .github/actionlint.yaml:

This change is needed, although CUDA Python is already pinned to actionlint's latest release, v1.7.12. That release predates GitHub's concurrency.queue key, so actionlint reports the valid queue: max setting as an unexpected key.

Keeping queue: max matters: GitHub uses it to retain multiple pending runs in the same concurrency group (up to 100), rather than canceling and replacing an earlier pending run whenever another one arrives. The health workflow serializes events for each workflow and trigger scope so that two runs cannot update the same rolling issue concurrently. Without queue: max, a burst of workflow completions could discard intermediate failure or recovery events, leaving an incomplete incident history. The setting lets us serialize those updates without intentionally dropping events. GitHub documents the behavior in Queueing multiple pending runs.

The actionlint configuration change is deliberately narrow: it suppresses only the exact queue parser diagnostic, and only for .github/workflows/ci-workflow-health.yml. All other actionlint checks remain enabled. The existing self-hosted-runner label configuration predates this PR and is independent of this compatibility suppression.

For completeness: support for queue: max is proposed in actionlint PR #654, but the PR is still open, and it is against our policy to depend on an unmerged third-party PR. Pinning to that implementation would also mean executing unreleased fork code during pre-commit checks, which is a worse tradeoff than a precise local suppression.

Once queue support is merged and included in an actionlint release, we can update the frozen actionlint pin and remove this temporary suppression.

[github-actions]

Doc Preview CI
Preview removed because the pull request was closed or merged.