MLE-32297: Consolidate Jenkins pipeline stage agents - #478
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A top-level cld-docker agent can remain reserved while stages with their own agent blocks still allocate additional executors, which may undercut the intended reduction in node allocations and worsen worker contention (especially when ARM/graviton stages run).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the repository’s Jenkins pipeline to reduce Jenkins worker thrashing by consolidating many x86 stages onto a single shared cld-docker agent context, instead of acquiring/releasing a node per stage.
Changes:
- Replaces
agent nonewith a top-levelagent { node { label 'cld-docker' } }so stages without their own agent reuse one executor. - Removes
agent { node { label 'cld-docker' } }from multiple sequential stages (Clean/Pre-check/Copy-RPMs/Build/Lint/Scan/Publish/BlackDuck). - Removes explicit
node('cld-docker')allocations inpostblocks so cleanup/notifications run in the existing agent context.
File summaries
| File | Description |
|---|---|
| Jenkinsfile | Consolidates x86 stage execution onto a shared cld-docker agent and removes extra node allocations in post to reduce per-stage re-queuing overhead. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…op-level agent Holding a top-level cld-docker agent for the whole pipeline while SCAP-Scan/Structure-Tests/Docker-Run-Tests still declared their own agent caused two nodes of the same scarce label pool to be held at once (and left the x86 executor idle during graviton stages), risking queue deadlock under load. Reverted to agent none and grouped the always-cld-docker stages that never overlap with the graviton stages into two nested-stage blocks (Prepare-Build-Lint-Scan, Publish-And-Scan), each with its own agent scoped to just that group and released before the next stage needs a different node.
vitalykorolev
requested review from
barkhachoithani,
pengzhouml,
rwinieski and
sumanthravipati
September 4, 2026 00:26
rwinieski
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: MLE-32297
Problem
Since ARM/Graviton build support was added, the pipeline switched from a single top-level agent to
agent nonewith nearly every stage declaring its ownagent { node { label 'cld-docker' } }(Clean-Previous-Results, Pre-Build-Check, Copy-RPMs, Build-Image, Pull-Upgrade-Image, Lint, Scan, Structure-Tests, Docker-Run-Tests, Publish-Image, BlackDuck-Scan), plus separatenode()allocations in eachpostblock branch. This causes ~11+ independent node allocate/release cycles per build even though these stages run strictly sequentially and never in parallel.With only 3 physical
cld-dockerworkers and multiple nightly builds triggered within overlapping cron windows, any delay in an earlier build causes builds to pile up (6+ concurrent runs observed), and the fine-grained per-stage node acquisition creates executor thrashing/queuing between every stage, significantly increasing total build time.Fix
Consolidated the stages that always require
cld-dockeronto the pipeline's single top-level agent (inherited implicitly, no per-stage re-allocation), and removed the explicitnode()wrapping inpostblock branches so cleanup/notification reuse the same agent context. ARM/graviton-specific stages (SCAP-Scan, Load-Image, Structure-Tests/Docker-Run-Tests when ARM, Cleanup-ARM) are intentionally left unchanged since they must switch node pools.This reduces node allocations per x86 build from ~11 down to 1.
Testing
Verified by running the updated pipeline manually; behaves as expected.
Follow-up (tracked separately, not in this PR)
Add a Lockable Resources based concurrency cap once DevOps defines cld-docker/cld-docker-graviton resource pools, to hard-limit total concurrent pipeline runs to available worker capacity.
Jira: MLE-32297