Skip to content

OCPNODE-4623: e2e for Block runc on RHEL 10 via OSImageURL stream class inspection - #31433

Open
asahay19 wants to merge 1 commit into
openshift:mainfrom
asahay19:4623
Open

OCPNODE-4623: e2e for Block runc on RHEL 10 via OSImageURL stream class inspection#31433
asahay19 wants to merge 1 commit into
openshift:mainfrom
asahay19:4623

Conversation

@asahay19

@asahay19 asahay19 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds disruptive e2e coverage for the osImageURL runc-on-RHEL-10 guard (OCPNODE-4518 / MCO stream-class inspection), in the existing runc RHCOS 10 upgrade guard suite alongside the osImageStream cases.

The new test verifies that when a pool uses runc via ContainerRuntimeConfig and targets RHCOS 10 through a MachineConfig osImageURL override (with no MachineConfigPool.spec.osImageStream), MCO blocks at render time (RenderDegraded), sets cluster upgradeability to Upgradeable=False, and leaves the node on RHCOS 9 with runc.

Key design points:

  • MCP omits osImageStream entirely — osImageURL and osImageStream on the same pool are mutually exclusive and produce a different render error, not the runc guard.
  • Guard assertion matches MCO’s message: targets a RHEL 10 OS image where runc is not available; fails fast if the stream+URL conflict message appears instead.
  • On clusters whose OSImageStream default is rhel-10, the test temporarily pins RHCOS 9 via a baseline osImageURL MC, removes it, then applies the RHEL 10 URL so only one user URL MC is active at guard time.

Summary by CodeRabbit

  • Tests

    • Added coverage for RHEL 9-to-10 upgrades using osImageURL when runc is the default runtime.
    • Verifies upgrades are blocked, rendering enters the expected degraded state, and nodes remain ready without an unintended rollout.
    • Added cleanup and validation for temporary upgrade configurations and machine settings.
  • Documentation

    • Documented the separate osImageURL upgrade-guard scenario, expected behavior, configuration constraints, and related references.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 27, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 27, 2026

Copy link
Copy Markdown

@asahay19: This pull request references OCPNODE-4623 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

This PR adds disruptive e2e coverage for the osImageURL runc-on-RHEL-10 guard (OCPNODE-4518 / MCO stream-class inspection), in the existing runc RHCOS 10 upgrade guard suite alongside the osImageStream cases.

The new test verifies that when a pool uses runc via ContainerRuntimeConfig and targets RHCOS 10 through a MachineConfig osImageURL override (with no MachineConfigPool.spec.osImageStream), MCO blocks at render time (RenderDegraded), sets cluster upgradeability to Upgradeable=False, and leaves the node on RHCOS 9 with runc.

Key design points:

  • MCP omits osImageStream entirely — osImageURL and osImageStream on the same pool are mutually exclusive and produce a different render error, not the runc guard.
  • Guard assertion matches MCO’s message: targets a RHEL 10 OS image where runc is not available; fails fast if the stream+URL conflict message appears instead.
  • On clusters whose OSImageStream default is rhel-10, the test temporarily pins RHCOS 9 via a baseline osImageURL MC, removes it, then applies the RHEL 10 URL so only one user URL MC is active at guard time.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Walkthrough

The test suite adds RHEL 9→10 osImageURL upgrade-guard coverage when runc is the default runtime, including resource setup, RenderDegraded and Upgradeable assertions, node-state checks, cleanup, and documentation.

Changes

Runc osImageURL upgrade guard

Layer / File(s) Summary
osImageURL test assets and configurable runtime resources
test/extended/node/runc_upgrade_cases.go
Adds helpers and state for creating osImageURL MachineConfigs, resolving OSImages, configuring runc CRCs, and constructing an osImageURL-capable MCP.
osImageURL runc guard execution
test/extended/node/runc_upgrade_cases.go
Adds the RHEL 9→10 guard test, validates the expected RenderDegraded message and blocked Upgradeable condition, and confirms the node remains ready on RHCOS 9 with runc.
Resource cleanup and scenario documentation
test/extended/node/runc_upgrade_cases.go, test/extended/node/runc_upgrade_cases.md
Cleans up tracked CRCs and MachineConfigs and documents the osImageURL guard separately from UC6A.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RuncUpgradeTest
  participant MachineConfigPool
  participant ClusterVersionOperator
  participant Node
  RuncUpgradeTest->>MachineConfigPool: apply RHEL 10 osImageURL override
  MachineConfigPool-->>RuncUpgradeTest: report osImageURL+runc RenderDegraded guard
  RuncUpgradeTest->>ClusterVersionOperator: inspect Upgradeable condition
  ClusterVersionOperator-->>RuncUpgradeTest: report upgrade blocked
  RuncUpgradeTest->>Node: inspect readiness, OS version, and runtime
  Node-->>RuncUpgradeTest: remain ready on RHCOS 9 with runc
Loading
🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding e2e coverage for blocking runc on RHEL 10 via OSImageURL inspection.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The Ginkgo titles are static literals; no timestamps, generated names, or variable interpolation were found in the added tests.
Test Structure And Quality ✅ Passed The added Ginkgo tests are scoped to single upgrade-guard scenarios, use BeforeEach/AfterEach cleanup, and all cluster waits include explicit timeouts.
Microshift Test Compatibility ✅ Passed The new e2e test suite already skips MicroShift in BeforeEach, so its use of MachineConfig, ClusterVersion, and other non-MicroShift APIs is protected.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The added e2e is wrapped by a Describe that skips SingleReplicaTopology before any worker-labeling/rollout logic runs.
Topology-Aware Scheduling Compatibility ✅ Passed PR only adds e2e test/docs; no deployment manifests, operator code, or controllers, and no new topology-sensitive scheduling constraints.
Ote Binary Stdout Contract ✅ Passed PASS: runc_upgrade_cases.go has no main/TestMain/init/RunSpecs or fmt.Print/log stdout writes; only test-scoped BeforeEach/It helpers and framework.Logf.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The new e2e test uses only cluster APIs/OSImageStream and node state; I found no hardcoded IPv4, localhost, or public internet connectivity.
No-Weak-Crypto ✅ Passed Touched files add test logic only; scans found no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, crypto imports, or constant-time secret comparisons.
Container-Privileges ✅ Passed No added manifest or pod spec sets privileged, host* IPC/PID/network, SYS_ADMIN, root, or allowPrivilegeEscalation; new code only creates MCP/MachineConfig/CRC objects.
No-Sensitive-Data-In-Logs ✅ Passed New logs only print pool names and MCO render-status text; no passwords, tokens, PII, or other sensitive values were added.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: asahay19
Once this PR has been reviewed and has the lgtm label, please assign cpmeadors for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
test/extended/node/runc_upgrade_cases.go (1)

298-316: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Cleanup drain-wait will short-circuit for this test because the pool is still degraded.

This test (unlike UC6A) never recovers the pool to a healthy state before the It ends — it's left RenderDegraded/Upgradeable=False by design. After deleting the CRC/MCs here, the shared drain wait at Line 314 calls WaitForMCP(..., WaitMCPWithMachineCount(0)) without WaitMCPAllowDegraded(). Per WaitForMCP's contract, it returns an error on the first poll that still observes degraded/renderDegraded == true, before ever checking whether the pool actually drained — so for this test the "wait for machine count 0" step will effectively no-op (logged, not asserted) instead of confirming a real drain before deleteMachineConfigPool runs.

🔧 Proposed fix
 		if nodeName != "" && testPoolName != "" {
-			if err := WaitForMCP(ctx, mcClient, testPoolName, 10*time.Minute, WaitMCPWithMachineCount(0)); err != nil {
+			if err := WaitForMCP(ctx, mcClient, testPoolName, 10*time.Minute, WaitMCPWithMachineCount(0), WaitMCPAllowDegraded()); err != nil {
 				framework.Logf("cleanup: failed waiting for MCP %s machine count 0: %v", testPoolName, err)
 			}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/node/runc_upgrade_cases.go` around lines 298 - 316, Update the
cleanup drain wait in the nodeName/testPoolName block to call WaitForMCP with
WaitMCPAllowDegraded() alongside WaitMCPWithMachineCount(0), allowing the test’s
intentionally degraded pool to be polled until its machine count reaches zero
before deleteMachineConfigPool proceeds.
🧹 Nitpick comments (1)
test/extended/node/runc_upgrade_cases.go (1)

593-598: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove the unused createRuncURLGuardPool helper.

The URL-guard test creates the pool via createOSImageURLUpgradeMCP and the CRC via createRuncGuardCRC directly, and there are no other calls to this helper. Keeping unused helpers makes the test harder to follow.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/node/runc_upgrade_cases.go` around lines 593 - 598, Remove the
unused createRuncURLGuardPool helper. Keep the URL-guard test flow using
createOSImageURLUpgradeMCP and createRuncGuardCRC directly, since there are no
callers that require the wrapper.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@test/extended/node/runc_upgrade_cases.go`:
- Around line 298-316: Update the cleanup drain wait in the
nodeName/testPoolName block to call WaitForMCP with WaitMCPAllowDegraded()
alongside WaitMCPWithMachineCount(0), allowing the test’s intentionally degraded
pool to be polled until its machine count reaches zero before
deleteMachineConfigPool proceeds.

---

Nitpick comments:
In `@test/extended/node/runc_upgrade_cases.go`:
- Around line 593-598: Remove the unused createRuncURLGuardPool helper. Keep the
URL-guard test flow using createOSImageURLUpgradeMCP and createRuncGuardCRC
directly, since there are no callers that require the wrapper.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 6cb03e40-faa0-4c10-ab7c-2c39016aa8d6

📥 Commits

Reviewing files that changed from the base of the PR and between 621e171 and d0144a6.

📒 Files selected for processing (2)
  • test/extended/node/runc_upgrade_cases.go
  • test/extended/node/runc_upgrade_cases.md

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jul 27, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@asahay19: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-ovn-upgrade d0144a6 link true /test e2e-gcp-ovn-upgrade
ci/prow/e2e-gcp-ovn d0144a6 link true /test e2e-gcp-ovn
ci/prow/e2e-aws-ovn-fips d0144a6 link true /test e2e-aws-ovn-fips

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-trt

openshift-trt Bot commented Jul 27, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: d0144a6

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volume-modify [FeatureGate:VolumeAttributesClass] [Feature:VolumeAttributesClass] should recover from invalid target VAC by updating PVC to new valid VAC" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] volume-modify [FeatureGate:VolumeAttributesClass] [Feature:VolumeAttributesClass] should recover from invalid target VAC by updating PVC to new valid VAC" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-gcp-csi Medium - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-gcp-csi Medium - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-gcp-csi Medium - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" is a new test, and was only seen in one job.

New tests seen in this PR at sha: d0144a6

  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volume-modify [FeatureGate:VolumeAttributesClass] [Feature:VolumeAttributesClass] should recover from invalid target VAC by updating PVC to new valid VAC" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] volume-modify [FeatureGate:VolumeAttributesClass] [Feature:VolumeAttributesClass] should recover from invalid target VAC by updating PVC to new valid VAC" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" [Total: 1, Pass: 1, Fail: 0, Flake: 0]

degradedPoolUpgradeableMessage = "One or more machine config pools are degraded"

// MCO rejects MachineConfig osImageURL override when MCP spec.osImageStream is also set.
osImageURLStreamConflictMessage = "cannot override MachineConfig osImageURL and set MachineConfigPool spec.osImageStream.name simultaneously"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't need to check this, just checking osImageURLRuncRHEL10GuardMessage is enough

Comment on lines +232 to +239
useRHEL9BaselineMC := clusterDefaultStream == streamRHEL10
if useRHEL9BaselineMC {
g.By("Pinning pool to RHCOS 9 via osImageURL because cluster OSImageStream default is rhel-10")
rhel9Image, err := osImageFromStream(ctx, mcClient, streamRHEL9)
o.Expect(err).NotTo(o.HaveOccurred())
o.Expect(createPoolOSImageURLMachineConfig(ctx, mcClient, runcURLGuardRHEL9MCName, runcRHCOS10URLGuardPool, rhel9Image)).To(o.Succeed())
cleanupRHEL9URLMC = true
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get why we need this. Can you explain?
Why don't we use just one mcp?

Comment on lines +808 to +809
return strings.Contains(message, osImageURLRuncRHEL10GuardMessage) &&
strings.Contains(strings.ToLower(message), "runc")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is redundant. osImageURLRuncRHEL10GuardMessage has "runc"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants