OCPEDGE-2827: tolerate one NotReady CP node in EnsureNodesReady for degraded TNF - #31442
OCPEDGE-2827: tolerate one NotReady CP node in EnsureNodesReady for degraded TNF#31442Neilhamza wants to merge 1 commit into
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
openshift-ci-robot
added
the
jira/valid-reference
|
@Neilhamza: This pull request references OCPEDGE-2827 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
Walkthrough
ChangesNode readiness handling
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
❌ Failed checks (1 error)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Neilhamza The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/extended/node/node_utils.go`:
- Around line 1006-1009: Update isControlPlaneNode to return (bool, error),
preserving the classification result while propagating failures from the
Nodes().Get lookup instead of converting them to false. Update EnsureNodesReady
and all callers to handle and assert the returned error so API lookup failures
are reported distinctly from a NotReady node.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 573730f8-9b15-4f19-ae06-966c3ffb9ad1
📒 Files selected for processing (1)
test/extended/node/node_utils.go
| func isControlPlaneNode(ctx context.Context, oc *exutil.CLI, nodeName string) bool { | ||
| node, err := oc.AdminKubeClient().CoreV1().Nodes().Get(ctx, nodeName, metav1.GetOptions{}) | ||
| if err != nil { | ||
| return false |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Propagate the node lookup error.
Line [1007] can fail, but Line [1008] silently converts that failure into false. A transient API error will then fall through to the readiness assertion and be reported as a NotReady node instead of an inability to classify it. Return (bool, error) and propagate/assert the error from EnsureNodesReady, or reuse the node returned by the initial list.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/extended/node/node_utils.go` around lines 1006 - 1009, Update
isControlPlaneNode to return (bool, error), preserving the classification result
while propagating failures from the Nodes().Get lookup instead of converting
them to false. Update EnsureNodesReady and all callers to handle and assert the
returned error so API lookup failures are reported distinctly from a NotReady
node.
Source: Path instructions
5df8d5b to
37eeb67
Compare
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/8e038bb0-8aa5-11f1-9537-77918f9bbb03-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/04a6a3b0-8aa6-11f1-9fe7-da6c8b8c8de2-0 |
|
Scheduling required tests: |
|
/retest |
|
/payload-aggregate periodic-ci-openshift-release-main-nightly-4.23-e2e-metal-ovn-two-node-fencing-degraded 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f4990ee0-8b1d-11f1-9b3a-26d65f39e96e-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn 5 |
|
/retest |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fa2112e0-8b1d-11f1-8f1b-d09c5195f877-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn 5 |
|
@Neilhamza: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9d156510-8b22-11f1-9ddd-15883f653acd-0 |
…egraded TNF EnsureNodesReady checks that all nodes are Ready before running tests. On intentionally degraded Two Node Fencing clusters (one master shut down), this blocks every test suite that uses the check (13 files). When DEGRADED_NODE=true (sets exutil.ClusterDegraded) and the cluster topology is DualReplica (TNF), tolerate exactly one NotReady node if it is a control-plane node. This follows the same pattern established in PR#30649 for cert collection on degraded clusters. The check remains strict for: - Non-degraded clusters (no env var) - Non-TNF topologies - More than one NotReady node - NotReady worker nodes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
37eeb67 to
9713c89
Compare
|
@coderabbitai perform full review |
|
@Neilhamza: This pull request references OCPEDGE-2827 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "5.0." or "openshift-5.0.", but it targets "openshift-4.23" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Action performedFull review triggered. |
2 participants
Summary
On intentionally degraded Two Node Fencing (TNF) clusters where one master is shut down,
EnsureNodesReadyinBeforeEachhooks blocks all 13 test files undertest/extended/node/because it fails on any NotReady node with no bypass.This change adds a targeted carve-out: when
DEGRADED_NODE=true(already wired via PR #30649) AND the cluster topology isDualReplica(TNF), tolerate exactly one NotReady control-plane node. This follows the sameClusterDegraded && IsTwoNodeFencing()pattern used intest/extended/operators/certs.go.The check remains strict for non-degraded clusters, non-TNF topologies, multiple NotReady nodes, and NotReady worker nodes.
Why not skip the tests?
Adding tests to
TEST_SKIPSrequires maintenance as new tests are added to the 13 affected files. Since all 13 files funnel throughEnsureNodesReady, fixing the single choke point automatically covers future tests.Failing CI lanes
periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-degradedperiodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-dualstack-degradedJira: https://redhat.atlassian.net/browse/OCPEDGE-2827
Summary by CodeRabbit