Skip to content

MON-4439: monitoring: ensure platform prometheus targets are discovered via EndpointSlices - #31431

Open
machine424 wants to merge 1 commit into
openshift:mainfrom
machine424:endpoo
Open

MON-4439: monitoring: ensure platform prometheus targets are discovered via EndpointSlices#31431
machine424 wants to merge 1 commit into
openshift:mainfrom
machine424:endpoo

Conversation

@machine424

@machine424 machine424 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

from #30611

Summary by CodeRabbit

  • Tests
    • Expanded Prometheus target validation coverage.
    • Added verification that targets are discovered through EndpointSlice without legacy endpoint discovery metadata.
    • Improved target fetching and scrape-state verification for more consistent test results.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 27, 2026
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Walkthrough

Prometheus target retrieval is centralized in a shared helper, and target data now exposes discovered labels. A new test detects targets discovered through Kubernetes Endpoints instead of EndpointSlices.

Changes

Prometheus target validation

Layer / File(s) Summary
Centralized target fetching
test/extended/prometheus/prometheus.go
The shared helper fetches and unmarshals Prometheus targets, validates active targets, and replaces duplicated authorization-test retrieval paths.
EndpointSlice discovery validation
test/extended/prometheus/prometheus.go
The target model includes discovered labels, and a test reports jobs carrying Kubernetes Endpoints discovery metadata.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Test Structure And Quality ⚠️ Warning fetchPrometheusTargets ignores its ctx and helper.GetURLWithToken polls on context.Background(), so canceled specs can still hang; the new test also has bare expectations. Thread the spec ctx through fetchPrometheusTargets/GetURLWithToken (or its polling loop) and add failure messages to the new Expect(err) checks.
Microshift Test Compatibility ⚠️ Warning Added Ginkgo test queries Prometheus in openshift-monitoring, which MicroShift doesn’t provide, and it lacks any MicroShift skip/tag guard. Skip this Describe/It on MicroShift with [Skipped:MicroShift] or an exutil.IsMicroShiftCluster() check before touching Prometheus/openshift-monitoring.
✅ Passed checks (13 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All Ginkgo titles in the changed file are static strings; the new test name contains no generated or runtime values.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The new endpointslice Prometheus test only inspects target discovery metadata and does not require multiple nodes, HA behavior, or any SNO skip guard.
Topology-Aware Scheduling Compatibility ✅ Passed Only test/extended/prometheus/prometheus.go changed, adding Prometheus target validation/helpers; no manifests, controllers, node selectors, affinity, or replica logic were introduced.
Ote Binary Stdout Contract ✅ Passed PASS: The only stdout write is fmt.Printf inside a helper invoked from an It block; no main/init/TestMain/BeforeSuite/setup code writes to stdout.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The added Prometheus test uses cluster route/service DNS only; no IPv4 literals, IP-family assumptions, or public internet connectivity were found.
No-Weak-Crypto ✅ Passed The changed Prometheus test code only fetches/inspects targets; it adds no weak ciphers, custom crypto, or secret/token comparisons.
Container-Privileges ✅ Passed HEAD changes only test/extended/prometheus/prometheus.go; no K8s manifests or privilege-related fields were added.
No-Sensitive-Data-In-Logs ✅ Passed No new log statements expose secrets/PII; the added flake only lists component jobs, and the helper does not log the bearer token.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: verifying Prometheus target discovery via EndpointSlices.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from jan--f and slashpai July 27, 2026 08:00
@machine424

Copy link
Copy Markdown
Contributor Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: machine424

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 27, 2026
@machine424

machine424 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

/retitle MON-4439: monitoring: ensure platform prometheus targets are discovered via EndpointSlices

@openshift-ci openshift-ci Bot changed the title WIP: monitoring: ensure platform prometheus targets are discovered via EndpointSlices MON-4439: monitoring: ensure platform prometheus targets are discovered via EndpointSlices Jul 27, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 27, 2026

Copy link
Copy Markdown

@machine424: This pull request references MON-4439 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

from #30611

Summary by CodeRabbit

  • Tests
  • Expanded Prometheus target validation coverage.
  • Added verification that targets are discovered through EndpointSlice without legacy endpoint discovery metadata.
  • Improved target fetching and scrape-state verification for more consistent test results.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 27, 2026
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 27, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended/prometheus/prometheus.go`:
- Around line 1003-1004: Update fetchPrometheusTargets to pass its ctx to
helper.GetURLWithToken, then thread that context through the helper’s
retry/polling loop instead of using context.Background(). Ensure cancellation or
timeout terminates the request and retry loop promptly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 2b6c0f47-c94a-40c5-9868-30af2f91d558

📥 Commits

Reviewing files that changed from the base of the PR and between 6e168d7 and cb49ebe.

📒 Files selected for processing (1)
  • test/extended/prometheus/prometheus.go

Comment on lines +1003 to +1004
func fetchPrometheusTargets(ctx context.Context, prometheusURL, bearerToken string) (*prometheusTargets, error) {
contents, err := helper.GetURLWithToken(helper.MustJoinUrlPath(prometheusURL, "api/v1/targets"), bearerToken)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show the relevant function and nearby call sites.
sed -n '980,1045p' test/extended/prometheus/prometheus.go
printf '\n---- call sites ----\n'
rg -n "fetchPrometheusTargets\(" test/extended/prometheus/prometheus.go

printf '\n---- helper definition ----\n'
rg -n "func GetURLWithToken|GetURLWithToken\(" -S .

Repository: openshift/origin

Length of output: 3505


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the helper implementation once its location is known.
helper_file=$(rg -l "func GetURLWithToken" -S . | head -n 1 || true)
if [ -n "${helper_file}" ]; then
  echo "HELPER_FILE=$helper_file"
  sed -n '1,220p' "$helper_file"
fi

Repository: openshift/origin

Length of output: 8259


Thread ctx through the Prometheus target fetch
fetchPrometheusTargets drops its ctx, and helper.GetURLWithToken retries on context.Background() for up to a minute. A canceled or timed-out spec can still block in this request path; pass the spec context into the helper and its polling loop.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/prometheus/prometheus.go` around lines 1003 - 1004, Update
fetchPrometheusTargets to pass its ctx to helper.GetURLWithToken, then thread
that context through the helper’s retry/polling loop instead of using
context.Background(). Ensure cancellation or timeout terminates the request and
retry loop promptly.

Source: Path instructions

@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@machine424: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vsphere-ovn cb49ebe link true /test e2e-vsphere-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-trt

openshift-trt Bot commented Jul 27, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: cb49ebe

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volume-modify [FeatureGate:VolumeAttributesClass] [Feature:VolumeAttributesClass] should recover from invalid target VAC by updating PVC to new valid VAC" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] volume-modify [FeatureGate:VolumeAttributesClass] [Feature:VolumeAttributesClass] should recover from invalid target VAC by updating PVC to new valid VAC" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-aws-csi Medium - "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-gcp-csi Medium - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-gcp-csi Medium - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" is a new test, and was only seen in one job.
pull-ci-openshift-origin-main-e2e-gcp-csi Medium - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" is a new test, and was only seen in one job.

New tests seen in this PR at sha: cb49ebe

  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volume-modify [FeatureGate:VolumeAttributesClass] [Feature:VolumeAttributesClass] should recover from invalid target VAC by updating PVC to new valid VAC" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] volume-modify [FeatureGate:VolumeAttributesClass] [Feature:VolumeAttributesClass] should recover from invalid target VAC by updating PVC to new valid VAC" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" [Total: 1, Pass: 1, Fail: 0, Flake: 0]
  • "[sig-instrumentation] Prometheus [apigroup:image.openshift.io] when installed on the cluster should only discover targets via endpointslice [Suite:openshift/conformance/parallel]" [Total: 5, Pass: 5, Fail: 0, Flake: 0]

@machine424

Copy link
Copy Markdown
Contributor Author

/payload 5.0 ci blocking

@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@machine424: trigger 5 job(s) of type blocking for the ci release of OCP 5.0

  • periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade
  • periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-azure-ovn-upgrade
  • periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-upgrade
  • periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aks
  • periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0ca52800-89cc-11f1-9a5e-0e2cccf21c79-0

@machine424

Copy link
Copy Markdown
Contributor Author

/payload 5.0 nightly blocking

@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@machine424: trigger 14 job(s) of type blocking for the nightly release of OCP 5.0

  • periodic-ci-openshift-release-main-ci-5.0-e2e-aws-upgrade-ovn-single-node
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips
  • periodic-ci-openshift-release-main-ci-5.0-e2e-azure-ovn-upgrade
  • periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-gcp-ovn-rt-upgrade
  • periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-azure-aks-ovn-conformance
  • periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-serial-1of2
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-serial-2of2
  • periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview
  • periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview-serial-1of3
  • periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview-serial-2of3
  • periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-techpreview-serial-3of3
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv4
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0fe6e580-89cc-11f1-985e-9140e3791094-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants