Skip to content

[PR-07] REST-Client and Operator Observability Baseline #375

Closed as not planned
Closed as not planned

Description

@amuraru

Implementation status: REST instrumentation is in #395; lifecycle-domain metrics/Events remain assigned to draft #397. Workload measurements are not complete.

Delivery bundle: A: Foundation. See #395 and #368 for the four-PR stack.

Parent: #393
Roadmap: #368
Plan item: PR-07
Priority: P2
Dependencies: None

Kubernetes support: Rolling upstream-maintained minors; use the oldest/newest boundaries pinned in config/kubernetes-versions.env. See #368 for the current policy. Unrun jobs are not passed validation.

Problem Statement

Evidence: main.go does not opt into the additional v0.25 REST-client metrics. Core controller/resource code does not register operator-specific collectors or emit transition Events.

Proposed Solution

Files: main.go, existing logging/resource/controller call sites, existing monitoring documentation/chart integration, and a small dedicated metrics source/test file if no suitable package exists.

Interface contract: Use controller-runtime's registry and RegisterRESTClientMetrics for request latency, retry, size, and rate-limiter latency as justified. Keep labels bounded; never label by task UUID or arbitrary error text.

  • Add collector tests asserting the selected REST metric families are registered and representative operator counters/gauges change correctly.
  • Add transition Event tests for significant lifecycle failures/progress, not one Event per poll.
  • Define operator metrics for pending finalization, external task outcomes/progress, and write conflicts without replacing existing controller-runtime metrics.
  • Document dashboard queries and alert/runbook meaning; baseline queue delay, reconcile duration, API requests/bytes, RSS, and startup/failover time on a reproducible workload.
  • Validate package tests plus a scrape test. Record workload cardinalities and results; do not invent performance targets or claimed savings.

Done when: Subsequent optimization PRs have repeatable before/after measurements and useful failure signals.

Delivery: Four consolidated PRs, approved November 16, 2026: A -> B -> C; D branches from A. Foundation is #395. The 22 PR-XX identifiers are work-item IDs, not separate PRs. Preserve every work-item acceptance criterion and native dependency. No gated experiment is automatically enabled, and no issue is complete merely because one PR references it. D must satisfy its native #375 gate, including lifecycle-domain signals assigned to B; branching from A does not waive that merge prerequisite. Write focused regression tests first; record exact commands/results and run make check plus this issue's real-cluster/measurement gates.

Alternatives Considered

Twenty-two individual PRs add unnecessary integration overhead, while one omnibus PR obscures correctness and rollout boundaries. Deliver this scope in its approved dependency-aligned bundle. Do not expand into deferred consistency, storage, or concurrency changes without their own decision gates.

Additional Context

Baseline: 95138e0d; detailed roadmap scope: #368, section PR-07. The roadmap #368 and its native sub-issues are the planning source of truth. No planning document is committed in the final PR diffs. Re-read affected source and overlapping work before implementation. Historical baseline test passes are not validation of this change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions