Skip to content

[PR-06] Managed Alert HTTP Lifecycle and Correct HA Routing #374

Closed as not planned

Description

@amuraru

Delivery bundle: D: HA and secure deployment. See #395 and #368 for the four-PR stack.

Parent: #392
Roadmap: #368
Plan item: PR-06
Priority: P1
Dependencies: None

Kubernetes support: Rolling upstream-maintained minors; use the oldest/newest boundaries pinned in config/kubernetes-versions.env. See #368 for the current policy. Unrun jobs are not passed validation.

Problem Statement

Evidence: controllers/alertmanager_controller.go:40-51 starts a leader-only plain runnable, ignores listener errors, and omits context shutdown. The alert Service selects all operator replicas. Deduplication includes process-local state in internal/alertmanager/currentalert/current_alerts.go.

Proposed Solution

Files: controllers/alertmanager_controller.go, relevant internal/alertmanager/ handlers/state/tests, charts/kafka-operator/templates/alertmanager-service.yaml, charts/kafka-operator/templates/operator-deployment-with-webhook.yaml, and values/schema.

Interface contract: A managed server returns bind/start errors and shuts down with context. Preserve one effective mutation/deduplication domain. Routing must not send alerts to ready standbys without a listener.

  • Add bind-conflict and shutdown tests using an ephemeral listener; assert returned errors and prompt closure.
  • Select the HA model before coding: leader-aware forwarding/routing, or all-replica ingress handing work into a durable leader-processed queue. Do not merely turn off leader election around the current process-local mutation handler.
  • Use manager.Server or an equivalently lifecycle-correct runnable with bounded shutdown, and implement/test the selected routing boundary.
  • Keep webhook availability on standbys; do not fix alerts by making all nonleaders globally unready.
  • Validate focused alert/controller tests and a Kind two-replica test delivering duplicate alerts continuously through the Service during leader termination.

Done when: Alerts reach a serving endpoint through failover without duplicate effective scaling actions.

Delivery: Four consolidated PRs, approved November 16, 2026: A -> B -> C; D branches from A. Foundation is #395. The 22 PR-XX identifiers are work-item IDs, not separate PRs. Preserve every work-item acceptance criterion and native dependency. No gated experiment is automatically enabled, and no issue is complete merely because one PR references it. D must satisfy its native #375 gate, including lifecycle-domain signals assigned to B; branching from A does not waive that merge prerequisite. Write focused regression tests first; record exact commands/results and run make check plus this issue's real-cluster/measurement gates.

Alternatives Considered

Twenty-two individual PRs add unnecessary integration overhead, while one omnibus PR obscures correctness and rollout boundaries. Deliver this scope in its approved dependency-aligned bundle. Do not expand into deferred consistency, storage, or concurrency changes without their own decision gates.

Additional Context

Baseline: 95138e0d; detailed roadmap scope: #368, section PR-06. The roadmap #368 and its native sub-issues are the planning source of truth. No planning document is committed in the final PR diffs. Re-read affected source and overlapping work before implementation. Historical baseline test passes are not validation of this change.

Activity

  1. added theissue type on Nov 16, 2026
  2. amuraru commented on Nov 17, 2026

    @amuraru
    Author

    Alert bind/shutdown, HA routing, and effective mutation-domain safety are consolidated into #392 under #368. Coordinate with existing #365; this closure does not claim the serving fix shipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions