Repository navigation
Description
Delivery bundle: D: HA and secure deployment. See #395 and #368 for the four-PR stack.
Parent: #392
Roadmap: #368
Plan item: PR-06
Priority: P1
Dependencies: None
Kubernetes support: Rolling upstream-maintained minors; use the oldest/newest boundaries pinned in config/kubernetes-versions.env. See #368 for the current policy. Unrun jobs are not passed validation.
Problem Statement
Evidence: controllers/alertmanager_controller.go:40-51 starts a leader-only plain runnable, ignores listener errors, and omits context shutdown. The alert Service selects all operator replicas. Deduplication includes process-local state in internal/alertmanager/currentalert/current_alerts.go.
Proposed Solution
Files: controllers/alertmanager_controller.go, relevant internal/alertmanager/ handlers/state/tests, charts/kafka-operator/templates/alertmanager-service.yaml, charts/kafka-operator/templates/operator-deployment-with-webhook.yaml, and values/schema.
Interface contract: A managed server returns bind/start errors and shuts down with context. Preserve one effective mutation/deduplication domain. Routing must not send alerts to ready standbys without a listener.
- Add bind-conflict and shutdown tests using an ephemeral listener; assert returned errors and prompt closure.
- Select the HA model before coding: leader-aware forwarding/routing, or all-replica ingress handing work into a durable leader-processed queue. Do not merely turn off leader election around the current process-local mutation handler.
- Use
manager.Serveror an equivalently lifecycle-correct runnable with bounded shutdown, and implement/test the selected routing boundary. - Keep webhook availability on standbys; do not fix alerts by making all nonleaders globally unready.
- Validate focused alert/controller tests and a Kind two-replica test delivering duplicate alerts continuously through the Service during leader termination.
Done when: Alerts reach a serving endpoint through failover without duplicate effective scaling actions.
Delivery: Four consolidated PRs, approved November 16, 2026: A -> B -> C; D branches from A. Foundation is #395. The 22 PR-XX identifiers are work-item IDs, not separate PRs. Preserve every work-item acceptance criterion and native dependency. No gated experiment is automatically enabled, and no issue is complete merely because one PR references it. D must satisfy its native #375 gate, including lifecycle-domain signals assigned to B; branching from A does not waive that merge prerequisite. Write focused regression tests first; record exact commands/results and run make check plus this issue's real-cluster/measurement gates.
Alternatives Considered
Twenty-two individual PRs add unnecessary integration overhead, while one omnibus PR obscures correctness and rollout boundaries. Deliver this scope in its approved dependency-aligned bundle. Do not expand into deferred consistency, storage, or concurrency changes without their own decision gates.
Additional Context
Baseline: 95138e0d; detailed roadmap scope: #368, section PR-06. The roadmap #368 and its native sub-issues are the planning source of truth. No planning document is committed in the final PR diffs. Re-read affected source and overlapping work before implementation. Historical baseline test passes are not validation of this change.
Activity
amuraru commentedon Nov 17, 2026
Alert bind/shutdown, HA routing, and effective mutation-domain safety are consolidated into #392 under #368. Coordinate with existing #365; this closure does not claim the serving fix shipped.