Skip to content

[PR-22] Safe Concurrency Across Independent KafkaClusters #390

Closed as not planned
Closed as not planned

Description

@amuraru

Delivery bundle: C: Convergence, APIs, and gated scalability. See #395 and #368 for the four-PR stack.

Parent: #393
Roadmap: #368
Plan item: PR-22
Priority: P3
Dependencies: PR-01, PR-03, PR-07, PR-10

Kubernetes support: Rolling upstream-maintained minors; use the oldest/newest boundaries pinned in config/kubernetes-versions.env. See #368 for the current policy. Unrun jobs are not passed validation.

Problem Statement

Evidence: CruiseControlOperationReconciler stores mutable scaler state on the reconciler (controllers/cruisecontroloperation_controller.go:72) and schedules across multiple CRs for one external cluster.

Proposed Solution

Files: Cruise Control operation/task controllers, scale adapters, manager/controller options, relevant tests and HA/restart scenarios.

Interface contract: The scaler is reconcile-local. Serialize scheduling by KafkaCluster identity, not merely operation namespace/name; different operation keys for the same cluster must not submit conflicting actions. Persisted state and PR-01 recovery remain authoritative across restart/failover.

  • Before enabling concurrency, define the per-cluster serialization design and how existing operation requests map to it. Do not rely solely on an in-memory lock for restart safety.
  • Add two-cluster parallelism and same-cluster/different-operation-key tests; assert no scaler cross-contamination or simultaneous incompatible submissions.
  • Add submit-then-crash/restart and leader-change tests using available Cruise Control task recovery. Document any residual external acknowledgment ambiguity.
  • Run race tests on affected controllers/adapters. Raise concurrency only for independent clusters after the tests pass.
  • Validate go test -race ./controllers ./pkg/scale, controller envtest, Kind HA/restart tests, and PR-07 throughput/fairness measurements.

Done when: Independent clusters can progress concurrently without weakening single-cluster execution safety.

Delivery: Four consolidated PRs, approved November 16, 2026: A -> B -> C; D branches from A. Foundation is #395. The 22 PR-XX identifiers are work-item IDs, not separate PRs. Preserve every work-item acceptance criterion and native dependency. No gated experiment is automatically enabled, and no issue is complete merely because one PR references it. D must satisfy its native #375 gate, including lifecycle-domain signals assigned to B; branching from A does not waive that merge prerequisite. Write focused regression tests first; record exact commands/results and run make check plus this issue's real-cluster/measurement gates.

Alternatives Considered

Twenty-two individual PRs add unnecessary integration overhead, while one omnibus PR obscures correctness and rollout boundaries. Deliver this scope in its approved dependency-aligned bundle. Do not expand into deferred consistency, storage, or concurrency changes without their own decision gates.

Additional Context

Baseline: 95138e0d; detailed roadmap scope: #368, section PR-22. The roadmap #368 and its native sub-issues are the planning source of truth. No planning document is committed in the final PR diffs. Re-read affected source and overlapping work before implementation. Historical baseline test passes are not validation of this change.

Related historical context: #301 (closed) covers Cruise Control restart races. Preserve its resolution and avoid duplicating that narrower fix.

Activity

  1. added theissue type on Nov 16, 2026
  2. amuraru commented on Nov 17, 2026

    @amuraru
    Author

    Concurrency increases are removed from active scope under #368: no demonstrated throughput bottleneck justifies new serialization/recovery machinery. Existing execution safety stays in #391; no worker increase claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions