Repository navigation
Description
Delivery bundle: C: Convergence, APIs, and gated scalability. See #395 and #368 for the four-PR stack.
Parent: #393
Roadmap: #368
Plan item: PR-21
Priority: P2
Dependencies: PR-07, PR-11
Kubernetes support: Rolling upstream-maintained minors; use the oldest/newest boundaries pinned in config/kubernetes-versions.env. See #368 for the current policy. Unrun jobs are not passed validation.
Problem Statement
Evidence: Successful topic/user reconciles return an empty result; Kubernetes resync is not a substitute for timely external drift checks. Other paths use fixed short polling intervals for progress/readiness.
Proposed Solution
Files: KafkaTopic/User and Cruise Control controllers, common requeue helpers, relevant external clients, configuration/docs, and polling tests.
Interface contract: Use watches for Kubernetes dependencies delivered by PR-11. Poll external Kafka/Cruise Control state only where needed, using bounded intervals, staggering, and a documented aggregate request/connection budget.
- Add tests for external drift without Kubernetes events, long-running external progress, many simultaneously created topics, and dependency recovery.
- Distinguish terminal invalid configuration, expected progress, and retryable failures. Use
TerminalErroronly when a reliable relevant event can trigger recovery; retain retry paths for external availability. - Choose cadence from measured detection/throughput requirements and expose configuration only where users need it. Keep TTL cleanup time-driven.
- Do not shorten global cache
SyncPeriodor open a new expensive Kafka admin connection for every topic at an aggressive default cadence. - Validate focused controller tests and compare detection delay, connection churn, API volume, and queue fairness on the PR-07 workload.
Done when: External drift is detected within the documented bound without polling Kubernetes dependencies unnecessarily.
Delivery: Four consolidated PRs, approved November 16, 2026: A -> B -> C; D branches from A. Foundation is #395. The 22 PR-XX identifiers are work-item IDs, not separate PRs. Preserve every work-item acceptance criterion and native dependency. No gated experiment is automatically enabled, and no issue is complete merely because one PR references it. D must satisfy its native #375 gate, including lifecycle-domain signals assigned to B; branching from A does not waive that merge prerequisite. Write focused regression tests first; record exact commands/results and run make check plus this issue's real-cluster/measurement gates.
Alternatives Considered
Twenty-two individual PRs add unnecessary integration overhead, while one omnibus PR obscures correctness and rollout boundaries. Deliver this scope in its approved dependency-aligned bundle. Do not expand into deferred consistency, storage, or concurrency changes without their own decision gates.
Additional Context
Baseline: 95138e0d; detailed roadmap scope: #368, section PR-21. The roadmap #368 and its native sub-issues are the planning source of truth. No planning document is committed in the final PR diffs. Re-read affected source and overlapping work before implementation. Historical baseline test passes are not validation of this change.
Activity
amuraru commentedon Nov 17, 2026
New external-drift polling is removed from active scope under #368: no agreed detection contract or measured budget currently justifies it. Required deletion/progress observation remains with its concrete fix; no drift guarantee added.