Skip to content

fix(controller): honor cancellation during reconciliation - #117

Merged
amuraru merged 1 commit into
masterfrom
fix/controller-cancellation
Nov 18, 2026
Merged

amuraru merged 1 commit into
masterfrom
fix/controller-cancellation

Conversation

@amuraru

@amuraru amuraru commented Nov 18, 2026

Copy link
Copy Markdown

Change log description

  • Propagate reconciliation cancellation through private helpers and Kubernetes requests without changing worker concurrency.
  • Give each ZooKeeper operation scope a fresh, owned session with a 30-second operation deadline, bounded by earlier parent deadlines and separate from the existing five-second session timeout.
  • Interrupt stalled handshakes and requests by closing only the owned session/socket, and wait for cancellation cleanup rather than leaving detached operation workers.

Purpose of the change

Implements PR-11 under #101. This does not close the broader workstream.

Health probes remain separate in #116. No storage, write-convergence, Helm retry, or worker-concurrency changes are included.

What the code does

Threads context.Context through controller helpers and every Kubernetes request. Request logging uses ctrl.LoggerFrom(ctx) instead of assigning the shared reconciler logger.

Changes all ZookeeperClient operations except Close() to accept context, and replaces the shared client with NewZkClient func() zk.ZookeeperClient. Updates controller mocks, the operator entry point, YAML-exporter constructor, e2e manager constructor, and tagged integration callers. Each metadata initialization or scale-metadata scope creates and closes its own session.

Adds a named zk.OperationTimeout of 30 seconds. DNS lookup, session readiness, and node operations honor cancellation; the controller shares one deadline across each metadata scope. The session timeout remains five seconds. A context-aware dialer retains the owned transport so cancellation can interrupt an incomplete handshake, which upstream Close() alone does not interrupt promptly. Cleanup is idempotent and waits for the upstream session loop and cancellation callback to finish.

Regression tests cover pre-canceled work without new mutations, blocked Kubernetes and ZooKeeper work, a real TCP peer that never completes session readiness, the earlier-parent and default 30-second deadline boundaries, stopping child-node creation after cancellation, and independently keyed reconciliations with isolated session ownership.

Cancellation does not roll back a mutation already accepted by a server. Upstream Close() can add up to its one-second close-request wait while cleanup completes. The only expected overlap with #116 is the nearby reconciler constructor in main.go; this PR changes its client factory, not manager health configuration.

How to verify it

Red before production changes: go test ./controllers ./pkg/zk -run '^(TestCanceledReconcileStopsClientWork|TestZookeeperOperationDeadline)$' -count=1 failed as intended: a pre-canceled reconcile made one update, a blocked helper ignored cancellation, and a connected-but-never-ready ZooKeeper session exceeded its parent deadline.

Passed locally on the final implementation:

go test -race ./controllers ./pkg/zk -run '^(TestCanceledReconcileStopsClientWork|TestZookeeperOperationDeadline|TestZookeeperCancellationStopsBlockedOperations|TestZookeeperCanceledCreateDoesNotCreateChild|TestZookeeperPreCanceledConnect)$' -count=1
go test -race ./controllers -run '^TestCanceledReconcileStopsClientWork$' -count=10
make test
make check
go vet ./...
git diff --check
golangci-lint run --new-from-rev=4b580e5daecb97df1b153e2fd8a98f7ad187cc8c ./...
go test -tags=zkintegration ./pkg/zk -run '^$'
go test -tags=envtest ./controllers -run '^$'
go test ./test/e2e/... -run '^$'
go test -race -tags=zkintegration ./pkg/zk -run '^TestMetadataCreateRetryPreservesExistingData$' -count=1 -v
KUBEBUILDER_ASSETS="$(setup-envtest use 1.34.1 -p path --installed-only)" go test -tags=envtest ./controllers -run '^TestEnvtest' -count=1 -v
docker build --build-arg GO_VERSION="$(awk '/^go /{print $2}' go.mod | cut -d. -f1,2)" -t zookeeper-operator:pr11-cancellation-local .

make test includes race detection; the changed-code linter reported 0 issues, matching CI's only-new-issues policy. The unrestricted golangci-lint run ./... still reports existing legacy findings outside this change.

Passed manual shutdown e2e on disposable Kubernetes 1.34.3: deployed the image built from this worktree using the existing e2e manifests, and routed a scale operation to an owned TCP server that accepted connections but never completed a ZooKeeper handshake. Scaled the operator Deployment to zero during that blocked operation (kubectl scale deployment/zookeeper-operator --replicas=0, followed by kubectl wait --for=delete pod/<operator-pod> --timeout=10s). The blocked reconciliation returned context canceled, the manager logged All workers finished, the peer observed socket closure, and shutdown completed in 0.53 seconds. The StatefulSet remained at three replicas rather than applying the requested five. The disposable cluster was removed.

The full existing Kubernetes e2e suite was not run locally; the e2e label requests it in GitHub Actions. No Java toolchain commands were needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@amuraru amuraru added the e2e label Nov 18, 2026
@amuraru
amuraru merged commit b05c2a8 into master Nov 18, 2026
13 of 15 checks passed
@amuraru
amuraru deleted the fix/controller-cancellation branch November 18, 2026 18:44
amuraru added a commit that referenced this pull request Nov 18, 2026
Keep the health lifecycle test compatible with the reconciler-local ZooKeeper client factory introduced by #117.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
amuraru added a commit that referenced this pull request Nov 18, 2026
Update the health-test fixture to use the reconciler factory introduced by #117.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
amuraru added a commit that referenced this pull request Nov 18, 2026
…112)

* fix(controller): skip unchanged ConfigMap and ServiceAccount writes

Guard owned data before existing updates while preserving conflicts and subsequent StatefulSet reconciliation. Cover no-ops, changes, removals and real API-server conflict retries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(operator): adapt health envtest to client factory

Keep the health lifecycle test compatible with the reconciler-local ZooKeeper client factory introduced by #117.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant