Skip to content

feat(operator): expose health and readiness probes - #116

Merged
amuraru merged 4 commits into
masterfrom
feat/operator-health-probes
Nov 18, 2026
Merged

amuraru merged 4 commits into
masterfrom
feat/operator-health-probes

Conversation

@amuraru

@amuraru amuraru commented Nov 18, 2026 •

Copy link
Copy Markdown

Change log description

  • Add controller-runtime health and readiness endpoints on --health-probe-bind-address (default :8081), without changing metrics binding.
  • Add named-port HTTP probes to Helm and the Kustomize base; e2e overlays inherit them without a public Service.
  • Test real cache initialization and shutdown with no ensembles; document operator readiness separately from ensemble readiness.

Purpose of the change

Implements PR-12 under #101. This does not close the broader workstream or implement its deferred items.

What the code does

Uses the existing manager health listener and health-check registration APIs. /healthz is a process-liveness ping and does not depend on ZooKeeper availability.

SetupWithManager declares the primary resource and owned resources once, then uses those same objects for both watch registration and eager informer initialization. There is no separate resource list in main.go to keep synchronized: adding an owned resource to the controller's declaration configures both behaviors. Controller-runtime otherwise registers those sources lazily, allowing its initial cache-sync phase to finish with an empty cache. /readyz becomes successful only after the actual watched caches initialize and fails when shutdown begins, including while controller work drains. Health-check setup itself contains no resource-type knowledge.

Initialized standby managers may be ready without holding the leader lease. Requiring leadership would block the default single-replica rolling update from replacing its old leader. Neither endpoint claims that managed ensembles are ready.

Both deployment paths expose named container port health:8081, with /healthz every 10 seconds and /readyz every 5 seconds, each with failure threshold 3. Metrics configuration and ZooKeeper ensemble status semantics are unchanged.

The existing rendered-contract suite covers Helm, default Kustomize, and the inheriting e2e overlay. The new in-process manager test uses Kubernetes 1.34.1 envtest with no ensembles: it blocks the real initial informer request, verifies readiness after the empty caches synchronize, and holds shutdown open while checking HTTP responses. There is no ZooKeeper failure mock, scaling scenario, or cluster/StatefulSet fixture. The existing make test-envtest target now includes the root package and selects opt-in files through the envtest build tag, without a test-name filter, so CI and releases exercise new tagged tests regardless of their names. Ordinary tests in those packages also run because Go build tags are additive.

How to verify it

Passed locally on this branch:

  • make check — passed formatting and license checks.
  • make test — passed the full ordinary Go suite with race detection; rerun immediately before publication.
  • go test -tags=envtest . ./controllers -count=1 -race -v — passed with existing Kubernetes 1.34.1 assets selected through KUBEBUILDER_ASSETS.
  • go test -tags=manifests ./test/manifests -race -count=1 — passed all rendered contracts.
  • helm lint --strict charts/zookeeper-operator — passed.
  • golangci-lint run --new-from-rev=4b580e5daecb97df1b153e2fd8a98f7ad187cc8c ./... — passed, zero new issues.
  • golangci-lint run --build-tags=envtest,manifests --new-from-rev=4b580e5daecb97df1b153e2fd8a98f7ad187cc8c . ./controllers ./test/manifests — passed, zero new issues.
  • go vet ./... — passed.
  • python3 -B -m unittest discover -s scripts -p 'test_*.py' — passed, 35 tests.
  • git diff --check — passed.

Actual Kubernetes validation used a disposable kind v1.34.3 cluster with the repository-pinned node image:

kindest/node:v1.34.3@sha256:08497ee19eace7b4b5348db5c6a1591d7752b164530a36f855cb0f2bdcbadd48

The final source-built linux/arm64 operator image, including the colocated watch/informer setup, passed startup and kubectl rollout restart for both Helm and the Kustomize/e2e-base deployment. kubectl rollout status deployment/zookeeper-operator --timeout=90s passed in both test namespaces. Both operator Pods were 1/1 Ready with zero restarts. Real Pod-proxy requests to /healthz and /readyz returned ok with zero ZooKeeperCluster objects in the cluster. The test cluster and task-specific image tag were removed afterward.

Not run: the full ZooKeeper ensemble e2e suite or independent code review. The make test-envtest installation wrapper was not rerun; its updated Go test selection was run directly with cached 1.34.1 assets and additional race detection. No dependencies were added.

amuraru and others added 4 commits November 18, 2026 17:57
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove test-name filtering so newly added tagged tests run automatically.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove ZooKeeper outage mocks and scaling fixtures; retain real cache startup, zero-ensemble readiness, and draining shutdown regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Derive controller watches and eager informer setup from the same primary and owned resource declarations. Keep health checks free of resource-type knowledge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@amuraru amuraru added the e2e label Nov 18, 2026
@amuraru
amuraru merged commit eead081 into master Nov 18, 2026
15 checks passed
@amuraru
amuraru deleted the feat/operator-health-probes branch November 18, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant