Skip to content

fix(healthcheck): cold-start readiness gate for lazily-created, unprobed checkers - #13897

Closed
aanogueira wants to merge 1 commit into
apache:masterfrom
aanogueira:andre.nogueira/healthcheck-worker-status-reconcile
Closed

fix(healthcheck): cold-start readiness gate for lazily-created, unprobed checkers#13897
aanogueira wants to merge 1 commit into
apache:masterfrom
aanogueira:andre.nogueira/healthcheck-worker-status-reconcile

Conversation

@aanogueira

Copy link
Copy Markdown

Problem

A freshly created health-check target defaults to internal_health = healthy with zero probes (add_target's hardcoded is_healthy=true), so a pod that restarts while its backend is already unhealthy briefly routes real traffic to it until enough active probes correct the target's state. Separately, checker creation itself is entirely lazy -- seeded only by fetch_checker() on the live request path -- so an idle-but-critical upstream with no prior traffic would never get a checker built at all ahead of a readiness check.

There is currently no way to tell "healthy" (a real check passed) apart from "healthy" (the zero-probe default) from outside the checker, so nothing can gate readiness on "has this actually been checked yet."

Changes

  • apisix/healthcheck_manager.lua: adds two accessors for a readiness plugin to use --
    • ensure_checker(resource_path): proactively seeds a checker for a resource even with zero prior traffic, reusing the existing timer_create_checker construction path. Also resolves domain-name upstream nodes via parse_domain_in_up up front -- that resolution otherwise only happens on the live request path, so a checker built ahead of traffic would start probing under an unresolved identity and get silently rebuilt (wiping its probe count) the moment real traffic first resolved the domain.
    • is_resource_probed(resource_path): true only once every target of the resource's checker has had enough real active-check attempts for its state to have actually converged -- a single attempt is not always enough: with e.g. unhealthy.http_failures = 2 configured, internal_health only converges after two consecutive attempts. Computes the required attempt threshold from the checker's own config (max(unhealthy.http_failures, .tcp_failures, .timeouts, healthy.successes)) and delegates to a companion module function, resty.healthcheck.all_targets_probed(name, shm_name, min_attempts), proposed as a separate change against lua-resty-healthcheck-api7 (the vendored library this repo depends on): a per-target probe-attempt counter in shm, incremented each time an active check is actually dispatched for a target (success, failure, or timeout all count -- attempted, not "healthy"), queryable from any worker.
  • t/node/healthcheck-fresh-node-default-healthy.t: new tests covering lazy checker creation (fetch_checker returns false until the next timer tick), ensure_checker building a checker with zero prior traffic, all_targets_probed flipping only after a real probe, and the multi-attempt threshold behavior specifically (stays false after 1 attempt when min_attempts=2, flips true only after the 2nd).

Validated end to end in a local kind cluster: pod-restart-while-unhealthy shows zero leaked requests before or after the readiness transition.

Dependency note

The probe-counter shm mechanism lives in the vendored lua-resty-healthcheck-api7 library, not this repo. The library-side change is being proposed as a standalone PR against api7/lua-resty-healthcheck.

Testing

  • t/node/healthcheck-fresh-node-default-healthy.t (this repo)
  • Local kind end-to-end trial: pod restart while backend already unhealthy, comparing leaked traffic and the readiness transition, stock vs. patched

…bed checkers (PS-12691)

A freshly created health-check target defaults to internal_health=healthy
with zero probes (add_target's hardcoded is_healthy=true), so a pod that
restarts while its backend is already unhealthy briefly routes real traffic
to it until enough active probes correct the target's state. Checker
creation itself is also entirely lazy, seeded only by fetch_checker() on the
live request path -- an idle-but-critical upstream with no prior traffic
would never get a checker built ahead of a readiness check at all.

There is currently no way to tell "healthy" (a real check passed) apart from
"healthy" (the zero-probe default) from outside the checker, so nothing can
gate readiness on "has this actually been checked yet."

Adds two accessors to apisix/healthcheck_manager.lua for a readiness plugin
to use:

- ensure_checker(resource_path): proactively seeds a checker for a resource
  even with zero prior traffic, reusing the existing timer_create_checker
  construction path. Also resolves domain-name upstream nodes via
  parse_domain_in_up up front -- that resolution otherwise only happens on
  the live request path, so a checker built ahead of traffic would start
  probing under an unresolved identity and get silently rebuilt (wiping its
  probe count) the moment real traffic first resolves the domain.
- is_resource_probed(resource_path): true only once every target of the
  resource's checker has had enough real active-check attempts for its
  state to have actually converged -- a single attempt is not always
  enough: with e.g. unhealthy.http_failures = 2 configured, internal_health
  only converges after two consecutive attempts. Computes the required
  attempt threshold from the checker's own config
  (max(unhealthy.http_failures, .tcp_failures, .timeouts,
  healthy.successes)) and delegates to a companion module function,
  resty.healthcheck.all_targets_probed(name, shm_name, min_attempts),
  proposed as a separate change against lua-resty-healthcheck-api7 (the
  vendored library this repo depends on): a per-target probe-attempt
  counter in shm, incremented each time an active check is actually
  dispatched for a target (success, failure, or timeout all count --
  attempted, not "healthy"), queryable from any worker.

t/node/healthcheck-fresh-node-default-healthy.t adds coverage for: lazy
checker creation (fetch_checker returns false until the next timer tick),
ensure_checker building a checker with zero prior traffic,
all_targets_probed flipping only after a real probe, and the multi-attempt
threshold behavior specifically (stays false after 1 attempt when
min_attempts=2, flips true only after the 2nd).

Validated end to end in a local kind cluster: pod-restart-while-unhealthy
shows zero leaked requests before or after the readiness transition.

Signed-off-by: Andre Nogueira <aanogueira@protonmail.com>
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Aug 28, 2026
@aanogueira aanogueira closed this Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant