fix(upstream): keep last resolved address on transient DNS failure - #13903
Open
janiussyafiq wants to merge 2 commits into
Open
fix(upstream): keep last resolved address on transient DNS failure#13903janiussyafiq wants to merge 2 commits into
janiussyafiq wants to merge 2 commits into
Conversation
A transient DNS failure (timeout, SERVFAIL) dropped the domain node from the upstream node list. The node churn rebuilt the health checker, and the re-added target starts as healthy, so a node that was correctly marked unhealthy received traffic again without passing any recovery probe. Keep the last resolved address when the query cannot be completed, so the node list stays stable and health state survives. An authoritative NXDOMAIN answer still removes the node.
httpc:request_uri can return nil, err; dereferencing res.status would raise instead of reporting the HTTP error. Use the same nil-safe pattern as TEST 2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When an upstream uses a domain name in
nodes, a transient DNS failure (timeout, SERVFAIL) during periodic re-resolution dropped the node from the resolved node list.When DNS recovered moments later, the node was re-added.
That drop/re-add changes the node list, which rebuilds the health checker, and a re-added target always starts as healthy.
As a result, a node that active checks had correctly marked unhealthy received traffic again without passing any recovery probe, and workers could disagree about its state because each of them resolves independently.
The DNS failure says nothing about the backend's health, so it should not destroy the health checker's state.
This PR:
core.dns.client.resolve()/core.resolver.parse_domain()as a third value, so callers can tell an authoritative NXDOMAIN answer apart from a query that could not be completed at allsort_by_key_hosttie-break on port, so two nodes sharing one IP cannot sort unstably and report a phantom node changeVerified against an end-to-end reproduction (3-node upstream, one node failing its health endpoint, DNS server blipped for 8 seconds): without the fix the unhealthy node returns to rotation on all workers after the blip; with the fix it receives no traffic and stays unhealthy.
Which issue(s) this PR fixes:
Fixes #13888
Checklist