Repository navigation
Conversation
…cancels an in-flight RPC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #337
Problem
Config listening must survive a reconnect — but it doesn't.
When a reconnect closes the old gRPC channel, grpc aio reports the cancelled in-flight listen RPC as
asyncio.CancelledError, aBaseExceptionthat escapes everyexcept Exceptionin the SDK. The listen task dies silently — no error is ever logged, and config changes stop reaching listeners until restart.Changes
Important
Also included: exponential backoff for immediate re-listens — a separate request-storm risk, not caused by the race above.
After a change notification the client re-listens immediately to drain follow-up changes. If changes never settle, this is an RTT-limited request loop (measured ~1600 QPS from one client). Now the interval starts at 100 ms, doubles per consecutive round with changes, caps at 5s, and resets on a clean round: the degraded rate converges to the normal heartbeat rate, and normal draining is unaffected (first re-listens are still sub-second). Pleased to split this into its own PR if preferred.
How Verified
With an in-process mock gRPC server (built on the SDK's own pb2 stubs) that can suspend the listen RPC(refer to the issue description), the race becomes 100% deterministic: suspend the listen request → reconnect → old channel close cancels the in-flight RPC. Two scenarios were run before/after the fix:
CancelledError; 0 listen requests after reconnect; shutdown re-raisesCompatibility / Risks
timeout_millis, then a properNacosException).Task.cancelling()is 3.11+. On 3.10 the code cannot distinguish genuine from spurious cancellation and always translates toNacosException. Consequence: on 3.10, cancelling a task that is awaiting an RPC surfaces asNacosExceptioninstead ofCancelledError(the task stops at its next cancellation point). All SDK-internal cancellation paths (shutdown) behave correctly either way.