Skip to content

fix #337: config listen loop silently dies when reconnect cancels an in-flight RPC - #338

Open
zmz789 wants to merge 1 commit into
nacos-group:masterfrom
zmz789:fix/config-listen-cancelled-on-reconnect
Open

zmz789 wants to merge 1 commit into
nacos-group:masterfrom
zmz789:fix/config-listen-cancelled-on-reconnect

Conversation

@zmz789

@zmz789 zmz789 commented Oct 12, 2026

Copy link
Copy Markdown

Fixes #337

Problem

Config listening must survive a reconnect — but it doesn't.

When a reconnect closes the old gRPC channel, grpc aio reports the cancelled in-flight listen RPC as asyncio.CancelledError, a BaseException that escapes every except Exception in the SDK. The listen task dies silently — no error is ever logged, and config changes stop reaching listeners until restart.

Changes

  1. Root fix (transport layer): when a connection close cancels an in-flight RPC, surface it as a normal retryable error so the existing retry/reconnect path handles it — genuine task cancellation still propagates. One change at the single RPC egress protects config, naming and ai at once.
  2. Harden the config listen loop: an unexpected exception in a listen round is now logged and the loop continues, instead of the task dying silently.
  3. Fail loudly at shutdown: if the listen task is already dead, log an explicit ERROR instead of re-raising the stale exception and aborting the rest of cleanup.

Important

Also included: exponential backoff for immediate re-listens — a separate request-storm risk, not caused by the race above.

After a change notification the client re-listens immediately to drain follow-up changes. If changes never settle, this is an RTT-limited request loop (measured ~1600 QPS from one client). Now the interval starts at 100 ms, doubles per consecutive round with changes, caps at 5s, and resets on a clean round: the degraded rate converges to the normal heartbeat rate, and normal draining is unaffected (first re-listens are still sub-second). Pleased to split this into its own PR if preferred.

How Verified

With an in-process mock gRPC server (built on the SDK's own pb2 stubs) that can suspend the listen RPC(refer to the issue description), the race becomes 100% deterministic: suspend the listen request → reconnect → old channel close cancels the in-flight RPC. Two scenarios were run before/after the fix:

before fix after fix
reconnect with in-flight listen RPC listen task killed by CancelledError; 0 listen requests after reconnect; shutdown re-raises task alive; server keeps receiving listen requests; config change delivered to listener; clean shutdown
server returns changes forever + query keeps failing ~5000 listen requests / 3 s (RTT-limited storm) 2 / 3 s (exponential backoff converges to heartbeat rate)

Compatibility / Risks

  • Retry behavior: a spuriously-cancelled RPC now retries (up to 3 attempts within the request timeout) instead of crashing the caller. Retried operations (publish/register) are idempotent in their normal usage; failure propagation becomes slower but cleaner (up to timeout_millis, then a proper NacosException).
  • Python 3.10 caveat: Task.cancelling() is 3.11+. On 3.10 the code cannot distinguish genuine from spurious cancellation and always translates to NacosException. Consequence: on 3.10, cancelling a task that is awaiting an RPC surfaces as NacosException instead of CancelledError (the task stops at its next cancellation point). All SDK-internal cancellation paths (shutdown) behave correctly either way.
  • Log volume: the catch-all can log one stack per loop round; rounds are naturally ≥5 s apart, and on the worst path (persistent change storm) the backoff converges to the same 5 s cadence.
  • Scope note: naming and ai send RPCs through the same transport code path, so this PR fixes the same silent-death bug for them too. What they don't get yet is the loop-level catch-all — log any unexpected exception and keep the loop running — which this PR applies to the config listen loop only; doing the same for naming/ai loops is left to a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Config listen loop silently dies when reconnect cancels an in-flight RPC

1 participant