Conversation
`create_replica_connection`'s final `else` restores the attempt that `write_command` consumed via `decr_check_attempted`, making `attempts_remaining` net-zero across a pass. The `== 0` arm of `finish_or_retry_command` is therefore unreachable, so the command goes back onto `Router::retry_buffer`, and the `while let .. pop_front()` loop that drains it pops the same command forever. Nothing on that path polls a tokio resource, so the loop's `write_command` await never returns `Pending` and the worker never returns to the scheduler. On a multi-threaded runtime the sibling worker then waits indefinitely in `park_condvar`, having failed to acquire the driver lock, so the time driver stops being polled and every `tokio::time` bound in the process goes inert -- including the one that would set the `timed_out` flag this same loop checks. Both exits from the loop are self-sealed. The four sibling branches in the same function restore the attempt correctly: they hand the command back over the router's mpsc channel, whose `poll_proceed` returns `Pending` once the coop budget is spent, so an unbounded retry there still lets the runtime make progress. The `cfg(not(replicas))` twin of this branch -- same condition, same error message, same two calls in the same order -- consumes the attempt and terminates.
A replica that accepts TCP but never answers -- a paused container, a blackholed route -- blocks the whole client, not just replica reads. The kernel in the peer's netns completes the TCP handshake without the process, so `connection_timeout` never fires. What hangs is the RESP handshake in `Replicas::add_connection`, which passes `transport.setup` a `None` timeout and therefore waits `internal_command_timeout` (10s by default). That runs inline on the router task, which is the only reader of the command channel, so while it waits `rx.recv()` is not polled and every queued command waits with it -- including commands routed to a primary that is perfectly healthy. `add_replica_with_policy` and `sync_replicas_with_policy` then made that unbounded rather than merely slow. Both looped on the reconnection policy, and `ReconnectPolicy` documents `max_attempts: 0` as unlimited (`incr_with_max`: `if max != 0 && curr >= max`), so `next_reconnection_delay` never returned `Err` and neither loop had an exit. `ReplicaConfig::ignore_reconnection_errors` -- on by default, whose whole purpose is to serve the command from the primary instead -- sits on the error path of the first of those loops and so was unreachable for the entire class of failure it was written for. Config cannot substitute for this. A finite `max_attempts` does arm the replica escape hatch, but the policy is one per client: exhausting it on the primary returns `Err` from `reconnect_with_policy`, which breaks the `read_or_write` loop and ends the router task, killing the client for good. `lazy_connections = false` only moves the block from one of these loops to the other. So both paths now make a single attempt and report failure: - the command path attempts `add_connection` inline, which lets `ignore_reconnection_errors` do its documented job and fall back to the primary. `add_replica_with_policy` had no other caller and is removed. - `sync_replicas_with_policy` becomes `sync_replicas_once`. Errors the policy would never have retried are still swallowed, as before; a transient error is now returned instead of retried forever. Both callers already deliver it over their response channel and return `Ok(())`, so the router task is unaffected. Neither path touches the global reconnection counter any more. Spending it drained a budget shared with the primary connection, and the reset on success masked earlier primary failures. Behaviour change worth noting: `Client::sync_replicas()` can now return a transient error where it previously blocked until the sync succeeded. Replica connections are still retried -- by the next command routed to a replica under `lazy_connections`, and by the routing refresh on every cluster sync.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A replica that accepts TCP but never answers -- a paused container, a
blackholed route -- blocks the whole client, not just replica reads.
The kernel in the peer's netns completes the TCP handshake without the
process, so
connection_timeoutnever fires. What hangs is the RESPhandshake in
Replicas::add_connection, which passestransport.setupa
Nonetimeout and therefore waitsinternal_command_timeout(10s bydefault). That runs inline on the router task, which is the only reader
of the command channel, so while it waits
rx.recv()is not polled andevery queued command waits with it -- including commands routed to a
primary that is perfectly healthy.
add_replica_with_policyandsync_replicas_with_policythen made thatunbounded rather than merely slow. Both looped on the reconnection
policy, and
ReconnectPolicydocumentsmax_attempts: 0as unlimited(
incr_with_max:if max != 0 && curr >= max), sonext_reconnection_delaynever returnedErrand neither loop had anexit.
ReplicaConfig::ignore_reconnection_errors-- on by default,whose whole purpose is to serve the command from the primary instead --
sits on the error path of the first of those loops and so was
unreachable for the entire class of failure it was written for.
Config cannot substitute for this. A finite
max_attemptsdoes arm thereplica escape hatch, but the policy is one per client: exhausting it on
the primary returns
Errfromreconnect_with_policy, which breaks theread_or_writeloop and ends the router task, killing the client forgood.
lazy_connections = falseonly moves the block from one of theseloops to the other.
So both paths now make a single attempt and report failure:
add_connectioninline, which letsignore_reconnection_errorsdo its documented job and fall back tothe primary.
add_replica_with_policyhad no other caller and isremoved.
sync_replicas_with_policybecomessync_replicas_once. Errors thepolicy would never have retried are still swallowed, as before; a
transient error is now returned instead of retried forever. Both
callers already deliver it over their response channel and return
Ok(()), so the router task is unaffected.Neither path touches the global reconnection counter any more. Spending
it drained a budget shared with the primary connection, and the reset on
success masked earlier primary failures.
Behaviour change worth noting:
Client::sync_replicas()can now returna transient error where it previously blocked until the sync succeeded.
Replica connections are still retried -- by the next command routed to a
replica under
lazy_connections, and by the routing refresh on everycluster sync.