Skip to content

fix(router): bound the replica connection paths independently of ReconnectPolicy - #381

Open
neogenie wants to merge 2 commits into
aembke:mainfrom
neogenie:fix/replica-connect-blocks-router
Open

neogenie wants to merge 2 commits into
aembke:mainfrom
neogenie:fix/replica-connect-blocks-router

Conversation

@neogenie

@neogenie neogenie commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

A replica that accepts TCP but never answers -- a paused container, a
blackholed route -- blocks the whole client, not just replica reads.

The kernel in the peer's netns completes the TCP handshake without the
process, so connection_timeout never fires. What hangs is the RESP
handshake in Replicas::add_connection, which passes transport.setup
a None timeout and therefore waits internal_command_timeout (10s by
default). That runs inline on the router task, which is the only reader
of the command channel, so while it waits rx.recv() is not polled and
every queued command waits with it -- including commands routed to a
primary that is perfectly healthy.

add_replica_with_policy and sync_replicas_with_policy then made that
unbounded rather than merely slow. Both looped on the reconnection
policy, and ReconnectPolicy documents max_attempts: 0 as unlimited
(incr_with_max: if max != 0 && curr >= max), so
next_reconnection_delay never returned Err and neither loop had an
exit. ReplicaConfig::ignore_reconnection_errors -- on by default,
whose whole purpose is to serve the command from the primary instead --
sits on the error path of the first of those loops and so was
unreachable for the entire class of failure it was written for.

Config cannot substitute for this. A finite max_attempts does arm the
replica escape hatch, but the policy is one per client: exhausting it on
the primary returns Err from reconnect_with_policy, which breaks the
read_or_write loop and ends the router task, killing the client for
good. lazy_connections = false only moves the block from one of these
loops to the other.

So both paths now make a single attempt and report failure:

  • the command path attempts add_connection inline, which lets
    ignore_reconnection_errors do its documented job and fall back to
    the primary. add_replica_with_policy had no other caller and is
    removed.
  • sync_replicas_with_policy becomes sync_replicas_once. Errors the
    policy would never have retried are still swallowed, as before; a
    transient error is now returned instead of retried forever. Both
    callers already deliver it over their response channel and return
    Ok(()), so the router task is unaffected.

Neither path touches the global reconnection counter any more. Spending
it drained a budget shared with the primary connection, and the reset on
success masked earlier primary failures.

Behaviour change worth noting: Client::sync_replicas() can now return
a transient error where it previously blocked until the sync succeeded.
Replica connections are still retried -- by the next command routed to a
replica under lazy_connections, and by the routing refresh on every
cluster sync.

`create_replica_connection`'s final `else` restores the attempt that
`write_command` consumed via `decr_check_attempted`, making
`attempts_remaining` net-zero across a pass. The `== 0` arm of
`finish_or_retry_command` is therefore unreachable, so the command goes back
onto `Router::retry_buffer`, and the `while let .. pop_front()` loop that
drains it pops the same command forever.

Nothing on that path polls a tokio resource, so the loop's `write_command`
await never returns `Pending` and the worker never returns to the scheduler.
On a multi-threaded runtime the sibling worker then waits indefinitely in
`park_condvar`, having failed to acquire the driver lock, so the time driver
stops being polled and every `tokio::time` bound in the process goes inert --
including the one that would set the `timed_out` flag this same loop checks.
Both exits from the loop are self-sealed.

The four sibling branches in the same function restore the attempt correctly:
they hand the command back over the router's mpsc channel, whose
`poll_proceed` returns `Pending` once the coop budget is spent, so an
unbounded retry there still lets the runtime make progress. The
`cfg(not(replicas))` twin of this branch -- same condition, same error
message, same two calls in the same order -- consumes the attempt and
terminates.
A replica that accepts TCP but never answers -- a paused container, a
blackholed route -- blocks the whole client, not just replica reads.

The kernel in the peer's netns completes the TCP handshake without the
process, so `connection_timeout` never fires. What hangs is the RESP
handshake in `Replicas::add_connection`, which passes `transport.setup`
a `None` timeout and therefore waits `internal_command_timeout` (10s by
default). That runs inline on the router task, which is the only reader
of the command channel, so while it waits `rx.recv()` is not polled and
every queued command waits with it -- including commands routed to a
primary that is perfectly healthy.

`add_replica_with_policy` and `sync_replicas_with_policy` then made that
unbounded rather than merely slow. Both looped on the reconnection
policy, and `ReconnectPolicy` documents `max_attempts: 0` as unlimited
(`incr_with_max`: `if max != 0 && curr >= max`), so
`next_reconnection_delay` never returned `Err` and neither loop had an
exit. `ReplicaConfig::ignore_reconnection_errors` -- on by default,
whose whole purpose is to serve the command from the primary instead --
sits on the error path of the first of those loops and so was
unreachable for the entire class of failure it was written for.

Config cannot substitute for this. A finite `max_attempts` does arm the
replica escape hatch, but the policy is one per client: exhausting it on
the primary returns `Err` from `reconnect_with_policy`, which breaks the
`read_or_write` loop and ends the router task, killing the client for
good. `lazy_connections = false` only moves the block from one of these
loops to the other.

So both paths now make a single attempt and report failure:

- the command path attempts `add_connection` inline, which lets
  `ignore_reconnection_errors` do its documented job and fall back to
  the primary. `add_replica_with_policy` had no other caller and is
  removed.
- `sync_replicas_with_policy` becomes `sync_replicas_once`. Errors the
  policy would never have retried are still swallowed, as before; a
  transient error is now returned instead of retried forever. Both
  callers already deliver it over their response channel and return
  `Ok(())`, so the router task is unaffected.

Neither path touches the global reconnection counter any more. Spending
it drained a budget shared with the primary connection, and the reset on
success masked earlier primary failures.

Behaviour change worth noting: `Client::sync_replicas()` can now return
a transient error where it previously blocked until the sync succeeded.
Replica connections are still retried -- by the next command routed to a
replica under `lazy_connections`, and by the routing refresh on every
cluster sync.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant