fix(cache): judge stale endpoints from the client's own cluster topology, not an admin command - #155
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Four moderate correctness issues remain in topology refresh, identity matching, and scan disabling.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Reworks stale Redis endpoint detection to use refreshed client topology without requiring admin commands.
Changes:
- Refreshes cluster membership through
ConfigureAsync. - Disables unsupported scans after one telemetry event.
- Adds unit, integration, and documentation coverage.
Required changes (moderate, 1 vote each):
- Preserve both hostname and endpoint identities when checking membership.
- Treat
ConfigureAsync()returningfalseas inconclusive. - Make multiplexer identity validation and scan disabling atomic.
- Only judge membership from topology known to have been refreshed.
File summaries
| File | Description |
|---|---|
tests/UiPath.Caching.Tests/Redis/RedisConnectorStaleEndpointTests.cs |
Tests revised stale-scan behavior. |
tests/UiPath.Caching.Tests/Redis/RedisConnectorIntegrationTests.cs |
Tests refresh under default non-admin configuration. |
src/UiPath.Caching/Redis/RedisConnector.cs |
Implements topology refresh and scan disabling. |
src/UiPath.Caching/Redis/ClusterTopologyReader.cs |
Adds cluster-topology membership reading. |
docs/reference/settings.md |
Documents configuration behavior. |
docs/how-to/resilience.md |
Explains topology-based stale endpoint detection. |
CHANGELOG.md |
Records the fix. |
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 4
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
3efc6fd to
3c5e5cf
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Two moderate topology-reliability issues must be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Balanced
3c5e5cf to
8028f20
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Transient topology refresh failures can incorrectly disable stale-endpoint scanning permanently.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
src/UiPath.Caching/Redis/RedisConnector.cs:489
- The PR and resilience docs state that a refresh leaving no configured node connected must be inconclusive, but this branch accepts a replaced configuration without rechecking
server.IsConnected. If the handshake installs the configuration and that connection then drops beforeConfigureAsynccompletes, the scan can still judge membership and force a reconnect from a node that is no longer connected. Only accept the changed configuration while its server remains connected; otherwise retry next interval.
else
{
return new(Conclusive: true, _topologyReader.GetMembers(after));
}
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Balanced
8028f20 to
cd87c1b
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Two moderate topology-handling issues could prevent correct stale-endpoint recovery.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
src/UiPath.Caching/Redis/RedisConnector.cs:516
_nullTopologyRefreshesis mutated here before the caller verifies that this multiplexer is still current. IfForceReconnectswaps multiplexers whileConfigureAsyncis pending, the discarded old result still increments (or resets) this shared counter, so a replacement can disable scanning before receiving three of its own null-topology results. Scope/reset the streak per multiplexer, or update it only after the identity check under the swap lock.
return neverConfigured && ++_nullTopologyRefreshes >= NullTopologyRefreshLimit ? new(Conclusive: true, Members: null) : ClusterMembership.Inconclusive;
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Balanced
cd87c1b to
84ce583
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The integration test has a critical compilation issue and does not prove that CLUSTER NODES reaches Redis.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Balanced
84ce583 to
0b32a72
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Two moderate race-related correctness issues could cause premature reconnection or scan disabling.
Review details
Suppressed comments (2)
src/UiPath.Caching/Redis/RedisConnector.cs:493
- A replaced configuration is accepted without re-checking that its server is still connected. If the configured node installs a topology and then disconnects during the refresh, this returns a conclusive result and may force a reconnect, despite the stated behavior that a refresh leaving no connected node must defer judgment. Gate the changed configuration on
server.IsConnected; the existing test only covers the null-configuration branch.
else
{
_nullTopologyRefreshes = 0;
return new(Conclusive: true, _topologyReader.GetMembers(after));
src/UiPath.Caching/Redis/RedisConnector.cs:504
- This counter is mutated before
ScanStaleEndpointsAsyncverifies that the inspected multiplexer is still current. In theclusterConfigurationKnown: falseswap scenario, the old scan increments the streak afterForceReconnectswaps multiplexers; the later identity check prevents immediate disabling but leaves that increment behind, so the replacement can be permanently disabled after fewer than three of its own null-topology refreshes. Tie the streak to the judged multiplexer or commit counter updates only after validating its identity.
// A node with no configuration may have lost only the topology reply; a persistent absence means it cannot answer it.
return neverConfigured && ++_nullTopologyRefreshes >= NullTopologyRefreshLimit ? new(Conclusive: true, Members: null) : ClusterMembership.Inconclusive;
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Balanced
c506cda to
047135f
Compare
…ogy, not an admin command CLUSTER NODES is admin-gated client-side, so under the default allowAdmin=false the scan threw RedisCommandException every interval and never reconnected. The scan now refreshes the topology through ConfigureAsync, whose CLUSTER NODES is an internal call, and reads the ClusterConfiguration of the connected configured endpoints, the only ones a non-initial reconfigure re-handshakes, and only once the refresh has replaced the instance. A skipped or unlanded refresh, or no connected configured endpoint, is inconclusive and retried next interval. When a refreshed node reports no cluster configuration on three consecutive refreshes the scan emits one Redis.StaleEndpointScanDisabled event and stops. Nodes the cluster flags fail or still has in handshake are absent from the topology the client dials and count as departed. A down node the topology still lists is re-checked only once per threshold and reported once through Redis.StaleEndpointStillAMember. A refresh that keeps throwing is tracked and backed off exponentially, up to an hour between attempts, instead of once per interval. Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V9jRarnMLeZRZ5rYySRhkh
047135f to
5645c0f
Compare
|



Follow-up to #153, from review feedback on the merged PR.
The bug
The membership check called
IServer.ClusterNodesRawAsync().CLUSTERis an admin command in StackExchange.Redis, so with the defaultallowAdmin=falsethe client refuses it before it leaves the process with aRedisCommandException. That type derives fromException, notRedisServerException, so the "not a cluster" filter never matched. The exception fell through to the outer catch, was tracked, and nothing reconnected. Every scan interval repeated it. The unit tests mockedIServer, so the client-side gate was never on the test path. Verified with a probe against a live server on 3.1.13: bothClusterNodesRawAsyncandExecuteAsync("CLUSTER", "NODES")are refused, since the string command maps back toRedisCommand.CLUSTER.The fix
IConnectionMultiplexer.ConfigureAsync(), which re-runs the handshake on every connected node. That handshake issuesCLUSTER NODESas an internal call (SetInternalCall, written directly to the connection), so the admin gate does not apply. The scan then readsIServer.ClusterConfigurationfrom a connected node and compares nodeEndPoints directly. The hand-writtenCLUSTER NODESparser goes away.CLUSTER NODES), membership can never be judged. The scan emits oneRedis.StaleEndpointScanDisabledevent, disposes its timer, and stops for the lifetime of the connector. This addresses the second review point: a permanent permission problem no longer produces an unbounded telemetry stream.ClusterConfigurationhas no public constructor, so the read of it sits behind a small internalIClusterTopologyReaderseam that tests replace.Tests
RefreshClusterMembership_ReachesTheServer_UnderTheDefaultConnectionStringruns the real refresh against the CI Redis with the default connection string. It skips itself if the connection string setsallowAdmin. This is the test that would have caught the original bug.Docs
resilience.md,settings.mdand the changelog describe the admin-free mechanism and the disable behaviour.🤖 Generated with Claude Code
https://claude.ai/code/session_01V9jRarnMLeZRZ5rYySRhkh