fix(cache): rebuild the Redis connection when a discovered cluster node is retired

[cosmin-staicu]

Why

On 2026-09-03 an Azure Managed Redis patch on redisaas-prd-aue-01 replaced the cluster node VMs and retired two node endpoints (4.195.18.22:8502 and :8503). Two long-lived control-plane pods kept them in the StackExchange.Redis multiplexer and retried them every ~2 s for 19 hours, logging It was not possible to connect to the redis server(s) <ip:port>/Interactive. ConnectTimeout at Error until a deployment replaced the pods. Traffic was not affected: commands went to the surviving nodes, so nothing in the library reacted.

StackExchange.Redis never forgets a server it has discovered, and only a fresh multiplexer re-reads the topology. Neither existing watchdog covers this case: hang detection needs commands stuck on the primary, and the planned-maintenance probe needs AzureRedisEvents, which Microsoft documents as Azure Cache for Redis Basic/Standard/Premium only. Azure Managed Redis does not publish it. Microsoft's guidance for Azure Managed Redis is the ForceReconnect pattern this PR automates.

What

• RedisConnector scans the multiplexer's servers every StaleEndpointScanInterval (30 s). A topology-discovered server that stays disconnected for StaleEndpointThreshold (5 min) is checked against CLUSTER NODES on a connected server. If it is no longer a member, the connector emits Redis.StaleEndpointDetected and calls the existing ForceReconnect().
• Configured endpoints are never judged stale. A node that is down but still a cluster member is left to StackExchange.Redis. A non-cluster server (CLUSTER NODES fails) is ignored.
• New options, on by default: EnableStaleEndpointDetection, StaleEndpointThreshold, StaleEndpointScanInterval. RedisConnector gains a constructor overload that takes a TimeProvider; the shipped constructor stays and uses TimeProvider.System.
• RedisHealthCheck reports ConnectionMultiplexer.DisconnectedEndPoints in its data.
• Docs: a "Redis connection self-healing" section in the resilience how-to covering all three watchdogs and the Azure Managed Redis note, settings reference rows, health-check recipe note, sample appsettings.all.json, changelog.
• Cluster membership is read with IServer.ClusterNodesRawAsync(): a node-local read under the default AllowAdmin=false, and a plain string the tests can substitute. The scan only reconnects the multiplexer it inspected, so a reconnect that lands while CLUSTER NODES is pending is not followed by a second rebuild.

Tests

Eleven new tests in RedisConnectorStaleEndpointTests drive the scan directly with a fake clock and a substituted CLUSTER NODES reply: reconnect on a retired node past the threshold, no action before the threshold, no action when the node is still a member, configured endpoints ignored, non-cluster server ignored, clock reset when the node reconnects in between, no-op before first connect, no second rebuild when the multiplexer is swapped mid-check, plus the CLUSTER NODES parser and endpoint formatting.

Full suite: 1554 passed on net8.0 and net10.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V9jRarnMLeZRZ5rYySRhkh

[github-actions]

🔎 Maintainer heads-up: automated triage flagged this PR as potentially material, so it may need a signed CLA in addition to the DCO sign-off.

Strong signals

• adds public API surface (PublicAPI.Unshipped.txt in src/UiPath.Caching)

This is advisory only — the bot does not decide. Please judge against the CLA criteria (material, product-critical, patent-sensitive, corporate contributor, broad commercial use). Note that thresholds can be gamed by splitting PRs, so use your judgement.

• If a CLA is needed → add the cla-required label (a contributor comment with signing steps is posted automatically).
• If it is not needed → replace needs-cla-review with cla-not-required so later pushes don't re-flag it.

[Copilot]

🟡 Changes recommended

The default configuration blocks the raw cluster command, and the change also introduces race, malformed-response, and binary-compatibility issues.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds automatic Redis cluster recovery when retired discovered endpoints remain in a multiplexer.

Changes:

• Adds configurable stale-endpoint detection and reconnection.
• Reports disconnected endpoints through health checks.
• Adds tests, documentation, sample configuration, and changelog entries.

File summaries

File	Description
tests/UiPath.Caching.Tests/Redis/RedisConnectorStaleEndpointTests.cs	Tests stale-node scanning and parsing.
src/UiPath.Caching/Redis/RedisHealthCheck.cs	Reports disconnected endpoints.
src/UiPath.Caching/Redis/RedisConnector.cs	Implements detection and reconnection.
src/UiPath.Caching/Redis/RedisConnectionOptions.cs	Adds detection settings.
src/UiPath.Caching/PublicAPI.Unshipped.txt	Records API changes.
samples/UiPath.Caching.Sample/appsettings.all.json	Demonstrates settings.
docs/reference/settings.md	Documents configuration options.
docs/recipes/redis-health-check.md	Documents health-check output.
docs/how-to/resilience.md	Describes Redis self-healing.
CHANGELOG.md	Records the feature.

Review details

Suppressed comments (1)

src/UiPath.Caching/Redis/RedisConnector.cs:430

• An unusable non-null reply is treated as an authoritative empty topology. ParseClusterNodeAddresses intentionally returns an empty set for values such as "garbage", so this path classifies every overdue endpoint as retired and forces a reconnect even though no connected server confirmed membership. Since a valid CLUSTER NODES response includes at least the responding node, treat an empty parsed set as null/inconclusive.

            return reply.IsNull ? null : ParseClusterNodeAddresses((string?)reply);

• Files reviewed: 10/10 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Malformed or extended cluster topology responses can incorrectly classify active endpoints as retired and trigger unnecessary reconnects.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/UiPath.Caching/Redis/RedisConnector.cs:475

• The current CLUSTER NODES address grammar permits auxiliary fields after the hostname (ip:port@cport,hostname,aux=value). Slicing everything after the first comma produces an identity such as node-a.internal,shard-id=x:6379, so a disconnected server represented by its hostname will not match its still-present member and can cause an unnecessary reconnect. Extract only the hostname field up to the next comma.

            var comma = at < 0 ? -1 : address.IndexOf(',', at);
            if (comma >= 0 && comma + 1 < address.Length)
            {
                addresses.Add($"{address[(comma + 1)..]}:{hostPort[(portSeparator + 1)..]}");
            }

• Files reviewed: 10/10 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Endpoint canonicalization and overly broad Redis error handling can cause false reconnects or silently disable detection.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/UiPath.Caching/Redis/RedisConnector.cs:475

• The membership comparison keeps Redis' textual IP spelling, while FormatEndPoint uses IPAddress.ToString(). Equivalent IPv6 forms (for example, an expanded cluster-announce-ip versus the compressed IPEndPoint returned by StackExchange.Redis) therefore compare unequal, so a disconnected node that is still present can be classified as retired and trigger repeated rebuilds. Parse and canonicalize the host portion before adding it to addresses.

• Files reviewed: 10/10 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Failed reconnect attempts reset outage tracking and unnecessarily delay the next recovery attempt.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/UiPath.Caching/Redis/RedisConnector.cs:371

• Do not clear the outage timestamps before the asynchronous reconnect has succeeded. If creating the replacement multiplexer fails (or another reconnect wins the _reconnecting race), the old multiplexer remains active, but the next scan starts a new five-minute threshold instead of retrying. Keeping these entries is safe because the next scan against a successful replacement prunes endpoints that disappeared.

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Elapsed-time tracking uses an adjustable wall clock, and the automatic timer path lacks coverage.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/UiPath.Caching/Redis/RedisConnector.cs:384

• This measures the disconnect threshold with UTC wall-clock time. TimeProvider.System.GetUtcNow() can move forward or backward when the system clock is corrected, so a clock jump can either classify a newly disconnected node as stale immediately or postpone detection beyond the configured threshold. Track the observation with GetTimestamp() and compare via GetElapsedTime() so “stayed disconnected” is based on monotonic elapsed time; update the fake provider/tests to advance timestamps as well.
  src/UiPath.Caching/Redis/RedisConnector.cs:64
• The tests explicitly disable stale-endpoint detection and invoke ScanStaleEndpointsAsync directly, so the enabled-by-default production path here is never exercised. Add a deterministic test that enables detection and verifies the configured timer actually triggers a scan/reconnect (ideally by creating the timer through the injected TimeProvider, rather than relying on real delays).

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The cluster-node parser mishandles valid endpoint fields containing auxiliary metadata, potentially causing unnecessary reconnects.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/UiPath.Caching/Redis/RedisConnector.cs:479

• CLUSTER NODES permits comma-separated auxiliary fields after the hostname (for example ip:port@cport,node-a,shard-id=abc). Taking the remainder of the field makes the recorded alias node-a,shard-id=abc:port, so a disconnected DnsEndPoint for node-a is falsely considered absent and triggers a reconnect. Parse only the first comma-delimited hostname segment before adding it.

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Non-canonical IPv6 addresses can be falsely classified as retired cluster nodes.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/UiPath.Caching/Redis/RedisConnector.cs:489

• The membership comparison preserves the raw IPv6 spelling from CLUSTER NODES, while IServer.EndPoint is an IPEndPoint whose IPAddress.ToString() is canonicalized. Redis permits announced IPv6 addresses such as 2001:0db8:0:0:0:0:0:1, so that live membership entry will not match the server formatted as 2001:db8::1:port; once disconnected, it is falsely treated as retired and can trigger repeated reconnects. Normalize IP literals before adding them to the membership set (and add an expanded-IPv6 test).

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

The implementation handles concurrency and failure cases defensively and is supported by focused tests and documentation.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
91.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud