Skip to content

Report consecutive connect failures in TNode and rate-limit the repeated log - #225

Open
resetius wants to merge 1 commit into
masterfrom
fix-tnode-connect-failure-logging
Open

resetius wants to merge 1 commit into
masterfrom
fix-tnode-connect-failure-logging

Conversation

@resetius

Copy link
Copy Markdown
Owner

Fixes #218

TNode retries a failed connect every second forever. That is deliberate: an embedder may start peers in any order, and a peer that is down for a while must not take the node down. Giving up or throwing would break such embedders, so this PR keeps the retry loop and changes only what is reported.

Chosen option: rate-limited, counted connect-error logging. It needs no new API or config and fits the library's existing std::cerr logging.

  • The node counts consecutive failed connects and resets the counter on success.
  • Each error line now carries the count, e.g. Error connecting to 127.0.0.1:9000: connect: Connection refused (13 consecutive failures). The first 3 failures are logged, then every 10th, so a permanently failing peer shows a growing counter instead of an unbounded stream of identical lines.
  • TNode::ConsecutiveConnectFailures() exposes the counter (not on INode) so an embedder or test can detect a stuck peer.

Not done: backoff with a cap, and reporting a peer's bind failure at startup. Backoff changes reconnect latency for everyone, and a bind failure is only known to the peer's own process; neither fits a small non-breaking change. A failed-after-N error or callback can be built on the counter if wanted.

Test: test_node_counts_connect_failures checks that the counter grows while nobody listens and returns to 0 after a successful connect.

Both this PR and the one for #219 touch DoConnect() but different lines; whichever merges second should rebase trivially or with a small mechanical conflict.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Linux)

334 tests  +1   334 ✅ +1   25s ⏱️ +2s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 4c3b507. ± Comparison against base commit 85489f0.

@github-actions

Copy link
Copy Markdown

Unit Test Results (MacOS)

255 tests  +1   255 ✅ +1   29s ⏱️ -3s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 4c3b507. ± Comparison against base commit 85489f0.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Windows)

176 tests  +1   176 ✅ +1   49s ⏱️ +5s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 4c3b507. ± Comparison against base commit 85489f0.

@resetius

Copy link
Copy Markdown
Owner Author

Independent review of head 4c3b5075d7ad8dcb031f94e0d57b86cd3a00038b. Verdict: no P1/P2 findings. The change is non-breaking and not spammy. Whether it closes #218 is the owner's call (P3 below).

What I checked

  • TNode reconnects forever to a peer that failed to bind, and the run hangs without an error #218 and the diff. The retry loop is unchanged (every 1 s). The counter goes up in the catch, is reset on success (coroio/actors/node.hpp:177, :182) and is exposed as TNode::ConsecutiveConnectFailures() (:136). It is not on INode, and the class is a header-only template, so INode users and embedders see no API change.
  • Logging. Failures 1 to 3 are logged, then every 10th (:183-185). I ran a node against a closed port for 25 s on macOS. It printed 5 lines, for failures 1, 2, 3, 10 and 20; master prints 26. ConsecutiveConnectFailures() returned 26. Nothing new throws and reconnect timing is the same. Only the volume and format of stderr change.
  • The test passes on kqueue, poll and select (macOS) and on epoll, uring and poll (Linux, poller switched locally). It uses the new accessor, so it checks the counter, not the run hang from the issue.
  • Full ctest with ASan on macOS and on rpi5 (Linux 7.3): all passed. CI: every job on this head is green.
  • Merge: this PR conflicts textually with Flush TNode output queued while connecting once the connection completes #223 in tests/test_actors.cpp. Both add a test after test_node_reconnects_to_dead_peer and an ADD_TEST at the same line. node.hpp merges cleanly. The fix is mechanical.

Findings

@resetius resetius added the agent-reviewed An independent review agent checked this PR; the verdict is in a PR comment label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-reviewed An independent review agent checked this PR; the verdict is in a PR comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TNode reconnects forever to a peer that failed to bind, and the run hangs without an error

1 participant