Skip to content

Stop connect_async's host walk where libpq stops it - #153

Merged
isdaniel merged 1 commit into
mainfrom
fix/startup-reconnect-liveness
Nov 1, 2026
Merged

isdaniel merged 1 commit into
mainfrom
fix/startup-reconnect-liveness

Conversation

@isdaniel

@isdaniel isdaniel commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR fixes them, adds regression tests for each, and documents the behaviour changes for the next release.

What changed

Area Problem (measured on main) Fix
Reconnect (stream.rs) timeout(d, es.next_event()) with d shorter than a reconnect never recovers: the dropped future takes the half-built connection with it, and the next call dials again from scratch. With d = 2 ms and one walsender kill: 9,459 connections in 50 s (rustls), 5,454 (libpq), and no events after the fault (d = 50–500 ms: 0–3 events). A private recovering flag is raised by recover_connection and lowered only on success. EventStream::next_event / next_event_before then park the stream in the poll future poll_next already uses, so the reconnect survives the caller's timeout.
Reconnect cancel (stream.rs) token.cancel() during a stuck reconnect waited out the connect attempt (2.8–3.8 s). The retry loop selects on the cancellation token during a reconnect: 0.13 ms.
libpq runtime (libpq.rs) new() now runs IDENTIFY_SYSTEM, which registered the AsyncFd on the runtime current at new() for the life of the connection. start() under another runtime failed (...is being shutdown) or hung (8 s). exec_async registers the socket for its own command and releases it, unless a COPY holds a registration. START_REPLICATION / COPY OUT still make the long-lived registration under the runtime that starts them.
connection_timeout = ZERO (retry.rs) Returned Timeout("0ns") after the first failed attempt, ignoring max_attempts. Zero means no budget; retries as for any other value.
Cancelled startup command (stream.rs) After a dropped start() / slot_confirmed_flush_lsn() / timeline_history(), every later command failed with a non-permanent Protocol error, so a retry loop never rebuilt the stream. Mapped to PermanentConnection at the stream layer; next_event_with_retry still reconnects.
Aligned snapshot (aligned.rs) TRUNCATE, table-rewriting ALTER TABLE and DROP + CREATE are not MVCC-safe: committed after the consistent point, they make COPY return an empty table while finish() says Ok. Under a truncating writer, 60/60 captures were silently empty. capture takes ACCESS SHARE ... NOWAIT on the requested tables, then fails if one changed since the snapshot (relfilenode / name). Lock alone is not enough (58/60 still empty): there is a window between the consistent point and the lock.
Aligned snapshot identity (capture_aligned) With a multi-host libpq conninfo the side connection could land on another cluster and return its rows. Refuses a side connection whose server_identity differs (PermanentConnection).
Tests Two integration tests and the startup heartbeat tests assumed 500 ms / 50 ms. Wait for the server (or leave margin) instead. One strict-sink control now fails when the commit_lsn filter is broken.
Docs / CI README and rustdoc omitted the acknowledge-before-copy step (50 transactions re-applied after a reconnect), the DDL lock, cancel-safety of next_row, and the xid semantics. Added. README gets an "Unreleased: changes since 0.9.0" section for the eventual upgrade note. New MSRV 1.87 job (library check only, because proptest 1.12 needs 1.88). Examples job uses --locked. Example, fuzz and load-test lockfiles refreshed. pq-sys 0.7.6 → 0.7.8 (bindgen regeneration only).

Behaviour changes to review

  • While a parked reconnect is in flight, EventStream::inner(), inner_mut() and current_lsn() panic until the next next_event() returns (documented; same class as an abandoned poll_next). Feedback methods are unaffected.
  • AlignedSnapshot::capture can now return an error (a requested table changed since the snapshot, or a conflicting lock is held at that instant). ALTER TABLE, TRUNCATE, VACUUM FULL and CLUSTER on the requested tables wait until finish(), and writes queued behind them wait too (up to their lock_timeout). Callers should retry a refused capture.
  • A cancelled startup command now returns PermanentConnection instead of Protocol.
  • connection_timeout = ZERO retries instead of failing on the first attempt.

Type of change

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor / internal cleanup
  • Documentation
  • CI / build

Checklist

  • make before-git-push passes locally (check, build, format, audit, test, doc-check)
  • Tests added/updated for the change (coverage stays ≥ 90%)
  • Both backends build (--features rustls-tls if connection/streaming code changed)
  • Docs / README updated if public API changed

@codspeed

codspeed Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 50 untouched benchmarks


Comparing fix/startup-reconnect-liveness (960b3d1) with main (2b35e0e)

Open in CodSpeed

@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.09774% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.48%. Comparing base (2b35e0e) to head (960b3d1).

Files with missing lines Patch % Lines
src/stream.rs 98.84% 3 Missing ⚠️
src/retry.rs 93.33% 2 Missing ⚠️
src/connection/libpq.rs 99.52% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #153      +/-   ##
==========================================
+ Coverage   96.38%   96.48%   +0.09%     
==========================================
  Files          36       36              
  Lines       28657    29290     +633     
==========================================
+ Hits        27621    28259     +638     
+ Misses       1036     1031       -5     
Flag Coverage Δ
integration 64.35% <82.31%> (+0.69%) ⬆️
unit 93.42% <95.48%> (+0.44%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical issues can silently omit snapshot rows after leaf-partition rewrites and leave aligned-snapshot handoffs waiting indefinitely while retaining table locks after a keepalive-only reconnect.

4 open issues
What changed in this PR

This PR improves connection recovery and aligned-snapshot reliability in pg-walstream, with regression tests and documentation for the changed behavior.

Changes:

  • Preserve reconnect progress across repeated read timeouts, improve cancellation handling, and release libpq command socket registrations between commands.
  • Add aligned-snapshot table locks, change detection, and side-connection identity checks.
  • Expand regression coverage, document transaction and upgrade semantics, add Rust 1.87 compatibility checks, and refresh dependencies.
File Description
src/​types.rs Clarifies message xid semantics and tests JSON compatibility.
src/​stream.rs Updates recovery, cancellation, startup errors, and identity checks.
src/​snapshot/​mod.rs Corrects the catalog/COPY behavior comment.
src/​snapshot/​aligned.rs Adds table protection, identity validation, and usage guidance.
src/​retry.rs Treats a zero connection budget as unlimited.
src/​protocol.rs Documents and tests per-change streamed xids.
src/​connection/​libpq.rs Releases command registrations and adds connection regression tests.
README.md Documents upgrade changes and safer snapshot merging.
load-tests/​Cargo.lock Refreshes load-test dependencies.
integration-tests/​reconnect_recovery.rs Tests slow reconnects and cancelled startup recovery.
integration-tests/​initial_snapshot.rs Waits for server state before cancelling snapshot creation.
integration-tests/​aligned_snapshot.rs Strengthens overlap checks and tests concurrent table changes.
fuzz/​Cargo.lock Updates the fuzzing smallvec dependency.
examples/​typed-deserialization/​Cargo.lock Refreshes example dependencies.
examples/​tokio-spawn-streaming/​Cargo.lock Refreshes example dependencies.
examples/​safe-transaction-consumer/​Cargo.lock Refreshes example dependencies.
examples/​raw-xlogdata/​Cargo.lock Refreshes example dependencies.
examples/​rate-limited-streaming/​Cargo.lock Refreshes example dependencies.
examples/​polling/​Cargo.lock Refreshes example dependencies.
examples/​pg-basebackup/​Cargo.lock Refreshes example dependencies.
examples/​initial-snapshot/​Cargo.lock Refreshes example dependencies.
examples/​derive-router/​Cargo.lock Refreshes example dependencies.
examples/​binary-column-access/​Cargo.lock Refreshes example dependencies.
examples/​basic-streaming/​Cargo.lock Refreshes example dependencies.
examples/​arbitrary-fuzzing/​Cargo.lock Updates the example’s smallvec dependency.
Cargo.toml Raises pq-sys to 0.7.8.
Cargo.lock Updates pq-sys and equivalent.
.github/​workflows/​ci-checks.yml Adds Rust 1.87 checks and enforces example lockfiles.

Review effort: Balanced

Comment thread src/snapshot/aligned.rs Resolved
Comment thread src/stream.rs Resolved
Comment thread integration-tests/reconnect_recovery.rs Resolved
Comment thread src/stream.rs Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical defects allow post-snapshot partition attachments to affect copied data and resumed bounded reads to report completion while leaving intervening transactions unread.

2 open issues
4 resolved since last review

Review effort: Balanced

Comment thread src/snapshot/aligned.rs Resolved
Comment thread src/stream.rs Resolved
connect_async continued to the next address after every failed attempt, so a
bad password was tried once per host and address (and twice that under
prefer-standby), where PQconnectdb tries it once. Against a PG17 server,
`host=P,P,P password=wrong` produced 3 failed logins per new() on this tree
and 1 with PQconnectdb; `dbname=nosuchdb host=P,P` made 2 attempts against 1.

libpq's rule (fe-connect.c, 14 to 18) is not "authentication failures": once
a connection to an address is up, any failure ends the connect, except
SQLSTATE 57P03 ("cannot connect now"), a target_session_attrs mismatch, a
connect_timeout expiry and a connection that never came up. An attempt that
fails while libpq was past the TCP connect and short of a completed startup
(no server_version ParameterStatus yet) now ends the walk, unless the server
said 57P03. A failed connection exposes no SQLSTATE, so 57P03 is recognised
by the server's wording ("the database system is ...", identical in 14 to
18); a server with a non-English lc_messages would end the walk there. The
final error is unchanged and stays Authentication for a bad password.

Measured against PG17 (primary, hot standby, hot_standby=off standby that
answers 57P03, dead port), 20 host-selection scenarios sync vs async: every
walking case (dead host first, standby/read-write/prefer-standby selection,
57P03 first, localhost, invalid name with hostaddr) is unchanged, and the
failed-login count drops from 3 to 1 per new().

Register a libpq command's socket with the runtime driving it

new() now runs IDENTIFY_SYSTEM through exec_async, which registered the
connection's AsyncFd on the runtime current at new() and kept it for the life
of the connection (at 0.9.0 registration happened in start()). A stream built
under one runtime and started under another then failed or hung: with the
first runtime dropped, start() returned "A Tokio 1.x context was found, but it
is being shutdown"; with it kept but never driven, start() waited forever for
a readiness event that its reactor would never deliver.

exec_async now registers the dup for its own command and releases it after,
unless a COPY already holds a registration. START_REPLICATION and COPY OUT
still register the long-lived one at their own entry point, so the
registration that serves reads is made by start() under the runtime that
starts the stream. The read and feedback paths are untouched.
@isdaniel
isdaniel force-pushed the fix/startup-reconnect-liveness branch from 5d1d2ed to 960b3d1 Compare November 1, 2026 12:50
@isdaniel
isdaniel merged commit f706d1f into main Nov 1, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants