Repository navigation
Stop connect_async's host walk where libpq stops it - #153
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #153 +/- ##
==========================================
+ Coverage 96.38% 96.48% +0.09%
==========================================
Files 36 36
Lines 28657 29290 +633
==========================================
+ Hits 27621 28259 +638
+ Misses 1036 1031 -5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical issues can silently omit snapshot rows after leaf-partition rewrites and leave aligned-snapshot handoffs waiting indefinitely while retaining table locks after a keepalive-only reconnect.
4 open issues
What changed in this PR
This PR improves connection recovery and aligned-snapshot reliability in pg-walstream, with regression tests and documentation for the changed behavior.
Changes:
- Preserve reconnect progress across repeated read timeouts, improve cancellation handling, and release libpq command socket registrations between commands.
- Add aligned-snapshot table locks, change detection, and side-connection identity checks.
- Expand regression coverage, document transaction and upgrade semantics, add Rust 1.87 compatibility checks, and refresh dependencies.
| File | Description |
|---|---|
src/types.rs |
Clarifies message xid semantics and tests JSON compatibility. |
src/stream.rs |
Updates recovery, cancellation, startup errors, and identity checks. |
src/snapshot/mod.rs |
Corrects the catalog/COPY behavior comment. |
src/snapshot/aligned.rs |
Adds table protection, identity validation, and usage guidance. |
src/retry.rs |
Treats a zero connection budget as unlimited. |
src/protocol.rs |
Documents and tests per-change streamed xids. |
src/connection/libpq.rs |
Releases command registrations and adds connection regression tests. |
README.md |
Documents upgrade changes and safer snapshot merging. |
load-tests/Cargo.lock |
Refreshes load-test dependencies. |
integration-tests/reconnect_recovery.rs |
Tests slow reconnects and cancelled startup recovery. |
integration-tests/initial_snapshot.rs |
Waits for server state before cancelling snapshot creation. |
integration-tests/aligned_snapshot.rs |
Strengthens overlap checks and tests concurrent table changes. |
fuzz/Cargo.lock |
Updates the fuzzing smallvec dependency. |
examples/typed-deserialization/Cargo.lock |
Refreshes example dependencies. |
examples/tokio-spawn-streaming/Cargo.lock |
Refreshes example dependencies. |
examples/safe-transaction-consumer/Cargo.lock |
Refreshes example dependencies. |
examples/raw-xlogdata/Cargo.lock |
Refreshes example dependencies. |
examples/rate-limited-streaming/Cargo.lock |
Refreshes example dependencies. |
examples/polling/Cargo.lock |
Refreshes example dependencies. |
examples/pg-basebackup/Cargo.lock |
Refreshes example dependencies. |
examples/initial-snapshot/Cargo.lock |
Refreshes example dependencies. |
examples/derive-router/Cargo.lock |
Refreshes example dependencies. |
examples/binary-column-access/Cargo.lock |
Refreshes example dependencies. |
examples/basic-streaming/Cargo.lock |
Refreshes example dependencies. |
examples/arbitrary-fuzzing/Cargo.lock |
Updates the example’s smallvec dependency. |
Cargo.toml |
Raises pq-sys to 0.7.8. |
Cargo.lock |
Updates pq-sys and equivalent. |
.github/workflows/ci-checks.yml |
Adds Rust 1.87 checks and enforces example lockfiles. |
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Critical defects allow post-snapshot partition attachments to affect copied data and resumed bounded reads to report completion while leaving intervening transactions unread.
2 open issues
4 resolved since last review
Review effort: Balanced
connect_async continued to the next address after every failed attempt, so a
bad password was tried once per host and address (and twice that under
prefer-standby), where PQconnectdb tries it once. Against a PG17 server,
`host=P,P,P password=wrong` produced 3 failed logins per new() on this tree
and 1 with PQconnectdb; `dbname=nosuchdb host=P,P` made 2 attempts against 1.
libpq's rule (fe-connect.c, 14 to 18) is not "authentication failures": once
a connection to an address is up, any failure ends the connect, except
SQLSTATE 57P03 ("cannot connect now"), a target_session_attrs mismatch, a
connect_timeout expiry and a connection that never came up. An attempt that
fails while libpq was past the TCP connect and short of a completed startup
(no server_version ParameterStatus yet) now ends the walk, unless the server
said 57P03. A failed connection exposes no SQLSTATE, so 57P03 is recognised
by the server's wording ("the database system is ...", identical in 14 to
18); a server with a non-English lc_messages would end the walk there. The
final error is unchanged and stays Authentication for a bad password.
Measured against PG17 (primary, hot standby, hot_standby=off standby that
answers 57P03, dead port), 20 host-selection scenarios sync vs async: every
walking case (dead host first, standby/read-write/prefer-standby selection,
57P03 first, localhost, invalid name with hostaddr) is unchanged, and the
failed-login count drops from 3 to 1 per new().
Register a libpq command's socket with the runtime driving it
new() now runs IDENTIFY_SYSTEM through exec_async, which registered the
connection's AsyncFd on the runtime current at new() and kept it for the life
of the connection (at 0.9.0 registration happened in start()). A stream built
under one runtime and started under another then failed or hung: with the
first runtime dropped, start() returned "A Tokio 1.x context was found, but it
is being shutdown"; with it kept but never driven, start() waited forever for
a readiness event that its reactor would never deliver.
exec_async now registers the dup for its own command and releases it after,
unless a COPY already holds a registration. START_REPLICATION and COPY OUT
still register the long-lived one at their own entry point, so the
registration that serves reads is made by start() under the runtime that
starts the stream. The read and feedback paths are untouched.
5d1d2ed to
960b3d1
Compare


Summary
This PR fixes them, adds regression tests for each, and documents the behaviour changes for the next release.
What changed
main)stream.rs)timeout(d, es.next_event())withdshorter than a reconnect never recovers: the dropped future takes the half-built connection with it, and the next call dials again from scratch. Withd = 2 msand one walsender kill: 9,459 connections in 50 s (rustls), 5,454 (libpq), and no events after the fault (d= 50–500 ms: 0–3 events).recoveringflag is raised byrecover_connectionand lowered only on success.EventStream::next_event/next_event_beforethen park the stream in the poll futurepoll_nextalready uses, so the reconnect survives the caller's timeout.stream.rs)token.cancel()during a stuck reconnect waited out the connect attempt (2.8–3.8 s).libpq.rs)new()now runsIDENTIFY_SYSTEM, which registered theAsyncFdon the runtime current atnew()for the life of the connection.start()under another runtime failed (...is being shutdown) or hung (8 s).exec_asyncregisters the socket for its own command and releases it, unless a COPY holds a registration.START_REPLICATION/ COPY OUT still make the long-lived registration under the runtime that starts them.connection_timeout = ZERO(retry.rs)Timeout("0ns")after the first failed attempt, ignoringmax_attempts.stream.rs)start()/slot_confirmed_flush_lsn()/timeline_history(), every later command failed with a non-permanentProtocolerror, so a retry loop never rebuilt the stream.PermanentConnectionat the stream layer;next_event_with_retrystill reconnects.aligned.rs)TRUNCATE, table-rewritingALTER TABLEandDROP+CREATEare not MVCC-safe: committed after the consistent point, they makeCOPYreturn an empty table whilefinish()saysOk. Under a truncating writer, 60/60 captures were silently empty.capturetakesACCESS SHARE ... NOWAITon the requested tables, then fails if one changed since the snapshot (relfilenode/ name). Lock alone is not enough (58/60 still empty): there is a window between the consistent point and the lock.capture_aligned)server_identitydiffers (PermanentConnection).commit_lsnfilter is broken.next_row, and the xid semantics.checkonly, becauseproptest 1.12needs 1.88). Examples job uses--locked. Example, fuzz and load-test lockfiles refreshed.pq-sys0.7.6 → 0.7.8 (bindgen regeneration only).Behaviour changes to review
EventStream::inner(),inner_mut()andcurrent_lsn()panic until the nextnext_event()returns (documented; same class as an abandonedpoll_next). Feedback methods are unaffected.AlignedSnapshot::capturecan now return an error (a requested table changed since the snapshot, or a conflicting lock is held at that instant).ALTER TABLE,TRUNCATE,VACUUM FULLandCLUSTERon the requested tables wait untilfinish(), and writes queued behind them wait too (up to theirlock_timeout). Callers should retry a refused capture.PermanentConnectioninstead ofProtocol.connection_timeout = ZEROretries instead of failing on the first attempt.Type of change
Checklist
make before-git-pushpasses locally (check, build, format, audit, test, doc-check)--features rustls-tlsif connection/streaming code changed)