Settle startup queries when their connection keeps closing - fixes #1223, fixes #1193 - #1241
JakeThomson wants to merge 2 commits into
Conversation
When a connection closes while its first query is parked as initial, closed() reconnects without clearing the dead backend's query and error response, and without any delay: - a backend terminated during the array type fetch replays its error onto the next connection: the parked query fails with it, and the internal fetch rejects with nothing handling it (porsager#1223) - a peer that closes cleanly during startup makes the connection reconnect with zero delay forever, and the query never settles (porsager#1193) - a peer that fails every array type fetch, as PgBouncer does during server_login_retry, must still settle the query with its own error within connect_timeout once the replay is gone - a failed array type fetch on a live connection is an unhandled rejection All four fail on master.
…rsager#1223, fixes porsager#1193 closed() reconnected a connection with a parked initial query before clearing the dead backend's query and errorResponse, so the next connection's first ReadyForQuery replayed them (porsager#1223). It also skipped the backoff, so a peer that kept closing spun zero-delay reconnects that neither connect_timeout nor max_lifetime could bound (porsager#1193). Clear the stale state and pace these reconnects with the shared backoff. The run of consecutive closes is bounded by connect_timeout, after which the parked query rejects with the last error the peer sent, or with CONNECTION_CLOSED. Any successful connect resets the run, including a reserve() whose initial is dropped before the array type fetch. Clearing the state alone is not enough: the replay was what settled the query after a server error followed by a close, so without the bound that case becomes the porsager#1193 loop. fetchArrayTypes() is now caught as well. If it fails on a live connection, errored() has already settled initial, and connected() re-arms needsTypes on the next connect.
cat-a-guerra
left a comment
There was a problem hiding this comment.
Reviewed and tested — this looks right to me, and it fixes both cases on my side. Ran the two standalone scripts (no Postgres needed, Node 24.19.0, max: 1, connect_timeout: 10):
| Scenario | master | this branch |
|---|---|---|
Peer fails every array-type fetch (FATAL 08P01 + close) |
rejected 08P01 in 9ms — via the stale replay this PR removes |
rejected 08P01 after 8.95s, 7 attempts |
| Clean close during startup (#1193) | never settled after 15s, 7,676 connection attempts | rejected CONNECTION_CLOSED after 6.9s, 7 attempts |
Matches the numbers in your description. I also like that the parked query now rejects with the peer's own error rather than a generic CONNECTION_CLOSED — the wrong-error replay is what made this hard for us to diagnose in the first place.
I traced the closeRunStart reset paths looking for a stale marker leaking into a later connection (a run that starts during one flap and then bounds an unrelated later one) and did not find a case: the marker is only set in closed() when initial is live, and every path that reaches a healthy ReadyForQuery clears it — the initial.reserve branch before the type fetch, and the options.shared.retries = retries = closeRunStart = 0 line after it.
Two minor notes, neither blocking:
-
options.connect_timeout || 30and an explicitly disabled timeout.timer()treats a falsysecondsas "no timer" (if (!seconds) return { cancel: noop, start: noop }), soconnect_timeout: 0today means no connect timeout at all. With this change that configuration gets a 30-second bound on the startup-close path. That seems like the better behaviour to me — unbounded is the #1193 hang — but it may be worth honouring0as unbounded, or calling the change out in the README next toconnect_timeout. -
The budget's delay comes from the pool-wide retry count.
delayis derived fromoptions.shared.retries, which every connection increments, whilecloseRunStartis per connection. So on a pool where several connections flap at once, an individual connection's run can exhaust theconnect_timeoutbudget in fewer attempts than it would alone. Probably what you want (a pool-wide problem should give up sooner), just noting the asymmetry in case it surprises someone reading the bound as per-connection.
The tests read well: reusing the suite's connect_timeout: 1 keeps the two bounded cases at about a second instead of 30, and attempts < 10 is the right kind of smoke bound given master produces thousands. Good call covering the non-fatal fetch error too — that path was not in our patch and would have bitten us eventually.
One optional addition: Failed array type fetch on a live connection is not an unhandled rejection asserts the absence of the rejection, but not the claim in the description that the connection stays usable. If the fake peer answered the follow-up query rather than socket.end() on Terminate, that test could also assert the query succeeds — the difference between "no crash" and "still works".
Thanks for picking this up and for the thorough tests.
Fixes #1223
Fixes #1193
When a connection closes while its first query is parked as
initial,closed()returnsreconnect()before clearing the dead backend'squeryanderrorResponse, and before applying any backoff. The next connection's firstReadyForQueryreplays the stale error: the parked query fails with an error from a backend it never ran on, and the internal array type fetch rejects with nothing handling it (#1223). A peer that closes without an error spins zero-delay reconnects thatconnect_timeoutandmax_lifetimecannot bound, and the query never settles (#1193).This clears the stale state and paces those reconnects with the shared backoff. The run of consecutive closes is bounded by
connect_timeout, after which the parked query rejects with the last error the peer sent, orCONNECTION_CLOSED. Any successful connect resets the run, including areserve()whoseinitialis dropped before the type fetch. The two have to land together: the replay was what settled the query after a server error followed by a close, which PgBouncer does throughoutserver_login_retry, so clearing it alone turns that case into the #1193 loop.fetchArrayTypes()is also caught now, for a failure on a live connection, whereerrored()has already settledinitial.This follows the suggested fix in #1223 and the time-budget bound from the #1193 thread. Thanks to @cat-a-guerra and @sen-pac for the analysis.
target_session_attrsscan that only ever sees closes now also stops afterconnect_timeoutinstead of scanning indefinitely. The existingMultiple hoststest passes.