Skip to content

Settle startup queries when their connection keeps closing - fixes #1223, fixes #1193 - #1241

Open
JakeThomson wants to merge 2 commits into
porsager:masterfrom
JakeThomson:fix-startup-reconnect
Open

JakeThomson wants to merge 2 commits into
porsager:masterfrom
JakeThomson:fix-startup-reconnect

Conversation

@JakeThomson

Copy link
Copy Markdown

Fixes #1223
Fixes #1193

When a connection closes while its first query is parked as initial, closed() returns reconnect() before clearing the dead backend's query and errorResponse, and before applying any backoff. The next connection's first ReadyForQuery replays the stale error: the parked query fails with an error from a backend it never ran on, and the internal array type fetch rejects with nothing handling it (#1223). A peer that closes without an error spins zero-delay reconnects that connect_timeout and max_lifetime cannot bound, and the query never settles (#1193).

This clears the stale state and paces those reconnects with the shared backoff. The run of consecutive closes is bounded by connect_timeout, after which the parked query rejects with the last error the peer sent, or CONNECTION_CLOSED. Any successful connect resets the run, including a reserve() whose initial is dropped before the type fetch. The two have to land together: the replay was what settled the query after a server error followed by a close, which PgBouncer does throughout server_login_retry, so clearing it alone turns that case into the #1193 loop. fetchArrayTypes() is also caught now, for a failure on a live connection, where errored() has already settled initial.

This follows the suggested fix in #1223 and the time-budget bound from the #1193 thread. Thanks to @cat-a-guerra and @sen-pac for the analysis.

When a connection closes while its first query is parked as initial,
closed() reconnects without clearing the dead backend's query and error
response, and without any delay:

- a backend terminated during the array type fetch replays its error
  onto the next connection: the parked query fails with it, and the
  internal fetch rejects with nothing handling it (porsager#1223)
- a peer that closes cleanly during startup makes the connection
  reconnect with zero delay forever, and the query never settles (porsager#1193)
- a peer that fails every array type fetch, as PgBouncer does during
  server_login_retry, must still settle the query with its own error
  within connect_timeout once the replay is gone
- a failed array type fetch on a live connection is an unhandled
  rejection

All four fail on master.
…rsager#1223, fixes porsager#1193

closed() reconnected a connection with a parked initial query before
clearing the dead backend's query and errorResponse, so the next
connection's first ReadyForQuery replayed them (porsager#1223). It also skipped
the backoff, so a peer that kept closing spun zero-delay reconnects that
neither connect_timeout nor max_lifetime could bound (porsager#1193).

Clear the stale state and pace these reconnects with the shared backoff.
The run of consecutive closes is bounded by connect_timeout, after which
the parked query rejects with the last error the peer sent, or with
CONNECTION_CLOSED. Any successful connect resets the run, including a
reserve() whose initial is dropped before the array type fetch.

Clearing the state alone is not enough: the replay was what settled the
query after a server error followed by a close, so without the bound
that case becomes the porsager#1193 loop.

fetchArrayTypes() is now caught as well. If it fails on a live
connection, errored() has already settled initial, and connected()
re-arms needsTypes on the next connect.

@cat-a-guerra cat-a-guerra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and tested — this looks right to me, and it fixes both cases on my side. Ran the two standalone scripts (no Postgres needed, Node 24.19.0, max: 1, connect_timeout: 10):

Scenario master this branch
Peer fails every array-type fetch (FATAL 08P01 + close) rejected 08P01 in 9ms — via the stale replay this PR removes rejected 08P01 after 8.95s, 7 attempts
Clean close during startup (#1193) never settled after 15s, 7,676 connection attempts rejected CONNECTION_CLOSED after 6.9s, 7 attempts

Matches the numbers in your description. I also like that the parked query now rejects with the peer's own error rather than a generic CONNECTION_CLOSED — the wrong-error replay is what made this hard for us to diagnose in the first place.

I traced the closeRunStart reset paths looking for a stale marker leaking into a later connection (a run that starts during one flap and then bounds an unrelated later one) and did not find a case: the marker is only set in closed() when initial is live, and every path that reaches a healthy ReadyForQuery clears it — the initial.reserve branch before the type fetch, and the options.shared.retries = retries = closeRunStart = 0 line after it.

Two minor notes, neither blocking:

  1. options.connect_timeout || 30 and an explicitly disabled timeout. timer() treats a falsy seconds as "no timer" (if (!seconds) return { cancel: noop, start: noop }), so connect_timeout: 0 today means no connect timeout at all. With this change that configuration gets a 30-second bound on the startup-close path. That seems like the better behaviour to me — unbounded is the #1193 hang — but it may be worth honouring 0 as unbounded, or calling the change out in the README next to connect_timeout.

  2. The budget's delay comes from the pool-wide retry count. delay is derived from options.shared.retries, which every connection increments, while closeRunStart is per connection. So on a pool where several connections flap at once, an individual connection's run can exhaust the connect_timeout budget in fewer attempts than it would alone. Probably what you want (a pool-wide problem should give up sooner), just noting the asymmetry in case it surprises someone reading the bound as per-connection.

The tests read well: reusing the suite's connect_timeout: 1 keeps the two bounded cases at about a second instead of 30, and attempts < 10 is the right kind of smoke bound given master produces thousands. Good call covering the non-fatal fetch error too — that path was not in our patch and would have bitten us eventually.

One optional addition: Failed array type fetch on a live connection is not an unhandled rejection asserts the absence of the rejection, but not the claim in the description that the connection stays usable. If the fake peer answered the follow-up query rather than socket.end() on Terminate, that test could also assert the query succeeds — the difference between "no crash" and "still works".

Thanks for picking this up and for the thorough tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants