fix(deps): keep the backend alive when a Postgres server process dies

[yannickmonney]

What changed

postgres.js 3.4.7, our pin, crashes the backend process when a PostgreSQL server process dies under an open transaction or a reserved connection. 3.4.9, the latest release, does the same.

• Where it reaches: an open transaction means sql.begin, so also transactSerializable; a reserved connection means sql.reserve.
• What kills the server process: a crash, the OOM killer, an immediate shutdown, docker kill.
• The crash: an uncaught TypeError: Cannot read properties of null (reading 'write') at connection.js nextWrite. closed() nulls the socket, and then begin()'s ROLLBACK, or the reservation's next statement, is written to it from an Immediate.
• Upstream: Error when attempting to write to a null socket porsager/postgres#1154 and Duplicate folder name is correctly blocked, but the error message exposes internal server details #1208, and the guard PRs Unable to configure the default model user group/team #1168 and Customer list combines name and email in a single column instead of showing them separately #1209, all open.

This PR patches the pinned postgres@3.4.7 through the root patchedDependencies (bun patch, the option chosen on #4041). Each build the package ships gets the same change: src/ (import, Bun), cjs/src/ (require) and cf/src/ (workerd).

1. connection.js nextWrite: Guard nextWrite against a closed socket - fixes #1208 porsager/postgres#1209's guard, verbatim. A write to a closed connection rejects the pending statements with CONNECTION_CLOSED instead of throwing.
2. connection.js closed(), beyond Customer list combines name and email in a single column instead of showing them separately #1209 (marked Tale:):
   • It settles what is still pending after a socket error too.
   • It drops a cancelled write together with its timer.
   • It no longer leaves the settled statement in query.
3. index.js, beyond Customer list combines name and email in a single column instead of showing them separately #1209 (marked Tale:):
   • A transaction's or a reservation's statement runs only while it still holds its connection (c.reserved is still its own); otherwise it rejects with CONNECTION_CLOSED.
   • Statements queued in begin()'s or reserve()'s local queue reject when the connection closes.
   • release() changes nothing once the reservation no longer holds its connection.

Each "beyond #1209" hunk has a case in the regression suite that fails without it (see the map below).

Also in this PR:

• Dockerfiles. Every Dockerfile that installs from the root manifests copies patches/: the platform's workspace-deps and pruner stages, web, docs, ui-docs, ai-gateway and the plop service template. bun install refuses to run without the patch file.
• Turbo. services/platform/turbo.json hashes patches/** for test, and the turbo-inputs guard knows about it.
• Docs. patches/README.md records the why, the proof, the behaviour change and the removal condition. .agents/repo.md states the rule.

No uncaughtException filter: strict uncaught-exception handling is unchanged.

Remove the patch when a postgres.js release fixes the write to a closed connection, what closed() leaves pending, and the pool's hand-back of a closed connection. Then:

• bump the pin in services/platform and packages/shared;
• drop the patch and its patchedDependencies entry;
• keep connection-loss.test.ts, which must pass on that release unpatched.

Finding → commit → test

All tests are in services/platform/backend/db/connection-loss.test.ts. Each runs through both the ESM and the CJS build, and the suite asserts which file each build loaded.

#	Finding	Commit	Test
1	Uncaught TypeError when a transaction's server process dies (the issue)	c782a4d01	"rejects the transaction, survives its rollback, and the pool reconnects"
2	An idle transaction's next statement after the loss (crash); with #1209 alone, the closed connection is parked in full and a one-connection pool hangs	c782a4d01	"rejects the next statement of a transaction that lost its connection while idle"
3	A dead transaction's statement runs on the connection the pool reconnected and commits on its own (unpatched too; real PG left ['stray'] in the table)	c782a4d01	"does not run a dead transaction's statement on the connection the pool reconnected"
4	Reserved connection: crash on the next statement (porsager/postgres#1208); with #1209 alone, release() hands the dead connection back and every later statement fails	c782a4d01	"rejects a reserved connection's statements, and release() does not hand the dead connection back"
5	On a reset, ROLLBACK is written between the 'error' and 'close' events. closed(true) never settles it, so it takes the reconnected connection's first answer and the pool hangs. A cancelled write's timer also stayed set	47ba92bdd	"settles a transaction whose connection was reset, and the next connection answers its own statements"
6	Statements queued behind a busy connection never settle when it closes: a hang on the 57P01 path of transactSerializable's queued retry	47ba92bdd	"rejects the statements queued behind a busy connection when it closes"
7	sql.end() never resolves after a loss: the closed connection kept its settled statement. sandbox/service.ts ends a pool with no timeout	47ba92bdd	every case requires the ended pool to close by itself (under 900 ms of its 1 s timeout); real PG: end-after-kill9 and plain-kill9-end now resolve
8	release() after the loss cleared the next holder's c.reserved	47ba92bdd	by reading; the release case runs on one connection
—	Controls: 57P01 termination (the fake server now reads nothing after FATAL, as a real server process does) and a healthy commit and rollback		"settles a transaction an administrator terminated (FATAL 57P01)"; "still commits a healthy transaction and rolls back a failing one"

Findings 5–8 came from an independent read-only review pass on c782a4d01.

How the suite works. postgres.js runs in a child process (connection-loss.child.mjs) against a fake PG v3 server in the test. The server ends the socket the way the kernel does for a killed server process, or resets it. The cut cases run on a one-connection pool.

Each case asserts:

• the child printed nothing on stderr (Node 23+'s TimeoutNegativeWarning for postgres.js's reconnect timer is filtered);
• it ran every step and exited 0 by itself;
• each pending operation settled with a catchable error;
• no false success: the server saw no COMMIT and no stray statement;
• the pool still serves statements;
• the ended pool closed by itself.

Evidence

Same source and environment for all of these: Node 22.21.1 (the image's), and my own PostgreSQL 16.14 on a free local port with synthetic data only.

Committed suite (16 tests), with each postgres package swapped in:

postgres package	Result
unpatched (exactly d6dd53ae0's install)	12 failed / 4 passed
#1209's guard alone	10 failed / 6 passed
c782a4d01's patch	4 failed / 12 passed (the reset and queued cases)
this head's patch	16 / 16 passed

Real PostgreSQL, child-process rig with a real kill -9. Each scenario ran through both builds.

Scenario	Unpatched 3.4.7 (rounds u1–u3)	This head (round p7)
kill -9 of the transaction's own server process	uncaught TypeError at connection.js:250:22, exit 1	CONNECTION_CLOSED, pool reconnects, end in 3 ms
kill -9 of another session's server process (crash recovery)	crash, exit 1	CONNECTION_CLOSED, pool reconnects
idle transaction, next statement (also over 1024 B, one connection)	crash	CONNECTION_CLOSED, pool reconnects
idle transaction, pool reconnects for another caller, next INSERT	INSERT commits on its own; table holds ['stray']	rejected; table empty
sql.reserve, pending statement plus the next one; release on one connection	crash	CONNECTION_CLOSED, pool reconnects
sql.end() with no timeout after the loss	hangs (plain query) or crash	resolves
pg_terminate_backend (57P01); healthy commit and rollback	no crash; ok	no crash; ok

Shipped paths

• ESM and CommonJS. The suite resolves postgres by its bare name from the platform, through the package's exports. It asserts that ESM loaded node_modules/postgres/src/index.js and CJS node_modules/postgres/cjs/src/index.js. cf/src/ carries the same hunks, checked by marker and hash.

• Clean frozen install (git archive of each head, bun 1.4.2, --frozen-lockfile): exit 0 for both c782a4d01 and 47ba92bdd, and the six patched files are byte-identical to the patched worktree.

• Image install steps, replayed with the image's bun 1.3.12 from the files stage 1 COPYs, for both heads:

  • stage 1 (HUSKY=0 bun install): exit 0, patched;
  • the pruner (bun install --production on stage 1's manifests and lockfile): exit 0, patched;
  • stage 1 without patches/: exit 1, Couldn't find patch file: 'patches/postgres@3.4.7.patch'.

  Main's stage 1 rewrites bun.lock the same way ("Removed: 9", replayed at d6dd53ae0), so that predates this PR.

• CI at c782a4d01.

  • Build platform ran [workspace-deps 6/21] COPY patches/ ./patches/ and bun 1.3.12's install (1946 packages), then [pruner 2/18] COPY --from=workspace-deps /app/patches/ … and bun install --production (1689 packages).
  • Unit's @tale/platform:test was a cache miss, executing, with connection-loss.test.ts at 12/12.
  • The web, docs, ui-docs and ai-gateway container tests, and Backend integration, passed.

  CI on this head is reported in a PR comment.

Sweep and observations

• sql.begin users. transactSerializable (packages/shared/src/db/serializable.ts, 64 backend/lib files) and direct sql.begin calls, the migrations in backend/db/migrate.ts included.

• sql.reserve users. The knowledge corpus bootstrap (core/knowledge/ddl.ts: ROLLBACK after a failed statement, then release()), and transactSerializable's queued retry (reserved.unsafe('BEGIN' / 'ROLLBACK' / 'COMMIT')). Both take the paths fixed here.

• Retry classification, as observed; nothing changed.

  • isTransientDbError (packages/shared/src/db/retry.ts) does not retry CONNECTION_CLOSED, the code a killed server process produced in every real kill -9 run here. Its message regex wants "connection closed" with a space. So transactSerializable does not replay such a transaction, and a COMMIT with an unknown outcome is not rerun.
  • ECONNRESET (the 57P01 path, sometimes) is retried, as TALE-406's fast-shutdown rounds found safe.

• Migration-retry question (fix(platform): wait out a restarting database when the backend boots #4031): kept separate, observed live, not changed. isDatabaseUnavailable counts CONNECTION_CLOSED as unavailable (backend/db/unavailable.ts). I ran runBootMigrations from a scratch worktree with an uncommitted probe migration that kills its own server process (COPY … TO PROGRAM 'kill -9 $PPID'), on PG 16.14 with Node 22.21.1:

  Probe	Unpatched	Patched (c782a4d01)
  killed once	the boot process dies at the first cut (exit 1)	rerun after 1 s, applied on attempt 2, done in 9.5 s
  kills on every attempt	—	8 attempts in 70 s (delays 1, 2, 4, 5, 5, 5, 5 s), so 8 crash recoveries of the whole cluster; then rejected CONNECTION_CLOSED

  Each crash ended the session, so the session-level advisory lock went with it and each rerun took the lock again. Whether to cap the reruns after the lock is taken is fix(platform): wait out a restarting database when the backend boots #4031's decision.

• Not changed, upstream: a sql.reserve() waiting for a connection can be dropped when another connection closes and the pool reconnects that one for it. onclose hands the request to the reconnect, and the reconnect's type fetch discards it.

• Behaviour change: a statement issued on a transaction handle after its COMMIT, or on a released reservation, now rejects with CONNECTION_CLOSED instead of running on the pool's connection outside the transaction. A grep found no platform caller doing either.

• CI path filters (follow-up; this token has no workflow scope). build.yml's paths, checks.yml's Backend integration filter and security.yml do not list patches/**, so a later edit to the patch alone would skip the image builds and the real-Postgres suite. This PR also changes package.json and bun.lock, so it is not affected.

• Docs. No page claims the backend crashes or survives a database crash, and the knowledge-Postgres crash page is about BM25 index damage.

• Security: no new surface. The ownership rule stops a dead transaction's statement from running outside it.

• Locales, migrations, UI: none.

• Manual tests: none; the suite owns this.

• Open PRs: none touches these files.

Checks run locally

• connection-loss.test.ts: 16/16 on Node 24.21, and 16/16 in the Node 22.21.1 matrix.
• tests/guards/dockerfile-fail-closed.guard.test.ts, tests/guards/turbo-inputs.guard.test.ts (32/32), backend/db/migrate.test.ts and backend/db/unavailable.test.ts: pass (at c782a4d01; the follow-up touches none of their inputs).
• oxlint --type-aware on the suite and the child: 0 problems.
• Scoped tsc --noEmit: 0 errors.
• oxfmt on every changed file it formats, commitlint on both commits, and opengrep on the new files: 0 findings.
• Not run here: the whole platform suite, whole-workspace tsc/lint, knip and bun run check. CI runs them.

Closes #4041. Refs #4031.

[yannickmonney]

CI on the exact head 47ba92bdda862a392387047d5e2948f62bcb22de: 63 passed, 11 skipped, 1 failed.

Passed:

• Unit: @tale/platform:test was a cache miss, executing, and ✓ server backend/db/connection-loss.test.ts (16 tests) ran.
• Build platform, Smoke test, Validate images and Scan platform.
• Backend integration.
• Type check, Lint, Format, Knip and UI.
• The web, docs and AI-gateway container tests.

Failed: UI docs container test (job 110637687411), cancelled; not caused by this change.

• The ui-docs image's Vite SSR build stalled right after ✓ 2295 modules transformed (23:50:49Z). Nothing more was logged until the step was cancelled at 00:03:44Z ("The operation was canceled").
• On c782a4d01, the same check, with the same ui-docs Dockerfile and app sources, passed in about 3 minutes (job 110627850731, 23:18:44–23:21:54Z).
• 47ba92bdd changes only patches/postgres@3.4.7.patch, patches/README.md and the platform's connection-loss suite. ui-docs does not use postgres.js.

My dispatch allows no re-run of this check from me, so I leave it to the reviewer or a maintainer.

[yannickmonney]

Independent review: accepted at tree aa5a2c50990c8407b24e6816da47b49dab084253 (head 609bf018d9476b89799b1ee4f8bb9c1eb03366be, an empty re-run commit on 47ba92bdd)

• Reviewer: Tale agent update dashboard ui (#508) #2 3f9fdcee-79e7-430b-a430-ba179d5b5132, run 83ca77ca-f3c4-4a41-bb14-090024a62d3e (TALE-359 review-merge occurrence, 2026-10-02 03:45 Europe/Zurich).
• Implementation: agent optimize rag action #4 5f5307c9-4cbd-416a-a456-8bb667fc584d, run 3890e406. The empty commit is the manager's, run 4e5d2018. The reviewer is distinct.
• Merge: it merges cleanly with main 05543548e.
• A read-only review agent ported the fake server, rebuilt the swap matrix and ran one-change-removed mutations. I re-ran what I relied on.

Verdict: the fix is right and needed. In a real kill -9 run, unpatched 3.4.7 kills the Node process; the patched build survives and the pool recovers. All P3s are latent or test-side, so I'm merging.

Evidence

• Real Postgres 16 (my private cluster): I ran a pool with a live transaction, a reserved connection and a query loop, then kill -9'd the backend serving the transaction.
  • Unpatched (main's 3.4.7): UNCAUGHT Cannot read properties of null (reading 'write'), the crash in bug(platform): a Postgres server process that dies under an open transaction crashes the backend (postgres.js null-socket write) #4041.
  • Patched: the process stays alive.
    • The transaction and the reserved statement reject with CONNECTION_CLOSED.
    • The loop rides out crash recovery: 20 errors (CONNECTION_CLOSED, 57P03), 155 successes.
    • A fresh query after recovery succeeds.
• The suite: connection-loss.test.ts passes 16/16 locally, on Node 24. With unpatched 3.4.7 swapped in, 12 fail and 4 pass, the author's figure exactly. The review agent reproduced the whole matrix: Customer list combines name and email in a single column instead of showing them separately #1209 alone 10/6, c782a4d01 4/12, head 16/16.
• The patch applies to the installed 3.4.7: all six pre-image hashes match. src/, cjs/src/ and cf/src/ get identical changes. The Customer list combines name and email in a single column instead of showing them separately #1209 guard is verbatim. The installed package carries the Tale: hunks.
• Ownership rule: c.reserved is set only when BEGIN runs or a reservation starts, and cleared on close, on idle after COMMIT/ROLLBACK, or on release.
  • Savepoints, pipelined statements and statements issued after awaiting other I/O still pass.
  • A dead transaction's or reservation's statements can no longer reach the reconnected connection.
• Images: every root-lockfile install copies patches/ before bun install (platform stage 1 and pruner, web, docs, ui-docs, ai-gateway, the plop template).
  • sandbox has its own lockfile; sandbox-runtime installs a skill's own package.
  • No .dockerignore excludes patches/.
  • postgres@3.4.7 exists only in the root lockfile.
• CI at the exact head (merge ref onto 05543548e):
  • All seven workflows succeeded: Checks 36949340275, Build 36949340224, E2E 36949340264, CLI 36949340228, SAST 36949340226, Security 36949340237 and Commitlint 36949340225.
  • 64 checks succeeded, among them Build platform, Smoke test, Validate images, the AI gateway, Docs, UI docs and Web container tests, Bun audit, and Backend integration (1264/1264 checks across 211/211 lanes).
  • Not counted as success: 1 neutral Trivy, and 10 skipped by design (7× Candidate gate, Attach to release, the 2 fork-PR jobs).
  • Unit replayed @tale/platform:test hash c8ddae65a. It executed with identical inputs in job 110637602498 at 47ba92bdd, on the same main: 996 files, connection-loss 16/16.

Non-blocking (P3), for follow-up

1. A connection lost during a .cursor() callback crashes the process, because of the patch's own query = null hunk.

   • Where: closed() nulls query. Upstream's unchanged PortalSuspended (connection.js:839-848) then reads query.cursorRows and query.portal after its await, and its catch calls query.reject on null.
   • Proved: the head's build exits 1 with TypeError … reading 'reject'. Customer list combines name and email in a single column instead of showing them separately #1209 alone, and the fix below, exit 0. Unpatched 3.4.7 also crashes, through nextWrite.
   • Exposure: no platform or package code calls .cursor(), which I checked by grep. Fix it before anything does.
   • Fix, checked by the review agent in all three builds: in PortalSuspended, keep const q = query before the await, return if q !== query afterwards, and use q.reject in the catch.

2. The suite pins 5 of the patch's 9 changes. With one change removed at a time, these four still pass 8/8 ESM:

   • the Customer list combines name and email in a single column instead of showing them separately #1209 guard (no tested path reaches it now);
   • dropping the stale write timer;
   • query = null: only sql.end({timeout:1}) gets slow, 1001 ms;
   • the reserve() ownership check: without it, a dead reservation's statement runs inside another caller's transaction. The fake server logged begin, tx_one, stray_r, tx_two, commit.

   That last one is the cross-request property this patch exists for. Add a reserve() variant of the "idle transaction reconnected" case, an end() timing case, and a cursor case with the fix.

3. A misleading error code. A manual COMMIT inside sql.begin, or a statement after COMMIT or on a released reservation, now rejects CONNECTION_CLOSED. The platform reads that as "database unavailable" (backend/db/unavailable.ts:44): a 503 with no error report. A use-after-commit bug would hide as an outage. Today, no manual COMMIT and no reuse after release exist, which I checked by grep. A distinct code would be safer.

4. Upstream: cursor errors mid-batch are swallowed, so a for await silently ends early. Not introduced here.

5. Small timing margin in the suite: about 80 ms between the server's close and the child's sleep (connection-loss.test.ts:115, child.mjs:88).

Also known, and the author's to route: patches/** is missing from the build.yml / checks.yml path filters (a maintainer item, workflow scope), and the reserve-waiter residual in patches/README.md.

Not covered: no production contact. I didn't drive a browser, because the change has no UI surface.