Skip to content

(janitor/flaky-test) fix unhandled pg Pool error in admin-permissions-migration test - #5769

Merged
pandemicsyn merged 1 commit into
mainfrom
janitor/flaky-test-admin-permissions-migration
Aug 31, 2026
Merged

(janitor/flaky-test) fix unhandled pg Pool error in admin-permissions-migration test#5769
pandemicsyn merged 1 commit into
mainfrom
janitor/flaky-test-admin-permissions-migration

Conversation

@kilo-code-bot

@kilo-code-bot kilo-code-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Flaky test

apps/web/src/lib/admin/admin-permissions-migration.test.ts > admin permissions migration › backfills only admins that matched the previous grant-authority predicate

Evidence

  • Failed in CI run 33368508201 (sha 9293124762296beb3f614efa2cb365260a46644f) with:
    FAIL src/lib/admin/admin-permissions-migration.test.ts (23.551 s)
      ● admin permissions migration › backfills only admins that matched the previous grant-authority predicate
    
        Unhandled error. (error: terminating connection due to administrator command
          ...
          code: '57P01', ... routine: 'ProcessInterrupts',
          client: Client { ... database: 'admin_permissions_migration_02ad7696cd8444d3a5ad410a71c358e1', ... }
    
  • Reviewed every other test-job failure across the last 120 failing CI runs on this repo (gh run list --workflow=CI, 08-29 through 08-31). All other test/typecheck/lint failures were either the dependency-cycle-check (madge) step misreported under the test job name, or typecheck/lint failures — none were a second occurrence of this or any other named-test failure, i.e. this test fails intermittently and is the only reproducible named-test flake in that window (frequency: 1 observed failure across ~890 test-job runs of this suite in the sampled window).
  • The rest of the suite (890 other test files, 12,847 tests) passed in the same run; only this one test failed, and the failure is a Postgres FATAL: terminating connection due to administrator command (57P01) surfacing as a Jest "Unhandled error" rather than a normal assertion failure — i.e. the test's own logic never disagreed with expectations.

Root cause

This test:

  1. Creates a scratch database and a testPool = new Pool(...) connected to it.
  2. In its finally block, calls await testPool.end() and then immediately DROP DATABASE ... WITH (FORCE) against that same database.

Pool#end() resolves once the underlying client sockets are told to close, but does not guarantee the server has finished processing that close. DROP DATABASE ... WITH (FORCE) force-terminates any backend still associated with the target database at the moment it runs. When the socket teardown and the forced termination race — which depends on runner/network timing and is therefore nondeterministic — the termination FATAL can still arrive on a socket the pool considers already closing. pg-pool re-emits that as an 'error' event on the Pool. Neither adminPool nor testPool in this test had an 'error' listener attached, so Node treats the event as an unhandled error, which Jest attributes to whichever test happens to be executing when it lands — producing an intermittent, timing-dependent failure unrelated to the test's assertions.

This is exactly the scenario the app's own primary/replica pools already guard against in apps/web/src/lib/drizzle.ts:

// In test mode neither exits: `pool.end()` during cleanup emits idle-client
// errors, and a listener must stay attached so Node does not treat the emit as
// an unhandled throw.

This test's ad-hoc pg.Pool instances never got the same treatment.

Fix

Attach an 'error' listener (log-and-continue, matching the production pattern) to both adminPool and testPool in this test. This does not change what the test asserts, add retries/timeouts/sleeps, or touch production code — it only ensures that a raced, already-expected idle-client error during teardown surfaces as a handled event instead of crashing the test run.

Why this is deterministic

The fix does not depend on timing: an 'error' listener on a Pool is always safe to attach and always prevents the "unhandled error" crash regardless of whether the race actually happens on a given run. The test's real assertions (migration backfill behavior) are unchanged and continue to run to completion before the pools are torn down.

Validation

  • pnpm exec oxlint --config .oxlintrc.json apps/web/src/lib/admin/admin-permissions-migration.test.ts → 0 warnings/errors.
  • pnpm exec oxfmt --list-different apps/web/src/lib/admin/admin-permissions-migration.test.ts → no diff.
  • apps/web: pnpm exec tsgo --noEmit → clean.
  • This sandbox cannot start the local Docker Postgres/Redis services (per environment constraints), so the migration test itself could not be executed here. The change is additive (event listeners only) and does not alter any awaited control flow, so it cannot change the test's pass/fail outcome when the race does not occur, and it converts the race outcome into a handled no-op instead of a crash when it does.

…-migration test

The migration test creates its own `pg.Pool` instances and force-drops its
scratch database in the `finally` block. `Pool#end()` resolves once the
client sockets are told to close, but the server's `DROP DATABASE ... WITH
(FORCE)` termination can still land on a socket that has not finished
closing. node-postgres re-emits that as an 'error' event on the pool, and
with no listener attached Node treats it as an unhandled error, failing
whichever test happens to be running when it surfaces.

The app's own pools in drizzle.ts already attach an 'error' listener for
exactly this reason (see the comment there). Do the same for the pools
this test creates.
@kilo-code-bot kilo-code-bot Bot added the janitor Automated dead-code/duplication cleanup label Aug 31, 2026
@kilo-code-bot

kilo-code-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (1 files)
  • apps/web/src/lib/admin/admin-permissions-migration.test.ts

Reviewed by grok-4.6 · Input: 56K · Output: 4.8K · Cached: 97.5K

Review guidance: REVIEW.md from base branch main

@pandemicsyn
pandemicsyn merged commit dbe8aa2 into main Aug 31, 2026
17 checks passed
@pandemicsyn
pandemicsyn deleted the janitor/flaky-test-admin-permissions-migration branch August 31, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

janitor Automated dead-code/duplication cleanup

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant