From ca6942072374cef4aa2d6f7f43d78f0e733da07f Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Sat, 24 Oct 2026 17:56:06 -0700 Subject: [PATCH 1/2] docs: explain why pg-sprite requires session affinity Every pool proves session affinity and refuses transaction- and statement-mode poolers, but no doc explained why or how to connect. --- docs/README.md | 1 + docs/session-affinity.md | 180 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 181 insertions(+) create mode 100644 docs/session-affinity.md diff --git a/docs/README.md b/docs/README.md index b682d65a..0dc7a147 100644 --- a/docs/README.md +++ b/docs/README.md @@ -50,6 +50,7 @@ Aurora-only. Why that combination is the product is [vision.md](vision.md); star | [lint-report.md](lint-report.md) | The **lint report contract** — the versioned JSON shape `pg-sprite lint` emits for offline CI gating: finding fields (verbatim SQL, line/column), the codes table, severities and exit behavior, the offline-conservatism rules, and how the contract versions relative to the plan report. | | [suggest-report.md](suggest-report.md) | The **suggest report contract** — the versioned JSON shape `pg-sprite suggest` emits for offline advice: the typed caveat vocabulary (what changes about how you must run a safer form, and what a failed step leaves behind), the typed guidance codes for rewrites the planner cannot construct, and the operation → safer form → caveats table (pinned by test). | | [progress-report.md](progress-report.md) | The **progress report contract** — the versioned JSON snapshot a caller receives when polling a running change through the `*WithProgress` entry points: phases and operations vocabularies, the terminal-freeze rule, server-observed and engine-measured work counters, and polling semantics (pinned by test). | +| [session-affinity.md](session-affinity.md) | **Why pg-sprite needs one PostgreSQL session per connection** — the session state it depends on (timeouts, the table lock, a catalog-first `search_path`), what breaks behind a transaction- or statement-mode pooler, the session-affinity proof every pool runs, which endpoints pass, and how to connect. | | [engine-role.md](engine-role.md) | The **engine-role provisioning contract** — the tiered minimum access a PostgreSQL user needs to run schema changes against tables it does not own: role membership for owner-gated DDL, schema `CREATE` for index builds and shadow objects, `SET ROLE` for owner-correct shadow creation, replication access for CDC, and the explicit list of powers the engine role must *not* have. Preflight refusals name the missing `GRANT` and point here. | | [invalid-index-recovery.md](invalid-index-recovery.md) | The **operator runbook** for the native path's invalid-index outcomes — an invalid index the executor found or left. How the typed states are told apart, which ones `RebuildAbandonedIndex` recovers on its own (and the lock-and-identity proof that makes its drop safe), when the entry may be another actor's healthy in-flight build, and what to check when the executor could prove nothing. | | [testing.md](testing.md) | The **test-suite guide** — how to run the suite (unit, per-major, all supported majors, compose database), current coverage, the remaining executor-phase test obligations, and the vanilla-PostgreSQL-matrix vs real-Aurora validation boundary. | diff --git a/docs/session-affinity.md b/docs/session-affinity.md new file mode 100644 index 00000000..6b4bbc03 --- /dev/null +++ b/docs/session-affinity.md @@ -0,0 +1,180 @@ +# Session affinity: why pg-sprite needs one PostgreSQL session per connection + +**pg-sprite requires that every connection it opens stays on one PostgreSQL server session +for as long as the connection is open.** That holds for a direct connection and for a pooler +in session mode. It does not hold behind a transaction-mode or statement-mode pooler, and +pg-sprite refuses to run there before it executes anything. This page explains what depends +on session affinity, how pg-sprite proves it, what it refuses, and how to connect. + +## Table of contents + +- [Is it mandatory?](#is-it-mandatory) +- [What depends on one session](#what-depends-on-one-session) +- [What breaks without it](#what-breaks-without-it) +- [How pg-sprite proves it](#how-pg-sprite-proves-it) +- [Which endpoints pass](#which-endpoints-pass) +- [How to connect](#how-to-connect) +- [What the proof does not cover](#what-the-proof-does-not-cover) +- [Where it is enforced](#where-it-is-enforced) + +## Is it mandatory? + +Yes, for every command that connects to a database. `migrate` (including `--dry-run` and +`--desired`), `diff`, `pull` and `status` all build their pool through `dbconn.NewPool`, which +proves session affinity before it returns and fails if the proof fails. There is no flag or +option that skips the proof. The commands that never connect (`fmt`, `lint`, `suggest`, +`capabilities`) don't need it. + +A library caller is held to the same rule: [`migrate.Run`](../pkg/migrate/migrate.go) and the +other entry points take the pool `dbconn.NewPool` returns, and the table lock session proves +affinity again on its own connection. + +## What depends on one session + +Each item below is state PostgreSQL keeps **per server session**, not per client connection. +If the connection's next statement lands on a different backend, the state is not there. + +| Session state | What pg-sprite uses it for | Invariant | +|---|---|---| +| `lock_timeout` and `statement_timeout`, sent as startup parameters and read back by `NewPool` | Bound every lock wait and every statement, so a queued `ALTER` can never sit at the head of the lock queue and block every reader and writer of the table | [LK-2](invariants.md#lk-2--exactly-one-access-exclusive-window-and-every-strong-lock-is-bounded) | +| A session-level advisory lock on a dedicated connection (`pg_try_advisory_lock(0x70677370, hashtext(…))`) | Make sure at most one pg-sprite run works on a table at a time | [LK-1](invariants.md#lk-1--at-most-one-migration-runs-per-table) | +| A catalog-first `search_path`, set once per physical connection | Stop a user schema from shadowing `pg_class`, `hashtext` and other catalog names in the engine's unqualified reads | [CO-9](invariants.md#co-9--decision-reads-resolve-to-the-real-catalog) | + +`SET LOCAL` work inside a transaction (the role switch and guards in a chunk copy, for +example) needs only the transaction, not the session. The three rows above are why the +session itself has to be stable. + +## What breaks without it + +Behind a **transaction-mode** pooler, the client connection is bound to a backend only for the +length of one transaction. Between transactions the pooler can hand the next statement to any +backend. Example: one pg-sprite connection whose session state landed on backend A, followed +by an `ALTER` in its next transaction: + +```diagram + client connection pooler (transaction mode) backends + ───────────────── ───────────────────────── ──────── + session state is set: ──▶ lands on backend A ──▶ A: lock_timeout=3s, + timeouts, search_path catalog-first path + SELECT pg_try_advisory_lock ──▶ runs on backend A ──▶ A: holds the table lock + → true + (idle between transactions) A goes back to the pool; + another client now runs on + A and "holds" pg-sprite's lock + ALTER TABLE ... (next tx) ──▶ runs on backend B ──▶ B: lock_timeout=0 + (server default: wait + forever) +``` + +(pg-sprite sends its timeouts as startup parameters. A transaction-mode PgBouncer rejects +startup parameters it doesn't track, or is configured to drop them; `NewPool` reads the +values back and refuses a connection that doesn't carry them, with +`ErrSessionSettingsDiscarded`. The proof below covers the case where the settings did land, +but only on one backend.) + +Three things went wrong, and none is visible to the client: + +1. **The `ALTER` runs unbounded.** Backend B never saw `lock_timeout`, so if the table is busy + the `ALTER` waits at the head of the lock queue indefinitely, and every query on the table + queues behind it. That is the outage the timeouts exist to prevent. +2. **The table lock no longer excludes anyone.** The advisory lock lives on backend A, which + now serves another client. A second pg-sprite run whose statement lands on A sees the lock + as its own, and two runs can work on the same table at once. +3. **Catalog reads may resolve differently.** Backend B may carry a `search_path` that lets a + user schema shadow catalog names. + +Behind a **statement-mode** pooler it is worse: every statement can land on a different +backend, and the pooler refuses multi-statement transactions outright, which pg-sprite needs +for every chunk copy, guarded write, checkpoint save and cutover. + +## How pg-sprite proves it + +Reading back `lock_timeout` is not enough on its own: an idle transaction-mode pooler may hand +the same backend back twice and report the settings healthily. So after the read-back, +`NewPool` forces the rebind a pooler would eventually make, and checks for it: + +```diagram + conn (one of the pool's connections) second connection (throwaway pool of 1) + ──── ───────────────────────────────────── + t1 pg_try_advisory_lock(random key) → true + t2 BEGIN (pins a backend for the whole tx) + t3 pg_try_advisory_lock(same key) + healthy: false (conn's session holds it) + t4 does pg_locks show my own backend + as the holder of the key? + healthy: yes + t5 ROLLBACK + t6 pg_advisory_unlock(key) → true + healthy: true (conn is still the session that took it) +``` + +- **Direct connection or session-mode pooler:** `conn` is one session throughout, so every + reading is the healthy one, and there are no false positives to trade off. +- **Transaction-mode pooler:** the open transaction at t2 takes the backend `conn` used at t1, + so `conn`'s next statement lands elsewhere. At least one reading fails: the second + connection takes a lock `conn` "holds", `conn` is no longer the holder in `pg_locks`, or + `conn` can't release what it took. `NewPool` returns `ErrNoSessionAffinity` with the remedy. +- **Statement-mode pooler:** the pooler refuses the `BEGIN` at t2, and `NewPool` fails before + any work runs. +- The key is freshly random on every call, so concurrent proofs never contend and a probe lock + stranded on a pooled backend can't block anything later. +- The proof needs **one spare connection** to the server, or pooler, beyond the pool's own. + +The same proof runs on the table lock's dedicated connection before the lock is attempted +(`dbconn.AcquireTableLock`), so the lock is never taken on a connection that can't keep it. + +## Which endpoints pass + +| Endpoint | Result | +|---|---| +| Direct connection to PostgreSQL (including Aurora/RDS writer endpoints) | Passes | +| PgBouncer, or another pooler, in **session** mode | Passes. The proof needs one spare server connection in the pooler's pool. | +| PgBouncer in **transaction** mode | Refused with `ErrNoSessionAffinity` | +| PgBouncer in **statement** mode | Refused before any work runs: PgBouncer rejects the proof's `BEGIN` | +| Supabase Supavisor session endpoint | Passes ([supabase.md](supabase.md)) | +| Supabase Supavisor transaction endpoint | Refused with `ErrNoSessionAffinity`, even with named prepared statements disabled ([supabase.md](supabase.md)) | + +The transaction-mode and session-mode PgBouncer results are tested against a real PgBouncer +([testing.md](testing.md)). + +## How to connect + +- **Point pg-sprite at a direct endpoint or a session-mode endpoint.** Applications can keep + using a transaction-mode pooler; pg-sprite is one more client of the same database, not a + replacement for the application's connection path. +- **On hosted PostgreSQL** the session endpoint is usually the same host on a different port. + The refusal's remedy says so. +- **Size the pooler for pg-sprite:** in session mode each pg-sprite connection holds a server + connection for its whole life. A run needs its pool, the dedicated table lock session, the + proof's spare connection, and (for copy-and-swap on a live table) replication connections, + which never go through a transaction-mode pooler. +- **Keep idle limits longer than the table lock keepalive.** The lock session confirms it still + holds the lock every 10 seconds by default. A pooler or server idle timeout + (`idle_session_timeout`, a pooler's own idle limit) shorter than that can end the session, + and pg-sprite then aborts the run (LK-1, fail closed). + +## What the proof does not cover + +- **It is evidence, not certainty.** An idle transaction-mode pooler with spare backends can in + principle answer every reading healthily. The proof is a guard against the configuration an + operator lands on by following a hosted platform's default connection string, not a + substitute for pointing the engine at the right endpoint. +- **It runs once per pool, on one connection.** Connections the pool opens later are not + re-proven; they inherit the endpoint the first one proved. +- **Dedicated sessions from `dbconn.ConnectSession`** are not proven. They are used for work + that is not tied to one session the way a session-level lock is: a transaction-scoped + advisory lock on a slot name, and the identity check of a copy-and-swap connection. +- **Replication connections** are not proven. They are opened from the same connection + settings and must reach the same database; a transaction-mode pooler can't carry them. + +## Where it is enforced + +- [`pkg/dbconn/dbconn.go`](../pkg/dbconn/dbconn.go): `NewPool` reads the timeouts back + (`verifySessionSettings`) and runs the proof (`proveSessionAffinity`). +- [`pkg/dbconn/affinity.go`](../pkg/dbconn/affinity.go): `ProveSessionAffinity`, the proof. +- [`pkg/dbconn/sessionstate.go`](../pkg/dbconn/sessionstate.go): `ErrNoSessionAffinity`, + `ErrSessionSettingsDiscarded`, and the remedy text. +- [`pkg/dbconn/table_lock.go`](../pkg/dbconn/table_lock.go): `AcquireTableLock` proves + affinity on the lock session before attempting the lock. +- [invariants.md, LK-1](invariants.md#lk-1--at-most-one-migration-runs-per-table) and + [LK-2](invariants.md#lk-2--exactly-one-access-exclusive-window-and-every-strong-lock-is-bounded). From fcf4f9997aa8df2b1481d01101ebbf1cbdcac7d3 Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Fri, 30 Oct 2026 15:49:44 +1100 Subject: [PATCH 2/2] docs: describe the affinity proof as evidence, not certification A transaction pooler with spare backends can pass the proof, so the page no longer promises it is always refused and names both refusal errors. It also follows the statement-mode refusal: SELECT 1 before the BEGIN, then ErrNoSessionAffinity. --- docs/session-affinity.md | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/docs/session-affinity.md b/docs/session-affinity.md index 6b4bbc03..f3128fa0 100644 --- a/docs/session-affinity.md +++ b/docs/session-affinity.md @@ -2,8 +2,10 @@ **pg-sprite requires that every connection it opens stays on one PostgreSQL server session for as long as the connection is open.** That holds for a direct connection and for a pooler -in session mode. It does not hold behind a transaction-mode or statement-mode pooler, and -pg-sprite refuses to run there before it executes anything. This page explains what depends +in session mode. It does not hold behind a transaction-mode or statement-mode pooler, so pg-sprite must +be pointed at a direct or session-mode endpoint. It checks before it executes anything and +refuses an endpoint the check catches, but a check that passes is evidence, not +certification (see [How pg-sprite proves it](#how-pg-sprite-proves-it)). This page explains what depends on session affinity, how pg-sprite proves it, what it refuses, and how to connect. ## Table of contents @@ -91,13 +93,14 @@ for every chunk copy, guarded write, checkpoint save and cutover. Reading back `lock_timeout` is not enough on its own: an idle transaction-mode pooler may hand the same backend back twice and report the settings healthily. So after the read-back, -`NewPool` forces the rebind a pooler would eventually make, and checks for it: +`NewPool` tries to force the rebind a pooler would eventually make, and checks for it: ```diagram conn (one of the pool's connections) second connection (throwaway pool of 1) ──── ───────────────────────────────────── t1 pg_try_advisory_lock(random key) → true - t2 BEGIN (pins a backend for the whole tx) + t2 SELECT 1 (takes a backend first) + BEGIN (pins it for the whole tx) t3 pg_try_advisory_lock(same key) healthy: false (conn's session holds it) t4 does pg_locks show my own backend @@ -110,12 +113,17 @@ the same backend back twice and report the settings healthily. So after the read - **Direct connection or session-mode pooler:** `conn` is one session throughout, so every reading is the healthy one, and there are no false positives to trade off. -- **Transaction-mode pooler:** the open transaction at t2 takes the backend `conn` used at t1, - so `conn`'s next statement lands elsewhere. At least one reading fails: the second - connection takes a lock `conn` "holds", `conn` is no longer the holder in `pg_locks`, or - `conn` can't release what it took. `NewPool` returns `ErrNoSessionAffinity` with the remedy. -- **Statement-mode pooler:** the pooler refuses the `BEGIN` at t2, and `NewPool` fails before - any work runs. +- **Transaction-mode pooler:** the proof tries to expose the rebind. The open transaction at + t2 is meant to take the backend `conn` used at t1, so that `conn`'s next statement lands + elsewhere; when it does, at least one reading fails: the second connection takes a lock + `conn` "holds", `conn` is no longer the holder in `pg_locks`, or `conn` can't release what it + took. `NewPool` then returns `ErrNoSessionAffinity` with the remedy. A pooler with spare + backends can hand the second connection a different backend, and then every reading is + the healthy one: **a passing proof does not make a transaction-mode endpoint safe.** Point + pg-sprite at a direct or session-mode endpoint whether or not the proof passes. +- **Statement-mode pooler:** the second connection has just run `SELECT 1`, so a pooler that + refuses the `BEGIN` at t2 keeps no backend across statements. `NewPool` returns + `ErrNoSessionAffinity` before any work runs. - The key is freshly random on every call, so concurrent proofs never contend and a probe lock stranded on a pooled backend can't block anything later. - The proof needs **one spare connection** to the server, or pooler, beyond the pool's own. @@ -129,13 +137,14 @@ The same proof runs on the table lock's dedicated connection before the lock is |---|---| | Direct connection to PostgreSQL (including Aurora/RDS writer endpoints) | Passes | | PgBouncer, or another pooler, in **session** mode | Passes. The proof needs one spare server connection in the pooler's pool. | -| PgBouncer in **transaction** mode | Refused with `ErrNoSessionAffinity` | -| PgBouncer in **statement** mode | Refused before any work runs: PgBouncer rejects the proof's `BEGIN` | +| PgBouncer in **transaction** mode | Not supported. Usually refused with `ErrNoSessionAffinity`, or with `ErrSessionSettingsDiscarded` when the pooler rejects or drops the timeout startup parameters. A proof that passes does not make it supported. | +| PgBouncer in **statement** mode | Not supported. Refused with `ErrNoSessionAffinity` (PgBouncer rejects the proof's `BEGIN`), or with `ErrSessionSettingsDiscarded` as above. | | Supabase Supavisor session endpoint | Passes ([supabase.md](supabase.md)) | -| Supabase Supavisor transaction endpoint | Refused with `ErrNoSessionAffinity`, even with named prepared statements disabled ([supabase.md](supabase.md)) | +| Supabase Supavisor transaction endpoint | Not supported. Refused with `ErrNoSessionAffinity` in testing, even with named prepared statements disabled ([supabase.md](supabase.md)) | -The transaction-mode and session-mode PgBouncer results are tested against a real PgBouncer -([testing.md](testing.md)). +The session-, transaction- and statement-mode PgBouncer results are tested against a real +PgBouncer ([testing.md](testing.md)). Both refusals name the same remedy: connect through a +direct or session-mode endpoint. ## How to connect