Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,8 @@ refusal — never a silently wrong or incomplete result:
which keeps its values. A `DROP INDEX` run under `--accept-blocking` needs
`--allow-destructive` too: accepting the lock is not accepting the loss,
and the refusal lists the checks (`idx_scan` on every replica, foreign
keys, uniqueness) that catch an index the application still uses.
keys, uniqueness) that catch an index the application still uses, with
the connected server's own scan count for the index in `index_usage`.
- **Destructive desired-state plans are refused as a whole** — `migrate --desired`
creates missing tables and converges supported changes, but never infers permission
to discard live structure. A new primary key or unique constraint over a live
Expand Down
2 changes: 1 addition & 1 deletion SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The invariant registry (invariant IDs referenced below) lives in
| Package | Core? | Status | Invariants enforced |
| --- | --- | --- | --- |
| `pkg/dbconn` — pool defaults, terminate-blockers, retries, RDS TLS, advisory table lock, replication connection | ✅ core | table-lock primitive exists; `ConnectReplication` dials one replication-mode connection with the pool's TLS, timeouts, and `BeforeConnect` hook; wiring into executing modes planned | LK-1 primitive; LK-2 primitives; CO-9 (session hook and `LocalSearchPath`) |
| `pkg/preflight` — precondition verifier, refusals | ✅ core | exists; copy-and-swap target proof (shape and dependents) and cluster/volume environment check gate the copy-and-swap route before any shadow object is created; table usage evidence (`ReadTableUsage`, `TableUsage.Judge`) exists with no caller yet | ST-6, RF-1..RF-5, RF-11, CO-9 (every proof read is `pg_catalog`-qualified) |
| `pkg/preflight` — precondition verifier, refusals | ✅ core | exists; copy-and-swap target proof (shape and dependents) and cluster/volume environment check gate the copy-and-swap route before any shadow object is created; table usage evidence (`ReadTableUsage`, `TableUsage.Judge`) exists with no caller yet; index usage evidence (`ReadIndexUsage`) is reported on a refused `DROP INDEX` | ST-6, RF-1..RF-5, RF-11, CO-9 (every proof read is `pg_catalog`-qualified) |
| `pkg/executor` — bounded optimistic attempt; native concurrent index build with invalid-index recovery; native sequence executor for the safer idioms | ✅ core | exists (Phase 1: attempt-under-budget; Phase 3.1: concurrent index build; Phase 3.2: sequence executor) | LK-2 (attempt bound + the CONCURRENTLY wait-policy exception), CO-9 (qualified proof reads), ST-9 (create owner verified, never repaired) |
| `pkg/checksum` — chunk verifier, divergence policy, repair, continuous checker | ✅ core | the chunk `Verifier` exists (one read-only `REPEATABLE READ` transaction per chunk in which the chunk is cut and both digests read, so all three share a snapshot and the cut is bounded, every column cast to the shadow's type on both sides, the same guard as the copier — owner role, the copy statement's `search_path` (the catalog alone, or the shadow build's path when the proof carries conversions), `ACCESS SHARE` on both relations, lock confirmation, relation-OID check — and a `Report` of mismatched chunks up to the landed watermark); `Check` runs the pass under an explicit `DivergencePolicy` (zero value refused; `ParseDivergencePolicy` for configuration, which names `abort` or `repair` and refuses `recopy`), aborts with the shadow untouched or recopies every differing chunk with the copier's chunk statement in one guarded transaction — every chunk's rows deleted before any are put back, so a unique value the source moved between two chunks lands — then rereads each chunk under `repair`, or commits without the reread under `recopy` (the reconcile mode's policy, for a source written while the pass runs; a recopy that collides with a shadow row because the source moved a unique value from a clean chunk into a differing one is rolled back and the pass compares and recopies again, a bounded number of times, counting each collision), returning the committed repairs with any refusal, and mints `CleanWatermark` and `VerifiedShadow` (private constructors) only from a pass that found nothing and repaired nothing; `CheckLive` is the pass for a source written while the catch-up applies the stream — a differing chunk is reread once the catch-up has confirmed past the insert position it was read at and judged against the earlier read (settled, hot and reread again up to a bound, or a confirmed divergence), an unsettled chunk is an `UnresolvedError` that mints nothing, a live repair holds the catch-up's flushes off the shadow in Go for as long as it recopies (`LiveOptions.Flushes`, a `FlushGate` the applier's `Catchup` satisfies, required under `repair`) with `SHARE MODE` on the shadow against every other writer, resumes them once the recopy has committed, and settles each repaired chunk the same way, and `recopy` is refused; the pass is the progress tracker's `WorkSource` while it runs (chunks compared, rows hashed, chunks mismatched, chunks repaired, chunks reread, all from memory); the continuous checker that runs `CheckLive` on a loop is planned | CO-1, CO-2, CO-3, CO-9, LK-1, LK-2 |
| `pkg/copier` — shadow-table chunked copy | ✅ core | contract types and the keyset `Chunker` exist (row-count chunks over the proven key, first chunk open below and last open above, a cut frontier for the applier's discard rule, time-targeted sizing) and the parallel `Copier` (one bounded never-overwriting insert per chunk under the table lock session — each column the proof's conversions name selected as its `USING` expression, every other bare — each in a guarded transaction — owner role, the copy statement's `search_path` (the catalog alone, or the shadow build's path when the copy carries conversions), `ACCESS SHARE` on both relations, lock confirmation, relation-OID check — frontier-ordered in-flight registry, a resume that first clears the shadow above the watermark, `Position.Classify` for the applier, and the copy step's `progress.WorkSource` — rows from committed chunks, the source's catalog row count, both tables' measured sizes; every cut and measurement in a bounded read-only transaction with the catalog alone on its `search_path`) exist; every refusal is a `*RefusalError` carrying a typed `RefusalCause` from the closed set in [docs/refusal-classes.md](docs/refusal-classes.md#copier-refusals-keyed-on-copierrefusalcause) | CO-4 (chunk coverage, copy SQL shape, in-flight registry), CO-9, LK-1, LK-2 (every statement on the caller's pool bounds itself), LK-3 |
Expand Down
1 change: 1 addition & 0 deletions demo/tour.sh
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,7 @@ execute_accepted_needs_loss() {
assert_eq "refusal exit of [$sql]" 2 "$status"
assert_eq "reason of [$sql]" "destructive-change" "$(jq -r '.reason' <<<"$out")"
assert_eq "table of [$sql]" "$table" "$(jq -r '.table' <<<"$out")"
assert_eq "index_usage.idx_scan is a number in [$sql]" number "$(jq -r '.index_usage.idx_scan | type' <<<"$out")"
else
"$PGS" migrate --url "$PG_DSN" --alter "$sql" --accept-blocking "$table" --statement-timeout 30s \
|| echo "(refused, exit $? — expected 2: a dropped index also needs --allow-destructive)"
Expand Down
8 changes: 6 additions & 2 deletions docs/invariants.md
Original file line number Diff line number Diff line change
Expand Up @@ -1413,7 +1413,8 @@ Each refusal is a preflight **error with a stated reason** — never a warning,
reason. A `DROP INDEX` — unique or not — is gate-refused first; when `--accept-blocking`
accepts that refusal, the drop still runs only if `--allow-destructive` names the same
owning table, and otherwise it is refused with `destructive-change` before anything
executes, the detail listing the checks that catch an index the application still uses.
executes, the detail listing the checks that catch an index the application still uses and
the verdict's `index_usage` carrying the connected server's live evidence for the index.
Accepting the lock is not accepting the loss. Outage-risk statements are covered by
RF-5 and AB-1..AB-5 (`--force`, `--accept-blocking`) and by the LK-2 budgets.
*Enforced:* `pkg/migrate` (`run`'s destructive gate through `destructiveOperations` and
Expand All @@ -1429,7 +1430,10 @@ Each refusal is a preflight **error with a stated reason** — never a warning,
`TestRunRefusesAnUnacknowledgedDropColumnRoutedToCopyAndSwap`,
`TestRunNeedsBothForceAndTheDestructiveAcknowledgement`,
`TestRunAllowDestructiveDoesNotLiftADropIndexRefusal`,
`TestRunAcceptBlockingNeedsTheDestructiveAcknowledgementToDropAnIndex`, `TestAdmitDestructive`,
`TestRunAcceptBlockingNeedsTheDestructiveAcknowledgementToDropAnIndex`,
`TestReadIndexUsageCountsEachIndexsOwnScans`,
`TestReadIndexUsageSumsAPartitionedIndexsLeafIndexes`,
`TestCheckLeafStatisticsListedRefusesAnUnlistedLeaf`, `TestAdmitDestructive`,
`TestMigrateDryRunReportsTheDestructiveRefusal`, `TestAllowDestructiveFlagGrammar`,
`demo/tour.sh` (`refused_destructive`). *Source:*
[design principles](design-principles.md#declarative-review-first-workflow)
Expand Down
2 changes: 1 addition & 1 deletion docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ not escape hatches:
| `ADD CONSTRAINT ... USING INDEX` on a partitioned parent | PostgreSQL does not support adopting an existing index on a partitioned parent in any supported version. pg-sprite refuses before execution. |
| `ADD PRIMARY KEY` / `ADD UNIQUE` with a storage parameter the online sequence cannot render | The online sequence builds the key's index with `CREATE UNIQUE INDEX CONCURRENTLY` and carries the key's `WITH (...)` parameters onto it, each name quoted and each value a quoted literal. A parameter it cannot render faithfully, one whose value is not a plain word, number, or string, contains a backslash, or whose name is namespaced, would be dropped or changed, so no sequence is built. Running the statement as written builds the index under `ACCESS EXCLUSIVE`, so it is refused as `rewrite-required` with the `unique-index-then-constraint` guidance. Build the index concurrently with the parameter, then attach it with `ADD CONSTRAINT ... USING INDEX`. |
| `ADD FOREIGN KEY ... NOT VALID` on a partitioned parent | PostgreSQL does not support this before version 18, so pg-sprite refuses it on versions 14–17. It is supported on version 18 and later. |
| `DROP INDEX` and `REINDEX`, plain or `CONCURRENTLY` | `migrate` never runs index maintenance online-safely. The plain form is refused with `index-statement` and names its `CONCURRENTLY` idiom; the concurrent form is refused as already safe, so run it directly against the database. `migrate --accept-blocking SCHEMA.TABLE` runs a single-relation plain `DROP INDEX`, `REINDEX INDEX`, or `REINDEX TABLE` as-is under bounded budgets and exits 3 ([lock-budgeted-passthrough.md](lock-budgeted-passthrough.md)). A `DROP INDEX`, unique or not, also needs `--allow-destructive` naming the same table: without it the drop is refused with `destructive-change` (exit 2) before anything runs, and the detail lists the checks to make first — `idx_scan` is 0 on the primary and every replica over a full application cycle, no foreign key relies on the index, and nothing needs a unique index's guarantee. `diff`'s hint to run a removed index through `migrate --alter` ends in this refusal; run `DROP INDEX CONCURRENTLY` directly. |
| `DROP INDEX` and `REINDEX`, plain or `CONCURRENTLY` | `migrate` never runs index maintenance online-safely. The plain form is refused with `index-statement` and names its `CONCURRENTLY` idiom; the concurrent form is refused as already safe, so run it directly against the database. `migrate --accept-blocking SCHEMA.TABLE` runs a single-relation plain `DROP INDEX`, `REINDEX INDEX`, or `REINDEX TABLE` as-is under bounded budgets and exits 3 ([lock-budgeted-passthrough.md](lock-budgeted-passthrough.md)). A `DROP INDEX`, unique or not, also needs `--allow-destructive` naming the same table: without it the drop is refused with `destructive-change` (exit 2) before anything runs, and the detail lists the checks to make first — `idx_scan` is 0 on the primary and every replica over a full application cycle, no foreign key relies on the index, and nothing needs a unique index's guarantee. The refusal's `index_usage` carries the connected server's own evidence — `idx_scan`, `stats_reset`, `last_idx_scan` (PostgreSQL 16+), uniqueness, and whether `track_counts` was on — which covers that one server only, never its replicas. `diff`'s hint to run a removed index through `migrate --alter` ends in this refusal; run `DROP INDEX CONCURRENTLY` directly. |
| `RENAME COLUMN` and `RENAME TO` | Without `--force`, refused with `not-native-safe-rewrite-required`, nothing executed: the catalog flip is instant, but every running application instance that still queries the old name breaks the moment it commits, and no safer sequence can be built from the statement alone because the dual-write and the read switch live in the application. The plan report, `suggest`, and the verdict carry the manual path as typed guidance — `expand-contract-column` for a column (add the new column, dual-write and backfill, switch reads, drop the old column as its own reviewed change) and `coordinated-table-rename` for a table (coordinate the rename with the deploy that adopts the new name, or keep the old name readable through a view until every instance has moved) — and `lint` reports the rename as an error. `--force SCHEMA.TABLE` runs the submitted rename as a blind bounded attempt under the usual budgets and size guard, for the operator who has coordinated the deploy; on a table above `--max-table-size` (default 1 GiB) the forced rename is refused as `not-native-safe-table-too-large` until that threshold is raised. |
| `ALTER INDEX ... RENAME TO` on an index that backs a primary-key, unique, or exclusion constraint | Refused with `not-native-safe-rewrite-required`, class `by-design`, guidance `coordinated-constraint-rename`, nothing executed: PostgreSQL renames the constraint with its index, and `ON CONFLICT ON CONSTRAINT` and error handling that name the old constraint break the moment it commits. `--force` does not apply to an index rename. A plain index rename runs. The refusal needs the catalog, so offline `lint` and `suggest` do not report it. |
| `ATTACH PARTITION` | Without `--force`, refused with `not-native-safe-rewrite-required`, nothing executed: the engine does not yet construct the validated `CHECK` that lets the attach skip its scan. Pre-prove the bound yourself, then run the attach directly. `--force SCHEMA.TABLE` runs the blocking `ATTACH PARTITION` as submitted under bounded budgets, taking the blocking lock. |
Expand Down
11 changes: 10 additions & 1 deletion docs/lock-budgeted-passthrough.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,16 @@ A `DROP INDEX` needs both acknowledgements, each naming the owning table: `--acc
accepts the lock, and `--allow-destructive` accepts losing the index (RF-10). The second is
checked after the owning table resolves and before anything executes; without it the drop is
refused with `destructive-change`, and the detail lists the checks that catch an index the
application still uses. `REINDEX INDEX` and `REINDEX TABLE` discard nothing and need only
application still uses. The refusal also carries `index_usage`, the connected server's live
evidence for the index (`preflight.ReadIndexUsage`, keyed on the index OID the owning-table
lookup resolved): its `idx_scan` since `stats_reset` (or since the index was created, if
later, which the server does not record), `last_idx_scan` on PostgreSQL 16 and later, whether
it is unique, whether `track_counts` was on, and whether a saved role or database setting
turns it off. A partitioned index has no statistics of its own, so its `idx_scan` is the sum
over its leaf indexes and `partitions` counts them. It is one server's view —
replicas keep their own counters — so it is evidence for the operator, not a gate. Its times
are UTC. The read takes no lock; if it fails, including when the server lists no statistics
for a leaf index, nothing executes and the error is returned instead of the refusal. `REINDEX INDEX` and `REINDEX TABLE` discard nothing and need only
`--accept-blocking`.

The value names the table whose lock the operator accepts, because that is the relation the
Expand Down
8 changes: 7 additions & 1 deletion docs/schemabot-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -634,7 +634,13 @@ and PostgreSQL `lock_timeout`). Contract points for the PostgreSQL adapter:
executed. An acknowledged run's verdict carries `allowed_destructive: true`, the
machine-readable audit record to store beside the operator's consent. The acknowledgement
covers only the loss: a statement that also needs `Force` still needs it, and a `DROP INDEX`
run with `Options.AcceptBlocking` needs both, each naming the index's owning table.
run with `Options.AcceptBlocking` needs both, each naming the index's owning table; its
refusal carries `index_usage` (the writer's `idx_scan`, summed over the leaf indexes of a
partitioned index with `partitions` counting them, `stats_reset`, `last_idx_scan`,
uniqueness, `track_counts`, `track_counts_off_by_setting`; times in UTC) to show the
operator before asking for that consent. If that read fails, `Run` returns the error with a
zero verdict and nothing executed rather than the refusal: route it as a retryable
failure, like a connection fault, not as a refusal.
`RunDesired`
rejects `AllowDestructive` (`ErrAllowDestructiveNotSupported`) and keeps refusing a
destructive plan whole, so a declarative drop the operator approves runs as its own
Expand Down
6 changes: 5 additions & 1 deletion docs/table-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,11 @@ those statistics can prove. This page explains what the check reads, how it deci
`DROP TABLE` is still refused by both front doors today
([capabilities](capabilities.md#deliberately-operator-owned)). The check described here is
a read-only primitive, `preflight.ReadTableUsage` and `TableUsage.Judge`, that a later
drop path will consult. Nothing calls it yet.
drop path will consult. Nothing calls it yet. Its index-level sibling,
`preflight.ReadIndexUsage`, reads one index's `idx_scan`, `last_idx_scan`, and uniqueness
(summed over the leaf indexes of a partitioned index) under the same limits, including the
saved `track_counts` settings check, and is reported, never judged, on a refused `DROP INDEX`
([lock-budgeted passthrough](lock-budgeted-passthrough.md#flag-and-front-door)).

## Read this first: what the check can see

Expand Down
1 change: 1 addition & 0 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,7 @@ same package cover the rest of it.
| Library front door (`diffplan.Plan`): ordered routed plan, missing-table, no-op, copy-and-swap refusal, never-writes, deterministic fingerprint | [diffplan unit](../pkg/diffplan/diffplan_test.go), [diffplan integration](../pkg/diffplan/diffplan_integration_test.go) |
| Bounded optimistic native attempt and table preflight | [pkg/executor](../pkg/executor/optimistic_integration_test.go), [pkg/preflight](../pkg/preflight/preflight_integration_test.go) |
| Table usage evidence (`ReadTableUsage`, `Judge`): window edges, recent scans and lock holders, PostgreSQL 14/15 unknown, resets never idle, one statistics snapshot per read, `track_counts` turned off by a saved setting, partitioned tables judged across every leaf partition | [pkg/preflight](../pkg/preflight/table_usage_integration_test.go), [coverage](../pkg/preflight/table_usage_coverage_integration_test.go), [partitions](../pkg/preflight/table_usage_partitions_integration_test.go), [verdict (unit)](../pkg/preflight/table_usage_verdict_test.go), [partition aggregation (unit)](../pkg/preflight/table_usage_partitions_test.go) |
| Index usage evidence (`ReadIndexUsage`): each index's own scan count, uniqueness, last scan time on PostgreSQL 16+, a partitioned index summed over its leaf indexes at any depth, `track_counts` turned off by a saved setting, a non-index OID refused | [pkg/preflight](../pkg/preflight/index_usage_integration_test.go), [partitions](../pkg/preflight/index_usage_partitions_integration_test.go), [evidence and detail (unit)](../pkg/migrate/index_usage_evidence_test.go) |
| Safer-sequence runner: autocommit steps, committed-prefix semantics on a mid-sequence failure | [pkg/executor](../pkg/executor/sequence_integration_test.go) |
| Concurrent index build: caller-owned session, invalid-index detection into a typed state, and the proven recovery (`RebuildAbandonedIndex`, `DropAbandonedIndex`) | [pkg/executor](../pkg/executor/recover_integration_test.go) |
| Greenfield `CREATE TABLE` path: shape refusals, claimed-name collisions, post-commit name verification | [pkg/executor](../pkg/executor/create_integration_test.go) |
Expand Down
Loading
Loading