Skip to content

Add a schema_by_table.sql payload to the bundle - #17

Merged
steven-hubbard merged 5 commits into
mainfrom
schema-by-table-payload
Sep 29, 2026
Merged

steven-hubbard merged 5 commits into
mainfrom
schema-by-table-payload

Conversation

@lin-crl

@lin-crl lin-crl commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

pg_dump writes DDL in phase order — every CREATE TABLE, then every default, then every constraint, then every index — so one table's definition ends up scattered over six or more places. That is the wrong shape for migration review, which works a table at a time.

This adds a second schema payload, schema_by_table.sql, carrying the same statements regrouped for reading:

  • a preamble of the objects that are not table-scoped
  • one section per table, ordered by foreign-key dependency so a referenced table is always defined first
  • within a section: CREATE TABLE (with its owned sequence and defaults), primary key and other constraints, indexes, foreign keys, comments, then the replica-identity index beside the ALTER that selects it
  • a trailing section for views, triggers, and anything spanning tables

schema.sql is untouched and stays the byte-for-byte authority — render_schema_by_table() is a pure function over the dump text it already has in hand, so there are no new catalog queries, no new SQL constants, and no second subprocess call site. VERSION goes to 0.3.0.

Commits

  1. Add a schema_by_table.sql payload to the bundle — the renderer, the payload wiring, and goldens.
  2. Cover the schema shapes the regrouping can get wrong — fixtures for the shapes most likely to break it.
  3. Stop treating the Postgres role name as a bundle secret — fixes a false positive in an existing test (see below).

On the fixtures

The renderer was only exercised against ordinary tables. The second commit adds the shapes where regrouping can actually go wrong: a statement that names two tables at once, a relation that is a table to one catalog and a view to another, and a constraint that carries its own index.

The integration fixture gains a materialized view with an index, plain and partial exclusion constraints, and all three explicit replica identities, with assertions that each lands where it must.

Partitioning gets a recorded dump instead of a live fixture. require_supported_layout rejects a partitioned source outright, so the profiler cannot reach those shapes at all — but the renderer is a pure function over dump text and can be handed one directly. That golden covers RANGE and LIST parents, a DEFAULT partition, a partition attached after the fact, two-level sub-partitioning, and a partitioned index, including the ATTACH PARTITION statements that stop replaying if either table they name moves past them.

On the third commit

test_no_credential_appears_in_any_bundle searched the bundle for PGUSER as well as PGPASSWORD. A role name is not a secret, and the bundle records identity deliberately — source.database and source.kind are published fields — so searching for a role name reports a collision rather than a leak.

The default superuser hits this every time: postgres matches "kind": "postgres", which appears in every bundle the tool can produce. The failure printed the whole archive as raw bytes, naming neither the member nor the field. The test now checks the password alone and says which bundle it reached. test_no_credential_appears_on_stderr is unchanged and still covers the user name, where it genuinely must not appear.

Test plan

  • python3 -m unittest — 422 tests pass (415 before)
  • python3 dbprofiler.py --check-safety — OK, no new SQL constants outside the audit
  • python3 -m unittest integration_test against PostgreSQL 16.15 — 79 tests pass (71 before)
  • No SQL lost: multiset diff of non-comment, whitespace-normalized lines between schema.sql and schema_by_table.sql from the same run is empty
  • The regrouped payload replays into a fresh database with 0 errors, producing a catalog identical to the raw dump's — same objects, constraints, indexes, and replica identities
  • The partitioned golden replays too, reproducing all 28 catalog objects including partition bounds and index attachments
  • Credential check verified both ways: it now passes with PGUSER=postgres (previously failed every time), and still fails when a password really is present in the bundle

lin-crl and others added 4 commits September 28, 2026 16:59
pg_dump writes DDL in phase order, so reading one table's definition means
jumping between six or more places in schema.sql. That is the wrong shape for
migration review, which works table by table.

Emit the same statements regrouped: a preamble of objects that are not
table-scoped, one section per table in foreign-key dependency order, and a
trailing section for views, triggers, and anything spanning tables. Within a
section the order is CREATE TABLE, primary key and indexes, foreign keys,
comments, then the replica-identity index beside the ALTER that selects it.

No SQL is rewritten and nothing is dropped -- the renderer is a pure function
over the dump text, so schema.sql stays the byte-for-byte authority and the
regrouped file still replays.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
The regrouped payload was only ever exercised against ordinary tables. The
shapes most likely to break it were untested: a statement that names two
tables at once, a relation that is a table to one catalog and a view to
another, and a constraint that carries its own index.

Add to the integration fixture a materialized view with an index, plain and
partial exclusion constraints, and all three explicit replica identities,
with assertions that each lands where it has to. Exclusion constraints sort
with the primary key, REPLICA IDENTITY USING INDEX stays beside the index it
names, and the materialized view and its index go cross-table.

Partitioning gets a recorded dump instead. require_supported_layout refuses
a partitioned source, so the profiler cannot reach those shapes at all, but
the renderer is a pure function over dump text and can be handed one
directly -- including the ATTACH PARTITION statements that stop replaying if
either table they name moves past them.

Collect the fixture table list into one constant; four assertions compared
against the whole set and adding a table missed three of them.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
test_no_credential_appears_in_any_bundle searched the bundle for PGUSER as
well as PGPASSWORD. A role name is not a secret, and the bundle records
identity deliberately: source.database and source.kind are published fields.
Searching for a role name therefore reports a collision rather than a leak.

The default superuser hits this every time -- `postgres` matches
"kind": "postgres", which appears in every bundle the tool can produce -- and
a role named after its database matches source.database. The failure printed
the whole archive as raw bytes, naming neither the member nor the field, so
the one test most worth trusting was the easiest to dismiss.

Check the password alone, and say which bundle it reached. The user name is
still covered where it is genuinely not allowed to appear:
test_no_credential_appears_on_stderr is unchanged.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
I001 on the new collections import and E501 on an ATTACH PARTITION regex,
both introduced by the two commits before this one.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@steven-hubbard
steven-hubbard merged commit 6ea24b8 into main Sep 29, 2026
8 checks passed
@lin-crl
lin-crl deleted the schema-by-table-payload branch September 29, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants