Repository navigation
Add a schema_by_table.sql payload to the bundle - #17
Merged
Merged
Conversation
pg_dump writes DDL in phase order, so reading one table's definition means jumping between six or more places in schema.sql. That is the wrong shape for migration review, which works table by table. Emit the same statements regrouped: a preamble of objects that are not table-scoped, one section per table in foreign-key dependency order, and a trailing section for views, triggers, and anything spanning tables. Within a section the order is CREATE TABLE, primary key and indexes, foreign keys, comments, then the replica-identity index beside the ALTER that selects it. No SQL is rewritten and nothing is dropped -- the renderer is a pure function over the dump text, so schema.sql stays the byte-for-byte authority and the regrouped file still replays. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
The regrouped payload was only ever exercised against ordinary tables. The shapes most likely to break it were untested: a statement that names two tables at once, a relation that is a table to one catalog and a view to another, and a constraint that carries its own index. Add to the integration fixture a materialized view with an index, plain and partial exclusion constraints, and all three explicit replica identities, with assertions that each lands where it has to. Exclusion constraints sort with the primary key, REPLICA IDENTITY USING INDEX stays beside the index it names, and the materialized view and its index go cross-table. Partitioning gets a recorded dump instead. require_supported_layout refuses a partitioned source, so the profiler cannot reach those shapes at all, but the renderer is a pure function over dump text and can be handed one directly -- including the ATTACH PARTITION statements that stop replaying if either table they name moves past them. Collect the fixture table list into one constant; four assertions compared against the whole set and adding a table missed three of them. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
test_no_credential_appears_in_any_bundle searched the bundle for PGUSER as well as PGPASSWORD. A role name is not a secret, and the bundle records identity deliberately: source.database and source.kind are published fields. Searching for a role name therefore reports a collision rather than a leak. The default superuser hits this every time -- `postgres` matches "kind": "postgres", which appears in every bundle the tool can produce -- and a role named after its database matches source.database. The failure printed the whole archive as raw bytes, naming neither the member nor the field, so the one test most worth trusting was the easiest to dismiss. Check the password alone, and say which bundle it reached. The user name is still covered where it is genuinely not allowed to appear: test_no_credential_appears_on_stderr is unchanged. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
I001 on the new collections import and E501 on an ATTACH PARTITION regex, both introduced by the two commits before this one. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pg_dumpwrites DDL in phase order — everyCREATE TABLE, then every default, then every constraint, then every index — so one table's definition ends up scattered over six or more places. That is the wrong shape for migration review, which works a table at a time.This adds a second schema payload,
schema_by_table.sql, carrying the same statements regrouped for reading:CREATE TABLE(with its owned sequence and defaults), primary key and other constraints, indexes, foreign keys, comments, then the replica-identity index beside theALTERthat selects itschema.sqlis untouched and stays the byte-for-byte authority —render_schema_by_table()is a pure function over the dump text it already has in hand, so there are no new catalog queries, no new SQL constants, and no secondsubprocesscall site.VERSIONgoes to 0.3.0.Commits
On the fixtures
The renderer was only exercised against ordinary tables. The second commit adds the shapes where regrouping can actually go wrong: a statement that names two tables at once, a relation that is a table to one catalog and a view to another, and a constraint that carries its own index.
The integration fixture gains a materialized view with an index, plain and partial exclusion constraints, and all three explicit replica identities, with assertions that each lands where it must.
Partitioning gets a recorded dump instead of a live fixture.
require_supported_layoutrejects a partitioned source outright, so the profiler cannot reach those shapes at all — but the renderer is a pure function over dump text and can be handed one directly. That golden covers RANGE and LIST parents, a DEFAULT partition, a partition attached after the fact, two-level sub-partitioning, and a partitioned index, including theATTACH PARTITIONstatements that stop replaying if either table they name moves past them.On the third commit
test_no_credential_appears_in_any_bundlesearched the bundle forPGUSERas well asPGPASSWORD. A role name is not a secret, and the bundle records identity deliberately —source.databaseandsource.kindare published fields — so searching for a role name reports a collision rather than a leak.The default superuser hits this every time:
postgresmatches"kind": "postgres", which appears in every bundle the tool can produce. The failure printed the whole archive as raw bytes, naming neither the member nor the field. The test now checks the password alone and says which bundle it reached.test_no_credential_appears_on_stderris unchanged and still covers the user name, where it genuinely must not appear.Test plan
python3 -m unittest— 422 tests pass (415 before)python3 dbprofiler.py --check-safety— OK, no new SQL constants outside the auditpython3 -m unittest integration_testagainst PostgreSQL 16.15 — 79 tests pass (71 before)schema.sqlandschema_by_table.sqlfrom the same run is emptyPGUSER=postgres(previously failed every time), and still fails when a password really is present in the bundle