Repository navigation
Add generate and reproduce: from a bundle to a loaded reproduction - #19
Open
steven-hubbard wants to merge 2 commits into
Open
steven-hubbard wants to merge 2 commits into
steven-hubbard wants to merge 2 commits into
Conversation
Contributor
Author
|
@lin-crl can you take a look? |
dbprofiler collected a bundle and stopped there. Turning one into a reproduction database was hand-work, redone per database. `generate` derives a dbworkload kit from profile.json alone: the generation definition, a CockroachDB schema grouped one section per table with index recommendations, an IMPORT INTO emitter, a driver script, and stubs for the two things per-column statistics cannot supply -- keys that must agree across tables, and the workload itself. What the contract cannot express (CHECK constraints, partial-index predicates, expression keys) is flagged in place rather than dropped, and schema.sql stays authoritative for the real schema. `reproduce` drives a kit into the target named by DBPROFILER_TARGET_URL, applying the schema, generating the data, loading in foreign-key order, validating every foreign key and refreshing statistics. How much is supplied decides where it starts: nothing collects first, --bundle derives a kit, --kit drives an existing one. Both go through the existing single child-process call site, so audit_subprocess_usage() is unchanged and credentials still reach psql only through the environment. Statements issued against a target are TARGET_SQL_* constants audited on their own rule -- ANALYZE belongs there, COUNT( and CREATE STATISTICS do not -- and EMIT_* templates are audited for credentials. check_safety() now reports all three categories. A target URL recorded into a kit is stripped of its userinfo first: a kit is a directory of files that gets copied and shared. Regenerating over an existing kit refreshes the derived files but keeps an edited keys.py or workload.py, which are the only parts of a kit that cannot be rebuilt from the bundle. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
main grew schema_by_table.sql, a regrouped pg_dump published as a bundle payload. This branch grew schema.crdb.sql, a CockroachDB schema synthesized from profile.json into a generated kit. They answer the same question for different readers, so both are kept. The bundle's copy stays authoritative for the source schema: it is the dump regrouped, so it carries CHECK constraints, partial-index predicates and expression keys, none of which the contract records. The kit's copy targets CockroachDB -- STRING, BYTES, INT4 rather than a widening INT -- and carries the index recommendations, which need the contract rather than the dump. The only conflict was in test_dbprofiler.py, where both sides appended test classes to the end of the module. No names collide; both sets are kept. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
steven-hubbard
force-pushed
the
kit-generate
branch
from
October 28, 2026 18:32
0b08e5e to
a572af2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dbprofiler collected a bundle and stopped there. Turning one into a
reproduction database was hand-work, redone per database.
generatederives a dbworkload kit from profile.json alone: the generationdefinition, a CockroachDB schema grouped one section per table with index
recommendations, an IMPORT INTO emitter, a driver script, and stubs for the
two things per-column statistics cannot supply -- keys that must agree across
tables, and the workload itself. What the contract cannot express (CHECK
constraints, partial-index predicates, expression keys) is flagged in place
rather than dropped, and schema.sql stays authoritative for the real schema.
reproducedrives a kit into the target named by DBPROFILER_TARGET_URL,applying the schema, generating the data, loading in foreign-key order,
validating every foreign key and refreshing statistics. How much is supplied
decides where it starts: nothing collects first, --bundle derives a kit,
--kit drives an existing one.
Both go through the existing single child-process call site, so
audit_subprocess_usage() is unchanged and credentials still reach psql only
through the environment. Statements issued against a target are TARGET_SQL_*
constants audited on their own rule -- ANALYZE belongs there, COUNT( and
CREATE STATISTICS do not -- and EMIT_* templates are audited for credentials.
check_safety() now reports all three categories.
A target URL recorded into a kit is stripped of its userinfo first: a kit is
a directory of files that gets copied and shared. Regenerating over an
existing kit refreshes the derived files but keeps an edited keys.py or
workload.py, which are the only parts of a kit that cannot be rebuilt from
the bundle.
Co-Authored-By: roachdev-claude roachdev-claude-bot@cockroachlabs.com