Repository navigation
fix(core): bound what a SQLite or DuckDB catalog read holds - #4287
Merged
Merged
Conversation
…ables that name the parent
…e quadratic name search
… constraint scan, count child keys once
…ts by commas, group save rows by key once
… off duckdb_columns
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CatalogBudgetbounds what one catalog read holds at 64 MB (ByteLimit::CATALOG_DEFAULT), counted asValue::retained_bytescounts text. Past it the read fails withDriverError::CatalogTooLarge, codedriver.catalog_too_large, category Internal.pragma_foreign_key_listruns on a table only once its cost, worked out from the CREATE TABLE, fits the budget. SQLite compiled a copy of a column's name for every REFERENCES on it, 330 MB for a 120 KB table.CatalogTooLarge. The error lists the first ten groups.duckdb_tables()andduckdb_dependencies(). The keys into a table no longer join the child's CREATE TABLE onto every key, a parent's CREATE TABLE is read once, and a dump reads each child's parents as one list.core/ARCHITECTURE.md,linux/docs/error-handling.md,linux/docs/adding-drivers.md.Test plan
tablepro-driver-sqlite --test catalog_memory: a crafted file (a column carrying REFERENCES 4,000 times, a 64 KiB-named child with 4,000 keys, 4,000 keys copying a 64 KiB primary key, broken rows and broken keys of 64 KiB-named tables, 60 children of 256 KiB each, 3,000 rows each breaking 1,000 keys) fails each catalog read and the save withCatalogTooLarge; the heap stays under twice the budget and SQLite's own memory under 32 MB. Without the key list check SQLite reached 331 MB.tablepro-driver-duckdb --test catalog_memory: 1,000 keys into one parent read in under 16 MB and a CHECK over 600 long-named columns stops at the budget; in a file with a 64 KiB-named child of 600 keys every constraint read stops before DuckDB runs it, and the table list still reads.catalog::a_large_catalog_reads_whole: 5,000 tables keyed twice each to one table, and a table with 500 keys, read whole and in order. It passes on main too; the keys into the table take 0.2 s here and 1.4 s on main.catalog::keys_into_a_table_are_found_however_its_name_is_written, and the DuckDB dump with a parent name holding U+001F.cargo test -p tablepro-core -p tablepro-driver-sqlite -p tablepro-driver-duckdb -p tablepro-test-support.