Skip to content

fix(core): bound what a SQLite or DuckDB catalog read holds - #4287

Merged
datlechin merged 11 commits into
mainfrom
fix/sqlite-catalog-memory
Nov 15, 2026
Merged

datlechin merged 11 commits into
mainfrom
fix/sqlite-catalog-memory

Conversation

@datlechin

@datlechin datlechin commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Core: CatalogBudget bounds what one catalog read holds at 64 MB (ByteLimit::CATALOG_DEFAULT), counted as Value::retained_bytes counts text. Past it the read fails with DriverError::CatalogTooLarge, code driver.catalog_too_large, category Internal.
  • SQLite: catalog rows stream instead of being collected, and each kept text and each per-key copy is charged. Key lists are no longer sorted by SQLite, which held every row first.
  • SQLite: pragma_foreign_key_list runs on a table only once its cost, worked out from the CREATE TABLE, fits the budget. SQLite compiled a copy of a column's name for every REFERENCES on it, 330 MB for a 120 KB table.
  • SQLite: the keys into a table read every table up to 64 KiB whose key list is cheap to compile in one query, counting each child's keys as they pass and keeping its CREATE TABLE once; other tables are read one at a time, searched in linear time for the parent's name and costed first.
  • SQLite: each CREATE TABLE is parsed once per read for key comparisons, and parents are grouped by name instead of searched per key.
  • SQLite: a save's foreign key check holds each table's name and each broken key once, and each broken row as a key index and a rowid, all charged to one budget with the groups its error names; past it the save rolls back with CatalogTooLarge. The error lists the first ten groups.
  • DuckDB: catalog reads are charged the same way. A constraint read runs only once its result and DuckDB's largest chunk of constraint rows, which it builds for every table before filtering, fit; both are bounded from duckdb_tables() and duckdb_dependencies(). The keys into a table no longer join the child's CREATE TABLE onto every key, a parent's CREATE TABLE is read once, and a dump reads each child's parents as one list.
  • Linux and Windows word the new code.
  • Docs: core ADR 0029, core/ARCHITECTURE.md, linux/docs/error-handling.md, linux/docs/adding-drivers.md.

Test plan

  • tablepro-driver-sqlite --test catalog_memory: a crafted file (a column carrying REFERENCES 4,000 times, a 64 KiB-named child with 4,000 keys, 4,000 keys copying a 64 KiB primary key, broken rows and broken keys of 64 KiB-named tables, 60 children of 256 KiB each, 3,000 rows each breaking 1,000 keys) fails each catalog read and the save with CatalogTooLarge; the heap stays under twice the budget and SQLite's own memory under 32 MB. Without the key list check SQLite reached 331 MB.
  • tablepro-driver-duckdb --test catalog_memory: 1,000 keys into one parent read in under 16 MB and a CHECK over 600 long-named columns stops at the budget; in a file with a 64 KiB-named child of 600 keys every constraint read stops before DuckDB runs it, and the table list still reads.
  • catalog::a_large_catalog_reads_whole: 5,000 tables keyed twice each to one table, and a table with 500 keys, read whole and in order. It passes on main too; the keys into the table take 0.2 s here and 1.4 s on main.
  • catalog::keys_into_a_table_are_found_however_its_name_is_written, and the DuckDB dump with a parent name holding U+001F.
  • cargo test -p tablepro-core -p tablepro-driver-sqlite -p tablepro-driver-duckdb -p tablepro-test-support.

@datlechin
datlechin merged commit 1c03bcd into main Nov 15, 2026
31 of 34 checks passed
@datlechin
datlechin deleted the fix/sqlite-catalog-memory branch November 15, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant