Skip to content

fix(core): read each SQLite catalog in one transaction and cap a schema statement at 10 MB - #4291

Merged
datlechin merged 13 commits into
mainfrom
fix/sqlite-catalog-limits
Nov 15, 2026
Merged

datlechin merged 13 commits into
mainfrom
fix/sqlite-catalog-limits

Conversation

@datlechin

@datlechin datlechin commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Summary

  • SQLite: every catalog read runs in one read transaction (a savepoint released at its end), so a schema change by another connection between its statements can no longer make the keys into a table skip a child. A stop that cuts a catalog read's lock wait short still reads as cancelled or timed out.
  • SQLite: the workspace connections set SQLITE_LIMIT_SQL_LENGTH to 10,000,000 bytes. SQLite applies it to each CREATE statement as it reads the schema, so a file with a longer one fails to open as a malformed schema (driver.server, SQLite error 11), and SQLite ignores whatever follows a stored statement, so each catalog read that parses a stored text takes it only up to the same cap and fails with CatalogTooLarge past it.
  • SQLite: a stored text is kept only up to where its first statement's terminator starts, as SQLite reads it (ScriptPlan::first_statement_end, lexed without keeping tokens). A semicolon inside parentheses ends nothing, since SQLite reads it as a virtual table's module argument. A tail of 50,000 additions after a CREATE INDEX overflowed the stack in sqlparser and aborted the process; a table's tail reached the CREATE TABLE reader, a rebuild's source and dumps.
  • SQLite: a stored text's size comes from octet_length, which reads it from the record without the text, and the keys into a table count only the text of tables up to 64 KiB. A 25 MB tail cost 125 MB of SQLite memory on every table's definition before; now none.
  • SQLite: the floor moves from 3.37 to 3.43 for octet_length; every system with GTK 4.22 ships 3.46 or newer.
  • SQLite: the workspace connections set SQLITE_LIMIT_EXPR_DEPTH to the default 1,000, since a build can raise it and sqlparser drops and renders an expression one call per level.
  • SQLite: a sqlparser parse runs only once the text, at 512 bytes for each byte (about 420 measured), fits the budget. An index's statement is parsed only for an expression key or a WHERE, and a generated column's expression alone, not its whole CREATE TABLE. An 8 MB generated expression held 2.2 GB before.
  • SQLite: index key offsets come from one forward pass, not a count from each line's start. An 80 KB one-line partial index took 3.3 s; now 0.02 s.
  • SQLite: a table's dependent views and triggers come from one pass over the names they mention, not a rescan of every row per kept view. 10,000 views on a table beside 10,000 others took 3.7 s per definition read, and views stored in reverse order grew with the cube.
  • The key list cost check and the 64 MB catalog budget stay: no SQLite limit bounds what pragma_foreign_key_list compiles (about S²/52 bytes for a statement of S bytes, 331 MB from a 117 KB table), SQLITE_LIMIT_VDBE_OP only stops it after 153 MB at 25,000 ops, and sqlite3_hard_heap_limit64 is process-wide.
  • DuckDB is unchanged: it has no per-connection statement or schema limit.
  • Docs: core ADR 0029, linux/docs/drivers/sqlite.md.

Deliberately not fixed here

Pre-existing, in code this PR does not change, each needing its own design:

  • SQLite compiles pragma foreign_key_check with a virtual generated column's expression once per key: an empty table whose g AS (a IN (<5,000 zeros>)) carries 200 REFERENCES compiles to 217 MB (Codex, SQLite 3.54) though foreign_key_list_bytes costs it at 103 KB. Each such expansion needs its own estimate; bounding what SQLite allocates for one statement needs a counting allocator (SQLITE_CONFIG_MALLOC) armed per read, or the work in a process of its own.
  • A dump holds every stored statement at once with no total budget (ADR 0029 lists dumps as not budgeted), so hundreds of statements just under the 10 MB cap make it hold gigabytes. Needs a streaming dump read.
  • SQLite expands chained views inside pragma_table_list (18 views: 120 MB, 10 s), and the DuckDB duckdb_columns()/duckdb_constraints() scans.

Test plan

  • connection::tests::a_catalog_read_runs_in_one_transaction_and_ends_it: the read's statements run in one transaction, which ends after a failed read too.
  • catalog::a_create_statement_past_ten_million_bytes_does_not_open: a 9 MB CREATE TABLE opens and reads; one past 10 MB fails the connect with SQLite error 11.
  • catalog::a_stored_create_text_past_the_cap_is_refused_before_it_is_parsed: an 11 MB tail after a stored statement opens, and the reads that would parse it fail with CatalogTooLarge.
  • lock_wait::a_catalog_read_waiting_for_a_lock_stops_as_the_stop_it_was: cancelled and timed-out lock waits read as such, and the next read and write work.
  • catalog_memory: a 1 MB generated expression stops at the budget, a 100 KB one-line partial index reads within it, a table beside an 11 MB tail reads with under 8 MB of SQLite memory, and the keys of a table with a 4 MB tail under the cap read within 16 MB.
  • index_sql::tests::a_long_one_line_key_list_reads_in_one_pass and keys_after_wide_characters_on_their_line_keep_their_text.
  • table_definition::tests::a_wide_table_parses_only_its_generated_expressions and a_generated_expression_too_long_to_parse_stops_at_the_budget.
  • catalog::a_stored_text_is_read_to_its_first_statement: a table's and an index's tails are cut; on the previous head this test aborted with a stack overflow.
  • script_corpus::the_first_statement_ends_where_its_terminator_starts: quotes, comments, brackets and trigger bodies, and an untouched text stays whole.
  • runtime_version::tests::sqlite_runtime_version_below_3_43_is_unsupported replaces the 3.37 test, on purpose.
  • statement_limit::tests::a_stored_text_keeps_its_first_statement_and_a_module_argument_its_semicolon.
  • dependents::tests::views_stored_before_the_views_they_name_are_kept: a 2,000-view chain stored in reverse order.
  • catalog::a_large_catalog_reads_whole and the existing catalog_memory crafted-file checks still pass.
  • cargo test -p tablepro-driver-sqlite.

@datlechin
datlechin enabled auto-merge (squash) November 15, 2026 14:23
@datlechin
datlechin merged commit 304a8de into main Nov 15, 2026
34 checks passed
@datlechin
datlechin deleted the fix/sqlite-catalog-limits branch November 17, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant