Repository navigation
fix(core): read each SQLite catalog in one transaction and cap a schema statement at 10 MB - #4291
Merged
Merged
Conversation
…ma statement at 10 MB
…topped lock wait reads as cancelled or timed out
…cap before parsing it
…r and require SQLite 3.43
… when cutting a stored statement
…ust past the statement cap
datlechin
enabled auto-merge (squash)
November 15, 2026 14:23
…nown limit in ADR 0029
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SQLITE_LIMIT_SQL_LENGTHto 10,000,000 bytes. SQLite applies it to each CREATE statement as it reads the schema, so a file with a longer one fails to open as a malformed schema (driver.server, SQLite error 11), and SQLite ignores whatever follows a stored statement, so each catalog read that parses a stored text takes it only up to the same cap and fails withCatalogTooLargepast it.ScriptPlan::first_statement_end, lexed without keeping tokens). A semicolon inside parentheses ends nothing, since SQLite reads it as a virtual table's module argument. A tail of 50,000 additions after a CREATE INDEX overflowed the stack in sqlparser and aborted the process; a table's tail reached the CREATE TABLE reader, a rebuild's source and dumps.octet_length, which reads it from the record without the text, and the keys into a table count only the text of tables up to 64 KiB. A 25 MB tail cost 125 MB of SQLite memory on every table's definition before; now none.octet_length; every system with GTK 4.22 ships 3.46 or newer.SQLITE_LIMIT_EXPR_DEPTHto the default 1,000, since a build can raise it and sqlparser drops and renders an expression one call per level.pragma_foreign_key_listcompiles (about S²/52 bytes for a statement of S bytes, 331 MB from a 117 KB table),SQLITE_LIMIT_VDBE_OPonly stops it after 153 MB at 25,000 ops, andsqlite3_hard_heap_limit64is process-wide.linux/docs/drivers/sqlite.md.Deliberately not fixed here
Pre-existing, in code this PR does not change, each needing its own design:
pragma foreign_key_checkwith a virtual generated column's expression once per key: an empty table whoseg AS (a IN (<5,000 zeros>))carries 200 REFERENCES compiles to 217 MB (Codex, SQLite 3.54) thoughforeign_key_list_bytescosts it at 103 KB. Each such expansion needs its own estimate; bounding what SQLite allocates for one statement needs a counting allocator (SQLITE_CONFIG_MALLOC) armed per read, or the work in a process of its own.pragma_table_list(18 views: 120 MB, 10 s), and the DuckDBduckdb_columns()/duckdb_constraints()scans.Test plan
connection::tests::a_catalog_read_runs_in_one_transaction_and_ends_it: the read's statements run in one transaction, which ends after a failed read too.catalog::a_create_statement_past_ten_million_bytes_does_not_open: a 9 MB CREATE TABLE opens and reads; one past 10 MB fails the connect with SQLite error 11.catalog::a_stored_create_text_past_the_cap_is_refused_before_it_is_parsed: an 11 MB tail after a stored statement opens, and the reads that would parse it fail withCatalogTooLarge.lock_wait::a_catalog_read_waiting_for_a_lock_stops_as_the_stop_it_was: cancelled and timed-out lock waits read as such, and the next read and write work.catalog_memory: a 1 MB generated expression stops at the budget, a 100 KB one-line partial index reads within it, a table beside an 11 MB tail reads with under 8 MB of SQLite memory, and the keys of a table with a 4 MB tail under the cap read within 16 MB.index_sql::tests::a_long_one_line_key_list_reads_in_one_passandkeys_after_wide_characters_on_their_line_keep_their_text.table_definition::tests::a_wide_table_parses_only_its_generated_expressionsanda_generated_expression_too_long_to_parse_stops_at_the_budget.catalog::a_stored_text_is_read_to_its_first_statement: a table's and an index's tails are cut; on the previous head this test aborted with a stack overflow.script_corpus::the_first_statement_ends_where_its_terminator_starts: quotes, comments, brackets and trigger bodies, and an untouched text stays whole.runtime_version::tests::sqlite_runtime_version_below_3_43_is_unsupportedreplaces the 3.37 test, on purpose.statement_limit::tests::a_stored_text_keeps_its_first_statement_and_a_module_argument_its_semicolon.dependents::tests::views_stored_before_the_views_they_name_are_kept: a 2,000-view chain stored in reverse order.catalog::a_large_catalog_reads_wholeand the existingcatalog_memorycrafted-file checks still pass.cargo test -p tablepro-driver-sqlite.