From c54252aac34d1e7683cf50cba8b080615643798f Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 01:25:28 +0700 Subject: [PATCH 1/9] fix(core): bound what a SQLite or DuckDB catalog read holds --- core/ARCHITECTURE.md | 2 +- core/crates/core/src/byte_limit.rs | 2 + core/crates/core/src/catalog_budget.rs | 120 ++++ core/crates/core/src/error.rs | 13 +- core/crates/core/src/lib.rs | 2 + core/crates/drivers/duckdb/src/catalog/mod.rs | 302 +++++--- core/crates/drivers/duckdb/src/connection.rs | 18 +- core/crates/drivers/duckdb/src/dump.rs | 60 +- .../drivers/duckdb/tests/catalog_memory.rs | 110 +++ core/crates/drivers/sqlite/src/catalog/mod.rs | 40 +- .../sqlite/src/catalog/table_definition.rs | 212 +++--- .../sqlite/src/ddl/create_table_source.rs | 50 +- core/crates/drivers/sqlite/src/ddl/mod.rs | 2 +- .../sqlite/src/foreign_key_verification.rs | 213 ++++-- core/crates/drivers/sqlite/src/metadata.rs | 678 ++++++++++++------ core/crates/drivers/sqlite/src/write_batch.rs | 115 +-- .../drivers/sqlite/tests/catalog_memory.rs | 174 +++++ .../sqlite/tests/integration/catalog.rs | 97 ++- core/crates/test-support/src/heap_peak.rs | 55 ++ core/crates/test-support/src/lib.rs | 2 + ...0029-a-catalog-read-keeps-a-byte-budget.md | 30 + core/docs/decisions/README.md | 1 + linux/crates/app/src/ui/error_text/driver.rs | 4 + linux/crates/app/src/ui/error_text/tests.rs | 9 + linux/docs/adding-drivers.md | 2 +- linux/docs/error-handling.md | 3 +- .../TablePro.App/Strings/en-US/Resources.resw | 3 + 27 files changed, 1740 insertions(+), 579 deletions(-) create mode 100644 core/crates/core/src/catalog_budget.rs create mode 100644 core/crates/drivers/duckdb/tests/catalog_memory.rs create mode 100644 core/crates/drivers/sqlite/tests/catalog_memory.rs create mode 100644 core/crates/test-support/src/heap_peak.rs create mode 100644 core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md diff --git a/core/ARCHITECTURE.md b/core/ARCHITECTURE.md index bb2889d361..fd8f9302aa 100644 --- a/core/ARCHITECTURE.md +++ b/core/ARCHITECTURE.md @@ -128,7 +128,7 @@ How an app reaches a database: the contract in `tablepro-core`, a transport that ### The contract -`DatabaseDriver` describes an engine before any connection exists, through `DriverCapabilities`, and opens a `Connection`. The `Connection` serves the workspace from a pool; every call takes `CallOptions` (a deadline and a cancel token) and runs on a task the connection owns (`OwnedCalls`), so a stop reaches the server instead of leaving the statement running. Saves report what they left in the database through a `WriteLedger`. Each editor tab gets an `EditorSession` of its own that streams results into a `ResultSink`, and the `Session` that opens it puts a `ByteLimitSink` in front of that sink, so a run stops keeping rows once they pass `RunOptions::byte_limit` bytes across its results, the row that crosses it kept whole ([ADR 0020](docs/decisions/0020-a-run-keeps-a-byte-limit-of-rows.md)). [docs/adding-drivers.md](../linux/docs/adding-drivers.md) is the full contract, [ADR 0016](../linux/docs/decisions/0016-driver-contract-v2.md) the reasons for it, and the engine pages say how each driver meets it: [PostgreSQL](../linux/docs/drivers/postgresql.md), [MySQL and MariaDB](../linux/docs/drivers/mysql.md), [SQLite](../linux/docs/drivers/sqlite.md), [DuckDB](../linux/docs/drivers/duckdb.md), [SQL Server](../linux/docs/drivers/sql-server.md), [ClickHouse](../linux/docs/drivers/clickhouse.md), [Cassandra and ScyllaDB](../linux/docs/drivers/cassandra.md), [Redis](../linux/docs/drivers/redis.md), [MongoDB](../linux/docs/drivers/mongodb.md). An engine without SQL says so through `DriverCapabilities::data_model`: the workspace lists a key-value engine's keys and opens them in key tabs ([ADR 0021](../linux/docs/decisions/0021-engines-without-sql.md)), and opens a document engine's collections in table tabs whose statements are the engine's shell calls ([ADR 0026](../linux/docs/decisions/0026-documents-in-the-table-tab.md)). +`DatabaseDriver` describes an engine before any connection exists, through `DriverCapabilities`, and opens a `Connection`. The `Connection` serves the workspace from a pool; every call takes `CallOptions` (a deadline and a cancel token) and runs on a task the connection owns (`OwnedCalls`), so a stop reaches the server instead of leaving the statement running. Saves report what they left in the database through a `WriteLedger`. Each editor tab gets an `EditorSession` of its own that streams results into a `ResultSink`, and the `Session` that opens it puts a `ByteLimitSink` in front of that sink, so a run stops keeping rows once they pass `RunOptions::byte_limit` bytes across its results, the row that crosses it kept whole ([ADR 0020](docs/decisions/0020-a-run-keeps-a-byte-limit-of-rows.md)). A file engine's catalog read holds at most `ByteLimit::CATALOG_DEFAULT` bytes of names and definitions through `CatalogBudget`, and fails `DriverError::CatalogTooLarge` past it ([ADR 0029](docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md)). [docs/adding-drivers.md](../linux/docs/adding-drivers.md) is the full contract, [ADR 0016](../linux/docs/decisions/0016-driver-contract-v2.md) the reasons for it, and the engine pages say how each driver meets it: [PostgreSQL](../linux/docs/drivers/postgresql.md), [MySQL and MariaDB](../linux/docs/drivers/mysql.md), [SQLite](../linux/docs/drivers/sqlite.md), [DuckDB](../linux/docs/drivers/duckdb.md), [SQL Server](../linux/docs/drivers/sql-server.md), [ClickHouse](../linux/docs/drivers/clickhouse.md), [Cassandra and ScyllaDB](../linux/docs/drivers/cassandra.md), [Redis](../linux/docs/drivers/redis.md), [MongoDB](../linux/docs/drivers/mongodb.md). An engine without SQL says so through `DriverCapabilities::data_model`: the workspace lists a key-value engine's keys and opens them in key tabs ([ADR 0021](../linux/docs/decisions/0021-engines-without-sql.md)), and opens a document engine's collections in table tabs whose statements are the engine's shell calls ([ADR 0026](../linux/docs/decisions/0026-documents-in-the-table-tab.md)). ### SQL scripts diff --git a/core/crates/core/src/byte_limit.rs b/core/crates/core/src/byte_limit.rs index 18a6031a81..1a12adc3d2 100644 --- a/core/crates/core/src/byte_limit.rs +++ b/core/crates/core/src/byte_limit.rs @@ -9,6 +9,8 @@ pub struct ByteLimit(NonZeroU64); impl ByteLimit { // SI bytes, as the Linux banner prints them: "256.0 MB". pub const EDITOR_DEFAULT: Self = Self(NonZeroU64::new(256_000_000).expect("256 MB is not zero")); + // About a thousand times the largest real table's columns, keys and indexes, and a million table names. + pub const CATALOG_DEFAULT: Self = Self(NonZeroU64::new(64_000_000).expect("64 MB is not zero")); pub fn new(bytes: u64) -> Option { NonZeroU64::new(bytes).map(Self) diff --git a/core/crates/core/src/catalog_budget.rs b/core/crates/core/src/catalog_budget.rs new file mode 100644 index 0000000000..c2d10438eb --- /dev/null +++ b/core/crates/core/src/catalog_budget.rs @@ -0,0 +1,120 @@ +use crate::byte_limit::ByteLimit; +use crate::error::DriverError; + +// A file's catalog can repeat one long name across thousands of keys, so what a read holds grows with both; this +// bounds it the way `retained_bytes` measures a result's text. +#[derive(Debug)] +pub struct CatalogBudget { + limit: u64, + held: u64, +} + +impl CatalogBudget { + pub fn new(limit: ByteLimit) -> Self { + Self { + limit: limit.get(), + held: 0, + } + } + + pub fn charge(&mut self, bytes: usize) -> Result<(), DriverError> { + self.held = self.held.saturating_add(u64::try_from(bytes).unwrap_or(u64::MAX)); + match self.held > self.limit { + true => Err(DriverError::CatalogTooLarge { + limit_bytes: self.limit, + }), + false => Ok(()), + } + } + + pub fn charge_text(&mut self, text: &str) -> Result<(), DriverError> { + self.charge(text_bytes(text)) + } + + pub fn keep(&mut self, text: String) -> Result { + self.charge_text(&text)?; + Ok(text) + } + + pub fn keep_optional(&mut self, text: Option) -> Result, DriverError> { + text.map(|text| self.keep(text)).transpose() + } + + pub fn charge_texts<'a>(&mut self, texts: impl IntoIterator) -> Result<(), DriverError> { + texts.into_iter().try_for_each(|text| self.charge_text(text)) + } + + // For a text the read drops before it reads the next one, such as each child's CREATE TABLE. + pub fn release_text(&mut self, text: &str) { + self.held = self + .held + .saturating_sub(u64::try_from(text_bytes(text)).unwrap_or(u64::MAX)); + } +} + +impl Default for CatalogBudget { + fn default() -> Self { + Self::new(ByteLimit::CATALOG_DEFAULT) + } +} + +fn text_bytes(text: &str) -> usize { + size_of::().saturating_add(text.len()) +} + +#[cfg(test)] +mod tests { + use super::*; + + const SLOT: usize = size_of::(); + + fn budget(bytes: u64) -> CatalogBudget { + CatalogBudget::new(ByteLimit::new(bytes).unwrap()) + } + + #[test] + fn a_text_costs_its_slot_and_its_length() { + let mut budget = budget((SLOT + 5) as u64); + + assert_eq!(budget.keep("héll".to_owned()).as_deref(), Ok("héll")); + assert_eq!( + budget.charge_text("x"), + Err(DriverError::CatalogTooLarge { + limit_bytes: (SLOT + 5) as u64 + }) + ); + } + + #[test] + fn the_limit_itself_still_fits() { + let mut budget = budget(100); + + assert_eq!(budget.charge(100), Ok(())); + assert!(budget.charge(1).is_err()); + } + + #[test] + fn a_released_text_makes_room_again() { + let mut budget = budget((SLOT * 2 + 8) as u64); + let text = "a".repeat(8); + + budget.charge_text(&text).unwrap(); + budget.release_text(&text); + budget.charge_text(&text).unwrap(); + assert_eq!(budget.keep_optional(Some(String::new())), Ok(Some(String::new()))); + assert!(budget.charge_texts([&"b".to_owned()]).is_err()); + } + + #[test] + fn the_default_is_the_catalog_limit() { + let mut budget = CatalogBudget::default(); + + assert_eq!(budget.charge(64_000_000), Ok(())); + assert_eq!( + budget.charge(1), + Err(DriverError::CatalogTooLarge { + limit_bytes: 64_000_000 + }) + ); + } +} diff --git a/core/crates/core/src/error.rs b/core/crates/core/src/error.rs index 3568fbc27e..6312d587fc 100644 --- a/core/crates/core/src/error.rs +++ b/core/crates/core/src/error.rs @@ -143,6 +143,10 @@ pub enum DriverError { #[error("this engine has no {feature}")] Unsupported { feature: UnsupportedFeature }, + + // A crafted file can repeat a long name across thousands of keys, so a catalog read stops at a byte budget. + #[error("the catalog read would hold more than {limit_bytes} bytes")] + CatalogTooLarge { limit_bytes: u64 }, } impl DriverError { @@ -176,6 +180,7 @@ impl DriverError { Self::Decode { .. } => "driver.decode", Self::Protocol(..) => "driver.protocol", Self::Unsupported { .. } => "driver.unsupported", + Self::CatalogTooLarge { .. } => "driver.catalog_too_large", } } } @@ -216,7 +221,9 @@ impl DriverError { | Self::FileIo { .. } => ErrorCategory::File, Self::Busy(_) | Self::EditorSessionBusy => ErrorCategory::Busy, Self::Config(_) | Self::Unsupported { .. } | Self::DatabaseNotFound { .. } => ErrorCategory::Configuration, - Self::GuardUnsupported { .. } | Self::Decode { .. } | Self::Protocol(_) => ErrorCategory::Internal, + Self::GuardUnsupported { .. } | Self::Decode { .. } | Self::Protocol(_) | Self::CatalogTooLarge { .. } => { + ErrorCategory::Internal + } } } @@ -366,6 +373,10 @@ mod tests { (DriverError::Cancelled.code(), "driver.cancelled"), (DriverError::auth(None).code(), "driver.auth"), (DriverError::server("no such table").code(), "driver.server"), + ( + DriverError::CatalogTooLarge { limit_bytes: 1 }.code(), + "driver.catalog_too_large", + ), (DriverError::Config(ConfigError::EmptyHost).code(), "driver.config"), (ConfigError::EmptyHost.code(), "config.empty_host"), ( diff --git a/core/crates/core/src/lib.rs b/core/crates/core/src/lib.rs index c36800dbc4..73874815da 100644 --- a/core/crates/core/src/lib.rs +++ b/core/crates/core/src/lib.rs @@ -7,6 +7,7 @@ mod byte_limit; mod byte_limited_sink; mod call_options; mod capabilities; +mod catalog_budget; mod certificate_file_problem; mod client_identity; pub mod column; @@ -114,6 +115,7 @@ pub use capabilities::{ CommitResolutionSupport, DataModel, DriverCapabilities, EndpointKind, EstimateQuality, ReadOnlyEnforcement, ScriptExecution, StatementStop, }; +pub use catalog_budget::CatalogBudget; pub use certificate_file_problem::CertificateFileProblem; pub use client_identity::ClientIdentity; pub use column::ColumnInfo; diff --git a/core/crates/drivers/duckdb/src/catalog/mod.rs b/core/crates/drivers/duckdb/src/catalog/mod.rs index d7ec7f67fa..6fdf6ed3aa 100644 --- a/core/crates/drivers/duckdb/src/catalog/mod.rs +++ b/core/crates/drivers/duckdb/src/catalog/mod.rs @@ -6,7 +6,7 @@ mod declared_columns; -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use tablepro_core::column::{ColumnDefault, ColumnInfo, ColumnType, SqlExpression}; use tablepro_core::ddl::{ @@ -21,7 +21,7 @@ use tablepro_core::meta::{ }; use tablepro_core::sql_syntax::{LiteralToken, MAX_EXPRESSION_LEN, SqlGrammar, parse_expression}; use tablepro_core::value::Value; -use tablepro_core::{DriverError, ForeignKeyInfo}; +use tablepro_core::{CatalogBudget, DriverError, ForeignKeyInfo}; use self::declared_columns::declared_columns; use crate::duck_type::column_type_from_text; @@ -39,8 +39,14 @@ fn schema_of(table: &TableRef) -> &str { .unwrap_or(DEFAULT_SCHEMA) } -/// Rows of a catalogue query whose columns are all text or NULL. -fn text_rows(link: &Link, sql: &str, params: &[&str]) -> Result>>, DriverError> { +/// Rows of a catalogue query whose columns are all text or NULL, each +/// value counted against `budget` as it is read. +fn text_rows( + link: &Link, + sql: &str, + params: &[&str], + budget: &mut CatalogBudget, +) -> Result>>, DriverError> { let mut statement = link.connection.prepare(sql).map_err(map_error)?; let mut rows = statement .query(duckdb::params_from_iter(params.iter())) @@ -50,7 +56,7 @@ fn text_rows(link: &Link, sql: &str, params: &[&str]) -> Result>(index).map_err(map_error)?); + values.push(budget.keep_optional(row.get::<_, Option>(index).map_err(map_error)?)?); } read.push(values); } @@ -61,6 +67,11 @@ fn text(row: &[Option], index: usize) -> String { row.get(index).cloned().flatten().unwrap_or_default() } +// Moved out rather than copied, so a row of long names is not held twice. +fn take(row: &mut [Option], index: usize) -> String { + row.get_mut(index).and_then(Option::take).unwrap_or_default() +} + fn flag(row: &[Option], index: usize) -> bool { row.get(index) .and_then(Option::as_deref) @@ -72,6 +83,7 @@ pub(crate) fn list_schemas(link: &Link, catalog: &str) -> Result link, "SELECT schema_name FROM duckdb_schemas() WHERE database_name = ? ORDER BY schema_name", &[catalog], + &mut CatalogBudget::default(), )?; Ok(rows .iter() @@ -86,7 +98,12 @@ pub(crate) fn list_schemas(link: &Link, catalog: &str) -> Result .collect()) } -pub(crate) fn list_objects(link: &Link, catalog: &str, scope: &ObjectScope) -> Result, DriverError> { +pub(crate) fn list_objects( + link: &Link, + catalog: &str, + scope: &ObjectScope, + budget: &mut CatalogBudget, +) -> Result, DriverError> { let schema = match scope { ObjectScope::DefaultSchema => Some(DEFAULT_SCHEMA), ObjectScope::Schema(schema) => Some(schema.as_str()), @@ -106,8 +123,8 @@ pub(crate) fn list_objects(link: &Link, catalog: &str, scope: &ObjectScope) -> R while let Some(row) = rows.next().map_err(map_error)? { let kind: String = row.get(2).map_err(map_error)?; objects.push(ObjectInfo { - schema: Some(row.get(0).map_err(map_error)?), - name: row.get(1).map_err(map_error)?, + schema: Some(budget.keep(row.get(0).map_err(map_error)?)?), + name: budget.keep(row.get(1).map_err(map_error)?)?, kind: match kind.as_str() { "view" => ObjectKind::View, _ => ObjectKind::Table, @@ -120,7 +137,12 @@ pub(crate) fn list_objects(link: &Link, catalog: &str, scope: &ObjectScope) -> R /// A table's columns, with the primary key marked and the generated /// columns told apart from defaults. -pub(crate) fn fetch_columns(link: &Link, catalog: &str, table: &TableRef) -> Result, DriverError> { +pub(crate) fn fetch_columns( + link: &Link, + catalog: &str, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result, DriverError> { let schema = schema_of(table); let rows = text_rows( link, @@ -128,19 +150,23 @@ pub(crate) fn fetch_columns(link: &Link, catalog: &str, table: &TableRef) -> Res FROM duckdb_columns() WHERE database_name = ? AND schema_name = ? AND table_name = ? \ ORDER BY column_index", &[catalog, schema, &table.name], + budget, )?; - let keys = primary_key(link, catalog, table)? + let keys = primary_key(link, catalog, table, budget)? .map(|key| key.columns) .unwrap_or_default(); - let generated: HashSet = table_sql(link, catalog, table)? + let sql = table_sql(link, catalog, table, budget)?; + let generated: HashSet = sql + .as_deref() .map(|sql| { - declared_columns(&sql) + declared_columns(sql) .into_iter() .filter(|column| column.generated) .map(|column| column.name) .collect() }) .unwrap_or_default(); + release(budget, sql); Ok(rows .iter() .map(|row| { @@ -196,13 +222,26 @@ pub(crate) fn classify_default(text: Option<&str>, column_type: &ColumnType) -> /// The CREATE TABLE statement DuckDB keeps for a table, which a view has /// none of. -fn table_sql(link: &Link, catalog: &str, table: &TableRef) -> Result, DriverError> { - let rows = text_rows( +fn table_sql( + link: &Link, + catalog: &str, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result, DriverError> { + let mut rows = text_rows( link, "SELECT sql FROM duckdb_tables() WHERE database_name = ? AND schema_name = ? AND table_name = ?", &[catalog, schema_of(table), &table.name], + budget, )?; - Ok(rows.first().and_then(|row| row.first().cloned().flatten())) + Ok(rows.first_mut().and_then(|row| row.first_mut().and_then(Option::take))) +} + +// For a text read, used and dropped before the next one is read. +fn release(budget: &mut CatalogBudget, text: Option) { + if let Some(text) = text { + budget.release_text(&text); + } } /// The columns of a table's PRIMARY KEY and UNIQUE constraints, in key @@ -211,6 +250,7 @@ fn key_constraints( link: &Link, catalog: &str, table: &TableRef, + budget: &mut CatalogBudget, ) -> Result)>, DriverError> { let rows = text_rows( link, @@ -220,10 +260,11 @@ fn key_constraints( AND constraint_type IN ('PRIMARY KEY', 'UNIQUE') \ ORDER BY constraint_index", &[catalog, schema_of(table), &table.name], + budget, )?; Ok(rows - .iter() - .map(|row| (text(row, 0), text(row, 1), split_names(&text(row, 2)))) + .into_iter() + .map(|mut row| (take(&mut row, 0), take(&mut row, 1), split_names(&take(&mut row, 2)))) .collect()) } @@ -231,8 +272,9 @@ pub(crate) fn primary_key( link: &Link, catalog: &str, table: &TableRef, + budget: &mut CatalogBudget, ) -> Result, DriverError> { - Ok(key_constraints(link, catalog, table)? + Ok(key_constraints(link, catalog, table, budget)? .into_iter() .find(|(_, kind, _)| kind == "PRIMARY KEY") .map(|(name, _, columns)| PrimaryKeyConstraint { @@ -244,8 +286,13 @@ pub(crate) fn primary_key( /// The table's indexes: the ones created with CREATE INDEX, and the /// index behind each PRIMARY KEY and UNIQUE constraint, which DuckDB /// lists only as the constraint. -pub(crate) fn fetch_indexes(link: &Link, catalog: &str, table: &TableRef) -> Result, DriverError> { - let mut indexes: Vec = key_constraints(link, catalog, table)? +pub(crate) fn fetch_indexes( + link: &Link, + catalog: &str, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result, DriverError> { + let mut indexes: Vec = key_constraints(link, catalog, table, budget)? .into_iter() .map(|(name, kind, columns)| IndexInfo { name, @@ -262,7 +309,7 @@ pub(crate) fn fetch_indexes(link: &Link, catalog: &str, table: &TableRef) -> Res definition: None, }) .collect(); - let columns: HashSet = fetch_columns(link, catalog, table)? + let columns: HashSet = fetch_columns(link, catalog, table, budget)? .into_iter() .map(|column| column.name) .collect(); @@ -274,6 +321,7 @@ pub(crate) fn fetch_indexes(link: &Link, catalog: &str, table: &TableRef) -> Res FROM duckdb_indexes() WHERE database_name = ? AND schema_name = ? AND table_name = ? \ ORDER BY index_name", &[catalog, schema_of(table), &table.name], + budget, )?; let quote = |name: &str| format!("\"{}\"", name.replace('"', "\"\"")); let target = format!("{}.{}.{}", quote(catalog), quote(schema_of(table)), quote(&table.name)); @@ -363,6 +411,7 @@ pub(crate) fn fetch_foreign_keys( link: &Link, catalog: &str, table: &TableRef, + budget: &mut CatalogBudget, ) -> Result, DriverError> { let rows = text_rows( link, @@ -373,20 +422,23 @@ pub(crate) fn fetch_foreign_keys( WHERE database_name = ? AND schema_name = ? AND table_name = ? AND constraint_type = 'FOREIGN KEY' \ ORDER BY constraint_index", &[catalog, schema_of(table), &table.name], + budget, )?; - let schema = schema_of(table).to_owned(); - Ok(rows - .iter() - .map(|row| ForeignKeyInfo { - name: text(row, 0), - columns: split_names(&text(row, 2)), - ref_schema: Some(schema.clone()), - ref_table: text(row, 1), - ref_columns: split_names(&text(row, 3)), - on_delete: Some(ReferentialAction::NoAction), - on_update: Some(ReferentialAction::NoAction), + let schema = schema_of(table); + rows.into_iter() + .map(|mut row| { + // Each key holds its own copy of the schema's name, which the file chose. + Ok(ForeignKeyInfo { + name: take(&mut row, 0), + columns: split_names(&take(&mut row, 2)), + ref_schema: Some(budget.keep(schema.to_owned())?), + ref_table: take(&mut row, 1), + ref_columns: split_names(&take(&mut row, 3)), + on_delete: Some(ReferentialAction::NoAction), + on_update: Some(ReferentialAction::NoAction), + }) }) - .collect()) + .collect() } /// The table's own keys, each with whether `=` on the referenced columns @@ -396,12 +448,22 @@ pub(crate) fn outbound_foreign_keys( link: &Link, catalog: &str, table: &TableRef, + budget: &mut CatalogBudget, ) -> Result, DriverError> { - let keys = fetch_foreign_keys(link, catalog, table)?; + let keys = fetch_foreign_keys(link, catalog, table, budget)?; let mut outbound = Vec::with_capacity(keys.len()); + // Each parent's CREATE TABLE once, however many keys point at it. + let mut parents: HashMap<(Option, String), Vec> = HashMap::new(); for key in keys { - let referenced = TableRef::new(key.ref_schema.clone(), key.ref_table.clone()); - let collated = collated_columns(link, catalog, &referenced)?; + let parent = (key.ref_schema.clone(), key.ref_table.clone()); + let collated = match parents.get(&parent) { + Some(collated) => collated, + None => { + let referenced = TableRef::new(key.ref_schema.clone(), key.ref_table.clone()); + let collated = collated_columns(link, catalog, &referenced, budget)?; + parents.entry(parent).or_insert(collated) + } + }; let parent_compares_alike = !key .ref_columns .iter() @@ -416,16 +478,25 @@ pub(crate) fn outbound_foreign_keys( /// The columns of a table declared with a collation, as its stored SQL /// spells them. -fn collated_columns(link: &Link, catalog: &str, table: &TableRef) -> Result, DriverError> { - Ok(table_sql(link, catalog, table)? +fn collated_columns( + link: &Link, + catalog: &str, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result, DriverError> { + let sql = table_sql(link, catalog, table, budget)?; + let collated = sql + .as_deref() .map(|sql| { - declared_columns(&sql) + declared_columns(sql) .into_iter() .filter(|column| column.collation.is_some()) .map(|column| column.name) .collect() }) - .unwrap_or_default()) + .unwrap_or_default(); + release(budget, sql); + Ok(collated) } fn split_names(joined: &str) -> Vec { @@ -443,41 +514,44 @@ pub(crate) fn inbound_foreign_keys( link: &Link, catalog: &str, table: &TableRef, + budget: &mut CatalogBudget, ) -> Result, DriverError> { + // Not the child's CREATE TABLE on every key: it grows with the keys the child declares. let rows = text_rows( link, "SELECT c.constraint_name, c.schema_name, c.table_name, \ array_to_string(c.constraint_column_names, chr(31)), \ - array_to_string(c.referenced_column_names, chr(31)), t.sql \ + array_to_string(c.referenced_column_names, chr(31)) \ FROM duckdb_constraints() c \ - JOIN duckdb_tables() t \ - ON t.database_name = c.database_name AND t.schema_name = c.schema_name AND t.table_name = c.table_name \ WHERE c.database_name = ? AND c.schema_name = ? AND c.referenced_table = ? \ AND c.constraint_type = 'FOREIGN KEY' \ ORDER BY c.table_name, c.constraint_index", &[catalog, schema_of(table), &table.name], + budget, )?; - Ok(rows - .iter() - .map(|row| { - let columns = split_names(&text(row, 3)); - let collated: Vec = declared_columns(&text(row, 5)) - .into_iter() - .filter(|column| column.collation.is_some()) - .map(|column| column.name) - .collect(); - let compares_alike = !columns - .iter() - .any(|column| collated.iter().any(|name| name.eq_ignore_ascii_case(column))); - InboundForeignKey { - name: text(row, 0), - table: TableRef::new(Some(text(row, 1)), text(row, 2)), - child_compares_alike: compares_alike, - columns, - referenced_columns: split_names(&text(row, 4)), - } - }) - .collect()) + let mut keys = Vec::with_capacity(rows.len()); + let mut child: Option<(TableRef, Vec)> = None; + for mut row in rows { + let referencing = TableRef::new(Some(take(&mut row, 1)), take(&mut row, 2)); + // The rows come by child, so each child's CREATE TABLE is read once. + if child.as_ref().is_none_or(|(read, _)| *read != referencing) { + let collated = collated_columns(link, catalog, &referencing, budget)?; + child = Some((referencing.clone(), collated)); + } + let collated: &[String] = child.as_ref().map_or(&[], |(_, collated)| collated); + let columns = split_names(&take(&mut row, 3)); + let compares_alike = !columns + .iter() + .any(|column| collated.iter().any(|name| name.eq_ignore_ascii_case(column))); + keys.push(InboundForeignKey { + name: take(&mut row, 0), + table: referencing, + child_compares_alike: compares_alike, + columns, + referenced_columns: split_names(&take(&mut row, 4)), + }); + } + Ok(keys) } /// How a row is named again: by the primary key, else by a UNIQUE key @@ -486,14 +560,19 @@ pub(crate) fn inbound_foreign_keys( /// over a column declared with a collation: DuckDB keeps the key unique /// byte for byte, while `=` and ORDER BY compare under the collation, so /// `'ABC'` and `'abc'` are two rows one predicate finds. -pub(crate) fn row_identity(link: &Link, catalog: &str, table: &TableRef) -> Result { - let collated = collated_columns(link, catalog, table)?; +pub(crate) fn row_identity( + link: &Link, + catalog: &str, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result { + let collated = collated_columns(link, catalog, table, budget)?; let compares_bytes = |columns: &[String]| { !columns .iter() .any(|column| collated.iter().any(|name| name.eq_ignore_ascii_case(column))) }; - let constraints = key_constraints(link, catalog, table)?; + let constraints = key_constraints(link, catalog, table, budget)?; if let Some((_, _, columns)) = constraints .iter() .find(|(_, kind, columns)| kind == "PRIMARY KEY" && compares_bytes(columns)) @@ -502,7 +581,7 @@ pub(crate) fn row_identity(link: &Link, catalog: &str, table: &TableRef) -> Resu columns: columns.clone(), }); } - let not_null: HashSet = fetch_columns(link, catalog, table)? + let not_null: HashSet = fetch_columns(link, catalog, table, budget)? .into_iter() .filter(|column| !column.nullable) .map(|column| column.name) @@ -526,6 +605,7 @@ pub(crate) fn table_estimate(link: &Link, catalog: &str, table: &TableRef) -> Re "SELECT CAST(estimated_size AS VARCHAR) FROM duckdb_tables() \ WHERE database_name = ? AND schema_name = ? AND table_name = ?", &[catalog, schema_of(table), &table.name], + &mut CatalogBudget::default(), )?; let estimate = rows .first() @@ -542,10 +622,11 @@ pub(crate) fn fetch_table_definition( link: &Link, catalog: &str, table: &TableRef, + budget: &mut CatalogBudget, ) -> Result { - let create_sql = table_sql(link, catalog, table)?; - let columns = fetch_columns(link, catalog, table)?; - let check_columns = check_constraints(link, catalog, table)?; + let create_sql = table_sql(link, catalog, table, budget)?; + let columns = fetch_columns(link, catalog, table, budget)?; + let check_columns = check_constraints(link, catalog, table, budget)?; let collations: Vec<(String, String)> = create_sql .as_deref() .map(declared_columns) @@ -581,18 +662,25 @@ pub(crate) fn fetch_table_definition( ColumnDefinition::new(info, attributes) }) .collect(); - let indexes = fetch_indexes(link, catalog, table)?; - let source = create_sql.map(|create_statement| TableSource { - create_statement, - dependents: dependent_sources(link, catalog, table).unwrap_or_default(), - }); + let indexes = fetch_indexes(link, catalog, table, budget)?; + let source = match create_sql { + Some(create_statement) => Some(TableSource { + create_statement, + // Missing indexes would be dropped by a rebuild, so a definition past the budget is no definition. + dependents: match dependent_sources(link, catalog, table, budget) { + Err(error @ DriverError::CatalogTooLarge { .. }) => return Err(error), + read => read.unwrap_or_default(), + }, + }), + None => None, + }; Ok(TableDefinition { table: TableRef::new(Some(schema_of(table).to_owned()), table.name.clone()), columns: definitions, - primary_key: primary_key(link, catalog, table)?, + primary_key: primary_key(link, catalog, table, budget)?, indexes, - foreign_keys: fetch_foreign_keys(link, catalog, table)?, - inbound_foreign_keys: inbound_foreign_keys(link, catalog, table)?, + foreign_keys: fetch_foreign_keys(link, catalog, table, budget)?, + inbound_foreign_keys: inbound_foreign_keys(link, catalog, table, budget)?, source, options: None, }) @@ -605,6 +693,7 @@ fn generation_expression(link: &Link, catalog: &str, table: &TableRef, column: & "SELECT column_default FROM duckdb_columns() \ WHERE database_name = ? AND schema_name = ? AND table_name = ? AND column_name = ?", &[catalog, schema_of(table), &table.name, column], + &mut CatalogBudget::default(), ) .ok()?; rows.first() @@ -620,6 +709,7 @@ fn check_constraints( link: &Link, catalog: &str, table: &TableRef, + budget: &mut CatalogBudget, ) -> Result, DriverError> { let rows = text_rows( link, @@ -627,50 +717,62 @@ fn check_constraints( WHERE database_name = ? AND schema_name = ? AND table_name = ? AND constraint_type = 'CHECK' \ ORDER BY constraint_index", &[catalog, schema_of(table), &table.name], + budget, )?; - let mut checks: std::collections::HashMap> = std::collections::HashMap::new(); - for row in &rows { - let Some(expression) = row.get(1).cloned().flatten() else { - continue; - }; - for column in split_names(&text(row, 0)) { - checks.entry(column).or_default().push(expression.clone()); + // A CHECK over every column is each column's, so its text is held once per column it names. + let mut checks: std::collections::HashMap> = std::collections::HashMap::new(); + for (index, row) in rows.iter().enumerate() { + if row.get(1).is_some_and(Option::is_some) { + for column in split_names(&text(row, 0)) { + checks.entry(column).or_default().push(index); + } } } - Ok(checks + let expression = |index: &usize| { + rows.get(*index) + .map_or("", |row| row.get(1).and_then(Option::as_deref).unwrap_or("")) + }; + checks .into_iter() .map(|(column, expressions)| { let joined = match expressions.as_slice() { - [only] => only.clone(), + [only] => expression(only).to_owned(), many => many .iter() - .map(|one| format!("({one})")) + .map(|one| format!("({})", expression(one))) .collect::>() .join(" AND "), }; - (column, SqlExpression::from_catalog_text(joined)) + Ok((column, SqlExpression::from_catalog_text(budget.keep(joined)?))) }) - .collect()) + .collect() } /// The indexes created on the table, which a table rebuilt from its /// CREATE TABLE text would lose. -fn dependent_sources(link: &Link, catalog: &str, table: &TableRef) -> Result, DriverError> { +fn dependent_sources( + link: &Link, + catalog: &str, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result, DriverError> { let rows = text_rows( link, "SELECT index_name, sql FROM duckdb_indexes() \ WHERE database_name = ? AND schema_name = ? AND table_name = ? AND sql IS NOT NULL ORDER BY index_name", &[catalog, schema_of(table), &table.name], + budget, )?; - Ok(rows - .iter() - .map(|row| SchemaObjectSource { - kind: SchemaObjectKind::Index, - name: text(row, 0), - table: table.name.clone(), - sql: text(row, 1), + rows.into_iter() + .map(|mut row| { + Ok(SchemaObjectSource { + kind: SchemaObjectKind::Index, + name: take(&mut row, 0), + table: budget.keep(table.name.clone())?, + sql: take(&mut row, 1), + }) }) - .collect()) + .collect() } #[cfg(test)] diff --git a/core/crates/drivers/duckdb/src/connection.rs b/core/crates/drivers/duckdb/src/connection.rs index f4ac5823bb..cb4554ed26 100644 --- a/core/crates/drivers/duckdb/src/connection.rs +++ b/core/crates/drivers/duckdb/src/connection.rs @@ -8,7 +8,7 @@ use tablepro_core::dump::{DumpCatalog, DumpScope}; use tablepro_core::meta::{IndexInfo, ObjectInfo, ObjectScope, RowIdentity, SchemaInfo, TableEstimate, TableRef}; use tablepro_core::statement::Statement; use tablepro_core::{ - CallOptions, ColumnInfo, CommitResolution, CommitToken, Connection, DriverError, EditorSession, + CallOptions, CatalogBudget, ColumnInfo, CommitResolution, CommitToken, Connection, DriverError, EditorSession, EditorSessionOptions, FileEndpoint, ForeignKeyInfo, LinkState, LinkWatch, LivenessPolicy, LossPhase, OwnedCalls, QueryResult, ReadOnlyRefusal, ReadStatement, ResultSink, RowLimit, ServerInfo, SinkFlow, StopRequest, StreamSummary, TimeoutPhase, WorkspaceSettings, WriteAtomicity, WriteBatch, WriteKind, WriteLedger, WriteReport, @@ -269,7 +269,7 @@ impl Connection for DuckDbConnection { async fn list_objects(&self, scope: &ObjectScope, options: CallOptions) -> Result, DriverError> { let scope = scope.clone(); self.read(options, move |link, catalog| { - catalog::list_objects(link, catalog, &scope) + catalog::list_objects(link, catalog, &scope, &mut CatalogBudget::default()) }) .await } @@ -277,7 +277,7 @@ impl Connection for DuckDbConnection { async fn fetch_columns(&self, table: &TableRef, options: CallOptions) -> Result, DriverError> { let table = table.clone(); self.read(options, move |link, catalog| { - catalog::fetch_columns(link, catalog, &table) + catalog::fetch_columns(link, catalog, &table, &mut CatalogBudget::default()) }) .await } @@ -285,7 +285,7 @@ impl Connection for DuckDbConnection { async fn fetch_indexes(&self, table: &TableRef, options: CallOptions) -> Result, DriverError> { let table = table.clone(); self.read(options, move |link, catalog| { - catalog::fetch_indexes(link, catalog, &table) + catalog::fetch_indexes(link, catalog, &table, &mut CatalogBudget::default()) }) .await } @@ -297,7 +297,7 @@ impl Connection for DuckDbConnection { ) -> Result, DriverError> { let table = table.clone(); self.read(options, move |link, catalog| { - catalog::fetch_foreign_keys(link, catalog, &table) + catalog::fetch_foreign_keys(link, catalog, &table, &mut CatalogBudget::default()) }) .await } @@ -309,7 +309,7 @@ impl Connection for DuckDbConnection { ) -> Result, DriverError> { let table = table.clone(); self.read(options, move |link, catalog| { - catalog::outbound_foreign_keys(link, catalog, &table) + catalog::outbound_foreign_keys(link, catalog, &table, &mut CatalogBudget::default()) }) .await } @@ -321,7 +321,7 @@ impl Connection for DuckDbConnection { ) -> Result, DriverError> { let table = table.clone(); self.read(options, move |link, catalog| { - catalog::inbound_foreign_keys(link, catalog, &table) + catalog::inbound_foreign_keys(link, catalog, &table, &mut CatalogBudget::default()) }) .await } @@ -335,7 +335,7 @@ impl Connection for DuckDbConnection { // One transaction, so the parts come from the same moment. self.read(options, move |link, catalog| { batch(link, "BEGIN TRANSACTION READ ONLY")?; - let definition = catalog::fetch_table_definition(link, catalog, &table); + let definition = catalog::fetch_table_definition(link, catalog, &table, &mut CatalogBudget::default()); link.hold_stops(); batch(link, "ROLLBACK")?; definition @@ -359,7 +359,7 @@ impl Connection for DuckDbConnection { async fn row_identity(&self, table: &TableRef, options: CallOptions) -> Result { let table = table.clone(); self.read(options, move |link, catalog| { - catalog::row_identity(link, catalog, &table) + catalog::row_identity(link, catalog, &table, &mut CatalogBudget::default()) }) .await } diff --git a/core/crates/drivers/duckdb/src/dump.rs b/core/crates/drivers/duckdb/src/dump.rs index 00df54ed7d..27a34c6afb 100644 --- a/core/crates/drivers/duckdb/src/dump.rs +++ b/core/crates/drivers/duckdb/src/dump.rs @@ -4,6 +4,8 @@ //! triggers EXPORT leaves out. Names carry their schema and never the //! catalog, which is the file's own name. +use std::collections::HashMap; + use duckdb::types::Value as DuckValue; use tablepro_core::DriverError; use tablepro_core::dialect::SqlDialect; @@ -60,10 +62,13 @@ const TABLES_SQL: &str = "SELECT schema_name, table_name, sql FROM duckdb_tables WHERE database_name = ? AND (? IS NULL OR schema_name = ?) AND NOT internal AND NOT temporary \ ORDER BY schema_name, table_name"; -/// DuckDB keeps a foreign key's parent in the child's schema. -const FOREIGN_KEYS_SQL: &str = "SELECT schema_name, table_name, referenced_table FROM duckdb_constraints() \ +/// DuckDB keeps a foreign key's parent in the child's schema. One row +/// per child, not per key, which would repeat the child's name. +const FOREIGN_KEYS_SQL: &str = "SELECT schema_name, table_name, \ + string_agg(referenced_table, chr(31) ORDER BY constraint_index) \ + FROM duckdb_constraints() \ WHERE database_name = ? AND (? IS NULL OR schema_name = ?) AND constraint_type = 'FOREIGN KEY' \ - ORDER BY schema_name, table_name, constraint_index"; + GROUP BY schema_name, table_name ORDER BY schema_name, table_name"; /// The index behind a PRIMARY KEY or UNIQUE has no text: its table's /// CREATE makes it. @@ -321,7 +326,9 @@ fn add_tables(reader: &Reader<'_>, dump: &mut DumpCatalog) -> Result<(), DriverE Ok(( row.get::<_, String>(0)?, row.get::<_, String>(1)?, - row.get::<_, Option>(2)?, + row.get::<_, Option>(2)? + .map(|parents| parents.split('\u{1f}').map(str::to_owned).collect()) + .unwrap_or_default(), )) })?; let listed: Vec<&Named> = tables.iter().filter(|table| table.text().is_some()).collect(); @@ -341,22 +348,31 @@ fn add_tables(reader: &Reader<'_>, dump: &mut DumpCatalog) -> Result<(), DriverE /// DuckDB checks a foreign key as each statement ends, and has no way /// to add one after the rows, so a table's rows follow its parents', and /// a table whose key points at itself takes a row per statement. -fn dump_tables(tables: &[&Named], keys: &[(String, String, Option)]) -> Vec { +fn dump_tables(tables: &[&Named], keys: &[(String, String, Vec)]) -> Vec { + let parents_of: HashMap<(&str, &str), &[String]> = keys + .iter() + .map(|(schema, child, parents)| ((schema.as_str(), child.as_str()), parents.as_slice())) + .collect(); + // A key keeps its parent as the CREATE spelled it, and DuckDB folds the case of names. + let mut declared_names: HashMap<(&str, String), &str> = HashMap::new(); + for table in tables { + declared_names + .entry((table.schema.as_str(), table.name.to_ascii_lowercase())) + .or_insert(table.name.as_str()); + } tables .iter() .map(|table| { let mut dumped = DumpTable::new(TableRef::new(Some(table.schema.clone()), table.name.clone())); - let parents = keys - .iter() - .filter(|(schema, child, _)| *schema == table.schema && *child == table.name) - .filter_map(|(_, _, parent)| parent.as_deref()); + let parents = parents_of + .get(&(table.schema.as_str(), table.name.as_str())) + .copied() + .unwrap_or_default(); for parent in parents { - // A key keeps its parent as the CREATE spelled it, and - // DuckDB folds the case of names. - let declared = tables - .iter() - .find(|other| other.schema == table.schema && other.name.eq_ignore_ascii_case(parent)) - .map_or(parent, |other| other.name.as_str()); + let declared = declared_names + .get(&(table.schema.as_str(), parent.to_ascii_lowercase())) + .copied() + .unwrap_or(parent.as_str()); if declared == table.name { dumped.one_row_per_insert = true; continue; @@ -503,8 +519,12 @@ mod tests { } } - fn key(schema: &str, child: &str, parent: &str) -> (String, String, Option) { - (schema.to_owned(), child.to_owned(), Some(parent.to_owned())) + fn key(schema: &str, child: &str, parents: &[&str]) -> (String, String, Vec) { + ( + schema.to_owned(), + child.to_owned(), + parents.iter().map(|parent| (*parent).to_owned()).collect(), + ) } #[test] @@ -544,11 +564,7 @@ mod tests { fn a_table_follows_its_parents_and_a_self_reference_loads_a_row_at_a_time() { let (parent, child, other) = (named("app", "parent"), named("app", "child"), named("main", "parent")); let tables = [&child, &parent, &other]; - let keys = [ - key("app", "child", "Parent"), - key("app", "child", "parent"), - key("app", "child", "child"), - ]; + let keys = [key("app", "child", &["Parent", "parent", "child"])]; let dumped = dump_tables(&tables, &keys); diff --git a/core/crates/drivers/duckdb/tests/catalog_memory.rs b/core/crates/drivers/duckdb/tests/catalog_memory.rs new file mode 100644 index 0000000000..e3eed29e1d --- /dev/null +++ b/core/crates/drivers/duckdb/tests/catalog_memory.rs @@ -0,0 +1,110 @@ +// Catalog reads of a crafted file hold what they return, and stop at the catalog budget. A test binary of its own +// because it counts every heap allocation the process makes, which another test running beside it would add to. Add +// no other test here. + +use std::time::Duration; + +use tablepro_core::meta::TableRef; +use tablepro_core::{ + ByteLimit, CallOptions, ConnectEnvironment, ConnectOptions, ConnectTarget, DatabaseDriver, DriverError, + FileEndpoint, WorkspaceSettings, +}; +use tablepro_driver_duckdb::DuckDbDriver; +use tablepro_test_support::conformance::test_liveness; +use tablepro_test_support::{HeapCounter, heap_peak_of}; +use tokio::time::Instant; +use tokio_util::sync::CancellationToken; + +#[global_allocator] +static HEAP: HeapCounter = HeapCounter; + +// The budget counts the text a read holds, not the vectors and maps around it. +const BOUND: usize = 2 * ByteLimit::CATALOG_DEFAULT.get() as usize; + +const KEYS: usize = 1_000; +const NAMED_KEYS: usize = 600; + +// DuckDB takes seconds to create and open more than this: +// - a child declaring 1,000 keys into one parent, whose CREATE TABLE, 40 KB, was read again for each of them; +// - a child with a 64 KiB name and 600 keys, each named after it, and its name again on each key into the parent; +// - a CHECK naming 600 columns of 200-byte names, which is each column's. Flat: DuckDB binds a nested one on the +// calling thread's stack. +fn crafted_schema() -> String { + let column_names: Vec = (0..600) + .map(|index| format!("c{index:03}{}", "c".repeat(196))) + .collect(); + let columns: Vec = column_names.iter().map(|name| format!("{name} INTEGER")).collect(); + format!( + "CREATE TABLE parent (id INTEGER PRIMARY KEY); \ + CREATE TABLE child (a INTEGER{}); \ + CREATE TABLE named_parent (id INTEGER PRIMARY KEY); \ + CREATE TABLE {} (a INTEGER{}); \ + CREATE TABLE checked ({}, CHECK (greatest({}) > 0)); \ + CHECKPOINT;", + ", FOREIGN KEY (a) REFERENCES parent (id)".repeat(KEYS), + "n".repeat(64 * 1024), + ", FOREIGN KEY (a) REFERENCES named_parent (id)".repeat(NAMED_KEYS), + columns.join(", "), + column_names.join(", "), + ) +} + +fn call() -> CallOptions { + CallOptions::new( + Some(Instant::now() + Duration::from_secs(120)), + CancellationToken::new(), + ) +} + +fn too_large(read: &str, result: Result) { + let limit = ByteLimit::CATALOG_DEFAULT.get(); + assert!( + matches!(result, Err(DriverError::CatalogTooLarge { limit_bytes }) if limit_bytes == limit), + "{read}: {:?}", + result.err() + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_crafted_catalog_holds_what_it_returns() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("crafted.duckdb"); + duckdb::Connection::open(&path) + .unwrap() + .execute_batch(&crafted_schema()) + .unwrap(); + let options = ConnectOptions { + target: ConnectTarget::File(FileEndpoint::open_existing(path).unwrap()), + workspace_settings: WorkspaceSettings::workspace(true), + liveness: test_liveness(), + environment: ConnectEnvironment::new("test"), + }; + let connection = DuckDbDriver.connect(options, CancellationToken::new()).await.unwrap(); + let table = |name: &str| TableRef::new(Some("main".to_owned()), name); + + let (outbound, peak) = heap_peak_of(connection.fetch_outbound_foreign_keys(&table("child"), call())).await; + assert_eq!(outbound.unwrap().len(), KEYS); + assert!(peak < BOUND / 8, "the keys of child held {peak} bytes"); + let (inbound, peak) = heap_peak_of(connection.fetch_inbound_foreign_keys(&table("parent"), call())).await; + assert_eq!(inbound.unwrap().len(), KEYS); + assert!(peak < BOUND / 8, "the keys into parent held {peak} bytes"); + let (definition, peak) = heap_peak_of(connection.fetch_table_definition(&table("parent"), call())).await; + assert_eq!(definition.unwrap().inbound_foreign_keys.len(), KEYS); + assert!(peak < BOUND / 8, "the definition of parent held {peak} bytes"); + + let named = table(&"n".repeat(64 * 1024)); + let (inbound, peak) = heap_peak_of(connection.fetch_inbound_foreign_keys(&table("named_parent"), call())).await; + too_large("keys into named_parent", inbound); + assert!(peak < BOUND, "the keys into named_parent held {peak} bytes"); + let (keys, peak) = heap_peak_of(connection.fetch_foreign_keys(&named, call())).await; + assert_eq!( + keys.map(|keys| keys.len()), + Ok(NAMED_KEYS), + "its own keys, named once each, fit" + ); + assert!(peak < BOUND, "the keys of the long child held {peak} bytes"); + let (checked, peak) = heap_peak_of(connection.fetch_table_definition(&table("checked"), call())).await; + too_large("definition of checked", checked); + assert!(peak < BOUND, "the definition of checked held {peak} bytes"); + connection.close().await.unwrap(); +} diff --git a/core/crates/drivers/sqlite/src/catalog/mod.rs b/core/crates/drivers/sqlite/src/catalog/mod.rs index f2e278239f..75238c02a4 100644 --- a/core/crates/drivers/sqlite/src/catalog/mod.rs +++ b/core/crates/drivers/sqlite/src/catalog/mod.rs @@ -20,7 +20,7 @@ use tablepro_core::value::{SqlDecimal, Value}; pub(crate) use dependents::{SchemaRow, select_dependents}; pub use runtime_version::{SQLITE_MIN_VERSION, SqliteVersion, check_runtime_version}; pub(crate) use sqlite_column_row::{SqliteColumnRow, TableFacts}; -pub(crate) use table_definition::{InboundKeyRow, foreign_key_names, generated_expressions, inbound_foreign_keys}; +pub(crate) use table_definition::{InboundKeyColumn, foreign_key_names, generated_expressions, inbound_foreign_keys}; /// The schema unqualified names resolve to. pub(crate) const MAIN_SCHEMA: &str = "main"; @@ -45,11 +45,17 @@ pub(crate) const INDEX_LIST_SQL: &str = pub(crate) const INDEX_XINFO_SQL: &str = "SELECT cid, name, \"desc\" FROM pragma_index_xinfo(?1, ?2) WHERE key = 1 ORDER BY seqno"; -pub(crate) const FOREIGN_KEY_LIST_SQL: &str = "SELECT id, \"table\", \"from\", \"to\", on_update, on_delete \ - FROM pragma_foreign_key_list(?1, ?2) ORDER BY id, seq"; +/// Not sorted: a sort holds every row before the first, and a column +/// can carry REFERENCES thousands of times, its long name on each row. +/// The reader orders a key's columns by `seq`. +pub(crate) const FOREIGN_KEY_LIST_SQL: &str = "SELECT id, seq, \"table\", \"from\", \"to\", on_update, on_delete \ + FROM pragma_foreign_key_list(?1, ?2)"; -pub(crate) const FOREIGN_KEY_CHECK_SQL: &str = - "SELECT \"table\", rowid, parent, fkid FROM pragma_foreign_key_check(?1, ?2)"; +// A key column as `FOREIGN_KEY_LIST_SQL` reads it: its `seq`, the child's column and the parent's, if named. +pub(crate) type ListedKeyColumn = (i64, String, Option); + +// Not the table and parent names, which repeat on every row that breaks a key. +pub(crate) const FOREIGN_KEY_CHECK_SQL: &str = "SELECT rowid, fkid FROM pragma_foreign_key_check(?1, ?2)"; /// `sqlite_schema` of one schema, which has to be named in the FROM. pub(crate) fn objects_sql(schema_sql: &str) -> String { @@ -82,15 +88,27 @@ pub(crate) fn dependents_sql(schema_sql: &str) -> String { ) } -/// `child_keys` counts every key of the child, wherever it points, since -/// the ids number them all. +/// Each column of another table's key that points at the table, not +/// sorted, as `FOREIGN_KEY_LIST_SQL`. The child's CREATE TABLE comes +/// from `inbound_children_sql`, once per child: it grows with the keys +/// it declares. pub(crate) fn inbound_keys_sql(schema_sql: &str) -> String { format!( - "SELECT m.name AS child, m.sql AS child_sql, \ - (SELECT count(DISTINCT k.id) FROM pragma_foreign_key_list(m.name, ?2) k) AS child_keys, \ - f.id AS id, f.\"from\" AS \"from\", f.\"to\" AS \"to\" \ + "SELECT m.name AS child, f.id AS id, f.seq AS seq, f.\"from\" AS \"from\", f.\"to\" AS \"to\" \ FROM {schema_sql}.sqlite_schema m, pragma_foreign_key_list(m.name, ?2) f \ - WHERE m.type = 'table' AND f.\"table\" = ?1 COLLATE NOCASE ORDER BY m.name, f.id, f.seq" + WHERE m.type = 'table' AND f.\"table\" = ?1 COLLATE NOCASE" + ) +} + +/// The tables with a key pointing at the table, each once. `child_keys` +/// counts every key of the child, wherever it points, since the ids +/// number them all. +pub(crate) fn inbound_children_sql(schema_sql: &str) -> String { + format!( + "SELECT m.name AS child, m.sql AS sql, \ + (SELECT count(DISTINCT k.id) FROM pragma_foreign_key_list(m.name, ?2) k) AS child_keys \ + FROM {schema_sql}.sqlite_schema m WHERE m.type = 'table' AND EXISTS \ + (SELECT 1 FROM pragma_foreign_key_list(m.name, ?2) f WHERE f.\"table\" = ?1 COLLATE NOCASE)" ) } diff --git a/core/crates/drivers/sqlite/src/catalog/table_definition.rs b/core/crates/drivers/sqlite/src/catalog/table_definition.rs index c21ce7d190..b4c5db6973 100644 --- a/core/crates/drivers/sqlite/src/catalog/table_definition.rs +++ b/core/crates/drivers/sqlite/src/catalog/table_definition.rs @@ -1,4 +1,4 @@ -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use sqlparser::ast::{ColumnOption, Statement}; use sqlparser::dialect::SQLiteDialect; @@ -6,72 +6,69 @@ use sqlparser::parser::Parser; use tablepro_core::column::SqlExpression; use tablepro_core::ddl::InboundForeignKey; use tablepro_core::meta::TableRef; +use tablepro_core::{CatalogBudget, DriverError}; -use crate::ddl::declared_foreign_key_names; - -/// One row of the inbound foreign key query: a key column of another -/// table that points at this one. +// A column of another table's key that points at this one. `to` is None for a key naming no columns, which means +// the parent's primary key. #[derive(Debug, Clone, PartialEq, Eq)] -pub(crate) struct InboundKeyRow { - pub child: String, - /// The child's CREATE TABLE, where the key's declared name is. - pub child_sql: Option, - /// How many foreign keys the child has, here or elsewhere. - pub child_keys: usize, +pub(crate) struct InboundKeyColumn { pub id: i64, + pub seq: i64, pub from: String, - /// `None` when the key names no columns, which means the parent's - /// primary key. pub to: Option, } -/// Group key columns into keys, in the order the query returned them. -/// A key that names no parent columns points at `primary_key`. +// One child's keys in pragma id order, each named by its id in `names`. A key naming no parent columns points at +// `primary_key`. pub(crate) fn inbound_foreign_keys( schema: Option<&str>, - rows: Vec, + child: &str, + mut columns: Vec, + names: &[String], primary_key: &[String], -) -> Vec { + budget: &mut CatalogBudget, +) -> Result, DriverError> { + columns.sort_by_key(|column| (column.id, column.seq)); let mut keys: Vec = Vec::new(); - let mut positions: HashMap<(String, i64), usize> = HashMap::new(); let mut unnamed_targets: Vec = Vec::new(); - let mut names: HashMap> = HashMap::new(); - for row in rows { - let child_names = names.entry(row.child.clone()).or_insert_with(|| { - let declared = row.child_sql.as_deref().and_then(declared_foreign_key_names); - foreign_key_names(&row.child, declared, row.child_keys) - }); - let name = usize::try_from(row.id) - .ok() - .and_then(|id| child_names.get(id)) - .cloned() - .unwrap_or_else(|| foreign_key_name(&row.child, row.id)); - let position = *positions.entry((row.child.clone(), row.id)).or_insert_with(|| { + let mut last_id = None; + for column in columns { + if last_id != Some(column.id) { + last_id = Some(column.id); + let name = usize::try_from(column.id) + .ok() + .and_then(|id| names.get(id)) + .cloned() + .unwrap_or_else(|| foreign_key_name(child, column.id)); + // Each key holds its own copy of the child's name, which can be long. keys.push(InboundForeignKey { child_compares_alike: true, - name, - table: TableRef::new(schema.map(str::to_owned), row.child.clone()), + name: budget.keep(name)?, + table: TableRef::new( + budget.keep_optional(schema.map(str::to_owned))?, + budget.keep(child.to_owned())?, + ), columns: Vec::new(), referenced_columns: Vec::new(), }); unnamed_targets.push(false); - keys.len() - 1 - }); - let (Some(key), Some(unnamed)) = (keys.get_mut(position), unnamed_targets.get_mut(position)) else { + } + let (Some(key), Some(unnamed)) = (keys.last_mut(), unnamed_targets.last_mut()) else { continue; }; - key.columns.push(row.from); - match row.to { + key.columns.push(column.from); + match column.to { Some(to) => key.referenced_columns.push(to), None => *unnamed = true, } } for (key, unnamed) in keys.iter_mut().zip(unnamed_targets) { if unnamed { + budget.charge_texts(primary_key)?; key.referenced_columns = primary_key.to_vec(); } } - keys + Ok(keys) } /// The name of a key declared without one, made from the child table @@ -80,24 +77,37 @@ pub(crate) fn foreign_key_name(table: &str, id: i64) -> String { format!("fk_{table}_{id}") } -/// A name for each of a table's `count` foreign keys, by pragma id. The -/// pragmas do not report the name a CONSTRAINT clause gave a key, so it -/// comes from the table's CREATE TABLE, `declared`. A key without one, -/// or with a name an earlier key took, gets a made-up one. -pub(crate) fn foreign_key_names(table: &str, declared: Option>>, count: usize) -> Vec { +// The pragmas do not report the name a CONSTRAINT clause gave a key, so it comes from the CREATE TABLE, `declared`; +// a key without one, or with a name an earlier key took, gets a made-up one. +pub(crate) fn foreign_key_names( + table: &str, + declared: Option>>, + count: usize, + budget: &mut CatalogBudget, +) -> Result, DriverError> { let declared = declared.filter(|declared| declared.len() == count).unwrap_or_default(); - let mut names: Vec = Vec::with_capacity(count); + let mut names: Vec = Vec::new(); + // Folded as SQLite folds names, so a table with thousands of keys is not checked key against key. Only a + // declared name can be taken already, so a table that declares none keeps no second copy. + let mut taken: HashSet = HashSet::new(); for id in 0..count { let made_up = || foreign_key_name(table, i64::try_from(id).unwrap_or(i64::MAX)); let name = declared .get(id) .cloned() .flatten() - .filter(|name| !names.iter().any(|taken| taken.eq_ignore_ascii_case(name))) + .filter(|name| !taken.contains(&name.to_ascii_lowercase())) .unwrap_or_else(made_up); + let name = budget.keep(name)?; + if !declared.is_empty() { + taken.insert(budget.keep(name.to_ascii_lowercase())?); + } names.push(name); } - names + for folded in &taken { + budget.release_text(folded); + } + Ok(names) } /// The expressions of the generated columns, read from the table's own @@ -135,56 +145,67 @@ pub(crate) fn generated_expressions(create_statement: &str) -> HashMap) -> InboundKeyRow { - InboundKeyRow { - child: child.to_owned(), - child_sql: None, - child_keys: 3, + fn column(id: i64, seq: i64, from: &str, to: Option<&str>) -> InboundKeyColumn { + InboundKeyColumn { id, + seq, from: from.to_owned(), to: to.map(str::to_owned), } } + fn keys_of( + child: &str, + columns: Vec, + child_sql: Option<&str>, + child_keys: usize, + ) -> Vec { + let mut budget = CatalogBudget::default(); + let declared = child_sql.and_then(crate::ddl::declared_foreign_key_names); + let names = foreign_key_names(child, declared, child_keys, &mut budget).unwrap(); + inbound_foreign_keys(None, child, columns, &names, &["id".to_owned()], &mut budget).unwrap() + } + #[test] fn a_key_without_target_columns_points_at_the_primary_key() { - let rows = vec![ - key_row("lines", 0, "order_id", None), - key_row("notes", 0, "order_a", Some("a")), - key_row("notes", 0, "order_b", Some("b")), - ]; - - let keys = inbound_foreign_keys(None, rows, &["id".to_owned()]); - - assert_eq!(keys.len(), 2); - assert_eq!(keys[0].name, "fk_lines_0"); - assert_eq!(keys[0].table, TableRef::unqualified("lines")); - assert_eq!(keys[0].referenced_columns, ["id"]); - assert_eq!(keys[1].columns, ["order_a", "order_b"]); - assert_eq!(keys[1].referenced_columns, ["a", "b"]); + let lines = keys_of("lines", vec![column(0, 0, "order_id", None)], None, 3); + let notes = keys_of( + "notes", + vec![column(0, 1, "order_b", Some("b")), column(0, 0, "order_a", Some("a"))], + None, + 3, + ); + + assert_eq!(lines.len(), 1); + assert_eq!(lines[0].name, "fk_lines_0"); + assert_eq!(lines[0].table, TableRef::unqualified("lines")); + assert_eq!(lines[0].referenced_columns, ["id"]); + assert_eq!(notes.len(), 1); + assert_eq!( + notes[0].columns, + ["order_a", "order_b"], + "in key order whatever order they came in" + ); + assert_eq!(notes[0].referenced_columns, ["a", "b"]); } #[test] fn a_declared_constraint_name_is_the_key_name() { let child_sql = "CREATE TABLE lines (a INTEGER CONSTRAINT to_orders REFERENCES orders, b INTEGER REFERENCES \ orders, CONSTRAINT to_orders FOREIGN KEY (a) REFERENCES orders)"; - let rows = [(0, "a"), (1, "b"), (2, "a")].map(|(id, from)| InboundKeyRow { - child_sql: Some(child_sql.to_owned()), - ..key_row("lines", id, from, None) - }); + let columns = || { + vec![ + column(2, 0, "a", None), + column(0, 0, "a", None), + column(1, 0, "b", None), + ] + }; - let keys = inbound_foreign_keys(None, rows.to_vec(), &["id".to_owned()]); + let keys = keys_of("lines", columns(), Some(child_sql), 3); let names: Vec<&str> = keys.iter().map(|key| key.name.as_str()).collect(); assert_eq!(names, ["to_orders", "fk_lines_1", "fk_lines_2"]); - let miscounted: Vec = rows - .iter() - .map(|row| InboundKeyRow { - child_keys: 4, - ..row.clone() - }) - .collect(); - let unsure: Vec = inbound_foreign_keys(None, miscounted, &["id".to_owned()]) + let unsure: Vec = keys_of("lines", columns(), Some(child_sql), 4) .into_iter() .map(|key| key.name) .collect(); @@ -194,10 +215,41 @@ mod tests { "names read from text that declares fewer keys than the child has" ); assert_eq!( - foreign_key_names("t", Some(vec![Some("x".to_owned())]), 2), - ["fk_t_0", "fk_t_1"], + foreign_key_names("t", Some(vec![Some("x".to_owned())]), 2, &mut CatalogBudget::default()), + Ok(vec!["fk_t_0".to_owned(), "fk_t_1".to_owned()]), "a list that does not match the catalogue is not used" ); + assert_eq!( + foreign_key_names( + "t", + Some(vec![Some("Same".to_owned()), Some("SAME".to_owned())]), + 2, + &mut CatalogBudget::default() + ), + Ok(vec!["Same".to_owned(), "fk_t_1".to_owned()]), + "a name an earlier key took, in other case, is not taken again" + ); + } + + #[test] + fn copies_of_a_long_primary_key_stop_at_the_budget() { + let primary_key: Vec = (0..10).map(|column| format!("{column}{}", "k".repeat(1_000))).collect(); + let columns: Vec = (0..1_000).map(|id| column(id, 0, "a", None)).collect(); + let mut budget = CatalogBudget::new(tablepro_core::ByteLimit::new(1_000_000).unwrap()); + + let names = foreign_key_names("child", None, 1_000, &mut budget).unwrap(); + let read = inbound_foreign_keys(None, "child", columns, &names, &primary_key, &mut budget); + + assert_eq!(read, Err(DriverError::CatalogTooLarge { limit_bytes: 1_000_000 })); + } + + #[test] + fn made_up_names_of_a_long_table_stop_at_the_budget() { + let mut budget = CatalogBudget::new(tablepro_core::ByteLimit::new(1_000_000).unwrap()); + + let names = foreign_key_names(&"t".repeat(10_000), None, 1_000, &mut budget); + + assert_eq!(names, Err(DriverError::CatalogTooLarge { limit_bytes: 1_000_000 })); } #[test] diff --git a/core/crates/drivers/sqlite/src/ddl/create_table_source.rs b/core/crates/drivers/sqlite/src/ddl/create_table_source.rs index fb7ab199d7..abbc545092 100644 --- a/core/crates/drivers/sqlite/src/ddl/create_table_source.rs +++ b/core/crates/drivers/sqlite/src/ddl/create_table_source.rs @@ -9,7 +9,7 @@ //! so what a rebuild does not change goes into the new table as it was //! written. -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use std::ops::Range; use tablepro_core::ddl::TableDefinition; @@ -316,13 +316,28 @@ pub(crate) fn declared_foreign_key_names(sql: &str) -> Option Some(names) } -/// How a CREATE TABLE has `column` compare its values. `None` when the -/// text does not read or names no such column. -pub(crate) fn declared_comparison(sql: &str, column: &str) -> Option { - let source = CreateTableSource::read(sql).ok()?; - let (_, declared) = source - .columns() - .find(|(_, declared)| declared.name.eq_ignore_ascii_case(column))?; +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct DeclaredComparisons(HashMap); + +impl DeclaredComparisons { + // Read once per table, not once per key column: a table can declare thousands of keys. + pub(crate) fn read(sql: &str) -> Option { + let source = CreateTableSource::read(sql).ok()?; + let mut columns = HashMap::new(); + for (_, declared) in source.columns() { + columns + .entry(declared.name.to_ascii_lowercase()) + .or_insert_with(|| comparison(&source, declared)); + } + Some(Self(columns)) + } + + pub(crate) fn of(&self, column: &str) -> Option<&crate::catalog::Comparison> { + self.0.get(&column.to_ascii_lowercase()) + } +} + +fn comparison(source: &CreateTableSource<'_>, declared: &SourceColumn) -> crate::catalog::Comparison { let type_text = kept_type_name(declared.type_range.clone().map_or("", |range| source.text(range))); let collation = declared .segments @@ -334,7 +349,7 @@ pub(crate) fn declared_comparison(sql: &str, column: &str) -> Option, + rows: Vec<(usize, Option)>, +} + +// A foreign key of one child table that some row breaks. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct BrokenKey { pub table: String, - /// `None` for a WITHOUT ROWID child table, which has no rowid to - /// name the row by, and for one whose rows the batch renumbers. - pub rowid: Option, pub parent: String, pub key: ViolatedKey, } +impl Violations { + pub(crate) fn add_key(&mut self, key: BrokenKey) -> usize { + self.keys.push(key); + self.keys.len() - 1 + } + + pub(crate) fn add_row(&mut self, key: usize, rowid: Option) { + self.rows.push((key, rowid)); + } + + pub(crate) fn extend(&mut self, other: Violations) { + let offset = self.keys.len(); + self.keys.extend(other.keys); + self.rows + .extend(other.rows.into_iter().map(|(key, rowid)| (key + offset, rowid))); + } +} + /// The foreign key a row breaks, by its columns rather than its number: /// a rebuild numbers the keys again, and can trade one key for another /// to the same parent. @@ -69,19 +93,22 @@ impl Renames { /// Name `baseline` rows by the names their tables and key columns /// have after the batch, so a renamed table or key compares with /// itself. - pub(crate) fn apply(&self, baseline: Vec) -> Vec { - baseline - .into_iter() - .map(|violation| Violation { - key: ViolatedKey { - columns: self.column_names(&violation.table, violation.key.columns), - parent_columns: self.column_names(&violation.parent, violation.key.parent_columns), - }, - table: self.table_name(violation.table), - rowid: violation.rowid, - parent: self.table_name(violation.parent), - }) - .collect() + pub(crate) fn apply(&self, baseline: Violations) -> Violations { + Violations { + keys: baseline + .keys + .into_iter() + .map(|broken| BrokenKey { + key: ViolatedKey { + columns: self.column_names(&broken.table, broken.key.columns), + parent_columns: self.column_names(&broken.parent, broken.key.parent_columns), + }, + table: self.table_name(broken.table), + parent: self.table_name(broken.parent), + }) + .collect(), + rows: baseline.rows, + } } fn table_name(&self, name: String) -> String { @@ -102,48 +129,59 @@ impl Renames { /// The rows of `after` that `before` does not account for, compared as /// multisets. Rows without a rowid are all alike, so for them it is the /// count per child, parent and key that compares. -pub(crate) fn introduced(before: &[Violation], after: &[Violation]) -> Vec { - let mut remaining: HashMap = HashMap::new(); - for violation in before { - *remaining.entry(key(violation)).or_default() += 1; +pub(crate) fn introduced(before: &Violations, after: &Violations) -> Vec { + let mut ids: HashMap = HashMap::new(); + let mut id_of = |broken: &BrokenKey| { + let next = ids.len(); + *ids.entry(key(broken)).or_insert(next) + }; + let before_ids: Vec = before.keys.iter().map(&mut id_of).collect(); + let after_ids: Vec = after.keys.iter().map(&mut id_of).collect(); + let mut remaining: HashMap<(usize, Option), usize> = HashMap::new(); + for (key, rowid) in &before.rows { + if let Some(id) = before_ids.get(*key) { + *remaining.entry((*id, *rowid)).or_default() += 1; + } } let mut introduced: Vec = Vec::new(); - for violation in after { - if let Some(left) = remaining.get_mut(&key(violation)) + let mut groups: HashMap<(String, String), usize> = HashMap::new(); + for (key, rowid) in &after.rows { + let (Some(id), Some(broken)) = (after_ids.get(*key), after.keys.get(*key)) else { + continue; + }; + if let Some(left) = remaining.get_mut(&(*id, *rowid)) && *left > 0 { *left -= 1; continue; } - let table = violation.table.to_lowercase(); - let parent = violation.parent.to_lowercase(); - match introduced - .iter_mut() - .find(|group| group.table.to_lowercase() == table && group.parent.to_lowercase() == parent) - { + let group = (broken.table.to_lowercase(), broken.parent.to_lowercase()); + match groups.get(&group).and_then(|index| introduced.get_mut(*index)) { Some(group) => group.count += 1, - None => introduced.push(IntroducedViolations { - table: violation.table.clone(), - parent: violation.parent.clone(), - count: 1, - }), + None => { + groups.insert(group, introduced.len()); + introduced.push(IntroducedViolations { + table: broken.table.clone(), + parent: broken.parent.clone(), + count: 1, + }); + } } } introduced } -type ViolationKey = (String, Option, String, Vec, Vec); +type ViolationKey = (String, String, Vec, Vec); /// SQLite names tables and columns case-insensitively, and so does the /// comparison. -fn key(violation: &Violation) -> ViolationKey { +fn key(broken: &BrokenKey) -> ViolationKey { let folded = |names: &[String]| names.iter().map(|name| name.to_lowercase()).collect(); ( - violation.table.to_lowercase(), - violation.rowid, - violation.parent.to_lowercase(), - folded(&violation.key.columns), - folded(&violation.key.parent_columns), + broken.table.to_lowercase(), + broken.parent.to_lowercase(), + folded(&broken.key.columns), + folded(&broken.key.parent_columns), ) } @@ -172,21 +210,38 @@ pub(crate) fn violation_error(introduced: &[IntroducedViolations]) -> DriverErro mod tests { use super::*; - fn violation(table: &str, rowid: Option, parent: &str) -> Violation { - keyed(table, rowid, parent, &["parent"], &["id"]) + type Row<'a> = (&'a str, Option, &'a str, &'a [&'a str], &'a [&'a str]); + + fn violation<'a>(table: &'a str, rowid: Option, parent: &'a str) -> Row<'a> { + (table, rowid, parent, &["parent"], &["id"]) + } + + fn keyed<'a>( + table: &'a str, + rowid: Option, + parent: &'a str, + columns: &'a [&'a str], + parent_columns: &'a [&'a str], + ) -> Row<'a> { + (table, rowid, parent, columns, parent_columns) } - fn keyed(table: &str, rowid: Option, parent: &str, columns: &[&str], parent_columns: &[&str]) -> Violation { + // One key per row, as two keys of the same columns compare alike. + fn found(rows: &[Row<'_>]) -> Violations { let owned = |names: &[&str]| names.iter().map(|name| (*name).to_owned()).collect(); - Violation { - table: table.to_owned(), - rowid, - parent: parent.to_owned(), - key: ViolatedKey { - columns: owned(columns), - parent_columns: owned(parent_columns), - }, + let mut violations = Violations::default(); + for (table, rowid, parent, columns, parent_columns) in rows { + let key = violations.add_key(BrokenKey { + table: (*table).to_owned(), + parent: (*parent).to_owned(), + key: ViolatedKey { + columns: owned(columns), + parent_columns: owned(parent_columns), + }, + }); + violations.add_row(key, *rowid); } + violations } fn renames(before: &str, after: &str, columns: &[(&str, &str)]) -> Renames { @@ -215,7 +270,7 @@ mod tests { violation("lines", Some(2), "orders"), ]; - let introduced = introduced(&renames.apply(before), &after); + let introduced = introduced(&renames.apply(found(&before)), &found(&after)); assert_eq!( introduced, @@ -233,7 +288,7 @@ mod tests { let after = vec![keyed("children", Some(1), "parents", &["b"], &["id"])]; assert_eq!( - introduced(&before, &after), + introduced(&found(&before), &found(&after)), [IntroducedViolations { table: "children".to_owned(), parent: "parents".to_owned(), @@ -254,7 +309,7 @@ mod tests { keyed("grandchildren", Some(4), "children", &["child"], &["parent_id"]), ]; - assert!(introduced(&renames.apply(before), &after).is_empty()); + assert!(introduced(&renames.apply(found(&before)), &found(&after)).is_empty()); } #[test] @@ -267,9 +322,47 @@ mod tests { violation("tags", None, "posts"), ]; - assert!(introduced(&before, &same).is_empty()); - assert_eq!(introduced(&before, &more).len(), 1); - assert_eq!(introduced(&before, &more)[0].count, 1); + assert!(introduced(&found(&before), &found(&same)).is_empty()); + assert_eq!(introduced(&found(&before), &found(&more)).len(), 1); + assert_eq!(introduced(&found(&before), &found(&more))[0].count, 1); + } + + #[test] + fn rows_of_one_key_compare_one_for_one_and_name_their_group_in_order() { + let mut before = Violations::default(); + let broken = |table: &str, parent: &str| BrokenKey { + table: table.to_owned(), + parent: parent.to_owned(), + key: ViolatedKey { + columns: vec!["a".to_owned()], + parent_columns: Vec::new(), + }, + }; + let key = before.add_key(broken("t", "p")); + for rowid in 0..1_000 { + before.add_row(key, Some(rowid)); + } + let mut after = before.clone(); + let other = after.add_key(broken("t", "q")); + after.add_row(other, Some(5)); + after.add_row(key, Some(1_000)); + after.add_row(other, Some(6)); + + assert_eq!( + introduced(&before, &after), + [ + IntroducedViolations { + table: "t".to_owned(), + parent: "q".to_owned(), + count: 2 + }, + IntroducedViolations { + table: "t".to_owned(), + parent: "p".to_owned(), + count: 1 + }, + ] + ); } #[test] diff --git a/core/crates/drivers/sqlite/src/metadata.rs b/core/crates/drivers/sqlite/src/metadata.rs index 8745ac145b..43200758a0 100644 --- a/core/crates/drivers/sqlite/src/metadata.rs +++ b/core/crates/drivers/sqlite/src/metadata.rs @@ -3,8 +3,9 @@ //! `catalog` owns the SQL and what the rows mean. This runs it. use std::collections::btree_map::Entry; -use std::collections::{BTreeMap, HashMap}; +use std::collections::{BTreeMap, HashMap, hash_map}; +use futures::TryStreamExt; use sqlx::sqlite::{SqliteConnection as SqlxConnection, SqliteRow}; use sqlx::{AssertSqlSafe, Row}; use tablepro_core::column::ColumnInfo; @@ -17,13 +18,13 @@ use tablepro_core::meta::{ RowIdentity, SchemaInfo, TableEstimate, TableRef, }; use tablepro_core::sql_syntax::{SqlGrammar, qualify_created_name, terminated}; -use tablepro_core::{DriverError, ForeignKeyInfo, ServerCode, ServerDiagnostics, UnsupportedFeature}; +use tablepro_core::{CatalogBudget, DriverError, ForeignKeyInfo, ServerCode, ServerDiagnostics, UnsupportedFeature}; use crate::catalog::{ - self, InboundKeyRow, MAIN_SCHEMA, SchemaRow, SqliteColumnRow, TableFacts, foreign_key_names, generated_expressions, - inbound_foreign_keys, select_dependents, + self, InboundKeyColumn, ListedKeyColumn, MAIN_SCHEMA, SchemaRow, SqliteColumnRow, TableFacts, foreign_key_names, + generated_expressions, inbound_foreign_keys, select_dependents, }; -use crate::ddl::{declared_comparison, declared_descending_key, declared_foreign_key_names}; +use crate::ddl::{DeclaredComparisons, declared_descending_key, declared_foreign_key_names}; use crate::errors::{SQLITE_ERROR, map_error}; use crate::index_sql::read_index_sql; @@ -69,18 +70,17 @@ pub(crate) async fn list_objects( ObjectScope::Schema(schema) => vec![schema.clone()], ObjectScope::AllSchemas => schema_names(connection).await?, }; + let mut budget = CatalogBudget::default(); let mut objects = Vec::new(); for schema in schemas { - let rows = sqlx::query(AssertSqlSafe(catalog::objects_sql("e(&schema)))) - .fetch_all(&mut *connection) - .await - .map_err(map_error)?; - for row in rows { - let name = text(&row, "name")?; + let mut rows = sqlx::query(AssertSqlSafe(catalog::objects_sql("e(&schema)))).fetch(&mut *connection); + while let Some(row) = rows.try_next().await.map_err(map_error)? { + let name = budget.keep(text(&row, "name")?)?; + let schema = budget.keep_optional(qualifier(&schema))?; objects.push(match text(&row, "type")?.as_str() { - "trigger" => ObjectInfo::trigger(qualifier(&schema), name, text(&row, "tbl_name")?), - "view" => ObjectInfo::new(qualifier(&schema), name, ObjectKind::View), - _ => ObjectInfo::new(qualifier(&schema), name, ObjectKind::Table), + "trigger" => ObjectInfo::trigger(schema, name, budget.keep(text(&row, "tbl_name")?)?), + "view" => ObjectInfo::new(schema, name, ObjectKind::View), + _ => ObjectInfo::new(schema, name, ObjectKind::Table), }); } } @@ -121,7 +121,7 @@ pub(crate) async fn fetch_columns( connection: &mut SqlxConnection, table: &TableRef, ) -> Result, DriverError> { - let shape = table_shape(connection, table).await?; + let shape = table_shape(connection, table, &mut CatalogBudget::default()).await?; Ok(shape .visible_columns() .map(|column| column.column_info(&shape.facts)) @@ -132,54 +132,79 @@ pub(crate) async fn fetch_indexes( connection: &mut SqlxConnection, table: &TableRef, ) -> Result, DriverError> { - let shape = table_shape(connection, table).await?; - indexes(connection, table, &shape).await + let mut budget = CatalogBudget::default(); + let shape = table_shape(connection, table, &mut budget).await?; + indexes(connection, table, &shape, &mut budget).await } pub(crate) async fn fetch_foreign_keys( connection: &mut SqlxConnection, table: &TableRef, ) -> Result, DriverError> { - let keys = foreign_key_rows(connection, table).await?; + let mut budget = CatalogBudget::default(); + let keys = foreign_key_rows(connection, table, &mut budget).await?; let create = match keys.is_empty() { true => None, - false => create_statement(connection, table).await?, + false => create_statement(connection, table, &mut budget).await?, }; - resolve_foreign_keys(connection, table, keys, create.as_deref()).await + resolve_foreign_keys(connection, table, keys, create.as_deref(), &mut budget).await } pub(crate) async fn fetch_outbound_foreign_keys( connection: &mut SqlxConnection, table: &TableRef, ) -> Result, DriverError> { - let keys = fetch_foreign_keys(connection, table).await?; - let child_sql = match keys.is_empty() { - true => None, - false => create_statement(connection, table).await?, - }; - let mut parents: HashMap> = HashMap::new(); - let mut outbound = Vec::with_capacity(keys.len()); - for key in keys { - let parent_sql = match parents.get(&key.ref_table) { - Some(sql) => sql.clone(), - None => { - let parent = TableRef::new(table.schema.clone(), key.ref_table.clone()); - let sql = create_statement(connection, &parent).await?; - parents.insert(key.ref_table.clone(), sql.clone()); - sql + let mut budget = CatalogBudget::default(); + let keys = foreign_key_rows(connection, table, &mut budget).await?; + if keys.is_empty() { + return Ok(Vec::new()); + } + let create = create_statement(connection, table, &mut budget).await?; + let keys = resolve_foreign_keys(connection, table, keys, create.as_deref(), &mut budget).await?; + let child = create.as_deref().map(DeclaredComparisons::read); + release(&mut budget, create); + let mut parents: HashMap> = HashMap::new(); + for (index, key) in keys.iter().enumerate() { + match parents.entry(key.ref_table.to_ascii_lowercase()) { + hash_map::Entry::Occupied(entry) => entry.into_mut().push(index), + hash_map::Entry::Vacant(entry) => { + budget.charge_text(entry.key())?; + entry.insert(vec![index]); } + } + } + let mut alike = vec![true; keys.len()]; + // Each parent's CREATE TABLE is read and parsed once, however many keys point at it. + for indices in parents.values() { + let Some(first) = indices.first().and_then(|index| keys.get(*index)) else { + continue; }; - let parent_compares_alike = compares_alike( - child_sql.as_deref(), - parent_sql.as_deref(), - key.columns.iter().zip(&key.ref_columns), - ); - outbound.push(OutboundForeignKey { + let parent_sql = create_statement( + connection, + &TableRef::new(table.schema.clone(), first.ref_table.clone()), + &mut budget, + ) + .await?; + let parent = parent_sql.as_deref().map(DeclaredComparisons::read); + release(&mut budget, parent_sql); + for index in indices { + if let (Some(key), Some(alike)) = (keys.get(*index), alike.get_mut(*index)) { + *alike = compares_alike( + child.as_ref(), + parent.as_ref(), + key.columns.iter().zip(&key.ref_columns), + ); + } + } + } + Ok(keys + .into_iter() + .zip(alike) + .map(|(key, parent_compares_alike)| OutboundForeignKey { key, parent_compares_alike, - }); - } - Ok(outbound) + }) + .collect()) } /// The table's foreign keys by pragma id, each with whether it points @@ -187,40 +212,53 @@ pub(crate) async fn fetch_outbound_foreign_keys( async fn foreign_key_rows( connection: &mut SqlxConnection, table: &TableRef, + budget: &mut CatalogBudget, ) -> Result, DriverError> { let schema = schema_of(table); - let rows = sqlx::query(catalog::FOREIGN_KEY_LIST_SQL) + let mut rows = sqlx::query(catalog::FOREIGN_KEY_LIST_SQL) .bind(table.name.as_str()) .bind(schema) - .fetch_all(&mut *connection) - .await - .map_err(map_error)?; - let mut keys: BTreeMap = BTreeMap::new(); - for row in rows { + .fetch(&mut *connection); + let mut keys: BTreeMap)> = BTreeMap::new(); + while let Some(row) = rows.try_next().await.map_err(map_error)? { let id: i64 = row.try_get("id").map_err(map_error)?; - let to: Option = row.try_get("to").map_err(map_error)?; - let (key, to_primary) = match keys.entry(id) { + let (_, columns) = match keys.entry(id) { Entry::Occupied(entry) => entry.into_mut(), Entry::Vacant(entry) => entry.insert(( ForeignKeyInfo { name: String::new(), columns: Vec::new(), - ref_schema: table.schema.clone(), - ref_table: text(&row, "table")?, + ref_schema: budget.keep_optional(table.schema.clone())?, + ref_table: budget.keep(text(&row, "table")?)?, ref_columns: Vec::new(), on_delete: referential_action(&text(&row, "on_delete")?), on_update: referential_action(&text(&row, "on_update")?), }, - false, + Vec::new(), )), }; - key.columns.push(text(&row, "from")?); - match to { - Some(to) => key.ref_columns.push(to), - None => *to_primary = true, - } + // A column can carry REFERENCES many times over, so its declared name comes back once per key. + columns.push(( + row.try_get("seq").map_err(map_error)?, + budget.keep(text(&row, "from")?)?, + budget.keep_optional(row.try_get("to").map_err(map_error)?)?, + )); } - Ok(keys) + Ok(keys + .into_iter() + .map(|(id, (mut key, mut columns))| { + columns.sort_by_key(|(seq, _, _)| *seq); + let mut to_primary = false; + for (_, from, to) in columns { + key.columns.push(from); + match to { + Some(to) => key.ref_columns.push(to), + None => to_primary = true, + } + } + (id, (key, to_primary)) + }) + .collect()) } /// Name the keys from the table's CREATE TABLE, and spell each parent @@ -236,35 +274,58 @@ async fn resolve_foreign_keys( table: &TableRef, keys: BTreeMap, create_statement: Option<&str>, + budget: &mut CatalogBudget, ) -> Result, DriverError> { let declared = create_statement.and_then(declared_foreign_key_names); - let mut names = foreign_key_names(&table.name, declared, keys.len()).into_iter(); - let mut parents: Vec<(String, Option)> = Vec::new(); - let mut resolved = Vec::with_capacity(keys.len()); - for (mut key, to_primary) in keys.into_values() { - key.name = names.next().unwrap_or_default(); - let known = parents - .iter() - .position(|(written, _)| written.eq_ignore_ascii_case(&key.ref_table)); - let index = match known { - Some(index) => index, - None => { - let parent = TableRef::new(table.schema.clone(), key.ref_table.clone()); - let found = declared_table(connection, &parent).await?; - parents.push((key.ref_table.clone(), found)); - parents.len() - 1 + let names = foreign_key_names(&table.name, declared, keys.len(), budget)?; + let mut resolved: Vec<(ForeignKeyInfo, bool)> = keys + .into_values() + .zip(names) + .map(|((mut key, to_primary), name)| { + key.name = name; + (key, to_primary) + }) + .collect(); + // SQLite folds ASCII letters alone when it compares names. + let mut parents: Vec<(String, Vec)> = Vec::new(); + let mut by_name: HashMap = HashMap::new(); + for (index, (key, _)) in resolved.iter().enumerate() { + match by_name.entry(key.ref_table.to_ascii_lowercase()) { + hash_map::Entry::Occupied(entry) => { + if let Some((_, indices)) = parents.get_mut(*entry.get()) { + indices.push(index); + } } + hash_map::Entry::Vacant(entry) => { + budget.charge_text(entry.key())?; + entry.insert(parents.len()); + parents.push((key.ref_table.clone(), vec![index])); + } + } + } + for (written, indices) in parents { + let parent = TableRef::new(table.schema.clone(), written); + let Some(found) = declared_table(connection, &parent, budget).await? else { + continue; }; - if let Some((_, Some(parent))) = parents.get(index) { - key.ref_table = parent.name.clone(); + let primary_key = found.primary_key(); + for index in indices { + let Some((key, to_primary)) = resolved.get_mut(index) else { + continue; + }; + key.ref_table = budget.keep(found.name.clone())?; key.ref_columns = match to_primary { - true => parent.primary_key(), - false => key.ref_columns.iter().map(|column| parent.spelling(column)).collect(), + // Each key naming no columns holds its own copy of the parent's whole primary key. + true => { + budget.charge_texts(&primary_key)?; + primary_key.clone() + } + false => key.ref_columns.iter().map(|column| found.spelling(column)).collect(), }; } - resolved.push(key); + release_columns(budget, &found.columns); } - Ok(resolved) + Ok(resolved.into_iter().map(|(key, _)| key).collect()) } /// A table's declared name and columns. @@ -298,6 +359,7 @@ fn declared_spelling<'a>(written: &str, mut declared: impl Iterator Result, DriverError> { let schema = schema_of(table); let name: Option = sqlx::query_scalar(catalog::DECLARED_TABLE_NAME_SQL) @@ -309,95 +371,197 @@ async fn declared_table( let Some(name) = name else { return Ok(None); }; - let columns = sqlx::query(catalog::TABLE_XINFO_SQL) - .bind(name.as_str()) - .bind(schema) - .fetch_all(&mut *connection) - .await - .map_err(map_error)? - .iter() - .map(SqliteColumnRow::from_row) - .collect::, _>>()?; + let columns = column_rows(connection, &name, schema, budget).await?; Ok(Some(DeclaredTable { name, columns })) } +async fn column_rows( + connection: &mut SqlxConnection, + table: &str, + schema: &str, + budget: &mut CatalogBudget, +) -> Result, DriverError> { + let mut rows = sqlx::query(catalog::TABLE_XINFO_SQL) + .bind(table) + .bind(schema) + .fetch(&mut *connection); + let mut columns = Vec::new(); + while let Some(row) = rows.try_next().await.map_err(map_error)? { + let column = SqliteColumnRow::from_row(&row)?; + budget.charge_text(&column.name)?; + budget.charge_text(&column.declared_type)?; + budget.charge_texts(&column.default)?; + columns.push(column); + } + Ok(columns) +} + +fn release_columns(budget: &mut CatalogBudget, columns: &[SqliteColumnRow]) { + for column in columns { + budget.release_text(&column.name); + budget.release_text(&column.declared_type); + if let Some(default) = &column.default { + budget.release_text(default); + } + } +} + pub(crate) async fn fetch_inbound_foreign_keys( connection: &mut SqlxConnection, table: &TableRef, ) -> Result, DriverError> { - let shape = table_shape(connection, table).await?; - inbound_keys(connection, table, &shape).await + let mut budget = CatalogBudget::default(); + let shape = table_shape(connection, table, &mut budget).await?; + inbound_keys(connection, table, &shape, &mut budget).await } /// The keys of the schema's other tables that point at `table`, its /// columns spelled as `shape` declares them, since a child's REFERENCES -/// clause can spell them otherwise. +/// clause can spell them otherwise. Ordered by child, then key. async fn inbound_keys( connection: &mut SqlxConnection, table: &TableRef, shape: &TableShape, + budget: &mut CatalogBudget, ) -> Result, DriverError> { + let mut children = inbound_columns(connection, table, budget).await?; + if children.is_empty() { + return Ok(Vec::new()); + } + let parent_sql = create_statement(connection, table, budget).await?; + let parent = Parent { + table, + shape, + declared: parent_sql.as_deref().map(DeclaredComparisons::read), + primary_key: shape.primary_key(), + }; + release(budget, parent_sql); + let mut keys = Vec::new(); let schema_sql = quote(schema_of(table)); - let rows = sqlx::query(AssertSqlSafe(catalog::inbound_keys_sql(&schema_sql))) + let mut rows = sqlx::query(AssertSqlSafe(catalog::inbound_children_sql(&schema_sql))) .bind(table.name.as_str()) .bind(schema_of(table)) - .fetch_all(&mut *connection) - .await - .map_err(map_error)? - .iter() - .map(|row| { - Ok(InboundKeyRow { - child: text(row, "child")?, - child_sql: row.try_get("child_sql").map_err(map_error)?, - child_keys: usize::try_from(row.try_get::("child_keys").map_err(map_error)?).unwrap_or(0), - id: row.try_get("id").map_err(map_error)?, - from: text(row, "from")?, - to: row.try_get("to").map_err(map_error)?, - }) - }) - .collect::, DriverError>>()?; - let child_sql: HashMap> = rows - .iter() - .map(|row| (row.child.clone(), row.child_sql.clone())) - .collect(); - let parent_sql = match rows.is_empty() { - true => None, - false => create_statement(connection, table).await?, - }; - let mut keys = inbound_foreign_keys(table.schema.as_deref(), rows, &shape.primary_key()); - for key in &mut keys { - for column in &mut key.referenced_columns { - *column = declared_spelling(column, shape.columns.iter().map(|declared| declared.name.as_str())); - } - let child = child_sql.get(&key.table.name).and_then(Option::as_deref); - key.child_compares_alike = compares_alike( - child, - parent_sql.as_deref(), - key.columns.iter().zip(&key.referenced_columns), - ); + .fetch(&mut *connection); + while let Some(row) = rows.try_next().await.map_err(map_error)? { + let name: &str = row.try_get("child").map_err(map_error)?; + let Some((child, columns)) = children.remove_entry(name) else { + continue; + }; + let sql = budget.keep_optional(row.try_get("sql").map_err(map_error)?)?; + let count = usize::try_from(row.try_get::("child_keys").map_err(map_error)?).unwrap_or(0); + keys.extend(parent.keys_of(child, columns, sql, count, budget)?); + } + // A child the second read no longer saw keeps made-up names. + for (child, columns) in children { + keys.extend(parent.keys_of(child, columns, None, 0, budget)?); } + keys.sort_by(|left, right| left.table.name.cmp(&right.table.name)); Ok(keys) } +/// Each child's key columns, its name read once rather than kept on +/// every row. +async fn inbound_columns( + connection: &mut SqlxConnection, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result>, DriverError> { + let schema_sql = quote(schema_of(table)); + let mut rows = sqlx::query(AssertSqlSafe(catalog::inbound_keys_sql(&schema_sql))) + .bind(table.name.as_str()) + .bind(schema_of(table)) + .fetch(&mut *connection); + let mut children: HashMap> = HashMap::new(); + while let Some(row) = rows.try_next().await.map_err(map_error)? { + let column = InboundKeyColumn { + id: row.try_get("id").map_err(map_error)?, + seq: row.try_get("seq").map_err(map_error)?, + from: budget.keep(text(&row, "from")?)?, + to: budget.keep_optional(row.try_get("to").map_err(map_error)?)?, + }; + let child: &str = row.try_get("child").map_err(map_error)?; + match children.get_mut(child) { + Some(columns) => columns.push(column), + None => { + children.insert(budget.keep(child.to_owned())?, vec![column]); + } + } + } + Ok(children) +} + +/// The table the inbound keys point at, read once for all its children. +struct Parent<'a> { + table: &'a TableRef, + shape: &'a TableShape, + declared: Option>, + primary_key: Vec, +} + +impl Parent<'_> { + /// One child's keys, from its CREATE TABLE, which is let go before + /// the next child's is read. + fn keys_of( + &self, + child: String, + columns: Vec, + sql: Option, + count: usize, + budget: &mut CatalogBudget, + ) -> Result, DriverError> { + let names = foreign_key_names( + &child, + sql.as_deref().and_then(declared_foreign_key_names), + count, + budget, + )?; + let declared = sql.as_deref().map(DeclaredComparisons::read); + release(budget, sql); + let mut keys = inbound_foreign_keys( + self.table.schema.as_deref(), + &child, + columns, + &names, + &self.primary_key, + budget, + )?; + for name in &names { + budget.release_text(name); + } + budget.release_text(&child); + for key in &mut keys { + for column in &mut key.referenced_columns { + *column = declared_spelling(column, self.shape.columns.iter().map(|declared| declared.name.as_str())); + } + key.child_compares_alike = compares_alike( + declared.as_ref(), + self.declared.as_ref(), + key.columns.iter().zip(&key.referenced_columns), + ); + } + Ok(keys) + } +} + /// SQLite compares a key under the parent columns' affinity and /// collation, so `=` on a column declared otherwise finds other rows /// than the key matches. A definition that does not read says nothing /// either way, so the key is not followed. A missing table is followed, /// to say that it is missing. fn compares_alike<'a>( - child_sql: Option<&str>, - parent_sql: Option<&str>, + child: Option<&Option>, + parent: Option<&Option>, pairs: impl IntoIterator, ) -> bool { - let (Some(child_sql), Some(parent_sql)) = (child_sql, parent_sql) else { + let (Some(child), Some(parent)) = (child, parent) else { return true; }; pairs.into_iter().all(|(child_column, parent_column)| { match ( - declared_comparison(child_sql, child_column), - declared_comparison(parent_sql, parent_column), + child.as_ref().and_then(|declared| declared.of(child_column)), + parent.as_ref().and_then(|declared| declared.of(parent_column)), ) { - (Some(child), Some(parent)) => child.alike(&parent), + (Some(child), Some(parent)) => child.alike(parent), _ => false, } }) @@ -409,7 +573,8 @@ pub(crate) async fn row_identity( connection: &mut SqlxConnection, table: &TableRef, ) -> Result { - let shape = table_shape(connection, table).await?; + let mut budget = CatalogBudget::default(); + let shape = table_shape(connection, table, &mut budget).await?; // A view has no rowid and no key, and a virtual table's rowid is // whatever its module makes of it. if shape.kind != "table" { @@ -425,13 +590,16 @@ pub(crate) async fn row_identity( .filter(|column| column.not_null) .map(|column| column.name.as_str()) .collect(); - let unique = indexes(connection, table, &shape).await?.into_iter().find(|index| { - index.can_identify_rows() - && index - .key_columns() - .iter() - .all(|column| not_null.iter().any(|name| name.eq_ignore_ascii_case(column))) - }); + let unique = indexes(connection, table, &shape, &mut budget) + .await? + .into_iter() + .find(|index| { + index.can_identify_rows() + && index + .key_columns() + .iter() + .all(|column| not_null.iter().any(|name| name.eq_ignore_ascii_case(column))) + }); if let Some(index) = unique { return Ok(RowIdentity::UniqueKey { columns: index.key_columns(), @@ -484,9 +652,10 @@ pub(crate) async fn fetch_table_definition( connection: &mut SqlxConnection, table: &TableRef, ) -> Result { - let shape = table_shape(connection, table).await?; + let mut budget = CatalogBudget::default(); + let shape = table_shape(connection, table, &mut budget).await?; let schema_sql = quote(schema_of(table)); - let create_statement = create_statement(connection, table).await?; + let create_statement = create_statement(connection, table, &mut budget).await?; let expressions = match ( &create_statement, @@ -511,16 +680,23 @@ pub(crate) async fn fetch_table_definition( name: None, columns: key_columns, }); - let indexes = indexes(connection, table, &shape).await?; - let foreign_keys = foreign_key_rows(connection, table).await?; - let foreign_keys = resolve_foreign_keys(connection, table, foreign_keys, create_statement.as_deref()).await?; + let indexes = indexes(connection, table, &shape, &mut budget).await?; + let foreign_keys = foreign_key_rows(connection, table, &mut budget).await?; + let foreign_keys = resolve_foreign_keys( + connection, + table, + foreign_keys, + create_statement.as_deref(), + &mut budget, + ) + .await?; - let inbound = inbound_keys(connection, table, &shape).await?; + let inbound = inbound_keys(connection, table, &shape, &mut budget).await?; let source = match create_statement { Some(create_statement) => Some(TableSource { create_statement, - dependents: dependents(connection, &schema_sql, &table.name).await?, + dependents: dependents(connection, &schema_sql, &table.name, &mut budget).await?, }), None => None, }; @@ -538,9 +714,13 @@ pub(crate) async fn fetch_table_definition( } /// The CREATE statement SQLite stored for the table. -async fn create_statement(connection: &mut SqlxConnection, table: &TableRef) -> Result, DriverError> { +async fn create_statement( + connection: &mut SqlxConnection, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result, DriverError> { let schema_sql = quote(schema_of(table)); - Ok(sqlx::query(AssertSqlSafe(catalog::object_sql(&schema_sql))) + let sql = sqlx::query(AssertSqlSafe(catalog::object_sql(&schema_sql))) .bind(table.name.as_str()) .fetch_optional(&mut *connection) .await @@ -548,20 +728,25 @@ async fn create_statement(connection: &mut SqlxConnection, table: &TableRef) -> .map(|row| row.try_get::, _>("sql")) .transpose() .map_err(map_error)? - .flatten()) + .flatten(); + budget.keep_optional(sql) +} + +fn release(budget: &mut CatalogBudget, text: Option) { + if let Some(text) = text { + budget.release_text(&text); + } } async fn dependents( connection: &mut SqlxConnection, schema_sql: &str, table: &str, + budget: &mut CatalogBudget, ) -> Result, DriverError> { - let rows = sqlx::query(AssertSqlSafe(catalog::dependents_sql(schema_sql))) - .fetch_all(&mut *connection) - .await - .map_err(map_error)?; - let mut schema_rows = Vec::with_capacity(rows.len()); - for row in rows { + let mut rows = sqlx::query(AssertSqlSafe(catalog::dependents_sql(schema_sql))).fetch(&mut *connection); + let mut schema_rows = Vec::new(); + while let Some(row) = rows.try_next().await.map_err(map_error)? { let kind = match text(&row, "type")?.as_str() { "index" => SchemaObjectKind::Index, "trigger" => SchemaObjectKind::Trigger, @@ -569,9 +754,9 @@ async fn dependents( }; schema_rows.push(SchemaRow { kind, - name: text(&row, "name")?, - table: text(&row, "tbl_name")?, - sql: text(&row, "sql")?, + name: budget.keep(text(&row, "name")?)?, + table: budget.keep(text(&row, "tbl_name")?)?, + sql: budget.keep(text(&row, "sql")?)?, }); } Ok(select_dependents(table, schema_rows)) @@ -581,69 +766,27 @@ async fn indexes( connection: &mut SqlxConnection, table: &TableRef, shape: &TableShape, + budget: &mut CatalogBudget, ) -> Result, DriverError> { let schema = schema_of(table); - let schema_sql = quote(schema); - let list = sqlx::query(catalog::INDEX_LIST_SQL) - .bind(table.name.as_str()) - .bind(schema) - .fetch_all(&mut *connection) - .await - .map_err(map_error)?; - let mut indexes = Vec::with_capacity(list.len() + 1); - for entry in list { - let name = text(&entry, "name")?; - let unique: i64 = entry.try_get("unique").map_err(map_error)?; - let partial: i64 = entry.try_get("partial").map_err(map_error)?; - let primary = text(&entry, "origin")? == "pk"; - let stored_sql: Option = sqlx::query_scalar(AssertSqlSafe(catalog::index_sql(&schema_sql))) - .bind(name.as_str()) - .fetch_optional(&mut *connection) - .await - .map_err(map_error)? - .flatten(); - let parsed = stored_sql.as_deref().and_then(read_index_sql); - let key_rows = sqlx::query(catalog::INDEX_XINFO_SQL) - .bind(name.as_str()) + let mut listed = Vec::new(); + { + let mut rows = sqlx::query(catalog::INDEX_LIST_SQL) + .bind(table.name.as_str()) .bind(schema) - .fetch_all(&mut *connection) - .await - .map_err(map_error)?; - let mut keys = Vec::with_capacity(key_rows.len()); - for (position, key) in key_rows.iter().enumerate() { - let cid: i64 = key.try_get("cid").map_err(map_error)?; - let descending = key.try_get::("desc").map_err(map_error)? != 0; - let column: Option = key.try_get("name").map_err(map_error)?; - keys.push(match (cid, column) { - (-2, _) => IndexKey::Expression { - sql: parsed - .as_ref() - .and_then(|parsed| parsed.keys.get(position).cloned()) - .unwrap_or_default(), - descending, - }, - (-1, _) => IndexKey::Column { - name: "rowid".to_owned(), - descending, - }, - (_, column) => IndexKey::Column { - name: column.unwrap_or_default(), - descending, - }, - }); + .fetch(&mut *connection); + while let Some(entry) = rows.try_next().await.map_err(map_error)? { + listed.push(( + budget.keep(text(&entry, "name")?)?, + entry.try_get::("unique").map_err(map_error)? != 0, + entry.try_get::("partial").map_err(map_error)? != 0, + text(&entry, "origin")? == "pk", + )); } - let predicate = match partial { - 0 => None, - _ => parsed.and_then(|parsed| parsed.predicate), - }; - indexes.push(IndexInfo { - name, - keys, - unique: unique != 0, - primary, - predicate, - definition: stored_sql.map(|sql| in_schema(&sql, schema)), - }); + } + let mut indexes = Vec::with_capacity(listed.len() + 1); + for (name, unique, partial, primary) in listed { + indexes.push(index(connection, schema, name, unique, partial, primary, budget).await?); } // An INTEGER PRIMARY KEY is the rowid itself and has no index of its // own, but the structure tab still shows the key. @@ -667,7 +810,80 @@ async fn indexes( Ok(indexes) } -async fn table_shape(connection: &mut SqlxConnection, table: &TableRef) -> Result { +async fn index( + connection: &mut SqlxConnection, + schema: &str, + name: String, + unique: bool, + partial: bool, + primary: bool, + budget: &mut CatalogBudget, +) -> Result { + let stored_sql: Option = sqlx::query_scalar(AssertSqlSafe(catalog::index_sql("e(schema)))) + .bind(name.as_str()) + .fetch_optional(&mut *connection) + .await + .map_err(map_error)? + .flatten(); + let stored_sql = budget.keep_optional(stored_sql)?; + let parsed = stored_sql.as_deref().and_then(read_index_sql); + let mut keys = Vec::new(); + let mut rows = sqlx::query(catalog::INDEX_XINFO_SQL) + .bind(name.as_str()) + .bind(schema) + .fetch(&mut *connection); + while let Some(key) = rows.try_next().await.map_err(map_error)? { + let position = keys.len(); + let cid: i64 = key.try_get("cid").map_err(map_error)?; + let descending = key.try_get::("desc").map_err(map_error)? != 0; + let column: Option = budget.keep_optional(key.try_get("name").map_err(map_error)?)?; + keys.push(match (cid, column) { + (-2, _) => IndexKey::Expression { + sql: budget.keep( + parsed + .as_ref() + .and_then(|parsed| parsed.keys.get(position).cloned()) + .unwrap_or_default(), + )?, + descending, + }, + (-1, _) => IndexKey::Column { + name: "rowid".to_owned(), + descending, + }, + (_, column) => IndexKey::Column { + name: column.unwrap_or_default(), + descending, + }, + }); + } + let predicate = match partial { + false => None, + true => parsed.and_then(|parsed| parsed.predicate), + }; + let definition = match stored_sql { + Some(sql) => { + let definition = budget.keep(in_schema(&sql, schema))?; + budget.release_text(&sql); + Some(definition) + } + None => None, + }; + Ok(IndexInfo { + name, + keys, + unique, + primary, + predicate, + definition, + }) +} + +async fn table_shape( + connection: &mut SqlxConnection, + table: &TableRef, + budget: &mut CatalogBudget, +) -> Result { let schema = schema_of(table); let listed = sqlx::query(catalog::TABLE_LIST_SQL) .bind(table.name.as_str()) @@ -682,15 +898,7 @@ async fn table_shape(connection: &mut SqlxConnection, table: &TableRef) -> Resul let without_rowid = listed.try_get::("wr").map_err(map_error)? != 0; let strict = listed.try_get::("strict").map_err(map_error)? != 0; - let columns = sqlx::query(catalog::TABLE_XINFO_SQL) - .bind(table.name.as_str()) - .bind(schema) - .fetch_all(&mut *connection) - .await - .map_err(map_error)? - .iter() - .map(SqliteColumnRow::from_row) - .collect::, _>>()?; + let columns = column_rows(connection, &table.name, schema, budget).await?; let single_column_key = columns.iter().filter(|column| column.pk > 0).count() == 1; let mut facts = TableFacts { strict, @@ -699,9 +907,11 @@ async fn table_shape(connection: &mut SqlxConnection, table: &TableRef) -> Resul descending_key: false, }; if let Some(candidate) = columns.iter().find(|column| column.is_rowid_candidate(&facts)) { - facts.descending_key = create_statement(connection, table) - .await? - .is_some_and(|sql| declared_descending_key(&sql, &candidate.name)); + let sql = create_statement(connection, table, budget).await?; + facts.descending_key = sql + .as_deref() + .is_some_and(|sql| declared_descending_key(sql, &candidate.name)); + release(budget, sql); } Ok(TableShape { kind, facts, columns }) } diff --git a/core/crates/drivers/sqlite/src/write_batch.rs b/core/crates/drivers/sqlite/src/write_batch.rs index 27cb580cef..7ad777b908 100644 --- a/core/crates/drivers/sqlite/src/write_batch.rs +++ b/core/crates/drivers/sqlite/src/write_batch.rs @@ -9,22 +9,26 @@ //! and the batch then rolls back. use std::collections::HashMap; +use std::collections::hash_map::Entry; use std::time::Duration; +use futures::TryStreamExt; use sqlx::sqlite::SqliteConnection as SqlxConnection; use sqlx::{AssertSqlSafe, Connection as _, Row}; use tablepro_core::meta::TableRef; use tablepro_core::statement::{GuardedStatement, RowGuard}; use tablepro_core::{ - BusyReason, DriverError, ForeignKeyCheck, ForeignKeyMode, LossPhase, SessionFate, StopReason, StopRequest, - TimeoutPhase, WriteBatch, WriteLedger, WriteReport, + BusyReason, CatalogBudget, DriverError, ForeignKeyCheck, ForeignKeyMode, LossPhase, SessionFate, StopReason, + StopRequest, TimeoutPhase, WriteBatch, WriteLedger, WriteReport, }; use crate::autocommit::is_autocommit; -use crate::catalog::{FOREIGN_KEY_CHECK_SQL, FOREIGN_KEY_LIST_SQL, MAIN_SCHEMA}; +use crate::catalog::{ + DECLARED_TABLE_NAME_SQL, FOREIGN_KEY_CHECK_SQL, FOREIGN_KEY_LIST_SQL, ListedKeyColumn, MAIN_SCHEMA, +}; use crate::encode::bind_all; use crate::errors::{SQLITE_IOERR, is_interrupt, map_error, primary_of}; -use crate::foreign_key_verification::{Renames, ViolatedKey, Violation, introduced, violation_error}; +use crate::foreign_key_verification::{BrokenKey, Renames, ViolatedKey, Violations, introduced, violation_error}; use crate::interrupt::InterruptFlag; use crate::result_stream::{count, reports_changes}; use crate::sqlite_pooled::SqlitePooled; @@ -126,7 +130,7 @@ impl Write<'_> { return Err(self.failure(0, error, fate)); } }, - None => Vec::new(), + None => Violations::default(), }; let mut rows_affected = Vec::with_capacity(self.batch.steps.len()); @@ -300,8 +304,8 @@ fn stop_cause(reason: StopReason) -> DriverError { } /// The violations already there, named as they will be after the batch. -async fn baseline(connection: &mut SqlxConnection, checks: &[ForeignKeyCheck]) -> Result, DriverError> { - let mut found = Vec::new(); +async fn baseline(connection: &mut SqlxConnection, checks: &[ForeignKeyCheck]) -> Result { + let mut found = Violations::default(); let mut renames = Renames::default(); for check in checks { let Some(before) = &check.before else { @@ -316,8 +320,8 @@ async fn baseline(connection: &mut SqlxConnection, checks: &[ForeignKeyCheck]) - async fn violations_after( connection: &mut SqlxConnection, checks: &[ForeignKeyCheck], -) -> Result, DriverError> { - let mut found = Vec::new(); +) -> Result { + let mut found = Violations::default(); for check in checks { found.extend(violations(connection, &check.after, check.rowids_renumbered).await?); } @@ -331,56 +335,87 @@ async fn violations( connection: &mut SqlxConnection, table: &TableRef, rowids_renumbered: bool, -) -> Result, DriverError> { +) -> Result { let schema = table .schema .as_deref() .filter(|schema| !schema.is_empty()) .unwrap_or(MAIN_SCHEMA); - let rows = sqlx::query(FOREIGN_KEY_CHECK_SQL) + let mut rows: Vec<(i64, Option)> = Vec::new(); + { + let mut read = sqlx::query(FOREIGN_KEY_CHECK_SQL) + .bind(table.name.as_str()) + .bind(schema) + .fetch(&mut *connection); + while let Some(row) = read.try_next().await.map_err(map_error)? { + let rowid: Option = row.try_get("rowid").map_err(map_error)?; + rows.push((row.try_get("fkid").map_err(map_error)?, rowid)); + } + } + if rows.is_empty() { + return Ok(Violations::default()); + } + let declared: Option = sqlx::query_scalar(DECLARED_TABLE_NAME_SQL) .bind(table.name.as_str()) .bind(schema) - .fetch_all(&mut *connection) + .fetch_optional(&mut *connection) .await .map_err(map_error)?; - if rows.is_empty() { - return Ok(Vec::new()); - } + let name = declared.unwrap_or_else(|| table.name.clone()); let keys = foreign_keys(connection, table, schema).await?; - rows.iter() - .map(|row| { - let rowid: Option = row.try_get("rowid").map_err(map_error)?; - let fkid: i64 = row.try_get("fkid").map_err(map_error)?; - Ok(Violation { - table: row.try_get("table").map_err(map_error)?, - rowid: rowid.filter(|_| !rowids_renumbered), - parent: row.try_get("parent").map_err(map_error)?, - key: keys.get(&fkid).cloned().unwrap_or_default(), + let mut found = Violations::default(); + let mut positions: HashMap = HashMap::new(); + for (fkid, rowid) in rows { + let position = *positions.entry(fkid).or_insert_with(|| { + let (parent, key) = keys.get(&fkid).cloned().unwrap_or_default(); + found.add_key(BrokenKey { + table: name.clone(), + parent, + key, }) - }) - .collect() + }); + found.add_row(position, rowid.filter(|_| !rowids_renumbered)); + } + Ok(found) } -/// The table's foreign keys by the id `foreign_key_check` names them by. +/// The table's foreign keys, with the parent each names, by the id +/// `foreign_key_check` names them by. async fn foreign_keys( connection: &mut SqlxConnection, table: &TableRef, schema: &str, -) -> Result, DriverError> { - let rows = sqlx::query(FOREIGN_KEY_LIST_SQL) +) -> Result, DriverError> { + let mut budget = CatalogBudget::default(); + let mut rows = sqlx::query(FOREIGN_KEY_LIST_SQL) .bind(table.name.as_str()) .bind(schema) - .fetch_all(&mut *connection) - .await - .map_err(map_error)?; - let mut keys: HashMap = HashMap::new(); - for row in rows { + .fetch(&mut *connection); + let mut keys: HashMap)> = HashMap::new(); + while let Some(row) = rows.try_next().await.map_err(map_error)? { let id: i64 = row.try_get("id").map_err(map_error)?; - let from: String = row.try_get("from").map_err(map_error)?; - let to: Option = row.try_get("to").map_err(map_error)?; - let key = keys.entry(id).or_default(); - key.columns.push(from); - key.parent_columns.extend(to); + let column = ( + row.try_get("seq").map_err(map_error)?, + budget.keep(row.try_get("from").map_err(map_error)?)?, + budget.keep_optional(row.try_get("to").map_err(map_error)?)?, + ); + match keys.entry(id) { + Entry::Occupied(entry) => entry.into_mut().1.push(column), + Entry::Vacant(entry) => { + entry.insert((budget.keep(row.try_get("table").map_err(map_error)?)?, vec![column])); + } + } } - Ok(keys) + Ok(keys + .into_iter() + .map(|(id, (parent, mut columns))| { + columns.sort_by_key(|(seq, _, _)| *seq); + let mut key = ViolatedKey::default(); + for (_, from, to) in columns { + key.columns.push(from); + key.parent_columns.extend(to); + } + (id, (parent, key)) + }) + .collect()) } diff --git a/core/crates/drivers/sqlite/tests/catalog_memory.rs b/core/crates/drivers/sqlite/tests/catalog_memory.rs new file mode 100644 index 0000000000..6072474fc0 --- /dev/null +++ b/core/crates/drivers/sqlite/tests/catalog_memory.rs @@ -0,0 +1,174 @@ +// Catalog reads of a crafted file stop at the catalog budget. A test binary of its own because it counts every heap +// allocation the process makes, which another test running beside it would add to. Add no other test here. + +use std::sync::Arc; +use std::time::Duration; + +use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection}; +use sqlx::{Connection as _, Executor as _}; +use tablepro_core::meta::{ObjectScope, TableRef}; +use tablepro_core::statement::{GuardedStatement, StatementBuilder}; +use tablepro_core::{ + ByteLimit, CallOptions, ConnectEnvironment, ConnectOptions, ConnectTarget, Connection, DatabaseDriver, DriverError, + FileEndpoint, ForeignKeyCheck, ForeignKeyMode, WorkspaceSettings, WriteBatch, WriteKind, +}; +use tablepro_driver_sqlite::SqliteDriver; +use tablepro_test_support::conformance::test_liveness; +use tablepro_test_support::{HeapCounter, heap_peak_of as peak_of}; +use tokio::time::Instant; +use tokio_util::sync::CancellationToken; + +#[global_allocator] +static HEAP: HeapCounter = HeapCounter; + +// The budget counts the text a read holds, not the vectors and maps around it. +const BOUND: usize = 2 * ByteLimit::CATALOG_DEFAULT.get() as usize; + +const LONG: usize = 64 * 1024; +const KEYS: usize = 4_000; +const BROKEN_ROWS: usize = 20_000; + +fn quoted(name: &str) -> String { + format!("\"{name}\"") +} + +// Each shape costs the file a few hundred kilobytes and, read without a bound, a gigabyte or more: +// - one column carrying REFERENCES 4,000 times, which the pragma names on every key; +// - a child with a 64 KiB name and 4,000 keys, each of which names the child; +// - 4,000 keys naming no columns of a parent whose primary key is 64 KiB of names, each a copy of it; +// - 20,000 rows of a table with a 64 KiB name that break its key, which a check named on every row. +fn crafted_schema() -> Vec { + let column = "c".repeat(LONG); + let child = "n".repeat(LONG); + let key_columns: Vec = (0..64) + .map(|index| format!("k{index:02}{}", "k".repeat(1_021))) + .collect(); + let declared: Vec = key_columns + .iter() + .map(|name| format!("{} INTEGER", quoted(name))) + .collect(); + let listed: Vec = key_columns.iter().map(|name| quoted(name)).collect(); + vec![ + "CREATE TABLE repeated_parent (id INTEGER PRIMARY KEY)".to_owned(), + format!( + "CREATE TABLE repeated ({} INTEGER{})", + quoted(&column), + " REFERENCES repeated_parent".repeat(KEYS) + ), + "CREATE TABLE named_parent (id INTEGER PRIMARY KEY)".to_owned(), + format!( + "CREATE TABLE {} (a INTEGER{})", + quoted(&child), + ", FOREIGN KEY (a) REFERENCES named_parent".repeat(KEYS) + ), + format!( + "CREATE TABLE wide_key ({}, PRIMARY KEY ({}))", + declared.join(", "), + listed.join(", ") + ), + format!( + "CREATE TABLE pointer (a INTEGER{})", + ", FOREIGN KEY (a) REFERENCES wide_key".repeat(KEYS) + ), + format!("CREATE TABLE {} (a INTEGER REFERENCES named_parent)", quoted(&broken())), + format!( + "WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < {BROKEN_ROWS}) \ + INSERT INTO {} SELECT i FROM n", + quoted(&broken()) + ), + ] +} + +fn broken() -> String { + "b".repeat(LONG) +} + +fn call() -> CallOptions { + CallOptions::new( + Some(Instant::now() + Duration::from_secs(120)), + CancellationToken::new(), + ) +} + +async fn open(path: &std::path::Path) -> Result, DriverError> { + let options = ConnectOptions { + target: ConnectTarget::File(FileEndpoint::open_existing(path.to_path_buf())?), + workspace_settings: WorkspaceSettings::workspace(false), + liveness: test_liveness(), + environment: ConnectEnvironment::new("test"), + }; + SqliteDriver.connect(options, CancellationToken::new()).await +} + +fn too_large(read: &str, result: Result) { + let limit = ByteLimit::CATALOG_DEFAULT.get(); + assert!( + matches!(result, Err(DriverError::CatalogTooLarge { limit_bytes }) if limit_bytes == limit), + "{read}: {:?}", + result.err() + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_crafted_catalog_stops_at_the_budget() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("crafted.sqlite3"); + let options = SqliteConnectOptions::new() + .filename(&path) + .create_if_missing(true) + .foreign_keys(false); + let mut writer = SqliteConnection::connect_with(&options).await.unwrap(); + for sql in crafted_schema() { + writer.execute(sqlx::AssertSqlSafe(sql)).await.unwrap(); + } + writer.close().await.unwrap(); + let connection = open(&path).await.unwrap(); + let table = |name: &str| TableRef::unqualified(name); + let child = table(&"n".repeat(LONG)); + + let (objects, peak) = peak_of(connection.list_objects(&ObjectScope::DefaultSchema, call())).await; + assert_eq!(objects.unwrap().len(), 7); + assert!(peak < BOUND, "listing the tables held {peak} bytes"); + let (columns, _) = peak_of(connection.fetch_columns(&table("repeated"), call())).await; + assert_eq!(columns.unwrap().len(), 1, "a long name alone is read"); + + for (name, table) in [ + ("repeated", table("repeated")), + ("the long child", child.clone()), + ("pointer", table("pointer")), + ] { + let (read, peak) = peak_of(connection.fetch_foreign_keys(&table, call())).await; + too_large(&format!("keys of {name}"), read); + assert!(peak < BOUND, "keys of {name} held {peak} bytes"); + let (read, peak) = peak_of(connection.fetch_outbound_foreign_keys(&table, call())).await; + too_large(&format!("outbound keys of {name}"), read); + assert!(peak < BOUND, "outbound keys of {name} held {peak} bytes"); + let (read, peak) = peak_of(connection.fetch_table_definition(&table, call())).await; + too_large(&format!("definition of {name}"), read); + assert!(peak < BOUND, "definition of {name} held {peak} bytes"); + } + for parent in ["repeated_parent", "named_parent", "wide_key"] { + let (read, peak) = peak_of(connection.fetch_inbound_foreign_keys(&table(parent), call())).await; + too_large(&format!("keys into {parent}"), read); + assert!(peak < BOUND, "keys into {parent} held {peak} bytes"); + let (read, peak) = peak_of(connection.fetch_table_definition(&table(parent), call())).await; + too_large(&format!("definition of {parent}"), read); + assert!(peak < BOUND, "definition of {parent} held {peak} bytes"); + } + + // A rebuild checks the keys its table breaks before and after; the rows that broke them already are not its own. + let mut builder = StatementBuilder::new(); + builder.push_sql(&format!("UPDATE {} SET a = a WHERE 0", quoted(&broken()))); + let batch = WriteBatch::new( + WriteKind::SchemaChanges, + table(&broken()), + vec![GuardedStatement::unguarded(builder.finish())], + ForeignKeyMode::SuspendedThenVerified { + checks: vec![ForeignKeyCheck::same_table(table(&broken()))], + }, + ); + let (written, peak) = peak_of(connection.write_batch(&batch, call())).await; + assert!(written.is_ok(), "{written:?}"); + assert!(peak < BOUND / 8, "checking {BROKEN_ROWS} broken rows held {peak} bytes"); + connection.close().await.unwrap(); +} diff --git a/core/crates/drivers/sqlite/tests/integration/catalog.rs b/core/crates/drivers/sqlite/tests/integration/catalog.rs index b494cd01e2..403c81e084 100644 --- a/core/crates/drivers/sqlite/tests/integration/catalog.rs +++ b/core/crates/drivers/sqlite/tests/integration/catalog.rs @@ -1,10 +1,12 @@ //! Column kinds from SQLite's structured catalogue, and table //! definitions with their dependents. +use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection}; +use sqlx::{Connection as _, Executor as _}; use tablepro_core::Value; use tablepro_core::column::{ColumnDefault, ColumnKind, IntegerKind, IntegerWidth}; use tablepro_core::ddl::{GeneratedStorage, IdentityStyle, SchemaObjectKind}; -use tablepro_core::meta::TableRef; +use tablepro_core::meta::{ObjectScope, TableRef}; use crate::support::{TestFile, apply, call}; @@ -306,6 +308,99 @@ async fn inbound_keys_read_alone_are_the_definitions() { connection.close().await.unwrap(); } +// A large real schema stays under the catalog budget: every table, and every key into a table thousands of others +// point at, read whole and in order. SQLite takes time quadratic in the tables to create them, hence not 50,000. +#[tokio::test] +async fn a_large_catalog_reads_whole() { + const CHILDREN: usize = 5_000; + const WIDE_KEYS: usize = 500; + let file = TestFile::new().await; + let mut script = String::from("BEGIN; CREATE TABLE users (id INTEGER PRIMARY KEY, email TEXT UNIQUE);"); + for child in 0..CHILDREN { + script.push_str(&format!( + "CREATE TABLE child_{child:05} (id INTEGER PRIMARY KEY, owner INTEGER REFERENCES users, email TEXT, \ + CONSTRAINT by_email_{child:05} FOREIGN KEY (email) REFERENCES users (email));" + )); + } + let columns: Vec = (0..WIDE_KEYS * 2).map(|column| format!("c{column} INTEGER")).collect(); + let keys: Vec = (0..WIDE_KEYS) + .map(|column| format!("FOREIGN KEY (c{column}) REFERENCES users")) + .collect(); + script.push_str(&format!( + "CREATE TABLE wide ({}, {}); COMMIT;", + columns.join(", "), + keys.join(", ") + )); + let mut writer = SqliteConnection::connect_with(&SqliteConnectOptions::new().filename(&file.path)) + .await + .unwrap(); + writer.execute(sqlx::AssertSqlSafe(script)).await.unwrap(); + writer.close().await.unwrap(); + let connection = file.open().await; + let users = TableRef::unqualified("users"); + let objects = connection + .list_objects(&ObjectScope::DefaultSchema, call()) + .await + .unwrap(); + let inbound = connection.fetch_inbound_foreign_keys(&users, call()).await.unwrap(); + let definition = connection.fetch_table_definition(&users, call()).await.unwrap(); + let wide = connection + .fetch_outbound_foreign_keys(&TableRef::unqualified("wide"), call()) + .await + .unwrap(); + + assert_eq!(objects.len(), CHILDREN + 2); + assert_eq!(inbound.len(), CHILDREN * 2 + WIDE_KEYS); + type Key<'a> = (&'a str, &'a str, &'a [String], &'a [String], bool); + let first: Vec> = inbound[..2] + .iter() + .map(|key| { + ( + key.table.name.as_str(), + key.name.as_str(), + key.columns.as_slice(), + key.referenced_columns.as_slice(), + key.child_compares_alike, + ) + }) + .collect(); + assert_eq!( + first, + [ + ( + "child_00000", + "by_email_00000", + &["email".to_owned()][..], + &["email".to_owned()][..], + true + ), + ( + "child_00000", + "fk_child_00000_1", + &["owner".to_owned()][..], + &["id".to_owned()][..], + true + ), + ] + ); + assert!( + inbound.windows(2).all(|pair| pair[0].table.name <= pair[1].table.name), + "ordered by child" + ); + assert_eq!(inbound.last().map(|key| key.table.name.as_str()), Some("wide")); + assert_eq!(definition.inbound_foreign_keys, inbound); + assert_eq!(wide.len(), WIDE_KEYS); + assert!( + wide.iter() + .enumerate() + .all(|(id, key)| key.key.name == format!("fk_wide_{id}") + && key.key.ref_table == "users" + && key.key.ref_columns == ["id"] + && key.parent_compares_alike) + ); + connection.close().await.unwrap(); +} + #[tokio::test] async fn dependents_and_source_are_read() { let file = TestFile::new().await; diff --git a/core/crates/test-support/src/heap_peak.rs b/core/crates/test-support/src/heap_peak.rs new file mode 100644 index 0000000000..3c299d8fa9 --- /dev/null +++ b/core/crates/test-support/src/heap_peak.rs @@ -0,0 +1,55 @@ +use std::alloc::{GlobalAlloc, Layout, System}; +use std::future::Future; +use std::sync::atomic::{AtomicUsize, Ordering}; + +// The global allocator of a test binary that bounds what a read holds on the heap, on every platform. It counts the +// whole process, so such a binary holds that one test alone. +pub struct HeapCounter; + +static HELD: AtomicUsize = AtomicUsize::new(0); +static PEAK: AtomicUsize = AtomicUsize::new(0); + +fn grew(by: usize) { + let held = HELD.fetch_add(by, Ordering::Relaxed).saturating_add(by); + PEAK.fetch_max(held, Ordering::Relaxed); +} + +// SAFETY: every call goes to the system allocator with the caller's own arguments; the counters only watch. +unsafe impl GlobalAlloc for HeapCounter { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + // SAFETY: the caller keeps `alloc`'s contract, which is the system allocator's. + let pointer = unsafe { System.alloc(layout) }; + if !pointer.is_null() { + grew(layout.size()); + } + pointer + } + + unsafe fn dealloc(&self, pointer: *mut u8, layout: Layout) { + // SAFETY: `pointer` came from this allocator, which is the system's, with `layout`. + unsafe { System.dealloc(pointer, layout) }; + HELD.fetch_sub(layout.size(), Ordering::Relaxed); + } + + unsafe fn realloc(&self, pointer: *mut u8, layout: Layout, new_size: usize) -> *mut u8 { + // SAFETY: as for `dealloc`, with the caller's `new_size`. + let moved = unsafe { System.realloc(pointer, layout, new_size) }; + if !moved.is_null() { + match new_size.checked_sub(layout.size()) { + Some(more) => grew(more), + None => { + HELD.fetch_sub(layout.size() - new_size, Ordering::Relaxed); + } + } + } + moved + } +} + +// How far the heap rose above where it stood when `read` started, while it ran. Without `HeapCounter` installed it is 0. +pub async fn heap_peak_of(read: impl Future) -> (T, usize) { + let start = HELD.load(Ordering::Relaxed); + PEAK.store(start, Ordering::Relaxed); + let value = read.await; + (value, PEAK.load(Ordering::Relaxed).saturating_sub(start)) +} diff --git a/core/crates/test-support/src/lib.rs b/core/crates/test-support/src/lib.rs index 066d78d4a7..d48fa551de 100644 --- a/core/crates/test-support/src/lib.rs +++ b/core/crates/test-support/src/lib.rs @@ -25,6 +25,7 @@ mod fake_reply; mod fake_run; mod fake_secret_vault; mod fault; +mod heap_peak; pub mod history_cases; mod replies; mod round_trip_counter; @@ -49,6 +50,7 @@ pub use fake_reply::FakeReply; pub use fake_run::{FakeRun, RunWait}; pub use fake_secret_vault::{Call, FakeSecretVault}; pub use fault::FaultTransport; +pub use heap_peak::{HeapCounter, heap_peak_of}; pub use replies::Replies; pub use round_trip_counter::{RoundTripCounter, StreamTally}; pub use snippet_text::expand_snippet; diff --git a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md new file mode 100644 index 0000000000..b64e0b56fc --- /dev/null +++ b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md @@ -0,0 +1,30 @@ +# 0029: A catalog read keeps a byte budget + +- **Status**: Accepted +- **Date**: 2026-11-15 + +## Context + +A file engine reads its catalog from the file the user opened, and the file chooses every name in it. SQLite's pragmas repeat a name on every key that uses it: a column that carries REFERENCES thousands of times comes back with its name on every key, a child's name on every key into its parent, and a key that names no parent columns took a copy of the parent's whole primary key. The keys into a table also carried the child's whole CREATE TABLE on each key column, and that text grows with the keys the child declares. DuckDB's keys into a table did the same with the child's CREATE TABLE. A file of a few hundred kilobytes made one read of a table's keys hold gigabytes, and each read held all its rows before it looked at the first. + +## Decision + +`CatalogBudget` in `tablepro-core` bounds what one catalog read holds at `ByteLimit::CATALOG_DEFAULT`, 64,000,000 bytes, counted as `Value::retained_bytes` counts text: a string's slot and its length. Past it the read fails with `DriverError::CatalogTooLarge { limit_bytes }`, code `driver.catalog_too_large`, category `Internal`. + +- SQLite and DuckDB charge each text a read keeps as it arrives, and each copy made per key before it is made. A text read and dropped before the next one, as each child's CREATE TABLE, is released. +- Each text of the file is read once per call. SQLite streams catalog rows, unsorted where a sort would hold every row before the first, reads each child's CREATE TABLE once in a second query, and parses each CREATE TABLE once. DuckDB reads each child's and each parent's CREATE TABLE once. +- A save's foreign key check reads a rowid and a key id for each row that breaks a key, not the table's and parent's names. + +## Rationale + +- **An error, not a cut list.** A table definition plans structure changes, and a rebuild planned from a list cut short would drop what was left out. +- **64 MB.** A schema of 5,000 tables keyed to one table reads its keys in a few megabytes, and a million table names fit under it. +- **In the driver.** The driver knows which texts it copies per key, and no app has to pass a limit. + +## Consequences + +- Large real schemas read as before; the keys into a table read faster, without a child's CREATE TABLE on each row. +- One value is read before it is charged. SQLite bounds a value at `SQLITE_MAX_LENGTH` and parsed the text when it opened the file. +- DuckDB holds a catalog query's whole result before the first row, and names each constraint after its table. The budget bounds what the driver keeps, not that. +- Not budgeted: the schema list, reads of one value, and the SQL dumps, which read each object once. +- The apps word the code: Linux in `linux/crates/app/src/ui/error_text/driver.rs`, Windows in `Resources.resw`. diff --git a/core/docs/decisions/README.md b/core/docs/decisions/README.md index e197123ccc..a8fad9dd23 100644 --- a/core/docs/decisions/README.md +++ b/core/docs/decisions/README.md @@ -36,3 +36,4 @@ The format, the statuses and how a later ADR changes an earlier one follow [the | [0026](0026-the-windows-app-signs-in-through-the-ffi.md) | The Windows app signs in to its TablePro account through the FFI | Accepted | `Core::open_account` builds the account with `AppClient::WINDOWS`, the Core's secret store for the refresh token, a `HostBrowser` callback and `MachineId::derive_machine_guid` of the registry's MachineGuid, keyed as Linux keys `/etc/machine-id`; `status()` is a snapshot and `AccountListener` only signals. | | [0027](0027-share-links.md) | Share links go through the account, with the snapshot and the secret scan in the core | Accepted; masked columns and the secret scan amended by 0028 | `Account` adds the share calls through the generated `/v2` client, `fetch_shared` without a token; problem codes stay `Refused` with their own `code()`; `build_snapshot` writes raw JSON cells by the export's lossless rules up to the plan's rows; a table page shares `build_view_sql`, its export with the filter's values written in; `secret_scan` warns; the deep link takes one 43-character base62 token. | | [0028](0028-data-masking.md) | A shared result masks its columns before it leaves the machine, with the macOS app's output | Accepted | `tablepro_core::masking` gives the macOS styles byte for byte; `SharePlan` applies the sheet's Keep, Mask or Remove per column for the snapshot, its preview and summary; the snapshot names each masked column's style and keeps to the macOS body budget; `masking::detect` suggests masks from names, values and the SQL by PLAN-MASKING 8.3. | +| [0029](0029-a-catalog-read-keeps-a-byte-budget.md) | A catalog read keeps a byte budget | Accepted | `CatalogBudget` bounds what one SQLite or DuckDB catalog read holds at 64 MB, counting each text kept and each copy made per key, and fails `DriverError::CatalogTooLarge` past it; each text of the file is read once per call, rows stream unsorted, and a save's foreign key check reads no names per row. | diff --git a/linux/crates/app/src/ui/error_text/driver.rs b/linux/crates/app/src/ui/error_text/driver.rs index 0e19e289d5..21921d1081 100644 --- a/linux/crates/app/src/ui/error_text/driver.rs +++ b/linux/crates/app/src/ui/error_text/driver.rs @@ -340,6 +340,10 @@ fn internal_message(error: &DriverError) -> String { &[("column", column)], ), DriverError::Protocol(_) => crate::i18n::gettext("The driver could not read the server's reply."), + DriverError::CatalogTooLarge { limit_bytes } => crate::i18n::gettext_f( + "The database's structure is larger than TablePro reads at once ({size}).", + &[("size", gtk4::glib::format_size(*limit_bytes).as_str())], + ), _ => crate::i18n::gettext("The driver hit an internal error."), } } diff --git a/linux/crates/app/src/ui/error_text/tests.rs b/linux/crates/app/src/ui/error_text/tests.rs index de00476bdf..5dcecba876 100644 --- a/linux/crates/app/src/ui/error_text/tests.rs +++ b/linux/crates/app/src/ui/error_text/tests.rs @@ -90,6 +90,15 @@ fn every_category_has_a_message() { assert_eq!(messages.len(), cases.len(), "two categories share a message"); } +#[test] +fn a_catalog_past_the_budget_names_the_limit() { + let message = driver_message(&DriverError::CatalogTooLarge { + limit_bytes: 64_000_000, + }); + + assert!(message.contains("64.0"), "{message}"); +} + #[test] fn server_message_includes_code_detail_hint() { let diagnostics = ServerDiagnostics { diff --git a/linux/docs/adding-drivers.md b/linux/docs/adding-drivers.md index e71709fdfd..5d891a3117 100644 --- a/linux/docs/adding-drivers.md +++ b/linux/docs/adding-drivers.md @@ -164,7 +164,7 @@ The `Connection` is a workspace's link to one server: the pool behind the sideba | `server_info`, `dialect`, `read_only` | Fixed at connect. | | `link_state`, `lost` | A `LinkWatch` shared by every call. `lost()` resolves when the link is gone: a failed probe, or a failed call that a check outside the pool confirmed. One dead backend in the pool does not resolve it, so the session monitor can act on it. | | `probe` | Whether the server answers. It never waits for a pool slot: with every slot busy it opens a connection of its own. | -| `list_schemas`, `list_objects`, `fetch_columns`, `fetch_indexes`, `fetch_foreign_keys`, `fetch_outbound_foreign_keys`, `fetch_inbound_foreign_keys`, `fetch_table_definition`, `row_identity`, `table_estimate` | Catalogue reads. `fetch_table_definition` reads columns, keys and dependents in one call so they come from the same moment. `fetch_outbound_foreign_keys` and `fetch_inbound_foreign_keys` return the keys that definition lists, read alone for a table tab's cell menu. Each key names its other table as the sidebar does, and its columns as declared. `parent_compares_alike` and `child_compares_alike` are false when a value of one side, as text read under the other side's column type, can find other rows than the key matches: the two sides differ in type or collation and the engine compares them through a conversion. | +| `list_schemas`, `list_objects`, `fetch_columns`, `fetch_indexes`, `fetch_foreign_keys`, `fetch_outbound_foreign_keys`, `fetch_inbound_foreign_keys`, `fetch_table_definition`, `row_identity`, `table_estimate` | Catalogue reads. `fetch_table_definition` reads columns, keys and dependents in one call so they come from the same moment. `fetch_outbound_foreign_keys` and `fetch_inbound_foreign_keys` return the keys that definition lists, read alone for a table tab's cell menu. Each key names its other table as the sidebar does, and its columns as declared. `parent_compares_alike` and `child_compares_alike` are false when a value of one side, as text read under the other side's column type, can find other rows than the key matches: the two sides differ in type or collation and the engine compares them through a conversion. A file engine's read holds at most a `CatalogBudget` and fails `CatalogTooLarge` past it (core ADR 0029). | | `fetch_character_sets` | The character sets the server offers, each with its default collation and every collation it takes, for the Structure tab's pickers. A read of its own after the definition, so its failure leaves the rest of the tab working. The default answers `Unsupported` (`UnsupportedFeature::CharacterSets`). | | `query_bound` | A generated read that keeps at most `limit` rows and reads one more to set `truncated`. | | `count_rows` | A generated count. | diff --git a/linux/docs/error-handling.md b/linux/docs/error-handling.md index 50667d0d59..6eaa427cf5 100644 --- a/linux/docs/error-handling.md +++ b/linux/docs/error-handling.md @@ -41,6 +41,7 @@ Errors are typed enums, built with `thiserror` wherever they cross a crate bound | `Decode { column, detail }` | The driver could not read a value. A bug on our side, not the server's. The detail is for the log. | | `Protocol(String)` | The server or the client library said something the driver did not expect. The text is for the log. | | `Unsupported { feature }` | The engine has no such feature. `UnsupportedFeature` names it, and the app writes the sentence. | +| `CatalogTooLarge { limit_bytes }` | A catalog read would hold more than its budget, as a crafted file makes it ([core ADR 0029](../../core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md)). | Rules: @@ -67,7 +68,7 @@ Rules: | `WriteOutcome` | `RolledBack`, `RowGuardFailed`, `CommitOutcomeUnknown`, `PartiallyApplied` | Says what is in the database now. Only `RolledBack` and `RowGuardFailed` may say nothing changed. | | `File` | the four file variants | Names the path. A connect that failed on the file is not retried. | | `Busy` | `Busy`, `EditorSessionBusy` | Says to try again shortly, that another program or tab holds the file's lock, or to wait for the tab's statement. | -| `Internal` | `GuardUnsupported`, `Decode`, `Protocol`; `SshFailure::Protocol` and `SshFailure::Internal` | Says what the driver could not do: check a save, read a value, read the reply. | +| `Internal` | `GuardUnsupported`, `Decode`, `Protocol`, `CatalogTooLarge`; `SshFailure::Protocol` and `SshFailure::Internal` | Says what the driver could not do: check a save, read a value, read the reply, read a catalog past its budget. | ## `ServerDiagnostics` diff --git a/windows/src/TablePro.App/Strings/en-US/Resources.resw b/windows/src/TablePro.App/Strings/en-US/Resources.resw index bef28b743a..5d2b7eefd2 100644 --- a/windows/src/TablePro.App/Strings/en-US/Resources.resw +++ b/windows/src/TablePro.App/Strings/en-US/Resources.resw @@ -2337,6 +2337,9 @@ New fingerprint: {2} This database doesn't support that. + + The database's structure is larger than TablePro reads at once. + Data or structure From 4acb8c631f3dfea412ca9caa4d353565f34d4661 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 01:29:31 +0700 Subject: [PATCH 2/9] fix(core): read a DuckDB dump's parents as a list, not text split on a separator --- core/crates/drivers/duckdb/src/dump.rs | 6 ++-- .../duckdb/tests/integration/catalog.rs | 29 +++++++++++++++++++ 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/core/crates/drivers/duckdb/src/dump.rs b/core/crates/drivers/duckdb/src/dump.rs index 27a34c6afb..24fd33f317 100644 --- a/core/crates/drivers/duckdb/src/dump.rs +++ b/core/crates/drivers/duckdb/src/dump.rs @@ -65,7 +65,7 @@ const TABLES_SQL: &str = "SELECT schema_name, table_name, sql FROM duckdb_tables /// DuckDB keeps a foreign key's parent in the child's schema. One row /// per child, not per key, which would repeat the child's name. const FOREIGN_KEYS_SQL: &str = "SELECT schema_name, table_name, \ - string_agg(referenced_table, chr(31) ORDER BY constraint_index) \ + list(referenced_table ORDER BY constraint_index) \ FROM duckdb_constraints() \ WHERE database_name = ? AND (? IS NULL OR schema_name = ?) AND constraint_type = 'FOREIGN KEY' \ GROUP BY schema_name, table_name ORDER BY schema_name, table_name"; @@ -326,9 +326,7 @@ fn add_tables(reader: &Reader<'_>, dump: &mut DumpCatalog) -> Result<(), DriverE Ok(( row.get::<_, String>(0)?, row.get::<_, String>(1)?, - row.get::<_, Option>(2)? - .map(|parents| parents.split('\u{1f}').map(str::to_owned).collect()) - .unwrap_or_default(), + texts(row.get(2)?).into_iter().flatten().collect(), )) })?; let listed: Vec<&Named> = tables.iter().filter(|table| table.text().is_some()).collect(); diff --git a/core/crates/drivers/duckdb/tests/integration/catalog.rs b/core/crates/drivers/duckdb/tests/integration/catalog.rs index 7a5ed7634e..b6ac813fd4 100644 --- a/core/crates/drivers/duckdb/tests/integration/catalog.rs +++ b/core/crates/drivers/duckdb/tests/integration/catalog.rs @@ -1,6 +1,7 @@ //! What the catalogue reads report for a DuckDB file. use tablepro_core::column::ColumnDefault; +use tablepro_core::dump::DumpScope; use tablepro_core::meta::{IndexKey, ObjectKind, ObjectScope, RowCountEstimate, RowIdentity, TableRef}; use tablepro_core::{RunEnd, Value}; @@ -434,3 +435,31 @@ async fn a_schema_named_like_the_file_is_read_and_changed() { assert_eq!(dialect.portable_name(&table), "\"test\".\"t\""); connection.close().await.unwrap(); } + +// A dump reads each child's parents as one list, which a separator character inside a quoted name cannot split. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_dump_orders_a_child_after_each_parent_it_names() { + let file = TestFile::new(); + let connection = file.open().await; + apply( + connection.as_ref(), + &[ + "CREATE TABLE \"z\u{1f}parent\" (id INTEGER PRIMARY KEY)", + "CREATE TABLE other (id INTEGER PRIMARY KEY)", + "CREATE TABLE achild (a INTEGER REFERENCES \"z\u{1f}parent\" (id), b INTEGER REFERENCES other (id), \ + c INTEGER REFERENCES \"Z\u{1f}PARENT\" (id))", + ], + ) + .await + .unwrap(); + + let dump = connection + .read_dump_catalog(&DumpScope::Database, call()) + .await + .unwrap(); + + let child = dump.tables.iter().find(|table| table.table.name == "achild").unwrap(); + let parents: Vec<&str> = child.references.iter().map(|parent| parent.name.as_str()).collect(); + assert_eq!(parents, ["z\u{1f}parent", "other"]); + connection.close().await.unwrap(); +} From 1d4896dbadaa9da314a91fe1a59770a7216c3c51 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 01:49:09 +0700 Subject: [PATCH 3/9] fix(core): cost a SQLite key list before compiling it, and ask only tables that name the parent --- core/crates/core/src/catalog_budget.rs | 21 ++ core/crates/drivers/duckdb/src/catalog/mod.rs | 13 +- core/crates/drivers/sqlite/src/catalog/mod.rs | 57 +++-- .../sqlite/src/ddl/create_table_source.rs | 39 ++++ core/crates/drivers/sqlite/src/ddl/mod.rs | 4 +- .../sqlite/src/foreign_key_verification.rs | 216 +++++++++++------- core/crates/drivers/sqlite/src/metadata.rs | 154 +++++++++---- core/crates/drivers/sqlite/src/write_batch.rs | 31 ++- .../drivers/sqlite/tests/catalog_memory.rs | 126 +++++++--- .../sqlite/tests/integration/catalog.rs | 29 +++ ...0029-a-catalog-read-keeps-a-byte-budget.md | 14 +- core/docs/decisions/README.md | 2 +- 12 files changed, 515 insertions(+), 191 deletions(-) diff --git a/core/crates/core/src/catalog_budget.rs b/core/crates/core/src/catalog_budget.rs index c2d10438eb..36300cbf51 100644 --- a/core/crates/core/src/catalog_budget.rs +++ b/core/crates/core/src/catalog_budget.rs @@ -27,6 +27,17 @@ impl CatalogBudget { } } + // For work the engine does and lets go of within the read, such as a statement it compiles: it has to fit beside + // what the read holds, and holds nothing after. + pub fn fits(&self, bytes: usize) -> Result<(), DriverError> { + match self.held.saturating_add(u64::try_from(bytes).unwrap_or(u64::MAX)) > self.limit { + true => Err(DriverError::CatalogTooLarge { + limit_bytes: self.limit, + }), + false => Ok(()), + } + } + pub fn charge_text(&mut self, text: &str) -> Result<(), DriverError> { self.charge(text_bytes(text)) } @@ -105,6 +116,16 @@ mod tests { assert!(budget.charge_texts([&"b".to_owned()]).is_err()); } + #[test] + fn work_that_fits_beside_what_is_held_holds_nothing_after() { + let mut budget = budget(100); + budget.charge(60).unwrap(); + + assert_eq!(budget.fits(40), Ok(())); + assert!(budget.fits(41).is_err()); + assert_eq!(budget.charge(40), Ok(()), "fits charged nothing"); + } + #[test] fn the_default_is_the_catalog_limit() { let mut budget = CatalogBudget::default(); diff --git a/core/crates/drivers/duckdb/src/catalog/mod.rs b/core/crates/drivers/duckdb/src/catalog/mod.rs index 6fdf6ed3aa..ce5910ca4a 100644 --- a/core/crates/drivers/duckdb/src/catalog/mod.rs +++ b/core/crates/drivers/duckdb/src/catalog/mod.rs @@ -461,6 +461,7 @@ pub(crate) fn outbound_foreign_keys( None => { let referenced = TableRef::new(key.ref_schema.clone(), key.ref_table.clone()); let collated = collated_columns(link, catalog, &referenced, budget)?; + budget.charge_texts(parent.0.iter().chain([&parent.1]))?; parents.entry(parent).or_insert(collated) } }; @@ -485,7 +486,7 @@ fn collated_columns( budget: &mut CatalogBudget, ) -> Result, DriverError> { let sql = table_sql(link, catalog, table, budget)?; - let collated = sql + let collated: Vec = sql .as_deref() .map(|sql| { declared_columns(sql) @@ -496,9 +497,16 @@ fn collated_columns( }) .unwrap_or_default(); release(budget, sql); + budget.charge_texts(&collated)?; Ok(collated) } +fn release_names(budget: &mut CatalogBudget, names: &[String]) { + for name in names { + budget.release_text(name); + } +} + fn split_names(joined: &str) -> Vec { match joined.is_empty() { true => Vec::new(), @@ -535,6 +543,9 @@ pub(crate) fn inbound_foreign_keys( let referencing = TableRef::new(Some(take(&mut row, 1)), take(&mut row, 2)); // The rows come by child, so each child's CREATE TABLE is read once. if child.as_ref().is_none_or(|(read, _)| *read != referencing) { + if let Some((_, collated)) = &child { + release_names(budget, collated); + } let collated = collated_columns(link, catalog, &referencing, budget)?; child = Some((referencing.clone(), collated)); } diff --git a/core/crates/drivers/sqlite/src/catalog/mod.rs b/core/crates/drivers/sqlite/src/catalog/mod.rs index 75238c02a4..6fbcc7aa21 100644 --- a/core/crates/drivers/sqlite/src/catalog/mod.rs +++ b/core/crates/drivers/sqlite/src/catalog/mod.rs @@ -88,30 +88,55 @@ pub(crate) fn dependents_sql(schema_sql: &str) -> String { ) } -/// Each column of another table's key that points at the table, not -/// sorted, as `FOREIGN_KEY_LIST_SQL`. The child's CREATE TABLE comes -/// from `inbound_children_sql`, once per child: it grows with the keys -/// it declares. -pub(crate) fn inbound_keys_sql(schema_sql: &str) -> String { +/// The tables whose CREATE TABLE names the table: the ones a key can +/// point at it from. SQLite matches a key's parent folding ASCII letters +/// alone, as `lower` does, and a name can be written with its quote +/// character doubled. `small` is a CREATE TABLE of at most 64 KiB whose +/// key list cannot cost SQLite more than `?3` bytes to compile: no more +/// key columns than REFERENCES and commas, each with three names no +/// longer than the text and 512 bytes of opcodes. +fn referring_tables(schema_sql: &str) -> String { format!( - "SELECT m.name AS child, f.id AS id, f.seq AS seq, f.\"from\" AS \"from\", f.\"to\" AS \"to\" \ - FROM {schema_sql}.sqlite_schema m, pragma_foreign_key_list(m.name, ?2) f \ - WHERE m.type = 'table' AND f.\"table\" = ?1 COLLATE NOCASE" + "WITH named AS (SELECT rowid, name, sql, length(CAST(sql AS BLOB)) AS bytes, \ + (length(sql) - length(replace(lower(sql), 'references', ''))) / 10 \ + + length(sql) - length(replace(sql, ',', '')) AS key_columns \ + FROM {schema_sql}.sqlite_schema WHERE type = 'table' AND sql IS NOT NULL AND ( \ + instr(lower(sql), lower(?1)) > 0 \ + OR instr(lower(sql), lower(replace(?1, '\"', '\"\"'))) > 0 \ + OR instr(lower(sql), lower(replace(?1, '`', '``'))) > 0 \ + OR instr(lower(sql), lower(replace(?1, '''', ''''''))) > 0)), \ + referring AS (SELECT rowid, name, sql, \ + bytes <= 65536 AND (key_columns + 1) * (3 * bytes + 512) <= ?3 AS small FROM named)" ) } -/// The tables with a key pointing at the table, each once. `child_keys` -/// counts every key of the child, wherever it points, since the ids -/// number them all. -pub(crate) fn inbound_children_sql(schema_sql: &str) -> String { +/// Each key column of the small referring tables that points at the +/// table, with the child's CREATE TABLE on each row, not sorted: a sort +/// holds every row before the first, and a child's rows come together. +pub(crate) fn small_children_keys_sql(schema_sql: &str) -> String { format!( - "SELECT m.name AS child, m.sql AS sql, \ - (SELECT count(DISTINCT k.id) FROM pragma_foreign_key_list(m.name, ?2) k) AS child_keys \ - FROM {schema_sql}.sqlite_schema m WHERE m.type = 'table' AND EXISTS \ - (SELECT 1 FROM pragma_foreign_key_list(m.name, ?2) f WHERE f.\"table\" = ?1 COLLATE NOCASE)" + "{} SELECT m.name AS child, m.sql AS sql, f.id AS id, f.seq AS seq, f.\"from\" AS \"from\", \ + f.\"to\" AS \"to\", (SELECT count(DISTINCT k.id) FROM pragma_foreign_key_list(m.name, ?2) k) AS child_keys \ + FROM referring m, pragma_foreign_key_list(m.name, ?2) f \ + WHERE m.small AND f.\"table\" = ?1 COLLATE NOCASE", + referring_tables(schema_sql) ) } +/// The other referring tables, read one at a time. +pub(crate) fn large_children_sql(schema_sql: &str) -> String { + format!( + "{} SELECT rowid, name FROM referring WHERE NOT small ORDER BY name", + referring_tables(schema_sql) + ) +} + +/// One schema row by rowid, which SQLite finds without reading the +/// others, as a lookup by name would for each of thousands of children. +pub(crate) fn schema_row_sql(schema_sql: &str) -> String { + format!("SELECT name, sql FROM {schema_sql}.sqlite_schema WHERE rowid = ?1") +} + /// `sqlite_stat1` exists only once ANALYZE has run, so its presence is /// checked before it is read. pub(crate) fn has_table_sql(schema_sql: &str) -> String { diff --git a/core/crates/drivers/sqlite/src/ddl/create_table_source.rs b/core/crates/drivers/sqlite/src/ddl/create_table_source.rs index abbc545092..90f30618c5 100644 --- a/core/crates/drivers/sqlite/src/ddl/create_table_source.rs +++ b/core/crates/drivers/sqlite/src/ddl/create_table_source.rs @@ -352,6 +352,27 @@ fn comparison(source: &CreateTableSource<'_>, declared: &SourceColumn) -> crate: crate::catalog::Comparison::of(&type_text, collation.as_deref()) } +// SQLite compiles `PRAGMA foreign_key_list` with one row per key column in its program, each with its own copy of +// the parent's name, the column's and the parent column's: a column carrying REFERENCES thousands of times costs its +// name thousands of times before the first row. `ROW` covers a row's opcodes and its action names. Text that does +// not read is costed as if each REFERENCES in it named the whole text. +pub(crate) fn foreign_key_list_bytes(sql: &str) -> usize { + const ROW: usize = 512; + let Ok(source) = CreateTableSource::read(sql) else { + let references = sql.to_ascii_lowercase().matches("references").count(); + return references.saturating_mul(sql.len().saturating_add(ROW)); + }; + source + .foreign_keys() + .iter() + .flat_map(|key| { + key.columns.iter().enumerate().map(|(position, column)| { + ROW + key.target.table.len() + column.len() + key.target.columns.get(position).map_or(0, String::len) + }) + }) + .fold(0, usize::saturating_add) +} + /// A type name as SQLite keeps it, which is what its affinity and the /// pragmas come from: a quoted name loses its quotes and whatever /// follows them, as `sqlite3Dequote` does. @@ -967,6 +988,24 @@ mod tests { assert_eq!(DeclaredComparisons::read("CREATE TABLE t (a 'unclosed)"), None); } + #[test] + fn the_key_list_costs_each_key_column_its_names() { + let column = "c".repeat(1_000); + let repeated = format!("CREATE TABLE t (\"{column}\" INTEGER{})", " REFERENCES p".repeat(100)); + + assert_eq!(foreign_key_list_bytes("CREATE TABLE t (a INTEGER)"), 0); + assert_eq!( + foreign_key_list_bytes("CREATE TABLE t (a, b, FOREIGN KEY (a, b) REFERENCES pp (x, yy))"), + (512 + 2 + 1 + 1) + (512 + 2 + 1 + 2) + ); + assert_eq!(foreign_key_list_bytes(&repeated), 100 * (512 + 1 + 1_000)); + assert_eq!( + foreign_key_list_bytes("CREATE TABLE t (a REFERENCES p, b 'unclosed)"), + "CREATE TABLE t (a REFERENCES p, b 'unclosed)".len() + 512, + "text that does not read" + ); + } + #[test] fn identifiers_in_every_quote_style_read_as_their_names() { let source = read("CREATE TABLE \"we\"\"ird\" (\"a\"\"b\" TEXT, `c``d` INT, [e f] REAL, 'g' BLOB, plain)"); diff --git a/core/crates/drivers/sqlite/src/ddl/mod.rs b/core/crates/drivers/sqlite/src/ddl/mod.rs index ce151b3b45..9ca9adc879 100644 --- a/core/crates/drivers/sqlite/src/ddl/mod.rs +++ b/core/crates/drivers/sqlite/src/ddl/mod.rs @@ -24,7 +24,9 @@ use tablepro_core::statement::{Statement, StatementBuilder}; use tablepro_core::{ForeignKeyInfo, Value}; use self::create_table_source::CreateTableSource; -pub(crate) use self::create_table_source::{DeclaredComparisons, declared_descending_key, declared_foreign_key_names}; +pub(crate) use self::create_table_source::{ + DeclaredComparisons, declared_descending_key, declared_foreign_key_names, foreign_key_list_bytes, +}; use self::declared_shape::readable; use crate::catalog::MAIN_SCHEMA; use crate::dialect::SqliteDialect; diff --git a/core/crates/drivers/sqlite/src/foreign_key_verification.rs b/core/crates/drivers/sqlite/src/foreign_key_verification.rs index 7451a26f28..d2d6053413 100644 --- a/core/crates/drivers/sqlite/src/foreign_key_verification.rs +++ b/core/crates/drivers/sqlite/src/foreign_key_verification.rs @@ -14,24 +14,37 @@ use tablepro_core::{DriverError, ServerCode, ServerDiagnostics}; use crate::errors::{SQLITE_CONSTRAINT, SQLITE_CONSTRAINT_FOREIGNKEY}; -// The rows of `foreign_key_check`, in its order. A file can hold millions of rows that break one key, so each row -// is the key's index and its rowid rather than its own copy of the names. A rowid is None for a WITHOUT ROWID child -// table, which has no rowid to name the row by, and for one whose rows the batch renumbers. +// The rows of `foreign_key_check` for each child table checked, in its order. #[derive(Debug, Clone, Default, PartialEq, Eq)] pub(crate) struct Violations { + tables: Vec, +} + +// A file can hold millions of rows that break thousands of keys of one table with a long name, so the table's name is +// held once, each key once, and each row as the index of the key it breaks and its rowid. A rowid is None for a +// WITHOUT ROWID child table, which has no rowid to name the row by, and for one whose rows the batch renumbers. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct TableViolations { + table: String, keys: Vec, rows: Vec<(usize, Option)>, } -// A foreign key of one child table that some row breaks. #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) struct BrokenKey { - pub table: String, pub parent: String, pub key: ViolatedKey, } -impl Violations { +impl TableViolations { + pub(crate) fn new(table: String) -> Self { + Self { + table, + keys: Vec::new(), + rows: Vec::new(), + } + } + pub(crate) fn add_key(&mut self, key: BrokenKey) -> usize { self.keys.push(key); self.keys.len() - 1 @@ -40,12 +53,17 @@ impl Violations { pub(crate) fn add_row(&mut self, key: usize, rowid: Option) { self.rows.push((key, rowid)); } +} + +impl Violations { + pub(crate) fn push(&mut self, table: TableViolations) { + if !table.rows.is_empty() { + self.tables.push(table); + } + } pub(crate) fn extend(&mut self, other: Violations) { - let offset = self.keys.len(); - self.keys.extend(other.keys); - self.rows - .extend(other.rows.into_iter().map(|(key, rowid)| (key + offset, rowid))); + self.tables.extend(other.tables); } } @@ -95,19 +113,25 @@ impl Renames { /// itself. pub(crate) fn apply(&self, baseline: Violations) -> Violations { Violations { - keys: baseline - .keys + tables: baseline + .tables .into_iter() - .map(|broken| BrokenKey { - key: ViolatedKey { - columns: self.column_names(&broken.table, broken.key.columns), - parent_columns: self.column_names(&broken.parent, broken.key.parent_columns), - }, - table: self.table_name(broken.table), - parent: self.table_name(broken.parent), + .map(|checked| TableViolations { + keys: checked + .keys + .into_iter() + .map(|broken| BrokenKey { + key: ViolatedKey { + columns: self.column_names(&checked.table, broken.key.columns), + parent_columns: self.column_names(&broken.parent, broken.key.parent_columns), + }, + parent: self.table_name(broken.parent), + }) + .collect(), + table: self.table_name(checked.table), + rows: checked.rows, }) .collect(), - rows: baseline.rows, } } @@ -130,59 +154,82 @@ impl Renames { /// multisets. Rows without a rowid are all alike, so for them it is the /// count per child, parent and key that compares. pub(crate) fn introduced(before: &Violations, after: &Violations) -> Vec { - let mut ids: HashMap = HashMap::new(); - let mut id_of = |broken: &BrokenKey| { - let next = ids.len(); - *ids.entry(key(broken)).or_insert(next) - }; - let before_ids: Vec = before.keys.iter().map(&mut id_of).collect(); - let after_ids: Vec = after.keys.iter().map(&mut id_of).collect(); + let mut ids = KeyIds::default(); let mut remaining: HashMap<(usize, Option), usize> = HashMap::new(); - for (key, rowid) in &before.rows { - if let Some(id) = before_ids.get(*key) { - *remaining.entry((*id, *rowid)).or_default() += 1; + for checked in &before.tables { + let (_, keys) = ids.of(checked); + for (key, rowid) in &checked.rows { + if let Some(id) = keys.get(*key) { + *remaining.entry((*id, *rowid)).or_default() += 1; + } } } let mut introduced: Vec = Vec::new(); - let mut groups: HashMap<(String, String), usize> = HashMap::new(); - for (key, rowid) in &after.rows { - let (Some(id), Some(broken)) = (after_ids.get(*key), after.keys.get(*key)) else { - continue; - }; - if let Some(left) = remaining.get_mut(&(*id, *rowid)) - && *left > 0 - { - *left -= 1; - continue; - } - let group = (broken.table.to_lowercase(), broken.parent.to_lowercase()); - match groups.get(&group).and_then(|index| introduced.get_mut(*index)) { - Some(group) => group.count += 1, - None => { - groups.insert(group, introduced.len()); - introduced.push(IntroducedViolations { - table: broken.table.clone(), - parent: broken.parent.clone(), - count: 1, - }); + let mut groups: HashMap<(usize, String), usize> = HashMap::new(); + for checked in &after.tables { + let (table, keys) = ids.of(checked); + let parents: Vec = checked.keys.iter().map(|broken| broken.parent.to_lowercase()).collect(); + for (key, rowid) in &checked.rows { + let (Some(id), Some(broken), Some(parent)) = (keys.get(*key), checked.keys.get(*key), parents.get(*key)) + else { + continue; + }; + if let Some(left) = remaining.get_mut(&(*id, *rowid)) + && *left > 0 + { + *left -= 1; + continue; + } + match groups + .get(&(table, parent.clone())) + .and_then(|index| introduced.get_mut(*index)) + { + Some(group) => group.count += 1, + None => { + groups.insert((table, parent.clone()), introduced.len()); + introduced.push(IntroducedViolations { + table: checked.table.clone(), + parent: broken.parent.clone(), + count: 1, + }); + } } } } introduced } -type ViolationKey = (String, String, Vec, Vec); - -/// SQLite names tables and columns case-insensitively, and so does the -/// comparison. -fn key(broken: &BrokenKey) -> ViolationKey { - let folded = |names: &[String]| names.iter().map(|name| name.to_lowercase()).collect(); - ( - broken.table.to_lowercase(), - broken.parent.to_lowercase(), - folded(&broken.key.columns), - folded(&broken.key.parent_columns), - ) +// SQLite names tables and columns case-insensitively, and so does the comparison. A table's name is folded once, +// whatever number of keys it breaks. +#[derive(Default)] +struct KeyIds { + tables: HashMap, + keys: HashMap<(usize, String, Vec, Vec), usize>, +} + +impl KeyIds { + fn of(&mut self, checked: &TableViolations) -> (usize, Vec) { + let next = self.tables.len(); + let table = *self.tables.entry(checked.table.to_lowercase()).or_insert(next); + let folded = |names: &[String]| names.iter().map(|name| name.to_lowercase()).collect(); + let keys = checked + .keys + .iter() + .map(|broken| { + let next = self.keys.len(); + *self + .keys + .entry(( + table, + broken.parent.to_lowercase(), + folded(&broken.key.columns), + folded(&broken.key.parent_columns), + )) + .or_insert(next) + }) + .collect(); + (table, keys) + } } /// The error a batch that introduced violations is rolled back with, @@ -226,20 +273,31 @@ mod tests { (table, rowid, parent, columns, parent_columns) } - // One key per row, as two keys of the same columns compare alike. + // One key per row, as two keys of the same columns compare alike, and one table per name in the order it comes. fn found(rows: &[Row<'_>]) -> Violations { let owned = |names: &[&str]| names.iter().map(|name| (*name).to_owned()).collect(); - let mut violations = Violations::default(); + let mut tables: Vec = Vec::new(); for (table, rowid, parent, columns, parent_columns) in rows { - let key = violations.add_key(BrokenKey { - table: (*table).to_owned(), + let position = match tables.iter().position(|checked| checked.table == *table) { + Some(position) => position, + None => { + tables.push(TableViolations::new((*table).to_owned())); + tables.len() - 1 + } + }; + let checked = &mut tables[position]; + let key = checked.add_key(BrokenKey { parent: (*parent).to_owned(), key: ViolatedKey { columns: owned(columns), parent_columns: owned(parent_columns), }, }); - violations.add_row(key, *rowid); + checked.add_row(key, *rowid); + } + let mut violations = Violations::default(); + for checked in tables { + violations.push(checked); } violations } @@ -329,24 +387,26 @@ mod tests { #[test] fn rows_of_one_key_compare_one_for_one_and_name_their_group_in_order() { - let mut before = Violations::default(); - let broken = |table: &str, parent: &str| BrokenKey { - table: table.to_owned(), + let broken = |parent: &str| BrokenKey { parent: parent.to_owned(), key: ViolatedKey { columns: vec!["a".to_owned()], parent_columns: Vec::new(), }, }; - let key = before.add_key(broken("t", "p")); + let mut checked = TableViolations::new("t".to_owned()); + let key = checked.add_key(broken("p")); for rowid in 0..1_000 { - before.add_row(key, Some(rowid)); + checked.add_row(key, Some(rowid)); } - let mut after = before.clone(); - let other = after.add_key(broken("t", "q")); - after.add_row(other, Some(5)); - after.add_row(key, Some(1_000)); - after.add_row(other, Some(6)); + let mut before = Violations::default(); + before.push(checked.clone()); + let other = checked.add_key(broken("q")); + checked.add_row(other, Some(5)); + checked.add_row(key, Some(1_000)); + checked.add_row(other, Some(6)); + let mut after = Violations::default(); + after.push(checked); assert_eq!( introduced(&before, &after), diff --git a/core/crates/drivers/sqlite/src/metadata.rs b/core/crates/drivers/sqlite/src/metadata.rs index 43200758a0..b51338725d 100644 --- a/core/crates/drivers/sqlite/src/metadata.rs +++ b/core/crates/drivers/sqlite/src/metadata.rs @@ -3,7 +3,7 @@ //! `catalog` owns the SQL and what the rows mean. This runs it. use std::collections::btree_map::Entry; -use std::collections::{BTreeMap, HashMap, hash_map}; +use std::collections::{BTreeMap, HashMap, HashSet, hash_map}; use futures::TryStreamExt; use sqlx::sqlite::{SqliteConnection as SqlxConnection, SqliteRow}; @@ -18,13 +18,15 @@ use tablepro_core::meta::{ RowIdentity, SchemaInfo, TableEstimate, TableRef, }; use tablepro_core::sql_syntax::{SqlGrammar, qualify_created_name, terminated}; -use tablepro_core::{CatalogBudget, DriverError, ForeignKeyInfo, ServerCode, ServerDiagnostics, UnsupportedFeature}; +use tablepro_core::{ + ByteLimit, CatalogBudget, DriverError, ForeignKeyInfo, ServerCode, ServerDiagnostics, UnsupportedFeature, +}; use crate::catalog::{ self, InboundKeyColumn, ListedKeyColumn, MAIN_SCHEMA, SchemaRow, SqliteColumnRow, TableFacts, foreign_key_names, generated_expressions, inbound_foreign_keys, select_dependents, }; -use crate::ddl::{DeclaredComparisons, declared_descending_key, declared_foreign_key_names}; +use crate::ddl::{DeclaredComparisons, declared_descending_key, declared_foreign_key_names, foreign_key_list_bytes}; use crate::errors::{SQLITE_ERROR, map_error}; use crate::index_sql::read_index_sql; @@ -142,11 +144,8 @@ pub(crate) async fn fetch_foreign_keys( table: &TableRef, ) -> Result, DriverError> { let mut budget = CatalogBudget::default(); - let keys = foreign_key_rows(connection, table, &mut budget).await?; - let create = match keys.is_empty() { - true => None, - false => create_statement(connection, table, &mut budget).await?, - }; + let create = create_statement(connection, table, &mut budget).await?; + let keys = foreign_key_rows(connection, table, create.as_deref(), &mut budget).await?; resolve_foreign_keys(connection, table, keys, create.as_deref(), &mut budget).await } @@ -155,11 +154,11 @@ pub(crate) async fn fetch_outbound_foreign_keys( table: &TableRef, ) -> Result, DriverError> { let mut budget = CatalogBudget::default(); - let keys = foreign_key_rows(connection, table, &mut budget).await?; + let create = create_statement(connection, table, &mut budget).await?; + let keys = foreign_key_rows(connection, table, create.as_deref(), &mut budget).await?; if keys.is_empty() { return Ok(Vec::new()); } - let create = create_statement(connection, table, &mut budget).await?; let keys = resolve_foreign_keys(connection, table, keys, create.as_deref(), &mut budget).await?; let child = create.as_deref().map(DeclaredComparisons::read); release(&mut budget, create); @@ -212,8 +211,10 @@ pub(crate) async fn fetch_outbound_foreign_keys( async fn foreign_key_rows( connection: &mut SqlxConnection, table: &TableRef, + create_statement: Option<&str>, budget: &mut CatalogBudget, ) -> Result, DriverError> { + budget.fits(create_statement.map_or(0, foreign_key_list_bytes))?; let schema = schema_of(table); let mut rows = sqlx::query(catalog::FOREIGN_KEY_LIST_SQL) .bind(table.name.as_str()) @@ -424,10 +425,6 @@ async fn inbound_keys( shape: &TableShape, budget: &mut CatalogBudget, ) -> Result, DriverError> { - let mut children = inbound_columns(connection, table, budget).await?; - if children.is_empty() { - return Ok(Vec::new()); - } let parent_sql = create_statement(connection, table, budget).await?; let parent = Parent { table, @@ -436,58 +433,115 @@ async fn inbound_keys( primary_key: shape.primary_key(), }; release(budget, parent_sql); - let mut keys = Vec::new(); + let mut keys = small_children_keys(connection, &parent, budget).await?; let schema_sql = quote(schema_of(table)); - let mut rows = sqlx::query(AssertSqlSafe(catalog::inbound_children_sql(&schema_sql))) - .bind(table.name.as_str()) - .bind(schema_of(table)) + let mut large = Vec::new(); + { + let mut rows = sqlx::query_as::<_, (i64, String)>(AssertSqlSafe(catalog::large_children_sql(&schema_sql))) + .bind(table.name.as_str()) + .bind(schema_of(table)) + .bind(i64::try_from(ByteLimit::CATALOG_DEFAULT.get()).unwrap_or(i64::MAX)) + .fetch(&mut *connection); + while let Some((rowid, child)) = rows.try_next().await.map_err(map_error)? { + large.push((rowid, budget.keep(child)?)); + } + } + // One child at a time: its CREATE TABLE is let go before the next is read. + for (rowid, child) in large { + let row: Option<(String, Option)> = sqlx::query_as(AssertSqlSafe(catalog::schema_row_sql(&schema_sql))) + .bind(rowid) + .fetch_optional(&mut *connection) + .await + .map_err(map_error)?; + // A row another connection gave to another table since is not this child's. + let sql = budget.keep_optional(row.filter(|(name, _)| *name == child).and_then(|(_, sql)| sql))?; + budget.fits(sql.as_deref().map_or(0, foreign_key_list_bytes))?; + let (columns, count) = key_columns_into(connection, &child, table, budget).await?; + match columns.is_empty() { + true => { + release(budget, sql); + budget.release_text(&child); + } + false => keys.extend(parent.keys_of(child, columns, sql, count, budget)?), + } + } + keys.sort_by(|left, right| left.table.name.cmp(&right.table.name)); + Ok(keys) +} + +/// The keys of the small referring tables, in one read: a child's rows +/// come together, each with its CREATE TABLE, kept from its first row. +async fn small_children_keys( + connection: &mut SqlxConnection, + parent: &Parent<'_>, + budget: &mut CatalogBudget, +) -> Result, DriverError> { + let schema_sql = quote(schema_of(parent.table)); + let mut rows = sqlx::query(AssertSqlSafe(catalog::small_children_keys_sql(&schema_sql))) + .bind(parent.table.name.as_str()) + .bind(schema_of(parent.table)) + .bind(i64::try_from(ByteLimit::CATALOG_DEFAULT.get()).unwrap_or(i64::MAX)) .fetch(&mut *connection); + let mut keys = Vec::new(); + let mut child: Option<(String, Option, usize, Vec)> = None; while let Some(row) = rows.try_next().await.map_err(map_error)? { let name: &str = row.try_get("child").map_err(map_error)?; - let Some((child, columns)) = children.remove_entry(name) else { - continue; - }; - let sql = budget.keep_optional(row.try_get("sql").map_err(map_error)?)?; - let count = usize::try_from(row.try_get::("child_keys").map_err(map_error)?).unwrap_or(0); - keys.extend(parent.keys_of(child, columns, sql, count, budget)?); + if child.as_ref().is_none_or(|(read, ..)| read != name) { + if let Some((read, sql, count, columns)) = child.take() { + keys.extend(parent.keys_of(read, columns, sql, count, budget)?); + } + child = Some(( + budget.keep(name.to_owned())?, + budget.keep_optional(row.try_get("sql").map_err(map_error)?)?, + usize::try_from(row.try_get::("child_keys").map_err(map_error)?).unwrap_or(0), + Vec::new(), + )); + } + if let Some((_, _, _, columns)) = &mut child { + columns.push(InboundKeyColumn { + id: row.try_get("id").map_err(map_error)?, + seq: row.try_get("seq").map_err(map_error)?, + from: budget.keep(text(&row, "from")?)?, + to: budget.keep_optional(row.try_get("to").map_err(map_error)?)?, + }); + } } - // A child the second read no longer saw keeps made-up names. - for (child, columns) in children { - keys.extend(parent.keys_of(child, columns, None, 0, budget)?); + if let Some((read, sql, count, columns)) = child { + keys.extend(parent.keys_of(read, columns, sql, count, budget)?); } - keys.sort_by(|left, right| left.table.name.cmp(&right.table.name)); Ok(keys) } -/// Each child's key columns, its name read once rather than kept on -/// every row. -async fn inbound_columns( +/// The columns of the child's keys that point at `parent`, and how many +/// keys the child has wherever they point, since the ids number them +/// all. SQLite compares a key's parent folding ASCII letters alone. +async fn key_columns_into( connection: &mut SqlxConnection, - table: &TableRef, + child: &str, + parent: &TableRef, budget: &mut CatalogBudget, -) -> Result>, DriverError> { - let schema_sql = quote(schema_of(table)); - let mut rows = sqlx::query(AssertSqlSafe(catalog::inbound_keys_sql(&schema_sql))) - .bind(table.name.as_str()) - .bind(schema_of(table)) +) -> Result<(Vec, usize), DriverError> { + let mut rows = sqlx::query(catalog::FOREIGN_KEY_LIST_SQL) + .bind(child) + .bind(schema_of(parent)) .fetch(&mut *connection); - let mut children: HashMap> = HashMap::new(); + let mut ids = HashSet::new(); + let mut columns = Vec::new(); while let Some(row) = rows.try_next().await.map_err(map_error)? { - let column = InboundKeyColumn { - id: row.try_get("id").map_err(map_error)?, + let id: i64 = row.try_get("id").map_err(map_error)?; + ids.insert(id); + let target: &str = row.try_get("table").map_err(map_error)?; + if !target.eq_ignore_ascii_case(&parent.name) { + continue; + } + columns.push(InboundKeyColumn { + id, seq: row.try_get("seq").map_err(map_error)?, from: budget.keep(text(&row, "from")?)?, to: budget.keep_optional(row.try_get("to").map_err(map_error)?)?, - }; - let child: &str = row.try_get("child").map_err(map_error)?; - match children.get_mut(child) { - Some(columns) => columns.push(column), - None => { - children.insert(budget.keep(child.to_owned())?, vec![column]); - } - } + }); } - Ok(children) + Ok((columns, ids.len())) } /// The table the inbound keys point at, read once for all its children. @@ -681,7 +735,7 @@ pub(crate) async fn fetch_table_definition( columns: key_columns, }); let indexes = indexes(connection, table, &shape, &mut budget).await?; - let foreign_keys = foreign_key_rows(connection, table, &mut budget).await?; + let foreign_keys = foreign_key_rows(connection, table, create_statement.as_deref(), &mut budget).await?; let foreign_keys = resolve_foreign_keys( connection, table, diff --git a/core/crates/drivers/sqlite/src/write_batch.rs b/core/crates/drivers/sqlite/src/write_batch.rs index 7ad777b908..3d47e98eeb 100644 --- a/core/crates/drivers/sqlite/src/write_batch.rs +++ b/core/crates/drivers/sqlite/src/write_batch.rs @@ -24,11 +24,14 @@ use tablepro_core::{ use crate::autocommit::is_autocommit; use crate::catalog::{ - DECLARED_TABLE_NAME_SQL, FOREIGN_KEY_CHECK_SQL, FOREIGN_KEY_LIST_SQL, ListedKeyColumn, MAIN_SCHEMA, + DECLARED_TABLE_NAME_SQL, FOREIGN_KEY_CHECK_SQL, FOREIGN_KEY_LIST_SQL, ListedKeyColumn, MAIN_SCHEMA, object_sql, }; +use crate::ddl::foreign_key_list_bytes; use crate::encode::bind_all; use crate::errors::{SQLITE_IOERR, is_interrupt, map_error, primary_of}; -use crate::foreign_key_verification::{BrokenKey, Renames, ViolatedKey, Violations, introduced, violation_error}; +use crate::foreign_key_verification::{ + BrokenKey, Renames, TableViolations, ViolatedKey, Violations, introduced, violation_error, +}; use crate::interrupt::InterruptFlag; use crate::result_stream::{count, reports_changes}; use crate::sqlite_pooled::SqlitePooled; @@ -361,22 +364,19 @@ async fn violations( .fetch_optional(&mut *connection) .await .map_err(map_error)?; - let name = declared.unwrap_or_else(|| table.name.clone()); + let mut found = TableViolations::new(declared.unwrap_or_else(|| table.name.clone())); let keys = foreign_keys(connection, table, schema).await?; - let mut found = Violations::default(); let mut positions: HashMap = HashMap::new(); for (fkid, rowid) in rows { let position = *positions.entry(fkid).or_insert_with(|| { let (parent, key) = keys.get(&fkid).cloned().unwrap_or_default(); - found.add_key(BrokenKey { - table: name.clone(), - parent, - key, - }) + found.add_key(BrokenKey { parent, key }) }); found.add_row(position, rowid.filter(|_| !rowids_renumbered)); } - Ok(found) + let mut violations = Violations::default(); + violations.push(found); + Ok(violations) } /// The table's foreign keys, with the parent each names, by the id @@ -387,6 +387,17 @@ async fn foreign_keys( schema: &str, ) -> Result, DriverError> { let mut budget = CatalogBudget::default(); + let schema_sql = format!("\"{}\"", schema.replace('"', "\"\"")); + let create: Option<(String, Option)> = sqlx::query_as(AssertSqlSafe(object_sql(&schema_sql))) + .bind(table.name.as_str()) + .fetch_optional(&mut *connection) + .await + .map_err(map_error)?; + budget.fits( + create + .and_then(|(_, sql)| sql) + .map_or(0, |sql| foreign_key_list_bytes(&sql)), + )?; let mut rows = sqlx::query(FOREIGN_KEY_LIST_SQL) .bind(table.name.as_str()) .bind(schema) diff --git a/core/crates/drivers/sqlite/tests/catalog_memory.rs b/core/crates/drivers/sqlite/tests/catalog_memory.rs index 6072474fc0..d442b1a0d0 100644 --- a/core/crates/drivers/sqlite/tests/catalog_memory.rs +++ b/core/crates/drivers/sqlite/tests/catalog_memory.rs @@ -27,6 +27,7 @@ const BOUND: usize = 2 * ByteLimit::CATALOG_DEFAULT.get() as usize; const LONG: usize = 64 * 1024; const KEYS: usize = 4_000; const BROKEN_ROWS: usize = 20_000; +const BULKY_CHILDREN: usize = 60; fn quoted(name: &str) -> String { format!("\"{name}\"") @@ -36,7 +37,9 @@ fn quoted(name: &str) -> String { // - one column carrying REFERENCES 4,000 times, which the pragma names on every key; // - a child with a 64 KiB name and 4,000 keys, each of which names the child; // - 4,000 keys naming no columns of a parent whose primary key is 64 KiB of names, each a copy of it; -// - 20,000 rows of a table with a 64 KiB name that break its key, which a check named on every row. +// - 20,000 rows of a table with a 64 KiB name that break its key, which a check named on every row; +// - one row of a table with a 64 KiB name that breaks 4,000 keys, which the check named on every key; +// - 60 children of one parent, each CREATE TABLE 256 KiB long, which SQLite queued 50 at a time. fn crafted_schema() -> Vec { let column = "c".repeat(LONG); let child = "n".repeat(LONG); @@ -76,9 +79,59 @@ fn crafted_schema() -> Vec { INSERT INTO {} SELECT i FROM n", quoted(&broken()) ), + format!( + "CREATE TABLE {} (a INTEGER{})", + quoted(&broken_keys()), + " REFERENCES named_parent".repeat(KEYS) + ), + format!("INSERT INTO {} VALUES (1)", quoted(&broken_keys())), + "CREATE TABLE bulky_parent (id INTEGER PRIMARY KEY)".to_owned(), ] + .into_iter() + .chain((0..BULKY_CHILDREN).map(|child| { + format!( + "CREATE TABLE bulky_{child:02} (a INTEGER REFERENCES bulky_parent /* {} */)", + "x".repeat(256 * 1024) + ) + })) + .collect() +} + +fn broken_keys() -> String { + "k".repeat(LONG) +} + +// A rebuild checks the keys its table breaks before and after; the rows that broke them already are not its own. +fn checked_save(table: &str) -> WriteBatch { + let mut builder = StatementBuilder::new(); + builder.push_sql(&format!("UPDATE {} SET a = a WHERE 0", quoted(table))); + WriteBatch::new( + WriteKind::SchemaChanges, + TableRef::unqualified(table), + vec![GuardedStatement::unguarded(builder.finish())], + ForeignKeyMode::SuspendedThenVerified { + checks: vec![ForeignKeyCheck::same_table(TableRef::unqualified(table))], + }, + ) +} + +// How far the heap and SQLite's own memory, which the heap counter does not see, rose while `read` ran. SQLite's is 0 +// where the library keeps no count. +async fn measured(read: impl std::future::Future) -> (T, usize, i64) { + // SAFETY: these calls only read and reset SQLite's own counters. + let start = unsafe { + libsqlite3_sys::sqlite3_memory_highwater(1); + libsqlite3_sys::sqlite3_memory_used() + }; + let (value, heap) = peak_of(read).await; + // SAFETY: as above. + let native = unsafe { libsqlite3_sys::sqlite3_memory_highwater(1) } - start; + (value, heap, native) } +// SQLite compiled a crafted table's key list, a copy of its long name for each key, into gigabytes. +const NATIVE_BOUND: i64 = 32 << 20; + fn broken() -> String { "b".repeat(LONG) } @@ -126,10 +179,10 @@ async fn a_crafted_catalog_stops_at_the_budget() { let table = |name: &str| TableRef::unqualified(name); let child = table(&"n".repeat(LONG)); - let (objects, peak) = peak_of(connection.list_objects(&ObjectScope::DefaultSchema, call())).await; - assert_eq!(objects.unwrap().len(), 7); - assert!(peak < BOUND, "listing the tables held {peak} bytes"); - let (columns, _) = peak_of(connection.fetch_columns(&table("repeated"), call())).await; + let (objects, heap, _) = measured(connection.list_objects(&ObjectScope::DefaultSchema, call())).await; + assert_eq!(objects.unwrap().len(), 9 + BULKY_CHILDREN); + assert!(heap < BOUND, "listing the tables held {heap} bytes"); + let (columns, _, _) = measured(connection.fetch_columns(&table("repeated"), call())).await; assert_eq!(columns.unwrap().len(), 1, "a long name alone is read"); for (name, table) in [ @@ -137,38 +190,55 @@ async fn a_crafted_catalog_stops_at_the_budget() { ("the long child", child.clone()), ("pointer", table("pointer")), ] { - let (read, peak) = peak_of(connection.fetch_foreign_keys(&table, call())).await; + let (read, heap, native) = measured(connection.fetch_foreign_keys(&table, call())).await; too_large(&format!("keys of {name}"), read); - assert!(peak < BOUND, "keys of {name} held {peak} bytes"); - let (read, peak) = peak_of(connection.fetch_outbound_foreign_keys(&table, call())).await; + assert!( + heap < BOUND && native < NATIVE_BOUND, + "keys of {name}: {heap}, {native}" + ); + let (read, heap, native) = measured(connection.fetch_outbound_foreign_keys(&table, call())).await; too_large(&format!("outbound keys of {name}"), read); - assert!(peak < BOUND, "outbound keys of {name} held {peak} bytes"); - let (read, peak) = peak_of(connection.fetch_table_definition(&table, call())).await; + assert!( + heap < BOUND && native < NATIVE_BOUND, + "outbound keys of {name}: {heap}, {native}" + ); + let (read, heap, native) = measured(connection.fetch_table_definition(&table, call())).await; too_large(&format!("definition of {name}"), read); - assert!(peak < BOUND, "definition of {name} held {peak} bytes"); + assert!( + heap < BOUND && native < NATIVE_BOUND, + "definition of {name}: {heap}, {native}" + ); } for parent in ["repeated_parent", "named_parent", "wide_key"] { - let (read, peak) = peak_of(connection.fetch_inbound_foreign_keys(&table(parent), call())).await; + let (read, heap, native) = measured(connection.fetch_inbound_foreign_keys(&table(parent), call())).await; too_large(&format!("keys into {parent}"), read); - assert!(peak < BOUND, "keys into {parent} held {peak} bytes"); - let (read, peak) = peak_of(connection.fetch_table_definition(&table(parent), call())).await; + assert!( + heap < BOUND && native < NATIVE_BOUND, + "keys into {parent}: {heap}, {native}" + ); + let (read, heap, native) = measured(connection.fetch_table_definition(&table(parent), call())).await; too_large(&format!("definition of {parent}"), read); - assert!(peak < BOUND, "definition of {parent} held {peak} bytes"); + assert!( + heap < BOUND && native < NATIVE_BOUND, + "definition of {parent}: {heap}, {native}" + ); } - // A rebuild checks the keys its table breaks before and after; the rows that broke them already are not its own. - let mut builder = StatementBuilder::new(); - builder.push_sql(&format!("UPDATE {} SET a = a WHERE 0", quoted(&broken()))); - let batch = WriteBatch::new( - WriteKind::SchemaChanges, - table(&broken()), - vec![GuardedStatement::unguarded(builder.finish())], - ForeignKeyMode::SuspendedThenVerified { - checks: vec![ForeignKeyCheck::same_table(table(&broken()))], - }, + for (name, table) in [("broken rows", broken()), ("broken keys", broken_keys())] { + let (written, heap, native) = measured(connection.write_batch(&checked_save(&table), call())).await; + assert!(written.is_ok(), "{name}: {written:?}"); + assert!( + heap < BOUND / 8 && native < NATIVE_BOUND, + "checking the {name}: {heap}, {native}" + ); + } + + // A child's CREATE TABLE is read alone, never 50 of them queued in SQLite's reader. + let (inbound, heap, native) = measured(connection.fetch_inbound_foreign_keys(&table("bulky_parent"), call())).await; + assert_eq!(inbound.unwrap().len(), BULKY_CHILDREN); + assert!( + heap < BOUND / 8 && native < 8 << 20, + "keys into bulky_parent: {heap}, {native}" ); - let (written, peak) = peak_of(connection.write_batch(&batch, call())).await; - assert!(written.is_ok(), "{written:?}"); - assert!(peak < BOUND / 8, "checking {BROKEN_ROWS} broken rows held {peak} bytes"); connection.close().await.unwrap(); } diff --git a/core/crates/drivers/sqlite/tests/integration/catalog.rs b/core/crates/drivers/sqlite/tests/integration/catalog.rs index 403c81e084..8987257169 100644 --- a/core/crates/drivers/sqlite/tests/integration/catalog.rs +++ b/core/crates/drivers/sqlite/tests/integration/catalog.rs @@ -401,6 +401,35 @@ async fn a_large_catalog_reads_whole() { connection.close().await.unwrap(); } +// Only tables whose CREATE TABLE names the parent are asked for their keys, so a name written with its quote doubled, +// or in other case, still finds them. +#[tokio::test] +async fn keys_into_a_table_are_found_however_its_name_is_written() { + let file = TestFile::new().await; + let connection = file.open().await; + apply( + connection.as_ref(), + &[ + "CREATE TABLE \"we\"\"ird's\" (id INTEGER PRIMARY KEY)", + "CREATE TABLE doubled (id INTEGER PRIMARY KEY, parent INTEGER REFERENCES \"WE\"\"IRD's\")", + "CREATE TABLE quoted (id INTEGER PRIMARY KEY, parent INTEGER REFERENCES 'we\"ird''s' (id))", + "CREATE TABLE ticked (id INTEGER PRIMARY KEY, parent INTEGER REFERENCES `we\"ird's`)", + "CREATE TABLE unrelated (id INTEGER PRIMARY KEY, other INTEGER REFERENCES doubled)", + ], + ) + .await + .unwrap(); + + let inbound = connection + .fetch_inbound_foreign_keys(&TableRef::unqualified("we\"ird's"), call()) + .await + .unwrap(); + + let children: Vec<&str> = inbound.iter().map(|key| key.table.name.as_str()).collect(); + assert_eq!(children, ["doubled", "quoted", "ticked"]); + connection.close().await.unwrap(); +} + #[tokio::test] async fn dependents_and_source_are_read() { let file = TestFile::new().await; diff --git a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md index b64e0b56fc..a47ffbda8a 100644 --- a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md +++ b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md @@ -5,26 +5,28 @@ ## Context -A file engine reads its catalog from the file the user opened, and the file chooses every name in it. SQLite's pragmas repeat a name on every key that uses it: a column that carries REFERENCES thousands of times comes back with its name on every key, a child's name on every key into its parent, and a key that names no parent columns took a copy of the parent's whole primary key. The keys into a table also carried the child's whole CREATE TABLE on each key column, and that text grows with the keys the child declares. DuckDB's keys into a table did the same with the child's CREATE TABLE. A file of a few hundred kilobytes made one read of a table's keys hold gigabytes, and each read held all its rows before it looked at the first. +A file engine reads its catalog from the file the user opened, and the file chooses every name in it. SQLite's pragmas repeat a name on every key that uses it: a column that carries REFERENCES thousands of times comes back with its name on every key, a child's name on every key into its parent, and a key that names no parent columns took a copy of the parent's whole primary key. SQLite also compiles `PRAGMA foreign_key_list` with a copy of those names for each key column, so a 300 KB table cost it over 300 MB before the first row, and the keys into any table ran that pragma on every table of the schema. The keys into a table carried the child's whole CREATE TABLE on each key column, in SQLite and in DuckDB, and that text grows with the keys the child declares. A file of a few hundred kilobytes made one read of a table's keys hold gigabytes. ## Decision `CatalogBudget` in `tablepro-core` bounds what one catalog read holds at `ByteLimit::CATALOG_DEFAULT`, 64,000,000 bytes, counted as `Value::retained_bytes` counts text: a string's slot and its length. Past it the read fails with `DriverError::CatalogTooLarge { limit_bytes }`, code `driver.catalog_too_large`, category `Internal`. - SQLite and DuckDB charge each text a read keeps as it arrives, and each copy made per key before it is made. A text read and dropped before the next one, as each child's CREATE TABLE, is released. -- Each text of the file is read once per call. SQLite streams catalog rows, unsorted where a sort would hold every row before the first, reads each child's CREATE TABLE once in a second query, and parses each CREATE TABLE once. DuckDB reads each child's and each parent's CREATE TABLE once. -- A save's foreign key check reads a rowid and a key id for each row that breaks a key, not the table's and parent's names. +- Each text of the file is read once per call, and SQLite's catalog rows stream, unsorted where a sort would hold every row first. +- SQLite runs `pragma_foreign_key_list` on a table only once the key list's cost, worked out from its CREATE TABLE (`foreign_key_list_bytes`), fits the budget. The keys into a table ask only the tables whose CREATE TABLE names it: those of at most 64 KiB whose key list cannot cost more than the budget in one read, the others one at a time. +- A save's foreign key check holds each checked table's name once and each row as a key index and a rowid. +- DuckDB reads each child's and each parent's CREATE TABLE once, and charges the column names it keeps from them. ## Rationale - **An error, not a cut list.** A table definition plans structure changes, and a rebuild planned from a list cut short would drop what was left out. - **64 MB.** A schema of 5,000 tables keyed to one table reads its keys in a few megabytes, and a million table names fit under it. -- **In the driver.** The driver knows which texts it copies per key, and no app has to pass a limit. +- **In the driver.** The driver knows which texts it copies per key and what SQLite compiles for them, and no app has to pass a limit. ## Consequences -- Large real schemas read as before; the keys into a table read faster, without a child's CREATE TABLE on each row. -- One value is read before it is charged. SQLite bounds a value at `SQLITE_MAX_LENGTH` and parsed the text when it opened the file. +- Large real schemas read as before, and the keys into a table read faster: 5,000 children in 0.2 s, against 1.4 s. +- One value is read before it is charged, and sqlx queues up to 50 rows ahead of the reader. SQLite bounds a value at `SQLITE_MAX_LENGTH`, and a CREATE TABLE over 64 KiB is read alone. - DuckDB holds a catalog query's whole result before the first row, and names each constraint after its table. The budget bounds what the driver keeps, not that. - Not budgeted: the schema list, reads of one value, and the SQL dumps, which read each object once. - The apps word the code: Linux in `linux/crates/app/src/ui/error_text/driver.rs`, Windows in `Resources.resw`. diff --git a/core/docs/decisions/README.md b/core/docs/decisions/README.md index a8fad9dd23..aaec84f125 100644 --- a/core/docs/decisions/README.md +++ b/core/docs/decisions/README.md @@ -36,4 +36,4 @@ The format, the statuses and how a later ADR changes an earlier one follow [the | [0026](0026-the-windows-app-signs-in-through-the-ffi.md) | The Windows app signs in to its TablePro account through the FFI | Accepted | `Core::open_account` builds the account with `AppClient::WINDOWS`, the Core's secret store for the refresh token, a `HostBrowser` callback and `MachineId::derive_machine_guid` of the registry's MachineGuid, keyed as Linux keys `/etc/machine-id`; `status()` is a snapshot and `AccountListener` only signals. | | [0027](0027-share-links.md) | Share links go through the account, with the snapshot and the secret scan in the core | Accepted; masked columns and the secret scan amended by 0028 | `Account` adds the share calls through the generated `/v2` client, `fetch_shared` without a token; problem codes stay `Refused` with their own `code()`; `build_snapshot` writes raw JSON cells by the export's lossless rules up to the plan's rows; a table page shares `build_view_sql`, its export with the filter's values written in; `secret_scan` warns; the deep link takes one 43-character base62 token. | | [0028](0028-data-masking.md) | A shared result masks its columns before it leaves the machine, with the macOS app's output | Accepted | `tablepro_core::masking` gives the macOS styles byte for byte; `SharePlan` applies the sheet's Keep, Mask or Remove per column for the snapshot, its preview and summary; the snapshot names each masked column's style and keeps to the macOS body budget; `masking::detect` suggests masks from names, values and the SQL by PLAN-MASKING 8.3. | -| [0029](0029-a-catalog-read-keeps-a-byte-budget.md) | A catalog read keeps a byte budget | Accepted | `CatalogBudget` bounds what one SQLite or DuckDB catalog read holds at 64 MB, counting each text kept and each copy made per key, and fails `DriverError::CatalogTooLarge` past it; each text of the file is read once per call, rows stream unsorted, and a save's foreign key check reads no names per row. | +| [0029](0029-a-catalog-read-keeps-a-byte-budget.md) | A catalog read keeps a byte budget | Accepted | `CatalogBudget` bounds what one SQLite or DuckDB catalog read holds at 64 MB, counting each text kept and each copy made per key, and fails `DriverError::CatalogTooLarge` past it; SQLite costs a key list before it compiles one, asks only the tables that name a parent for keys into it, and streams rows unsorted; a save's foreign key check names each table once. | From 0cf39b60821b7b18e34a85493446d89468517af6 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 01:52:19 +0700 Subject: [PATCH 4/9] docs(linux): note the catalog read bound in the changelog --- linux/changelog.d/4287.security.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 linux/changelog.d/4287.security.md diff --git a/linux/changelog.d/4287.security.md b/linux/changelog.d/4287.security.md new file mode 100644 index 0000000000..499dffe69d --- /dev/null +++ b/linux/changelog.d/4287.security.md @@ -0,0 +1 @@ +A crafted SQLite or DuckDB file no longer makes reading its tables' keys or structure fill memory. From 330fbe869ef647549320286deb0e24ed87f14a30 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 01:52:30 +0700 Subject: [PATCH 5/9] docs(linux): name the bug in the changelog fragment --- linux/changelog.d/4287.security.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/linux/changelog.d/4287.security.md b/linux/changelog.d/4287.security.md index 499dffe69d..78831b461a 100644 --- a/linux/changelog.d/4287.security.md +++ b/linux/changelog.d/4287.security.md @@ -1 +1 @@ -A crafted SQLite or DuckDB file no longer makes reading its tables' keys or structure fill memory. +Reading a table's keys or structure in a crafted SQLite or DuckDB file could take gigabytes of memory. From 119f8b5ca9677202f507f1b5163abac399602bf0 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 02:10:08 +0700 Subject: [PATCH 6/9] fix(core): bound DuckDB constraint reads and save checks, and drop the quadratic name search --- core/crates/core/src/catalog_budget.rs | 13 +- core/crates/drivers/duckdb/src/catalog/mod.rs | 57 ++++- core/crates/drivers/duckdb/src/dump.rs | 4 +- core/crates/drivers/sqlite/src/catalog/mod.rs | 45 ++-- .../sqlite/src/ddl/create_table_source.rs | 6 +- .../sqlite/src/foreign_key_verification.rs | 201 +++++++++++++----- core/crates/drivers/sqlite/src/metadata.rs | 46 +++- core/crates/drivers/sqlite/src/write_batch.rs | 45 ++-- .../drivers/sqlite/tests/catalog_memory.rs | 20 +- ...0029-a-catalog-read-keeps-a-byte-budget.md | 8 +- core/docs/decisions/README.md | 2 +- 11 files changed, 315 insertions(+), 132 deletions(-) diff --git a/core/crates/core/src/catalog_budget.rs b/core/crates/core/src/catalog_budget.rs index 36300cbf51..ab11fd7860 100644 --- a/core/crates/core/src/catalog_budget.rs +++ b/core/crates/core/src/catalog_budget.rs @@ -1,8 +1,7 @@ use crate::byte_limit::ByteLimit; use crate::error::DriverError; -// A file's catalog can repeat one long name across thousands of keys, so what a read holds grows with both; this -// bounds it the way `retained_bytes` measures a result's text. +// A file's catalog can repeat one long name across thousands of keys; counted as `retained_bytes` counts text. #[derive(Debug)] pub struct CatalogBudget { limit: u64, @@ -55,11 +54,13 @@ impl CatalogBudget { texts.into_iter().try_for_each(|text| self.charge_text(text)) } - // For a text the read drops before it reads the next one, such as each child's CREATE TABLE. + // For what the read drops before it reads the next, such as each child's CREATE TABLE. + pub fn release(&mut self, bytes: usize) { + self.held = self.held.saturating_sub(u64::try_from(bytes).unwrap_or(u64::MAX)); + } + pub fn release_text(&mut self, text: &str) { - self.held = self - .held - .saturating_sub(u64::try_from(text_bytes(text)).unwrap_or(u64::MAX)); + self.release(text_bytes(text)); } } diff --git a/core/crates/drivers/duckdb/src/catalog/mod.rs b/core/crates/drivers/duckdb/src/catalog/mod.rs index ce5910ca4a..ea49c27176 100644 --- a/core/crates/drivers/duckdb/src/catalog/mod.rs +++ b/core/crates/drivers/duckdb/src/catalog/mod.rs @@ -39,8 +39,7 @@ fn schema_of(table: &TableRef) -> &str { .unwrap_or(DEFAULT_SCHEMA) } -/// Rows of a catalogue query whose columns are all text or NULL, each -/// value counted against `budget` as it is read. +/// Rows of a catalogue query whose columns are all text or NULL. fn text_rows( link: &Link, sql: &str, @@ -63,6 +62,56 @@ fn text_rows( Ok(read) } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Constraints { + Own, + Referencing, +} + +// Constraints: REFERENCES, CHECK, UNIQUE and PRIMARY in the text and a NOT NULL per column; each names its table +// `{names}` times and a column, beside at most four copies of the text. DOUBLE, since the file chooses the numbers. +const CONSTRAINT_BYTES: &str = "CAST((strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'references', ''))) / 10 \ + + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'check', ''))) / 5 \ + + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'unique', ''))) / 6 \ + + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'primary', ''))) / 7 + t.column_count + 1 AS DOUBLE) \ + * ({names} * strlen(t.table_name) + 512 + coalesce((SELECT max(strlen(c.column_name)) FROM duckdb_columns() c \ + WHERE c.table_oid = t.table_oid), 0)) \ + + 4 * strlen(t.sql)"; + +// DuckDB holds a query's whole result before its first row and names each constraint after its table, so the result +// is bounded before the query runs. +fn constraints_fit( + link: &Link, + catalog: &str, + table: &TableRef, + from: Constraints, + budget: &CatalogBudget, +) -> Result<(), DriverError> { + // A constraint's name holds its table's, and the keys into a table also read the child's name. + let (names, tables) = match from { + Constraints::Own => ("1", "t.database_name = ? AND t.schema_name = ? AND t.table_name = ?"), + Constraints::Referencing => ( + "2", + "t.table_oid IN (SELECT d.refobjid FROM duckdb_dependencies() d \ + JOIN duckdb_tables() p ON d.objid = p.table_oid \ + WHERE p.database_name = ? AND p.schema_name = ? AND p.table_name = ?)", + ), + }; + let bytes_sql = CONSTRAINT_BYTES.replace("{names}", names); + let sql = format!("SELECT sum({bytes_sql}) FROM duckdb_tables() t WHERE {tables}"); + let bytes: Option = link + .connection + .query_row(&sql, duckdb::params![catalog, schema_of(table), table.name], |row| { + row.get(0) + }) + .map_err(map_error)?; + let bytes = bytes.unwrap_or(0.0); + budget.fits(match bytes < usize::MAX as f64 { + true => bytes as usize, + false => usize::MAX, + }) +} + fn text(row: &[Option], index: usize) -> String { row.get(index).cloned().flatten().unwrap_or_default() } @@ -252,6 +301,7 @@ fn key_constraints( table: &TableRef, budget: &mut CatalogBudget, ) -> Result)>, DriverError> { + constraints_fit(link, catalog, table, Constraints::Own, budget)?; let rows = text_rows( link, "SELECT constraint_name, constraint_type, array_to_string(constraint_column_names, chr(31)) \ @@ -413,6 +463,7 @@ pub(crate) fn fetch_foreign_keys( table: &TableRef, budget: &mut CatalogBudget, ) -> Result, DriverError> { + constraints_fit(link, catalog, table, Constraints::Own, budget)?; let rows = text_rows( link, "SELECT constraint_name, referenced_table, \ @@ -525,6 +576,7 @@ pub(crate) fn inbound_foreign_keys( budget: &mut CatalogBudget, ) -> Result, DriverError> { // Not the child's CREATE TABLE on every key: it grows with the keys the child declares. + constraints_fit(link, catalog, table, Constraints::Referencing, budget)?; let rows = text_rows( link, "SELECT c.constraint_name, c.schema_name, c.table_name, \ @@ -722,6 +774,7 @@ fn check_constraints( table: &TableRef, budget: &mut CatalogBudget, ) -> Result, DriverError> { + constraints_fit(link, catalog, table, Constraints::Own, budget)?; let rows = text_rows( link, "SELECT array_to_string(constraint_column_names, chr(31)), expression FROM duckdb_constraints() \ diff --git a/core/crates/drivers/duckdb/src/dump.rs b/core/crates/drivers/duckdb/src/dump.rs index 24fd33f317..1525dfa2dd 100644 --- a/core/crates/drivers/duckdb/src/dump.rs +++ b/core/crates/drivers/duckdb/src/dump.rs @@ -62,8 +62,8 @@ const TABLES_SQL: &str = "SELECT schema_name, table_name, sql FROM duckdb_tables WHERE database_name = ? AND (? IS NULL OR schema_name = ?) AND NOT internal AND NOT temporary \ ORDER BY schema_name, table_name"; -/// DuckDB keeps a foreign key's parent in the child's schema. One row -/// per child, not per key, which would repeat the child's name. +/// DuckDB keeps a foreign key's parent in the child's schema. +// One row per child: one per key would repeat the child's name. const FOREIGN_KEYS_SQL: &str = "SELECT schema_name, table_name, \ list(referenced_table ORDER BY constraint_index) \ FROM duckdb_constraints() \ diff --git a/core/crates/drivers/sqlite/src/catalog/mod.rs b/core/crates/drivers/sqlite/src/catalog/mod.rs index 6fbcc7aa21..3b8aba6507 100644 --- a/core/crates/drivers/sqlite/src/catalog/mod.rs +++ b/core/crates/drivers/sqlite/src/catalog/mod.rs @@ -45,9 +45,7 @@ pub(crate) const INDEX_LIST_SQL: &str = pub(crate) const INDEX_XINFO_SQL: &str = "SELECT cid, name, \"desc\" FROM pragma_index_xinfo(?1, ?2) WHERE key = 1 ORDER BY seqno"; -/// Not sorted: a sort holds every row before the first, and a column -/// can carry REFERENCES thousands of times, its long name on each row. -/// The reader orders a key's columns by `seq`. +// Unsorted: a sort holds every row before the first, and a column can carry REFERENCES thousands of times. pub(crate) const FOREIGN_KEY_LIST_SQL: &str = "SELECT id, seq, \"table\", \"from\", \"to\", on_update, on_delete \ FROM pragma_foreign_key_list(?1, ?2)"; @@ -88,51 +86,38 @@ pub(crate) fn dependents_sql(schema_sql: &str) -> String { ) } -/// The tables whose CREATE TABLE names the table: the ones a key can -/// point at it from. SQLite matches a key's parent folding ASCII letters -/// alone, as `lower` does, and a name can be written with its quote -/// character doubled. `small` is a CREATE TABLE of at most 64 KiB whose -/// key list cannot cost SQLite more than `?3` bytes to compile: no more -/// key columns than REFERENCES and commas, each with three names no -/// longer than the text and 512 bytes of opcodes. -fn referring_tables(schema_sql: &str) -> String { +// `small`: at most 64 KiB, with a key list SQLite cannot compile past ?3 bytes: no more key columns than REFERENCES +// and commas, each with three names no longer than the text and 512 bytes of opcodes. +fn sized_tables(schema_sql: &str) -> String { format!( - "WITH named AS (SELECT rowid, name, sql, length(CAST(sql AS BLOB)) AS bytes, \ + "WITH tables AS (SELECT rowid, name, sql, length(CAST(sql AS BLOB)) AS bytes, \ (length(sql) - length(replace(lower(sql), 'references', ''))) / 10 \ + length(sql) - length(replace(sql, ',', '')) AS key_columns \ - FROM {schema_sql}.sqlite_schema WHERE type = 'table' AND sql IS NOT NULL AND ( \ - instr(lower(sql), lower(?1)) > 0 \ - OR instr(lower(sql), lower(replace(?1, '\"', '\"\"'))) > 0 \ - OR instr(lower(sql), lower(replace(?1, '`', '``'))) > 0 \ - OR instr(lower(sql), lower(replace(?1, '''', ''''''))) > 0)), \ - referring AS (SELECT rowid, name, sql, \ - bytes <= 65536 AND (key_columns + 1) * (3 * bytes + 512) <= ?3 AS small FROM named)" + FROM {schema_sql}.sqlite_schema WHERE type = 'table' AND sql IS NOT NULL), \ + sized AS (SELECT rowid, name, sql, \ + bytes <= 65536 AND (key_columns + 1) * (3 * bytes + 512) <= ?3 AS small FROM tables)" ) } -/// Each key column of the small referring tables that points at the -/// table, with the child's CREATE TABLE on each row, not sorted: a sort -/// holds every row before the first, and a child's rows come together. +// Unsorted, so the rows stream; a child's rows come together. pub(crate) fn small_children_keys_sql(schema_sql: &str) -> String { format!( "{} SELECT m.name AS child, m.sql AS sql, f.id AS id, f.seq AS seq, f.\"from\" AS \"from\", \ f.\"to\" AS \"to\", (SELECT count(DISTINCT k.id) FROM pragma_foreign_key_list(m.name, ?2) k) AS child_keys \ - FROM referring m, pragma_foreign_key_list(m.name, ?2) f \ + FROM sized m, pragma_foreign_key_list(m.name, ?2) f \ WHERE m.small AND f.\"table\" = ?1 COLLATE NOCASE", - referring_tables(schema_sql) + sized_tables(schema_sql) ) } -/// The other referring tables, read one at a time. -pub(crate) fn large_children_sql(schema_sql: &str) -> String { +pub(crate) fn large_tables_sql(schema_sql: &str) -> String { format!( - "{} SELECT rowid, name FROM referring WHERE NOT small ORDER BY name", - referring_tables(schema_sql) + "{} SELECT rowid, name FROM sized WHERE NOT small ORDER BY name", + sized_tables(schema_sql) ) } -/// One schema row by rowid, which SQLite finds without reading the -/// others, as a lookup by name would for each of thousands of children. +// By rowid: a lookup by name scans the whole schema, once per child. pub(crate) fn schema_row_sql(schema_sql: &str) -> String { format!("SELECT name, sql FROM {schema_sql}.sqlite_schema WHERE rowid = ?1") } diff --git a/core/crates/drivers/sqlite/src/ddl/create_table_source.rs b/core/crates/drivers/sqlite/src/ddl/create_table_source.rs index 90f30618c5..4dde2d4daa 100644 --- a/core/crates/drivers/sqlite/src/ddl/create_table_source.rs +++ b/core/crates/drivers/sqlite/src/ddl/create_table_source.rs @@ -352,10 +352,8 @@ fn comparison(source: &CreateTableSource<'_>, declared: &SourceColumn) -> crate: crate::catalog::Comparison::of(&type_text, collation.as_deref()) } -// SQLite compiles `PRAGMA foreign_key_list` with one row per key column in its program, each with its own copy of -// the parent's name, the column's and the parent column's: a column carrying REFERENCES thousands of times costs its -// name thousands of times before the first row. `ROW` covers a row's opcodes and its action names. Text that does -// not read is costed as if each REFERENCES in it named the whole text. +// SQLite compiles `PRAGMA foreign_key_list` with a copy of the parent's, the column's and the parent column's names +// for each key column, so a column carrying REFERENCES thousands of times costs its name as often. pub(crate) fn foreign_key_list_bytes(sql: &str) -> usize { const ROW: usize = 512; let Ok(source) = CreateTableSource::read(sql) else { diff --git a/core/crates/drivers/sqlite/src/foreign_key_verification.rs b/core/crates/drivers/sqlite/src/foreign_key_verification.rs index d2d6053413..182d623729 100644 --- a/core/crates/drivers/sqlite/src/foreign_key_verification.rs +++ b/core/crates/drivers/sqlite/src/foreign_key_verification.rs @@ -7,10 +7,10 @@ //! and the check after it are compared, and only the difference fails //! the batch. -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use tablepro_core::ddl::ColumnRename; -use tablepro_core::{DriverError, ServerCode, ServerDiagnostics}; +use tablepro_core::{CatalogBudget, DriverError, ServerCode, ServerDiagnostics}; use crate::errors::{SQLITE_CONSTRAINT, SQLITE_CONSTRAINT_FOREIGNKEY}; @@ -20,16 +20,18 @@ pub(crate) struct Violations { tables: Vec, } -// A file can hold millions of rows that break thousands of keys of one table with a long name, so the table's name is -// held once, each key once, and each row as the index of the key it breaks and its rowid. A rowid is None for a -// WITHOUT ROWID child table, which has no rowid to name the row by, and for one whose rows the batch renumbers. +// A crafted file can break thousands of keys of a long-named table in millions of rows, so names are held once and +// rows only up to the budget, past which the table compares by count per key, as renumbered rows do. #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) struct TableViolations { table: String, keys: Vec, - rows: Vec<(usize, Option)>, + counts: Vec, + rows: Option)>>, } +const ROW_BYTES: usize = size_of::<(usize, Option)>(); + #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) struct BrokenKey { pub parent: String, @@ -41,23 +43,37 @@ impl TableViolations { Self { table, keys: Vec::new(), - rows: Vec::new(), + counts: Vec::new(), + rows: Some(Vec::new()), } } pub(crate) fn add_key(&mut self, key: BrokenKey) -> usize { self.keys.push(key); + self.counts.push(0); self.keys.len() - 1 } - pub(crate) fn add_row(&mut self, key: usize, rowid: Option) { - self.rows.push((key, rowid)); + pub(crate) fn add_row(&mut self, key: usize, rowid: Option, budget: &mut CatalogBudget) { + if let Some(count) = self.counts.get_mut(key) { + *count += 1; + } + let Some(rows) = &mut self.rows else { + return; + }; + match budget.charge(ROW_BYTES) { + Ok(()) => rows.push((key, rowid)), + Err(_) => { + budget.release(ROW_BYTES.saturating_mul(rows.len() + 1)); + self.rows = None; + } + } } } impl Violations { pub(crate) fn push(&mut self, table: TableViolations) { - if !table.rows.is_empty() { + if table.counts.iter().any(|count| *count > 0) { self.tables.push(table); } } @@ -129,6 +145,7 @@ impl Renames { }) .collect(), table: self.table_name(checked.table), + counts: checked.counts, rows: checked.rows, }) .collect(), @@ -155,48 +172,90 @@ impl Renames { /// count per child, parent and key that compares. pub(crate) fn introduced(before: &Violations, after: &Violations) -> Vec { let mut ids = KeyIds::default(); + let before_ids: Vec<(usize, Vec)> = before.tables.iter().map(|checked| ids.of(checked)).collect(); + let after_ids: Vec<(usize, Vec)> = after.tables.iter().map(|checked| ids.of(checked)).collect(); + // A table whose rows were let go on either side compares by count. + let counted: HashSet = before + .tables + .iter() + .zip(&before_ids) + .chain(after.tables.iter().zip(&after_ids)) + .filter(|(checked, _)| checked.rows.is_none()) + .map(|(_, (table, _))| *table) + .collect(); let mut remaining: HashMap<(usize, Option), usize> = HashMap::new(); - for checked in &before.tables { - let (_, keys) = ids.of(checked); - for (key, rowid) in &checked.rows { - if let Some(id) = keys.get(*key) { - *remaining.entry((*id, *rowid)).or_default() += 1; + let mut counts: HashMap = HashMap::new(); + for (checked, (table, keys)) in before.tables.iter().zip(&before_ids) { + match (counted.contains(table), &checked.rows) { + (false, Some(rows)) => { + for (key, rowid) in rows { + if let Some(id) = keys.get(*key) { + *remaining.entry((*id, *rowid)).or_default() += 1; + } + } + } + _ => { + for (id, count) in keys.iter().zip(&checked.counts) { + *counts.entry(*id).or_default() += count; + } } } } - let mut introduced: Vec = Vec::new(); - let mut groups: HashMap<(usize, String), usize> = HashMap::new(); - for checked in &after.tables { - let (table, keys) = ids.of(checked); - let parents: Vec = checked.keys.iter().map(|broken| broken.parent.to_lowercase()).collect(); - for (key, rowid) in &checked.rows { - let (Some(id), Some(broken), Some(parent)) = (keys.get(*key), checked.keys.get(*key), parents.get(*key)) - else { - continue; - }; - if let Some(left) = remaining.get_mut(&(*id, *rowid)) - && *left > 0 - { - *left -= 1; - continue; + let mut found = Introduced::default(); + for (checked, (table, keys)) in after.tables.iter().zip(&after_ids) { + match (counted.contains(table), &checked.rows) { + (false, Some(rows)) => { + for (key, rowid) in rows { + let Some(id) = keys.get(*key) else { + continue; + }; + if let Some(left) = remaining.get_mut(&(*id, *rowid)) + && *left > 0 + { + *left -= 1; + continue; + } + found.add(*table, checked, *key, 1); + } } - match groups - .get(&(table, parent.clone())) - .and_then(|index| introduced.get_mut(*index)) - { - Some(group) => group.count += 1, - None => { - groups.insert((table, parent.clone()), introduced.len()); - introduced.push(IntroducedViolations { - table: checked.table.clone(), - parent: broken.parent.clone(), - count: 1, - }); + _ => { + for (key, (id, count)) in keys.iter().zip(&checked.counts).enumerate() { + let before = counts.entry(*id).or_default(); + let kept = (*before).min(*count); + *before -= kept; + found.add(*table, checked, key, count - kept); } } } } - introduced + found.groups +} + +// The groups by child and parent, in the order their first row came. +#[derive(Default)] +struct Introduced { + groups: Vec, + positions: HashMap<(usize, String), usize>, +} + +impl Introduced { + fn add(&mut self, table: usize, checked: &TableViolations, key: usize, count: usize) { + let Some(broken) = checked.keys.get(key).filter(|_| count > 0) else { + return; + }; + let group = (table, broken.parent.to_lowercase()); + match self.positions.get(&group).and_then(|index| self.groups.get_mut(*index)) { + Some(found) => found.count += count, + None => { + self.positions.insert(group, self.groups.len()); + self.groups.push(IntroducedViolations { + table: checked.table.clone(), + parent: broken.parent.clone(), + count, + }); + } + } + } } // SQLite names tables and columns case-insensitively, and so does the comparison. A table's name is folded once, @@ -293,7 +352,7 @@ mod tests { parent_columns: owned(parent_columns), }, }); - checked.add_row(key, *rowid); + checked.add_row(key, *rowid, &mut CatalogBudget::default()); } let mut violations = Violations::default(); for checked in tables { @@ -394,17 +453,18 @@ mod tests { parent_columns: Vec::new(), }, }; + let budget = &mut CatalogBudget::default(); let mut checked = TableViolations::new("t".to_owned()); let key = checked.add_key(broken("p")); for rowid in 0..1_000 { - checked.add_row(key, Some(rowid)); + checked.add_row(key, Some(rowid), budget); } let mut before = Violations::default(); before.push(checked.clone()); let other = checked.add_key(broken("q")); - checked.add_row(other, Some(5)); - checked.add_row(key, Some(1_000)); - checked.add_row(other, Some(6)); + checked.add_row(other, Some(5), budget); + checked.add_row(key, Some(1_000), budget); + checked.add_row(other, Some(6), budget); let mut after = Violations::default(); after.push(checked); @@ -425,6 +485,51 @@ mod tests { ); } + #[test] + fn rows_past_the_budget_compare_by_count_per_key() { + let broken = |parent: &str| BrokenKey { + parent: parent.to_owned(), + key: ViolatedKey { + columns: vec!["a".to_owned()], + parent_columns: Vec::new(), + }, + }; + let rows = |count: i64, budget: &mut CatalogBudget| { + let mut checked = TableViolations::new("t".to_owned()); + let (p, q) = (checked.add_key(broken("p")), checked.add_key(broken("q"))); + for rowid in 0..count { + checked.add_row(p, Some(rowid), budget); + checked.add_row(q, Some(rowid), budget); + } + let mut found = Violations::default(); + found.push(checked); + found + }; + let budget = &mut CatalogBudget::new(tablepro_core::ByteLimit::new(4_096).unwrap()); + + let before = rows(1_000, budget); + let same = rows(1_000, budget); + let more = rows(1_001, budget); + + assert!(before.tables[0].rows.is_none(), "the rows passed the budget"); + assert!(introduced(&before, &same).is_empty()); + assert_eq!( + introduced(&before, &more), + [ + IntroducedViolations { + table: "t".to_owned(), + parent: "p".to_owned(), + count: 1 + }, + IntroducedViolations { + table: "t".to_owned(), + parent: "q".to_owned(), + count: 1 + }, + ] + ); + } + #[test] fn the_error_reads_as_a_sqlite_foreign_key_failure() { let error = violation_error(&[IntroducedViolations { diff --git a/core/crates/drivers/sqlite/src/metadata.rs b/core/crates/drivers/sqlite/src/metadata.rs index b51338725d..af769f7e77 100644 --- a/core/crates/drivers/sqlite/src/metadata.rs +++ b/core/crates/drivers/sqlite/src/metadata.rs @@ -418,7 +418,7 @@ pub(crate) async fn fetch_inbound_foreign_keys( /// The keys of the schema's other tables that point at `table`, its /// columns spelled as `shape` declares them, since a child's REFERENCES -/// clause can spell them otherwise. Ordered by child, then key. +/// clause can spell them otherwise. async fn inbound_keys( connection: &mut SqlxConnection, table: &TableRef, @@ -437,7 +437,7 @@ async fn inbound_keys( let schema_sql = quote(schema_of(table)); let mut large = Vec::new(); { - let mut rows = sqlx::query_as::<_, (i64, String)>(AssertSqlSafe(catalog::large_children_sql(&schema_sql))) + let mut rows = sqlx::query_as::<_, (i64, String)>(AssertSqlSafe(catalog::large_tables_sql(&schema_sql))) .bind(table.name.as_str()) .bind(schema_of(table)) .bind(i64::try_from(ByteLimit::CATALOG_DEFAULT.get()).unwrap_or(i64::MAX)) @@ -455,6 +455,11 @@ async fn inbound_keys( .map_err(map_error)?; // A row another connection gave to another table since is not this child's. let sql = budget.keep_optional(row.filter(|(name, _)| *name == child).and_then(|(_, sql)| sql))?; + if !sql.as_deref().is_some_and(|sql| names_table(sql, &table.name)) { + release(budget, sql); + budget.release_text(&child); + continue; + } budget.fits(sql.as_deref().map_or(0, foreign_key_list_bytes))?; let (columns, count) = key_columns_into(connection, &child, table, budget).await?; match columns.is_empty() { @@ -469,8 +474,6 @@ async fn inbound_keys( Ok(keys) } -/// The keys of the small referring tables, in one read: a child's rows -/// come together, each with its CREATE TABLE, kept from its first row. async fn small_children_keys( connection: &mut SqlxConnection, parent: &Parent<'_>, @@ -512,9 +515,19 @@ async fn small_children_keys( Ok(keys) } -/// The columns of the child's keys that point at `parent`, and how many -/// keys the child has wherever they point, since the ids number them -/// all. SQLite compares a key's parent folding ASCII letters alone. +// Linear time, since the file chooses both texts. SQLite folds ASCII letters alone, and a quoted name doubles its +// quote character. +fn names_table(sql: &str, table: &str) -> bool { + let sql = sql.to_ascii_lowercase(); + let table = table.to_ascii_lowercase(); + sql.contains(&table) + || ['"', '`', '\''].into_iter().any(|quote| { + let doubled: String = [quote, quote].iter().collect(); + table.contains(quote) && sql.contains(&table.replace(quote, &doubled)) + }) +} + +// The count is of every key of the child, wherever it points, since the ids number them all. async fn key_columns_into( connection: &mut SqlxConnection, child: &str, @@ -544,7 +557,6 @@ async fn key_columns_into( Ok((columns, ids.len())) } -/// The table the inbound keys point at, read once for all its children. struct Parent<'a> { table: &'a TableRef, shape: &'a TableShape, @@ -553,8 +565,6 @@ struct Parent<'a> { } impl Parent<'_> { - /// One child's keys, from its CREATE TABLE, which is let go before - /// the next child's is read. fn keys_of( &self, child: String, @@ -1034,3 +1044,19 @@ fn no_such_table(schema: &str, name: &str) -> DriverError { format!("no such table: {schema}.{name}"), )) } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_table_is_named_in_any_case_and_quote_style() { + let sql = "CREATE TABLE c (a REFERENCES \"WE\"\"IRD\", b REFERENCES 'it''s', d REFERENCES `x``y`)"; + + assert!(names_table(sql, "we\"ird")); + assert!(names_table(sql, "IT's")); + assert!(names_table(sql, "x`y")); + assert!(!names_table(sql, "other")); + assert!(!names_table(&"a".repeat(1 << 20), &format!("{}b", "a".repeat(1 << 14)))); + } +} diff --git a/core/crates/drivers/sqlite/src/write_batch.rs b/core/crates/drivers/sqlite/src/write_batch.rs index 3d47e98eeb..323fb3a19c 100644 --- a/core/crates/drivers/sqlite/src/write_batch.rs +++ b/core/crates/drivers/sqlite/src/write_batch.rs @@ -125,8 +125,10 @@ impl Write<'_> { } }; + // Shared by the check before the batch and the one after it, which are held together. + let mut budget = CatalogBudget::default(); let baseline = match checks { - Some(checks) => match baseline(&mut transaction, checks).await { + Some(checks) => match baseline(&mut transaction, checks, &mut budget).await { Ok(baseline) => baseline, Err(error) => { let fate = self.rollback(transaction).await; @@ -156,7 +158,7 @@ impl Write<'_> { } if let Some(checks) = checks { - let after = match violations_after(&mut transaction, checks).await { + let after = match violations_after(&mut transaction, checks, &mut budget).await { Ok(after) => after, Err(error) => { let fate = self.rollback(transaction).await; @@ -307,14 +309,18 @@ fn stop_cause(reason: StopReason) -> DriverError { } /// The violations already there, named as they will be after the batch. -async fn baseline(connection: &mut SqlxConnection, checks: &[ForeignKeyCheck]) -> Result { +async fn baseline( + connection: &mut SqlxConnection, + checks: &[ForeignKeyCheck], + budget: &mut CatalogBudget, +) -> Result { let mut found = Violations::default(); let mut renames = Renames::default(); for check in checks { let Some(before) = &check.before else { continue; }; - found.extend(violations(connection, before, check.rowids_renumbered).await?); + found.extend(violations(connection, before, check.rowids_renumbered, budget).await?); renames.add(&before.name, &check.after.name, &check.renamed_columns); } Ok(renames.apply(found)) @@ -323,10 +329,11 @@ async fn baseline(connection: &mut SqlxConnection, checks: &[ForeignKeyCheck]) - async fn violations_after( connection: &mut SqlxConnection, checks: &[ForeignKeyCheck], + budget: &mut CatalogBudget, ) -> Result { let mut found = Violations::default(); for check in checks { - found.extend(violations(connection, &check.after, check.rowids_renumbered).await?); + found.extend(violations(connection, &check.after, check.rowids_renumbered, budget).await?); } Ok(found) } @@ -338,26 +345,13 @@ async fn violations( connection: &mut SqlxConnection, table: &TableRef, rowids_renumbered: bool, + budget: &mut CatalogBudget, ) -> Result { let schema = table .schema .as_deref() .filter(|schema| !schema.is_empty()) .unwrap_or(MAIN_SCHEMA); - let mut rows: Vec<(i64, Option)> = Vec::new(); - { - let mut read = sqlx::query(FOREIGN_KEY_CHECK_SQL) - .bind(table.name.as_str()) - .bind(schema) - .fetch(&mut *connection); - while let Some(row) = read.try_next().await.map_err(map_error)? { - let rowid: Option = row.try_get("rowid").map_err(map_error)?; - rows.push((row.try_get("fkid").map_err(map_error)?, rowid)); - } - } - if rows.is_empty() { - return Ok(Violations::default()); - } let declared: Option = sqlx::query_scalar(DECLARED_TABLE_NAME_SQL) .bind(table.name.as_str()) .bind(schema) @@ -367,20 +361,25 @@ async fn violations( let mut found = TableViolations::new(declared.unwrap_or_else(|| table.name.clone())); let keys = foreign_keys(connection, table, schema).await?; let mut positions: HashMap = HashMap::new(); - for (fkid, rowid) in rows { + let mut rows = sqlx::query(FOREIGN_KEY_CHECK_SQL) + .bind(table.name.as_str()) + .bind(schema) + .fetch(&mut *connection); + while let Some(row) = rows.try_next().await.map_err(map_error)? { + let rowid: Option = row.try_get("rowid").map_err(map_error)?; + let fkid: i64 = row.try_get("fkid").map_err(map_error)?; let position = *positions.entry(fkid).or_insert_with(|| { let (parent, key) = keys.get(&fkid).cloned().unwrap_or_default(); found.add_key(BrokenKey { parent, key }) }); - found.add_row(position, rowid.filter(|_| !rowids_renumbered)); + found.add_row(position, rowid.filter(|_| !rowids_renumbered), budget); } let mut violations = Violations::default(); violations.push(found); Ok(violations) } -/// The table's foreign keys, with the parent each names, by the id -/// `foreign_key_check` names them by. +/// The table's foreign keys by the id `foreign_key_check` names them by. async fn foreign_keys( connection: &mut SqlxConnection, table: &TableRef, diff --git a/core/crates/drivers/sqlite/tests/catalog_memory.rs b/core/crates/drivers/sqlite/tests/catalog_memory.rs index d442b1a0d0..7182a3610a 100644 --- a/core/crates/drivers/sqlite/tests/catalog_memory.rs +++ b/core/crates/drivers/sqlite/tests/catalog_memory.rs @@ -28,6 +28,7 @@ const LONG: usize = 64 * 1024; const KEYS: usize = 4_000; const BROKEN_ROWS: usize = 20_000; const BULKY_CHILDREN: usize = 60; +const ORPHANS: usize = 3_000; fn quoted(name: &str) -> String { format!("\"{name}\"") @@ -39,7 +40,8 @@ fn quoted(name: &str) -> String { // - 4,000 keys naming no columns of a parent whose primary key is 64 KiB of names, each a copy of it; // - 20,000 rows of a table with a 64 KiB name that break its key, which a check named on every row; // - one row of a table with a 64 KiB name that breaks 4,000 keys, which the check named on every key; -// - 60 children of one parent, each CREATE TABLE 256 KiB long, which SQLite queued 50 at a time. +// - 60 children of one parent, each CREATE TABLE 256 KiB long, which SQLite queued 50 at a time; +// - 3,000 rows that each break 1,000 keys, three million broken keys from 30 KB of text and rows. fn crafted_schema() -> Vec { let column = "c".repeat(LONG); let child = "n".repeat(LONG); @@ -86,6 +88,14 @@ fn crafted_schema() -> Vec { ), format!("INSERT INTO {} VALUES (1)", quoted(&broken_keys())), "CREATE TABLE bulky_parent (id INTEGER PRIMARY KEY)".to_owned(), + format!( + "CREATE TABLE orphans (a INTEGER{})", + " REFERENCES named_parent".repeat(1_000) + ), + format!( + "WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < {ORPHANS}) \ + INSERT INTO orphans SELECT i FROM n" + ), ] .into_iter() .chain((0..BULKY_CHILDREN).map(|child| { @@ -180,7 +190,7 @@ async fn a_crafted_catalog_stops_at_the_budget() { let child = table(&"n".repeat(LONG)); let (objects, heap, _) = measured(connection.list_objects(&ObjectScope::DefaultSchema, call())).await; - assert_eq!(objects.unwrap().len(), 9 + BULKY_CHILDREN); + assert_eq!(objects.unwrap().len(), 10 + BULKY_CHILDREN); assert!(heap < BOUND, "listing the tables held {heap} bytes"); let (columns, _, _) = measured(connection.fetch_columns(&table("repeated"), call())).await; assert_eq!(columns.unwrap().len(), 1, "a long name alone is read"); @@ -232,6 +242,12 @@ async fn a_crafted_catalog_stops_at_the_budget() { "checking the {name}: {heap}, {native}" ); } + let (written, heap, native) = measured(connection.write_batch(&checked_save("orphans"), call())).await; + assert!(written.is_ok(), "orphans: {written:?}"); + assert!( + heap < BOUND && native < NATIVE_BOUND, + "checking the orphans: {heap}, {native}" + ); // A child's CREATE TABLE is read alone, never 50 of them queued in SQLite's reader. let (inbound, heap, native) = measured(connection.fetch_inbound_foreign_keys(&table("bulky_parent"), call())).await; diff --git a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md index a47ffbda8a..18a7179036 100644 --- a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md +++ b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md @@ -13,9 +13,9 @@ A file engine reads its catalog from the file the user opened, and the file choo - SQLite and DuckDB charge each text a read keeps as it arrives, and each copy made per key before it is made. A text read and dropped before the next one, as each child's CREATE TABLE, is released. - Each text of the file is read once per call, and SQLite's catalog rows stream, unsorted where a sort would hold every row first. -- SQLite runs `pragma_foreign_key_list` on a table only once the key list's cost, worked out from its CREATE TABLE (`foreign_key_list_bytes`), fits the budget. The keys into a table ask only the tables whose CREATE TABLE names it: those of at most 64 KiB whose key list cannot cost more than the budget in one read, the others one at a time. -- A save's foreign key check holds each checked table's name once and each row as a key index and a rowid. -- DuckDB reads each child's and each parent's CREATE TABLE once, and charges the column names it keeps from them. +- SQLite runs `pragma_foreign_key_list` on a table only once the key list's cost, worked out from its CREATE TABLE (`foreign_key_list_bytes`), fits the budget. The keys into a table ask every table of at most 64 KiB whose key list cannot cost more than the budget in one read; each other table is read alone, searched in linear time for the parent's name, and costed first. +- A save's foreign key check holds each checked table's name once and each row as a key index and a rowid, up to the budget; past it the table compares by count per key, as a table whose rows the save renumbers does. +- DuckDB bounds what a constraint read can return from `duckdb_tables()` before it runs, since it holds the whole result first. It reads each child's and each parent's CREATE TABLE once, and charges the column names it keeps from them. ## Rationale @@ -27,6 +27,6 @@ A file engine reads its catalog from the file the user opened, and the file choo - Large real schemas read as before, and the keys into a table read faster: 5,000 children in 0.2 s, against 1.4 s. - One value is read before it is charged, and sqlx queues up to 50 rows ahead of the reader. SQLite bounds a value at `SQLITE_MAX_LENGTH`, and a CREATE TABLE over 64 KiB is read alone. -- DuckDB holds a catalog query's whole result before the first row, and names each constraint after its table. The budget bounds what the driver keeps, not that. +- DuckDB names every constraint of a table after it as it scans, a chunk of rows at a time, whatever the query keeps. That is DuckDB's own work, and no read here bounds it. - Not budgeted: the schema list, reads of one value, and the SQL dumps, which read each object once. - The apps word the code: Linux in `linux/crates/app/src/ui/error_text/driver.rs`, Windows in `Resources.resw`. diff --git a/core/docs/decisions/README.md b/core/docs/decisions/README.md index aaec84f125..6a1070a7fa 100644 --- a/core/docs/decisions/README.md +++ b/core/docs/decisions/README.md @@ -36,4 +36,4 @@ The format, the statuses and how a later ADR changes an earlier one follow [the | [0026](0026-the-windows-app-signs-in-through-the-ffi.md) | The Windows app signs in to its TablePro account through the FFI | Accepted | `Core::open_account` builds the account with `AppClient::WINDOWS`, the Core's secret store for the refresh token, a `HostBrowser` callback and `MachineId::derive_machine_guid` of the registry's MachineGuid, keyed as Linux keys `/etc/machine-id`; `status()` is a snapshot and `AccountListener` only signals. | | [0027](0027-share-links.md) | Share links go through the account, with the snapshot and the secret scan in the core | Accepted; masked columns and the secret scan amended by 0028 | `Account` adds the share calls through the generated `/v2` client, `fetch_shared` without a token; problem codes stay `Refused` with their own `code()`; `build_snapshot` writes raw JSON cells by the export's lossless rules up to the plan's rows; a table page shares `build_view_sql`, its export with the filter's values written in; `secret_scan` warns; the deep link takes one 43-character base62 token. | | [0028](0028-data-masking.md) | A shared result masks its columns before it leaves the machine, with the macOS app's output | Accepted | `tablepro_core::masking` gives the macOS styles byte for byte; `SharePlan` applies the sheet's Keep, Mask or Remove per column for the snapshot, its preview and summary; the snapshot names each masked column's style and keeps to the macOS body budget; `masking::detect` suggests masks from names, values and the SQL by PLAN-MASKING 8.3. | -| [0029](0029-a-catalog-read-keeps-a-byte-budget.md) | A catalog read keeps a byte budget | Accepted | `CatalogBudget` bounds what one SQLite or DuckDB catalog read holds at 64 MB, counting each text kept and each copy made per key, and fails `DriverError::CatalogTooLarge` past it; SQLite costs a key list before it compiles one, asks only the tables that name a parent for keys into it, and streams rows unsorted; a save's foreign key check names each table once. | +| [0029](0029-a-catalog-read-keeps-a-byte-budget.md) | A catalog read keeps a byte budget | Accepted | `CatalogBudget` bounds what one SQLite or DuckDB catalog read holds at 64 MB, counting each text kept and each copy made per key, and fails `DriverError::CatalogTooLarge` past it; SQLite costs a key list before it compiles one and streams rows unsorted; DuckDB bounds a constraint read before it runs; a save's foreign key check falls back to counts past the budget. | From b9beee4708ac91ae4cf4bcad4e97d757fee1700b Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 02:27:56 +0700 Subject: [PATCH 7/9] fix(core): roll a save back past its check budget, bound every DuckDB constraint scan, count child keys once --- core/crates/drivers/duckdb/src/catalog/mod.rs | 45 ++-- .../drivers/duckdb/tests/catalog_memory.rs | 81 ++++--- core/crates/drivers/sqlite/src/catalog/mod.rs | 13 +- .../sqlite/src/foreign_key_verification.rs | 214 +++++++----------- core/crates/drivers/sqlite/src/metadata.rs | 70 ++++-- core/crates/drivers/sqlite/src/write_batch.rs | 17 +- .../drivers/sqlite/tests/catalog_memory.rs | 4 +- ...0029-a-catalog-read-keeps-a-byte-budget.md | 7 +- core/docs/decisions/README.md | 2 +- 9 files changed, 232 insertions(+), 221 deletions(-) diff --git a/core/crates/drivers/duckdb/src/catalog/mod.rs b/core/crates/drivers/duckdb/src/catalog/mod.rs index ea49c27176..4b977148de 100644 --- a/core/crates/drivers/duckdb/src/catalog/mod.rs +++ b/core/crates/drivers/duckdb/src/catalog/mod.rs @@ -68,18 +68,31 @@ enum Constraints { Referencing, } -// Constraints: REFERENCES, CHECK, UNIQUE and PRIMARY in the text and a NOT NULL per column; each names its table -// `{names}` times and a column, beside at most four copies of the text. DOUBLE, since the file chooses the numbers. -const CONSTRAINT_BYTES: &str = "CAST((strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'references', ''))) / 10 \ +// A table's constraints: REFERENCES, CHECK, UNIQUE and PRIMARY in its text, and a NOT NULL per column. DOUBLE, since +// the file chooses the numbers. +const CONSTRAINT_COUNT: &str = "CAST((strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'references', ''))) / 10 \ + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'check', ''))) / 5 \ + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'unique', ''))) / 6 \ - + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'primary', ''))) / 7 + t.column_count + 1 AS DOUBLE) \ - * ({names} * strlen(t.table_name) + 512 + coalesce((SELECT max(strlen(c.column_name)) FROM duckdb_columns() c \ - WHERE c.table_oid = t.table_oid), 0)) \ - + 4 * strlen(t.sql)"; + + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'primary', ''))) / 7 + t.column_count + 1 AS DOUBLE)"; + +// Beside its table's name, a constraint row names a column. +const ROW_BYTES: &str = "(512 + coalesce((SELECT max(strlen(c.column_name)) FROM duckdb_columns() c \ + WHERE c.table_oid = t.table_oid), 0))"; + +// DuckDB builds every table's constraint rows, a chunk of 2,048 at a time, before a query filters them, and holds the +// whole result before its first row; it names each constraint after its table. A read runs only once the result, +// with its table names `names` times a row and four copies of the text, and the largest chunk fit. +fn constraint_bounds(names: &str, tables: &str) -> String { + format!( + "SELECT \ + (SELECT sum({CONSTRAINT_COUNT} * ({names} * strlen(t.table_name) + {ROW_BYTES}) + 4 * strlen(t.sql)) \ + FROM duckdb_tables() t WHERE {tables}), \ + (SELECT max(least({CONSTRAINT_COUNT}, 2048) * (2 * strlen(t.table_name) + {ROW_BYTES})) FROM duckdb_tables() t) \ + + 3 * coalesce((SELECT sum(b) FROM (SELECT CAST(strlen(sql) AS DOUBLE) AS b FROM duckdb_tables() \ + ORDER BY b DESC LIMIT 2048)), 0)" + ) +} -// DuckDB holds a query's whole result before its first row and names each constraint after its table, so the result -// is bounded before the query runs. fn constraints_fit( link: &Link, catalog: &str, @@ -97,15 +110,15 @@ fn constraints_fit( WHERE p.database_name = ? AND p.schema_name = ? AND p.table_name = ?)", ), }; - let bytes_sql = CONSTRAINT_BYTES.replace("{names}", names); - let sql = format!("SELECT sum({bytes_sql}) FROM duckdb_tables() t WHERE {tables}"); - let bytes: Option = link + let (kept, built): (Option, Option) = link .connection - .query_row(&sql, duckdb::params![catalog, schema_of(table), table.name], |row| { - row.get(0) - }) + .query_row( + &constraint_bounds(names, tables), + duckdb::params![catalog, schema_of(table), table.name], + |row| Ok((row.get(0)?, row.get(1)?)), + ) .map_err(map_error)?; - let bytes = bytes.unwrap_or(0.0); + let bytes = kept.unwrap_or(0.0) + built.unwrap_or(0.0); budget.fits(match bytes < usize::MAX as f64 { true => bytes as usize, false => usize::MAX, diff --git a/core/crates/drivers/duckdb/tests/catalog_memory.rs b/core/crates/drivers/duckdb/tests/catalog_memory.rs index e3eed29e1d..b4577be10f 100644 --- a/core/crates/drivers/duckdb/tests/catalog_memory.rs +++ b/core/crates/drivers/duckdb/tests/catalog_memory.rs @@ -2,11 +2,12 @@ // because it counts every heap allocation the process makes, which another test running beside it would add to. Add // no other test here. +use std::sync::Arc; use std::time::Duration; -use tablepro_core::meta::TableRef; +use tablepro_core::meta::{ObjectScope, TableRef}; use tablepro_core::{ - ByteLimit, CallOptions, ConnectEnvironment, ConnectOptions, ConnectTarget, DatabaseDriver, DriverError, + ByteLimit, CallOptions, ConnectEnvironment, ConnectOptions, ConnectTarget, Connection, DatabaseDriver, DriverError, FileEndpoint, WorkspaceSettings, }; use tablepro_driver_duckdb::DuckDbDriver; @@ -24,12 +25,11 @@ const BOUND: usize = 2 * ByteLimit::CATALOG_DEFAULT.get() as usize; const KEYS: usize = 1_000; const NAMED_KEYS: usize = 600; -// DuckDB takes seconds to create and open more than this: +// DuckDB takes seconds to create and open more than this. One file a real schema could be: // - a child declaring 1,000 keys into one parent, whose CREATE TABLE, 40 KB, was read again for each of them; -// - a child with a 64 KiB name and 600 keys, each named after it, and its name again on each key into the parent; // - a CHECK naming 600 columns of 200-byte names, which is each column's. Flat: DuckDB binds a nested one on the // calling thread's stack. -fn crafted_schema() -> String { +fn large_schema() -> String { let column_names: Vec = (0..600) .map(|index| format!("c{index:03}{}", "c".repeat(196))) .collect(); @@ -37,18 +37,43 @@ fn crafted_schema() -> String { format!( "CREATE TABLE parent (id INTEGER PRIMARY KEY); \ CREATE TABLE child (a INTEGER{}); \ - CREATE TABLE named_parent (id INTEGER PRIMARY KEY); \ - CREATE TABLE {} (a INTEGER{}); \ CREATE TABLE checked ({}, CHECK (greatest({}) > 0)); \ CHECKPOINT;", ", FOREIGN KEY (a) REFERENCES parent (id)".repeat(KEYS), - "n".repeat(64 * 1024), - ", FOREIGN KEY (a) REFERENCES named_parent (id)".repeat(NAMED_KEYS), columns.join(", "), column_names.join(", "), ) } +// And one crafted: a child with a 64 KiB name and 600 keys, which DuckDB names after it whatever table a read asks for. +fn crafted_schema() -> String { + format!( + "CREATE TABLE named_parent (id INTEGER PRIMARY KEY); \ + CREATE TABLE other (id INTEGER PRIMARY KEY); \ + CREATE TABLE {} (a INTEGER{}); \ + CHECKPOINT;", + "n".repeat(64 * 1024), + ", FOREIGN KEY (a) REFERENCES named_parent (id)".repeat(NAMED_KEYS), + ) +} + +async fn open(directory: &std::path::Path, name: &str, schema: &str) -> Result, String> { + let path = directory.join(name); + duckdb::Connection::open(&path) + .and_then(|writer| writer.execute_batch(schema)) + .map_err(|error| error.to_string())?; + let options = ConnectOptions { + target: ConnectTarget::File(FileEndpoint::open_existing(path).map_err(|error| error.to_string())?), + workspace_settings: WorkspaceSettings::workspace(true), + liveness: test_liveness(), + environment: ConnectEnvironment::new("test"), + }; + DuckDbDriver + .connect(options, CancellationToken::new()) + .await + .map_err(|error| error.to_string()) +} + fn call() -> CallOptions { CallOptions::new( Some(Instant::now() + Duration::from_secs(120)), @@ -68,19 +93,8 @@ fn too_large(read: &str, result: Result) { #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn a_crafted_catalog_holds_what_it_returns() { let directory = tempfile::tempdir().unwrap(); - let path = directory.path().join("crafted.duckdb"); - duckdb::Connection::open(&path) - .unwrap() - .execute_batch(&crafted_schema()) - .unwrap(); - let options = ConnectOptions { - target: ConnectTarget::File(FileEndpoint::open_existing(path).unwrap()), - workspace_settings: WorkspaceSettings::workspace(true), - liveness: test_liveness(), - environment: ConnectEnvironment::new("test"), - }; - let connection = DuckDbDriver.connect(options, CancellationToken::new()).await.unwrap(); let table = |name: &str| TableRef::new(Some("main".to_owned()), name); + let connection = open(directory.path(), "large.duckdb", &large_schema()).await.unwrap(); let (outbound, peak) = heap_peak_of(connection.fetch_outbound_foreign_keys(&table("child"), call())).await; assert_eq!(outbound.unwrap().len(), KEYS); @@ -91,20 +105,21 @@ async fn a_crafted_catalog_holds_what_it_returns() { let (definition, peak) = heap_peak_of(connection.fetch_table_definition(&table("parent"), call())).await; assert_eq!(definition.unwrap().inbound_foreign_keys.len(), KEYS); assert!(peak < BOUND / 8, "the definition of parent held {peak} bytes"); - - let named = table(&"n".repeat(64 * 1024)); - let (inbound, peak) = heap_peak_of(connection.fetch_inbound_foreign_keys(&table("named_parent"), call())).await; - too_large("keys into named_parent", inbound); - assert!(peak < BOUND, "the keys into named_parent held {peak} bytes"); - let (keys, peak) = heap_peak_of(connection.fetch_foreign_keys(&named, call())).await; - assert_eq!( - keys.map(|keys| keys.len()), - Ok(NAMED_KEYS), - "its own keys, named once each, fit" - ); - assert!(peak < BOUND, "the keys of the long child held {peak} bytes"); let (checked, peak) = heap_peak_of(connection.fetch_table_definition(&table("checked"), call())).await; too_large("definition of checked", checked); assert!(peak < BOUND, "the definition of checked held {peak} bytes"); connection.close().await.unwrap(); + + let connection = open(directory.path(), "crafted.duckdb", &crafted_schema()) + .await + .unwrap(); + let (inbound, peak) = heap_peak_of(connection.fetch_inbound_foreign_keys(&table("named_parent"), call())).await; + too_large("keys into named_parent", inbound); + assert!(peak < BOUND / 8, "the keys into named_parent held {peak} bytes"); + let (keys, peak) = heap_peak_of(connection.fetch_foreign_keys(&table("other"), call())).await; + too_large("keys of an unrelated table", keys); + assert!(peak < BOUND / 8, "the keys of an unrelated table held {peak} bytes"); + let (objects, _) = heap_peak_of(connection.list_objects(&ObjectScope::DefaultSchema, call())).await; + assert_eq!(objects.unwrap().len(), 3, "the table list reads no constraint"); + connection.close().await.unwrap(); } diff --git a/core/crates/drivers/sqlite/src/catalog/mod.rs b/core/crates/drivers/sqlite/src/catalog/mod.rs index 3b8aba6507..ebde8972e0 100644 --- a/core/crates/drivers/sqlite/src/catalog/mod.rs +++ b/core/crates/drivers/sqlite/src/catalog/mod.rs @@ -99,13 +99,16 @@ fn sized_tables(schema_sql: &str) -> String { ) } -// Unsorted, so the rows stream; a child's rows come together. +// Every key of every small table, unsorted so the rows stream, a child's together: the child's keys are counted as +// they pass, and only those pointing here carry their names and the child's CREATE TABLE. pub(crate) fn small_children_keys_sql(schema_sql: &str) -> String { format!( - "{} SELECT m.name AS child, m.sql AS sql, f.id AS id, f.seq AS seq, f.\"from\" AS \"from\", \ - f.\"to\" AS \"to\", (SELECT count(DISTINCT k.id) FROM pragma_foreign_key_list(m.name, ?2) k) AS child_keys \ - FROM sized m, pragma_foreign_key_list(m.name, ?2) f \ - WHERE m.small AND f.\"table\" = ?1 COLLATE NOCASE", + "{} SELECT m.name AS child, f.id AS id, f.seq AS seq, \ + CASE WHEN f.\"table\" = ?1 COLLATE NOCASE THEN m.sql END AS sql, \ + CASE WHEN f.\"table\" = ?1 COLLATE NOCASE THEN f.\"from\" END AS \"from\", \ + CASE WHEN f.\"table\" = ?1 COLLATE NOCASE THEN f.\"to\" END AS \"to\", \ + f.\"table\" = ?1 COLLATE NOCASE AS points_here \ + FROM sized m, pragma_foreign_key_list(m.name, ?2) f WHERE m.small", sized_tables(schema_sql) ) } diff --git a/core/crates/drivers/sqlite/src/foreign_key_verification.rs b/core/crates/drivers/sqlite/src/foreign_key_verification.rs index 182d623729..9b8699d92e 100644 --- a/core/crates/drivers/sqlite/src/foreign_key_verification.rs +++ b/core/crates/drivers/sqlite/src/foreign_key_verification.rs @@ -7,7 +7,7 @@ //! and the check after it are compared, and only the difference fails //! the batch. -use std::collections::{HashMap, HashSet}; +use std::collections::HashMap; use tablepro_core::ddl::ColumnRename; use tablepro_core::{CatalogBudget, DriverError, ServerCode, ServerDiagnostics}; @@ -21,13 +21,12 @@ pub(crate) struct Violations { } // A crafted file can break thousands of keys of a long-named table in millions of rows, so names are held once and -// rows only up to the budget, past which the table compares by count per key, as renumbered rows do. +// every row and key is charged to the save's budget: past it the save stops rather than compare less. #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) struct TableViolations { table: String, keys: Vec, - counts: Vec, - rows: Option)>>, + rows: Vec<(usize, Option)>, } const ROW_BYTES: usize = size_of::<(usize, Option)>(); @@ -39,41 +38,36 @@ pub(crate) struct BrokenKey { } impl TableViolations { - pub(crate) fn new(table: String) -> Self { - Self { - table, + pub(crate) fn new(table: String, budget: &mut CatalogBudget) -> Result { + Ok(Self { + table: budget.keep(table)?, keys: Vec::new(), - counts: Vec::new(), - rows: Some(Vec::new()), - } + rows: Vec::new(), + }) } - pub(crate) fn add_key(&mut self, key: BrokenKey) -> usize { + pub(crate) fn add_key(&mut self, key: BrokenKey, budget: &mut CatalogBudget) -> Result { + budget.charge_text(&key.parent)?; + budget.charge_texts(key.key.columns.iter().chain(&key.key.parent_columns))?; self.keys.push(key); - self.counts.push(0); - self.keys.len() - 1 + Ok(self.keys.len() - 1) } - pub(crate) fn add_row(&mut self, key: usize, rowid: Option, budget: &mut CatalogBudget) { - if let Some(count) = self.counts.get_mut(key) { - *count += 1; - } - let Some(rows) = &mut self.rows else { - return; - }; - match budget.charge(ROW_BYTES) { - Ok(()) => rows.push((key, rowid)), - Err(_) => { - budget.release(ROW_BYTES.saturating_mul(rows.len() + 1)); - self.rows = None; - } - } + pub(crate) fn add_row( + &mut self, + key: usize, + rowid: Option, + budget: &mut CatalogBudget, + ) -> Result<(), DriverError> { + budget.charge(ROW_BYTES)?; + self.rows.push((key, rowid)); + Ok(()) } } impl Violations { pub(crate) fn push(&mut self, table: TableViolations) { - if table.counts.iter().any(|count| *count > 0) { + if !table.rows.is_empty() { self.tables.push(table); } } @@ -145,7 +139,6 @@ impl Renames { }) .collect(), table: self.table_name(checked.table), - counts: checked.counts, rows: checked.rows, }) .collect(), @@ -172,60 +165,29 @@ impl Renames { /// count per child, parent and key that compares. pub(crate) fn introduced(before: &Violations, after: &Violations) -> Vec { let mut ids = KeyIds::default(); - let before_ids: Vec<(usize, Vec)> = before.tables.iter().map(|checked| ids.of(checked)).collect(); - let after_ids: Vec<(usize, Vec)> = after.tables.iter().map(|checked| ids.of(checked)).collect(); - // A table whose rows were let go on either side compares by count. - let counted: HashSet = before - .tables - .iter() - .zip(&before_ids) - .chain(after.tables.iter().zip(&after_ids)) - .filter(|(checked, _)| checked.rows.is_none()) - .map(|(_, (table, _))| *table) - .collect(); let mut remaining: HashMap<(usize, Option), usize> = HashMap::new(); - let mut counts: HashMap = HashMap::new(); - for (checked, (table, keys)) in before.tables.iter().zip(&before_ids) { - match (counted.contains(table), &checked.rows) { - (false, Some(rows)) => { - for (key, rowid) in rows { - if let Some(id) = keys.get(*key) { - *remaining.entry((*id, *rowid)).or_default() += 1; - } - } - } - _ => { - for (id, count) in keys.iter().zip(&checked.counts) { - *counts.entry(*id).or_default() += count; - } + for checked in &before.tables { + let (_, keys) = ids.of(checked); + for (key, rowid) in &checked.rows { + if let Some(id) = keys.get(*key) { + *remaining.entry((*id, *rowid)).or_default() += 1; } } } let mut found = Introduced::default(); - for (checked, (table, keys)) in after.tables.iter().zip(&after_ids) { - match (counted.contains(table), &checked.rows) { - (false, Some(rows)) => { - for (key, rowid) in rows { - let Some(id) = keys.get(*key) else { - continue; - }; - if let Some(left) = remaining.get_mut(&(*id, *rowid)) - && *left > 0 - { - *left -= 1; - continue; - } - found.add(*table, checked, *key, 1); - } - } - _ => { - for (key, (id, count)) in keys.iter().zip(&checked.counts).enumerate() { - let before = counts.entry(*id).or_default(); - let kept = (*before).min(*count); - *before -= kept; - found.add(*table, checked, key, count - kept); - } + for checked in &after.tables { + let (table, keys) = ids.of(checked); + for (key, rowid) in &checked.rows { + let Some(id) = keys.get(*key) else { + continue; + }; + if let Some(left) = remaining.get_mut(&(*id, *rowid)) + && *left > 0 + { + *left -= 1; + continue; } + found.add(table, checked, *key); } } found.groups @@ -239,19 +201,19 @@ struct Introduced { } impl Introduced { - fn add(&mut self, table: usize, checked: &TableViolations, key: usize, count: usize) { - let Some(broken) = checked.keys.get(key).filter(|_| count > 0) else { + fn add(&mut self, table: usize, checked: &TableViolations, key: usize) { + let Some(broken) = checked.keys.get(key) else { return; }; let group = (table, broken.parent.to_lowercase()); match self.positions.get(&group).and_then(|index| self.groups.get_mut(*index)) { - Some(found) => found.count += count, + Some(found) => found.count += 1, None => { self.positions.insert(group, self.groups.len()); self.groups.push(IntroducedViolations { table: checked.table.clone(), parent: broken.parent.clone(), - count, + count: 1, }); } } @@ -340,19 +302,25 @@ mod tests { let position = match tables.iter().position(|checked| checked.table == *table) { Some(position) => position, None => { - tables.push(TableViolations::new((*table).to_owned())); + tables.push(TableViolations::new((*table).to_owned(), &mut CatalogBudget::default()).unwrap()); tables.len() - 1 } }; let checked = &mut tables[position]; - let key = checked.add_key(BrokenKey { - parent: (*parent).to_owned(), - key: ViolatedKey { - columns: owned(columns), - parent_columns: owned(parent_columns), - }, - }); - checked.add_row(key, *rowid, &mut CatalogBudget::default()); + let budget = &mut CatalogBudget::default(); + let key = checked + .add_key( + BrokenKey { + parent: (*parent).to_owned(), + key: ViolatedKey { + columns: owned(columns), + parent_columns: owned(parent_columns), + }, + }, + budget, + ) + .unwrap(); + checked.add_row(key, *rowid, budget).unwrap(); } let mut violations = Violations::default(); for checked in tables { @@ -454,17 +422,17 @@ mod tests { }, }; let budget = &mut CatalogBudget::default(); - let mut checked = TableViolations::new("t".to_owned()); - let key = checked.add_key(broken("p")); + let mut checked = TableViolations::new("t".to_owned(), budget).unwrap(); + let key = checked.add_key(broken("p"), budget).unwrap(); for rowid in 0..1_000 { - checked.add_row(key, Some(rowid), budget); + checked.add_row(key, Some(rowid), budget).unwrap(); } let mut before = Violations::default(); before.push(checked.clone()); - let other = checked.add_key(broken("q")); - checked.add_row(other, Some(5), budget); - checked.add_row(key, Some(1_000), budget); - checked.add_row(other, Some(6), budget); + let other = checked.add_key(broken("q"), budget).unwrap(); + checked.add_row(other, Some(5), budget).unwrap(); + checked.add_row(key, Some(1_000), budget).unwrap(); + checked.add_row(other, Some(6), budget).unwrap(); let mut after = Violations::default(); after.push(checked); @@ -486,48 +454,22 @@ mod tests { } #[test] - fn rows_past_the_budget_compare_by_count_per_key() { - let broken = |parent: &str| BrokenKey { - parent: parent.to_owned(), - key: ViolatedKey { - columns: vec!["a".to_owned()], - parent_columns: Vec::new(), - }, - }; - let rows = |count: i64, budget: &mut CatalogBudget| { - let mut checked = TableViolations::new("t".to_owned()); - let (p, q) = (checked.add_key(broken("p")), checked.add_key(broken("q"))); - for rowid in 0..count { - checked.add_row(p, Some(rowid), budget); - checked.add_row(q, Some(rowid), budget); - } - let mut found = Violations::default(); - found.push(checked); - found - }; + fn rows_past_the_budget_stop_the_check() { let budget = &mut CatalogBudget::new(tablepro_core::ByteLimit::new(4_096).unwrap()); - - let before = rows(1_000, budget); - let same = rows(1_000, budget); - let more = rows(1_001, budget); - - assert!(before.tables[0].rows.is_none(), "the rows passed the budget"); - assert!(introduced(&before, &same).is_empty()); - assert_eq!( - introduced(&before, &more), - [ - IntroducedViolations { - table: "t".to_owned(), + let mut checked = TableViolations::new("t".to_owned(), budget).unwrap(); + let key = checked + .add_key( + BrokenKey { parent: "p".to_owned(), - count: 1 - }, - IntroducedViolations { - table: "t".to_owned(), - parent: "q".to_owned(), - count: 1 + key: ViolatedKey::default(), }, - ] - ); + budget, + ) + .unwrap(); + + let stopped = (0..1_000).try_for_each(|rowid| checked.add_row(key, Some(rowid), budget)); + + assert_eq!(stopped, Err(DriverError::CatalogTooLarge { limit_bytes: 4_096 })); } #[test] diff --git a/core/crates/drivers/sqlite/src/metadata.rs b/core/crates/drivers/sqlite/src/metadata.rs index af769f7e77..11088a8a86 100644 --- a/core/crates/drivers/sqlite/src/metadata.rs +++ b/core/crates/drivers/sqlite/src/metadata.rs @@ -486,35 +486,67 @@ async fn small_children_keys( .bind(i64::try_from(ByteLimit::CATALOG_DEFAULT.get()).unwrap_or(i64::MAX)) .fetch(&mut *connection); let mut keys = Vec::new(); - let mut child: Option<(String, Option, usize, Vec)> = None; + let mut child: Option = None; while let Some(row) = rows.try_next().await.map_err(map_error)? { let name: &str = row.try_get("child").map_err(map_error)?; - if child.as_ref().is_none_or(|(read, ..)| read != name) { - if let Some((read, sql, count, columns)) = child.take() { - keys.extend(parent.keys_of(read, columns, sql, count, budget)?); + if child.as_ref().is_none_or(|read| read.name != name) { + if let Some(done) = child.take() { + keys.extend(done.keys(parent, budget)?); } - child = Some(( - budget.keep(name.to_owned())?, - budget.keep_optional(row.try_get("sql").map_err(map_error)?)?, - usize::try_from(row.try_get::("child_keys").map_err(map_error)?).unwrap_or(0), - Vec::new(), - )); + child = Some(ChildRows::new(budget.keep(name.to_owned())?)); } - if let Some((_, _, _, columns)) = &mut child { - columns.push(InboundKeyColumn { - id: row.try_get("id").map_err(map_error)?, - seq: row.try_get("seq").map_err(map_error)?, - from: budget.keep(text(&row, "from")?)?, - to: budget.keep_optional(row.try_get("to").map_err(map_error)?)?, - }); + let Some(read) = &mut child else { + continue; + }; + let id: i64 = row.try_get("id").map_err(map_error)?; + read.ids.insert(id); + if !row.try_get::("points_here").map_err(map_error)? { + continue; + } + if read.sql.is_none() { + read.sql = budget.keep_optional(row.try_get("sql").map_err(map_error)?)?; } + read.columns.push(InboundKeyColumn { + id, + seq: row.try_get("seq").map_err(map_error)?, + from: budget.keep(text(&row, "from")?)?, + to: budget.keep_optional(row.try_get("to").map_err(map_error)?)?, + }); } - if let Some((read, sql, count, columns)) = child { - keys.extend(parent.keys_of(read, columns, sql, count, budget)?); + if let Some(done) = child { + keys.extend(done.keys(parent, budget)?); } Ok(keys) } +// One child's rows of the small tables' key list, as they pass. +struct ChildRows { + name: String, + sql: Option, + ids: HashSet, + columns: Vec, +} + +impl ChildRows { + fn new(name: String) -> Self { + Self { + name, + sql: None, + ids: HashSet::new(), + columns: Vec::new(), + } + } + + fn keys(self, parent: &Parent<'_>, budget: &mut CatalogBudget) -> Result, DriverError> { + if self.columns.is_empty() { + budget.release_text(&self.name); + release(budget, self.sql); + return Ok(Vec::new()); + } + parent.keys_of(self.name, self.columns, self.sql, self.ids.len(), budget) + } +} + // Linear time, since the file chooses both texts. SQLite folds ASCII letters alone, and a quoted name doubles its // quote character. fn names_table(sql: &str, table: &str) -> bool { diff --git a/core/crates/drivers/sqlite/src/write_batch.rs b/core/crates/drivers/sqlite/src/write_batch.rs index 323fb3a19c..eaadf4c9bc 100644 --- a/core/crates/drivers/sqlite/src/write_batch.rs +++ b/core/crates/drivers/sqlite/src/write_batch.rs @@ -358,7 +358,7 @@ async fn violations( .fetch_optional(&mut *connection) .await .map_err(map_error)?; - let mut found = TableViolations::new(declared.unwrap_or_else(|| table.name.clone())); + let mut found = TableViolations::new(declared.unwrap_or_else(|| table.name.clone()), budget)?; let keys = foreign_keys(connection, table, schema).await?; let mut positions: HashMap = HashMap::new(); let mut rows = sqlx::query(FOREIGN_KEY_CHECK_SQL) @@ -368,11 +368,16 @@ async fn violations( while let Some(row) = rows.try_next().await.map_err(map_error)? { let rowid: Option = row.try_get("rowid").map_err(map_error)?; let fkid: i64 = row.try_get("fkid").map_err(map_error)?; - let position = *positions.entry(fkid).or_insert_with(|| { - let (parent, key) = keys.get(&fkid).cloned().unwrap_or_default(); - found.add_key(BrokenKey { parent, key }) - }); - found.add_row(position, rowid.filter(|_| !rowids_renumbered), budget); + let position = match positions.get(&fkid) { + Some(position) => *position, + None => { + let (parent, key) = keys.get(&fkid).cloned().unwrap_or_default(); + let position = found.add_key(BrokenKey { parent, key }, budget)?; + positions.insert(fkid, position); + position + } + }; + found.add_row(position, rowid.filter(|_| !rowids_renumbered), budget)?; } let mut violations = Violations::default(); violations.push(found); diff --git a/core/crates/drivers/sqlite/tests/catalog_memory.rs b/core/crates/drivers/sqlite/tests/catalog_memory.rs index 7182a3610a..a599c9de4c 100644 --- a/core/crates/drivers/sqlite/tests/catalog_memory.rs +++ b/core/crates/drivers/sqlite/tests/catalog_memory.rs @@ -41,7 +41,7 @@ fn quoted(name: &str) -> String { // - 20,000 rows of a table with a 64 KiB name that break its key, which a check named on every row; // - one row of a table with a 64 KiB name that breaks 4,000 keys, which the check named on every key; // - 60 children of one parent, each CREATE TABLE 256 KiB long, which SQLite queued 50 at a time; -// - 3,000 rows that each break 1,000 keys, three million broken keys from 30 KB of text and rows. +// - 3,000 rows that each break 1,000 keys, three million broken keys from 30 KB of text and rows, which stop a save. fn crafted_schema() -> Vec { let column = "c".repeat(LONG); let child = "n".repeat(LONG); @@ -243,7 +243,7 @@ async fn a_crafted_catalog_stops_at_the_budget() { ); } let (written, heap, native) = measured(connection.write_batch(&checked_save("orphans"), call())).await; - assert!(written.is_ok(), "orphans: {written:?}"); + too_large("checking the orphans", written.map_err(|error| error.cause().clone())); assert!( heap < BOUND && native < NATIVE_BOUND, "checking the orphans: {heap}, {native}" diff --git a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md index 18a7179036..a32b37065b 100644 --- a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md +++ b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md @@ -14,8 +14,8 @@ A file engine reads its catalog from the file the user opened, and the file choo - SQLite and DuckDB charge each text a read keeps as it arrives, and each copy made per key before it is made. A text read and dropped before the next one, as each child's CREATE TABLE, is released. - Each text of the file is read once per call, and SQLite's catalog rows stream, unsorted where a sort would hold every row first. - SQLite runs `pragma_foreign_key_list` on a table only once the key list's cost, worked out from its CREATE TABLE (`foreign_key_list_bytes`), fits the budget. The keys into a table ask every table of at most 64 KiB whose key list cannot cost more than the budget in one read; each other table is read alone, searched in linear time for the parent's name, and costed first. -- A save's foreign key check holds each checked table's name once and each row as a key index and a rowid, up to the budget; past it the table compares by count per key, as a table whose rows the save renumbers does. -- DuckDB bounds what a constraint read can return from `duckdb_tables()` before it runs, since it holds the whole result first. It reads each child's and each parent's CREATE TABLE once, and charges the column names it keeps from them. +- A save's foreign key check holds each checked table's name and each broken key once, and each broken row as a key index and a rowid, all charged to one budget for the checks before and after the save. Past it the save rolls back with `CatalogTooLarge`: a check by count alone could let the save move a broken key to other rows. +- DuckDB builds every table's constraint rows, a chunk at a time, before a query filters them, and holds the whole result before the first row. A constraint read runs only once both fit, bounded from `duckdb_tables()` and `duckdb_dependencies()`. It reads each child's and each parent's CREATE TABLE once, and charges the column names it keeps from them. ## Rationale @@ -27,6 +27,7 @@ A file engine reads its catalog from the file the user opened, and the file choo - Large real schemas read as before, and the keys into a table read faster: 5,000 children in 0.2 s, against 1.4 s. - One value is read before it is charged, and sqlx queues up to 50 rows ahead of the reader. SQLite bounds a value at `SQLITE_MAX_LENGTH`, and a CREATE TABLE over 64 KiB is read alone. -- DuckDB names every constraint of a table after it as it scans, a chunk of rows at a time, whatever the query keeps. That is DuckDB's own work, and no read here bounds it. +- In a DuckDB file with a crafted table, every read that touches constraints fails, the columns of any table among them, since DuckDB builds the crafted table's rows for each. The table list still reads. +- A save on a table with millions of rows that already break its keys rolls back rather than compare them. - Not budgeted: the schema list, reads of one value, and the SQL dumps, which read each object once. - The apps word the code: Linux in `linux/crates/app/src/ui/error_text/driver.rs`, Windows in `Resources.resw`. diff --git a/core/docs/decisions/README.md b/core/docs/decisions/README.md index 6a1070a7fa..d3e14d0bd3 100644 --- a/core/docs/decisions/README.md +++ b/core/docs/decisions/README.md @@ -36,4 +36,4 @@ The format, the statuses and how a later ADR changes an earlier one follow [the | [0026](0026-the-windows-app-signs-in-through-the-ffi.md) | The Windows app signs in to its TablePro account through the FFI | Accepted | `Core::open_account` builds the account with `AppClient::WINDOWS`, the Core's secret store for the refresh token, a `HostBrowser` callback and `MachineId::derive_machine_guid` of the registry's MachineGuid, keyed as Linux keys `/etc/machine-id`; `status()` is a snapshot and `AccountListener` only signals. | | [0027](0027-share-links.md) | Share links go through the account, with the snapshot and the secret scan in the core | Accepted; masked columns and the secret scan amended by 0028 | `Account` adds the share calls through the generated `/v2` client, `fetch_shared` without a token; problem codes stay `Refused` with their own `code()`; `build_snapshot` writes raw JSON cells by the export's lossless rules up to the plan's rows; a table page shares `build_view_sql`, its export with the filter's values written in; `secret_scan` warns; the deep link takes one 43-character base62 token. | | [0028](0028-data-masking.md) | A shared result masks its columns before it leaves the machine, with the macOS app's output | Accepted | `tablepro_core::masking` gives the macOS styles byte for byte; `SharePlan` applies the sheet's Keep, Mask or Remove per column for the snapshot, its preview and summary; the snapshot names each masked column's style and keeps to the macOS body budget; `masking::detect` suggests masks from names, values and the SQL by PLAN-MASKING 8.3. | -| [0029](0029-a-catalog-read-keeps-a-byte-budget.md) | A catalog read keeps a byte budget | Accepted | `CatalogBudget` bounds what one SQLite or DuckDB catalog read holds at 64 MB, counting each text kept and each copy made per key, and fails `DriverError::CatalogTooLarge` past it; SQLite costs a key list before it compiles one and streams rows unsorted; DuckDB bounds a constraint read before it runs; a save's foreign key check falls back to counts past the budget. | +| [0029](0029-a-catalog-read-keeps-a-byte-budget.md) | A catalog read keeps a byte budget | Accepted | `CatalogBudget` bounds what one SQLite or DuckDB catalog read holds at 64 MB, counting each text kept and each copy made per key, and fails `DriverError::CatalogTooLarge` past it; SQLite costs a key list before it compiles one and streams rows unsorted; DuckDB bounds a constraint read before it runs; a save's foreign key check rolls back past the budget. | From 864a6c3cdfbf62e6f5ed1123847c0f75c6fdba83 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 02:40:15 +0700 Subject: [PATCH 8/9] fix(core): bound DuckDB chunks across tables, cost unreadable key lists by commas, group save rows by key once --- core/crates/drivers/duckdb/src/catalog/mod.rs | 11 +++-- .../sqlite/src/ddl/create_table_source.rs | 13 +++--- .../sqlite/src/foreign_key_verification.rs | 42 ++++++++++++------- 3 files changed, 42 insertions(+), 24 deletions(-) diff --git a/core/crates/drivers/duckdb/src/catalog/mod.rs b/core/crates/drivers/duckdb/src/catalog/mod.rs index 4b977148de..a300d11c22 100644 --- a/core/crates/drivers/duckdb/src/catalog/mod.rs +++ b/core/crates/drivers/duckdb/src/catalog/mod.rs @@ -79,15 +79,18 @@ const CONSTRAINT_COUNT: &str = "CAST((strlen(lower(t.sql)) - strlen(replace(lowe const ROW_BYTES: &str = "(512 + coalesce((SELECT max(strlen(c.column_name)) FROM duckdb_columns() c \ WHERE c.table_oid = t.table_oid), 0))"; -// DuckDB builds every table's constraint rows, a chunk of 2,048 at a time, before a query filters them, and holds the -// whole result before its first row; it names each constraint after its table. A read runs only once the result, -// with its table names `names` times a row and four copies of the text, and the largest chunk fit. +// DuckDB builds every table's constraint rows, 2,048 to a chunk across tables, before a query filters them, and holds +// the whole result before its first row; it names each constraint after its table. A read runs only once the result, +// its table's name `names` times a row and four copies of the text, and the costliest 2,048 rows of any tables fit. fn constraint_bounds(names: &str, tables: &str) -> String { format!( "SELECT \ (SELECT sum({CONSTRAINT_COUNT} * ({names} * strlen(t.table_name) + {ROW_BYTES}) + 4 * strlen(t.sql)) \ FROM duckdb_tables() t WHERE {tables}), \ - (SELECT max(least({CONSTRAINT_COUNT}, 2048) * (2 * strlen(t.table_name) + {ROW_BYTES})) FROM duckdb_tables() t) \ + (SELECT coalesce(sum(bytes * greatest(0, least(rows, 2048 - (upto - rows)))), 0) FROM \ + (SELECT rows, bytes, sum(rows) OVER (ORDER BY bytes DESC ROWS UNBOUNDED PRECEDING) AS upto FROM \ + (SELECT least({CONSTRAINT_COUNT}, 2048) AS rows, 2 * strlen(t.table_name) + {ROW_BYTES} AS bytes \ + FROM duckdb_tables() t))) \ + 3 * coalesce((SELECT sum(b) FROM (SELECT CAST(strlen(sql) AS DOUBLE) AS b FROM duckdb_tables() \ ORDER BY b DESC LIMIT 2048)), 0)" ) diff --git a/core/crates/drivers/sqlite/src/ddl/create_table_source.rs b/core/crates/drivers/sqlite/src/ddl/create_table_source.rs index 4dde2d4daa..5cd00ffd8f 100644 --- a/core/crates/drivers/sqlite/src/ddl/create_table_source.rs +++ b/core/crates/drivers/sqlite/src/ddl/create_table_source.rs @@ -356,9 +356,11 @@ fn comparison(source: &CreateTableSource<'_>, declared: &SourceColumn) -> crate: // for each key column, so a column carrying REFERENCES thousands of times costs its name as often. pub(crate) fn foreign_key_list_bytes(sql: &str) -> usize { const ROW: usize = 512; + // Text this reader cannot follow, SQLite may: no more key columns than REFERENCES and commas, each with three names + // no longer than the text. let Ok(source) = CreateTableSource::read(sql) else { - let references = sql.to_ascii_lowercase().matches("references").count(); - return references.saturating_mul(sql.len().saturating_add(ROW)); + let columns = sql.to_ascii_lowercase().matches("references").count() + sql.matches(',').count() + 1; + return columns.saturating_mul(sql.len().saturating_mul(3).saturating_add(ROW)); }; source .foreign_keys() @@ -997,10 +999,11 @@ mod tests { (512 + 2 + 1 + 1) + (512 + 2 + 1 + 2) ); assert_eq!(foreign_key_list_bytes(&repeated), 100 * (512 + 1 + 1_000)); + let unread = "CREATE TABLE t (a, b, FOREIGN KEY (a, b) REFERENCES p (x, y)) /*"; assert_eq!( - foreign_key_list_bytes("CREATE TABLE t (a REFERENCES p, b 'unclosed)"), - "CREATE TABLE t (a REFERENCES p, b 'unclosed)".len() + 512, - "text that does not read" + foreign_key_list_bytes(unread), + (1 + 4 + 1) * (3 * unread.len() + 512), + "text that does not read, a key column per REFERENCES and comma" ); } diff --git a/core/crates/drivers/sqlite/src/foreign_key_verification.rs b/core/crates/drivers/sqlite/src/foreign_key_verification.rs index 9b8699d92e..1565c799f2 100644 --- a/core/crates/drivers/sqlite/src/foreign_key_verification.rs +++ b/core/crates/drivers/sqlite/src/foreign_key_verification.rs @@ -177,6 +177,7 @@ pub(crate) fn introduced(before: &Violations, after: &Violations) -> Vec> = vec![None; checked.keys.len()]; for (key, rowid) in &checked.rows { let Some(id) = keys.get(*key) else { continue; @@ -187,13 +188,17 @@ pub(crate) fn introduced(before: &Violations, after: &Violations) -> Vec, @@ -201,21 +206,28 @@ struct Introduced { } impl Introduced { - fn add(&mut self, table: usize, checked: &TableViolations, key: usize) { + fn group_of(&mut self, table: usize, checked: &TableViolations, key: usize) -> usize { let Some(broken) = checked.keys.get(key) else { - return; + return usize::MAX; }; - let group = (table, broken.parent.to_lowercase()); - match self.positions.get(&group).and_then(|index| self.groups.get_mut(*index)) { - Some(found) => found.count += 1, - None => { - self.positions.insert(group, self.groups.len()); - self.groups.push(IntroducedViolations { - table: checked.table.clone(), - parent: broken.parent.clone(), - count: 1, - }); - } + let next = self.groups.len(); + let group = *self + .positions + .entry((table, broken.parent.to_lowercase())) + .or_insert(next); + if group == next { + self.groups.push(IntroducedViolations { + table: checked.table.clone(), + parent: broken.parent.clone(), + count: 0, + }); + } + group + } + + fn add(&mut self, group: usize) { + if let Some(found) = self.groups.get_mut(group) { + found.count += 1; } } } From cffd74288b6d19ffc233ec754317181195204a68 Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Sun, 15 Nov 2026 02:47:53 +0700 Subject: [PATCH 9/9] fix(core): budget a save's violation groups and keep the DuckDB bound off duckdb_columns --- core/crates/drivers/duckdb/src/catalog/mod.rs | 6 +- .../sqlite/src/foreign_key_verification.rs | 77 +++++++++++++++---- core/crates/drivers/sqlite/src/write_batch.rs | 8 +- ...0029-a-catalog-read-keeps-a-byte-budget.md | 3 +- 4 files changed, 73 insertions(+), 21 deletions(-) diff --git a/core/crates/drivers/duckdb/src/catalog/mod.rs b/core/crates/drivers/duckdb/src/catalog/mod.rs index a300d11c22..e8b6679d34 100644 --- a/core/crates/drivers/duckdb/src/catalog/mod.rs +++ b/core/crates/drivers/duckdb/src/catalog/mod.rs @@ -75,9 +75,9 @@ const CONSTRAINT_COUNT: &str = "CAST((strlen(lower(t.sql)) - strlen(replace(lowe + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'unique', ''))) / 6 \ + (strlen(lower(t.sql)) - strlen(replace(lower(t.sql), 'primary', ''))) / 7 + t.column_count + 1 AS DOUBLE)"; -// Beside its table's name, a constraint row names a column. -const ROW_BYTES: &str = "(512 + coalesce((SELECT max(strlen(c.column_name)) FROM duckdb_columns() c \ - WHERE c.table_oid = t.table_oid), 0))"; +// Beside its table's name, a constraint row's own words. Not `duckdb_columns()`, which repeats each table's name on +// every column row it builds, for every table, before a filter. +const ROW_BYTES: &str = "512"; // DuckDB builds every table's constraint rows, 2,048 to a chunk across tables, before a query filters them, and holds // the whole result before its first row; it names each constraint after its table. A read runs only once the result, diff --git a/core/crates/drivers/sqlite/src/foreign_key_verification.rs b/core/crates/drivers/sqlite/src/foreign_key_verification.rs index 1565c799f2..a88b0a6fdd 100644 --- a/core/crates/drivers/sqlite/src/foreign_key_verification.rs +++ b/core/crates/drivers/sqlite/src/foreign_key_verification.rs @@ -163,7 +163,11 @@ impl Renames { /// The rows of `after` that `before` does not account for, compared as /// multisets. Rows without a rowid are all alike, so for them it is the /// count per child, parent and key that compares. -pub(crate) fn introduced(before: &Violations, after: &Violations) -> Vec { +pub(crate) fn introduced( + before: &Violations, + after: &Violations, + budget: &mut CatalogBudget, +) -> Result, DriverError> { let mut ids = KeyIds::default(); let mut remaining: HashMap<(usize, Option), usize> = HashMap::new(); for checked in &before.tables { @@ -189,12 +193,15 @@ pub(crate) fn introduced(before: &Violations, after: &Violations) -> Vec *group, + None => *group.insert(found.group_of(table, checked, *key, budget)?), + }; found.add(group); } } } - found.groups + Ok(found.groups) } // The groups by child and parent, in the order their first row came. Each key's group is looked up once, not on @@ -206,9 +213,16 @@ struct Introduced { } impl Introduced { - fn group_of(&mut self, table: usize, checked: &TableViolations, key: usize) -> usize { + // Each group holds the child's and the parent's names, which a crafted file makes long and many. + fn group_of( + &mut self, + table: usize, + checked: &TableViolations, + key: usize, + budget: &mut CatalogBudget, + ) -> Result { let Some(broken) = checked.keys.get(key) else { - return usize::MAX; + return Ok(usize::MAX); }; let next = self.groups.len(); let group = *self @@ -217,12 +231,12 @@ impl Introduced { .or_insert(next); if group == next { self.groups.push(IntroducedViolations { - table: checked.table.clone(), - parent: broken.parent.clone(), + table: budget.keep(checked.table.clone())?, + parent: budget.keep(broken.parent.clone())?, count: 0, }); } - group + Ok(group) } fn add(&mut self, group: usize) { @@ -268,8 +282,11 @@ impl KeyIds { /// The error a batch that introduced violations is rolled back with, /// in the form SQLite reports a foreign key failure itself. pub(crate) fn violation_error(introduced: &[IntroducedViolations]) -> DriverError { - let listed: Vec = introduced + // The first few say where to look; a crafted file can make thousands, each with long names. + const LISTED: usize = 10; + let mut listed: Vec = introduced .iter() + .take(LISTED) .map(|group| { format!( "{} rows of {} point at missing rows of {}", @@ -277,6 +294,9 @@ pub(crate) fn violation_error(introduced: &[IntroducedViolations]) -> DriverErro ) }) .collect(); + if introduced.len() > LISTED { + listed.push(format!("and {} more", introduced.len() - LISTED)); + } DriverError::reported(ServerDiagnostics::new( Some(ServerCode::Sqlite { primary: SQLITE_CONSTRAINT, @@ -367,7 +387,12 @@ mod tests { violation("lines", Some(2), "orders"), ]; - let introduced = introduced(&renames.apply(found(&before)), &found(&after)); + let introduced = introduced( + &renames.apply(found(&before)), + &found(&after), + &mut CatalogBudget::default(), + ) + .unwrap(); assert_eq!( introduced, @@ -385,7 +410,7 @@ mod tests { let after = vec![keyed("children", Some(1), "parents", &["b"], &["id"])]; assert_eq!( - introduced(&found(&before), &found(&after)), + introduced(&found(&before), &found(&after), &mut CatalogBudget::default()).unwrap(), [IntroducedViolations { table: "children".to_owned(), parent: "parents".to_owned(), @@ -406,7 +431,15 @@ mod tests { keyed("grandchildren", Some(4), "children", &["child"], &["parent_id"]), ]; - assert!(introduced(&renames.apply(found(&before)), &found(&after)).is_empty()); + assert!( + introduced( + &renames.apply(found(&before)), + &found(&after), + &mut CatalogBudget::default() + ) + .unwrap() + .is_empty() + ); } #[test] @@ -419,9 +452,21 @@ mod tests { violation("tags", None, "posts"), ]; - assert!(introduced(&found(&before), &found(&same)).is_empty()); - assert_eq!(introduced(&found(&before), &found(&more)).len(), 1); - assert_eq!(introduced(&found(&before), &found(&more))[0].count, 1); + assert!( + introduced(&found(&before), &found(&same), &mut CatalogBudget::default()) + .unwrap() + .is_empty() + ); + assert_eq!( + introduced(&found(&before), &found(&more), &mut CatalogBudget::default()) + .unwrap() + .len(), + 1 + ); + assert_eq!( + introduced(&found(&before), &found(&more), &mut CatalogBudget::default()).unwrap()[0].count, + 1 + ); } #[test] @@ -449,7 +494,7 @@ mod tests { after.push(checked); assert_eq!( - introduced(&before, &after), + introduced(&before, &after, budget).unwrap(), [ IntroducedViolations { table: "t".to_owned(), diff --git a/core/crates/drivers/sqlite/src/write_batch.rs b/core/crates/drivers/sqlite/src/write_batch.rs index eaadf4c9bc..40caef5aa7 100644 --- a/core/crates/drivers/sqlite/src/write_batch.rs +++ b/core/crates/drivers/sqlite/src/write_batch.rs @@ -165,7 +165,13 @@ impl Write<'_> { return Err(self.failure(self.batch.steps.len(), error, fate)); } }; - let introduced = introduced(&baseline, &after); + let introduced = match introduced(&baseline, &after, &mut budget) { + Ok(introduced) => introduced, + Err(error) => { + let fate = self.rollback(transaction).await; + return Err(self.failure(self.batch.steps.len(), error, fate)); + } + }; if !introduced.is_empty() { self.rollback(transaction).await; return Err(DriverError::RolledBack { diff --git a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md index a32b37065b..dc0a2b6645 100644 --- a/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md +++ b/core/docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md @@ -28,6 +28,7 @@ A file engine reads its catalog from the file the user opened, and the file choo - Large real schemas read as before, and the keys into a table read faster: 5,000 children in 0.2 s, against 1.4 s. - One value is read before it is charged, and sqlx queues up to 50 rows ahead of the reader. SQLite bounds a value at `SQLITE_MAX_LENGTH`, and a CREATE TABLE over 64 KiB is read alone. - In a DuckDB file with a crafted table, every read that touches constraints fails, the columns of any table among them, since DuckDB builds the crafted table's rows for each. The table list still reads. -- A save on a table with millions of rows that already break its keys rolls back rather than compare them. +- DuckDB's own catalog functions, `duckdb_columns()` and `duckdb_constraints()`, build a row for every column and constraint of every table before a filter, each with its table's name, and a column can repeat a constraint. The bound counts table names and the text, not a long column name repeated in its constraints; that work is DuckDB's. +- A save on a table with millions of rows that already break its keys rolls back rather than compare them. Its error lists the first ten tables and parents. - Not budgeted: the schema list, reads of one value, and the SQL dumps, which read each object once. - The apps word the code: Linux in `linux/crates/app/src/ui/error_text/driver.rs`, Windows in `Resources.resw`.