Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ How an app reaches a database: the contract in `tablepro-core`, a transport that

### The contract

`DatabaseDriver` describes an engine before any connection exists, through `DriverCapabilities`, and opens a `Connection`. The `Connection` serves the workspace from a pool; every call takes `CallOptions` (a deadline and a cancel token) and runs on a task the connection owns (`OwnedCalls`), so a stop reaches the server instead of leaving the statement running. Saves report what they left in the database through a `WriteLedger`. Each editor tab gets an `EditorSession` of its own that streams results into a `ResultSink`, and the `Session` that opens it puts a `ByteLimitSink` in front of that sink, so a run stops keeping rows once they pass `RunOptions::byte_limit` bytes across its results, the row that crosses it kept whole ([ADR 0020](docs/decisions/0020-a-run-keeps-a-byte-limit-of-rows.md)). [docs/adding-drivers.md](../linux/docs/adding-drivers.md) is the full contract, [ADR 0016](../linux/docs/decisions/0016-driver-contract-v2.md) the reasons for it, and the engine pages say how each driver meets it: [PostgreSQL](../linux/docs/drivers/postgresql.md), [MySQL and MariaDB](../linux/docs/drivers/mysql.md), [SQLite](../linux/docs/drivers/sqlite.md), [DuckDB](../linux/docs/drivers/duckdb.md), [SQL Server](../linux/docs/drivers/sql-server.md), [ClickHouse](../linux/docs/drivers/clickhouse.md), [Cassandra and ScyllaDB](../linux/docs/drivers/cassandra.md), [Redis](../linux/docs/drivers/redis.md), [MongoDB](../linux/docs/drivers/mongodb.md). An engine without SQL says so through `DriverCapabilities::data_model`: the workspace lists a key-value engine's keys and opens them in key tabs ([ADR 0021](../linux/docs/decisions/0021-engines-without-sql.md)), and opens a document engine's collections in table tabs whose statements are the engine's shell calls ([ADR 0026](../linux/docs/decisions/0026-documents-in-the-table-tab.md)).
`DatabaseDriver` describes an engine before any connection exists, through `DriverCapabilities`, and opens a `Connection`. The `Connection` serves the workspace from a pool; every call takes `CallOptions` (a deadline and a cancel token) and runs on a task the connection owns (`OwnedCalls`), so a stop reaches the server instead of leaving the statement running. Saves report what they left in the database through a `WriteLedger`. Each editor tab gets an `EditorSession` of its own that streams results into a `ResultSink`, and the `Session` that opens it puts a `ByteLimitSink` in front of that sink, so a run stops keeping rows once they pass `RunOptions::byte_limit` bytes across its results, the row that crosses it kept whole ([ADR 0020](docs/decisions/0020-a-run-keeps-a-byte-limit-of-rows.md)). A file engine's catalog read holds at most `ByteLimit::CATALOG_DEFAULT` bytes of names and definitions through `CatalogBudget`, and fails `DriverError::CatalogTooLarge` past it ([ADR 0029](docs/decisions/0029-a-catalog-read-keeps-a-byte-budget.md)). [docs/adding-drivers.md](../linux/docs/adding-drivers.md) is the full contract, [ADR 0016](../linux/docs/decisions/0016-driver-contract-v2.md) the reasons for it, and the engine pages say how each driver meets it: [PostgreSQL](../linux/docs/drivers/postgresql.md), [MySQL and MariaDB](../linux/docs/drivers/mysql.md), [SQLite](../linux/docs/drivers/sqlite.md), [DuckDB](../linux/docs/drivers/duckdb.md), [SQL Server](../linux/docs/drivers/sql-server.md), [ClickHouse](../linux/docs/drivers/clickhouse.md), [Cassandra and ScyllaDB](../linux/docs/drivers/cassandra.md), [Redis](../linux/docs/drivers/redis.md), [MongoDB](../linux/docs/drivers/mongodb.md). An engine without SQL says so through `DriverCapabilities::data_model`: the workspace lists a key-value engine's keys and opens them in key tabs ([ADR 0021](../linux/docs/decisions/0021-engines-without-sql.md)), and opens a document engine's collections in table tabs whose statements are the engine's shell calls ([ADR 0026](../linux/docs/decisions/0026-documents-in-the-table-tab.md)).

### SQL scripts

Expand Down
2 changes: 2 additions & 0 deletions core/crates/core/src/byte_limit.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ pub struct ByteLimit(NonZeroU64);
impl ByteLimit {
// SI bytes, as the Linux banner prints them: "256.0 MB".
pub const EDITOR_DEFAULT: Self = Self(NonZeroU64::new(256_000_000).expect("256 MB is not zero"));
// About a thousand times the largest real table's columns, keys and indexes, and a million table names.
pub const CATALOG_DEFAULT: Self = Self(NonZeroU64::new(64_000_000).expect("64 MB is not zero"));

pub fn new(bytes: u64) -> Option<Self> {
NonZeroU64::new(bytes).map(Self)
Expand Down
142 changes: 142 additions & 0 deletions core/crates/core/src/catalog_budget.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
use crate::byte_limit::ByteLimit;
use crate::error::DriverError;

// A file's catalog can repeat one long name across thousands of keys; counted as `retained_bytes` counts text.
#[derive(Debug)]
pub struct CatalogBudget {
limit: u64,
held: u64,
}

impl CatalogBudget {
pub fn new(limit: ByteLimit) -> Self {
Self {
limit: limit.get(),
held: 0,
}
}

pub fn charge(&mut self, bytes: usize) -> Result<(), DriverError> {
self.held = self.held.saturating_add(u64::try_from(bytes).unwrap_or(u64::MAX));
match self.held > self.limit {
true => Err(DriverError::CatalogTooLarge {
limit_bytes: self.limit,
}),
false => Ok(()),
}
}

// For work the engine does and lets go of within the read, such as a statement it compiles: it has to fit beside
// what the read holds, and holds nothing after.
pub fn fits(&self, bytes: usize) -> Result<(), DriverError> {
match self.held.saturating_add(u64::try_from(bytes).unwrap_or(u64::MAX)) > self.limit {
true => Err(DriverError::CatalogTooLarge {
limit_bytes: self.limit,
}),
false => Ok(()),
}
}

pub fn charge_text(&mut self, text: &str) -> Result<(), DriverError> {
self.charge(text_bytes(text))
}

pub fn keep(&mut self, text: String) -> Result<String, DriverError> {
self.charge_text(&text)?;
Ok(text)
}

pub fn keep_optional(&mut self, text: Option<String>) -> Result<Option<String>, DriverError> {
text.map(|text| self.keep(text)).transpose()
}

pub fn charge_texts<'a>(&mut self, texts: impl IntoIterator<Item = &'a String>) -> Result<(), DriverError> {
texts.into_iter().try_for_each(|text| self.charge_text(text))
}

// For what the read drops before it reads the next, such as each child's CREATE TABLE.
pub fn release(&mut self, bytes: usize) {
self.held = self.held.saturating_sub(u64::try_from(bytes).unwrap_or(u64::MAX));
}

pub fn release_text(&mut self, text: &str) {
self.release(text_bytes(text));
}
}

impl Default for CatalogBudget {
fn default() -> Self {
Self::new(ByteLimit::CATALOG_DEFAULT)
}
}

fn text_bytes(text: &str) -> usize {
size_of::<String>().saturating_add(text.len())
}

#[cfg(test)]
mod tests {
use super::*;

const SLOT: usize = size_of::<String>();

fn budget(bytes: u64) -> CatalogBudget {
CatalogBudget::new(ByteLimit::new(bytes).unwrap())
}

#[test]
fn a_text_costs_its_slot_and_its_length() {
let mut budget = budget((SLOT + 5) as u64);

assert_eq!(budget.keep("héll".to_owned()).as_deref(), Ok("héll"));
assert_eq!(
budget.charge_text("x"),
Err(DriverError::CatalogTooLarge {
limit_bytes: (SLOT + 5) as u64
})
);
}

#[test]
fn the_limit_itself_still_fits() {
let mut budget = budget(100);

assert_eq!(budget.charge(100), Ok(()));
assert!(budget.charge(1).is_err());
}

#[test]
fn a_released_text_makes_room_again() {
let mut budget = budget((SLOT * 2 + 8) as u64);
let text = "a".repeat(8);

budget.charge_text(&text).unwrap();
budget.release_text(&text);
budget.charge_text(&text).unwrap();
assert_eq!(budget.keep_optional(Some(String::new())), Ok(Some(String::new())));
assert!(budget.charge_texts([&"b".to_owned()]).is_err());
}

#[test]
fn work_that_fits_beside_what_is_held_holds_nothing_after() {
let mut budget = budget(100);
budget.charge(60).unwrap();

assert_eq!(budget.fits(40), Ok(()));
assert!(budget.fits(41).is_err());
assert_eq!(budget.charge(40), Ok(()), "fits charged nothing");
}

#[test]
fn the_default_is_the_catalog_limit() {
let mut budget = CatalogBudget::default();

assert_eq!(budget.charge(64_000_000), Ok(()));
assert_eq!(
budget.charge(1),
Err(DriverError::CatalogTooLarge {
limit_bytes: 64_000_000
})
);
}
}
13 changes: 12 additions & 1 deletion core/crates/core/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,10 @@ pub enum DriverError {

#[error("this engine has no {feature}")]
Unsupported { feature: UnsupportedFeature },

// A crafted file can repeat a long name across thousands of keys, so a catalog read stops at a byte budget.
#[error("the catalog read would hold more than {limit_bytes} bytes")]
CatalogTooLarge { limit_bytes: u64 },
}

impl DriverError {
Expand Down Expand Up @@ -176,6 +180,7 @@ impl DriverError {
Self::Decode { .. } => "driver.decode",
Self::Protocol(..) => "driver.protocol",
Self::Unsupported { .. } => "driver.unsupported",
Self::CatalogTooLarge { .. } => "driver.catalog_too_large",
}
}
}
Expand Down Expand Up @@ -216,7 +221,9 @@ impl DriverError {
| Self::FileIo { .. } => ErrorCategory::File,
Self::Busy(_) | Self::EditorSessionBusy => ErrorCategory::Busy,
Self::Config(_) | Self::Unsupported { .. } | Self::DatabaseNotFound { .. } => ErrorCategory::Configuration,
Self::GuardUnsupported { .. } | Self::Decode { .. } | Self::Protocol(_) => ErrorCategory::Internal,
Self::GuardUnsupported { .. } | Self::Decode { .. } | Self::Protocol(_) | Self::CatalogTooLarge { .. } => {
ErrorCategory::Internal
}
}
}

Expand Down Expand Up @@ -366,6 +373,10 @@ mod tests {
(DriverError::Cancelled.code(), "driver.cancelled"),
(DriverError::auth(None).code(), "driver.auth"),
(DriverError::server("no such table").code(), "driver.server"),
(
DriverError::CatalogTooLarge { limit_bytes: 1 }.code(),
"driver.catalog_too_large",
),
(DriverError::Config(ConfigError::EmptyHost).code(), "driver.config"),
(ConfigError::EmptyHost.code(), "config.empty_host"),
(
Expand Down
2 changes: 2 additions & 0 deletions core/crates/core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ mod byte_limit;
mod byte_limited_sink;
mod call_options;
mod capabilities;
mod catalog_budget;
mod certificate_file_problem;
mod client_identity;
pub mod column;
Expand Down Expand Up @@ -114,6 +115,7 @@ pub use capabilities::{
CommitResolutionSupport, DataModel, DriverCapabilities, EndpointKind, EstimateQuality, ReadOnlyEnforcement,
ScriptExecution, StatementStop,
};
pub use catalog_budget::CatalogBudget;
pub use certificate_file_problem::CertificateFileProblem;
pub use client_identity::ClientIdentity;
pub use column::ColumnInfo;
Expand Down
Loading
Loading