Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion core/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Each boundary error has a stable `code()`, such as `driver.timeout` or `ssh.host

`ffi` is the library an app outside Rust loads: the Windows app as a DLL, the Apple apps as a static library in an XCFramework, built without the `engines` feature by `apple/scripts/build-core.sh`. Core ADRs 0002, on the bindings, 0003, on result windows, and 0009, on the macOS slices, hold the reasons.

- One UniFFI API. `Core` lists the engines with the family each is listed under, opens a file, connects a saved connection or tests one from a form, and parses a connection URL with `tablepro_core::connection_url`, refusing one that needs a setting its record has no field for. Its `ConnectionLibrary` reads and changes the saved connections, groups, favourites and tags in `connections.json` and stores their secrets, and gives the Apple apps the library's CloudKit records and merges a pull into it (`tablepro_storage::cloudkit`). A `DatabaseSession` reads the catalog one level at a time, writes a table's sorted page at an offset and counts the table, writes a query re-run with a sort (`docs/decisions/0007-sorting-a-result.md`), reports and waits on the session's health, and opens a `QueryEditor`. `QueryEditor::run` returns a `QueryRun` at once; the app drains its events after a `RunListener` call, which comes only when no earlier one is waiting to be drained, and the last says which transaction the editor is left in. Each result set is a `ResultSet`, whose `sorted` gives the rows it holds in a column's order.
- One UniFFI API. `Core` lists the engines with the family each is listed under, opens a file, connects a saved connection or tests one from a form, and parses a connection URL with `tablepro_core::connection_url`, refusing one that needs a setting its record has no field for. Its `ConnectionLibrary` reads and changes the saved connections, groups, favourites and tags in `connections.json` and stores their secrets, and gives the Apple apps the library's CloudKit records and merges a pull into it (`tablepro_storage::cloudkit`). A `DatabaseSession` reads the catalog one level at a time, writes a table's sorted page at an offset and counts the table, writes a query re-run with a sort (`docs/decisions/0007-sorting-a-result.md`), says what Safe Mode lets an editor run do at the session's level and takes a level saved while it is open (`gate_run`, over `SafeModeLevel::gate`, and `set_safe_mode`), reports and waits on the session's health, and opens a `QueryEditor`. `QueryEditor::run` returns a `QueryRun` at once; the app drains its events after a `RunListener` call, which comes only when no earlier one is waiting to be drained, and the last says which transaction the editor is left in. Each result set is a `ResultSet`, whose `sorted` gives the rows it holds in a column's order.
- The grid reads a `ResultSet` through the C ABI in `src/window_abi.rs`: `tablepro_cursor_fetch` packs the display text of one visible range, and `tablepro_cursor_free` ends the cursor.
- The library owns one tokio runtime. Each async export runs on it and is aborted when the foreign side drops the call; a `CancelToken` stops it sooner, since UniFFI cannot cancel.
- A result set takes rows as the driver reads them. Each column keeps a kind byte, a raw `i64` and the exact text `tablepro_core::export::value_text` writes, or a binary value's bytes, in chunks of 65,536 rows. A window only cuts and copies.
Expand Down
19 changes: 0 additions & 19 deletions core/crates/drivers/clickhouse/src/call_client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -55,25 +55,6 @@ impl CallClient {
}
}

/// A read as the server runs it: never a write, and stopped by the
/// server itself once its bound passes. `None` leaves it unbounded.
/// `readonly` is the level `ClickHouseEndpoint::read_level` chose; none
/// leaves the login's own.
pub(crate) fn reading(
client: &clickhouse::Client,
bound: Option<Duration>,
readonly: Option<&str>,
) -> clickhouse::Client {
let client = client
.clone()
.with_setting("timeout_overflow_mode", "throw")
.with_setting("max_execution_time", whole_seconds(bound).to_string());
match readonly {
Some(level) => client.with_setting("readonly", level),
None => client,
}
}

/// The nearer of the call's deadline and the configured timeout.
pub(crate) fn read_bound(deadline: Option<Instant>, configured: Option<Duration>) -> Option<Duration> {
let remaining = deadline.map(|deadline| deadline.saturating_duration_since(Instant::now()));
Expand Down
72 changes: 65 additions & 7 deletions core/crates/drivers/clickhouse/src/cell_seed.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ pub(crate) enum CellPlan {
Integer,
Float32,
Float64,
Decimal,
Decimal {
scale: u32,
},
Bool,
Date,
DateTime,
Expand Down Expand Up @@ -58,7 +60,9 @@ impl CellPlan {
| DataTypeNode::Int256 => Self::Integer,
DataTypeNode::Float32 | DataTypeNode::BFloat16 => Self::Float32,
DataTypeNode::Float64 => Self::Float64,
DataTypeNode::Decimal(..) => Self::Decimal,
DataTypeNode::Decimal(_, scale, _) => Self::Decimal {
scale: u32::from(*scale),
},
DataTypeNode::Bool => Self::Bool,
DataTypeNode::Date | DataTypeNode::Date32 => Self::Date,
DataTypeNode::DateTime(_) | DataTypeNode::DateTime64(..) => Self::DateTime,
Expand Down Expand Up @@ -103,7 +107,7 @@ impl CellPlan {
Self::Integer => "Integer".to_owned(),
Self::Float32 => "Float32".to_owned(),
Self::Float64 => "Float64".to_owned(),
Self::Decimal => "Decimal".to_owned(),
Self::Decimal { .. } => "Decimal".to_owned(),
Self::Bool => "Bool".to_owned(),
Self::Date => "Date".to_owned(),
Self::DateTime => "DateTime".to_owned(),
Expand Down Expand Up @@ -153,7 +157,7 @@ impl<'de> DeserializeSeed<'de> for CellSeed<'_> {
let text = Option::<String>::deserialize(deserializer)?;
Ok(text.map_or(Value::Null, |text| temporal_or_uuid(plan, &text)))
}
CellPlan::Integer | CellPlan::Float32 | CellPlan::Float64 | CellPlan::Decimal | CellPlan::Bool => {
CellPlan::Integer | CellPlan::Float32 | CellPlan::Float64 | CellPlan::Decimal { .. } | CellPlan::Bool => {
let raw = <&RawValue>::deserialize(deserializer)?;
Ok(number(plan, raw.get()))
}
Expand All @@ -178,8 +182,13 @@ fn undecodable(plan: &CellPlan, reason: UndecodableReason) -> Value {
/// A number token, quoted or bare: the server quotes 64-bit and wider
/// integers, and the non-finite floats, so that JSON readers keep them.
fn number(plan: &CellPlan, token: &str) -> Value {
// A Nullable column takes its null before this, so a float's null is a NaN or an infinity, which a login that may
// not set `output_format_json_quote_denormals` gets as null.
if token == "null" {
return Value::Null;
return match plan {
CellPlan::Float32 | CellPlan::Float64 => undecodable(plan, UndecodableReason::OutOfRange),
_ => Value::Null,
};
}
let text = token
.strip_prefix('"')
Expand All @@ -189,7 +198,9 @@ fn number(plan: &CellPlan, token: &str) -> Value {
CellPlan::Integer => integer(text),
CellPlan::Float32 => f32::from_str(text).ok().map(Value::Float32),
CellPlan::Float64 => f64::from_str(text).ok().map(Value::Float64),
CellPlan::Decimal => SqlDecimal::from_str(text).ok().map(Value::Decimal),
CellPlan::Decimal { scale } => SqlDecimal::from_str(text)
.ok()
.map(|decimal| Value::Decimal(at_scale(decimal, *scale))),
CellPlan::Bool => match text {
"true" | "1" => Some(Value::Bool(true)),
"false" | "0" => Some(Value::Bool(false)),
Expand All @@ -200,6 +211,19 @@ fn number(plan: &CellPlan, token: &str) -> Value {
parsed.unwrap_or_else(|| undecodable(plan, UndecodableReason::OutOfRange))
}

// A login that may not set `output_format_decimal_trailing_zeros` gets `1.5` for a `Decimal(10, 2)` holding 1.50, which
// is not equal to the 1.50 an edit of it parses.
fn at_scale(decimal: SqlDecimal, scale: u32) -> SqlDecimal {
let padded = match decimal.parts() {
Some((negative, coefficient, written)) if written < scale => {
let zeros = "0".repeat(usize::try_from(scale - written).unwrap_or(0));
SqlDecimal::from_parts(negative, &format!("{coefficient}{zeros}"), scale).ok()
}
_ => None,
};
padded.unwrap_or(decimal)
}

/// An integer of any width, in the smallest variant that holds it, the
/// way an edit parses one, so a value read back compares equal to the
/// same value typed in.
Expand Down Expand Up @@ -377,11 +401,45 @@ mod tests {
assert_eq!(wide.to_string(), widest);
let decimal = "12345678901234567890123456789012345678901234567890123456.12345678901234567890";
assert_eq!(
read(&CellPlan::Decimal, decimal.as_bytes()),
read(&CellPlan::Decimal { scale: 20 }, decimal.as_bytes()),
Value::Decimal(decimal.parse().expect("a decimal"))
);
}

#[test]
fn a_decimal_written_without_its_trailing_zeros_reads_at_its_columns_scale() {
let plan = CellPlan::Decimal { scale: 2 };
let at = |text: &str| Value::Decimal(text.parse().expect("a decimal"));

assert_eq!(read(&plan, b"1.5"), at("1.50"));
assert_eq!(read(&plan, b"-0.1"), at("-0.10"));
assert_eq!(read(&plan, b"7"), at("7.00"));
assert_eq!(read(&plan, b"0"), at("0.00"));
assert_eq!(read(&plan, b"1.25"), at("1.25"));
assert_eq!(read(&CellPlan::Decimal { scale: 0 }, b"12"), at("12"));
}

#[test]
fn a_float_the_server_wrote_as_null_is_out_of_range_unless_its_column_is_nullable() {
let out_of_range = |type_name: &str| {
Value::Undecodable(Box::new(UndecodedValue {
type_name: type_name.to_owned(),
reason: UndecodableReason::OutOfRange,
}))
};

assert_eq!(read(&CellPlan::Float64, b"null"), out_of_range("Float64"));
assert_eq!(read(&CellPlan::Float32, b"null"), out_of_range("Float32"));
assert_eq!(
read(&CellPlan::Array(Box::new(CellPlan::Float64)), b"[1.5,null]"),
Value::Array(vec![Value::Float64(1.5), out_of_range("Float64")])
);
assert_eq!(
read(&CellPlan::Nullable(Box::new(CellPlan::Float64)), b"null"),
Value::Null
);
}

#[test]
fn non_finite_floats_arrive_quoted() {
let Value::Float32(nan) = read(&CellPlan::Float32, b"\"nan\"") else {
Expand Down
120 changes: 102 additions & 18 deletions core/crates/drivers/clickhouse/src/client_factory.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ const POOL_IDLE_TIMEOUT: Duration = Duration::from_secs(2);
/// certificate's common name instead of a password.
const CERTIFICATE_AUTH_HEADER: &str = "X-ClickHouse-SSL-Certificate-Auth";

/// The settings every request carries.
const COMMON_SETTINGS: [(&str, &str); 7] = [
// A Decimal is written at its column's scale, so a value read back
// keeps the scale it was stored with and compares equal to it.
Expand All @@ -47,16 +46,62 @@ const COMMON_SETTINGS: [(&str, &str); 7] = [
// `replication_wait_for_inactive_replica_timeout`, two minutes by
// default, and then end with UNFINISHED although the live replicas
// applied it.
("mutations_sync", "1"),
(MUTATIONS_SYNC, "1"),
// A read whose connection the app dropped stops on the server even
// when no KILL could be sent.
("cancel_http_readonly_queries_on_client_close", "1"),
];

pub(crate) const MAX_EXECUTION_TIME: &str = "max_execution_time";
pub(crate) const TIMEOUT_OVERFLOW_MODE: &str = "timeout_overflow_mode";
pub(crate) const MUTATIONS_SYNC: &str = "mutations_sync";

// A login at `readonly=1`, or one a profile constraint holds to a value, refuses the whole request that changes such a
// setting (164, 452), so the driver sends only what `system.settings` says this login may change.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct ChangeableSettings(Vec<&'static str>);

impl ChangeableSettings {
pub(crate) fn all() -> Self {
Self(Self::asked().collect())
}

pub(crate) fn asked() -> impl Iterator<Item = &'static str> {
COMMON_SETTINGS
.into_iter()
.map(|(name, _)| name)
.chain([MAX_EXECUTION_TIME, TIMEOUT_OVERFLOW_MODE])
}

pub(crate) fn listed(names: &[String]) -> Self {
Self(
Self::asked()
.filter(|asked| names.iter().any(|name| name == asked))
.collect(),
)
}

pub(crate) fn allows(&self, name: &str) -> bool {
self.0.contains(&name)
}

pub(crate) fn common(&self) -> impl Iterator<Item = (&'static str, &'static str)> + '_ {
COMMON_SETTINGS.into_iter().filter(|(name, _)| self.allows(name))
}

// A bound without the mode would let a profile's `timeout_overflow_mode=break` end a read at it with the rows read
// so far, as if they were all.
pub(crate) fn bounds_reads(&self) -> bool {
self.allows(MAX_EXECUTION_TIME) && self.allows(TIMEOUT_OVERFLOW_MODE)
}
}

// `readonly=2` still lets `INSERT INTO FUNCTION file(...)` write a file (measured on 26.8); 1 refuses it and still
// takes the settings a request carries. A login already at 1 or 2 refuses any change of `readonly`, so it keeps its own.
// takes the settings a request carries. A login at 2 refuses any change of `readonly`. One at 1 takes the 1 as no
// change, and it puts back a session that ran `SET readonly = 0`, which a profile marking `readonly`
// `changeable_in_readonly` lets it run.
pub(crate) fn strict_readonly(login_level: u8) -> Option<&'static str> {
(login_level == 0).then_some("1")
(login_level <= 1).then_some("1")
}

// User SQL and everything on a read-only connection is held strictly. The app's own reads keep 2, which still lets
Expand All @@ -69,17 +114,17 @@ pub(crate) fn read_level(read_only: bool, user_sql: bool, login_level: u8) -> Op
}

pub(crate) fn read_only_settings(login_level: u8) -> Vec<(&'static str, &'static str)> {
let mut settings = vec![("allow_ddl", "0")];
let mut settings = Vec::new();
// A login at 1 refuses the change, and `readonly=1` refuses the DDL it would.
if login_level != 1 {
settings.push(("allow_ddl", "0"));
}
settings.extend(strict_readonly(login_level).map(|level| ("readonly", level)));
settings
}

/// The client every request of one target goes through: hyper's pool
/// over the target's transport, the target's credentials and database,
/// and the settings every request carries.
///
/// The clickhouse crate's own TLS stays unused: TLS is negotiated here,
/// over the transport's stream, against the trust the target names.
// No setting goes on here: the login's first answer says which of the driver's it may change. The crate's own TLS
// stays unused, since TLS is negotiated over the transport's stream against the trust the target names.
pub(crate) fn base_client(
target: &NetworkTarget,
liveness: &LivenessPolicy,
Expand All @@ -106,14 +151,11 @@ pub(crate) fn base_client(
};
// A typed fetch then reads RowBinary, which names no column types, so no type name the server sends is parsed,
// however deep it nests, nor a list sized by the column count the server claims.
let mut client = client
let client = client
.with_url(service_url(scheme, target))
.with_validation(false)
.with_product_info(APPLICATION_NAME, environment.client_version.clone());
for (name, value) in COMMON_SETTINGS {
client = client.with_setting(name, value);
}
client = with_credentials(client, target)?;
let mut client = with_credentials(client, target)?;
if let Some(database) = target.database.as_deref().filter(|name| !name.is_empty()) {
client = client.with_database(database);
}
Expand Down Expand Up @@ -176,10 +218,46 @@ mod tests {

#[test]
fn a_login_with_a_readonly_level_of_its_own_keeps_it() {
assert_eq!(read_only_settings(1), vec![("allow_ddl", "0")]);
assert_eq!(read_only_settings(2), vec![("allow_ddl", "0")]);
}

#[test]
fn a_read_only_connection_holds_a_login_at_readonly_1_to_its_level_without_allow_ddl() {
assert_eq!(read_only_settings(1), vec![("readonly", "1")]);
}

#[test]
fn only_the_drivers_own_settings_the_server_lists_are_changeable() {
let listed = ["mutations_sync", "max_threads", MAX_EXECUTION_TIME].map(str::to_owned);

let changeable = ChangeableSettings::listed(&listed);

assert!(changeable.allows("mutations_sync"));
assert!(changeable.allows(MAX_EXECUTION_TIME));
assert!(!changeable.allows("max_threads"));
assert!(!changeable.allows("output_format_json_quote_denormals"));
assert_eq!(changeable.common().collect::<Vec<_>>(), vec![("mutations_sync", "1")]);
}

#[test]
fn a_read_is_bounded_on_the_server_only_with_its_overflow_mode() {
let bound_only = ChangeableSettings::listed(&[MAX_EXECUTION_TIME.to_owned()]);
let both = ChangeableSettings::listed(&[MAX_EXECUTION_TIME.to_owned(), TIMEOUT_OVERFLOW_MODE.to_owned()]);

assert!(!bound_only.bounds_reads());
assert!(both.bounds_reads());
assert!(ChangeableSettings::all().bounds_reads());
}

#[test]
fn a_login_that_may_change_nothing_gets_no_common_setting() {
let changeable = ChangeableSettings::listed(&[]);

assert_eq!(changeable.common().count(), 0);
assert!(!changeable.bounds_reads());
assert_eq!(ChangeableSettings::all().common().count(), COMMON_SETTINGS.len());
}

#[test]
fn user_sql_is_held_to_readonly_1_on_any_connection() {
assert_eq!(read_level(false, true, 0), Some("1"));
Expand All @@ -193,7 +271,13 @@ mod tests {
assert_eq!(read_level(false, true, 2), None);
assert_eq!(read_level(false, false, 2), Some("2"));
assert_eq!(read_level(false, false, 1), None);
assert_eq!(read_level(true, true, 1), None);
}

#[test]
fn user_sql_and_a_read_only_connection_hold_a_login_at_readonly_1_to_its_level() {
assert_eq!(read_level(false, true, 1), Some("1"));
assert_eq!(read_level(true, true, 1), Some("1"));
assert_eq!(read_level(true, false, 1), Some("1"));
}

#[test]
Expand Down
Loading
Loading