From 0af920cd86a0eaf697765212ecce61636fc5f20e Mon Sep 17 00:00:00 2001 From: Lee Hannigan Date: Fri, 23 Oct 2026 21:14:17 +0000 Subject: [PATCH 01/15] feat: store and report WarmThroughput on tables and global secondary indexes Since Terraform AWS provider 6.13.0 the aws_dynamodb_table resource waits for WarmThroughput.Status to read ACTIVE after every table and index create or update and treats a description without the member as not found, so an apply against ExtendDB polled until it timed out (#462). ExtendDB now accepts WarmThroughput on CreateTable, UpdateTable and GlobalSecondaryIndexUpdates (Create and Update), stores what the client sent, and reports the member with the shape, defaults, lifecycle and validation measured on Amazon DynamoDB on 2026-10-23: * Absent while the table or index is CREATING unless the request named it (then echoed as sent with Status UPDATING); 12000 read and 4000 write units on an on-demand table and its indexes, the provisioned units on a provisioned table and index; present with Status on every CreateTable, DescribeTable, UpdateTable and DeleteTable description. * An update reports the previous values with Status UPDATING for the control plane delay, then the new values ACTIVE; a decrease is validated against the reported values. A switch to PAY_PER_REQUEST raises a stored value to the on-demand minimum. A capacity change on a never-set provisioned table moves the reported default with the units. * Refusals carry the service's sentences: below the on-demand minimum, below the provisioned units, below the current value, each naming "table" or "index "; one action per GlobalSecondaryIndexUpdate element; one update per index; on an on-demand table an index Update may only carry OnDemandThroughput or WarmThroughput. * UpdateTable with a table-level WarmThroughput refuses companions in the measured order: Server-Side Encryption, TableClass, DeletionProtection (each with its own sentence), OnDemandThroughput (needs BillingMode), a GlobalSecondaryIndexUpdates Delete, then BillingMode, ProvisionedThroughput, StreamSpecification or vector index updates ("WarmThroughput must be the only operation in the request"). GlobalSecondaryIndexUpdates Update beside it is accepted. A request carrying only WarmThroughput is accepted. SSESpecification is now read on UpdateTableInput so it can be refused rather than dropped. * A restored table starts with the defaults; a GlobalSecondaryIndexOverride carrying WarmThroughput is refused with the service's sentence. Nothing acts on the values. There is no pre-warming and no scaling at the database level, and the stored numbers have no effect on throughput, throttling or storage; they exist so clients that read the member work. The account maximum the service enforces is not enforced here. Storage: one nullable warm_throughput column on tables and indexes on SQLite (schema plus shape convergence), PostgreSQL (001_schema.sql plus shape steps for existing catalogs) and Cassandra (migration V006, catalog version 0.0.5 so extenddb migrate applies it); a document field on MongoDB. Each backend stores one JSON record (the client's sides, the previous sides, and the UPDATING deadline); the wire member is derived in one place, TableDescription::prepare_for_response. On PostgreSQL, UpdateTable builds a new GSI before it answers (#401), which takes longer than the 0.25 s UPDATING window the index row is inserted with. The window of each GSI the request created with WarmThroughput starts again once the build has run, so the response reports it UPDATING as the service does. Tests: unit tests for the record, defaults and every rule; three dual-target wire files (lifecycle and validation, update shapes and billing changes, backup and restore) covering every measured row; a Terraform apply of the configuration from the provider issue plus a table with a GSI on every backend, with Terraform 1.17.0 and provider 6.70.0 pinned and checksum-asserted in CI; a Cassandra test for a catalog that has not run V006. Measured on live DynamoDB: the three wire files pass against the service (12, 13 and 1 tests) and against SQLite, PostgreSQL, MongoDB and Cassandra. Cassandra lib (61) and direct_integration (151) suites pass. Negative controls: with the member removed the Terraform apply fails with the provider's "couldn't find resource (21 retries)" after about three and a half minutes, which is the hang from the issue; with the validation removed the three refusal rows fail; five further built mutants (table member, index member, UPDATING deadline, only-operation rule, swapped defaults) each fail the wire file. Documented in docs/differences-from-dynamodb.md (Capacity and Throttling) and docs/getting-started.md. Closes #462. --- .github/workflows/integration-cassandra.yml | 4 +- .github/workflows/integration-mongodb.yml | 4 +- .github/workflows/integration.yml | 10 +- crates/core/src/types/mod.rs | 5 + crates/core/src/types/table.rs | 82 ++- crates/core/src/types/warm_throughput.rs | 345 ++++++++++++ .../core/src/types/warm_throughput/tests.rs | 338 ++++++++++++ crates/core/src/validation/mod.rs | 21 + crates/core/src/validation/warm_throughput.rs | 381 ++++++++++++++ .../src/validation/warm_throughput/tests.rs | 493 ++++++++++++++++++ crates/engine/src/backup/overrides.rs | 18 +- crates/engine/src/backup/tests.rs | 9 +- crates/engine/src/gsi_updates.rs | 333 ++++++++++++ crates/engine/src/import_export.rs | 1 + crates/engine/src/lib.rs | 2 + crates/engine/src/update_table.rs | 166 +++--- crates/engine/src/warm_throughput.rs | 452 ++++++++++++++++ .../catalog/V006__warm_throughput.cql | 18 + crates/storage-cassandra/src/backup_engine.rs | 1 + crates/storage-cassandra/src/create_table.rs | 64 ++- crates/storage-cassandra/src/lib.rs | 3 +- crates/storage-cassandra/src/migrations.rs | 6 +- crates/storage-cassandra/src/table_helpers.rs | 10 +- crates/storage-cassandra/src/update_table.rs | 91 +++- .../storage-cassandra/src/warm_throughput.rs | 107 ++++ crates/storage-cassandra/tests/common/mod.rs | 6 + .../tests/direct/backup_engine.rs | 67 ++- .../tests/direct/cassandra_engine.rs | 1 + .../storage-cassandra/tests/direct/streams.rs | 1 + .../tests/direct/table_engine.rs | 1 + .../tests/direct/warm_throughput_catalog.rs | 158 ++++++ .../tests/direct_integration.rs | 2 + .../tests/ttl_integration.rs | 4 + crates/storage-mongodb/src/backup_engine.rs | 2 + .../storage-mongodb/src/backup_live_tests.rs | 4 + crates/storage-mongodb/src/lib.rs | 1 + crates/storage-mongodb/src/table_engine.rs | 31 ++ crates/storage-mongodb/src/warm_throughput.rs | 169 ++++++ .../migrations/001_schema.sql | 8 + crates/storage-postgres/src/backup_engine.rs | 1 + crates/storage-postgres/src/create_table.rs | 42 +- crates/storage-postgres/src/delete_table.rs | 5 +- crates/storage-postgres/src/lib.rs | 1 + crates/storage-postgres/src/migrations.rs | 28 + crates/storage-postgres/src/table_helpers.rs | 18 +- crates/storage-postgres/src/update_table.rs | 26 +- .../storage-postgres/src/warm_throughput.rs | 168 ++++++ .../storage-postgres/tests/key_collation.rs | 1 + crates/storage-postgres/tests/twi_conflict.rs | 1 + .../tests/vector_control_plane.rs | 4 + crates/storage-sqlite/src/backup.rs | 10 +- crates/storage-sqlite/src/bootstrapper.rs | 7 +- crates/storage-sqlite/src/create_table.rs | 41 +- crates/storage-sqlite/src/lib.rs | 1 + crates/storage-sqlite/src/schema.rs | 39 ++ crates/storage-sqlite/src/table_helpers.rs | 13 +- crates/storage-sqlite/src/update_table.rs | 21 +- crates/storage-sqlite/src/warm_throughput.rs | 308 +++++++++++ crates/storage/src/backup_definition.rs | 1 + crates/storage/src/util/mod.rs | 5 + crates/storage/src/util/warm_throughput.rs | 148 ++++++ devtools/run-tests | 28 +- docs/differences-from-dynamodb.md | 1 + docs/getting-started.md | 4 + docs/manuals/01-architecture-guide.md | 2 +- docs/manuals/07-upgrade-manual.md | 15 +- tests/python/test_warm_throughput.py | 446 ++++++++++++++++ tests/python/test_warm_throughput_backup.py | 121 +++++ tests/python/test_warm_throughput_updates.py | 420 +++++++++++++++ tests/python/warm_throughput_helpers.py | 124 +++++ tests/terraform/install-terraform.sh | 47 ++ tests/terraform/warm-throughput/main.tf | 95 ++++ tests/terraform/warm-throughput/run.sh | 89 ++++ 73 files changed, 5501 insertions(+), 199 deletions(-) create mode 100644 crates/core/src/types/warm_throughput.rs create mode 100644 crates/core/src/types/warm_throughput/tests.rs create mode 100644 crates/core/src/validation/warm_throughput.rs create mode 100644 crates/core/src/validation/warm_throughput/tests.rs create mode 100644 crates/engine/src/gsi_updates.rs create mode 100644 crates/engine/src/warm_throughput.rs create mode 100644 crates/storage-cassandra/migrations/catalog/V006__warm_throughput.cql create mode 100644 crates/storage-cassandra/src/warm_throughput.rs create mode 100644 crates/storage-cassandra/tests/direct/warm_throughput_catalog.rs create mode 100644 crates/storage-mongodb/src/warm_throughput.rs create mode 100644 crates/storage-postgres/src/warm_throughput.rs create mode 100644 crates/storage-sqlite/src/warm_throughput.rs create mode 100644 crates/storage/src/util/warm_throughput.rs create mode 100644 tests/python/test_warm_throughput.py create mode 100644 tests/python/test_warm_throughput_backup.py create mode 100644 tests/python/test_warm_throughput_updates.py create mode 100644 tests/python/warm_throughput_helpers.py create mode 100755 tests/terraform/install-terraform.sh create mode 100644 tests/terraform/warm-throughput/main.tf create mode 100755 tests/terraform/warm-throughput/run.sh diff --git a/.github/workflows/integration-cassandra.yml b/.github/workflows/integration-cassandra.yml index 08ad150a8..439088d5e 100644 --- a/.github/workflows/integration-cassandra.yml +++ b/.github/workflows/integration-cassandra.yml @@ -145,8 +145,10 @@ jobs: run: pip install -r requirements.txt - name: Build release (cassandra backend) run: cargo build --release --no-default-features --features cassandra + - name: Install Terraform (pinned version, checksum asserted) + run: echo "TERRAFORM=$(tests/terraform/install-terraform.sh "$RUNNER_TEMP/terraform")" >> "$GITHUB_ENV" - name: Run Cassandra pytest suite - run: devtools/run-cassandra-tests -- --pytest --comprehensive --parallel + run: devtools/run-cassandra-tests -- --pytest --comprehensive --terraform --parallel cassandra-rust: runs-on: extenddb_ubuntu-2404_4-core diff --git a/.github/workflows/integration-mongodb.yml b/.github/workflows/integration-mongodb.yml index 3df405a75..8eb7ac6fc 100644 --- a/.github/workflows/integration-mongodb.yml +++ b/.github/workflows/integration-mongodb.yml @@ -48,8 +48,10 @@ jobs: run: pip install -r requirements.txt - name: Build release (mongodb backend) run: cargo build --release --no-default-features --features mongodb-test-hooks + - name: Install Terraform (pinned version, checksum asserted) + run: echo "TERRAFORM=$(tests/terraform/install-terraform.sh "$RUNNER_TEMP/terraform")" >> "$GITHUB_ENV" - name: Run MongoDB pytest suite - run: devtools/run-mongodb-tests -- --pytest --comprehensive --parallel + run: devtools/run-mongodb-tests -- --pytest --comprehensive --terraform --parallel mongodb-rust: runs-on: extenddb_ubuntu-2404_4-core diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 8a4818f75..58fa92517 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -89,13 +89,16 @@ jobs: - name: Install Python dependencies run: pip install -r requirements.txt + - name: Install Terraform (pinned version, checksum asserted) + run: echo "TERRAFORM=$(tests/terraform/install-terraform.sh "$RUNNER_TEMP/terraform")" >> "$GITHUB_ENV" + - name: Run integration tests env: EXTENDDB_TEST_ENDPOINT: https://127.0.0.1:18443 EXTENDDB_ADMIN_USER: admin EXTENDDB_ADMIN_PASSWORD: ${{ steps.init.outputs.admin_password }} EXTENDDB_TEST_PG_ADMIN_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 - run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" + run: devtools/run-tests --extenddb --pytest --comprehensive --terraform --parallel --filter "not import_export" # run-tests restarts the server daemonized to apply the import/export # config, and a daemonized server logs to syslog, not the job log. Without @@ -150,12 +153,15 @@ jobs: - name: Install Python dependencies run: pip install -r requirements.txt + - name: Install Terraform (pinned version, checksum asserted) + run: echo "TERRAFORM=$(tests/terraform/install-terraform.sh "$RUNNER_TEMP/terraform")" >> "$GITHUB_ENV" + - name: Run integration tests env: EXTENDDB_TEST_ENDPOINT: https://127.0.0.1:18443 EXTENDDB_ADMIN_USER: admin EXTENDDB_ADMIN_PASSWORD: ${{ steps.init.outputs.admin_password }} - run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" + run: devtools/run-tests --extenddb --pytest --comprehensive --terraform --parallel --filter "not import_export" # Same rationale as the postgres job: the daemonized server logs to # syslog, so this dump is the only server-side evidence on failure. diff --git a/crates/core/src/types/mod.rs b/crates/core/src/types/mod.rs index e21f75110..c8c0544ea 100755 --- a/crates/core/src/types/mod.rs +++ b/crates/core/src/types/mod.rs @@ -11,6 +11,7 @@ mod query; mod stream; mod table; mod transaction; +mod warm_throughput; pub use attribute_value::AttributeValue; pub use backup::{ @@ -76,3 +77,7 @@ pub use transaction::{ TransactGetItem, TransactGetItemsInput, TransactGetItemsOutput, TransactPut, TransactUpdate, TransactWriteItem, TransactWriteItemsInput, TransactWriteItemsOutput, }; +pub use warm_throughput::{ + ON_DEMAND_WARM_READ_UNITS, ON_DEMAND_WARM_WRITE_UNITS, WarmThroughput, WarmThroughputDefaults, + WarmThroughputDescription, WarmThroughputRecord, WarmThroughputStatus, epoch_seconds_now, +}; diff --git a/crates/core/src/types/table.rs b/crates/core/src/types/table.rs index fdebcab30..2982af087 100755 --- a/crates/core/src/types/table.rs +++ b/crates/core/src/types/table.rs @@ -303,6 +303,14 @@ pub struct GsiDescription { pub item_count: i64, #[serde(rename = "IndexArn")] pub index_arn: String, + /// Derived by `TableDescription::prepare_for_response` from + /// `warm_throughput_record`; backends leave this `None`. + #[serde(rename = "WarmThroughput", skip_serializing_if = "Option::is_none")] + pub warm_throughput: Option, + /// What the backend stored for this index, if a client ever set it. Not a + /// wire member; see [`super::WarmThroughputRecord`]. Nothing acts on it. + #[serde(skip)] + pub warm_throughput_record: Option, } /// Local secondary index description returned in responses. @@ -336,6 +344,12 @@ pub struct GsiInput { skip_serializing_if = "Option::is_none" )] pub provisioned_throughput: Option, + #[serde( + rename = "WarmThroughput", + default, + skip_serializing_if = "Option::is_none" + )] + pub warm_throughput: Option, } /// Local secondary index definition for `CreateTable` requests. @@ -822,6 +836,46 @@ impl TableDescription { } else if let Some(summary) = self.restore_summary.as_mut() { summary.restore_in_progress = true; } + self.populate_warm_throughput(super::epoch_seconds_now()); + } + + /// Derive the `WarmThroughput` member of the table and of each global + /// secondary index from the stored records (see + /// [`super::WarmThroughputRecord::describe_for`] for the measured rules). + /// Local secondary indexes carry none. Reached through + /// [`Self::prepare_for_response`]; `now` is epoch seconds. + pub fn populate_warm_throughput(&mut self, now: f64) { + let pay_per_request = self + .billing_mode_summary + .as_ref() + .is_some_and(|s| s.billing_mode == BillingMode::PayPerRequest); + let table_defaults = super::WarmThroughputDefaults::for_billing( + pay_per_request, + self.provisioned_throughput.read_capacity_units, + self.provisioned_throughput.write_capacity_units, + ); + self.warm_throughput = super::WarmThroughputRecord::describe_for( + self.warm_throughput_record.as_ref(), + self.table_status == TableStatus::Creating, + table_defaults, + now, + ); + for gsi in self.global_secondary_indexes.iter_mut().flatten() { + let (rcu, wcu) = gsi.provisioned_throughput.as_ref().map_or((0, 0), |pt| { + (pt.read_capacity_units, pt.write_capacity_units) + }); + let defaults = super::WarmThroughputDefaults::for_billing(pay_per_request, rcu, wcu); + // An index created with its table is CREATING while the table is; + // one added by UpdateTable is CREATING while it backfills. + let creating = + self.table_status == TableStatus::Creating || gsi.index_status == "CREATING"; + gsi.warm_throughput = super::WarmThroughputRecord::describe_for( + gsi.warm_throughput_record.as_ref(), + creating, + defaults, + now, + ); + } } /// Populate `TableThroughputModeSummary` as an exact mirror of @@ -901,6 +955,14 @@ pub struct TableDescription { pub on_demand_throughput: Option, #[serde(rename = "RestoreSummary", skip_serializing_if = "Option::is_none")] pub restore_summary: Option, + /// Derived by [`Self::prepare_for_response`] from `warm_throughput_record`, + /// the billing mode and the provisioned units; backends leave this `None`. + #[serde(rename = "WarmThroughput", skip_serializing_if = "Option::is_none")] + pub warm_throughput: Option, + /// What the backend stored for the table, if a client ever set it. Not a + /// wire member; see [`super::WarmThroughputRecord`]. Nothing acts on it. + #[serde(skip)] + pub warm_throughput_record: Option, } /// Details of a restore reported on the restored table's description. @@ -963,6 +1025,8 @@ pub struct CreateTableInput { pub table_class: Option, #[serde(rename = "OnDemandThroughput")] pub on_demand_throughput: Option, + #[serde(rename = "WarmThroughput", default)] + pub warm_throughput: Option, } /// `CreateTable` response body. @@ -1063,6 +1127,8 @@ pub struct CreateGsiAction { skip_serializing_if = "Option::is_none" )] pub provisioned_throughput: Option, + #[serde(rename = "WarmThroughput", default)] + pub warm_throughput: Option, } /// Delete an existing GSI from a table. @@ -1072,14 +1138,17 @@ pub struct DeleteGsiAction { pub index_name: String, } -/// Update provisioned throughput on an existing GSI. +/// Update an existing GSI. /// -/// Recognized by the deserializer but not yet implemented — the engine -/// returns a clear "not yet supported" error. +/// `WarmThroughput` is stored and reported (see [`super::WarmThroughputRecord`]). +/// A provisioned-throughput update is recognized by the deserializer but not +/// yet implemented: the engine returns a clear "not yet supported" error. #[derive(Debug, Clone, PartialEq, Deserialize)] pub struct UpdateGsiAction { #[serde(rename = "IndexName")] pub index_name: String, + #[serde(rename = "WarmThroughput", default)] + pub warm_throughput: Option, } /// `UpdateTable` request body. @@ -1110,6 +1179,13 @@ pub struct UpdateTableInput { /// unaffected. #[serde(rename = "VectorIndexUpdates", default)] pub vector_index_updates: Option>, + #[serde(rename = "WarmThroughput", default)] + pub warm_throughput: Option, + /// Accepted so that a request carrying it is recognised: `UpdateTable` + /// does not act on it (as before), but a table-level `WarmThroughput` + /// beside it is refused with the service's sentence. + #[serde(rename = "SSESpecification", default)] + pub sse_specification: Option, } /// `UpdateTable` response body. diff --git a/crates/core/src/types/warm_throughput.rs b/crates/core/src/types/warm_throughput.rs new file mode 100644 index 000000000..fadbef9c4 --- /dev/null +++ b/crates/core/src/types/warm_throughput.rs @@ -0,0 +1,345 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! `WarmThroughput` on tables and global secondary indexes. +//! +//! Measured against Amazon DynamoDB (us-east-1, 2026-10-23). The service +//! reports `WarmThroughput` as `{ReadUnitsPerSecond, WriteUnitsPerSecond, +//! Status}` on an ACTIVE table and on each of its ACTIVE global secondary +//! indexes; local secondary indexes carry none. A client that sends none gets +//! 12000 read and 4000 write units on an on-demand table and its indexes, and +//! the provisioned units on a provisioned table and each provisioned index. +//! +//! ExtendDB stores the values a client sends and reports them with the same +//! shape, defaults and validation. Nothing acts on them: there is no +//! pre-warming and no scaling at the database level, and the stored numbers +//! have no effect on throughput, throttling or storage. They exist so clients +//! that read the member (the Terraform AWS provider since 6.13.0 waits for +//! `Status: ACTIVE` after every table and index change) work. + +use serde::{Deserialize, Serialize}; + +/// Read units an on-demand table or index reports when the client sent none. +pub const ON_DEMAND_WARM_READ_UNITS: i64 = 12_000; +/// Write units an on-demand table or index reports when the client sent none. +pub const ON_DEMAND_WARM_WRITE_UNITS: i64 = 4_000; + +/// `WarmThroughput` as a client sends it on `CreateTable`, `UpdateTable`, a +/// `GlobalSecondaryIndexes` element and a `GlobalSecondaryIndexUpdates` +/// `Create` or `Update` action. Either member may be omitted. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] +pub struct WarmThroughput { + #[serde( + rename = "ReadUnitsPerSecond", + default, + skip_serializing_if = "Option::is_none" + )] + pub read_units_per_second: Option, + #[serde( + rename = "WriteUnitsPerSecond", + default, + skip_serializing_if = "Option::is_none" + )] + pub write_units_per_second: Option, +} + +/// The `Status` member of a reported `WarmThroughput`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum WarmThroughputStatus { + #[serde(rename = "ACTIVE")] + Active, + #[serde(rename = "UPDATING")] + Updating, +} + +/// `WarmThroughput` as the service reports it on `TableDescription` and +/// `GlobalSecondaryIndexDescription`. +/// +/// Both unit members are optional because the `CreateTable` response echoes a +/// partial request as sent: a request of `{ReadUnitsPerSecond: 20000}` was +/// answered with `{ReadUnitsPerSecond: 20000, Status: UPDATING}` and no write +/// member, and `DescribeTable` once ACTIVE then read `{20000, 4000, ACTIVE}`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub struct WarmThroughputDescription { + #[serde( + rename = "ReadUnitsPerSecond", + default, + skip_serializing_if = "Option::is_none" + )] + pub read_units_per_second: Option, + #[serde( + rename = "WriteUnitsPerSecond", + default, + skip_serializing_if = "Option::is_none" + )] + pub write_units_per_second: Option, + #[serde(rename = "Status")] + pub status: WarmThroughputStatus, +} + +/// What a backend persists for a table or index whose client set +/// `WarmThroughput`: the units the client sent (a side the client never sent +/// stays `None` and is reported from the defaults), the units the resource +/// reported before the last `UpdateTable` (reported again while `UPDATING`, +/// as the service does), and the epoch second until which the record reports +/// `UPDATING`. +/// +/// Stored as one JSON value in the catalog. Nothing reads these numbers except +/// the description builder: they do not pre-warm anything, scale anything or +/// change how a request is throttled or stored. +#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize, Default)] +pub struct WarmThroughputRecord { + #[serde(rename = "ReadUnitsPerSecond", default)] + pub read_units_per_second: Option, + #[serde(rename = "WriteUnitsPerSecond", default)] + pub write_units_per_second: Option, + /// The side the resource reported when the last `UpdateTable` was + /// validated (`None` means the default of the moment). Reported, and + /// decreases are validated against it, while the record is `UPDATING`. + #[serde(rename = "PreviousReadUnitsPerSecond", default)] + pub previous_read_units_per_second: Option, + #[serde(rename = "PreviousWriteUnitsPerSecond", default)] + pub previous_write_units_per_second: Option, + /// Epoch seconds. `Status` is `UPDATING` while `now < updating_until`. + #[serde(rename = "UpdatingUntil", default)] + pub updating_until: f64, +} + +/// The defaults a table or index reports when the client never set warm +/// throughput, and the values that fill a side the client left out. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct WarmThroughputDefaults { + pub read_units_per_second: i64, + pub write_units_per_second: i64, + /// On an on-demand resource a stored side below the default is reported + /// as the default: switching a provisioned table to PAY_PER_REQUEST + /// resets a below-minimum warm throughput to 12000/4000 (measured, rows + /// T7c and T7d). + pub pay_per_request: bool, +} + +impl WarmThroughputDefaults { + /// On-demand: 12000 read, 4000 write. Provisioned: the resource's own + /// `ReadCapacityUnits` and `WriteCapacityUnits`. Measured on a + /// PAY_PER_REQUEST table and on a 1/1 provisioned table with a 1/1 index. + #[must_use] + pub fn for_billing( + pay_per_request: bool, + read_capacity_units: i64, + write_capacity_units: i64, + ) -> Self { + if pay_per_request { + Self { + read_units_per_second: ON_DEMAND_WARM_READ_UNITS, + write_units_per_second: ON_DEMAND_WARM_WRITE_UNITS, + pay_per_request, + } + } else { + Self { + read_units_per_second: read_capacity_units, + write_units_per_second: write_capacity_units, + pay_per_request, + } + } + } + + /// The pair `(read, write)` of stored sides resolved against these + /// defaults: an unset side is the default; on an on-demand resource a + /// side below the default is the default. + #[must_use] + pub fn resolve(self, read: Option, write: Option) -> (i64, i64) { + let side = |stored: Option, default: i64| match stored { + Some(v) if self.pay_per_request => v.max(default), + Some(v) => v, + None => default, + }; + ( + side(read, self.read_units_per_second), + side(write, self.write_units_per_second), + ) + } + + /// The description an untouched resource reports once it is ACTIVE. + #[must_use] + pub fn describe(self) -> WarmThroughputDescription { + WarmThroughputDescription { + read_units_per_second: Some(self.read_units_per_second), + write_units_per_second: Some(self.write_units_per_second), + status: WarmThroughputStatus::Active, + } + } +} + +impl WarmThroughputRecord { + /// The record a `CreateTable` (or a GSI `Create`) writes: the request as + /// sent, reporting `UPDATING` until `updating_until`. + /// + /// The previous pair is the request itself: a created resource that is + /// past CREATING inside its window (an index added by `UpdateTable` is + /// `ACTIVE` here at once) reports the requested values with `UPDATING` + /// and then the same values with `ACTIVE`, as the service does for an + /// index being created (rows S8a, V5). A side the request leaves out + /// reports the default. + #[must_use] + pub fn from_request(spec: &WarmThroughput, updating_until: f64) -> Self { + Self { + read_units_per_second: spec.read_units_per_second, + write_units_per_second: spec.write_units_per_second, + previous_read_units_per_second: spec.read_units_per_second, + previous_write_units_per_second: spec.write_units_per_second, + updating_until, + } + } + + /// The record an `UpdateTable` writes over `existing` at `now`: each side + /// the request names replaces the stored side, a side it leaves out keeps + /// the stored value (or stays unset and keeps reporting the default), the + /// `UPDATING` deadline restarts, and the previous pair becomes what + /// `existing` reported at `now` (its own previous pair while it was still + /// `UPDATING`, its stored sides otherwise), so a request inside the window + /// keeps reporting the values the window started from. + /// + /// A `spec` with neither side (`WarmThroughput: {}`) changes no value and + /// restarts the window. The service answers `InternalFailure` with an empty + /// message to that request on `UpdateTable` (row S9b); ExtendDB accepts it + /// and does not imitate a service fault. + #[must_use] + pub fn apply_update( + existing: Option, + spec: &WarmThroughput, + updating_until: f64, + now: f64, + ) -> Self { + let existing = existing.unwrap_or_default(); + let (previous_read, previous_write) = if existing.is_updating_at(now) { + ( + existing.previous_read_units_per_second, + existing.previous_write_units_per_second, + ) + } else { + ( + existing.read_units_per_second, + existing.write_units_per_second, + ) + }; + Self { + read_units_per_second: spec + .read_units_per_second + .or(existing.read_units_per_second), + write_units_per_second: spec + .write_units_per_second + .or(existing.write_units_per_second), + previous_read_units_per_second: previous_read, + previous_write_units_per_second: previous_write, + updating_until, + } + } + + /// The effective units once the window has passed: the stored side where + /// set, the default otherwise (clamped on an on-demand resource). + #[must_use] + pub fn effective(&self, defaults: WarmThroughputDefaults) -> (i64, i64) { + defaults.resolve(self.read_units_per_second, self.write_units_per_second) + } + + /// The pair the resource reports at `now`: the previous pair while + /// `UPDATING` (measured: the service keeps reporting the values the + /// window started from, rows U2, S2a, S2c), the effective pair afterwards. + /// Decreases are validated against this pair (row U3: inside the window a + /// request carrying the previous values is accepted; row S2b: on a + /// settled table the same request is a decrease). + #[must_use] + pub fn reported(&self, defaults: WarmThroughputDefaults, now: f64) -> (i64, i64) { + if self.is_updating_at(now) { + defaults.resolve( + self.previous_read_units_per_second, + self.previous_write_units_per_second, + ) + } else { + self.effective(defaults) + } + } + + /// Whether the record still reports `UPDATING` at `now` (epoch seconds). + #[must_use] + pub fn is_updating_at(&self, now: f64) -> bool { + now < self.updating_until + } + + /// The description once the owning table or index is past CREATING: both + /// sides filled, the previous pair with `UPDATING` until the deadline, + /// then the new pair with `ACTIVE`. + #[must_use] + pub fn describe( + &self, + defaults: WarmThroughputDefaults, + now: f64, + ) -> WarmThroughputDescription { + let (read, write) = self.reported(defaults, now); + WarmThroughputDescription { + read_units_per_second: Some(read), + write_units_per_second: Some(write), + status: if self.is_updating_at(now) { + WarmThroughputStatus::Updating + } else { + WarmThroughputStatus::Active + }, + } + } + + /// The `CreateTable` response shape while the resource is still CREATING: + /// the request echoed as sent (a side the client left out is absent) with + /// `Status: UPDATING`. Measured on a `{ReadUnitsPerSecond: 20000}` request. + #[must_use] + pub fn echo_while_creating(&self) -> WarmThroughputDescription { + WarmThroughputDescription { + read_units_per_second: self.read_units_per_second, + write_units_per_second: self.write_units_per_second, + status: WarmThroughputStatus::Updating, + } + } + + /// The member a response carries for a resource in `status`, or `None` + /// when the service omits it. + /// + /// - No record and the resource is CREATING: absent (measured on the + /// `CreateTable` response and the first `DescribeTable`). A record with + /// neither side (a `WarmThroughput: {}` request) is the same: the + /// service answers no member while CREATING (row T1). + /// - No record otherwise: the defaults, `ACTIVE`. + /// - A record while CREATING: the request echoed, `UPDATING`. + /// - A record otherwise: both sides filled, the previous pair with + /// `UPDATING` until the deadline, then the new pair with `ACTIVE`. + #[must_use] + pub fn describe_for( + record: Option<&Self>, + creating: bool, + defaults: WarmThroughputDefaults, + now: f64, + ) -> Option { + match (record, creating) { + (None, true) => None, + (Some(record), true) + if record.read_units_per_second.is_none() + && record.write_units_per_second.is_none() => + { + None + } + (None, false) => Some(defaults.describe()), + (Some(record), true) => Some(record.echo_while_creating()), + (Some(record), false) => Some(record.describe(defaults, now)), + } + } +} + +/// Seconds since the Unix epoch, for the `UPDATING` deadline and its check. +#[must_use] +#[allow(clippy::cast_precision_loss)] +pub fn epoch_seconds_now() -> f64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0.0, |d| d.as_secs_f64()) +} + +#[cfg(test)] +mod tests; diff --git a/crates/core/src/types/warm_throughput/tests.rs b/crates/core/src/types/warm_throughput/tests.rs new file mode 100644 index 000000000..62cf714aa --- /dev/null +++ b/crates/core/src/types/warm_throughput/tests.rs @@ -0,0 +1,338 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Tests for the `WarmThroughput` shapes and the stored record. + +use super::*; + +fn on_demand() -> WarmThroughputDefaults { + WarmThroughputDefaults::for_billing(true, 0, 0) +} + +#[test] +fn on_demand_defaults_are_the_measured_12000_and_4000() { + assert_eq!( + on_demand(), + WarmThroughputDefaults { + read_units_per_second: 12_000, + write_units_per_second: 4_000, + pay_per_request: true, + } + ); +} + +#[test] +fn provisioned_defaults_are_the_provisioned_units() { + assert_eq!( + WarmThroughputDefaults::for_billing(false, 1, 1), + WarmThroughputDefaults { + read_units_per_second: 1, + write_units_per_second: 1, + pay_per_request: false, + } + ); +} + +/// Rows T7a to T7d: a read-only record `{100}` on a provisioned 5/5 table +/// reads `{100, 5}`, follows a capacity change to `{100, 10}`, and resets to +/// `{12000, 4000}` once the table is on-demand (a side below the on-demand +/// default is reported as the default), after which `12000/4000` is not a +/// decrease. +#[test] +fn an_on_demand_resource_reports_at_least_the_on_demand_defaults() { + let record = WarmThroughputRecord::from_request( + &WarmThroughput { + read_units_per_second: Some(100), + write_units_per_second: None, + }, + 0.0, + ); + assert_eq!( + record.effective(WarmThroughputDefaults::for_billing(false, 5, 5)), + (100, 5) + ); + assert_eq!( + record.effective(WarmThroughputDefaults::for_billing(false, 10, 10)), + (100, 10) + ); + assert_eq!(record.effective(on_demand()), (12_000, 4_000)); + // A stored side above the default is kept on an on-demand resource. + let high = WarmThroughputRecord::from_request( + &WarmThroughput { + read_units_per_second: Some(20_000), + write_units_per_second: Some(50), + }, + 0.0, + ); + assert_eq!(high.effective(on_demand()), (20_000, 4_000)); +} + +/// Row T1: `WarmThroughput: {}` on CreateTable reports no member while the +/// table is CREATING, and the defaults once ACTIVE. +#[test] +fn an_empty_request_reports_no_member_while_creating() { + let record = WarmThroughputRecord::from_request(&WarmThroughput::default(), 10.0); + assert_eq!( + WarmThroughputRecord::describe_for(Some(&record), true, on_demand(), 1.0), + None + ); + assert_eq!( + WarmThroughputRecord::describe_for(Some(&record), false, on_demand(), 11.0), + Some(on_demand().describe()) + ); +} + +/// Rows U2, S2a, S2c, U3 and S2b: while UPDATING the record reports the pair +/// the window started from, and that pair is what a decrease is measured +/// against; once settled the new pair is reported and compared. +#[test] +fn the_previous_pair_is_reported_and_compared_while_updating() { + // Never set (defaults 12000/4000), raised to 24000/8000 at t=0, window to 10. + let raised = WarmThroughputRecord::apply_update( + None, + &WarmThroughput { + read_units_per_second: Some(24_000), + write_units_per_second: Some(8_000), + }, + 10.0, + 0.0, + ); + assert_eq!(raised.reported(on_demand(), 5.0), (12_000, 4_000)); + assert_eq!( + raised.describe(on_demand(), 5.0), + WarmThroughputDescription { + read_units_per_second: Some(12_000), + write_units_per_second: Some(4_000), + status: WarmThroughputStatus::Updating, + } + ); + assert_eq!(raised.reported(on_demand(), 10.0), (24_000, 8_000)); + // U3: inside the window a request carrying the previous values validates + // against 12000/4000 and, applied, keeps reporting 12000/4000. + let inside = WarmThroughputRecord::apply_update( + Some(raised), + &WarmThroughput { + read_units_per_second: Some(12_000), + write_units_per_second: Some(4_000), + }, + 15.0, + 5.0, + ); + assert_eq!(inside.reported(on_demand(), 6.0), (12_000, 4_000)); + assert_eq!(inside.reported(on_demand(), 15.0), (12_000, 4_000)); + // S2c: the same values re-sent on a settled table report themselves UPDATING. + let same = WarmThroughputRecord::apply_update( + Some(raised), + &WarmThroughput { + read_units_per_second: Some(24_000), + write_units_per_second: Some(8_000), + }, + 20.0, + 11.0, + ); + assert_eq!(same.reported(on_demand(), 12.0), (24_000, 8_000)); + assert_eq!( + same.describe(on_demand(), 12.0).status, + WarmThroughputStatus::Updating + ); +} + +/// The service omits the member on a CREATING table that was created +/// without one, and reports the defaults ACTIVE once the table is. +#[test] +fn no_record_is_absent_while_creating_and_defaults_once_active() { + assert_eq!( + WarmThroughputRecord::describe_for(None, true, on_demand(), 0.0), + None + ); + let json = serde_json::to_value( + WarmThroughputRecord::describe_for(None, false, on_demand(), 0.0).expect("present"), + ) + .expect("serialises"); + assert_eq!( + json, + serde_json::json!({ + "ReadUnitsPerSecond": 12000, + "WriteUnitsPerSecond": 4000, + "Status": "ACTIVE" + }) + ); +} + +/// Measured: `{ReadUnitsPerSecond: 20000}` on CreateTable was echoed as +/// `{ReadUnitsPerSecond: 20000, Status: UPDATING}` with no write member, +/// and read back as `{20000, 4000, ACTIVE}` once the table was ACTIVE. +#[test] +fn a_partial_request_is_echoed_while_creating_and_filled_once_active() { + let spec = WarmThroughput { + read_units_per_second: Some(20_000), + write_units_per_second: None, + }; + let record = WarmThroughputRecord::from_request(&spec, 10.0); + let creating = serde_json::to_value( + WarmThroughputRecord::describe_for(Some(&record), true, on_demand(), 1.0).expect("present"), + ) + .expect("serialises"); + assert_eq!( + creating, + serde_json::json!({ "ReadUnitsPerSecond": 20000, "Status": "UPDATING" }) + ); + let active = WarmThroughputRecord::describe_for(Some(&record), false, on_demand(), 11.0) + .expect("present"); + assert_eq!( + active, + WarmThroughputDescription { + read_units_per_second: Some(20_000), + write_units_per_second: Some(4_000), + status: WarmThroughputStatus::Active, + } + ); +} + +#[test] +fn the_status_is_updating_until_the_deadline_then_active() { + let record = WarmThroughputRecord::from_request(&WarmThroughput::default(), 100.0); + assert_eq!( + record.describe(on_demand(), 99.9).status, + WarmThroughputStatus::Updating + ); + assert_eq!( + record.describe(on_demand(), 100.0).status, + WarmThroughputStatus::Active + ); +} + +/// An update names one side: that side changes, the other keeps its stored +/// value, and the deadline restarts. +#[test] +fn an_update_replaces_only_the_sides_it_names() { + let existing = WarmThroughputRecord { + read_units_per_second: Some(100), + write_units_per_second: Some(50), + previous_read_units_per_second: None, + previous_write_units_per_second: None, + updating_until: 5.0, + }; + let updated = WarmThroughputRecord::apply_update( + Some(existing), + &WarmThroughput { + read_units_per_second: Some(200), + write_units_per_second: None, + }, + 9.0, + 6.0, + ); + assert_eq!( + updated, + WarmThroughputRecord { + read_units_per_second: Some(200), + write_units_per_second: Some(50), + previous_read_units_per_second: Some(100), + previous_write_units_per_second: Some(50), + updating_until: 9.0, + } + ); + // No stored record: the unnamed side stays unset and reports the default. + let fresh = WarmThroughputRecord::apply_update( + None, + &WarmThroughput { + read_units_per_second: None, + write_units_per_second: Some(8_000), + }, + 9.0, + 6.0, + ); + assert_eq!(fresh.effective(on_demand()), (12_000, 8_000)); +} + +/// The persisted shape round-trips, and a record written without a +/// deadline (or a side) reads back. +#[test] +fn the_record_round_trips_through_json() { + let record = WarmThroughputRecord { + read_units_per_second: Some(1), + write_units_per_second: None, + previous_read_units_per_second: Some(3), + previous_write_units_per_second: None, + updating_until: 1.5, + }; + let text = serde_json::to_string(&record).expect("serialises"); + assert_eq!( + serde_json::from_str::(&text).expect("parses"), + record + ); + let sparse: WarmThroughputRecord = + serde_json::from_str(r#"{"WriteUnitsPerSecond": 7}"#).expect("parses"); + assert_eq!(sparse.write_units_per_second, Some(7)); + assert_eq!(sparse.updating_until, 0.0); +} + +#[test] +fn the_request_shape_accepts_either_side_alone() { + let read_only: WarmThroughput = + serde_json::from_str(r#"{"ReadUnitsPerSecond": 20000}"#).expect("parses"); + assert_eq!(read_only.read_units_per_second, Some(20_000)); + assert_eq!(read_only.write_units_per_second, None); + let empty: WarmThroughput = serde_json::from_str("{}").expect("parses"); + assert_eq!(empty, WarmThroughput::default()); +} + +/// Rows S8a and V5: an index created by `UpdateTable` is past CREATING at +/// once here, and inside its window reports what was requested with +/// `UPDATING`, then the same values `ACTIVE`; a side left out reports the +/// default. +#[test] +fn a_created_resource_reports_the_request_while_updating() { + let record = WarmThroughputRecord::from_request( + &WarmThroughput { + read_units_per_second: Some(30_000), + write_units_per_second: Some(8_000), + }, + 10.0, + ); + assert_eq!( + WarmThroughputRecord::describe_for(Some(&record), false, on_demand(), 5.0), + Some(WarmThroughputDescription { + read_units_per_second: Some(30_000), + write_units_per_second: Some(8_000), + status: WarmThroughputStatus::Updating, + }) + ); + assert_eq!( + WarmThroughputRecord::describe_for(Some(&record), false, on_demand(), 10.0), + Some(WarmThroughputDescription { + read_units_per_second: Some(30_000), + write_units_per_second: Some(8_000), + status: WarmThroughputStatus::Active, + }) + ); + let partial = WarmThroughputRecord::from_request( + &WarmThroughput { + read_units_per_second: Some(20_000), + write_units_per_second: None, + }, + 10.0, + ); + assert_eq!(partial.reported(on_demand(), 5.0), (20_000, 4_000)); +} + +/// Row S9b: the service refuses `UpdateTable` with `WarmThroughput: {}` with +/// an empty `InternalFailure`; ExtendDB accepts it, keeps both values and +/// restarts the window. +#[test] +fn an_empty_update_keeps_the_values_and_restarts_the_window() { + let settled = WarmThroughputRecord::apply_update( + None, + &WarmThroughput { + read_units_per_second: Some(24_000), + write_units_per_second: Some(8_000), + }, + 1.0, + 0.0, + ); + let empty = + WarmThroughputRecord::apply_update(Some(settled), &WarmThroughput::default(), 12.0, 2.0); + assert_eq!(empty.effective(on_demand()), (24_000, 8_000)); + assert_eq!(empty.reported(on_demand(), 3.0), (24_000, 8_000)); + assert!(empty.is_updating_at(3.0)); +} diff --git a/crates/core/src/validation/mod.rs b/crates/core/src/validation/mod.rs index 3decb4666..f6a382975 100755 --- a/crates/core/src/validation/mod.rs +++ b/crates/core/src/validation/mod.rs @@ -2,11 +2,21 @@ // SPDX-License-Identifier: Apache-2.0 pub mod number; pub mod vector_item; +pub mod warm_throughput; pub use vector_item::{ MAX_HASH_KEY_SIZE, MAX_INLINE_FILTER_SIZE, validate_vector_write, validate_vector_write_changed, vector_components, vector_norm, }; +pub use warm_throughput::{ + DELETE_GSI_WITH_WARM_THROUGHPUT, DELETION_PROTECTION_MUST_BE_ONLY_OPERATION, + INDEX_CHANGE_WITH_STREAM_CHANGE, ON_DEMAND_THROUGHPUT_NEEDS_BILLING_MODE, + SSE_MUST_BE_ONLY_OPERATION, TABLE_CLASS_MUST_BE_ONLY_OPERATION, + WARM_THROUGHPUT_MUST_BE_ONLY_OPERATION, WarmThroughputSubject, + validate_create_table_warm_throughput, validate_created_gsi_warm_throughput, + validate_warm_throughput_is_only_operation, validate_warm_throughput_on_create, + validate_warm_throughput_on_update, +}; use crate::error::{DynamoDbError, ErrorMessageKey, error_message}; use crate::limits::LimitsConfig; @@ -126,6 +136,7 @@ pub fn validate_create_table( ))?; validate_vector_indexes(input)?; validate_stream_specification(input)?; + validate_create_table_warm_throughput(input)?; Ok(()) } @@ -2289,6 +2300,7 @@ mod tests { deletion_protection_enabled: None, table_class: None, on_demand_throughput: None, + warm_throughput: None, } } @@ -2532,6 +2544,7 @@ mod tests { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]); assert!(validate_create_table(&input, &limits).is_ok()); } @@ -2564,6 +2577,7 @@ mod tests { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]); assert!(validate_create_table(&input, &limits).is_err()); } @@ -2592,6 +2606,7 @@ mod tests { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]); assert!(validate_create_table(&input, &limits).is_err()); } @@ -2626,6 +2641,7 @@ mod tests { read_capacity_units: 5, write_capacity_units: 5, }), + warm_throughput: None, }]); let err = validate_create_table(&input, &limits).unwrap_err(); assert!( @@ -2654,6 +2670,7 @@ mod tests { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]); assert!(validate_create_table(&input, &limits).is_ok()); } @@ -2699,6 +2716,7 @@ mod tests { write_capacity_units: write, } }), + warm_throughput: None, }]); input } @@ -2748,6 +2766,7 @@ mod tests { key_schema: vec![make_ks("gsi_pk", KeyType::Hash)], projection: include(0..gsi_end), provisioned_throughput: None, + warm_throughput: None, }]); input.local_secondary_indexes = Some(vec![crate::types::LsiInput { index_name: "my-lsi".to_owned(), @@ -3575,6 +3594,7 @@ mod tests { non_key_attributes: non_key, }, provisioned_throughput: None, + warm_throughput: None, }; input.global_secondary_indexes = Some(vec![gsi(None)]); @@ -3614,6 +3634,7 @@ mod tests { non_key_attributes: non_key, }, provisioned_throughput: None, + warm_throughput: None, }]); input }; diff --git a/crates/core/src/validation/warm_throughput.rs b/crates/core/src/validation/warm_throughput.rs new file mode 100644 index 000000000..75502195e --- /dev/null +++ b/crates/core/src/validation/warm_throughput.rs @@ -0,0 +1,381 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The measured `WarmThroughput` request rules (Amazon DynamoDB, us-east-1, +//! 2026-10-23). Every refusal is a `ValidationException` whose text is the +//! service's own. +//! +//! - On-demand, `CreateTable`: a side below the on-demand default (12000 read, +//! 4000 write) is refused with the "initial throughput for OnDemand" +//! sentence. Read is checked before write (a 0/0 request reports the read +//! message). +//! - Provisioned, `CreateTable`: a side below the resource's provisioned units +//! is refused with the "lower than ReadCapacityUnits of +//! ProvisionedThroughput" sentence (both member forms measured; a value +//! equal to the units is accepted). +//! - `UpdateTable`, either billing mode: a side below the CURRENT warm +//! throughput is refused with the "decreasing WarmThroughput is not +//! supported" sentence. The same values as current are accepted. +//! - Every sentence names its subject: `for table` on the table, `for index +//! ` on a global secondary index (measured on create and update, +//! both members). +//! - `UpdateTable`: a table-level `WarmThroughput` is an operation that must +//! stand alone, checked before the value rules. Measured per companion: +//! `SSESpecification`, `TableClass` and `DeletionProtectionEnabled` each +//! answer their own member's "must be the only operation" sentence (those +//! members have the same rule on their own); `OnDemandThroughput`, with or +//! without `BillingMode`, answers its own "can only be combined with +//! BillingMode" sentence; a `GlobalSecondaryIndexUpdates` `Delete` answers +//! its own sentence; `StreamSpecification`, `BillingMode` and +//! `ProvisionedThroughput` answer the WarmThroughput sentence; +//! a `GlobalSecondaryIndexUpdates` `Update` is accepted beside it (table and +//! index both go `UPDATING`). See [`validate_warm_throughput_is_only_operation`]. + +use crate::error::DynamoDbError; +use crate::types::{ + BillingMode, CreateGsiAction, CreateTableInput, GsiInput, ON_DEMAND_WARM_READ_UNITS, + ON_DEMAND_WARM_WRITE_UNITS, ProvisionedThroughput, UpdateTableInput, WarmThroughput, +}; + +const PREFIX: &str = "One or more parameter values were invalid: Requested "; + +const ON_DEMAND_MINIMUM_SUFFIX: &str = " is lower than initial throughput for OnDemand. See: \ + https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/\ + on-demand-capacity-mode.html#on-demand-capacity-mode-initial"; + +const DECREASE_SUFFIX: &str = " is lower than current WarmThroughput, decreasing \ + WarmThroughput is not supported"; + +/// The measured refusal for a table-level `WarmThroughput` sent beside a +/// `StreamSpecification`, `BillingMode` or `ProvisionedThroughput` change. +pub const WARM_THROUGHPUT_MUST_BE_ONLY_OPERATION: &str = "One or more parameter values were \ + invalid: WarmThroughput must be the only operation in the request"; + +/// The measured refusal for `TableClass` beside another operation. +pub const TABLE_CLASS_MUST_BE_ONLY_OPERATION: &str = "One or more parameter values were \ + invalid: TableClass modification must be the only operation in the request"; + +/// The measured refusal for `DeletionProtectionEnabled` beside another +/// operation. +pub const DELETION_PROTECTION_MUST_BE_ONLY_OPERATION: &str = "One or more parameter values \ + were invalid: DeletionProtection modification must be the only operation in the request"; + +/// The measured refusal for `OnDemandThroughput` beside a table +/// `WarmThroughput`, with or without `BillingMode` (rows T4a, T4d). +pub const ON_DEMAND_THROUGHPUT_NEEDS_BILLING_MODE: &str = "One or more parameter values were \ + invalid: OnDemandThroughput can only be combined with BillingMode in an UpdateTable request"; + +/// The measured refusal for `SSESpecification` beside another operation +/// (rows S1a to S1c). +pub const SSE_MUST_BE_ONLY_OPERATION: &str = "One or more parameter values were invalid: \ + Server-Side Encryption modification must be the only operation in the request"; + +/// The measured refusal for a `GlobalSecondaryIndexUpdates` `Delete` beside a +/// table `WarmThroughput` (row S9a; the index stays). +pub const DELETE_GSI_WITH_WARM_THROUGHPUT: &str = "One or more parameter values were invalid: \ + Delete global secondary index cannot be specified when updating WarmThroughput"; + +/// The measured refusal for a `GlobalSecondaryIndexUpdates` `Delete` beside a +/// table `WarmThroughput` and a `StreamSpecification` (row V3a). +pub const INDEX_CHANGE_WITH_STREAM_CHANGE: &str = "One or more parameter values were invalid: \ + You cannot create or delete index while changing stream status"; + +/// The resource a `WarmThroughput` sentence is about: the service writes +/// `for table` on the table and `for index ` on a global secondary +/// index (measured on create and update, both members). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WarmThroughputSubject<'a> { + Table, + Index(&'a str), +} + +impl std::fmt::Display for WarmThroughputSubject<'_> { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Table => f.write_str("table"), + Self::Index(name) => write!(f, "index {name}"), + } + } +} + +/// The on-demand minimum refusal for `member` (`ReadUnitsPerSecond` or +/// `WriteUnitsPerSecond`) on `subject`. Measured byte for byte on both +/// members, on a table and on an index. +#[must_use] +pub fn warm_throughput_below_on_demand_minimum( + member: &str, + subject: WarmThroughputSubject<'_>, +) -> String { + format!("{PREFIX}{member} for WarmThroughput for {subject}{ON_DEMAND_MINIMUM_SUFFIX}") +} + +/// The provisioned-units refusal for `member` and its provisioned +/// counterpart (`ReadCapacityUnits` or `WriteCapacityUnits`) on `subject`. +/// Measured on both members on a table and on the read member on an index. +#[must_use] +pub fn warm_throughput_below_provisioned( + member: &str, + capacity_member: &str, + subject: WarmThroughputSubject<'_>, +) -> String { + format!( + "{PREFIX}{member} for WarmThroughput for {subject} is lower than {capacity_member} of \ + ProvisionedThroughput" + ) +} + +/// The decrease refusal for `member` on `subject`. Measured on both members +/// on a table (either billing mode) and on an index. +#[must_use] +pub fn warm_throughput_decrease(member: &str, subject: WarmThroughputSubject<'_>) -> String { + format!("{PREFIX}{member} for WarmThroughput for {subject}{DECREASE_SUFFIX}") +} + +/// The `CreateTable` rule for one resource (the table or one index) against +/// the billing mode it is created under and, when provisioned, its units. +/// +/// # Errors +/// +/// `ValidationException` with the measured sentence for the first side that +/// is too low, read before write. +pub fn validate_warm_throughput_on_create( + spec: &WarmThroughput, + subject: WarmThroughputSubject<'_>, + billing_mode: BillingMode, + provisioned: Option<&ProvisionedThroughput>, +) -> Result<(), DynamoDbError> { + match billing_mode { + BillingMode::PayPerRequest => { + if spec + .read_units_per_second + .is_some_and(|r| r < ON_DEMAND_WARM_READ_UNITS) + { + return Err(DynamoDbError::ValidationException( + warm_throughput_below_on_demand_minimum("ReadUnitsPerSecond", subject), + )); + } + if spec + .write_units_per_second + .is_some_and(|w| w < ON_DEMAND_WARM_WRITE_UNITS) + { + return Err(DynamoDbError::ValidationException( + warm_throughput_below_on_demand_minimum("WriteUnitsPerSecond", subject), + )); + } + } + BillingMode::Provisioned => { + // A provisioned resource without units is refused by the throughput + // rules before this one runs; nothing to compare against here. + let Some(units) = provisioned else { + return Ok(()); + }; + if spec + .read_units_per_second + .is_some_and(|r| r < units.read_capacity_units) + { + return Err(DynamoDbError::ValidationException( + warm_throughput_below_provisioned( + "ReadUnitsPerSecond", + "ReadCapacityUnits", + subject, + ), + )); + } + if spec + .write_units_per_second + .is_some_and(|w| w < units.write_capacity_units) + { + return Err(DynamoDbError::ValidationException( + warm_throughput_below_provisioned( + "WriteUnitsPerSecond", + "WriteCapacityUnits", + subject, + ), + )); + } + } + } + Ok(()) +} + +/// The `UpdateTable` rule for one resource against its current effective warm +/// throughput: no side may go below the current value. Equal is accepted. +/// +/// # Errors +/// +/// `ValidationException` with the measured decrease sentence, read before write. +pub fn validate_warm_throughput_on_update( + spec: &WarmThroughput, + subject: WarmThroughputSubject<'_>, + current: (i64, i64), +) -> Result<(), DynamoDbError> { + if spec.read_units_per_second.is_some_and(|r| r < current.0) { + return Err(DynamoDbError::ValidationException( + warm_throughput_decrease("ReadUnitsPerSecond", subject), + )); + } + if spec.write_units_per_second.is_some_and(|w| w < current.1) { + return Err(DynamoDbError::ValidationException( + warm_throughput_decrease("WriteUnitsPerSecond", subject), + )); + } + Ok(()) +} + +/// The `UpdateTable` rules for a request carrying a table-level +/// `WarmThroughput`, checked before the value rules. Measured per companion +/// on an on-demand table with one GSI, all values otherwise valid +/// (2026-10-23; row labels are those of the live runs): +/// +/// | beside `WarmThroughput` | refusal | rows | +/// |-------------------------------------------|----------------------------------|------| +/// | `SSESpecification` | Server-Side Encryption must be the only operation | S1a, S1b, S1c | +/// | `TableClass` | TableClass must be the only operation | V3b, V4, S10b | +/// | `DeletionProtectionEnabled` | DeletionProtection must be the only operation | S1d, T4c, T5d | +/// | `OnDemandThroughput`, with or without `BillingMode` | OnDemandThroughput can only be combined with BillingMode | T4a, T4d | +/// | `GlobalSecondaryIndexUpdates` with a `Delete` | Delete global secondary index cannot be specified when updating WarmThroughput | S9a | +/// | `GlobalSecondaryIndexUpdates` with a `Delete` and a `StreamSpecification` | You cannot create or delete index while changing stream status | V3a | +/// | `StreamSpecification` | WarmThroughput must be the only operation | T5b | +/// | `ProvisionedThroughput`, no `BillingMode` | WarmThroughput must be the only operation | V2 | +/// | `BillingMode` and `ProvisionedThroughput` | WarmThroughput must be the only operation | UT2 | +/// | `GlobalSecondaryIndexUpdates` with an `Update` | accepted; table and index go UPDATING | | +/// +/// Order, as measured: `SSESpecification`, then `TableClass`, then +/// `DeletionProtectionEnabled`, then `OnDemandThroughput`, then the index +/// `Delete`, then the WarmThroughput rule. SSE outranks TableClass, +/// DeletionProtection and Streams (S1a, S1b, S1c); TableClass outranks +/// DeletionProtection and Streams, and outranks a `Delete` (V3b); the triple +/// TableClass, DeletionProtection and Streams answers the TableClass sentence +/// (V4); DeletionProtection outranks Streams (S1d); TableClass and +/// DeletionProtection outrank `OnDemandThroughput` (T4b, T4c). The position of +/// the index `Delete` relative to `SSESpecification`, `DeletionProtectionEnabled` +/// and `OnDemandThroughput` was not measured; it sits after them, where the +/// measured pairs allow. A missing table with a companion answers the +/// companion's sentence (S10b TableClass, T5b Streams, T5d DeletionProtection) +/// because this function runs before the table is read; a bad value alone, or +/// an index `Update` alone, reaches the table lookup and answers +/// `ResourceNotFoundException` (S10, T5a, T5c). +/// +/// TableClass, DeletionProtection and SSE have the same "only operation" rule +/// on their own, without WarmThroughput (measured), but ExtendDB does not apply +/// it elsewhere yet; here the companion's own sentence is used only when +/// WarmThroughput is present, so a request without WarmThroughput is never +/// refused by this function. `VectorIndexUpdates` is an ExtendDB extension and +/// is grouped with the WarmThroughput sentence, unmeasured. `AttributeDefinitions` +/// is not an operation. +/// +/// # Errors +/// +/// `ValidationException` with the measured sentence for the companion. +pub fn validate_warm_throughput_is_only_operation( + input: &UpdateTableInput, +) -> Result<(), DynamoDbError> { + if input.warm_throughput.is_none() { + return Ok(()); + } + let refuse = |sentence: &str| Err(DynamoDbError::ValidationException(sentence.to_owned())); + if input.sse_specification.is_some() { + return refuse(SSE_MUST_BE_ONLY_OPERATION); + } + if input.table_class.is_some() { + return refuse(TABLE_CLASS_MUST_BE_ONLY_OPERATION); + } + if input.deletion_protection_enabled.is_some() { + return refuse(DELETION_PROTECTION_MUST_BE_ONLY_OPERATION); + } + if input.on_demand_throughput.is_some() { + return refuse(ON_DEMAND_THROUGHPUT_NEEDS_BILLING_MODE); + } + if input + .global_secondary_index_updates + .iter() + .flatten() + .any(|u| u.delete.is_some()) + { + return refuse(if input.stream_specification.is_some() { + INDEX_CHANGE_WITH_STREAM_CHANGE + } else { + DELETE_GSI_WITH_WARM_THROUGHPUT + }); + } + let warm_rule = input.billing_mode.is_some() + || input.table_throughput_mode.is_some() + || input.provisioned_throughput.is_some() + || input.stream_specification.is_some() + || input + .vector_index_updates + .as_ref() + .is_some_and(|u| !u.is_empty()); + if warm_rule { + return refuse(WARM_THROUGHPUT_MUST_BE_ONLY_OPERATION); + } + Ok(()) +} + +/// `validate_warm_throughput_on_create` for the table and each of its global +/// secondary indexes in a `CreateTable` request. +/// +/// # Errors +/// +/// The first refusal, table first, then indexes in request order. +pub fn validate_create_table_warm_throughput( + input: &CreateTableInput, +) -> Result<(), DynamoDbError> { + let billing_mode = input.billing_mode.unwrap_or(BillingMode::Provisioned); + if let Some(spec) = &input.warm_throughput { + validate_warm_throughput_on_create( + spec, + WarmThroughputSubject::Table, + billing_mode, + input.provisioned_throughput.as_ref(), + )?; + } + for gsi in input.global_secondary_indexes.iter().flatten() { + validate_gsi_warm_throughput_on_create(gsi, billing_mode)?; + } + Ok(()) +} + +/// The create rule for one `GlobalSecondaryIndexes` element. +/// +/// # Errors +/// +/// See [`validate_warm_throughput_on_create`]. +pub fn validate_gsi_warm_throughput_on_create( + gsi: &GsiInput, + billing_mode: BillingMode, +) -> Result<(), DynamoDbError> { + match &gsi.warm_throughput { + Some(spec) => validate_warm_throughput_on_create( + spec, + WarmThroughputSubject::Index(&gsi.index_name), + billing_mode, + gsi.provisioned_throughput.as_ref(), + ), + None => Ok(()), + } +} + +/// The create rule for a `GlobalSecondaryIndexUpdates` `Create` action, under +/// the table's billing mode. +/// +/// # Errors +/// +/// See [`validate_warm_throughput_on_create`]. +pub fn validate_created_gsi_warm_throughput( + create: &CreateGsiAction, + billing_mode: BillingMode, +) -> Result<(), DynamoDbError> { + match &create.warm_throughput { + Some(spec) => validate_warm_throughput_on_create( + spec, + WarmThroughputSubject::Index(&create.index_name), + billing_mode, + create.provisioned_throughput.as_ref(), + ), + None => Ok(()), + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/core/src/validation/warm_throughput/tests.rs b/crates/core/src/validation/warm_throughput/tests.rs new file mode 100644 index 000000000..e2df21df0 --- /dev/null +++ b/crates/core/src/validation/warm_throughput/tests.rs @@ -0,0 +1,493 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Tests for the measured `WarmThroughput` request rules. Each test names the +//! rows of the live run it reproduces. + +use super::*; + +fn spec(read: Option, write: Option) -> WarmThroughput { + WarmThroughput { + read_units_per_second: read, + write_units_per_second: write, + } +} + +fn message(result: Result<(), DynamoDbError>) -> String { + match result.expect_err("refused") { + DynamoDbError::ValidationException(m) => m, + other => panic!("{other:?}"), + } +} + +const TABLE: WarmThroughputSubject<'static> = WarmThroughputSubject::Table; +const GSI: WarmThroughputSubject<'static> = WarmThroughputSubject::Index("gsi"); + +const ON_DEMAND_READ_TABLE: &str = "One or more parameter values were invalid: Requested \ + ReadUnitsPerSecond for WarmThroughput for table is lower than initial throughput for \ + OnDemand. See: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/\ + on-demand-capacity-mode.html#on-demand-capacity-mode-initial"; + +fn five_five() -> ProvisionedThroughput { + ProvisionedThroughput { + read_capacity_units: 5, + write_capacity_units: 5, + } +} + +/// Measured rows C2, C3 and C7: the whole strings. +#[test] +fn on_demand_create_below_the_minimum_reports_the_measured_sentence() { + let read = message(validate_warm_throughput_on_create( + &spec(Some(11_999), Some(4_000)), + TABLE, + BillingMode::PayPerRequest, + None, + )); + assert_eq!(read, ON_DEMAND_READ_TABLE); + let write = message(validate_warm_throughput_on_create( + &spec(Some(12_000), Some(3_999)), + TABLE, + BillingMode::PayPerRequest, + None, + )); + assert_eq!( + write, + ON_DEMAND_READ_TABLE.replace("ReadUnitsPerSecond", "WriteUnitsPerSecond") + ); + // 0/0 reports the read side: read is checked first. + let both = message(validate_warm_throughput_on_create( + &spec(Some(0), Some(0)), + TABLE, + BillingMode::PayPerRequest, + None, + )); + assert_eq!(both, read); +} + +/// Rows C1 and C4: the minimum itself and a partial request are accepted. +#[test] +fn on_demand_create_at_or_above_the_minimum_is_accepted() { + validate_warm_throughput_on_create( + &spec(Some(12_000), Some(4_000)), + TABLE, + BillingMode::PayPerRequest, + None, + ) + .expect("the minimum"); + validate_warm_throughput_on_create( + &spec(Some(20_000), None), + TABLE, + BillingMode::PayPerRequest, + None, + ) + .expect("read alone"); +} + +/// Rows C5, C6, P1 and P2: a provisioned 5/5 table refuses 1/1 (read form) +/// and 5/4 (write form), accepts 5/5 (equal) and 100/50. +#[test] +fn provisioned_create_below_the_units_reports_the_measured_sentence() { + let units = five_five(); + let read = message(validate_warm_throughput_on_create( + &spec(Some(1), Some(1)), + TABLE, + BillingMode::Provisioned, + Some(&units), + )); + assert_eq!( + read, + "One or more parameter values were invalid: Requested ReadUnitsPerSecond for \ + WarmThroughput for table is lower than ReadCapacityUnits of ProvisionedThroughput" + ); + let write = message(validate_warm_throughput_on_create( + &spec(Some(5), Some(4)), + TABLE, + BillingMode::Provisioned, + Some(&units), + )); + assert_eq!( + write, + "One or more parameter values were invalid: Requested WriteUnitsPerSecond for \ + WarmThroughput for table is lower than WriteCapacityUnits of ProvisionedThroughput" + ); + validate_warm_throughput_on_create( + &spec(Some(5), Some(5)), + TABLE, + BillingMode::Provisioned, + Some(&units), + ) + .expect("equal to the units"); + validate_warm_throughput_on_create( + &spec(Some(100), Some(50)), + TABLE, + BillingMode::Provisioned, + Some(&units), + ) + .expect("above the units"); +} + +/// Rows U1 to U5, U7, UT1 and P3: equal and higher are accepted, lower is +/// refused with the decrease sentence on either billing mode and member. +#[test] +fn update_below_current_reports_the_decrease_sentence() { + validate_warm_throughput_on_update(&spec(Some(12_000), Some(4_000)), TABLE, (12_000, 4_000)) + .expect("same values"); + validate_warm_throughput_on_update(&spec(Some(24_000), Some(8_000)), TABLE, (12_000, 4_000)) + .expect("higher"); + validate_warm_throughput_on_update(&spec(Some(30_000), None), TABLE, (12_000, 4_000)) + .expect("read alone"); + let read = message(validate_warm_throughput_on_update( + &spec(Some(11_999), Some(4_000)), + TABLE, + (12_000, 4_000), + )); + assert_eq!( + read, + "One or more parameter values were invalid: Requested ReadUnitsPerSecond for \ + WarmThroughput for table is lower than current WarmThroughput, decreasing \ + WarmThroughput is not supported" + ); + // U7: a provisioned table at 100/50 refuses 2/2 with the same sentence. + assert_eq!( + message(validate_warm_throughput_on_update( + &spec(Some(2), Some(2)), + TABLE, + (100, 50) + )), + read + ); + // UT1 and P3: the write form, whole. + let write = message(validate_warm_throughput_on_update( + &spec(Some(12_000), Some(3_999)), + TABLE, + (12_000, 4_000), + )); + assert_eq!( + write, + "One or more parameter values were invalid: Requested WriteUnitsPerSecond for \ + WarmThroughput for table is lower than current WarmThroughput, decreasing \ + WarmThroughput is not supported" + ); + assert_eq!( + message(validate_warm_throughput_on_update( + &spec(Some(5), Some(4)), + TABLE, + (5, 5) + )), + write + ); +} + +/// Rows G1, G2, G3, UG1 and UG2: the index form of every sentence, whole. +#[test] +fn index_sentences_name_the_index() { + assert_eq!( + message(validate_warm_throughput_on_create( + &spec(Some(11_999), Some(4_000)), + GSI, + BillingMode::PayPerRequest, + None, + )), + ON_DEMAND_READ_TABLE.replace("for table", "for index gsi") + ); + assert_eq!( + message(validate_warm_throughput_on_create( + &spec(Some(12_000), Some(3_999)), + GSI, + BillingMode::PayPerRequest, + None, + )), + "One or more parameter values were invalid: Requested WriteUnitsPerSecond for \ + WarmThroughput for index gsi is lower than initial throughput for OnDemand. See: \ + https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/\ + on-demand-capacity-mode.html#on-demand-capacity-mode-initial" + ); + let units = five_five(); + assert_eq!( + message(validate_warm_throughput_on_create( + &spec(Some(1), Some(1)), + GSI, + BillingMode::Provisioned, + Some(&units), + )), + "One or more parameter values were invalid: Requested ReadUnitsPerSecond for \ + WarmThroughput for index gsi is lower than ReadCapacityUnits of ProvisionedThroughput" + ); + assert_eq!( + message(validate_warm_throughput_on_update( + &spec(Some(11_999), Some(4_000)), + GSI, + (12_000, 4_000), + )), + "One or more parameter values were invalid: Requested ReadUnitsPerSecond for \ + WarmThroughput for index gsi is lower than current WarmThroughput, decreasing \ + WarmThroughput is not supported" + ); + assert_eq!( + message(validate_warm_throughput_on_update( + &spec(Some(12_000), Some(3_999)), + GSI, + (12_000, 4_000), + )), + "One or more parameter values were invalid: Requested WriteUnitsPerSecond for \ + WarmThroughput for index gsi is lower than current WarmThroughput, decreasing \ + WarmThroughput is not supported" + ); +} + +#[test] +fn create_table_checks_the_table_then_each_index_by_name() { + let input: CreateTableInput = serde_json::from_value(serde_json::json!({ + "TableName": "t", + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "AttributeDefinitions": [{"AttributeName": "pk", "AttributeType": "S"}], + "BillingMode": "PAY_PER_REQUEST", + "WarmThroughput": {"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 4000}, + "GlobalSecondaryIndexes": [{ + "IndexName": "by-g", + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "ALL"}, + "WarmThroughput": {"ReadUnitsPerSecond": 1, "WriteUnitsPerSecond": 1} + }] + })) + .expect("input"); + let msg = message(validate_create_table_warm_throughput(&input)); + assert_eq!( + msg, + ON_DEMAND_READ_TABLE.replace("for table", "for index by-g") + ); + let mut ok = input.clone(); + ok.global_secondary_indexes = None; + validate_create_table_warm_throughput(&ok).expect("table alone"); +} + +/// The per-companion rules, as measured (service-pairs run and the review +/// rounds of 2026-10-23): a table-level WarmThroughput alone is accepted; +/// SSE, TableClass and DeletionProtection companions answer their own +/// sentence, in that order; OnDemandThroughput answers its own, with or +/// without BillingMode; an index Delete answers its own; Streams, billing mode +/// and provisioned throughput answer the WarmThroughput sentence; an index +/// Update companion is accepted. +#[test] +fn a_table_warm_throughput_must_be_the_only_operation() { + let input = |extra: serde_json::Value| -> UpdateTableInput { + let mut v = serde_json::json!({ + "TableName": "t", + "WarmThroughput": {"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 4000} + }); + for (k, val) in extra.as_object().expect("object") { + v[k] = val.clone(); + } + serde_json::from_value(v).expect("input") + }; + let gsi_update = serde_json::json!({"GlobalSecondaryIndexUpdates": [{"Update": { + "IndexName": "gsi", "WarmThroughput": {"ReadUnitsPerSecond": 12000}}}]}); + validate_warm_throughput_is_only_operation(&input(serde_json::json!({}))).expect("alone"); + validate_warm_throughput_is_only_operation(&input(serde_json::json!({ + "AttributeDefinitions": [{"AttributeName": "pk", "AttributeType": "S"}] + }))) + .expect("attribute definitions are not an operation"); + validate_warm_throughput_is_only_operation(&input(gsi_update.clone())) + .expect("an index update beside a table WarmThroughput is accepted"); + + let sse = serde_json::json!({"Enabled": true}); + let stream_off = serde_json::json!({"StreamEnabled": false}); + let stream_on = serde_json::json!({"StreamEnabled": true, "StreamViewType": "NEW_IMAGE"}); + let delete = serde_json::json!([{"Delete": {"IndexName": "gsi"}}]); + let cases = [ + // S1a, S1b, S1c: SSE outranks TableClass, DeletionProtection and Streams. + ( + serde_json::json!({"SSESpecification": sse}), + SSE_MUST_BE_ONLY_OPERATION, + ), + ( + serde_json::json!({"SSESpecification": sse, "TableClass": "STANDARD"}), + SSE_MUST_BE_ONLY_OPERATION, + ), + ( + serde_json::json!({"SSESpecification": sse, "DeletionProtectionEnabled": false}), + SSE_MUST_BE_ONLY_OPERATION, + ), + ( + serde_json::json!({"SSESpecification": sse, "StreamSpecification": stream_on}), + SSE_MUST_BE_ONLY_OPERATION, + ), + ( + serde_json::json!({"TableClass": "STANDARD"}), + TABLE_CLASS_MUST_BE_ONLY_OPERATION, + ), + ( + serde_json::json!({"DeletionProtectionEnabled": true}), + DELETION_PROTECTION_MUST_BE_ONLY_OPERATION, + ), + // T4d: OnDemandThroughput without BillingMode. + ( + serde_json::json!({"OnDemandThroughput": {"MaxReadRequestUnits": 10}}), + ON_DEMAND_THROUGHPUT_NEEDS_BILLING_MODE, + ), + // T4a: with BillingMode the sentence is the same (the service names + // BillingMode in it, and still refuses). + ( + serde_json::json!({"BillingMode": "PAY_PER_REQUEST", + "OnDemandThroughput": {"MaxReadRequestUnits": 10}}), + ON_DEMAND_THROUGHPUT_NEEDS_BILLING_MODE, + ), + // T4b, T4c: TableClass and DeletionProtection outrank OnDemandThroughput. + ( + serde_json::json!({"TableClass": "STANDARD", + "OnDemandThroughput": {"MaxReadRequestUnits": 10}}), + TABLE_CLASS_MUST_BE_ONLY_OPERATION, + ), + ( + serde_json::json!({"DeletionProtectionEnabled": true, + "OnDemandThroughput": {"MaxReadRequestUnits": 10}}), + DELETION_PROTECTION_MUST_BE_ONLY_OPERATION, + ), + // S9a: an index Delete; V3b: TableClass outranks it; V3a: with Streams + // the service names the stream change. + ( + serde_json::json!({"GlobalSecondaryIndexUpdates": delete}), + DELETE_GSI_WITH_WARM_THROUGHPUT, + ), + ( + serde_json::json!({"GlobalSecondaryIndexUpdates": delete, "TableClass": "STANDARD"}), + TABLE_CLASS_MUST_BE_ONLY_OPERATION, + ), + ( + serde_json::json!({"GlobalSecondaryIndexUpdates": delete, + "StreamSpecification": stream_on}), + INDEX_CHANGE_WITH_STREAM_CHANGE, + ), + ( + serde_json::json!({"StreamSpecification": stream_off}), + WARM_THROUGHPUT_MUST_BE_ONLY_OPERATION, + ), + ( + serde_json::json!({"BillingMode": "PROVISIONED", + "ProvisionedThroughput": {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}}), + WARM_THROUGHPUT_MUST_BE_ONLY_OPERATION, + ), + // V2: ProvisionedThroughput without BillingMode. + ( + serde_json::json!({"ProvisionedThroughput": + {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}}), + WARM_THROUGHPUT_MUST_BE_ONLY_OPERATION, + ), + // S1d: DeletionProtection outranks Streams. + ( + serde_json::json!({"DeletionProtectionEnabled": true, + "StreamSpecification": stream_off}), + DELETION_PROTECTION_MUST_BE_ONLY_OPERATION, + ), + // V4: the triple answers the TableClass sentence. + ( + serde_json::json!({"TableClass": "STANDARD", "DeletionProtectionEnabled": true, + "StreamSpecification": stream_on}), + TABLE_CLASS_MUST_BE_ONLY_OPERATION, + ), + ]; + for (extra, expected) in cases { + assert_eq!( + message(validate_warm_throughput_is_only_operation(&input( + extra.clone() + ))), + expected, + "{extra}" + ); + } + // No table-level member: this function refuses nothing, whatever else is + // there (the companions' own rules are out of scope here). + let no_warm: UpdateTableInput = serde_json::from_value(serde_json::json!({ + "TableName": "t", + "DeletionProtectionEnabled": true, + "TableClass": "STANDARD", + "SSESpecification": sse, + "StreamSpecification": {"StreamEnabled": false}, + "GlobalSecondaryIndexUpdates": [{"Update": { + "IndexName": "gsi", "WarmThroughput": {"ReadUnitsPerSecond": 12000}}}] + })) + .expect("input"); + validate_warm_throughput_is_only_operation(&no_warm).expect("no table-level member"); +} + +/// The sentences, whole (rows S1a, S9a, V3a). +#[test] +fn the_companion_sentences_are_the_measured_strings() { + assert_eq!( + SSE_MUST_BE_ONLY_OPERATION, + "One or more parameter values were invalid: Server-Side Encryption modification must \ + be the only operation in the request" + ); + assert_eq!( + DELETE_GSI_WITH_WARM_THROUGHPUT, + "One or more parameter values were invalid: Delete global secondary index cannot be \ + specified when updating WarmThroughput" + ); + assert_eq!( + INDEX_CHANGE_WITH_STREAM_CHANGE, + "One or more parameter values were invalid: You cannot create or delete index while \ + changing stream status" + ); +} + +/// Rows S10, T5a, T5c (missing table): a bad value alone, and an index +/// `Update` alone, are not refused here, so the request reaches the table +/// lookup and the service's `ResourceNotFoundException`. Rows S10b, T5b, T5d: +/// with a companion this function answers first, before the table is read. +#[test] +fn a_missing_table_reaches_the_lookup_unless_a_companion_answers_first() { + let input = |v: serde_json::Value| -> UpdateTableInput { + let mut v = v; + v["TableName"] = serde_json::json!("missing"); + serde_json::from_value(v).expect("input") + }; + // S10, T5a: a value below any default, nothing else. + validate_warm_throughput_is_only_operation(&input(serde_json::json!({ + "WarmThroughput": {"ReadUnitsPerSecond": 11999, "WriteUnitsPerSecond": 4000} + }))) + .expect("S10: reaches the lookup"); + validate_warm_throughput_is_only_operation(&input(serde_json::json!({ + "WarmThroughput": {"ReadUnitsPerSecond": 1} + }))) + .expect("T5a: reaches the lookup"); + // T5c: an index Update alone. + validate_warm_throughput_is_only_operation(&input(serde_json::json!({ + "GlobalSecondaryIndexUpdates": [{"Update": { + "IndexName": "gsi", "WarmThroughput": {"ReadUnitsPerSecond": 12000}}}] + }))) + .expect("T5c: reaches the lookup"); + let with = |extra: serde_json::Value| { + let mut v = serde_json::json!({"WarmThroughput": {"ReadUnitsPerSecond": 12000}}); + v[extra + .as_object() + .expect("object") + .keys() + .next() + .expect("key") + .as_str()] = extra + .as_object() + .expect("object") + .values() + .next() + .expect("value") + .clone(); + message(validate_warm_throughput_is_only_operation(&input(v))) + }; + assert_eq!( + with(serde_json::json!({"TableClass": "STANDARD_INFREQUENT_ACCESS"})), + TABLE_CLASS_MUST_BE_ONLY_OPERATION, + "S10b" + ); + assert_eq!( + with(serde_json::json!({"StreamSpecification": + {"StreamEnabled": true, "StreamViewType": "NEW_IMAGE"}})), + WARM_THROUGHPUT_MUST_BE_ONLY_OPERATION, + "T5b" + ); + assert_eq!( + with(serde_json::json!({"DeletionProtectionEnabled": false})), + DELETION_PROTECTION_MUST_BE_ONLY_OPERATION, + "T5d" + ); +} diff --git a/crates/engine/src/backup/overrides.rs b/crates/engine/src/backup/overrides.rs index 5adf8c3c6..ac6b07eec 100644 --- a/crates/engine/src/backup/overrides.rs +++ b/crates/engine/src/backup/overrides.rs @@ -46,8 +46,12 @@ //! on-demand restore is accepted and not recorded (ExtendDB keeps no //! per-index on-demand maximum); on a PROVISIONED restore it contradicts the //! billing mode and is refused. A member of an entry outside the service -//! model's (`WarmThroughput`, for one) is refused, since ExtendDB cannot apply -//! it. An `OnDemandThroughputOverride` replaces the backup's on-demand +//! model's is refused, since ExtendDB cannot apply it. A GSI entry's +//! `WarmThroughput` is refused with the service's own sentence (measured +//! 2026-10-23, row S6d). A restored table and its indexes report the +//! on-demand warm throughput defaults, as the service's restore does (row +//! S6c); the backup records no warm throughput (row S6b). An +//! `OnDemandThroughputOverride` replaces the backup's on-demand //! throughput as a whole: a member it leaves out means no maximum, as on //! `CreateTable`. @@ -132,6 +136,12 @@ pub(super) fn gsi_throughput_required(index_name: &str) -> DynamoDbError { )) } +/// The service's refusal of a `WarmThroughput` member on a +/// `GlobalSecondaryIndexOverride` entry (measured, row S6d), after the +/// "One or more parameter values were invalid: " prefix. +const WARM_THROUGHPUT_OVERRIDE: &str = + "WarmThroughput is not supported on Global Secondary Indexes via RestoreTableFromBackup"; + fn invalid(detail: String) -> DynamoDbError { DynamoDbError::ValidationException(format!( "One or more parameter values were invalid: {detail}" @@ -357,6 +367,9 @@ fn apply_gsis( .map(|(entry, recorded)| { let given = &entry.index; let name = &given.index_name; + if entry.unknown.iter().any(|m| m == "WarmThroughput") { + return Err(invalid(WARM_THROUGHPUT_OVERRIDE.to_owned())); + } same_definition( name, &entry.unknown, @@ -383,6 +396,7 @@ fn apply_gsis( key_schema: recorded.key_schema.clone(), projection: recorded.projection.clone(), provisioned_throughput: given.provisioned_throughput.clone(), + warm_throughput: None, }) }) .collect() diff --git a/crates/engine/src/backup/tests.rs b/crates/engine/src/backup/tests.rs index f7bd86281..d42221b11 100644 --- a/crates/engine/src/backup/tests.rs +++ b/crates/engine/src/backup/tests.rs @@ -213,6 +213,8 @@ fn described(request: &RestoreRequest) -> TableDescription { index_size_bytes: 0, item_count: 0, index_arn: String::new(), + warm_throughput: None, + warm_throughput_record: None, }) .collect() }), @@ -258,6 +260,7 @@ fn gsi(name: &str, pk: &str, pt: Option) -> GsiInput { non_key_attributes: None, }, provisioned_throughput: pt, + warm_throughput: None, } } @@ -1350,11 +1353,13 @@ async fn index_overrides_the_service_refuses_are_refused() { BillingMode is PAY_PER_REQUEST", ), ( - "a member ExtendDB cannot apply", + "a warm throughput on a GSI entry", json!({ "GlobalSecondaryIndexOverride": [entry_with( "WarmThroughput", json!({ "ReadUnitsPerSecond": 12000 }), )] }), - "index gsi1: member WarmThroughput is not supported", + // Row S6d, whole: the service's own sentence. + "One or more parameter values were invalid: WarmThroughput is not supported on \ + Global Secondary Indexes via RestoreTableFromBackup", ), ( "an LSI member ExtendDB cannot apply", diff --git a/crates/engine/src/gsi_updates.rs b/crates/engine/src/gsi_updates.rs new file mode 100644 index 000000000..6ca3f7caf --- /dev/null +++ b/crates/engine/src/gsi_updates.rs @@ -0,0 +1,333 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The per-element rules for `UpdateTable`'s `GlobalSecondaryIndexUpdates`, +//! checked before any storage read. +//! +//! Measured against Amazon DynamoDB on 2026-10-23 (rows V1a, V1b, T3b, V3c, +//! T3d, T3e): +//! +//! - An element with two or more of `Create`, `Update`, `Delete`: +//! "Only one global secondary index action is allowed per +//! GlobalSecondaryIndexUpdate object" (V1a Update plus Create, V1b Create +//! plus Delete). +//! - Two actions naming the same index, in one element or in two: "Only one +//! global secondary index update per index is allowed simultaneously. +//! Index: " (T3b one element, V3c two). It is checked before the +//! one-action rule, so an Update and a Delete of one index answer it. +//! - An `Update` on an on-demand table that carries anything other than +//! `WarmThroughput` (a `ProvisionedThroughput`, or nothing at all): "The +//! only Updates for index: when TableThroughputMode is +//! PAY_PER_REQUEST can be to OnDemandThroughput, WarmThroughput" (T3d, +//! T3e). The billing mode is known only once the table is read, so that +//! rule lives in `warm_throughput::validate_warm_throughput_updates`; here +//! an `Update` without `WarmThroughput` on a table whose mode is not yet +//! known is refused as not yet supported (the service's sentence for a +//! provisioned table was not measured). +//! +//! A `Create` has a valid name, a key schema of defined attributes, and (when +//! given) a throughput of at least 1 unit each; a `Delete` has a valid name. + +use extenddb_core::error::DynamoDbError; +use extenddb_core::types::UpdateTableInput; + +/// Row V1a and V1b, whole. +pub const ONE_ACTION_PER_ELEMENT: &str = "One or more parameter values were invalid: Only one \ + global secondary index action is allowed per GlobalSecondaryIndexUpdate object"; + +/// Rows T3b and V3c: the sentence for `name`. +#[must_use] +pub fn one_update_per_index(name: &str) -> String { + format!( + "One or more parameter values were invalid: Only one global secondary index update per \ + index is allowed simultaneously. Index: {name}" + ) +} + +/// Rows T3d and T3e: the sentence for an on-demand table's index `name`. +#[must_use] +pub fn on_demand_update_members(name: &str) -> String { + format!( + "One or more parameter values were invalid: The only Updates for index: {name} when \ + TableThroughputMode is PAY_PER_REQUEST can be to OnDemandThroughput, WarmThroughput" + ) +} + +/// Whether any `Update` element carries nothing but its index name (a +/// `ProvisionedThroughput` on it is not a member of ExtendDB's action, so it +/// reads the same). Such an element needs the table's billing mode for its +/// sentence, so the caller reads the table first. +#[must_use] +pub(crate) fn has_bare_update(input: &UpdateTableInput) -> bool { + input + .global_secondary_index_updates + .iter() + .flatten() + .any(|u| { + u.update + .as_ref() + .is_some_and(|a| a.warm_throughput.is_none()) + }) +} + +/// Refuse every `Update` element that carries nothing but its index name: +/// on an on-demand table with the measured sentence (rows T3d, T3e); on a +/// provisioned table, where the service's sentence was not measured, with +/// ExtendDB's own "not yet supported". +/// +/// # Errors +/// +/// `ValidationException`. +pub(crate) fn refuse_bare_updates( + input: &UpdateTableInput, + pay_per_request: bool, +) -> Result<(), DynamoDbError> { + if let Some(upd) = input + .global_secondary_index_updates + .iter() + .flatten() + .filter_map(|u| u.update.as_ref()) + .find(|a| a.warm_throughput.is_none()) + { + return Err(DynamoDbError::ValidationException(if pay_per_request { + on_demand_update_members(&upd.index_name) + } else { + "UpdateGlobalSecondaryIndex is not yet supported".to_owned() + })); + } + Ok(()) +} + +/// Apply the element rules to `input`. +/// +/// # Errors +/// +/// `ValidationException` with the measured sentence (or the existing +/// ExtendDB sentence where the service's was not measured). +pub(crate) fn validate_gsi_updates(input: &UpdateTableInput) -> Result<(), DynamoDbError> { + let Some(updates) = &input.global_secondary_index_updates else { + return Ok(()); + }; + // Rows T3b and V3c: the per-index rule is applied to every name in every + // element before any element is looked at on its own, so an element that + // names one index twice (an Update and a Delete of `gsi`, row T3b) answers + // it rather than the one-action rule; two actions on different indexes + // (rows V1a, V1b) pass this and reach the one-action rule below. + let mut seen: Vec<&str> = Vec::new(); + for update in updates { + for name in [ + update.create.as_ref().map(|c| c.index_name.as_str()), + update.update.as_ref().map(|u| u.index_name.as_str()), + update.delete.as_ref().map(|d| d.index_name.as_str()), + ] + .into_iter() + .flatten() + { + if seen.contains(&name) { + return Err(DynamoDbError::ValidationException(one_update_per_index( + name, + ))); + } + seen.push(name); + } + } + for update in updates { + let actions = usize::from(update.create.is_some()) + + usize::from(update.update.is_some()) + + usize::from(update.delete.is_some()); + if actions > 1 { + return Err(DynamoDbError::ValidationException( + ONE_ACTION_PER_ELEMENT.to_owned(), + )); + } + if actions == 0 { + return Err(DynamoDbError::ValidationException( + "One or more parameter values were invalid: GlobalSecondaryIndexUpdate must \ + contain Create, Update, or Delete" + .to_owned(), + )); + } + if let Some(upd) = &update.update { + // An Update carrying WarmThroughput is stored (checked against the + // index's current values once the table is read). One carrying + // nothing else is refused by [`refuse_bare_updates`] once the + // table's billing mode is known. + extenddb_core::validation::validate_index_name(&upd.index_name)?; + } + if let Some(create) = &update.create { + extenddb_core::validation::validate_index_name(&create.index_name)?; + extenddb_core::validation::validate_index_throughput_minimum( + &create.index_name, + create.provisioned_throughput.as_ref(), + )?; + if create.key_schema.is_empty() { + return Err(DynamoDbError::ValidationException( + "One or more parameter values were invalid: KeySchema must not be empty for \ + GSI creation" + .to_owned(), + )); + } + let attr_defs = input.attribute_definitions.as_deref().unwrap_or(&[]); + for ks in &create.key_schema { + if !attr_defs + .iter() + .any(|ad| ad.attribute_name == ks.attribute_name) + { + return Err(DynamoDbError::ValidationException(format!( + "One or more parameter values were invalid: Some index key attributes \ + are not defined in AttributeDefinitions. Keys: [{}], \ + AttributeDefinitions: [{}]", + ks.attribute_name, + attr_defs + .iter() + .map(|ad| ad.attribute_name.as_str()) + .collect::>() + .join(", ") + ))); + } + } + } + if let Some(delete) = &update.delete { + extenddb_core::validation::validate_index_name(&delete.index_name)?; + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::{Value, json}; + + fn input(updates: Value) -> UpdateTableInput { + serde_json::from_value(json!({ + "TableName": "t", + "AttributeDefinitions": [{ "AttributeName": "g", "AttributeType": "S" }], + "GlobalSecondaryIndexUpdates": updates, + })) + .unwrap() + } + + fn message(result: Result<(), DynamoDbError>) -> String { + match result.expect_err("refused") { + DynamoDbError::ValidationException(m) => m, + other => panic!("{other:?}"), + } + } + + fn create(name: &str) -> Value { + json!({ "Create": { + "IndexName": name, + "KeySchema": [{ "AttributeName": "g", "KeyType": "HASH" }], + "Projection": { "ProjectionType": "ALL" }, + }}) + } + + fn warm_update(name: &str) -> Value { + json!({ "Update": { "IndexName": name, "WarmThroughput": { "ReadUnitsPerSecond": 12000 } } }) + } + + /// Rows V1a and V1b: an element with two actions, whichever two, and the + /// Create or Delete in it is never reached (the Create here keys on an + /// undefined attribute and would otherwise be refused differently). + #[test] + fn an_element_with_two_actions_is_refused_before_its_parts_are_checked() { + let mut mixed = warm_update("gsi"); + mixed["Create"] = json!({ + "IndexName": "bad", + "KeySchema": [{ "AttributeName": "nope", "KeyType": "HASH" }], + "Projection": { "ProjectionType": "ALL" }, + }); + assert_eq!( + message(validate_gsi_updates(&input(json!([mixed])))), + ONE_ACTION_PER_ELEMENT + ); + let mut update_delete = warm_update("gsi"); + update_delete["Delete"] = json!({ "IndexName": "other" }); + assert_eq!( + message(validate_gsi_updates(&input(json!([update_delete])))), + ONE_ACTION_PER_ELEMENT + ); + let mut create_delete = create("gsi-two"); + create_delete["Delete"] = json!({ "IndexName": "gsi" }); + assert_eq!( + message(validate_gsi_updates(&input(json!([create_delete])))), + ONE_ACTION_PER_ELEMENT + ); + } + + /// Row T3b: one element carrying an Update and a Delete of the same index + /// answers the per-index sentence, not the one-action sentence. + #[test] + fn one_element_naming_an_index_twice_answers_the_per_index_sentence() { + let mut same = warm_update("gsi"); + same["Delete"] = json!({ "IndexName": "gsi" }); + assert_eq!( + message(validate_gsi_updates(&input(json!([same])))), + one_update_per_index("gsi") + ); + } + + /// Rows T3b and V3c: two elements on one index. + #[test] + fn two_elements_on_one_index_are_refused() { + let msg = message(validate_gsi_updates(&input(json!([ + { "Delete": { "IndexName": "gsi" } }, + warm_update("gsi"), + ])))); + assert_eq!( + msg, + "One or more parameter values were invalid: Only one global secondary index update \ + per index is allowed simultaneously. Index: gsi" + ); + validate_gsi_updates(&input(json!([ + { "Delete": { "IndexName": "gsi" } }, + warm_update("other"), + ]))) + .expect("different indexes"); + } + + /// Rows T3d and T3e: an Update with nothing but the index name (or a + /// member ExtendDB's action does not carry) on an on-demand table answers + /// the measured sentence; on a provisioned table ExtendDB's own. + #[test] + fn a_bare_update_is_refused_with_the_billing_mode_sentence() { + let bare = input(json!([{ "Update": { "IndexName": "gsi" } }])); + validate_gsi_updates(&bare).expect("the element rules pass; the mode decides"); + assert!(has_bare_update(&bare)); + assert!(!has_bare_update(&input(json!([warm_update("gsi")])))); + assert_eq!( + message(refuse_bare_updates(&bare, true)), + "One or more parameter values were invalid: The only Updates for index: gsi when \ + TableThroughputMode is PAY_PER_REQUEST can be to OnDemandThroughput, WarmThroughput" + ); + assert!(message(refuse_bare_updates(&bare, false)).contains("not yet supported")); + refuse_bare_updates(&input(json!([warm_update("gsi")])), true).expect("warm update"); + } + + #[test] + fn a_created_gsi_throughput_below_one_is_refused() { + for (read, write) in [(0, 0), (0, 1), (1, 0), (-1, 5)] { + let mut c = create("gix"); + c["Create"]["ProvisionedThroughput"] = + json!({ "ReadCapacityUnits": read, "WriteCapacityUnits": write }); + let m = message(validate_gsi_updates(&input(json!([c])))); + assert!(m.ends_with("for index: gix"), "({read}, {write}): {m}"); + } + let mut ok = create("gix"); + ok["Create"]["ProvisionedThroughput"] = + json!({ "ReadCapacityUnits": 1, "WriteCapacityUnits": 1 }); + validate_gsi_updates(&input(json!([ok]))).unwrap(); + } + + #[test] + fn an_empty_element_and_an_undefined_key_attribute_are_refused() { + assert!(message(validate_gsi_updates(&input(json!([{}])))).contains("must contain")); + let mut c = create("gix"); + c["Create"]["KeySchema"] = json!([{ "AttributeName": "nope", "KeyType": "HASH" }]); + assert!( + message(validate_gsi_updates(&input(json!([c])))) + .contains("Some index key attributes are not defined") + ); + } +} diff --git a/crates/engine/src/import_export.rs b/crates/engine/src/import_export.rs index a6ecba4b9..dd9ad359d 100755 --- a/crates/engine/src/import_export.rs +++ b/crates/engine/src/import_export.rs @@ -357,6 +357,7 @@ fn create_table_input_from_params(tcp: &TableCreationParameters) -> CreateTableI deletion_protection_enabled: None, table_class: None, on_demand_throughput: None, + warm_throughput: None, } } diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index 74083fbfa..40592178f 100755 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -20,6 +20,7 @@ mod describe_table; mod expected; mod expression_helpers; mod get_item; +mod gsi_updates; mod import_export; mod import_export_io; mod index_helpers; @@ -41,6 +42,7 @@ mod ttl; mod update_item; mod update_table; mod vector_gate; +mod warm_throughput; pub use batch_get_item::handle_batch_get_item; pub use batch_write_item::handle_batch_write_item; diff --git a/crates/engine/src/update_table.rs b/crates/engine/src/update_table.rs index 6db763f82..061fd29de 100755 --- a/crates/engine/src/update_table.rs +++ b/crates/engine/src/update_table.rs @@ -88,18 +88,20 @@ pub async fn handle_update_table( .as_ref() .is_some_and(|u| !u.is_empty()); - // Validate: at least one field must be specified. + // Validate: at least one field must be specified. An UpdateTable carrying + // only WarmThroughput is accepted (measured 2026-10-23). if input.billing_mode.is_none() && input.provisioned_throughput.is_none() && input.deletion_protection_enabled.is_none() && input.stream_specification.is_none() && input.table_class.is_none() && input.on_demand_throughput.is_none() + && input.warm_throughput.is_none() && !has_gsi_updates && !has_vector_updates { return Err(DynamoDbError::ValidationException( - "At least one of BillingMode, ProvisionedThroughput, DeletionProtectionEnabled, StreamSpecification, or GlobalSecondaryIndexUpdates must be specified".to_owned(), + "At least one of BillingMode, ProvisionedThroughput, DeletionProtectionEnabled, StreamSpecification, GlobalSecondaryIndexUpdates, or WarmThroughput must be specified".to_owned(), )); } @@ -140,21 +142,39 @@ pub async fn handle_update_table( )); } - validate_gsi_updates(&input)?; + // A table-level WarmThroughput stands alone in its request, with the + // measured per-companion sentences (measured 2026-10-23). Checked before + // the value rules, as the service does: it fired on requests whose values + // were otherwise valid. GlobalSecondaryIndexUpdates is accepted beside it. + extenddb_core::validation::validate_warm_throughput_is_only_operation(&input)?; + + crate::gsi_updates::validate_gsi_updates(&input)?; // A created GSI's projected attributes count toward the table's limit // together with the indexes it already has: the total CreateTable checks, // and so the total a restore of this table's backups checks. - if let Some(updates) = &input.global_secondary_index_updates - && updates.iter().any(|u| { - u.create.as_ref().is_some_and(|c| { - c.projection - .non_key_attributes - .as_ref() - .is_some_and(|a| !a.is_empty()) + let projects_attributes = input + .global_secondary_index_updates + .as_ref() + .is_some_and(|u| { + u.iter().any(|u| { + u.create.as_ref().is_some_and(|c| { + c.projection + .non_key_attributes + .as_ref() + .is_some_and(|a| !a.is_empty()) + }) }) - }) - { + }); + // WarmThroughput may not go below the current values, a created index's + // follows the table's billing mode, and an index Update carrying nothing + // answers a sentence that names the mode: all need the table as it is. + let touches_warm = crate::warm_throughput::touches_warm_throughput(&input); + let bare_update = crate::gsi_updates::has_bare_update(&input); + // A switch to PAY_PER_REQUEST stores the on-demand minimum as the warm + // throughput of a provisioned table that is below it, so it needs the table. + let switches_to_on_demand = matches!(input.billing_mode, Some(BillingMode::PayPerRequest)); + if projects_attributes || touches_warm || bare_update || switches_to_on_demand { let existing = ctx .storage .describe_table( @@ -165,9 +185,29 @@ pub async fn handle_update_table( ) .await .map_err(|e| update_table_err_to_dynamo(e, &input.table_name))?; - extenddb_core::validation::validate_projected_attribute_total( - projected_after_gsi_updates(&existing, updates), - )?; + if let Some(updates) = &input.global_secondary_index_updates + && projects_attributes + { + extenddb_core::validation::validate_projected_attribute_total( + projected_after_gsi_updates(&existing, updates), + )?; + } + if bare_update { + let pay_per_request = existing + .billing_mode_summary + .as_ref() + .is_some_and(|s| s.billing_mode == BillingMode::PayPerRequest); + crate::gsi_updates::refuse_bare_updates(&input, pay_per_request)?; + } + if touches_warm { + crate::warm_throughput::validate_warm_throughput_updates(&input, &existing)?; + } + if switches_to_on_demand { + // After every validation: the client cannot send WarmThroughput + // beside BillingMode, so this is the only way both reach a backend. + input.warm_throughput = + crate::warm_throughput::on_demand_switch_warm_throughput(&existing); + } } let table_name = input.table_name.clone(); @@ -203,70 +243,6 @@ pub async fn handle_update_table( serialize_output(&output) } -/// The per-entry rules for `GlobalSecondaryIndexUpdates`, checked before any -/// storage read: each entry has exactly one of Create, Update, or Delete, -/// and a Create has a valid name, a key schema of defined attributes, and -/// (when given) a throughput of at least 1 unit each. -fn validate_gsi_updates(input: &UpdateTableInput) -> Result<(), DynamoDbError> { - if let Some(updates) = &input.global_secondary_index_updates { - for update in updates { - if update.create.is_some() && update.delete.is_some() { - return Err(DynamoDbError::ValidationException( - "One or more parameter values were invalid: Only one of Create or Delete can be specified per GlobalSecondaryIndexUpdate".to_owned(), - )); - } - if let Some(ref upd) = update.update { - // S3: Acknowledge the Update action but reject it as unsupported. - let _ = upd; - return Err(DynamoDbError::ValidationException( - "UpdateGlobalSecondaryIndex is not yet supported".to_owned(), - )); - } - if update.create.is_none() && update.delete.is_none() { - return Err(DynamoDbError::ValidationException( - "One or more parameter values were invalid: GlobalSecondaryIndexUpdate must contain Create, Update, or Delete".to_owned(), - )); - } - // M3: Validate index names the same way CreateTable does. - if let Some(create) = &update.create { - extenddb_core::validation::validate_index_name(&create.index_name)?; - extenddb_core::validation::validate_index_throughput_minimum( - &create.index_name, - create.provisioned_throughput.as_ref(), - )?; - if create.key_schema.is_empty() { - return Err(DynamoDbError::ValidationException( - "One or more parameter values were invalid: KeySchema must not be empty for GSI creation".to_owned(), - )); - } - // Validate that all key attributes are defined in AttributeDefinitions - let attr_defs = input.attribute_definitions.as_deref().unwrap_or(&[]); - for ks in &create.key_schema { - if !attr_defs - .iter() - .any(|ad| ad.attribute_name == ks.attribute_name) - { - return Err(DynamoDbError::ValidationException(format!( - "One or more parameter values were invalid: Some index key attributes are not defined in AttributeDefinitions. \ - Keys: [{}], AttributeDefinitions: [{}]", - ks.attribute_name, - attr_defs - .iter() - .map(|ad| ad.attribute_name.as_str()) - .collect::>() - .join(", ") - ))); - } - } - } - if let Some(delete) = &update.delete { - extenddb_core::validation::validate_index_name(&delete.index_name)?; - } - } - } - Ok(()) -} - /// The table's projected `NonKeyAttributes` once `updates` apply: its LSIs, /// its GSIs that are not deleted, and the GSIs created. fn projected_after_gsi_updates( @@ -351,34 +327,6 @@ mod tests { use super::*; use extenddb_storage::error::StorageError; - #[test] - fn a_created_gsi_throughput_below_one_is_refused() { - let input = |pt: Value| -> UpdateTableInput { - serde_json::from_value(serde_json::json!({ - "TableName": "t", - "AttributeDefinitions": [{ "AttributeName": "g", "AttributeType": "S" }], - "GlobalSecondaryIndexUpdates": [{ "Create": { - "IndexName": "gix", - "KeySchema": [{ "AttributeName": "g", "KeyType": "HASH" }], - "Projection": { "ProjectionType": "ALL" }, - "ProvisionedThroughput": pt, - }}], - })) - .unwrap() - }; - for (read, write) in [(0, 0), (0, 1), (1, 0), (-1, 5)] { - let pt = serde_json::json!({ "ReadCapacityUnits": read, "WriteCapacityUnits": write }); - match validate_gsi_updates(&input(pt)).unwrap_err() { - DynamoDbError::ValidationException(m) => { - assert!(m.ends_with("for index: gix"), "{m}"); - } - other => panic!("({read}, {write}): {other:?}"), - } - } - let ok = serde_json::json!({ "ReadCapacityUnits": 1, "WriteCapacityUnits": 1 }); - validate_gsi_updates(&input(ok)).unwrap(); - } - #[test] fn projected_total_counts_kept_indexes_and_created_ones() { use extenddb_core::types::{ @@ -400,6 +348,8 @@ mod tests { index_size_bytes: 0, item_count: 0, index_arn: String::new(), + warm_throughput: None, + warm_throughput_record: None, }, GsiDescription { index_name: "dropped".to_owned(), @@ -410,6 +360,8 @@ mod tests { index_size_bytes: 0, item_count: 0, index_arn: String::new(), + warm_throughput: None, + warm_throughput_record: None, }, ]), local_secondary_indexes: Some(vec![LsiDescription { diff --git a/crates/engine/src/warm_throughput.rs b/crates/engine/src/warm_throughput.rs new file mode 100644 index 000000000..6434be79c --- /dev/null +++ b/crates/engine/src/warm_throughput.rs @@ -0,0 +1,452 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! `UpdateTable` checks for `WarmThroughput`, against the table's current +//! description. +//! +//! The create-time rules need only the request; these need what the table +//! reports now (an update may not lower a value, and a created index follows +//! the table's billing mode), so they run after a `DescribeTable` of the +//! target. Measured against Amazon DynamoDB on 2026-10-23; see +//! `extenddb_core::validation::warm_throughput` for the sentences. + +use extenddb_core::error::DynamoDbError; +use extenddb_core::types::{ + BillingMode, ON_DEMAND_WARM_READ_UNITS, ON_DEMAND_WARM_WRITE_UNITS, TableDescription, + UpdateTableInput, WarmThroughput, WarmThroughputDefaults, epoch_seconds_now, +}; +use extenddb_core::validation::{ + WarmThroughputSubject, validate_created_gsi_warm_throughput, validate_warm_throughput_on_update, +}; + +/// Whether `input` carries any `WarmThroughput`, on the table or in a +/// `GlobalSecondaryIndexUpdates` action. +#[must_use] +pub(crate) fn touches_warm_throughput(input: &UpdateTableInput) -> bool { + input.warm_throughput.is_some() + || input + .global_secondary_index_updates + .iter() + .flatten() + .any(|u| { + u.update + .as_ref() + .is_some_and(|a| a.warm_throughput.is_some()) + || u.create + .as_ref() + .is_some_and(|a| a.warm_throughput.is_some()) + }) +} + +/// The warm throughput the table reports now (the pair the window started +/// from while UPDATING, measured rows U3 and S2b) and the billing mode it is +/// under, from a description the backend returned. +fn current_table_values(existing: &TableDescription) -> (BillingMode, (i64, i64)) { + let now = epoch_seconds_now(); + let pay_per_request = existing + .billing_mode_summary + .as_ref() + .is_some_and(|s| s.billing_mode == BillingMode::PayPerRequest); + let defaults = WarmThroughputDefaults::for_billing( + pay_per_request, + existing.provisioned_throughput.read_capacity_units, + existing.provisioned_throughput.write_capacity_units, + ); + let current = existing.warm_throughput_record.as_ref().map_or_else( + || { + ( + defaults.read_units_per_second, + defaults.write_units_per_second, + ) + }, + |r| r.reported(defaults, now), + ); + let mode = if pay_per_request { + BillingMode::PayPerRequest + } else { + BillingMode::Provisioned + }; + (mode, current) +} + +/// The `WarmThroughput` an `UpdateTable` that switches a provisioned table to +/// PAY_PER_REQUEST stores with the switch, or `None` when the switch changes +/// nothing about it. +/// +/// Measured against Amazon DynamoDB on 2026-10-23: a provisioned table whose +/// warm throughput is below the on-demand minimum reads 12000 read and 4000 +/// write units once it is on-demand, whether the client set it (`{100}` on a +/// 5/5 table, rows T7a to T7d; `100/50` on an explicit table, row S7c) or +/// never did (a never-set 10/10 table, row S7a); a side at or above the +/// minimum is kept. The values stay when the table is switched back to +/// provisioned (row S7b: `12000/4000` after the flip back to 5/5), so they are +/// stored, not derived from the billing mode at read time. Each side becomes +/// the larger of what the table settles on (its stored side, else its +/// provisioned units) and the on-demand minimum, and both sides are stored. +/// +/// The service answers the switch with the previous values and `Status: +/// ACTIVE` while the table is `UPDATING`; ExtendDB stores the values through +/// the ordinary update path, so the member reads `UPDATING` for +/// `control_plane_delay_seconds`, with the previous values only when the +/// record held them (a never-set table reports the on-demand values at +/// once). The client cannot send `WarmThroughput` beside `BillingMode` (the +/// only-operation rule refuses it), so a request reaching a backend with both +/// carries this reset. +/// +/// Global secondary indexes are not stored here: a provisioned index never +/// set reports the on-demand minimum while the table is on-demand, and its +/// own units again after a switch back; the service's behaviour for an index +/// was not measured. +#[must_use] +pub(crate) fn on_demand_switch_warm_throughput( + existing: &TableDescription, +) -> Option { + let already_on_demand = existing + .billing_mode_summary + .as_ref() + .is_some_and(|s| s.billing_mode == BillingMode::PayPerRequest); + if already_on_demand { + return None; + } + let provisioned = WarmThroughputDefaults::for_billing( + false, + existing.provisioned_throughput.read_capacity_units, + existing.provisioned_throughput.write_capacity_units, + ); + let record = existing.warm_throughput_record.as_ref(); + let (read, write) = record.map_or( + ( + provisioned.read_units_per_second, + provisioned.write_units_per_second, + ), + |r| r.effective(provisioned), + ); + let raised = ( + read.max(ON_DEMAND_WARM_READ_UNITS), + write.max(ON_DEMAND_WARM_WRITE_UNITS), + ); + let both_stored = record + .is_some_and(|r| r.read_units_per_second.is_some() && r.write_units_per_second.is_some()); + if both_stored && raised == (read, write) { + return None; + } + Some(WarmThroughput { + read_units_per_second: Some(raised.0), + write_units_per_second: Some(raised.1), + }) +} + +/// Apply the measured update rules to `input` against `existing`: +/// +/// - the table's `WarmThroughput` may not go below its current values +/// (the "for table" sentence); +/// - an `Update` action's `WarmThroughput` may not go below the named +/// index's current values (the "for index " sentence), and the +/// index must exist; +/// - a `Create` action's `WarmThroughput` follows the create rule under the +/// table's current billing mode. +/// +/// # Errors +/// +/// `ValidationException` with the measured sentence; +/// `ResourceNotFoundException` for an `Update` naming an index the table +/// does not have. +pub(crate) fn validate_warm_throughput_updates( + input: &UpdateTableInput, + existing: &TableDescription, +) -> Result<(), DynamoDbError> { + let (billing_mode, current) = current_table_values(existing); + let now = epoch_seconds_now(); + if let Some(spec) = &input.warm_throughput { + validate_warm_throughput_on_update(spec, WarmThroughputSubject::Table, current)?; + } + let pay_per_request = billing_mode == BillingMode::PayPerRequest; + for action in input.global_secondary_index_updates.iter().flatten() { + if let Some(create) = &action.create { + validate_created_gsi_warm_throughput(create, billing_mode)?; + } + if let Some(update) = &action.update + && let Some(spec) = &update.warm_throughput + { + let gsi = existing + .global_secondary_indexes + .iter() + .flatten() + .find(|g| g.index_name == update.index_name) + .ok_or_else(|| { + DynamoDbError::ResourceNotFoundException(format!( + "Requested resource not found: Index {} for table {}", + update.index_name, input.table_name + )) + })?; + let (rcu, wcu) = gsi.provisioned_throughput.as_ref().map_or((0, 0), |pt| { + (pt.read_capacity_units, pt.write_capacity_units) + }); + let defaults = WarmThroughputDefaults::for_billing(pay_per_request, rcu, wcu); + let current = gsi.warm_throughput_record.as_ref().map_or_else( + || { + ( + defaults.read_units_per_second, + defaults.write_units_per_second, + ) + }, + |r| r.reported(defaults, now), + ); + validate_warm_throughput_on_update( + spec, + WarmThroughputSubject::Index(&update.index_name), + current, + )?; + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use extenddb_core::types::{ + BillingModeSummary, GsiDescription, Projection, ProjectionType, + ProvisionedThroughputDescription, WarmThroughputRecord, + }; + use serde_json::json; + + fn on_demand_table(record: Option) -> TableDescription { + TableDescription { + table_name: "t".to_owned(), + billing_mode_summary: Some(BillingModeSummary { + billing_mode: BillingMode::PayPerRequest, + last_update_to_pay_per_request_date_time: None, + }), + warm_throughput_record: record, + global_secondary_indexes: Some(vec![GsiDescription { + index_name: "gsi".to_owned(), + key_schema: Vec::new(), + projection: Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }, + index_status: "ACTIVE".to_owned(), + provisioned_throughput: Some(ProvisionedThroughputDescription::default()), + index_size_bytes: 0, + item_count: 0, + index_arn: String::new(), + warm_throughput: None, + warm_throughput_record: None, + }]), + ..Default::default() + } + } + + fn input(v: serde_json::Value) -> UpdateTableInput { + let mut v = v; + v["TableName"] = json!("t"); + serde_json::from_value(v).expect("input") + } + + /// U4: below the on-demand default on a table that never set one. + #[test] + fn a_table_update_below_the_current_default_is_refused() { + let err = validate_warm_throughput_updates( + &input(json!({"WarmThroughput": {"ReadUnitsPerSecond": 11999}})), + &on_demand_table(None), + ) + .expect_err("refused"); + assert!(matches!(err, DynamoDbError::ValidationException(m) if m.contains("decreasing"))); + } + + /// U1 and U2: same and higher are accepted. + #[test] + fn a_table_update_at_or_above_current_is_accepted() { + let existing = on_demand_table(Some(WarmThroughputRecord { + read_units_per_second: Some(20_000), + write_units_per_second: None, + previous_read_units_per_second: None, + previous_write_units_per_second: None, + updating_until: 0.0, + })); + validate_warm_throughput_updates( + &input(json!({"WarmThroughput": {"ReadUnitsPerSecond": 20000, "WriteUnitsPerSecond": 4000}})), + &existing, + ) + .expect("same"); + validate_warm_throughput_updates( + &input(json!({"WarmThroughput": {"ReadUnitsPerSecond": 20001}})), + &existing, + ) + .expect("higher"); + // The stored read side is what "current" means. + validate_warm_throughput_updates( + &input(json!({"WarmThroughput": {"ReadUnitsPerSecond": 12000}})), + &existing, + ) + .expect_err("below the stored 20000"); + } + + /// U6: an index update follows the same rule against the index's values. + #[test] + fn an_index_update_is_checked_against_the_index_and_must_name_one() { + let existing = on_demand_table(None); + validate_warm_throughput_updates( + &input(json!({"GlobalSecondaryIndexUpdates": [{"Update": { + "IndexName": "gsi", + "WarmThroughput": {"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 4000} + }}]})), + &existing, + ) + .expect("same as the index default"); + let low = validate_warm_throughput_updates( + &input(json!({"GlobalSecondaryIndexUpdates": [{"Update": { + "IndexName": "gsi", + "WarmThroughput": {"WriteUnitsPerSecond": 3999} + }}]})), + &existing, + ) + .expect_err("below"); + assert!( + matches!(&low, DynamoDbError::ValidationException(m) + if m.contains("WriteUnitsPerSecond for WarmThroughput for index gsi is lower than current")), + "{low:?}" + ); + let missing = validate_warm_throughput_updates( + &input(json!({"GlobalSecondaryIndexUpdates": [{"Update": { + "IndexName": "nope", + "WarmThroughput": {"ReadUnitsPerSecond": 12000} + }}]})), + &existing, + ) + .expect_err("no such index"); + assert!(matches!( + missing, + DynamoDbError::ResourceNotFoundException(_) + )); + } + + /// A created index's WarmThroughput follows the create rule under the + /// table's billing mode. + #[test] + fn a_created_index_is_checked_under_the_table_billing_mode() { + let err = validate_warm_throughput_updates( + &input(json!({"GlobalSecondaryIndexUpdates": [{"Create": { + "IndexName": "new", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "ALL"}, + "WarmThroughput": {"ReadUnitsPerSecond": 1} + }}]})), + &on_demand_table(None), + ) + .expect_err("below the on-demand minimum"); + assert!( + matches!(&err, DynamoDbError::ValidationException(m) + if m.contains("for index new is lower than initial throughput for OnDemand")), + "{err:?}" + ); + } + + fn provisioned_table(units: i64, record: Option) -> TableDescription { + TableDescription { + table_name: "t".to_owned(), + provisioned_throughput: ProvisionedThroughputDescription { + read_capacity_units: units, + write_capacity_units: units, + ..Default::default() + }, + warm_throughput_record: record, + ..Default::default() + } + } + + fn stored(read: Option, write: Option) -> Option { + Some(WarmThroughputRecord { + read_units_per_second: read, + write_units_per_second: write, + previous_read_units_per_second: None, + previous_write_units_per_second: None, + updating_until: 0.0, + }) + } + + fn pair(w: Option) -> Option<(i64, i64)> { + w.map(|w| { + ( + w.read_units_per_second.expect("read"), + w.write_units_per_second.expect("write"), + ) + }) + } + + /// Rows S7a, S7c, T7c: a switch to on-demand stores at least the on-demand + /// minimum on both sides, from a never-set table, a read-only record and + /// an explicit pair below it. + #[test] + fn a_switch_to_on_demand_stores_the_minimum_for_a_low_warm_throughput() { + assert_eq!( + pair(on_demand_switch_warm_throughput(&provisioned_table( + 10, None + ))), + Some((12_000, 4_000)), + "S7a: never set, 10/10" + ); + assert_eq!( + pair(on_demand_switch_warm_throughput(&provisioned_table( + 10, + stored(Some(100), None) + ))), + Some((12_000, 4_000)), + "T7c: {{100}} on a table now at 10/10" + ); + assert_eq!( + pair(on_demand_switch_warm_throughput(&provisioned_table( + 10, + stored(Some(100), Some(50)) + ))), + Some((12_000, 4_000)), + "S7c: explicit 100/50" + ); + } + + /// A side at or above the minimum is kept; nothing is stored when both + /// sides are stored and already at or above it, and none when the table is + /// on-demand already. + #[test] + fn a_switch_to_on_demand_keeps_sides_at_or_above_the_minimum() { + assert_eq!( + pair(on_demand_switch_warm_throughput(&provisioned_table( + 10, + stored(Some(20_000), None) + ))), + Some((20_000, 4_000)) + ); + assert_eq!( + pair(on_demand_switch_warm_throughput(&provisioned_table( + 10, + stored(Some(30_000), Some(9_000)) + ))), + None + ); + assert_eq!( + on_demand_switch_warm_throughput(&on_demand_table(None)), + None + ); + // Provisioned units above the minimum are what the table settles on. + assert_eq!( + pair(on_demand_switch_warm_throughput(&provisioned_table( + 50_000, None + ))), + Some((50_000, 50_000)) + ); + } + + #[test] + fn touches_is_true_for_any_warm_throughput_member() { + assert!(!touches_warm_throughput(&input( + json!({"DeletionProtectionEnabled": true}) + ))); + assert!(touches_warm_throughput(&input( + json!({"WarmThroughput": {}}) + ))); + assert!(touches_warm_throughput(&input(json!({ + "GlobalSecondaryIndexUpdates": [{"Update": {"IndexName": "g", "WarmThroughput": {}}}] + })))); + } +} diff --git a/crates/storage-cassandra/migrations/catalog/V006__warm_throughput.cql b/crates/storage-cassandra/migrations/catalog/V006__warm_throughput.cql new file mode 100644 index 000000000..9e1f19a30 --- /dev/null +++ b/crates/storage-cassandra/migrations/catalog/V006__warm_throughput.cql @@ -0,0 +1,18 @@ +-- Copyright 2026 ExtendDB contributors +-- SPDX-License-Identifier: Apache-2.0 +-- WarmThroughput on tables and global secondary indexes (catalog version 0.0.5). +-- +-- tables.warm_throughput and indexes.warm_throughput hold the WarmThroughput a +-- client set, as one JSON text (see extenddb_core::types::WarmThroughputRecord). +-- The value is stored and reported by DescribeTable so clients that read the +-- member work. Nothing acts on it: there is no pre-warming and no scaling at +-- the database level, and it has no effect on throughput, throttling or +-- storage. A row written before this migration has no value and reports the +-- defaults for its billing mode. +-- +-- NOTE: the migration runner splits files on the semicolon character, +-- including inside comments. Do not use one in a comment. +ALTER TABLE tables ADD IF NOT EXISTS warm_throughput text; +ALTER TABLE indexes ADD IF NOT EXISTS warm_throughput text; + +INSERT INTO settings (key, value) VALUES ('catalog_version', '0.0.5'); diff --git a/crates/storage-cassandra/src/backup_engine.rs b/crates/storage-cassandra/src/backup_engine.rs index 7c070934e..d9d30ac0d 100755 --- a/crates/storage-cassandra/src/backup_engine.rs +++ b/crates/storage-cassandra/src/backup_engine.rs @@ -574,6 +574,7 @@ impl CassandraEngine { write_capacity_units: pt.write_capacity_units, } }), + warm_throughput: None, }) .collect(); let local_secondary_indexes = table diff --git a/crates/storage-cassandra/src/create_table.rs b/crates/storage-cassandra/src/create_table.rs index 5628926b4..4aa409fd4 100644 --- a/crates/storage-cassandra/src/create_table.rs +++ b/crates/storage-cassandra/src/create_table.rs @@ -10,7 +10,10 @@ use extenddb_core::types::{ TableStatus, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{index_arn, stream_arn, table_arn}; +use extenddb_storage::util::{ + index_arn, stream_arn, table_arn, warm_throughput_record_for_create, + warm_throughput_record_json, +}; use crate::CassandraEngine; use crate::backup_engine::RestoreHeartbeat; @@ -263,6 +266,17 @@ impl CassandraEngine { Value::from(deletion_protection), Value::from(status_transition_at), ]; + // WarmThroughput as sent, reported back and acted on by nothing (see + // catalog migration V006). UPDATING for the same delay the CREATING to + // ACTIVE flip takes. Named only when sent, for the same reason as the + // restore columns: a catalog that has not run V006 still serves an + // ordinary CreateTable. + let warm_record = + warm_throughput_record_for_create(input.warm_throughput.as_ref(), delay_seconds); + if let Some(record) = warm_record.as_ref() { + columns.push_str(", warm_throughput"); + values.push(Value::from(warm_throughput_record_json(record)?.as_str())); + } if let Some(r) = restore { columns.push_str( ", restore_source_backup_arn, restore_source_table_arn, restore_date_time, \ @@ -336,6 +350,7 @@ impl CassandraEngine { // Insert GSI metadata let mut gsi_index_ids: Vec = Vec::new(); + let mut gsi_warm_records = Vec::new(); if let Some(gsis) = &input.global_secondary_indexes { for gsi in gsis { let gsi_ks = serde_json::to_value(&gsi.key_schema) @@ -358,26 +373,39 @@ impl CassandraEngine { .map_err(|e| StorageError::Internal(e.to_string()))?; let index_id = uuid::Uuid::new_v4().to_string(); + let gsi_warm = + warm_throughput_record_for_create(gsi.warm_throughput.as_ref(), delay_seconds); + let mut index_values = vec![ + Value::from(table_id.as_str()), + Value::from(gsi.index_name.as_str()), + Value::from(index_id.as_str()), + Value::from(gsi_ks.to_string().as_str()), + Value::from(gsi_proj.to_string().as_str()), + match gsi_pt.as_ref() { + Some(v) => Value::from(v.to_string().as_str()), + None => Value::NotSet, + }, + ]; + // Named only when sent; see the table insert above. + let (warm_column, warm_placeholder) = match gsi_warm.as_ref() { + Some(record) => { + index_values + .push(Value::from(warm_throughput_record_json(record)?.as_str())); + (", warm_throughput", ", ?") + } + None => ("", ""), + }; + gsi_warm_records.push(gsi_warm); let insert_index_cql = format!( "INSERT INTO {catalog_keyspace}.indexes (table_id, index_name, index_id, index_type, \ - key_schema, projection, index_status, provisioned_throughput) \ - VALUES (?, ?, ?, 'GSI', ?, ?, 'ACTIVE', ?)" + key_schema, projection, index_status, provisioned_throughput{warm_column}) \ + VALUES (?, ?, ?, 'GSI', ?, ?, 'ACTIVE', ?{warm_placeholder})" ); self.session .query_with_values( &insert_index_cql, - cdrs_tokio::query::QueryValues::SimpleValues(vec![ - Value::from(table_id.as_str()), - Value::from(gsi.index_name.as_str()), - Value::from(index_id.as_str()), - Value::from(gsi_ks.to_string().as_str()), - Value::from(gsi_proj.to_string().as_str()), - match gsi_pt.as_ref() { - Some(v) => Value::from(v.to_string().as_str()), - None => Value::NotSet, - }, - ]), + cdrs_tokio::query::QueryValues::SimpleValues(index_values), ) .await .map_err(|e| { @@ -513,7 +541,8 @@ impl CassandraEngine { let gsis = input.global_secondary_indexes.as_ref().map(|gs| { gs.iter() - .map(|g| GsiDescription { + .zip(gsi_warm_records) + .map(|(g, warm)| GsiDescription { index_name: g.index_name.clone(), key_schema: g.key_schema.clone(), projection: g.projection.clone(), @@ -539,6 +568,8 @@ impl CassandraEngine { &input.table_name, &g.index_name, ), + warm_throughput: None, + warm_throughput_record: warm, }) .collect() }); @@ -644,6 +675,9 @@ impl CassandraEngine { restore_in_progress: true, }), vector_indexes: None, + // Derived by the engine from the record; see core. + warm_throughput: None, + warm_throughput_record: warm_record, }; Ok((description, heartbeat)) } diff --git a/crates/storage-cassandra/src/lib.rs b/crates/storage-cassandra/src/lib.rs index 8f49e9664..e4ba4c8f5 100644 --- a/crates/storage-cassandra/src/lib.rs +++ b/crates/storage-cassandra/src/lib.rs @@ -7,7 +7,7 @@ //! as the underlying database. It provides a DynamoDB-compatible API backed //! by Cassandra's distributed architecture. -pub const CATALOG_VERSION: &str = "0.0.4"; +pub const CATALOG_VERSION: &str = "0.0.5"; mod admin_store; mod authorization_store; @@ -33,6 +33,7 @@ pub mod table_engine; mod table_helpers; pub mod ttl_worker; mod update_table; +mod warm_throughput; mod worker_store; pub mod workers; diff --git a/crates/storage-cassandra/src/migrations.rs b/crates/storage-cassandra/src/migrations.rs index b1bc4fe20..4f70394fb 100644 --- a/crates/storage-cassandra/src/migrations.rs +++ b/crates/storage-cassandra/src/migrations.rs @@ -45,7 +45,7 @@ type CassandraSession = Session< >; /// Embedded catalog migration files, applied in order. -pub(crate) const CATALOG_MIGRATIONS: &[(&str, &str)] = &[ +pub const CATALOG_MIGRATIONS: &[(&str, &str)] = &[ ( "V001__initial_schema.cql", include_str!("../migrations/catalog/V001__initial_schema.cql"), @@ -66,6 +66,10 @@ pub(crate) const CATALOG_MIGRATIONS: &[(&str, &str)] = &[ "V005__backup_definitions_and_restore_registration.cql", include_str!("../migrations/catalog/V005__backup_definitions_and_restore_registration.cql"), ), + ( + "V006__warm_throughput.cql", + include_str!("../migrations/catalog/V006__warm_throughput.cql"), + ), ]; /// Embedded data migration files, applied in order. diff --git a/crates/storage-cassandra/src/table_helpers.rs b/crates/storage-cassandra/src/table_helpers.rs index 94b6eb0cf..b24531e2d 100644 --- a/crates/storage-cassandra/src/table_helpers.rs +++ b/crates/storage-cassandra/src/table_helpers.rs @@ -10,7 +10,7 @@ use extenddb_core::types::{ TableStatus, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{index_arn, stream_arn}; +use extenddb_storage::util::{index_arn, parse_warm_throughput_record, stream_arn}; use crate::CassandraEngine; @@ -165,6 +165,9 @@ impl CassandraEngine { .ok() .flatten() .and_then(|s: String| serde_json::from_str(&s).ok()); + // Absent on a catalog that has not run V006 (SELECT * tolerates it). + let warm_json: Option = table_row.get_by_name("warm_throughput").ok().flatten(); + let warm_throughput_record = parse_warm_throughput_record(warm_json.as_deref())?; // A restore target reports its registration for as long as the row // carries it; the engine shows it only while the table is CREATING. @@ -218,6 +221,7 @@ impl CassandraEngine { if index_type == "GSI" { let pt_str: Option = row.get_r_by_name("provisioned_throughput").ok(); + let warm_json: Option = row.get_by_name("warm_throughput").ok().flatten(); gsis.push(GsiDescription { index_name: index_name.clone(), @@ -228,6 +232,8 @@ impl CassandraEngine { index_size_bytes: 0, item_count: 0, index_arn: index_arn(&self.region, account_id, &table_name, &index_name), + warm_throughput: None, + warm_throughput_record: parse_warm_throughput_record(warm_json.as_deref())?, }); } else { lsis.push(LsiDescription { @@ -301,6 +307,8 @@ impl CassandraEngine { on_demand_throughput, restore_summary, vector_indexes: None, + warm_throughput: None, + warm_throughput_record, }) } } diff --git a/crates/storage-cassandra/src/update_table.rs b/crates/storage-cassandra/src/update_table.rs index 4a20b8618..187c6fa36 100644 --- a/crates/storage-cassandra/src/update_table.rs +++ b/crates/storage-cassandra/src/update_table.rs @@ -12,7 +12,9 @@ use extenddb_core::types::{ AttributeDefinition, BillingMode, KeySchemaElement, TableDescription, UpdateTableInput, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::effective_attribute_definitions; +use extenddb_storage::util::{ + effective_attribute_definitions, warm_throughput_record_for_create, warm_throughput_record_json, +}; use crate::CassandraEngine; use crate::cassandra_util::query_optional; @@ -151,6 +153,59 @@ impl CassandraEngine { add_update!("on_demand_throughput", odt_json.as_str()); } + // WarmThroughput on the table and on updated indexes: stored as sent, + // nothing acts on it. Validated by the engine against the current values. + // + // A request with a BillingMode and a WarmThroughput is the one the + // engine builds for a switch to PAY_PER_REQUEST (a client cannot send + // both). It must not fail on a catalog that has not run V006, where + // the column is absent: the switch is skipped past the warm throughput + // and the reported value still follows the on-demand minimum at read + // time. A request that names WarmThroughput on its own fails there, as + // documented. + let warm_json = match &input.warm_throughput { + Some(spec) => { + match self + .merged_table_warm_throughput(account_id, &input.table_name, spec) + .await + { + Ok(json) => Some(json), + Err(e) if input.billing_mode.is_some() => { + tracing::warn!( + "update_table: warm throughput not stored with the billing mode \ + change (catalog migration V006 not applied?): {e}" + ); + None + } + Err(e) => return Err(e), + } + } + None => None, + }; + if let Some(json) = warm_json.as_deref() { + add_update!("warm_throughput", json); + } + for update in input.global_secondary_index_updates.iter().flatten() { + if let Some(action) = &update.update + && let Some(spec) = &action.warm_throughput + { + let json = self + .merged_index_warm_throughput(&table_id, &action.index_name, spec) + .await?; + add_index_op!( + format!( + "UPDATE {catalog_ks}.indexes SET warm_throughput = ? \ + WHERE table_id = ? AND index_name = ?" + ), + QueryValues::SimpleValues(vec![ + Value::from(json.as_str()), + Value::from(table_id.as_str()), + Value::from(action.index_name.as_str()), + ]) + ); + } + } + // Stream specification: add to batch, then handle shard init separately // (init_stream_shards writes to a different keyspace and does its own batch). let mut needs_shard_init = false; @@ -272,21 +327,35 @@ impl CassandraEngine { .map_err(|e| StorageError::Internal(e.to_string()))? .unwrap_or_default(); + let mut index_values = vec![ + Value::from(table_id.as_str()), + Value::from(create.index_name.as_str()), + Value::from(index_id.as_str()), + Value::from(idx_ks_json.as_str()), + Value::from(proj_json.as_str()), + Value::from(pt_json.as_str()), + ]; + // WarmThroughput named only when sent, as in create_table. + let gsi_warm = warm_throughput_record_for_create( + create.warm_throughput.as_ref(), + self.control_plane_delay_secs().await, + ); + let (warm_column, warm_placeholder) = match gsi_warm.as_ref() { + Some(record) => { + index_values + .push(Value::from(warm_throughput_record_json(record)?.as_str())); + (", warm_throughput", ", ?") + } + None => ("", ""), + }; add_index_op!( format!( "INSERT INTO {catalog_ks}.indexes \ (table_id, index_name, index_id, index_type, key_schema, \ - projection, index_status, provisioned_throughput) \ - VALUES (?, ?, ?, 'GSI', ?, ?, 'CREATING', ?)" + projection, index_status, provisioned_throughput{warm_column}) \ + VALUES (?, ?, ?, 'GSI', ?, ?, 'CREATING', ?{warm_placeholder})" ), - QueryValues::SimpleValues(vec![ - Value::from(table_id.as_str()), - Value::from(create.index_name.as_str()), - Value::from(index_id.as_str()), - Value::from(idx_ks_json.as_str()), - Value::from(proj_json.as_str()), - Value::from(pt_json.as_str()), - ]) + QueryValues::SimpleValues(index_values) ); gsi_creates.push((index_id, create.index_name.clone())); surviving_index_key_schemas.push(create.key_schema.clone()); diff --git a/crates/storage-cassandra/src/warm_throughput.rs b/crates/storage-cassandra/src/warm_throughput.rs new file mode 100644 index 000000000..7581b9d3d --- /dev/null +++ b/crates/storage-cassandra/src/warm_throughput.rs @@ -0,0 +1,107 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Stored `WarmThroughput` for the Cassandra backend: the control-plane delay +//! the `UPDATING` window takes, and the `UpdateTable` merges. +//! +//! The record is written as sent (one JSON text per table and per index, see +//! catalog migration V006) and read back by DescribeTable; nothing pre-warms +//! or scales on it, and it has no effect on throughput, throttling or +//! storage. See `extenddb_core::types::WarmThroughputRecord`. + +use cdrs_tokio::types::IntoRustByName; +use extenddb_core::types::WarmThroughput; +use extenddb_storage::error::StorageError; +use extenddb_storage::util::{warm_throughput_record_for_update, warm_throughput_record_json}; + +use crate::CassandraEngine; +use crate::cassandra_util::query_optional; + +impl CassandraEngine { + /// `control_plane_delay_seconds` from the settings table (0.25 when + /// unset): the window a fresh record reports `UPDATING` for, the same + /// one the CREATING to ACTIVE flip uses. + pub(crate) async fn control_plane_delay_secs(&self) -> f64 { + let catalog_keyspace = self.catalog_keyspace(); + let query = format!("SELECT value FROM {catalog_keyspace}.settings WHERE key = ?"); + query_optional::( + &self.session, + &query, + cdrs_tokio::query_values!("control_plane_delay_seconds"), + "control_plane_delay_seconds", + ) + .await + .ok() + .flatten() + .and_then(|row| { + let value: String = row.get_r_by_name("value").ok()?; + value.parse::().ok() + }) + .unwrap_or(0.25) + } + + /// The JSON an `UpdateTable` stores in `tables.warm_throughput`: the + /// request merged over the current record. Read as its own statement so + /// a catalog that has not run V006 only fails a request that names the + /// member. + pub(crate) async fn merged_table_warm_throughput( + &self, + account_id: &str, + table_name: &str, + spec: &WarmThroughput, + ) -> Result { + let catalog_ks = self.catalog_keyspace(); + let row = query_optional::( + &self.session, + &format!( + "SELECT warm_throughput FROM {catalog_ks}.tables \ + WHERE account_id = ? AND table_name = ?" + ), + cdrs_tokio::query_values!(account_id, table_name), + "update_table warm throughput", + ) + .await? + .ok_or_else(|| StorageError::TableNotFound(table_name.to_owned()))?; + let existing: Option = row.get_by_name("warm_throughput").ok().flatten(); + let record = warm_throughput_record_for_update( + existing.as_deref(), + spec, + self.control_plane_delay_secs().await, + )?; + warm_throughput_record_json(&record) + } + + /// The JSON an `UpdateTable` stores in `indexes.warm_throughput` for a + /// `GlobalSecondaryIndexUpdates.Update`: the request merged over the + /// index's current record. `IndexNotFound` when the table has no such GSI. + pub(crate) async fn merged_index_warm_throughput( + &self, + table_id: &str, + index_name: &str, + spec: &WarmThroughput, + ) -> Result { + let catalog_ks = self.catalog_keyspace(); + let row = query_optional::( + &self.session, + &format!( + "SELECT index_type, warm_throughput FROM {catalog_ks}.indexes \ + WHERE table_id = ? AND index_name = ?" + ), + cdrs_tokio::query_values!(table_id, index_name), + "update_table gsi warm throughput", + ) + .await? + .ok_or_else(|| StorageError::IndexNotFound(index_name.to_owned()))?; + let index_type: Option = row.get_by_name("index_type").ok().flatten(); + if index_type.as_deref() != Some("GSI") { + return Err(StorageError::IndexNotFound(index_name.to_owned())); + } + let existing: Option = row.get_by_name("warm_throughput").ok().flatten(); + let record = warm_throughput_record_for_update( + existing.as_deref(), + spec, + self.control_plane_delay_secs().await, + )?; + warm_throughput_record_json(&record) + } +} diff --git a/crates/storage-cassandra/tests/common/mod.rs b/crates/storage-cassandra/tests/common/mod.rs index c1fd1e971..5a367bb51 100644 --- a/crates/storage-cassandra/tests/common/mod.rs +++ b/crates/storage-cassandra/tests/common/mod.rs @@ -262,6 +262,7 @@ impl TestTable { tags: None, table_class: None, deletion_protection_enabled: None, + warm_throughput: None, }; let table_desc = engine.create_table(account_id, input).await.unwrap(); @@ -326,6 +327,7 @@ impl TestTable { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }; let input = extenddb_core::types::CreateTableInput { @@ -344,6 +346,7 @@ impl TestTable { tags: None, table_class: None, deletion_protection_enabled: None, + warm_throughput: None, }; let table_desc = engine.create_table(&account_id, input).await.unwrap(); @@ -419,6 +422,7 @@ impl TestTable { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }; let input = extenddb_core::types::CreateTableInput { @@ -437,6 +441,7 @@ impl TestTable { tags: None, table_class: None, deletion_protection_enabled: None, + warm_throughput: None, }; let table_desc = engine.create_table(&account_id, input).await.unwrap(); @@ -534,6 +539,7 @@ impl TestTable { tags: None, table_class: None, deletion_protection_enabled: None, + warm_throughput: None, }; let table_desc = engine.create_table(&account_id, input).await.unwrap(); diff --git a/crates/storage-cassandra/tests/direct/backup_engine.rs b/crates/storage-cassandra/tests/direct/backup_engine.rs index 30fe919ac..ff5f8a663 100644 --- a/crates/storage-cassandra/tests/direct/backup_engine.rs +++ b/crates/storage-cassandra/tests/direct/backup_engine.rs @@ -48,21 +48,21 @@ fn key(id: &str, sort: Option<&str>) -> Item { key } -fn ks(name: &str, key_type: KeyType) -> KeySchemaElement { +pub(crate) fn ks(name: &str, key_type: KeyType) -> KeySchemaElement { KeySchemaElement { attribute_name: name.to_owned(), key_type, } } -fn attr(name: &str, attribute_type: ScalarAttributeType) -> AttributeDefinition { +pub(crate) fn attr(name: &str, attribute_type: ScalarAttributeType) -> AttributeDefinition { AttributeDefinition { attribute_name: name.to_owned(), attribute_type, } } -async fn activate_tables(engine: &CassandraEngine) { +pub(crate) async fn activate_tables(engine: &CassandraEngine) { tokio::time::sleep(Duration::from_millis(350)).await; engine .process_control_plane_transitions() @@ -113,7 +113,11 @@ async fn put_all(engine: &CassandraEngine, key_info: &TableKeyInfo, items: Vec TableDescription { +pub(crate) async fn describe( + engine: &CassandraEngine, + account_id: &str, + name: &str, +) -> TableDescription { engine .describe_table( account_id, @@ -657,6 +661,7 @@ async fn test_restore_reproduces_gsi_and_lsi_contents() { non_key_attributes: Some(vec!["value".to_owned()]), }, provisioned_throughput: None, + warm_throughput: None, }]), local_secondary_indexes: Some(vec![LsiInput { index_name: "ByLocal".to_owned(), @@ -1360,6 +1365,7 @@ async fn test_backup_without_definition_restores_as_legacy() { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]), ..Default::default() }, @@ -1571,26 +1577,42 @@ async fn test_restore_of_deleted_backup_is_not_found() { // ── Isolated catalogs (own keyspace prefix) ──────────────────────────────── -/// Catalog migrations V001-V004: the catalog shape before V005 added -/// `backups_by_arn.definition/active_restores` and `tables.restore_*`. -const CATALOG_THROUGH_V004: &[&str] = &[ +/// The catalog migration files in order, so a test can build the catalog +/// shape of an earlier release by applying the first N statement by +/// statement, as the runner does. +pub(crate) const CATALOG_MIGRATION_FILES: &[&str] = &[ include_str!("../../migrations/catalog/V001__initial_schema.cql"), include_str!("../../migrations/catalog/V002__backup_restore.cql"), include_str!("../../migrations/catalog/V003__account_scoped_idempotency.cql"), include_str!("../../migrations/catalog/V004__ttl_backfill_cursor.cql"), + include_str!("../../migrations/catalog/V005__backup_definitions_and_restore_registration.cql"), ]; /// An engine on its own keyspace prefix (so its sweeps and workers touch no /// other test's tables) with one provisioned account. With `through_v004` /// the catalog has run only catalog migrations V001-V004, as a catalog not /// yet migrated by `extenddb migrate --yes`; otherwise every migration. -async fn isolated_engine( +pub(crate) async fn isolated_engine( tag: &str, through_v004: bool, ) -> ( std::sync::Arc, String, crate::helpers::DropKeyspacesOnDrop, +) { + isolated_engine_through(tag, if through_v004 { Some(4) } else { None }).await +} + +/// [`isolated_engine`] with the catalog at the shape after the first +/// `through` catalog migrations (recorded in `schema_history` as the runner +/// records them), or fully migrated when `None`. +pub(crate) async fn isolated_engine_through( + tag: &str, + through: Option, +) -> ( + std::sync::Arc, + String, + crate::helpers::DropKeyspacesOnDrop, ) { let mut config = crate::helpers::test_config(); config.keyspace_prefix = format!("f4{tag}_{}", &crate::helpers::unique_test_id()[..10]); @@ -1609,26 +1631,29 @@ async fn isolated_engine( .await .unwrap(); } - if through_v004 { + if let Some(through) = through { // Statement by statement, as the migration runner does, recording - // versions 1-4 in schema_history as the runner did for a 0.0.3 catalog. + // the applied versions in schema_history as the runner did. session.query(format!("USE {catalog_ks}")).await.unwrap(); - for file in CATALOG_THROUGH_V004 { + for file in &CATALOG_MIGRATION_FILES[..through] { for statement in file.split(';').map(str::trim).filter(|s| !s.is_empty()) { - session.query(statement).await.expect("V001-V004 statement"); + session + .query(statement) + .await + .expect("earlier catalog statement"); } } - for version in 1..=4_i32 { + for version in 1..=i32::try_from(through).expect("small") { session .query_with_values( format!( "INSERT INTO {catalog_ks}.schema_history (version, description, applied_at) \ - VALUES (?, 'through_v004', toTimestamp(now()))" + VALUES (?, 'earlier_release', toTimestamp(now()))" ), cdrs_tokio::query_values!(version), ) .await - .expect("record V001-V004"); + .expect("record earlier catalog versions"); } } else { extenddb_storage_cassandra::migrations::run_catalog_migrations(&session, &catalog_ks) @@ -1651,8 +1676,8 @@ async fn isolated_engine( (engine, account_id, drop_keyspaces) } -/// The columns catalog migration V005 adds are absent from `keyspace`. -async fn assert_catalog_lacks_v005(engine: &CassandraEngine, keyspace: &str) { +/// The column names of every table in `keyspace`. +pub(crate) async fn catalog_columns(engine: &CassandraEngine, keyspace: &str) -> Vec { let rows = extenddb_storage_cassandra::cassandra_util::query_rows::( &engine.session_arc(), "SELECT column_name FROM system_schema.columns WHERE keyspace_name = ?", @@ -1672,6 +1697,12 @@ async fn assert_catalog_lacks_v005(engine: &CassandraEngine, keyspace: &str) { .unwrap() }) .collect(); + columns +} + +/// The columns catalog migration V005 adds are absent from `keyspace`. +async fn assert_catalog_lacks_v005(engine: &CassandraEngine, keyspace: &str) { + let columns = catalog_columns(engine, keyspace).await; assert!(columns.iter().any(|c| c == "table_status"), "{columns:?}"); for v005 in [ "restore_source_backup_arn", @@ -1713,6 +1744,7 @@ async fn test_table_operations_work_on_a_catalog_without_v005() { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]), ..Default::default() }, @@ -1979,6 +2011,7 @@ async fn test_restore_heartbeat_runs_during_target_ddl() { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }); lsis.push(LsiInput { index_name: format!("L{i}"), diff --git a/crates/storage-cassandra/tests/direct/cassandra_engine.rs b/crates/storage-cassandra/tests/direct/cassandra_engine.rs index 52a5306c6..abf431cc1 100644 --- a/crates/storage-cassandra/tests/direct/cassandra_engine.rs +++ b/crates/storage-cassandra/tests/direct/cassandra_engine.rs @@ -125,6 +125,7 @@ mod tests { deletion_protection_enabled: None, table_class: None, tags: None, + warm_throughput: None, }; // Create table diff --git a/crates/storage-cassandra/tests/direct/streams.rs b/crates/storage-cassandra/tests/direct/streams.rs index 865774d3a..496f338f4 100644 --- a/crates/storage-cassandra/tests/direct/streams.rs +++ b/crates/storage-cassandra/tests/direct/streams.rs @@ -65,6 +65,7 @@ async fn setup_stream_table( tags: None, table_class: None, deletion_protection_enabled: None, + warm_throughput: None, }; let desc = engine.create_table(&account_id, input).await.unwrap(); diff --git a/crates/storage-cassandra/tests/direct/table_engine.rs b/crates/storage-cassandra/tests/direct/table_engine.rs index 1005f113d..2f3fbdc17 100644 --- a/crates/storage-cassandra/tests/direct/table_engine.rs +++ b/crates/storage-cassandra/tests/direct/table_engine.rs @@ -61,6 +61,7 @@ mod tests { tags: None, deletion_protection_enabled: None, table_class: None, + warm_throughput: None, }; engine diff --git a/crates/storage-cassandra/tests/direct/warm_throughput_catalog.rs b/crates/storage-cassandra/tests/direct/warm_throughput_catalog.rs new file mode 100644 index 000000000..07fd629c9 --- /dev/null +++ b/crates/storage-cassandra/tests/direct/warm_throughput_catalog.rs @@ -0,0 +1,158 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Catalog migration V006 (the stored `WarmThroughput` columns) on a catalog +//! that has not run it: the shape a deployment migrated by the previous +//! release has. +//! +//! Ordinary table and item operations work there, as they do without V005. +//! A request that names the member (`CreateTable` or `UpdateTable` with +//! `WarmThroughput`) needs the column and fails until `extenddb migrate +//! --yes` applies V006; `DescribeTable` reports the defaults. The catalog +//! version constant moves to 0.0.5 so `extenddb migrate` sees the catalog as +//! behind and applies V006 (without the bump the command reported nothing to +//! do while the backup paths listed V006 as pending and refused). + +use extenddb_core::types::{ + CreateTableInput, DeleteTableInput, TableStatus, UpdateTableInput, WarmThroughputStatus, +}; +use extenddb_storage::TableEngine; +use extenddb_storage_cassandra::CATALOG_VERSION; + +use crate::backup_engine::{activate_tables, catalog_columns, describe, isolated_engine_through}; + +/// The constant and the last migration file agree, the same tripwire the +/// PostgreSQL runner has: adding a migration without moving the version +/// leaves `extenddb migrate` with nothing to do. +#[test] +fn the_catalog_version_is_the_one_v006_writes() { + assert_eq!(CATALOG_VERSION, "0.0.5"); + let v006 = include_str!("../../migrations/catalog/V006__warm_throughput.cql"); + assert!( + v006.contains("('catalog_version', '0.0.5')"), + "V006 must stamp the catalog version the binary expects" + ); + assert_eq!( + extenddb_storage_cassandra::migrations::CATALOG_MIGRATIONS.len(), + 6, + "a catalog migration was added or removed; move CATALOG_VERSION and this count" + ); +} + +#[tokio::test] +async fn test_table_operations_work_on_a_catalog_without_v006() { + if crate::helpers::skip_without_cassandra() { + return; + } + let (engine, account_id, _drop_keyspaces) = isolated_engine_through("v5", Some(5)).await; + let columns = catalog_columns(&engine, &engine.catalog_keyspace()).await; + assert!( + columns.iter().any(|c| c == "restore_heartbeat"), + "V005 present" + ); + assert!( + !columns.iter().any(|c| c == "warm_throughput"), + "V006 must not have run: {columns:?}" + ); + + let input = |name: &str, warm: Option| -> CreateTableInput { + let mut v = serde_json::json!({ + "TableName": name, + "KeySchema": [{"AttributeName": "id", "KeyType": "HASH"}], + "AttributeDefinitions": [ + {"AttributeName": "id", "AttributeType": "S"}, + {"AttributeName": "g", "AttributeType": "S"} + ], + "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [{ + "IndexName": "ByG", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "ALL"} + }] + }); + if let Some(w) = warm { + v["WarmThroughput"] = w; + } + serde_json::from_value(v).expect("input") + }; + // A CreateTable without the member works and describes the defaults. + engine + .create_table(&account_id, input("Unmigrated", None)) + .await + .expect("CreateTable without WarmThroughput on a catalog without V006"); + activate_tables(&engine).await; + let mut described = describe(&engine, &account_id, "Unmigrated").await; + assert_eq!(described.table_status, TableStatus::Active); + described.prepare_for_response(); + let wt = described.warm_throughput.expect("defaults"); + assert_eq!( + ( + wt.read_units_per_second, + wt.write_units_per_second, + wt.status + ), + (Some(12_000), Some(4_000), WarmThroughputStatus::Active) + ); + + // Naming the member needs the column. + let refused = engine + .create_table( + &account_id, + input( + "NeedsV006", + Some(serde_json::json!({"ReadUnitsPerSecond": 20000})), + ), + ) + .await; + assert!( + refused.is_err(), + "CreateTable with WarmThroughput before V006" + ); + let update: UpdateTableInput = serde_json::from_value(serde_json::json!({ + "TableName": "Unmigrated", + "WarmThroughput": {"ReadUnitsPerSecond": 20000} + })) + .expect("update"); + assert!( + engine.update_table(&account_id, update).await.is_err(), + "UpdateTable with WarmThroughput before V006" + ); + + // `extenddb migrate --yes` applies V006 (the runner skips the recorded + // files); the same engine then stores the member without a restart. + extenddb_storage_cassandra::migrations::run_catalog_migrations( + &engine.session_arc(), + &engine.catalog_keyspace(), + ) + .await + .expect("apply V006"); + let columns = catalog_columns(&engine, &engine.catalog_keyspace()).await; + assert_eq!( + columns.iter().filter(|c| *c == "warm_throughput").count(), + 2, + "tables and indexes both gain the column" + ); + let update: UpdateTableInput = serde_json::from_value(serde_json::json!({ + "TableName": "Unmigrated", + "WarmThroughput": {"ReadUnitsPerSecond": 20000} + })) + .expect("update"); + let mut updated = engine + .update_table(&account_id, update) + .await + .expect("UpdateTable with WarmThroughput after V006"); + updated.prepare_for_response(); + assert_eq!( + updated.warm_throughput.expect("member").status, + WarmThroughputStatus::Updating + ); + engine + .delete_table( + &account_id, + DeleteTableInput { + table_name: "Unmigrated".to_owned(), + }, + ) + .await + .expect("DeleteTable"); +} diff --git a/crates/storage-cassandra/tests/direct_integration.rs b/crates/storage-cassandra/tests/direct_integration.rs index 43eb28b81..77b27f4d1 100644 --- a/crates/storage-cassandra/tests/direct_integration.rs +++ b/crates/storage-cassandra/tests/direct_integration.rs @@ -57,3 +57,5 @@ mod transact_write_items; mod transaction_ledger; #[path = "direct/users.rs"] mod users; +#[path = "direct/warm_throughput_catalog.rs"] +mod warm_throughput_catalog; diff --git a/crates/storage-cassandra/tests/ttl_integration.rs b/crates/storage-cassandra/tests/ttl_integration.rs index 279d79492..c9c106bb5 100644 --- a/crates/storage-cassandra/tests/ttl_integration.rs +++ b/crates/storage-cassandra/tests/ttl_integration.rs @@ -513,6 +513,7 @@ async fn test_ttl_sweep_emits_service_remove_stream_record() { tags: None, table_class: None, deletion_protection_enabled: None, + warm_throughput: None, }, ) .await @@ -3104,6 +3105,7 @@ async fn test_ttl_enabled_table_rejects_new_async_gsi() { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }), update: None, delete: None, @@ -3122,6 +3124,8 @@ async fn test_ttl_enabled_table_rejects_new_async_gsi() { table_class: None, on_demand_throughput: None, vector_index_updates: None, + warm_throughput: None, + sse_specification: None, }, ) .await; diff --git a/crates/storage-mongodb/src/backup_engine.rs b/crates/storage-mongodb/src/backup_engine.rs index beedee97c..17aee41fe 100644 --- a/crates/storage-mongodb/src/backup_engine.rs +++ b/crates/storage-mongodb/src/backup_engine.rs @@ -1826,6 +1826,7 @@ impl BackupEngine for MongoEngine { &index_doc, "provisioned_throughput", )?, + warm_throughput: None, }); } "LSI" => { @@ -2438,6 +2439,7 @@ mod tests { read_capacity_units: 7, write_capacity_units: 8, }), + warm_throughput: None, }; let kept = restore_gsi_throughput("PROVISIONED", Some(vec![gsi.clone()])).expect("gsis"); assert!(kept[0].provisioned_throughput.is_some()); diff --git a/crates/storage-mongodb/src/backup_live_tests.rs b/crates/storage-mongodb/src/backup_live_tests.rs index 75ed23761..8c69b381e 100644 --- a/crates/storage-mongodb/src/backup_live_tests.rs +++ b/crates/storage-mongodb/src/backup_live_tests.rs @@ -167,6 +167,7 @@ async fn source_table_for(engine: &MongoEngine, account: &str, name: &str, items non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]), local_secondary_indexes: Some(vec![LsiInput { index_name: "l1".to_owned(), @@ -1688,6 +1689,8 @@ async fn live_update_table_on_a_creating_restore_target_is_refused() { table_class: None, on_demand_throughput: None, vector_index_updates: None, + warm_throughput: None, + sse_specification: None, }; let changes = vec![ UpdateTableInput { @@ -1720,6 +1723,7 @@ async fn live_update_table_on_a_creating_restore_target_is_refused() { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }), update: None, delete: None, diff --git a/crates/storage-mongodb/src/lib.rs b/crates/storage-mongodb/src/lib.rs index c1d28882f..459f97829 100644 --- a/crates/storage-mongodb/src/lib.rs +++ b/crates/storage-mongodb/src/lib.rs @@ -24,6 +24,7 @@ pub mod pushdown; mod stream_engine; mod table_engine; mod ttl_worker; +mod warm_throughput; mod worker_store; pub use bootstrapper::MongoBootstrapper; diff --git a/crates/storage-mongodb/src/table_engine.rs b/crates/storage-mongodb/src/table_engine.rs index aa8c5272c..9e8b31eca 100644 --- a/crates/storage-mongodb/src/table_engine.rs +++ b/crates/storage-mongodb/src/table_engine.rs @@ -218,6 +218,12 @@ impl MongoEngine { // table return ResourceNotFound, matching DynamoDB and the postgres // backend. let delay_secs = self.control_plane_delay_seconds().await; + // WarmThroughput as sent, reported back and acted on by nothing. + // UPDATING for the same delay the CREATING to ACTIVE flip takes. + let (warm_record, warm_bson) = crate::warm_throughput::warm_record_bson_for_create( + input.warm_throughput.as_ref(), + delay_secs, + )?; let (table_status, status_transition_at): (&str, bson::Bson) = if defer_active { // Restore path: enter CREATING with no scheduled transition. The // caller flips the table to ACTIVE (or schedules the transition) @@ -254,6 +260,7 @@ impl MongoEngine { "table_class": table_class_bson, "sse_specification": sse_spec_bson, "on_demand_throughput": on_demand_bson, + "warm_throughput": warm_bson, }; if let Some(provenance) = restore { provenance.insert_into(&mut table_doc); @@ -351,6 +358,11 @@ impl MongoEngine { let index_status = if defer_active { "CREATING" } else { "ACTIVE" }; let backfill_mode = if defer_active { Some("restore") } else { None }; + let (gsi_warm_record, gsi_warm_bson) = + crate::warm_throughput::warm_record_bson_for_create( + gsi.warm_throughput.as_ref(), + delay_secs, + )?; let mut index_doc = doc! { "_id": { "table_id": &table_id, "index_name": &gsi.index_name }, "index_id": &index_id, @@ -359,6 +371,7 @@ impl MongoEngine { "projection": projection_bson, "index_status": index_status, "provisioned_throughput": index_pt_bson.unwrap_or(bson::Bson::Null), + "warm_throughput": gsi_warm_bson, }; if let Some(mode) = backfill_mode { index_doc.insert("backfill_mode", mode); @@ -395,6 +408,8 @@ impl MongoEngine { index_size_bytes: 0, item_count: 0, index_arn: index_arn_val, + warm_throughput: None, + warm_throughput_record: gsi_warm_record, }); } Some(descs) @@ -557,6 +572,7 @@ impl MongoEngine { sse_description, table_class_summary, on_demand_throughput: input.on_demand_throughput, + warm_throughput_record: warm_record, // Fields for features this backend does not implement, vector // indexes today, take their defaults. Adding one to // TableDescription then does not break this build. @@ -879,6 +895,11 @@ impl MongoEngine { } } + // WarmThroughput on the table and on updated indexes: stored as sent, + // nothing acts on it. Validated by the engine against the current values. + self.store_warm_throughput_updates(account_id, &input) + .await?; + if clear_provisioned_throughput { let table_doc = tables_coll .find_one(doc! { @@ -977,6 +998,10 @@ impl MongoEngine { // Live writes during the backfill window sync through // sync_indexes (upserts) so the eventual state is // convergent regardless of interleaving. + let (_, gsi_warm_bson) = crate::warm_throughput::warm_record_bson_for_create( + create.warm_throughput.as_ref(), + self.control_plane_delay_seconds().await, + )?; let index_doc = doc! { "_id": { "table_id": &desc.table_id, "index_name": &create.index_name }, "index_id": &index_id, @@ -985,6 +1010,7 @@ impl MongoEngine { "projection": projection_bson, "index_status": "CREATING", "provisioned_throughput": pt_bson.unwrap_or(bson::Bson::Null), + "warm_throughput": gsi_warm_bson, }; self.catalog_db @@ -1548,6 +1574,10 @@ impl MongoEngine { index_size_bytes: 0, item_count: 0, index_arn: idx_arn, + warm_throughput: None, + warm_throughput_record: crate::warm_throughput::warm_record_from_doc( + idx_doc, + )?, }); } "LSI" => { @@ -1644,6 +1674,7 @@ impl MongoEngine { table_class_summary, on_demand_throughput, restore_summary, + warm_throughput_record: crate::warm_throughput::warm_record_from_doc(doc)?, // Fields for features this backend does not implement, vector // indexes today, take their defaults. Adding one to // TableDescription then does not break this build. diff --git a/crates/storage-mongodb/src/warm_throughput.rs b/crates/storage-mongodb/src/warm_throughput.rs new file mode 100644 index 000000000..be1605e92 --- /dev/null +++ b/crates/storage-mongodb/src/warm_throughput.rs @@ -0,0 +1,169 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Stored `WarmThroughput` for `MongoEngine`: the BSON shape of the record on +//! table and index documents, and the `UpdateTable` merges. +//! +//! The record is written as sent and read back by DescribeTable; nothing +//! pre-warms or scales on it, and it has no effect on throughput, throttling +//! or storage. See `extenddb_core::types::WarmThroughputRecord`. + +use bson::{Bson, Document, doc}; +use extenddb_core::types::{ + UpdateTableInput, WarmThroughput, WarmThroughputRecord, epoch_seconds_now, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::util::warm_throughput_record_for_create; + +use crate::MongoEngine; + +/// The `warm_throughput` field value for a record: the record as a document, +/// or `Null` when the client never set one. +pub(crate) fn warm_record_bson( + record: Option<&WarmThroughputRecord>, +) -> Result { + match record { + Some(r) => bson::to_bson(r).map_err(|e| StorageError::Internal(e.to_string())), + None => Ok(Bson::Null), + } +} + +/// The record on a table or index document, if any. +/// +/// # Errors +/// +/// `Internal` when the stored field is present but not a record. +pub(crate) fn warm_record_from_doc( + doc: &Document, +) -> Result, StorageError> { + match doc.get("warm_throughput") { + None | Some(Bson::Null) => Ok(None), + Some(b) => bson::from_bson(b.clone()) + .map(Some) + .map_err(|e| StorageError::Internal(format!("warm throughput record: {e}"))), + } +} + +/// The field a create path writes for a request's `WarmThroughput`. +pub(crate) fn warm_record_bson_for_create( + spec: Option<&WarmThroughput>, + delay_secs: f64, +) -> Result<(Option, Bson), StorageError> { + let record = warm_throughput_record_for_create(spec, delay_secs); + let bson = warm_record_bson(record.as_ref())?; + Ok((record, bson)) +} + +impl MongoEngine { + /// Store the request's table-level `WarmThroughput` and each + /// `GlobalSecondaryIndexUpdates.Update.WarmThroughput`, merged over what + /// is stored. Read-modify-write without a transaction, like the rest of + /// this backend's UpdateTable. The engine has already validated the + /// values against the current ones. + pub(crate) async fn store_warm_throughput_updates( + &self, + account_id: &str, + input: &UpdateTableInput, + ) -> Result<(), StorageError> { + let gsi_updates: Vec<_> = input + .global_secondary_index_updates + .iter() + .flatten() + .filter_map(|u| u.update.as_ref()) + .filter_map(|u| u.warm_throughput.as_ref().map(|wt| (&u.index_name, wt))) + .collect(); + if input.warm_throughput.is_none() && gsi_updates.is_empty() { + return Ok(()); + } + let now = epoch_seconds_now(); + let updating_until = now + self.control_plane_delay_seconds().await.max(0.0); + let tables_coll = self.catalog_db.collection::("tables"); + let table_filter = + doc! { "_id": { "account_id": account_id, "table_name": &input.table_name } }; + let table_doc = tables_coll + .find_one(table_filter.clone()) + .await + .map_err(|e| StorageError::Internal(e.to_string()))? + .ok_or_else(|| StorageError::TableNotFound(input.table_name.clone()))?; + + if let Some(spec) = &input.warm_throughput { + let record = WarmThroughputRecord::apply_update( + warm_record_from_doc(&table_doc)?, + spec, + updating_until, + now, + ); + tables_coll + .update_one( + table_filter, + doc! { "$set": { "warm_throughput": warm_record_bson(Some(&record))? } }, + ) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + + if gsi_updates.is_empty() { + return Ok(()); + } + let table_id = table_doc + .get_str("table_id") + .map_err(|_| StorageError::Internal("missing table_id".to_owned()))?; + let indexes_coll = self.catalog_db.collection::("indexes"); + for (index_name, spec) in gsi_updates { + let filter = doc! { + "_id": { "table_id": table_id, "index_name": index_name }, + "index_type": "GSI", + }; + let index_doc = indexes_coll + .find_one(filter.clone()) + .await + .map_err(|e| StorageError::Internal(e.to_string()))? + .ok_or_else(|| StorageError::IndexNotFound(index_name.clone()))?; + let record = WarmThroughputRecord::apply_update( + warm_record_from_doc(&index_doc)?, + spec, + updating_until, + now, + ); + indexes_coll + .update_one( + filter, + doc! { "$set": { "warm_throughput": warm_record_bson(Some(&record))? } }, + ) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_record_round_trips_through_bson_and_null_reads_as_none() { + let record = WarmThroughputRecord { + read_units_per_second: Some(20_000), + write_units_per_second: None, + previous_read_units_per_second: Some(12_000), + previous_write_units_per_second: None, + updating_until: 1.5, + }; + let doc = doc! { "warm_throughput": warm_record_bson(Some(&record)).expect("bson") }; + assert_eq!(warm_record_from_doc(&doc).expect("parses"), Some(record)); + let null = doc! { "warm_throughput": Bson::Null }; + assert_eq!(warm_record_from_doc(&null).expect("parses"), None); + assert_eq!(warm_record_from_doc(&doc! {}).expect("parses"), None); + assert_eq!(warm_record_bson(None).expect("bson"), Bson::Null); + } + + #[test] + fn garbage_in_the_field_is_an_internal_error() { + let bad = doc! { "warm_throughput": "not a record" }; + assert!(matches!( + warm_record_from_doc(&bad), + Err(StorageError::Internal(m)) if m.starts_with("warm throughput record:") + )); + } +} diff --git a/crates/storage-postgres/migrations/001_schema.sql b/crates/storage-postgres/migrations/001_schema.sql index 3203651f4..5aaf2669f 100644 --- a/crates/storage-postgres/migrations/001_schema.sql +++ b/crates/storage-postgres/migrations/001_schema.sql @@ -35,6 +35,12 @@ CREATE TABLE IF NOT EXISTS tables ( table_class TEXT, sse_specification JSONB, on_demand_throughput JSONB, + -- WarmThroughput as the client set it (see + -- extenddb_core::types::WarmThroughputRecord). Stored and reported only: + -- nothing pre-warms or scales on it, and it has no effect on throughput, + -- throttling or storage. A catalog created before this column existed + -- gets it from the shape step in migrations.rs. + warm_throughput JSONB, PRIMARY KEY (account_id, table_name), CONSTRAINT tables_table_id_unique UNIQUE (table_id) ); @@ -54,6 +60,8 @@ CREATE TABLE IF NOT EXISTS indexes ( index_status TEXT NOT NULL DEFAULT 'ACTIVE', provisioned_throughput JSONB, propagation_delay_ms INTEGER, + -- Same stored-only WarmThroughput record as tables.warm_throughput. + warm_throughput JSONB, PRIMARY KEY (table_id, index_name), CONSTRAINT indexes_table_id_fkey FOREIGN KEY (table_id) REFERENCES tables(table_id) ON DELETE CASCADE, diff --git a/crates/storage-postgres/src/backup_engine.rs b/crates/storage-postgres/src/backup_engine.rs index 20baee942..378b8d9a1 100755 --- a/crates/storage-postgres/src/backup_engine.rs +++ b/crates/storage-postgres/src/backup_engine.rs @@ -413,6 +413,7 @@ impl PostgresEngine { key_schema, projection, provisioned_throughput: throughput_from_catalog(ipt.as_ref()), + warm_throughput: None, }); } } diff --git a/crates/storage-postgres/src/create_table.rs b/crates/storage-postgres/src/create_table.rs index fc41c8c0b..eb61889fb 100755 --- a/crates/storage-postgres/src/create_table.rs +++ b/crates/storage-postgres/src/create_table.rs @@ -8,7 +8,7 @@ use extenddb_core::types::{ ProvisionedThroughputDescription, SseDescription, SseType, TableDescription, TableStatus, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{index_arn, stream_arn, table_arn}; +use extenddb_storage::util::{index_arn, stream_arn, table_arn, warm_throughput_record_for_create}; use crate::PostgresEngine; @@ -182,6 +182,18 @@ impl PostgresEngine { } } + // WarmThroughput as sent, reported back and acted on by nothing (see + // the shape step on `tables.warm_throughput`). UPDATING for the same + // delay the CREATING to ACTIVE flip takes. + let delay_secs = Self::control_plane_delay_secs(&mut tx).await?; + let warm_record = + warm_throughput_record_for_create(input.warm_throughput.as_ref(), delay_secs); + let warm_json = warm_record + .as_ref() + .map(serde_json::to_value) + .transpose() + .map_err(|e| StorageError::Internal(e.to_string()))?; + // Insert table metadata, returning creation timestamp and actual status. // Use PG error code 23505 for robust duplicate detection instead of string matching. // H-5: Insert as CREATING with a scheduled transition to ACTIVE, @@ -198,7 +210,8 @@ impl PostgresEngine { (account_id, table_name, key_schema, attribute_definitions, billing_mode, provisioned_throughput, stream_specification, table_status, creation_date_time, table_arn, table_id, deletion_protection_enabled, - status_transition_at, table_class, sse_specification, on_demand_throughput) + status_transition_at, table_class, sse_specification, on_demand_throughput, + warm_throughput) VALUES ($1, $2, $3, $4, $5, $6, $7, CASE WHEN $14 THEN 'CREATING' WHEN (SELECT secs FROM delay) = 0 @@ -209,7 +222,7 @@ impl PostgresEngine { THEN NULL ELSE NOW() + make_interval(secs => (SELECT secs FROM delay)) END, - $11, $12, $13) + $11, $12, $13, $15) RETURNING EXTRACT(EPOCH FROM creation_date_time)::FLOAT8, table_status", ) .bind(account_id) @@ -226,6 +239,7 @@ impl PostgresEngine { .bind(&sse_spec_json) .bind(&on_demand_json) .bind(defer_active) + .bind(&warm_json) .fetch_one(&mut *tx) .await; let (creation_epoch, actual_status): (f64, String) = match inserted { @@ -260,8 +274,17 @@ impl PostgresEngine { // ProvisionedThroughput) so DescribeTable can deserialize it without // failing on the missing NumberOfDecreasesToday field. let mut gsi_index_ids: Vec = Vec::new(); + let mut gsi_warm_records = Vec::new(); if let Some(gsis) = &input.global_secondary_indexes { for gsi in gsis { + let gsi_warm = + warm_throughput_record_for_create(gsi.warm_throughput.as_ref(), delay_secs); + let gsi_warm_json = gsi_warm + .as_ref() + .map(serde_json::to_value) + .transpose() + .map_err(|e| StorageError::Internal(e.to_string()))?; + gsi_warm_records.push(gsi_warm); let gsi_ks = serde_json::to_value(&gsi.key_schema) .map_err(|e| StorageError::Internal(e.to_string()))?; let gsi_proj = serde_json::to_value(&gsi.projection) @@ -285,8 +308,8 @@ impl PostgresEngine { sqlx::query( r"INSERT INTO indexes (table_id, index_name, index_id, index_type, key_schema, projection, - index_status, provisioned_throughput) - VALUES ($1, $2, $3, 'GSI', $4, $5, 'ACTIVE', $6)", + index_status, provisioned_throughput, warm_throughput) + VALUES ($1, $2, $3, 'GSI', $4, $5, 'ACTIVE', $6, $7)", ) .bind(&table_id) .bind(&gsi.index_name) @@ -294,6 +317,7 @@ impl PostgresEngine { .bind(&gsi_ks) .bind(&gsi_proj) .bind(&gsi_pt) + .bind(&gsi_warm_json) .execute(&mut *tx) .await .map_err(|e| StorageError::Internal(e.to_string()))?; @@ -552,7 +576,8 @@ impl PostgresEngine { let gsis = input.global_secondary_indexes.as_ref().map(|gs| { gs.iter() - .map(|g| GsiDescription { + .zip(gsi_warm_records) + .map(|(g, warm)| GsiDescription { index_name: g.index_name.clone(), key_schema: g.key_schema.clone(), projection: g.projection.clone(), @@ -578,6 +603,8 @@ impl PostgresEngine { &input.table_name, &g.index_name, ), + warm_throughput: None, + warm_throughput_record: warm, }) .collect() }); @@ -700,6 +727,9 @@ impl PostgresEngine { .as_ref() .map(|tc| serde_json::json!({ "TableClass": tc })), on_demand_throughput: input.on_demand_throughput, + // Derived by the engine from the record; see core. + warm_throughput: None, + warm_throughput_record: warm_record, // Every field is populated deliberately, with no // `..Default::default()` spread. This response is the complete // description of what was just created, so a new core field should diff --git a/crates/storage-postgres/src/delete_table.rs b/crates/storage-postgres/src/delete_table.rs index 7475eed6a..5bff85307 100755 --- a/crates/storage-postgres/src/delete_table.rs +++ b/crates/storage-postgres/src/delete_table.rs @@ -30,7 +30,8 @@ impl PostgresEngine { EXTRACT(EPOCH FROM creation_date_time)::FLOAT8 as creation_epoch, table_size_bytes, item_count, table_arn, table_id, deletion_protection_enabled, stream_label, - table_class, sse_specification, on_demand_throughput + table_class, sse_specification, on_demand_throughput, + warm_throughput FROM tables WHERE account_id = $1 AND table_name = $2 AND table_status IN ('ACTIVE', 'CREATING') FOR UPDATE", ) @@ -73,7 +74,7 @@ impl PostgresEngine { // Fetch indexes for the response description. let index_rows: Vec = sqlx::query_as( r"SELECT index_name, index_id, index_type, key_schema, projection, - index_status, provisioned_throughput + index_status, provisioned_throughput, warm_throughput FROM indexes WHERE table_id = $1", ) .bind(&row.table_id) diff --git a/crates/storage-postgres/src/lib.rs b/crates/storage-postgres/src/lib.rs index e9f1ae2e9..9e6d0ec1c 100755 --- a/crates/storage-postgres/src/lib.rs +++ b/crates/storage-postgres/src/lib.rs @@ -31,6 +31,7 @@ mod ttl_worker; mod update_table; mod vector; mod vector_search; +mod warm_throughput; mod worker_store; mod workers; diff --git a/crates/storage-postgres/src/migrations.rs b/crates/storage-postgres/src/migrations.rs index 2873bbc77..16cd4d8b8 100755 --- a/crates/storage-postgres/src/migrations.rs +++ b/crates/storage-postgres/src/migrations.rs @@ -136,6 +136,34 @@ const CATALOG_SHAPE_STEPS: &[ShapeStep] = &[ apply: "ALTER TABLE table_restores ALTER COLUMN restore_date_time DROP DEFAULT", at_serve: false, }, + // WarmThroughput as the client set it (one JSONB record per table and per + // index, see extenddb_core::types::WarmThroughputRecord). Stored and + // reported only: nothing pre-warms or scales on it, and it has no effect + // on throughput, throttling or storage. `001_schema.sql` creates the + // columns on a fresh catalog; these steps add them to a catalog that ran + // the earlier revision of that file. Nullable, so rows written before the + // column report the defaults for their billing mode. A shape step rather + // than a numbered migration, so no catalog version moves. + ShapeStep { + what: "add tables.warm_throughput", + needed: "SELECT EXISTS(SELECT 1 FROM information_schema.tables \ + WHERE table_schema = 'public' AND table_name = 'tables') \ + AND NOT EXISTS(SELECT 1 FROM information_schema.columns \ + WHERE table_schema = 'public' AND table_name = 'tables' \ + AND column_name = 'warm_throughput')", + apply: "ALTER TABLE tables ADD COLUMN IF NOT EXISTS warm_throughput JSONB", + at_serve: true, + }, + ShapeStep { + what: "add indexes.warm_throughput", + needed: "SELECT EXISTS(SELECT 1 FROM information_schema.tables \ + WHERE table_schema = 'public' AND table_name = 'indexes') \ + AND NOT EXISTS(SELECT 1 FROM information_schema.columns \ + WHERE table_schema = 'public' AND table_name = 'indexes' \ + AND column_name = 'warm_throughput')", + apply: "ALTER TABLE indexes ADD COLUMN IF NOT EXISTS warm_throughput JSONB", + at_serve: true, + }, ]; /// How long a startup shape step waits for its table lock, in seconds. An diff --git a/crates/storage-postgres/src/table_helpers.rs b/crates/storage-postgres/src/table_helpers.rs index 4dfb7f18f..d4d7fcad2 100755 --- a/crates/storage-postgres/src/table_helpers.rs +++ b/crates/storage-postgres/src/table_helpers.rs @@ -8,7 +8,7 @@ use extenddb_core::types::{ ProvisionedThroughputDescription, SseDescription, SseType, TableDescription, TableStatus, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{index_arn, stream_arn}; +use extenddb_storage::util::{index_arn, parse_warm_throughput_record, stream_arn}; use extenddb_storage::vector_catalog::{VectorIndexCatalogRow, vector_index_descriptions}; use crate::PostgresEngine; @@ -34,6 +34,7 @@ pub(crate) struct TableRow { pub table_class: Option, pub sse_specification: Option, pub on_demand_throughput: Option, + pub warm_throughput: Option, } /// Row type for index metadata queries. @@ -46,6 +47,7 @@ pub(crate) struct IndexRow { pub projection: serde_json::Value, pub index_status: String, pub provisioned_throughput: Option, + pub warm_throughput: Option, } /// A `vector_indexes` catalog row, in `FromRow` field order. @@ -112,7 +114,8 @@ impl PostgresEngine { EXTRACT(EPOCH FROM creation_date_time)::FLOAT8 as creation_epoch, table_size_bytes, item_count, table_arn, table_id, deletion_protection_enabled, stream_label, - table_class, sse_specification, on_demand_throughput + table_class, sse_specification, on_demand_throughput, + warm_throughput FROM tables WHERE account_id = $1 AND table_name = $2", ) .bind(account_id) @@ -125,7 +128,7 @@ impl PostgresEngine { let index_rows: Vec = sqlx::query_as( r"SELECT index_name, index_id, index_type, key_schema, projection, - index_status, provisioned_throughput + index_status, provisioned_throughput, warm_throughput FROM indexes WHERE table_id = $1", ) .bind(&row.table_id) @@ -193,6 +196,10 @@ impl PostgresEngine { .transpose() .map_err(|e: serde_json::Error| StorageError::Internal(e.to_string()))?; + let warm_json = idx + .warm_throughput + .as_ref() + .map(serde_json::Value::to_string); gsis.push(GsiDescription { index_name: idx.index_name.clone(), key_schema: ks, @@ -213,6 +220,8 @@ impl PostgresEngine { &row.table_name, &idx.index_name, ), + warm_throughput: None, + warm_throughput_record: parse_warm_throughput_record(warm_json.as_deref())?, }); } else { lsis.push(LsiDescription { @@ -330,6 +339,9 @@ impl PostgresEngine { on_demand_throughput: row .on_demand_throughput .and_then(|v| serde_json::from_value(v).ok()), + warm_throughput_record: parse_warm_throughput_record( + row.warm_throughput.map(|v| v.to_string()).as_deref(), + )?, // Vector indexes are read separately and applied by the caller: this // builder takes `index_rows` only, and the two callers differ in // whether they need them. `build_table_description` fills them in; diff --git a/crates/storage-postgres/src/update_table.rs b/crates/storage-postgres/src/update_table.rs index dec99a93b..9a3474414 100755 --- a/crates/storage-postgres/src/update_table.rs +++ b/crates/storage-postgres/src/update_table.rs @@ -7,7 +7,9 @@ use extenddb_core::types::{ AttributeDefinition, BillingMode, KeySchemaElement, TableDescription, UpdateTableInput, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{building_refuses, effective_attribute_definitions}; +use extenddb_storage::util::{ + building_refuses, effective_attribute_definitions, warm_throughput_record_for_create, +}; use crate::PostgresEngine; use crate::data::gsi_build::{ @@ -430,6 +432,10 @@ impl PostgresEngine { .map_err(|e| StorageError::Internal(e.to_string()))?; } + // WarmThroughput on the table and on updated indexes: stored as sent, + // nothing acts on it. Validated by the engine against the current values. + Self::store_warm_throughput_updates(&mut tx, account_id, &table_id, &input).await?; + // Apply GSI updates (create/delete). let mut created_index_ids: Vec = Vec::new(); let mut deleted_index_ids: Vec = Vec::new(); @@ -465,12 +471,20 @@ impl PostgresEngine { .transpose() .map_err(|e| StorageError::Internal(e.to_string()))?; + let gsi_warm = warm_throughput_record_for_create( + create.warm_throughput.as_ref(), + Self::control_plane_delay_secs(&mut tx).await?, + ) + .as_ref() + .map(serde_json::to_value) + .transpose() + .map_err(|e| StorageError::Internal(e.to_string()))?; let index_id = uuid::Uuid::new_v4().to_string(); sqlx::query( r"INSERT INTO indexes (table_id, index_name, index_id, index_type, key_schema, projection, - index_status, provisioned_throughput) - VALUES ($1, $2, $3, 'GSI', $4, $5, 'CREATING', $6)", + index_status, provisioned_throughput, warm_throughput) + VALUES ($1, $2, $3, 'GSI', $4, $5, 'CREATING', $6, $7)", ) .bind(&table_id) .bind(&create.index_name) @@ -478,6 +492,7 @@ impl PostgresEngine { .bind(&gsi_ks) .bind(&gsi_proj) .bind(&gsi_pt) + .bind(&gsi_warm) .execute(&mut *tx) .await .map_err(|e| StorageError::Internal(e.to_string()))?; @@ -1087,6 +1102,11 @@ impl PostgresEngine { } } + // The build outlasts a new GSI's WarmThroughput UPDATING window; start + // it again so the response reports it as the service does. + self.restart_created_gsi_warm_throughput(&table_id, &input) + .await?; + self.build_table_description(account_id, &input.table_name) .await } diff --git a/crates/storage-postgres/src/warm_throughput.rs b/crates/storage-postgres/src/warm_throughput.rs new file mode 100644 index 000000000..8d43d398d --- /dev/null +++ b/crates/storage-postgres/src/warm_throughput.rs @@ -0,0 +1,168 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Stored `WarmThroughput` for `PostgresEngine`: the control-plane delay the +//! `UPDATING` window takes, and the `UpdateTable` merges. +//! +//! The record is written as sent and read back by DescribeTable; nothing +//! pre-warms or scales on it, and it has no effect on throughput, throttling +//! or storage. See `extenddb_core::types::WarmThroughputRecord`. + +use extenddb_core::types::UpdateTableInput; +use extenddb_storage::error::StorageError; +use extenddb_storage::util::{ + warm_throughput_record_for_create, warm_throughput_record_for_update, +}; + +use crate::PostgresEngine; + +impl PostgresEngine { + /// `control_plane_delay_seconds` from the settings table (0.25 when + /// unset): the window a fresh record reports `UPDATING` for, the same + /// one the CREATING to ACTIVE flip uses. + /// + /// The two windows are timed by different clocks: the CREATING to ACTIVE + /// flip is scheduled from the database clock (`NOW()` in the create + /// INSERT), the record's `UPDATING` deadline is the application clock + /// (`epoch_seconds_now()`). On a database host whose clock differs from + /// the server's, the member can read `UPDATING` after the table is + /// `ACTIVE`, or `ACTIVE` before, by the difference. + pub(crate) async fn control_plane_delay_secs( + conn: &mut sqlx::PgConnection, + ) -> Result { + sqlx::query_scalar( + "SELECT COALESCE((SELECT value::FLOAT8 FROM settings \ + WHERE key = 'control_plane_delay_seconds'), 0.25)", + ) + .fetch_one(&mut *conn) + .await + .map_err(|e| StorageError::Internal(e.to_string())) + } + + /// Store the request's table-level `WarmThroughput` and each + /// `GlobalSecondaryIndexUpdates.Update.WarmThroughput`, merged over what + /// is stored, inside the caller's catalog transaction. The engine has + /// already validated the values against the current ones. + pub(crate) async fn store_warm_throughput_updates( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + account_id: &str, + table_id: &str, + input: &UpdateTableInput, + ) -> Result<(), StorageError> { + let gsi_updates: Vec<_> = input + .global_secondary_index_updates + .iter() + .flatten() + .filter_map(|u| u.update.as_ref()) + .filter_map(|u| u.warm_throughput.as_ref().map(|wt| (&u.index_name, wt))) + .collect(); + if input.warm_throughput.is_none() && gsi_updates.is_empty() { + return Ok(()); + } + let delay_secs = Self::control_plane_delay_secs(tx).await?; + let to_json = |record| { + serde_json::to_value(record).map_err(|e| StorageError::Internal(e.to_string())) + }; + + if let Some(spec) = &input.warm_throughput { + let existing: Option = sqlx::query_scalar( + "SELECT warm_throughput FROM tables WHERE account_id = $1 AND table_name = $2", + ) + .bind(account_id) + .bind(&input.table_name) + .fetch_one(&mut **tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + let existing = existing.map(|v| v.to_string()); + let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs)?; + sqlx::query( + "UPDATE tables SET warm_throughput = $1 WHERE account_id = $2 AND table_name = $3", + ) + .bind(to_json(record)?) + .bind(account_id) + .bind(&input.table_name) + .execute(&mut **tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + + for (index_name, spec) in gsi_updates { + let existing: Option> = sqlx::query_scalar( + "SELECT warm_throughput FROM indexes \ + WHERE table_id = $1 AND index_name = $2 AND index_type = 'GSI'", + ) + .bind(table_id) + .bind(index_name) + .fetch_optional(&mut **tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + let Some(existing) = existing else { + return Err(StorageError::IndexNotFound(index_name.clone())); + }; + let existing = existing.map(|v| v.to_string()); + let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs)?; + sqlx::query( + "UPDATE indexes SET warm_throughput = $1 WHERE table_id = $2 AND index_name = $3", + ) + .bind(to_json(record)?) + .bind(table_id) + .bind(index_name) + .execute(&mut **tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + Ok(()) + } + + /// Start the `UPDATING` window of each GSI this `UpdateTable` created with + /// `WarmThroughput` again, now that its build has run. + /// + /// The row is inserted with the window starting at the insert, inside the + /// catalog transaction, and the build that follows waits at least + /// `GRACE_PUBLISH` (1 s) before it publishes the index, longer than the + /// default 0.25 s window. Without this the response, which is described + /// after the build, would report the new index's `WarmThroughput` `ACTIVE` + /// where the service reports it `UPDATING`. + pub(crate) async fn restart_created_gsi_warm_throughput( + &self, + table_id: &str, + input: &UpdateTableInput, + ) -> Result<(), StorageError> { + let creates: Vec<_> = input + .global_secondary_index_updates + .iter() + .flatten() + .filter_map(|u| u.create.as_ref()) + .filter(|c| c.warm_throughput.is_some()) + .collect(); + if creates.is_empty() { + return Ok(()); + } + let mut conn = self + .pool + .acquire() + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + let delay_secs = Self::control_plane_delay_secs(&mut conn).await?; + for create in creates { + let Some(record) = + warm_throughput_record_for_create(create.warm_throughput.as_ref(), delay_secs) + else { + continue; + }; + let json = + serde_json::to_value(record).map_err(|e| StorageError::Internal(e.to_string()))?; + sqlx::query( + "UPDATE indexes SET warm_throughput = $1 \ + WHERE table_id = $2 AND index_name = $3 AND index_type = 'GSI'", + ) + .bind(json) + .bind(table_id) + .bind(&create.index_name) + .execute(&mut *conn) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + Ok(()) + } +} diff --git a/crates/storage-postgres/tests/key_collation.rs b/crates/storage-postgres/tests/key_collation.rs index 4988d17eb..9847a70cb 100644 --- a/crates/storage-postgres/tests/key_collation.rs +++ b/crates/storage-postgres/tests/key_collation.rs @@ -180,6 +180,7 @@ fn composite_input(table: &str) -> CreateTableInput { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]), ..Default::default() } diff --git a/crates/storage-postgres/tests/twi_conflict.rs b/crates/storage-postgres/tests/twi_conflict.rs index 5d7c6fa14..826cea567 100644 --- a/crates/storage-postgres/tests/twi_conflict.rs +++ b/crates/storage-postgres/tests/twi_conflict.rs @@ -365,6 +365,7 @@ async fn gsi_table(s: &Scratch) -> TableKeyInfo { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }; s.engine .create_table( diff --git a/crates/storage-postgres/tests/vector_control_plane.rs b/crates/storage-postgres/tests/vector_control_plane.rs index 6118f9a0a..5f49bfa2b 100644 --- a/crates/storage-postgres/tests/vector_control_plane.rs +++ b/crates/storage-postgres/tests/vector_control_plane.rs @@ -342,6 +342,8 @@ fn update_input(table: &str) -> UpdateTableInput { on_demand_throughput: None, table_throughput_mode: None, vector_index_updates: None, + warm_throughput: None, + sse_specification: None, } } @@ -2909,6 +2911,7 @@ async fn update_table_refuses_a_vector_index_named_like_an_existing_gsi() { key_schema: hash_key("gsipk"), projection: projection(ProjectionType::All), provisioned_throughput: None, + warm_throughput: None, }]); s.engine .create_table(ACCOUNT, input) @@ -2975,6 +2978,7 @@ async fn update_table_refuses_a_gsi_named_like_an_existing_vector_index() { key_schema: hash_key("pk"), projection: projection(ProjectionType::All), provisioned_throughput: None, + warm_throughput: None, }), update: None, delete: None, diff --git a/crates/storage-sqlite/src/backup.rs b/crates/storage-sqlite/src/backup.rs index 4cea564e2..bc249ef8c 100644 --- a/crates/storage-sqlite/src/backup.rs +++ b/crates/storage-sqlite/src/backup.rs @@ -210,6 +210,7 @@ impl SqliteEngine { key_schema, projection, provisioned_throughput: throughput_from_catalog(ipt.as_ref()), + warm_throughput: None, }); } } @@ -2163,12 +2164,17 @@ mod restore_tests { .expect("old backup item"); } - // The dry run names the rebuild, with the row count. + // The dry run names the rebuild, with the row count, and the two + // WarmThroughput columns the 0.0.3 schema predates. assert_eq!( crate::schema::pending_shape_migrations(&engine.pool) .await .expect("pending"), - vec!["rebuild backup_items (3 rows)".to_owned()] + vec![ + "rebuild backup_items (3 rows)".to_owned(), + "add tables.warm_throughput".to_owned(), + "add indexes.warm_throughput".to_owned(), + ] ); let (log_guard, logs) = crate::schema::log_capture::capture(); crate::schema::apply(&engine.pool).await.expect("migrate"); diff --git a/crates/storage-sqlite/src/bootstrapper.rs b/crates/storage-sqlite/src/bootstrapper.rs index 293a39c8d..218cc876e 100644 --- a/crates/storage-sqlite/src/bootstrapper.rs +++ b/crates/storage-sqlite/src/bootstrapper.rs @@ -675,9 +675,14 @@ mod migrate_report_tests { bootstrap.read_catalog_version().await.expect("version"), Some(bootstrap.expected_catalog_version()) ); + // The 0.0.3 schema also predates the stored WarmThroughput columns. assert_eq!( bootstrap.pending_data_migrations().await.expect("pending"), - vec!["rebuild backup_items (2 rows)".to_owned()] + vec![ + "rebuild backup_items (2 rows)".to_owned(), + "add tables.warm_throughput".to_owned(), + "add indexes.warm_throughput".to_owned(), + ] ); bootstrap.run_data_migrations().await.expect("apply"); assert!( diff --git a/crates/storage-sqlite/src/create_table.rs b/crates/storage-sqlite/src/create_table.rs index ae4ae05e7..2bb49959c 100644 --- a/crates/storage-sqlite/src/create_table.rs +++ b/crates/storage-sqlite/src/create_table.rs @@ -17,7 +17,10 @@ use extenddb_core::types::{ ProvisionedThroughputDescription, SseDescription, SseType, TableDescription, TableStatus, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{index_arn, stream_arn, table_arn}; +use extenddb_storage::util::{ + index_arn, stream_arn, table_arn, warm_throughput_record_for_create, + warm_throughput_record_json, +}; use crate::sqlite_util::{format_epoch_seconds, format_timestamp, is_unique_violation}; use crate::store::SqliteEngine; @@ -120,6 +123,16 @@ impl SqliteEngine { .and_then(|v| v.parse::().ok()) .unwrap_or(0.25); + // Stored as sent and reported back; nothing acts on it (see the + // schema comment on `tables.warm_throughput`). UPDATING for the same + // delay the CREATING to ACTIVE flip takes. + let warm_record = + warm_throughput_record_for_create(input.warm_throughput.as_ref(), delay_secs); + let warm_json = warm_record + .as_ref() + .map(warm_throughput_record_json) + .transpose()?; + let now = time::OffsetDateTime::now_utc(); let creation_ts = format_timestamp(now); #[allow(clippy::cast_precision_loss)] @@ -209,8 +222,8 @@ impl SqliteEngine { (account_id, table_name, key_schema, attribute_definitions, billing_mode, \ provisioned_throughput, stream_specification, table_status, creation_date_time, \ table_arn, table_id, deletion_protection_enabled, status_transition_at, \ - table_class, sse_specification, on_demand_throughput) \ - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + table_class, sse_specification, on_demand_throughput, warm_throughput) \ + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", ) .bind(account_id) .bind(&input.table_name) @@ -228,6 +241,7 @@ impl SqliteEngine { .bind(&input.table_class) .bind(&sse_json) .bind(&on_demand_json) + .bind(&warm_json) .execute(&mut *tx) .await; if let Err(e) = inserted { @@ -278,6 +292,7 @@ impl SqliteEngine { // GSI / LSI metadata. let mut gsi_ids: Vec = Vec::new(); + let mut gsi_warm_records = Vec::new(); if let Some(gsis) = &input.global_secondary_indexes { for gsi in gsis { let ks = serde_json::to_string(&gsi.key_schema) @@ -298,12 +313,19 @@ impl SqliteEngine { }) .transpose() .map_err(|e| StorageError::Internal(e.to_string()))?; + let gsi_warm = + warm_throughput_record_for_create(gsi.warm_throughput.as_ref(), delay_secs); + let gsi_warm_json = gsi_warm + .as_ref() + .map(warm_throughput_record_json) + .transpose()?; + gsi_warm_records.push(gsi_warm); let index_id = uuid::Uuid::new_v4().to_string(); sqlx::query( "INSERT INTO indexes \ (table_id, index_name, index_id, index_type, key_schema, projection, \ - index_status, provisioned_throughput) \ - VALUES (?, ?, ?, 'GSI', ?, ?, 'ACTIVE', ?)", + index_status, provisioned_throughput, warm_throughput) \ + VALUES (?, ?, ?, 'GSI', ?, ?, 'ACTIVE', ?, ?)", ) .bind(&table_id) .bind(&gsi.index_name) @@ -311,6 +333,7 @@ impl SqliteEngine { .bind(&ks) .bind(&proj) .bind(&pt) + .bind(&gsi_warm_json) .execute(&mut *tx) .await .map_err(|e| StorageError::Internal(e.to_string()))?; @@ -515,7 +538,8 @@ impl SqliteEngine { let gsis = input.global_secondary_indexes.as_ref().map(|gs| { gs.iter() - .map(|g| GsiDescription { + .zip(gsi_warm_records) + .map(|(g, warm)| GsiDescription { index_name: g.index_name.clone(), key_schema: g.key_schema.clone(), projection: g.projection.clone(), @@ -541,6 +565,8 @@ impl SqliteEngine { &input.table_name, &g.index_name, ), + warm_throughput: None, + warm_throughput_record: warm, }) .collect() }); @@ -654,6 +680,9 @@ impl SqliteEngine { .as_ref() .map(|tc| serde_json::json!({ "TableClass": tc })), on_demand_throughput: input.on_demand_throughput, + // Derived by the engine from the record; see core. + warm_throughput: None, + warm_throughput_record: warm_record, // Every field is populated deliberately, with no `..Default::default()` // spread. This response is the complete description of what was just // created, so a new core field should break this site and force a diff --git a/crates/storage-sqlite/src/lib.rs b/crates/storage-sqlite/src/lib.rs index 3953fc350..4b5823aea 100644 --- a/crates/storage-sqlite/src/lib.rs +++ b/crates/storage-sqlite/src/lib.rs @@ -45,6 +45,7 @@ mod table_helpers; mod update_table; mod vector_bench; mod vector_search; +mod warm_throughput; mod worker; mod workers; diff --git a/crates/storage-sqlite/src/schema.rs b/crates/storage-sqlite/src/schema.rs index d04ee5b56..36700da11 100644 --- a/crates/storage-sqlite/src/schema.rs +++ b/crates/storage-sqlite/src/schema.rs @@ -65,6 +65,11 @@ CREATE TABLE IF NOT EXISTS tables ( table_class TEXT, sse_specification TEXT, on_demand_throughput TEXT, + -- WarmThroughput as the client set it (JSON, see + -- extenddb_core::types::WarmThroughputRecord). Stored and reported only: + -- nothing pre-warms or scales on it, and it has no effect on throughput, + -- throttling or storage. + warm_throughput TEXT, PRIMARY KEY (account_id, table_name), CONSTRAINT tables_table_id_unique UNIQUE (table_id) ); @@ -84,6 +89,8 @@ CREATE TABLE IF NOT EXISTS indexes ( index_status TEXT NOT NULL DEFAULT 'ACTIVE', provisioned_throughput TEXT, propagation_delay_ms INTEGER, + -- Same stored-only WarmThroughput record as tables.warm_throughput. + warm_throughput TEXT, PRIMARY KEY (table_id, index_name), CONSTRAINT indexes_table_id_fkey FOREIGN KEY (table_id) REFERENCES tables(table_id) ON DELETE CASCADE, @@ -508,6 +515,7 @@ async fn refuse_unwritable_catalog(pool: &SqlitePool) -> OpResult<()> { async fn converge_table_shapes(pool: &SqlitePool) -> OpResult<()> { migrate_backup_items(pool).await?; migrate_table_restores(pool).await?; + migrate_warm_throughput_columns(pool).await?; Ok(()) } @@ -535,9 +543,40 @@ pub async fn pending_shape_migrations(pool: &SqlitePool) -> OpResult if table_restores_lacks_source_table_arn(pool).await? { pending.push("add table_restores.source_table_arn".to_owned()); } + for table in WARM_THROUGHPUT_TABLES { + if lacks_warm_throughput(pool, table).await? { + pending.push(format!("add {table}.warm_throughput")); + } + } Ok(pending) } +/// The catalog tables that carry a stored `WarmThroughput` record. +const WARM_THROUGHPUT_TABLES: [&str; 2] = ["tables", "indexes"]; + +async fn lacks_warm_throughput(pool: &SqlitePool, table: &'static str) -> OpResult { + Ok(table_exists(pool, table).await? && !has_column(pool, table, "warm_throughput").await?) +} + +/// Add `warm_throughput` to `tables` and `indexes` created by an earlier +/// 0.0.4 build. Rows it wrote keep NULL, which DescribeTable reports as the +/// defaults for the billing mode. +async fn migrate_warm_throughput_columns(pool: &SqlitePool) -> OpResult<()> { + for table in WARM_THROUGHPUT_TABLES { + if !lacks_warm_throughput(pool, table).await? { + continue; + } + tracing::info!("catalog upgrade: adding {table}.warm_throughput"); + sqlx::query(&format!( + "ALTER TABLE {table} ADD COLUMN warm_throughput TEXT" + )) + .execute(pool) + .await + .map_err(|e| OpError::Internal(format!("add {table}.warm_throughput: {e}")))?; + } + Ok(()) +} + async fn has_column(pool: &SqlitePool, table: &'static str, column: &str) -> OpResult { sqlx::query_scalar("SELECT EXISTS(SELECT 1 FROM pragma_table_info(?) WHERE name = ?)") .bind(table) diff --git a/crates/storage-sqlite/src/table_helpers.rs b/crates/storage-sqlite/src/table_helpers.rs index bf9288436..f76e20a49 100644 --- a/crates/storage-sqlite/src/table_helpers.rs +++ b/crates/storage-sqlite/src/table_helpers.rs @@ -11,7 +11,7 @@ use extenddb_core::types::{ SseDescription, SseType, TableDescription, TableStatus, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{index_arn, stream_arn}; +use extenddb_storage::util::{index_arn, parse_warm_throughput_record, stream_arn}; use extenddb_storage::vector_catalog::{VectorIndexCatalogRow, vector_index_descriptions}; use crate::data; @@ -38,6 +38,7 @@ pub(crate) struct TableRow { pub table_class: Option, pub sse_specification: Option, pub on_demand_throughput: Option, + pub warm_throughput: Option, } /// Catalog `indexes` row. @@ -51,6 +52,7 @@ pub(crate) struct IndexRow { pub projection: String, pub index_status: String, pub provisioned_throughput: Option, + pub warm_throughput: Option, } /// A `vector_indexes` catalog row, in `FromRow` field order. @@ -105,11 +107,11 @@ pub(crate) const TABLE_COLUMNS: &str = "table_name, key_schema, attribute_defini billing_mode, provisioned_throughput, stream_specification, table_status, \ creation_date_time, table_size_bytes, item_count, table_arn, table_id, \ deletion_protection_enabled, stream_label, table_class, sse_specification, \ - on_demand_throughput"; + on_demand_throughput, warm_throughput"; /// Columns selected for an `IndexRow`, in `FromRow` field order. pub(crate) const INDEX_COLUMNS: &str = "index_name, index_id, index_type, key_schema, \ - projection, index_status, provisioned_throughput"; + projection, index_status, provisioned_throughput, warm_throughput"; fn parse_json(s: &str, ctx: &str) -> Result { serde_json::from_str(s).map_err(|e| StorageError::Internal(format!("{ctx}: {e}"))) @@ -212,6 +214,10 @@ impl SqliteEngine { &row.table_name, &idx.index_name, ), + warm_throughput: None, + warm_throughput_record: parse_warm_throughput_record( + idx.warm_throughput.as_deref(), + )?, }); } else { lsis.push(LsiDescription { @@ -325,6 +331,7 @@ impl SqliteEngine { .on_demand_throughput .as_deref() .and_then(|s| serde_json::from_str(s).ok()), + warm_throughput_record: parse_warm_throughput_record(row.warm_throughput.as_deref())?, // Any core field this backend does not populate takes its default, so // adding one does not break this build. ..Default::default() diff --git a/crates/storage-sqlite/src/update_table.rs b/crates/storage-sqlite/src/update_table.rs index 8adff6173..7ceeec523 100644 --- a/crates/storage-sqlite/src/update_table.rs +++ b/crates/storage-sqlite/src/update_table.rs @@ -18,7 +18,10 @@ use extenddb_core::types::{ TableDescription, UpdateTableInput, VectorIndexSpecification, }; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{building_refuses, effective_attribute_definitions}; +use extenddb_storage::util::{ + building_refuses, effective_attribute_definitions, warm_throughput_record_for_create, + warm_throughput_record_json, +}; use crate::store::SqliteEngine; @@ -357,6 +360,10 @@ impl SqliteEngine { } } + // WarmThroughput on the table and on updated indexes: stored as sent, + // nothing acts on it. Validated by the engine against the current values. + Self::store_warm_throughput_updates(&mut tx, account_id, &table_id, &input).await?; + // GSI create/delete. let mut created: Vec = Vec::new(); let mut deleted: Vec = Vec::new(); @@ -404,12 +411,19 @@ impl SqliteEngine { }) .transpose() .map_err(|e| StorageError::Internal(e.to_string()))?; + let warm = warm_throughput_record_for_create( + create.warm_throughput.as_ref(), + Self::control_plane_delay_secs(&mut tx).await?, + ) + .as_ref() + .map(warm_throughput_record_json) + .transpose()?; let index_id = uuid::Uuid::new_v4().to_string(); sqlx::query( "INSERT INTO indexes \ (table_id, index_name, index_id, index_type, key_schema, projection, \ - index_status, provisioned_throughput) \ - VALUES (?, ?, ?, 'GSI', ?, ?, 'CREATING', ?)", + index_status, provisioned_throughput, warm_throughput) \ + VALUES (?, ?, ?, 'GSI', ?, ?, 'CREATING', ?, ?)", ) .bind(&table_id) .bind(&create.index_name) @@ -417,6 +431,7 @@ impl SqliteEngine { .bind(&ks) .bind(&proj) .bind(&pt) + .bind(&warm) .execute(&mut *tx) .await .map_err(|e| StorageError::Internal(e.to_string()))?; diff --git a/crates/storage-sqlite/src/warm_throughput.rs b/crates/storage-sqlite/src/warm_throughput.rs new file mode 100644 index 000000000..944ce56b4 --- /dev/null +++ b/crates/storage-sqlite/src/warm_throughput.rs @@ -0,0 +1,308 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The `UpdateTable` side of stored `WarmThroughput` for `SqliteEngine`. +//! +//! The record is written as sent and read back by DescribeTable; nothing +//! pre-warms or scales on it, and it has no effect on throughput, throttling +//! or storage. See `extenddb_core::types::WarmThroughputRecord`. + +use extenddb_core::types::UpdateTableInput; +use extenddb_storage::error::StorageError; +use extenddb_storage::util::{warm_throughput_record_for_update, warm_throughput_record_json}; + +use crate::store::SqliteEngine; + +impl SqliteEngine { + /// `control_plane_delay_seconds` from the settings table (0.25 when + /// unset): the window a fresh record reports `UPDATING` for, the same + /// one the CREATING to ACTIVE flip uses. + pub(crate) async fn control_plane_delay_secs( + conn: &mut sqlx::SqliteConnection, + ) -> Result { + Ok(sqlx::query_scalar::<_, String>( + "SELECT value FROM settings WHERE key = 'control_plane_delay_seconds'", + ) + .fetch_optional(&mut *conn) + .await + .map_err(|e| StorageError::Internal(e.to_string()))? + .and_then(|v| v.parse::().ok()) + .unwrap_or(0.25)) + } + + /// Store the request's table-level `WarmThroughput` and each + /// `GlobalSecondaryIndexUpdates.Update.WarmThroughput`, merged over what + /// is stored, inside the caller's catalog transaction. The engine has + /// already validated the values against the current ones. + pub(crate) async fn store_warm_throughput_updates( + tx: &mut sqlx::Transaction<'_, sqlx::Sqlite>, + account_id: &str, + table_id: &str, + input: &UpdateTableInput, + ) -> Result<(), StorageError> { + let gsi_updates: Vec<_> = input + .global_secondary_index_updates + .iter() + .flatten() + .filter_map(|u| u.update.as_ref()) + .filter_map(|u| u.warm_throughput.as_ref().map(|wt| (&u.index_name, wt))) + .collect(); + if input.warm_throughput.is_none() && gsi_updates.is_empty() { + return Ok(()); + } + let delay_secs = Self::control_plane_delay_secs(tx).await?; + + if let Some(spec) = &input.warm_throughput { + let existing: Option = sqlx::query_scalar( + "SELECT warm_throughput FROM tables WHERE account_id = ? AND table_name = ?", + ) + .bind(account_id) + .bind(&input.table_name) + .fetch_one(&mut **tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs)?; + sqlx::query( + "UPDATE tables SET warm_throughput = ? WHERE account_id = ? AND table_name = ?", + ) + .bind(warm_throughput_record_json(&record)?) + .bind(account_id) + .bind(&input.table_name) + .execute(&mut **tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + + for (index_name, spec) in gsi_updates { + let existing: Option> = sqlx::query_scalar( + "SELECT warm_throughput FROM indexes \ + WHERE table_id = ? AND index_name = ? AND index_type = 'GSI'", + ) + .bind(table_id) + .bind(index_name) + .fetch_optional(&mut **tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + let Some(existing) = existing else { + return Err(StorageError::IndexNotFound(index_name.clone())); + }; + let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs)?; + sqlx::query( + "UPDATE indexes SET warm_throughput = ? WHERE table_id = ? AND index_name = ?", + ) + .bind(warm_throughput_record_json(&record)?) + .bind(table_id) + .bind(index_name) + .execute(&mut **tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use crate::SqliteEngine; + use extenddb_core::types::{WarmThroughputStatus, epoch_seconds_now}; + use extenddb_storage::TableEngine; + use serde_json::json; + + async fn engine_with_delay(delay: &str) -> SqliteEngine { + let engine = SqliteEngine::new(":memory:", 1, "us-east-1", 409_600) + .await + .expect("engine"); + crate::schema::apply(&engine.pool).await.expect("schema"); + sqlx::query("UPDATE settings SET value = ? WHERE key = 'control_plane_delay_seconds'") + .bind(delay) + .execute(&engine.pool) + .await + .expect("delay"); + sqlx::query("INSERT INTO accounts (account_id, account_name) VALUES (?, 'default')") + .bind(ACCOUNT) + .execute(&engine.pool) + .await + .expect("account"); + engine + } + + const ACCOUNT: &str = "000000000000"; + + async fn describe(engine: &SqliteEngine) -> extenddb_core::types::TableDescription { + let mut desc = engine + .describe_table( + ACCOUNT, + extenddb_core::types::DescribeTableInput { + table_name: "t".to_owned(), + }, + ) + .await + .expect("describe"); + desc.prepare_for_response(); + desc + } + + fn on_demand_with_gsi( + warm: Option, + ) -> extenddb_core::types::CreateTableInput { + let mut v = json!({ + "TableName": "t", + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "g", "AttributeType": "S"} + ], + "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [{ + "IndexName": "gsi", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "ALL"} + }] + }); + if let Some(w) = warm { + v["WarmThroughput"] = w; + } + serde_json::from_value(v).expect("input") + } + + /// No request member: the on-demand defaults, ACTIVE, on the table and the + /// index, once the table is ACTIVE (delay 0 makes it so at once). + #[tokio::test] + async fn an_untouched_on_demand_table_reports_the_defaults_once_active() { + let engine = engine_with_delay("0").await; + engine + .create_table_impl(ACCOUNT, on_demand_with_gsi(None), false) + .await + .expect("create"); + let desc = describe(&engine).await; + let wt = desc.warm_throughput.expect("table member"); + assert_eq!( + ( + wt.read_units_per_second, + wt.write_units_per_second, + wt.status + ), + (Some(12_000), Some(4_000), WarmThroughputStatus::Active) + ); + let gsi = &desc.global_secondary_indexes.expect("gsis")[0]; + let gwt = gsi.warm_throughput.expect("index member"); + assert_eq!( + (gwt.read_units_per_second, gwt.write_units_per_second), + (Some(12_000), Some(4_000)) + ); + } + + /// The stored record survives the round trip and an UpdateTable merges + /// over it: the deadline makes it UPDATING for the delay, then ACTIVE. + #[tokio::test] + async fn a_stored_record_is_read_back_and_an_update_merges_over_it() { + let engine = engine_with_delay("0").await; + engine + .create_table_impl( + ACCOUNT, + on_demand_with_gsi(Some(json!({"ReadUnitsPerSecond": 20000}))), + false, + ) + .await + .expect("create"); + let desc = describe(&engine).await; + let wt = desc.warm_throughput.expect("table member"); + assert_eq!( + ( + wt.read_units_per_second, + wt.write_units_per_second, + wt.status + ), + (Some(20_000), Some(4_000), WarmThroughputStatus::Active) + ); + + // A long delay so the UPDATING window is observable. + sqlx::query("UPDATE settings SET value = '3600' WHERE key = 'control_plane_delay_seconds'") + .execute(&engine.pool) + .await + .expect("delay"); + let update: extenddb_core::types::UpdateTableInput = serde_json::from_value(json!({ + "TableName": "t", + "WarmThroughput": {"WriteUnitsPerSecond": 8000}, + "GlobalSecondaryIndexUpdates": [{"Update": { + "IndexName": "gsi", + "WarmThroughput": {"ReadUnitsPerSecond": 24000, "WriteUnitsPerSecond": 4000} + }}] + })) + .expect("update"); + // While UPDATING the previous pair is reported (as the service does). + let mut returned = engine.update_table(ACCOUNT, update).await.expect("update"); + returned.prepare_for_response(); + let wt = returned.warm_throughput.expect("table member"); + assert_eq!( + ( + wt.read_units_per_second, + wt.write_units_per_second, + wt.status + ), + (Some(20_000), Some(4_000), WarmThroughputStatus::Updating) + ); + let gsi = &returned.global_secondary_indexes.expect("gsis")[0]; + let gwt = gsi.warm_throughput.expect("index member"); + assert_eq!( + ( + gwt.read_units_per_second, + gwt.write_units_per_second, + gwt.status + ), + (Some(12_000), Some(4_000), WarmThroughputStatus::Updating) + ); + // Past the deadline the same record reads ACTIVE. + let mut later = engine + .describe_table( + ACCOUNT, + extenddb_core::types::DescribeTableInput { + table_name: "t".to_owned(), + }, + ) + .await + .expect("describe"); + later.populate_warm_throughput(epoch_seconds_now() + 3601.0); + let settled = later.warm_throughput.expect("member"); + assert_eq!( + ( + settled.read_units_per_second, + settled.write_units_per_second, + settled.status + ), + (Some(20_000), Some(8_000), WarmThroughputStatus::Active) + ); + let gsi = &later.global_secondary_indexes.expect("gsis")[0]; + let gwt = gsi.warm_throughput.expect("index member"); + assert_eq!( + (gwt.read_units_per_second, gwt.write_units_per_second), + (Some(24_000), Some(4_000)) + ); + } + + /// An Update naming an index the table does not have is IndexNotFound. + #[tokio::test] + async fn an_update_of_a_missing_index_is_index_not_found() { + let engine = engine_with_delay("0").await; + engine + .create_table_impl(ACCOUNT, on_demand_with_gsi(None), false) + .await + .expect("create"); + let update: extenddb_core::types::UpdateTableInput = serde_json::from_value(json!({ + "TableName": "t", + "GlobalSecondaryIndexUpdates": [{"Update": { + "IndexName": "nope", + "WarmThroughput": {"ReadUnitsPerSecond": 24000} + }}] + })) + .expect("update"); + let err = engine + .update_table(ACCOUNT, update) + .await + .expect_err("refused"); + assert!( + matches!(err, extenddb_storage::error::StorageError::IndexNotFound(ref n) if n == "nope"), + "{err:?}" + ); + } +} diff --git a/crates/storage/src/backup_definition.rs b/crates/storage/src/backup_definition.rs index 411b22ab6..b25d150f1 100644 --- a/crates/storage/src/backup_definition.rs +++ b/crates/storage/src/backup_definition.rs @@ -493,6 +493,7 @@ mod tests { read_capacity_units: 3, write_capacity_units: 4, }), + warm_throughput: None, }], local_secondary_indexes: Vec::new(), table_class: Some("STANDARD_INFREQUENT_ACCESS".to_owned()), diff --git a/crates/storage/src/util/mod.rs b/crates/storage/src/util/mod.rs index a762738e7..c9bb7a502 100755 --- a/crates/storage/src/util/mod.rs +++ b/crates/storage/src/util/mod.rs @@ -10,6 +10,7 @@ mod arn; mod backup_id; mod catalog_version; mod key; +mod warm_throughput; pub use arn::{ ARN_PARTITION, format_stream_label, index_arn, new_stream_label, parse_stream_arn, stream_arn, @@ -27,6 +28,10 @@ pub use key::{ merge_attribute_definitions, parse_sk, pk_to_text, recover_sort_key_definitions, sk_column, sk_column_n, sk_info, }; +pub use warm_throughput::{ + parse_warm_throughput_record, warm_throughput_record_for_create, + warm_throughput_record_for_update, warm_throughput_record_json, warm_throughput_updating_until, +}; /// Whether an UpdateTable's GSI actions must be refused because the GSIs named /// in `building` are `CREATING`: any Create, or a Delete of a GSI that is not diff --git a/crates/storage/src/util/warm_throughput.rs b/crates/storage/src/util/warm_throughput.rs new file mode 100644 index 000000000..8050e0985 --- /dev/null +++ b/crates/storage/src/util/warm_throughput.rs @@ -0,0 +1,148 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Persisting `WarmThroughput` records, shared by every backend. +//! +//! A backend stores one JSON value per table and per global secondary index +//! (see `extenddb_core::types::WarmThroughputRecord`) and loads it back into +//! the description's `warm_throughput_record`; the engine derives the wire +//! member from it. Nothing else reads the value: it does not pre-warm, scale, +//! throttle or size anything. + +use extenddb_core::types::{WarmThroughput, WarmThroughputRecord, epoch_seconds_now}; + +use crate::error::StorageError; + +/// The epoch second until which a record written now reports `UPDATING`: +/// `delay_secs` from now, the control-plane delay that also times the +/// CREATING to ACTIVE flip. Zero delay means ACTIVE at once. +#[must_use] +pub fn warm_throughput_updating_until(delay_secs: f64) -> f64 { + epoch_seconds_now() + delay_secs.max(0.0) +} + +/// The record a create path stores for a request's `WarmThroughput`, or +/// `None` when the client sent none (the defaults are then reported). +#[must_use] +pub fn warm_throughput_record_for_create( + spec: Option<&WarmThroughput>, + delay_secs: f64, +) -> Option { + spec.map(|s| WarmThroughputRecord::from_request(s, warm_throughput_updating_until(delay_secs))) +} + +/// The record an update path stores over the stored JSON (`None` when the +/// resource never had one). +/// +/// # Errors +/// +/// `Internal` when the stored JSON does not parse. +pub fn warm_throughput_record_for_update( + existing_json: Option<&str>, + spec: &WarmThroughput, + delay_secs: f64, +) -> Result { + let existing = parse_warm_throughput_record(existing_json)?; + let now = epoch_seconds_now(); + Ok(WarmThroughputRecord::apply_update( + existing, + spec, + now + delay_secs.max(0.0), + now, + )) +} + +/// The stored JSON of a record. +/// +/// # Errors +/// +/// `Internal` on a serialization failure. +pub fn warm_throughput_record_json(record: &WarmThroughputRecord) -> Result { + serde_json::to_string(record) + .map_err(|e| StorageError::Internal(format!("warm throughput record: {e}"))) +} + +/// A record from its stored JSON; `None` in, `None` out. +/// +/// # Errors +/// +/// `Internal` when the text does not parse as a record. +pub fn parse_warm_throughput_record( + json: Option<&str>, +) -> Result, StorageError> { + json.map(|text| { + serde_json::from_str(text) + .map_err(|e| StorageError::Internal(format!("warm throughput record: {e}"))) + }) + .transpose() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_create_without_the_member_stores_nothing() { + assert_eq!(warm_throughput_record_for_create(None, 0.25), None); + } + + #[test] + fn a_create_with_the_member_stores_it_updating_for_the_delay() { + let spec = WarmThroughput { + read_units_per_second: Some(20_000), + write_units_per_second: None, + }; + let record = warm_throughput_record_for_create(Some(&spec), 5.0).expect("stored"); + assert_eq!(record.read_units_per_second, Some(20_000)); + assert_eq!(record.write_units_per_second, None); + assert!(record.is_updating_at(epoch_seconds_now() + 4.0)); + assert!(!record.is_updating_at(epoch_seconds_now() + 6.0)); + } + + #[test] + fn an_update_merges_into_the_stored_json_and_round_trips() { + let stored = warm_throughput_record_json(&WarmThroughputRecord { + read_units_per_second: Some(100), + write_units_per_second: Some(50), + previous_read_units_per_second: None, + previous_write_units_per_second: None, + updating_until: 0.0, + }) + .expect("json"); + let updated = warm_throughput_record_for_update( + Some(&stored), + &WarmThroughput { + read_units_per_second: None, + write_units_per_second: Some(75), + }, + 0.0, + ) + .expect("merged"); + assert_eq!(updated.read_units_per_second, Some(100)); + assert_eq!(updated.write_units_per_second, Some(75)); + let again = parse_warm_throughput_record(Some( + &warm_throughput_record_json(&updated).expect("json"), + )) + .expect("parses") + .expect("present"); + assert_eq!( + (again.read_units_per_second, again.write_units_per_second), + ( + updated.read_units_per_second, + updated.write_units_per_second + ) + ); + // serde_json's float parser is not exact to the last bit without its + // float_roundtrip feature; a microsecond is far below what the status + // check can see. + assert!((again.updating_until - updated.updating_until).abs() < 1e-6); + } + + #[test] + fn garbage_in_the_column_is_an_internal_error_not_a_default() { + let err = parse_warm_throughput_record(Some("{\"x\":")).expect_err("refused"); + assert!( + matches!(err, StorageError::Internal(m) if m.starts_with("warm throughput record:")) + ); + } +} diff --git a/devtools/run-tests b/devtools/run-tests index 421a97b3a..17da66a7a 100755 --- a/devtools/run-tests +++ b/devtools/run-tests @@ -65,6 +65,7 @@ RUN_RUST_INTEGRATION=false RUN_PYTEST=false RUN_EXTERNAL=false RUN_COMPREHENSIVE=false +RUN_TERRAFORM=false RUN_CATALOG_CHECK=false # Options @@ -88,6 +89,8 @@ Suites (at least one required): --pytest Python integration tests --external External Java test suite --comprehensive Comprehensive Python suite + --terraform Terraform apply of tests/terraform/warm-throughput (installs + the pinned Terraform unless TERRAFORM names a binary) --all all suites --catalog-check post-test catalog integrity check @@ -115,7 +118,8 @@ while [[ $# -gt 0 ]]; do --pytest) RUN_PYTEST=true; shift ;; --external) RUN_EXTERNAL=true; shift ;; --comprehensive) RUN_COMPREHENSIVE=true; shift ;; - --all) RUN_RUST=true; RUN_RUST_INTEGRATION=true; RUN_PYTEST=true; RUN_EXTERNAL=true; RUN_COMPREHENSIVE=true; shift ;; + --terraform) RUN_TERRAFORM=true; shift ;; + --all) RUN_RUST=true; RUN_RUST_INTEGRATION=true; RUN_PYTEST=true; RUN_EXTERNAL=true; RUN_COMPREHENSIVE=true; RUN_TERRAFORM=true; shift ;; --release) RELEASE=true; shift ;; --filter) FILTER="$2"; shift 2 ;; --catalog-check) RUN_CATALOG_CHECK=true; shift ;; @@ -155,8 +159,8 @@ if [[ -z "$TARGET" ]]; then fi # --- Validate: at least one suite --- -if ! $RUN_RUST && ! $RUN_RUST_INTEGRATION && ! $RUN_PYTEST && ! $RUN_EXTERNAL && ! $RUN_COMPREHENSIVE && ! $RUN_CATALOG_CHECK; then - echo "error: at least one suite flag is required (--rust, --rust-integration, --pytest, --external, --comprehensive, --all, --catalog-check)" +if ! $RUN_RUST && ! $RUN_RUST_INTEGRATION && ! $RUN_PYTEST && ! $RUN_EXTERNAL && ! $RUN_COMPREHENSIVE && ! $RUN_TERRAFORM && ! $RUN_CATALOG_CHECK; then + echo "error: at least one suite flag is required (--rust, --rust-integration, --pytest, --external, --comprehensive, --terraform, --all, --catalog-check)" exit 1 fi @@ -218,7 +222,7 @@ validate_real_dynamodb() { # Only validate credentials for integration suites NEEDS_INTEGRATION=false -if $RUN_PYTEST || $RUN_EXTERNAL || $RUN_COMPREHENSIVE || $RUN_CATALOG_CHECK || $RUN_RUST_INTEGRATION; then +if $RUN_PYTEST || $RUN_EXTERNAL || $RUN_COMPREHENSIVE || $RUN_TERRAFORM || $RUN_CATALOG_CHECK || $RUN_RUST_INTEGRATION; then NEEDS_INTEGRATION=true fi @@ -552,6 +556,22 @@ if $RUN_COMPREHENSIVE; then echo "" fi +# --- Terraform (the provider's WarmThroughput wait, issue #462) --- +if $RUN_TERRAFORM; then + OUTFILE="discussions/test-terraform-${HASH}.txt" + echo "=== Terraform apply → ${OUTFILE} ===" + if [[ "$TARGET" == "real-dynamodb" ]]; then + echo " skipped: the configuration points at an ExtendDB endpoint" + elif EXTENDDB_TEST_ENDPOINT="$EXTENDDB_TEST_ENDPOINT" \ + tests/terraform/warm-throughput/run.sh 2>&1 | tee "$OUTFILE"; then + echo " ✓ Terraform apply passed" + else + echo " ✗ Terraform apply failed" + PASS=false + fi + echo "" +fi + # --- CLI lifecycle tests (postgres-only; run last — they start/stop their own servers) --- if $RUN_PYTEST \ && [[ "$TARGET" != "real-dynamodb" \ diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 2d6469b7e..a8d2f4ac8 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -145,6 +145,7 @@ The same on all four backends. |------|----------|------| | Provisioned throughput | Token bucket per table/partition | Token bucket per table/partition, matching DynamoDB's burst and refill behavior | | On-demand capacity | Automatic scaling | Fixed initial burst capacity (4000 WCU / 12000 RCU), no auto-scaling | +| Warm throughput | Pre-provisions partitions to the requested read and write units; raising it is a long-running operation (`Status: UPDATING` for minutes, the previous values reported meanwhile); a request above the account's table maximum is `LimitExceededException` | Stored and reported with the same shape, defaults and validation so clients that read it (the Terraform AWS provider since 6.13.0) work: the previous values are reported with `Status: UPDATING` for `control_plane_delay_seconds` after a change, then the new values `ACTIVE`; a decrease is measured against the reported values, as on the service. Nothing acts on the values: no pre-warming and no scaling at the database level. The account maximum is not known to ExtendDB and is not enforced. A provisioned table whose warm throughput was never set, or was set on one side only, reports its new units at once after a capacity change, where the service reports the old units until the table is `ACTIVE`. Switching a table to `PAY_PER_REQUEST` stores 12000 read and 4000 write units where its warm throughput is lower, and the stored values stay after a switch back to `PROVISIONED`, as on the service; the response to the switch reads the new values with `Status: UPDATING` for `control_plane_delay_seconds`, where the service answers with the previous values and `Status: ACTIVE` | | Throttling | Always on; throttles requests that exceed provisioned/burst capacity. No setting to disable | Configurable via `throttling_enabled` runtime setting (default: `true`) | ## Operations Not Implemented diff --git a/docs/getting-started.md b/docs/getting-started.md index 2f22f158a..ddfdcfdff 100755 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -292,6 +292,10 @@ By default, control plane operations (CreateTable, DeleteTable) emulate real Dyn control_plane_delay_seconds 10 ``` +### Warm Throughput + +`WarmThroughput` on `CreateTable`, `UpdateTable` and `GlobalSecondaryIndexUpdates` is stored and reported on `DescribeTable` with the shape, defaults (12000 read and 4000 write units on an on-demand table, the provisioned units on a provisioned one) and validation of Amazon DynamoDB, and `Status` reads `UPDATING` for the control plane delay after a change, then `ACTIVE`. Nothing acts on the values: there is no pre-warming and no scaling at the database level, and they have no effect on throughput, throttling or storage. + ### Credential Import Controls whether `extenddb manage import-access-key` is allowed (default: `true`). diff --git a/docs/manuals/01-architecture-guide.md b/docs/manuals/01-architecture-guide.md index c709f3de5..6402f592e 100755 --- a/docs/manuals/01-architecture-guide.md +++ b/docs/manuals/01-architecture-guide.md @@ -172,7 +172,7 @@ extenddb uses a dual-database architecture: - **Catalog database** (e.g., `extenddb_catalog`): Stores table metadata, account/user/group/role/policy definitions, access keys, settings, stream metadata, and metrics. Shared across all accounts. - **Data database** (e.g., `extenddb_catalog_data`): Stores user items, GSI/LSI data, and stream records. Each table gets its own PostgreSQL table. -The catalog version is 0.0.4 on PostgreSQL, SQLite, and Cassandra. PostgreSQL and SQLite store it in the `settings` table under the key `catalog_version` and check it at startup; Cassandra has no startup check (see the Upgrade Manual). +The catalog version is 0.0.4 on PostgreSQL and SQLite and 0.0.5 on Cassandra (the stored warm throughput columns, Cassandra migration V006). PostgreSQL and SQLite store it in the `settings` table under the key `catalog_version` and check it at startup; Cassandra has no startup check (see the Upgrade Manual). The MongoDB backend tracks its own catalog version, 0.0.2. A mismatch between the compiled-in version and the stored one prevents a PostgreSQL or SQLite server from starting; run `extenddb migrate` to upgrade. diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 23c1cb6f5..5673a6d72 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -202,7 +202,20 @@ To discard data written after the upgrade as well, restore `` from its ## Version History -### Catalog 0.0.4 (Current) +### Catalog 0.0.5 (Cassandra only) + +Adds the stored `WarmThroughput` record: `V006__warm_throughput.cql` adds a nullable `warm_throughput` text column to `tables` and to `indexes` and stamps `catalog_version` 0.0.5. Nothing acts on the value; it is stored and reported by `DescribeTable` (see the differences document). PostgreSQL and SQLite add the same two columns at `serve` start (and in `extenddb migrate`) without a version change, so their catalog stays 0.0.4. + +A Cassandra server from this release started before `extenddb migrate --yes` serves every table and item operation and reports the default warm throughput; a `CreateTable` or `UpdateTable` that names `WarmThroughput` fails with an internal error (`Undefined column name warm_throughput`) until the catalog is migrated, an `UpdateTable` that switches a table to `PAY_PER_REQUEST` succeeds without storing the on-demand warm throughput (the reported value still follows the on-demand minimum while the table is on-demand), and CreateBackup, RestoreTableFromBackup and DeleteBackup are refused as for V005. Run: + +``` +extenddb migrate --config extenddb.toml --cassandra-user # exit 3, lists V006 +extenddb migrate --config extenddb.toml --cassandra-user --yes # applies V006 +``` + +V006 only adds columns and the version row; a 0.0.4 Cassandra binary keeps working against a 0.0.5 catalog. + +### Catalog 0.0.4 (Current on PostgreSQL and SQLite) Adds the table definition a backup records and the registration of a running restore: diff --git a/tests/python/test_warm_throughput.py b/tests/python/test_warm_throughput.py new file mode 100644 index 000000000..93ffa31ed --- /dev/null +++ b/tests/python/test_warm_throughput.py @@ -0,0 +1,446 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""WarmThroughput on tables and global secondary indexes. + +Every assertion here is a row measured against Amazon DynamoDB (us-east-1, +2026-10-23). The file runs against ExtendDB and against the service with the +same assertions; the two differ only in timing, which the `is_extenddb` +fixture gates: ExtendDB reports `Status: UPDATING` for the control-plane delay +(a fraction of a second by default), the service for minutes, so the "returns +to ACTIVE" wait runs only against ExtendDB. + +ExtendDB stores and reports the values. Nothing acts on them. +""" + +from __future__ import annotations + +import pytest +from botocore.exceptions import ClientError + +from warm_throughput_helpers import ( # noqa: F401 (tables is a fixture) + DECREASE, + DELETION_PROTECTION_ONLY, + GSI, + LSI, + ON_DEMAND_DEFAULTS, + ON_DEMAND_MIN, + ON_DEMAND_NEEDS_BILLING, + ONLY_OPERATION, + TABLE_CLASS_ONLY, + _create, + _sentence, + _validation, + _wait_all_active, + _wait_warm_active, + tables, +) + + +class TestWarmThroughputLifecycle: + def test_on_demand_table_absent_while_creating_defaults_once_active( + self, dynamodb_client, tables + ): + """Rows: CreateTable response and first DescribeTable carry no member while + CREATING; ACTIVE reads {12000, 4000, ACTIVE} on the table and every GSI; + a local secondary index carries none.""" + name, created = _create( + dynamodb_client, + tables, + BillingMode="PAY_PER_REQUEST", + GlobalSecondaryIndexes=[GSI], + LocalSecondaryIndexes=[LSI], + ) + if created["TableStatus"] == "CREATING": + assert "WarmThroughput" not in created + assert all("WarmThroughput" not in g for g in created["GlobalSecondaryIndexes"]) + first = dynamodb_client.describe_table(TableName=name)["Table"] + if first["TableStatus"] == "CREATING": + assert "WarmThroughput" not in first + active = _wait_all_active(dynamodb_client, name) + assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + gsi = active["GlobalSecondaryIndexes"][0] + assert gsi["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + assert "WarmThroughput" not in active["LocalSecondaryIndexes"][0] + + def test_provisioned_table_and_index_default_to_their_units(self, dynamodb_client, tables): + """Row: a 1/1 provisioned table with a 1/1 GSI reads {1, 1, ACTIVE} on both.""" + name, created = _create( + dynamodb_client, + tables, + BillingMode="PROVISIONED", + ProvisionedThroughput={"ReadCapacityUnits": 1, "WriteCapacityUnits": 1}, + GlobalSecondaryIndexes=[ + {**GSI, "ProvisionedThroughput": {"ReadCapacityUnits": 1, "WriteCapacityUnits": 1}} + ], + ) + if created["TableStatus"] == "CREATING": + assert "WarmThroughput" not in created + active = _wait_all_active(dynamodb_client, name) + expected = {"ReadUnitsPerSecond": 1, "WriteUnitsPerSecond": 1, "Status": "ACTIVE"} + assert active["WarmThroughput"] == expected + assert active["GlobalSecondaryIndexes"][0]["WarmThroughput"] == expected + + def test_update_table_without_warm_throughput_reports_it_active( + self, dynamodb_client, tables + ): + """Row: an UpdateTable that only touches deletion protection carries the + member with Status ACTIVE in its response.""" + name, _ = _create(dynamodb_client, tables, BillingMode="PAY_PER_REQUEST") + _wait_all_active(dynamodb_client, name) + resp = dynamodb_client.update_table(TableName=name, DeletionProtectionEnabled=False) + assert resp["TableDescription"]["WarmThroughput"] == { + **ON_DEMAND_DEFAULTS, + "Status": "ACTIVE", + } + + def test_create_with_warm_throughput_is_present_at_once_and_partial_is_echoed( + self, dynamodb_client, tables + ): + """Rows C1 and C4: a request with WarmThroughput is answered with the member + at once, Status UPDATING, as requested (a read-only request echoes only the + read side in the CreateTable response), and reads back filled once ACTIVE.""" + name, created = _create( + dynamodb_client, + tables, + BillingMode="PAY_PER_REQUEST", + WarmThroughput={"ReadUnitsPerSecond": 20000}, + ) + if created["TableStatus"] == "CREATING": + assert created["WarmThroughput"] == {"ReadUnitsPerSecond": 20000, "Status": "UPDATING"} + else: + assert created["WarmThroughput"]["ReadUnitsPerSecond"] == 20000 + _wait_all_active(dynamodb_client, name) + active = _wait_warm_active(dynamodb_client, name, timeout=900) + assert active["WarmThroughput"] == { + "ReadUnitsPerSecond": 20000, + "WriteUnitsPerSecond": 4000, + "Status": "ACTIVE", + } + + both, created = _create( + dynamodb_client, + tables, + BillingMode="PAY_PER_REQUEST", + WarmThroughput={"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 4000}, + GlobalSecondaryIndexes=[ + {**GSI, "WarmThroughput": {"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 4000}} + ], + ) + if created["TableStatus"] == "CREATING": + assert created["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + assert created["GlobalSecondaryIndexes"][0]["WarmThroughput"] == { + **ON_DEMAND_DEFAULTS, + "Status": "UPDATING", + } + active = _wait_all_active(dynamodb_client, both) + active = _wait_warm_active(dynamodb_client, both, timeout=900) + assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + assert active["GlobalSecondaryIndexes"][0]["WarmThroughput"] == { + **ON_DEMAND_DEFAULTS, + "Status": "ACTIVE", + } + + def test_update_warm_throughput_goes_updating_then_active( + self, dynamodb_client, tables, is_extenddb + ): + """Rows U1, U2, U3, U5, U6, S2a, S2b and S2c: UpdateTable with WarmThroughput + is accepted and the response reports the PREVIOUS values with Status + UPDATING while TableStatus and IndexStatus stay ACTIVE; inside the window a + request carrying those previous values is accepted (U3); once settled the new + values read ACTIVE and the same request is a decrease (S2b). Only the waits + are gated: ExtendDB returns to ACTIVE within the control plane delay, the + service after minutes, so the settled assertions run on ExtendDB only.""" + name, _ = _create( + dynamodb_client, + tables, + BillingMode="PAY_PER_REQUEST", + GlobalSecondaryIndexes=[GSI], + ) + _wait_all_active(dynamodb_client, name) + + # U1: the same values as current, and nothing else in the request. + resp = dynamodb_client.update_table(TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS) + t = resp["TableDescription"] + assert t["TableStatus"] == "ACTIVE" + assert t["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + if is_extenddb: + _wait_warm_active(dynamodb_client, name) + + # U2 and S2a: higher values; the response carries the previous pair. + higher = {"ReadUnitsPerSecond": 24000, "WriteUnitsPerSecond": 8000} + t = dynamodb_client.update_table(TableName=name, WarmThroughput=higher)["TableDescription"] + assert t["TableStatus"] == "ACTIVE" + assert t["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + described = dynamodb_client.describe_table(TableName=name)["Table"]["WarmThroughput"] + if described["Status"] == "UPDATING": + assert described == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + # U3: inside the window the previous values are not a decrease. + t = dynamodb_client.update_table(TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS)[ + "TableDescription" + ] + assert t["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + if is_extenddb: + _wait_warm_active(dynamodb_client, name) + dynamodb_client.update_table(TableName=name, WarmThroughput=higher) + if is_extenddb: + active = _wait_warm_active(dynamodb_client, name) + assert active["WarmThroughput"] == {**higher, "Status": "ACTIVE"} + + # S2b: on the settled table the previous values are a decrease. + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table(TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS) + _validation(exc, _sentence("ReadUnitsPerSecond", "table", DECREASE)) + # S2c: the same values re-sent report themselves UPDATING. + t = dynamodb_client.update_table(TableName=name, WarmThroughput=higher)[ + "TableDescription" + ] + assert t["WarmThroughput"] == {**higher, "Status": "UPDATING"} + _wait_warm_active(dynamodb_client, name) + + # U5: read alone; the write side keeps its value. + dynamodb_client.update_table(TableName=name, WarmThroughput={"ReadUnitsPerSecond": 30000}) + active = _wait_warm_active(dynamodb_client, name) + assert active["WarmThroughput"] == { + "ReadUnitsPerSecond": 30000, + "WriteUnitsPerSecond": 8000, + "Status": "ACTIVE", + } + + # U6: a GSI through GlobalSecondaryIndexUpdates.Update (same values, so + # the previous and new pairs agree on both targets). + resp = dynamodb_client.update_table( + TableName=name, + GlobalSecondaryIndexUpdates=[ + {"Update": {"IndexName": "gsi", "WarmThroughput": ON_DEMAND_DEFAULTS}} + ], + ) + gsi = resp["TableDescription"]["GlobalSecondaryIndexes"][0] + assert gsi["IndexStatus"] == "ACTIVE" + assert gsi["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + assert resp["TableDescription"]["TableStatus"] == "ACTIVE" + if is_extenddb: + active = _wait_warm_active(dynamodb_client, name) + assert active["GlobalSecondaryIndexes"][0]["WarmThroughput"] == { + **ON_DEMAND_DEFAULTS, + "Status": "ACTIVE", + } + + def test_capacity_change_leaves_a_set_warm_throughput_alone(self, dynamodb_client, tables): + """Rows C6 and U8: a provisioned 5/5 table created with WarmThroughput 100/50 + keeps 100/50 ACTIVE after its capacity moves to 10/10.""" + name, created = _create( + dynamodb_client, + tables, + BillingMode="PROVISIONED", + ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, + WarmThroughput={"ReadUnitsPerSecond": 100, "WriteUnitsPerSecond": 50}, + ) + if created["TableStatus"] == "CREATING": + assert created["WarmThroughput"] == { + "ReadUnitsPerSecond": 100, + "WriteUnitsPerSecond": 50, + "Status": "UPDATING", + } + _wait_all_active(dynamodb_client, name) + _wait_warm_active(dynamodb_client, name, timeout=900) + resp = dynamodb_client.update_table( + TableName=name, + ProvisionedThroughput={"ReadCapacityUnits": 10, "WriteCapacityUnits": 10}, + ) + expected = {"ReadUnitsPerSecond": 100, "WriteUnitsPerSecond": 50, "Status": "ACTIVE"} + assert resp["TableDescription"]["WarmThroughput"] == expected + _wait_all_active(dynamodb_client, name) + assert dynamodb_client.describe_table(TableName=name)["Table"]["WarmThroughput"] == expected + + +class TestWarmThroughputValidation: + def test_on_demand_create_below_the_minimum(self, dynamodb_client, tables): + """Rows C2, C3 and C7: the on-demand minimum, read checked before write.""" + for warm, member in ( + ({"ReadUnitsPerSecond": 11999, "WriteUnitsPerSecond": 4000}, "ReadUnitsPerSecond"), + ({"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 3999}, "WriteUnitsPerSecond"), + ({"ReadUnitsPerSecond": 0, "WriteUnitsPerSecond": 0}, "ReadUnitsPerSecond"), + ): + with pytest.raises(ClientError) as exc: + _create(dynamodb_client, tables, BillingMode="PAY_PER_REQUEST", WarmThroughput=warm) + _validation(exc, _sentence(member, "table", ON_DEMAND_MIN)) + + def test_on_demand_gsi_create_below_the_minimum_names_the_index(self, dynamodb_client, tables): + """Rows G1 and G2: the index form of the on-demand sentence.""" + for warm, member in ( + ({"ReadUnitsPerSecond": 11999, "WriteUnitsPerSecond": 4000}, "ReadUnitsPerSecond"), + ({"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 3999}, "WriteUnitsPerSecond"), + ): + with pytest.raises(ClientError) as exc: + _create( + dynamodb_client, + tables, + BillingMode="PAY_PER_REQUEST", + GlobalSecondaryIndexes=[{**GSI, "WarmThroughput": warm}], + ) + _validation(exc, _sentence(member, "index gsi", ON_DEMAND_MIN)) + + def test_provisioned_create_below_the_units(self, dynamodb_client, tables): + """Row C5 (read form measured; the write form is the same sentence with the + member names substituted).""" + with pytest.raises(ClientError) as exc: + _create( + dynamodb_client, + tables, + BillingMode="PROVISIONED", + ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, + WarmThroughput={"ReadUnitsPerSecond": 1, "WriteUnitsPerSecond": 1}, + ) + _validation( + exc, + _sentence( + "ReadUnitsPerSecond", "table", + " is lower than ReadCapacityUnits of ProvisionedThroughput", + ), + ) + # P1: the write form. + with pytest.raises(ClientError) as exc: + _create( + dynamodb_client, + tables, + BillingMode="PROVISIONED", + ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, + WarmThroughput={"ReadUnitsPerSecond": 5, "WriteUnitsPerSecond": 4}, + ) + _validation( + exc, + _sentence( + "WriteUnitsPerSecond", "table", + " is lower than WriteCapacityUnits of ProvisionedThroughput", + ), + ) + # G3: the index form, on a 5/5 GSI. + with pytest.raises(ClientError) as exc: + _create( + dynamodb_client, + tables, + BillingMode="PROVISIONED", + ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, + GlobalSecondaryIndexes=[{ + **GSI, + "ProvisionedThroughput": {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, + "WarmThroughput": {"ReadUnitsPerSecond": 1, "WriteUnitsPerSecond": 1}, + }], + ) + _validation( + exc, + _sentence( + "ReadUnitsPerSecond", "index gsi", + " is lower than ReadCapacityUnits of ProvisionedThroughput", + ), + ) + # P2: equal to the units is accepted and reads back {5, 5, ACTIVE}. + name, _ = _create( + dynamodb_client, + tables, + BillingMode="PROVISIONED", + ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, + WarmThroughput={"ReadUnitsPerSecond": 5, "WriteUnitsPerSecond": 5}, + ) + _wait_all_active(dynamodb_client, name) + active = _wait_warm_active(dynamodb_client, name, timeout=900) + assert active["WarmThroughput"] == { + "ReadUnitsPerSecond": 5, + "WriteUnitsPerSecond": 5, + "Status": "ACTIVE", + } + + def test_update_below_current_is_a_decrease(self, dynamodb_client, tables): + """Rows U4 and U7: below the current value on either billing mode.""" + od, _ = _create(dynamodb_client, tables, BillingMode="PAY_PER_REQUEST") + _wait_all_active(dynamodb_client, od) + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=od, WarmThroughput={"ReadUnitsPerSecond": 11999, "WriteUnitsPerSecond": 4000} + ) + _validation(exc, _sentence("ReadUnitsPerSecond", "table", DECREASE)) + # UT1: the write form. + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=od, WarmThroughput={"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 3999} + ) + _validation(exc, _sentence("WriteUnitsPerSecond", "table", DECREASE)) + + pv, _ = _create( + dynamodb_client, + tables, + BillingMode="PROVISIONED", + ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, + WarmThroughput={"ReadUnitsPerSecond": 100, "WriteUnitsPerSecond": 50}, + ) + _wait_all_active(dynamodb_client, pv) + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=pv, WarmThroughput={"ReadUnitsPerSecond": 2, "WriteUnitsPerSecond": 2} + ) + _validation(exc, _sentence("ReadUnitsPerSecond", "table", DECREASE)) + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table(TableName=pv, WarmThroughput={"WriteUnitsPerSecond": 49}) + _validation(exc, _sentence("WriteUnitsPerSecond", "table", DECREASE)) + + def test_gsi_update_below_current_names_the_index(self, dynamodb_client, tables): + """Rows UG1 and UG2: the index form of the decrease sentence, both members.""" + name, _ = _create( + dynamodb_client, tables, BillingMode="PAY_PER_REQUEST", GlobalSecondaryIndexes=[GSI] + ) + _wait_all_active(dynamodb_client, name) + for warm, member in ( + ({"ReadUnitsPerSecond": 11999, "WriteUnitsPerSecond": 4000}, "ReadUnitsPerSecond"), + ({"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 3999}, "WriteUnitsPerSecond"), + ): + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=name, + GlobalSecondaryIndexUpdates=[{"Update": {"IndexName": "gsi", "WarmThroughput": warm}}], + ) + _validation(exc, _sentence(member, "index gsi", DECREASE)) + + def test_table_warm_throughput_beside_another_change(self, dynamodb_client, tables): + """The per-companion rules, measured on an on-demand table with one GSI + (values otherwise valid): TableClass and DeletionProtection companions + answer their own "must be the only operation" sentence; a + StreamSpecification companion, and BillingMode with ProvisionedThroughput, + answer the WarmThroughput sentence; an OnDemandThroughput companion without + BillingMode answers its own sentence; a GlobalSecondaryIndexUpdates Update + carrying the index's WarmThroughput is accepted and both go UPDATING.""" + name, _ = _create( + dynamodb_client, tables, BillingMode="PAY_PER_REQUEST", GlobalSecondaryIndexes=[GSI] + ) + _wait_all_active(dynamodb_client, name) + cases = [ + ({"TableClass": "STANDARD"}, TABLE_CLASS_ONLY), + ({"DeletionProtectionEnabled": False}, DELETION_PROTECTION_ONLY), + ({"OnDemandThroughput": {"MaxReadRequestUnits": 100000}}, ON_DEMAND_NEEDS_BILLING), + ({"StreamSpecification": {"StreamEnabled": False}}, ONLY_OPERATION), + ( + { + "BillingMode": "PROVISIONED", + "ProvisionedThroughput": {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, + }, + ONLY_OPERATION, + ), + ] + for extra, expected in cases: + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS, **extra + ) + _validation(exc, expected) + + resp = dynamodb_client.update_table( + TableName=name, + WarmThroughput=ON_DEMAND_DEFAULTS, + GlobalSecondaryIndexUpdates=[ + {"Update": {"IndexName": "gsi", "WarmThroughput": ON_DEMAND_DEFAULTS}} + ], + ) + t = resp["TableDescription"] + assert t["TableStatus"] == "ACTIVE" + assert t["WarmThroughput"]["Status"] == "UPDATING" + assert t["GlobalSecondaryIndexes"][0]["WarmThroughput"]["Status"] == "UPDATING" diff --git a/tests/python/test_warm_throughput_backup.py b/tests/python/test_warm_throughput_backup.py new file mode 100644 index 000000000..40ebbe9cd --- /dev/null +++ b/tests/python/test_warm_throughput_backup.py @@ -0,0 +1,121 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""WarmThroughput and backups (rows S6a to S6d, measured against Amazon DynamoDB +on 2026-10-23): a backup records no warm throughput, a restored table and its +indexes report the on-demand defaults whatever the source had set, and a +GlobalSecondaryIndexOverride entry carrying WarmThroughput is refused. The file +runs against ExtendDB and the service with the same assertions.""" + +from __future__ import annotations + +import time + +import pytest +from botocore.exceptions import ClientError + +from helpers import unique_name +from warm_throughput_helpers import ( # noqa: F401 (tables is a fixture) + GSI, + ON_DEMAND_DEFAULTS, + _create, + _validation, + _wait_all_active, + _wait_warm_active, + tables, +) + +GSI_OVERRIDE_WARM_THROUGHPUT = ( + "One or more parameter values were invalid: WarmThroughput is not supported on Global " + "Secondary Indexes via RestoreTableFromBackup" +) + + +def _wait_backup_available(client, arn: str, timeout: float = 900.0) -> dict: + deadline = time.monotonic() + timeout + while True: + desc = client.describe_backup(BackupArn=arn)["BackupDescription"] + if desc["BackupDetails"]["BackupStatus"] == "AVAILABLE": + return desc + if time.monotonic() > deadline: + raise TimeoutError(f"backup {arn} did not become AVAILABLE") + time.sleep(1) + + +def test_a_restored_table_reports_the_defaults_and_a_gsi_override_cannot_set_it( + dynamodb_client, tables +): + """Rows S6a to S6d.""" + name, _ = _create( + dynamodb_client, + tables, + BillingMode="PAY_PER_REQUEST", + WarmThroughput={"ReadUnitsPerSecond": 30000, "WriteUnitsPerSecond": 8000}, + GlobalSecondaryIndexes=[ + {**GSI, "WarmThroughput": {"ReadUnitsPerSecond": 24000, "WriteUnitsPerSecond": 8000}} + ], + ) + _wait_all_active(dynamodb_client, name) + _wait_warm_active(dynamodb_client, name, timeout=900) + # A just-created table refuses CreateBackup with + # ContinuousBackupsUnavailableException for a short while on the service + # (backups are still being enabled); ExtendDB accepts at once. Retried the + # way test_backup_restore_fidelity.py does. + deadline = time.monotonic() + 60 + while True: + try: + arn = dynamodb_client.create_backup( + TableName=name, BackupName=f"wt-{int(time.time())}" + )["BackupDetails"]["BackupArn"] + break + except ClientError as e: + code = e.response["Error"]["Code"] + if code != "ContinuousBackupsUnavailableException" or time.monotonic() >= deadline: + raise + time.sleep(2) + try: + # S6b: the backup records no warm throughput. + desc = _wait_backup_available(dynamodb_client, arn) + assert "WarmThroughput" not in desc["SourceTableDetails"] + for gsi in desc.get("SourceTableFeatureDetails", {}).get("GlobalSecondaryIndexes", []): + assert "WarmThroughput" not in gsi + + # S6c: no member while CREATING; the defaults once ACTIVE, on the table + # and the index, although the source had set larger values. + target = unique_name("wt-restore") + tables.append(target) + restored = dynamodb_client.restore_table_from_backup( + TargetTableName=target, BackupArn=arn + )["TableDescription"] + if restored["TableStatus"] == "CREATING": + assert "WarmThroughput" not in restored + assert all("WarmThroughput" not in g for g in restored["GlobalSecondaryIndexes"]) + _wait_all_active(dynamodb_client, target) + active = _wait_warm_active(dynamodb_client, target, timeout=900) + assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + assert active["GlobalSecondaryIndexes"][0]["WarmThroughput"] == { + **ON_DEMAND_DEFAULTS, + "Status": "ACTIVE", + } + + # S6d: an override entry may not carry WarmThroughput. + with pytest.raises(ClientError) as exc: + dynamodb_client.restore_table_from_backup( + TargetTableName=unique_name("wt-restore"), + BackupArn=arn, + GlobalSecondaryIndexOverride=[ + { + **GSI, + "WarmThroughput": { + "ReadUnitsPerSecond": 24000, + "WriteUnitsPerSecond": 8000, + }, + } + ], + ) + _validation(exc, GSI_OVERRIDE_WARM_THROUGHPUT) + finally: + try: + dynamodb_client.delete_backup(BackupArn=arn) + except ClientError: + pass diff --git a/tests/python/test_warm_throughput_updates.py b/tests/python/test_warm_throughput_updates.py new file mode 100644 index 000000000..3592a0bf3 --- /dev/null +++ b/tests/python/test_warm_throughput_updates.py @@ -0,0 +1,420 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""UpdateTable shapes around WarmThroughput measured in the review rounds of +2026-10-23: the GlobalSecondaryIndexUpdates element rules, the companions a +table-level WarmThroughput may and may not have, a GSI Create carrying +WarmThroughput, the empty request on CreateTable, and the defaults after +capacity and billing changes. Row labels refer to the service-r1, r2 and r3 +runs. The file runs against ExtendDB and the service; only waits and the +pre-existing IndexStatus divergence are gated on `is_extenddb`. +""" + +from __future__ import annotations + +import pytest +from botocore.exceptions import ClientError + +from helpers import unique_name +from warm_throughput_helpers import ( # noqa: F401 (tables is a fixture) + ATTRS, + DECREASE, + DELETION_PROTECTION_ONLY, + GSI, + ON_DEMAND_DEFAULTS, + ON_DEMAND_MIN, + ON_DEMAND_NEEDS_BILLING, + ONLY_OPERATION, + TABLE_CLASS_ONLY, + _create, + _sentence, + _validation, + _wait_all_active, + _wait_warm_active, + tables, +) + +ONE_ACTION = ( + "One or more parameter values were invalid: Only one global secondary index action is " + "allowed per GlobalSecondaryIndexUpdate object" +) +ONE_UPDATE_PER_INDEX = ( + "One or more parameter values were invalid: Only one global secondary index update per " + "index is allowed simultaneously. Index: gsi" +) +ON_DEMAND_UPDATE_MEMBERS = ( + "One or more parameter values were invalid: The only Updates for index: gsi when " + "TableThroughputMode is PAY_PER_REQUEST can be to OnDemandThroughput, WarmThroughput" +) +SSE_ONLY = ( + "One or more parameter values were invalid: Server-Side Encryption modification must be " + "the only operation in the request" +) +DELETE_WITH_WARM = ( + "One or more parameter values were invalid: Delete global secondary index cannot be " + "specified when updating WarmThroughput" +) +GSI_TWO = { + "IndexName": "gsi-two", + "KeySchema": [{"AttributeName": "G", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "KEYS_ONLY"}, +} + + +def _on_demand_with_gsi(client, tables) -> str: + name, _ = _create(client, tables, BillingMode="PAY_PER_REQUEST", GlobalSecondaryIndexes=[GSI]) + _wait_all_active(client, name) + return name + + +class TestGsiUpdateElements: + def test_an_element_with_two_actions_is_refused(self, dynamodb_client, tables): + """Rows V1a and V1b: Update plus Create, and Create plus Delete (the Create + keys on a defined attribute, so only the element rule can refuse it).""" + name = _on_demand_with_gsi(dynamodb_client, tables) + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=name, + AttributeDefinitions=ATTRS, + GlobalSecondaryIndexUpdates=[{ + "Update": {"IndexName": "gsi", "WarmThroughput": ON_DEMAND_DEFAULTS}, + "Create": GSI_TWO, + }], + ) + _validation(exc, ONE_ACTION) + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=name, + AttributeDefinitions=ATTRS, + GlobalSecondaryIndexUpdates=[{"Create": GSI_TWO, "Delete": {"IndexName": "gsi"}}], + ) + _validation(exc, ONE_ACTION) + # T3b: Update plus Delete of the same index in one element answers the + # per-index sentence. + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=name, + GlobalSecondaryIndexUpdates=[{ + "Update": {"IndexName": "gsi", "WarmThroughput": ON_DEMAND_DEFAULTS}, + "Delete": {"IndexName": "gsi"}, + }], + ) + _validation(exc, ONE_UPDATE_PER_INDEX) + t = dynamodb_client.describe_table(TableName=name)["Table"] + assert [g["IndexName"] for g in t["GlobalSecondaryIndexes"]] == ["gsi"] + + def test_two_elements_on_one_index_are_refused(self, dynamodb_client, tables): + """Row V3c: Delete gsi and Update gsi WarmThroughput in two elements.""" + name = _on_demand_with_gsi(dynamodb_client, tables) + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=name, + GlobalSecondaryIndexUpdates=[ + {"Delete": {"IndexName": "gsi"}}, + {"Update": {"IndexName": "gsi", "WarmThroughput": ON_DEMAND_DEFAULTS}}, + ], + ) + _validation(exc, ONE_UPDATE_PER_INDEX) + + def test_an_update_without_warm_throughput_names_the_billing_mode( + self, dynamodb_client, tables + ): + """Rows T3d and T3e: an Update carrying a ProvisionedThroughput, or nothing, + on an on-demand table.""" + name = _on_demand_with_gsi(dynamodb_client, tables) + for update in ( + {"IndexName": "gsi", "ProvisionedThroughput": {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}}, + {"IndexName": "gsi"}, + ): + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table( + TableName=name, GlobalSecondaryIndexUpdates=[{"Update": update}] + ) + _validation(exc, ON_DEMAND_UPDATE_MEMBERS) + + +INDEX_CHANGE_WITH_STREAM = ( + "One or more parameter values were invalid: You cannot create or delete index while " + "changing stream status" +) +STREAM_ON = {"StreamEnabled": True, "StreamViewType": "NEW_IMAGE"} +PROVISIONED_5 = {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5} + + +def _refused(client, expected: str, **kwargs) -> None: + with pytest.raises(ClientError) as exc: + client.update_table(**kwargs) + _validation(exc, expected) + + +class TestWarmThroughputCompanions: + """The companions a table-level WarmThroughput may not have (rows S1a to S1d, + S9a, T4a to T4d, V2, V3a, V3b, V4). Every request here is refused, so one + table serves all of them.""" + + def test_sse_and_the_ordering_of_the_member_sentences(self, dynamodb_client, tables): + name = _on_demand_with_gsi(dynamodb_client, tables) + sse = {"Enabled": True} + cases = [ + ({"SSESpecification": sse}, SSE_ONLY), + # SSE outranks TableClass (S1a), DeletionProtection (S1b), Streams (S1c). + ({"SSESpecification": sse, "TableClass": "STANDARD_INFREQUENT_ACCESS"}, SSE_ONLY), + ({"SSESpecification": sse, "DeletionProtectionEnabled": False}, SSE_ONLY), + ({"SSESpecification": sse, "StreamSpecification": STREAM_ON}, SSE_ONLY), + # DeletionProtection outranks Streams (S1d). + ( + {"DeletionProtectionEnabled": False, "StreamSpecification": STREAM_ON}, + DELETION_PROTECTION_ONLY, + ), + # V4: the triple answers the TableClass sentence. + ( + { + "TableClass": "STANDARD_INFREQUENT_ACCESS", + "DeletionProtectionEnabled": False, + "StreamSpecification": STREAM_ON, + }, + TABLE_CLASS_ONLY, + ), + ] + for extra, expected in cases: + _refused(dynamodb_client, expected, TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS, **extra) + + def test_on_demand_throughput_with_and_without_billing_mode(self, dynamodb_client, tables): + name = _on_demand_with_gsi(dynamodb_client, tables) + odt = {"MaxReadRequestUnits": 100000} + cases = [ + # T4a and T4d: the same sentence with and without BillingMode. + ({"BillingMode": "PAY_PER_REQUEST", "OnDemandThroughput": odt}, ON_DEMAND_NEEDS_BILLING), + ({"OnDemandThroughput": odt}, ON_DEMAND_NEEDS_BILLING), + # T4b and T4c: TableClass and DeletionProtection outrank it. + ({"TableClass": "STANDARD_INFREQUENT_ACCESS", "OnDemandThroughput": odt}, TABLE_CLASS_ONLY), + ({"DeletionProtectionEnabled": False, "OnDemandThroughput": odt}, DELETION_PROTECTION_ONLY), + ] + for extra, expected in cases: + _refused(dynamodb_client, expected, TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS, **extra) + + def test_provisioned_throughput_alone_is_not_an_allowed_companion(self, dynamodb_client, tables): + """Row V2.""" + name = _on_demand_with_gsi(dynamodb_client, tables) + _refused( + dynamodb_client, + ONLY_OPERATION, + TableName=name, + WarmThroughput=ON_DEMAND_DEFAULTS, + ProvisionedThroughput=PROVISIONED_5, + ) + + def test_index_delete_is_refused_and_the_index_stays(self, dynamodb_client, tables): + """Rows S9a, V3a and V3b: Delete beside a table WarmThroughput; TableClass + outranks it; with Streams the service names the stream change.""" + name = _on_demand_with_gsi(dynamodb_client, tables) + delete = [{"Delete": {"IndexName": "gsi"}}] + _refused( + dynamodb_client, DELETE_WITH_WARM, TableName=name, + WarmThroughput=ON_DEMAND_DEFAULTS, GlobalSecondaryIndexUpdates=delete, + ) + _refused( + dynamodb_client, TABLE_CLASS_ONLY, TableName=name, + WarmThroughput=ON_DEMAND_DEFAULTS, GlobalSecondaryIndexUpdates=delete, + TableClass="STANDARD_INFREQUENT_ACCESS", + ) + _refused( + dynamodb_client, INDEX_CHANGE_WITH_STREAM, TableName=name, + WarmThroughput=ON_DEMAND_DEFAULTS, GlobalSecondaryIndexUpdates=delete, + StreamSpecification=STREAM_ON, + ) + t = _wait_all_active(dynamodb_client, name) + assert [g["IndexName"] for g in t["GlobalSecondaryIndexes"]] == ["gsi"] + assert t["GlobalSecondaryIndexes"][0]["WarmThroughput"] == { + **ON_DEMAND_DEFAULTS, "Status": "ACTIVE" + } + + def test_missing_table(self, dynamodb_client): + """Rows S10, T5a, T5c: a bad value alone, and an index Update alone, are + ResourceNotFoundException. Rows S10b, T5b, T5d: a companion is answered + first. Only the code is compared for the not-found message.""" + missing = unique_name("wt-missing") + for kwargs in ( + {"WarmThroughput": {"ReadUnitsPerSecond": 11999, "WriteUnitsPerSecond": 4000}}, + {"WarmThroughput": {"ReadUnitsPerSecond": 1}}, + { + "GlobalSecondaryIndexUpdates": [ + {"Update": {"IndexName": "gsi", "WarmThroughput": ON_DEMAND_DEFAULTS}} + ] + }, + ): + with pytest.raises(ClientError) as exc: + dynamodb_client.update_table(TableName=missing, **kwargs) + assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException" + for extra, expected in ( + ({"TableClass": "STANDARD_INFREQUENT_ACCESS"}, TABLE_CLASS_ONLY), + ({"StreamSpecification": STREAM_ON}, ONLY_OPERATION), + ({"DeletionProtectionEnabled": False}, DELETION_PROTECTION_ONLY), + ): + _refused( + dynamodb_client, expected, TableName=missing, + WarmThroughput=ON_DEMAND_DEFAULTS, **extra, + ) + + +class TestWarmThroughputCreateShapes: + def test_create_with_an_empty_warm_throughput_reports_no_member_while_creating( + self, dynamodb_client, tables + ): + """Row T1: `WarmThroughput: {}` on CreateTable is accepted, the response + while CREATING carries no member, and the defaults read once ACTIVE.""" + name, created = _create( + dynamodb_client, tables, BillingMode="PAY_PER_REQUEST", WarmThroughput={} + ) + if created["TableStatus"] == "CREATING": + assert "WarmThroughput" not in created + active = _wait_all_active(dynamodb_client, name) + active = _wait_warm_active(dynamodb_client, name, timeout=900) + assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + + def test_gsi_create_with_warm_throughput_in_update_table( + self, dynamodb_client, tables, is_extenddb + ): + """Rows S8a and V5: a GSI created through GlobalSecondaryIndexUpdates with + WarmThroughput reports the values as sent with Status UPDATING, then the + same values ACTIVE. The service also reports the new index CREATING and the + table UPDATING; ExtendDB reports both ACTIVE at once (the index build is + immediate), so those two statuses are asserted on the service only.""" + name = _on_demand_with_gsi(dynamodb_client, tables) + sent = {"ReadUnitsPerSecond": 30000, "WriteUnitsPerSecond": 8000} + resp = dynamodb_client.update_table( + TableName=name, + AttributeDefinitions=ATTRS, + GlobalSecondaryIndexUpdates=[{"Create": {**GSI_TWO, "WarmThroughput": sent}}], + ) + t = resp["TableDescription"] + by_name = {g["IndexName"]: g for g in t["GlobalSecondaryIndexes"]} + assert by_name["gsi-two"]["WarmThroughput"] == {**sent, "Status": "UPDATING"} + assert by_name["gsi"]["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + if not is_extenddb: + assert t["TableStatus"] == "UPDATING" + assert by_name["gsi-two"]["IndexStatus"] == "CREATING" + described = dynamodb_client.describe_table(TableName=name)["Table"] + new = {g["IndexName"]: g for g in described["GlobalSecondaryIndexes"]}["gsi-two"] + if new["WarmThroughput"]["Status"] == "UPDATING": + assert new["WarmThroughput"] == {**sent, "Status": "UPDATING"} + _wait_all_active(dynamodb_client, name) + settled = _wait_warm_active(dynamodb_client, name, timeout=900) + by_name = {g["IndexName"]: g for g in settled["GlobalSecondaryIndexes"]} + assert by_name["gsi-two"]["WarmThroughput"] == {**sent, "Status": "ACTIVE"} + assert by_name["gsi"]["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + + def test_gsi_create_below_the_minimum_names_the_index(self, dynamodb_client, tables): + """Row S8b.""" + name = _on_demand_with_gsi(dynamodb_client, tables) + _refused( + dynamodb_client, + _sentence("ReadUnitsPerSecond", "index gsi3", ON_DEMAND_MIN), + TableName=name, + AttributeDefinitions=ATTRS, + GlobalSecondaryIndexUpdates=[{"Create": { + **GSI_TWO, + "IndexName": "gsi3", + "WarmThroughput": {"ReadUnitsPerSecond": 11999, "WriteUnitsPerSecond": 4000}, + }}], + ) + + +class TestWarmThroughputAcrossBillingChanges: + """The values a provisioned table reports around capacity and billing mode + changes (rows S3, S7a to S7c, T7a to T7d). Statuses the service reports while + the table is UPDATING for a capacity or billing change, and the previous + values it answers with, are asserted on the service only: ExtendDB changes + capacity and billing mode with the table ACTIVE, and answers a switch to + PAY_PER_REQUEST with the on-demand values.""" + + def test_switch_to_on_demand_raises_a_low_warm_throughput( + self, dynamodb_client, tables, is_extenddb + ): + """Rows T7a to T7d and S7c: a read-only record {100} on a provisioned 5/5 + table reads {100, 5}, follows a capacity change to {100, 10}, and reads + {12000, 4000} once the table is on-demand; the on-demand values are then + not a decrease.""" + name, _ = _create( + dynamodb_client, + tables, + BillingMode="PROVISIONED", + ProvisionedThroughput=PROVISIONED_5, + WarmThroughput={"ReadUnitsPerSecond": 100}, + ) + _wait_all_active(dynamodb_client, name) + active = _wait_warm_active(dynamodb_client, name, timeout=900) + assert active["WarmThroughput"] == { + "ReadUnitsPerSecond": 100, "WriteUnitsPerSecond": 5, "Status": "ACTIVE" + } + dynamodb_client.update_table( + TableName=name, + ProvisionedThroughput={"ReadCapacityUnits": 10, "WriteCapacityUnits": 10}, + ) + _wait_all_active(dynamodb_client, name) + assert dynamodb_client.describe_table(TableName=name)["Table"]["WarmThroughput"] == { + "ReadUnitsPerSecond": 100, "WriteUnitsPerSecond": 10, "Status": "ACTIVE" + } + resp = dynamodb_client.update_table(TableName=name, BillingMode="PAY_PER_REQUEST") + if not is_extenddb: + assert resp["TableDescription"]["WarmThroughput"] == { + "ReadUnitsPerSecond": 100, "WriteUnitsPerSecond": 10, "Status": "ACTIVE" + } + _wait_all_active(dynamodb_client, name) + active = _wait_warm_active(dynamodb_client, name, timeout=900) + assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + resp = dynamodb_client.update_table(TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS) + assert resp["TableDescription"]["WarmThroughput"] == { + **ON_DEMAND_DEFAULTS, "Status": "UPDATING" + } + + def test_never_set_warm_throughput_across_capacity_and_billing_changes( + self, dynamodb_client, tables, is_extenddb + ): + """Rows S3, S7a and S7b: a provisioned 5/5 table that never set warm + throughput reads {5, 5}, then {10, 10} after a capacity change, then + {12000, 4000} on-demand, and keeps {12000, 4000} after the switch back to + provisioned.""" + name, _ = _create( + dynamodb_client, tables, BillingMode="PROVISIONED", ProvisionedThroughput=PROVISIONED_5 + ) + _wait_all_active(dynamodb_client, name) + active = _wait_warm_active(dynamodb_client, name, timeout=900) + assert active["WarmThroughput"] == { + "ReadUnitsPerSecond": 5, "WriteUnitsPerSecond": 5, "Status": "ACTIVE" + } + + # S3: the service answers the capacity change with the old units. + resp = dynamodb_client.update_table( + TableName=name, + ProvisionedThroughput={"ReadCapacityUnits": 10, "WriteCapacityUnits": 10}, + ) + if not is_extenddb: + assert resp["TableDescription"]["WarmThroughput"] == { + "ReadUnitsPerSecond": 5, "WriteUnitsPerSecond": 5, "Status": "ACTIVE" + } + _wait_all_active(dynamodb_client, name) + assert dynamodb_client.describe_table(TableName=name)["Table"]["WarmThroughput"] == { + "ReadUnitsPerSecond": 10, "WriteUnitsPerSecond": 10, "Status": "ACTIVE" + } + + # S7a: on-demand. + resp = dynamodb_client.update_table(TableName=name, BillingMode="PAY_PER_REQUEST") + if not is_extenddb: + assert resp["TableDescription"]["WarmThroughput"] == { + "ReadUnitsPerSecond": 10, "WriteUnitsPerSecond": 10, "Status": "ACTIVE" + } + _wait_all_active(dynamodb_client, name) + active = _wait_warm_active(dynamodb_client, name, timeout=900) + assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + + # S7b: back to provisioned; the on-demand values stay. + resp = dynamodb_client.update_table( + TableName=name, BillingMode="PROVISIONED", ProvisionedThroughput=PROVISIONED_5 + ) + assert resp["TableDescription"]["WarmThroughput"] == { + **ON_DEMAND_DEFAULTS, "Status": "ACTIVE" + } + _wait_all_active(dynamodb_client, name) + active = _wait_warm_active(dynamodb_client, name, timeout=900) + assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} diff --git a/tests/python/warm_throughput_helpers.py b/tests/python/warm_throughput_helpers.py new file mode 100644 index 000000000..bd9204b4a --- /dev/null +++ b/tests/python/warm_throughput_helpers.py @@ -0,0 +1,124 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""Shared constants, waits and the `tables` fixture for the WarmThroughput +wire tests (`test_warm_throughput.py`, `test_warm_throughput_updates.py`). +Every sentence here was measured against Amazon DynamoDB on 2026-10-23.""" + +from __future__ import annotations + +import time + +import pytest + +from helpers import unique_name, wait_for_active, wait_for_deleted + +ON_DEMAND_DEFAULTS = {"ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 4000} +PREFIX = "One or more parameter values were invalid: Requested " +ON_DEMAND_MIN = ( + " is lower than initial throughput for OnDemand. See: " + "https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/" + "on-demand-capacity-mode.html#on-demand-capacity-mode-initial" +) +DECREASE = " is lower than current WarmThroughput, decreasing WarmThroughput is not supported" +ONLY_OPERATION = ( + "One or more parameter values were invalid: WarmThroughput must be the only operation " + "in the request" +) +TABLE_CLASS_ONLY = ( + "One or more parameter values were invalid: TableClass modification must be the only " + "operation in the request" +) +DELETION_PROTECTION_ONLY = ( + "One or more parameter values were invalid: DeletionProtection modification must be the " + "only operation in the request" +) +ON_DEMAND_NEEDS_BILLING = ( + "One or more parameter values were invalid: OnDemandThroughput can only be combined with " + "BillingMode in an UpdateTable request" +) + + +def _sentence(member: str, subject: str, suffix: str) -> str: + """`Requested for WarmThroughput for `.""" + return f"{PREFIX}{member} for WarmThroughput for {subject}{suffix}" + +PK = [{"AttributeName": "PK", "KeyType": "HASH"}, {"AttributeName": "SK", "KeyType": "RANGE"}] +ATTRS = [ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + {"AttributeName": "G", "AttributeType": "S"}, +] +GSI = { + "IndexName": "gsi", + "KeySchema": [{"AttributeName": "G", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "ALL"}, +} +LSI = { + "IndexName": "lsi", + "KeySchema": [{"AttributeName": "PK", "KeyType": "HASH"}, {"AttributeName": "G", "KeyType": "RANGE"}], + "Projection": {"ProjectionType": "KEYS_ONLY"}, +} + + +def _wait_all_active(client, name: str, timeout: float = 900.0) -> dict: + """The description once the table and every GSI are ACTIVE.""" + wait_for_active(client, name, timeout) + deadline = time.monotonic() + timeout + while True: + t = client.describe_table(TableName=name)["Table"] + if all(g["IndexStatus"] == "ACTIVE" for g in t.get("GlobalSecondaryIndexes", [])): + return t + if time.monotonic() > deadline: + raise TimeoutError(f"indexes of {name} did not become ACTIVE") + time.sleep(1) + + +def _wait_warm_active(client, name: str, timeout: float = 30.0) -> dict: + """The description once the table's and every GSI's WarmThroughput is ACTIVE.""" + deadline = time.monotonic() + timeout + while True: + t = client.describe_table(TableName=name)["Table"] + statuses = [t["WarmThroughput"]["Status"]] + [ + g["WarmThroughput"]["Status"] for g in t.get("GlobalSecondaryIndexes", []) + ] + if all(s == "ACTIVE" for s in statuses): + return t + if time.monotonic() > deadline: + raise TimeoutError(f"WarmThroughput of {name} stayed {statuses}") + time.sleep(0.2) + + +def _validation(exc_info, expected: str) -> None: + err = exc_info.value.response["Error"] + assert err["Code"] == "ValidationException", err + assert err["Message"] == expected, err["Message"] + + +@pytest.fixture() +def tables(dynamodb_client): + """Register created table names for deletion after the test.""" + names: list[str] = [] + yield names + for name in names: + try: + dynamodb_client.update_table(TableName=name, DeletionProtectionEnabled=False) + except Exception: + pass + try: + dynamodb_client.delete_table(TableName=name) + wait_for_deleted(dynamodb_client, name, 600) + except Exception: + pass + + +def _create(client, tables, **kwargs) -> tuple[str, dict]: + name = unique_name("wt") + tables.append(name) + # The G attribute is defined only when an index keys on it: an unused + # definition is refused by the service and by ExtendDB. + indexed = "GlobalSecondaryIndexes" in kwargs or "LocalSecondaryIndexes" in kwargs + attrs = ATTRS if indexed else ATTRS[:2] + args = {"TableName": name, "KeySchema": PK, "AttributeDefinitions": attrs} + args.update(kwargs) + return name, client.create_table(**args)["TableDescription"] diff --git a/tests/terraform/install-terraform.sh b/tests/terraform/install-terraform.sh new file mode 100755 index 000000000..ef87d6818 --- /dev/null +++ b/tests/terraform/install-terraform.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# +# Install a pinned Terraform release into a directory, checking the zip +# against the checksum pinned below and against the SHA256SUMS file HashiCorp +# publishes for the release. Prints the path of the binary. +# +# tests/terraform/install-terraform.sh [DIR] (default: $TMPDIR/terraform-) +# +# Same version and checksum for local runs and for CI (the integration +# workflows call this script), so a run cannot silently pick up a different +# Terraform. + +set -euo pipefail + +TERRAFORM_VERSION="1.17.0" +# sha256 of terraform_1.17.0_linux_amd64.zip, from +# https://releases.hashicorp.com/terraform/1.17.0/terraform_1.17.0_SHA256SUMS +TERRAFORM_SHA256="c168cab7c637a8769fdb90089ddd19de701bc6ff8aca64dc2e2185fe46f59f67" + +dir="${1:-${TMPDIR:-/tmp}/terraform-${TERRAFORM_VERSION}}" +mkdir -p "$dir" +if [[ -x "$dir/terraform" ]] && "$dir/terraform" version | grep -q "Terraform v${TERRAFORM_VERSION}\$"; then + echo "$dir/terraform" + exit 0 +fi + +zip="terraform_${TERRAFORM_VERSION}_linux_amd64.zip" +base="https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}" +curl -sSf -o "$dir/$zip" "$base/$zip" +curl -sSf -o "$dir/SHA256SUMS" "$base/terraform_${TERRAFORM_VERSION}_SHA256SUMS" + +published=$(awk -v f="$zip" '$2 == f { print $1 }' "$dir/SHA256SUMS") +actual=$(sha256sum "$dir/$zip" | awk '{ print $1 }') +if [[ "$published" != "$TERRAFORM_SHA256" ]]; then + echo "error: published checksum for $zip is $published, pinned $TERRAFORM_SHA256" >&2 + exit 1 +fi +if [[ "$actual" != "$TERRAFORM_SHA256" ]]; then + echo "error: $zip has checksum $actual, pinned $TERRAFORM_SHA256" >&2 + exit 1 +fi + +unzip -o -q "$dir/$zip" -d "$dir" +rm -f "$dir/$zip" "$dir/SHA256SUMS" +echo "$dir/terraform" diff --git a/tests/terraform/warm-throughput/main.tf b/tests/terraform/warm-throughput/main.tf new file mode 100644 index 000000000..f52a8498c --- /dev/null +++ b/tests/terraform/warm-throughput/main.tf @@ -0,0 +1,95 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# +# The configuration from hashicorp/terraform-provider-aws#44344 (the table +# whose apply hung against an emulator that reported no WarmThroughput), plus +# a second table with a global secondary index, against the ExtendDB under +# test. Since provider 6.13.0 the aws_dynamodb_table resource waits for +# WarmThroughput.Status to read ACTIVE after every table and index create or +# update; run.sh asserts the apply completes and a second plan has no changes. + +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "6.70.0" + } + } +} + +variable "endpoint" { + description = "The ExtendDB endpoint, for example https://127.0.0.1:18443" + type = string +} + +variable "access_key" { + type = string + sensitive = true +} + +variable "secret_key" { + type = string + sensitive = true +} + +variable "insecure" { + description = "Accept the server's self-signed TLS certificate" + type = bool + default = true +} + +provider "aws" { + region = "us-east-1" + access_key = var.access_key + secret_key = var.secret_key + insecure = var.insecure + + skip_credentials_validation = true + skip_metadata_api_check = true + skip_requesting_account_id = true + + endpoints { + dynamodb = var.endpoint + } +} + +resource "aws_dynamodb_table" "example" { + name = "example" + billing_mode = "PAY_PER_REQUEST" + + hash_key = "PK" + range_key = "SK" + + attribute { + name = "PK" + type = "S" + } + + attribute { + name = "SK" + type = "S" + } +} + +resource "aws_dynamodb_table" "with_index" { + name = "example-with-index" + billing_mode = "PAY_PER_REQUEST" + + hash_key = "PK" + + attribute { + name = "PK" + type = "S" + } + + attribute { + name = "G" + type = "S" + } + + global_secondary_index { + name = "gsi" + hash_key = "G" + projection_type = "ALL" + } +} diff --git a/tests/terraform/warm-throughput/run.sh b/tests/terraform/warm-throughput/run.sh new file mode 100755 index 000000000..e41483c48 --- /dev/null +++ b/tests/terraform/warm-throughput/run.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# +# Apply tests/terraform/warm-throughput/main.tf against an ExtendDB and assert +# the apply completes within five minutes and a second plan has no changes. +# The tables are destroyed afterwards. +# +# Environment: +# EXTENDDB_TEST_ENDPOINT the server, for example https://127.0.0.1:18443 +# AWS_ACCESS_KEY_ID a key the server accepts +# AWS_SECRET_ACCESS_KEY +# TERRAFORM the terraform binary (default: install-terraform.sh) +# +# tests/terraform/warm-throughput/run.sh +# +# Exit 0 when the apply and the plan both pass; non-zero otherwise, with the +# failing step named. + +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +endpoint="${EXTENDDB_TEST_ENDPOINT:?set EXTENDDB_TEST_ENDPOINT}" +: "${AWS_ACCESS_KEY_ID:?set AWS_ACCESS_KEY_ID}" +: "${AWS_SECRET_ACCESS_KEY:?set AWS_SECRET_ACCESS_KEY}" +terraform="${TERRAFORM:-$("$here/../install-terraform.sh")}" +[[ -x "$terraform" ]] || { echo "FAIL: terraform binary $terraform not found or not executable"; exit 1; } + +# A scratch working copy, so the repository never holds provider binaries, +# lock files or state. +work="$(mktemp -d "${TMPDIR:-/tmp}/extenddb-tf-warm-throughput.XXXXXX")" +# The tables use fixed names, so a run that leaves them behind makes the next +# apply fail with ResourceInUseException: remove them on every exit once the +# apply has started (a failed assertion included). +applied=0 +cleanup() { + if [[ $applied -eq 1 ]]; then + timeout 300 "$terraform" destroy -auto-approve -no-color >/dev/null 2>&1 || true + fi + rm -rf "$work" +} +trap cleanup EXIT +cp "$here/main.tf" "$work/" +cd "$work" + +export TF_IN_AUTOMATION=1 +export TF_INPUT=0 +export TF_VAR_endpoint="$endpoint" +export TF_VAR_access_key="$AWS_ACCESS_KEY_ID" +export TF_VAR_secret_key="$AWS_SECRET_ACCESS_KEY" +# Keep the provider's own credential chain out of it. +unset AWS_PROFILE AWS_SESSION_TOKEN + +echo "=== terraform init ($("$terraform" version | head -1))" +"$terraform" init -no-color >init.log 2>&1 || { cat init.log; echo "FAIL: terraform init"; exit 1; } + +echo "=== terraform apply (5 minute limit)" +applied=1 +set +e +timeout 300 "$terraform" apply -auto-approve -no-color >apply.log 2>&1 +apply_status=$? +set -e +if [[ $apply_status -ne 0 ]]; then + cat apply.log + if [[ $apply_status -eq 124 ]]; then + echo "FAIL: terraform apply did not finish within 300 s (the provider waits for WarmThroughput.Status ACTIVE)" + else + echo "FAIL: terraform apply exited $apply_status" + fi + exit 1 +fi +grep -E "Apply complete! Resources: 2 added" apply.log || { cat apply.log; echo "FAIL: expected 2 resources added"; exit 1; } + +echo "=== terraform plan (must report no changes)" +set +e +timeout 300 "$terraform" plan -detailed-exitcode -no-color >plan.log 2>&1 +plan_status=$? +set -e +if [[ $plan_status -ne 0 ]]; then + cat plan.log + echo "FAIL: terraform plan exited $plan_status (2 means the applied tables drifted from the configuration)" + exit 1 +fi +grep -E "No changes" plan.log || { cat plan.log; echo "FAIL: terraform plan did not report No changes"; exit 1; } + +echo "=== terraform destroy" +timeout 300 "$terraform" destroy -auto-approve -no-color >destroy.log 2>&1 || { cat destroy.log; echo "FAIL: terraform destroy"; exit 1; } +applied=0 +echo "PASS: apply completed, second plan had no changes, tables destroyed" From 641e663b7a0056c9b50267c95a91f1bbedc8852f Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:49:34 +0000 Subject: [PATCH 02/15] feat(storage): add the warm_throughput columns as catalog migration 6 main replaced the three upgrade mechanisms this branch used (#666, #681): serve refuses a catalog with pending migrations, so the PostgreSQL shape steps applied at serve time, SQLite's shape convergence and Cassandra's V006 with catalog version 0.0.5 cannot run any more. The columns are now migration 6, `warm_throughput`, in crates/storage/src/migrations/registry.rs, Breaking and convergent on every backend, with RELEASED_THROUGH left at 0: - PostgreSQL: migrations/0006_warm_throughput.sql, ADD COLUMN IF NOT EXISTS on tables and indexes, run in a transaction. - SQLite: the code body m0006_warm_throughput.rs, which reads pragma_table_info before each ALTER TABLE, with a golden test on the 0.1.14 catalog (tests/fixtures/golden/0006_warm_throughput.txt). - Cassandra: migrations/catalog/0006_warm_throughput.cql with two ALTER TABLE ... ADD IF NOT EXISTS and no catalog_version insert. V006 is gone. - MongoDB: NotApplicable, since the catalog collections have no validator. The merge left migration 1 and SQLite's schema.rs as main has them, so no warm_throughput hunk reached 0001_baseline.sql. The tests that pin the latest migration or a catalog's definitions after every migration now expect 6, and _as_released_catalog in tests/test_resource_tag_migration.py drops the two columns so migration 1's check of the v1.0 schema accepts the catalog. The upgrade manual drops this branch's Catalog 0.0.5 section, which collided with main's continuous-backups drop, and describes migration 6 under Current Status, including that going back to a release without it means restoring the backup taken before the upgrade. The verify examples in the install guides read migration 6, and docs/schema/{sqlite,postgres,cassandra}.txt are regenerated from a fresh init. Literals added on main since the branch point gain the new members: GsiInput in crates/storage/src/snapshot_conformance.rs and in two Cassandra direct tests, and UpdateTableInput and CreateGsiAction in another. The SQLite warm throughput tests build their catalog with migrations::apply_for_tests and insert control_plane_delay_seconds, which no migration seeds. --- .../catalog/0006_warm_throughput.cql | 13 ++++ .../catalog/V006__warm_throughput.cql | 18 ------ crates/storage-cassandra/src/create_table.rs | 7 +-- .../storage-cassandra/src/migrations/mod.rs | 9 +++ .../storage-cassandra/tests/direct/index.rs | 3 + .../transact_write_stream_batch_limit.rs | 1 + crates/storage-cassandra/tests/migrations.rs | 10 ++- .../tests/migrations_stopped.rs | 10 ++- .../tests/migrations_support/mod.rs | 30 ++++++++- crates/storage-mongodb/src/migrations/mod.rs | 4 ++ .../storage-mongodb/src/migrations/tests.rs | 3 +- .../storage-mongodb/tests/migrations_fresh.rs | 32 +++++----- .../tests/migrations_guards.rs | 4 +- .../tests/migrations_holders.rs | 4 +- .../storage-mongodb/tests/migrations_locks.rs | 4 +- .../tests/migrations_released.rs | 8 +-- .../migrations/0006_warm_throughput.sql | 12 ++++ crates/storage-postgres/src/create_table.rs | 4 +- crates/storage-postgres/src/migrations/mod.rs | 12 +++- crates/storage-sqlite/src/create_table.rs | 6 +- .../src/migrations/golden_tests.rs | 56 ++++++++++++++++- .../src/migrations/m0006_warm_throughput.rs | 62 +++++++++++++++++++ crates/storage-sqlite/src/migrations/mod.rs | 12 ++++ crates/storage-sqlite/src/migrations/tests.rs | 2 +- crates/storage-sqlite/src/warm_throughput.rs | 20 +++--- .../fixtures/golden/0006_warm_throughput.txt | 9 +++ crates/storage/src/migrations/registry.rs | 5 ++ crates/storage/src/snapshot_conformance.rs | 1 + devtools/check-postgres-catalog-lock | 4 +- docs/getting-started.md | 2 +- docs/manuals/04-quickstart-setup-guide.md | 2 +- docs/manuals/07-upgrade-manual.md | 6 +- docs/manuals/08-install-linux.md | 2 +- docs/manuals/09-install-macos.md | 2 +- tests/test_resource_tag_migration.py | 11 +++- 35 files changed, 300 insertions(+), 90 deletions(-) create mode 100644 crates/storage-cassandra/migrations/catalog/0006_warm_throughput.cql delete mode 100644 crates/storage-cassandra/migrations/catalog/V006__warm_throughput.cql create mode 100644 crates/storage-postgres/migrations/0006_warm_throughput.sql create mode 100644 crates/storage-sqlite/src/migrations/m0006_warm_throughput.rs create mode 100644 crates/storage-sqlite/tests/fixtures/golden/0006_warm_throughput.txt diff --git a/crates/storage-cassandra/migrations/catalog/0006_warm_throughput.cql b/crates/storage-cassandra/migrations/catalog/0006_warm_throughput.cql new file mode 100644 index 000000000..f92ce411a --- /dev/null +++ b/crates/storage-cassandra/migrations/catalog/0006_warm_throughput.cql @@ -0,0 +1,13 @@ +-- Copyright 2026 ExtendDB contributors +-- SPDX-License-Identifier: Apache-2.0 +-- +-- Migration 6, warm_throughput, Cassandra catalog keyspace: a nullable +-- warm_throughput text column on tables and on indexes, holding the +-- WarmThroughput a client set (one JSON record, see +-- extenddb_core::types::WarmThroughputRecord). Stored and reported only: +-- nothing pre-warms or scales on it. A row without a value reports the +-- defaults for its billing mode. ADD IF NOT EXISTS makes a second run change +-- nothing. The runner writes the catalog keyspace in front of each table. + +ALTER TABLE tables ADD IF NOT EXISTS warm_throughput text; +ALTER TABLE indexes ADD IF NOT EXISTS warm_throughput text; diff --git a/crates/storage-cassandra/migrations/catalog/V006__warm_throughput.cql b/crates/storage-cassandra/migrations/catalog/V006__warm_throughput.cql deleted file mode 100644 index 9e1f19a30..000000000 --- a/crates/storage-cassandra/migrations/catalog/V006__warm_throughput.cql +++ /dev/null @@ -1,18 +0,0 @@ --- Copyright 2026 ExtendDB contributors --- SPDX-License-Identifier: Apache-2.0 --- WarmThroughput on tables and global secondary indexes (catalog version 0.0.5). --- --- tables.warm_throughput and indexes.warm_throughput hold the WarmThroughput a --- client set, as one JSON text (see extenddb_core::types::WarmThroughputRecord). --- The value is stored and reported by DescribeTable so clients that read the --- member work. Nothing acts on it: there is no pre-warming and no scaling at --- the database level, and it has no effect on throughput, throttling or --- storage. A row written before this migration has no value and reports the --- defaults for its billing mode. --- --- NOTE: the migration runner splits files on the semicolon character, --- including inside comments. Do not use one in a comment. -ALTER TABLE tables ADD IF NOT EXISTS warm_throughput text; -ALTER TABLE indexes ADD IF NOT EXISTS warm_throughput text; - -INSERT INTO settings (key, value) VALUES ('catalog_version', '0.0.5'); diff --git a/crates/storage-cassandra/src/create_table.rs b/crates/storage-cassandra/src/create_table.rs index 28c22ae14..cab054d75 100644 --- a/crates/storage-cassandra/src/create_table.rs +++ b/crates/storage-cassandra/src/create_table.rs @@ -266,10 +266,9 @@ impl CassandraEngine { Value::from(status_transition_at), ]; // WarmThroughput as sent, reported back and acted on by nothing (see - // catalog migration V006). UPDATING for the same delay the CREATING to - // ACTIVE flip takes. Named only when sent, for the same reason as the - // restore columns: a catalog that has not run V006 still serves an - // ordinary CreateTable. + // catalog migration 6). UPDATING for the same delay the CREATING to + // ACTIVE flip takes. Named only when sent, so a table created without + // it writes no null cell. let warm_record = warm_throughput_record_for_create(input.warm_throughput.as_ref(), delay_seconds); if let Some(record) = warm_record.as_ref() { diff --git a/crates/storage-cassandra/src/migrations/mod.rs b/crates/storage-cassandra/src/migrations/mod.rs index aae33b64f..1198fe705 100644 --- a/crates/storage-cassandra/src/migrations/mod.rs +++ b/crates/storage-cassandra/src/migrations/mod.rs @@ -127,6 +127,15 @@ pub fn migration(id: MigrationId) -> Migration { "0005_drop_continuous_backups.cql" )], }, + MigrationId::WarmThroughput => Migration::Applies { + kind: Kind::Breaking, + convergent: true, + bodies: vec![file_body!( + StoreKind::Catalog, + "migrations/catalog", + "0006_warm_throughput.cql" + )], + }, } } diff --git a/crates/storage-cassandra/tests/direct/index.rs b/crates/storage-cassandra/tests/direct/index.rs index 0d5423a05..286e28645 100644 --- a/crates/storage-cassandra/tests/direct/index.rs +++ b/crates/storage-cassandra/tests/direct/index.rs @@ -1030,6 +1030,8 @@ async fn a_backfill_the_recovery_worker_started_ends_before_the_worker_returns() .update_table( &table.key_info.account_id, UpdateTableInput { + sse_specification: None, + warm_throughput: None, table_name: table.key_info.table_name.clone(), billing_mode: None, table_throughput_mode: None, @@ -1047,6 +1049,7 @@ async fn a_backfill_the_recovery_worker_started_ends_before_the_worker_returns() non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }), update: None, delete: None, diff --git a/crates/storage-cassandra/tests/direct/transact_write_stream_batch_limit.rs b/crates/storage-cassandra/tests/direct/transact_write_stream_batch_limit.rs index 025599b84..76676a85c 100644 --- a/crates/storage-cassandra/tests/direct/transact_write_stream_batch_limit.rs +++ b/crates/storage-cassandra/tests/direct/transact_write_stream_batch_limit.rs @@ -135,6 +135,7 @@ async fn two_gsi_table(engine: &CassandraEngine) -> (TableKeyInfo, DropKeyspaces non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }; let attribute_definitions = vec![ string_attribute("id"), diff --git a/crates/storage-cassandra/tests/migrations.rs b/crates/storage-cassandra/tests/migrations.rs index 3a2349632..a54149a92 100644 --- a/crates/storage-cassandra/tests/migrations.rs +++ b/crates/storage-cassandra/tests/migrations.rs @@ -176,12 +176,10 @@ async fn migration_1_creates_the_expected_definitions() { std::fs::write(&path, text).expect("write the expected file"); continue; } - let mut expected: postcheck::Definitions = - serde_json::from_str(&std::fs::read_to_string(&path).expect("read")).expect("json"); - // Migration 5 drops continuous_backups, which migration 1 creates. - expected - .columns - .retain(|name, _| !name.starts_with("continuous_backups.")); + let expected = migrations_support::after_every_migration( + dir, + serde_json::from_str(&std::fs::read_to_string(&path).expect("read")).expect("json"), + ); assert_eq!( postcheck::compare(&expected, &actual), Vec::::new(), diff --git a/crates/storage-cassandra/tests/migrations_stopped.rs b/crates/storage-cassandra/tests/migrations_stopped.rs index a1df7802f..8ef7b92f8 100644 --- a/crates/storage-cassandra/tests/migrations_stopped.rs +++ b/crates/storage-cassandra/tests/migrations_stopped.rs @@ -246,12 +246,10 @@ async fn assert_finished(session: &Arc, prefix: &str, released "{}/migrations/{dir}/0001_baseline.expected.json", env!("CARGO_MANIFEST_DIR") ); - let mut expected: postcheck::Definitions = - serde_json::from_str(&std::fs::read_to_string(&path).expect("read")).expect("json"); - // Migration 5 drops continuous_backups, which migration 1 creates. - expected - .columns - .retain(|name, _| !name.starts_with("continuous_backups.")); + let expected = migrations_support::after_every_migration( + dir, + serde_json::from_str(&std::fs::read_to_string(&path).expect("read")).expect("json"), + ); assert_eq!(expected, actual, "{at}: {dir}"); } let store = CassandraCatalogStore::catalog_only(session.clone(), prefix.to_owned()); diff --git a/crates/storage-cassandra/tests/migrations_support/mod.rs b/crates/storage-cassandra/tests/migrations_support/mod.rs index 11cd3485d..5157ec017 100644 --- a/crates/storage-cassandra/tests/migrations_support/mod.rs +++ b/crates/storage-cassandra/tests/migrations_support/mod.rs @@ -10,7 +10,7 @@ use std::time::Duration; use extenddb_storage_cassandra::cassandra_util::CassandraSession; use extenddb_storage_cassandra::engine::CassandraEngine; -use extenddb_storage_cassandra::migrations::wait_for_schema_agreement; +use extenddb_storage_cassandra::migrations::{postcheck, wait_for_schema_agreement}; use crate::helpers; @@ -34,6 +34,34 @@ pub fn new_prefix(tag: &str) -> (String, helpers::DropKeyspacesOnDrop) { (prefix, drops) } +/// The definitions a keyspace of store kind `dir` (`catalog` or `account`) +/// holds once every migration ran, from migration 1's expected file: +/// migration 5 drops `continuous_backups`, which migration 1 creates, and +/// migration 6 adds `warm_throughput` to the catalog's `tables` and +/// `indexes`. +pub fn after_every_migration( + dir: &str, + mut expected: postcheck::Definitions, +) -> postcheck::Definitions { + expected + .columns + .retain(|name, _| !name.starts_with("continuous_backups.")); + if dir == "catalog" { + for table in ["tables", "indexes"] { + expected.columns.insert( + format!("{table}.warm_throughput"), + postcheck::Column { + kind: "regular".to_owned(), + position: -1, + clustering_order: "none".to_owned(), + data_type: "text".to_owned(), + }, + ); + } + } + expected +} + /// Run `statement`, then wait until every live node agrees on the schema, /// so the next statement finds what this one created on whichever node it /// reaches (#637). diff --git a/crates/storage-mongodb/src/migrations/mod.rs b/crates/storage-mongodb/src/migrations/mod.rs index 9a3c88598..dca7d21e6 100644 --- a/crates/storage-mongodb/src/migrations/mod.rs +++ b/crates/storage-mongodb/src/migrations/mod.rs @@ -137,6 +137,10 @@ pub fn migration(id: MigrationId) -> Migration { m0005_drop_continuous_backups::SOURCE, )], }, + MigrationId::WarmThroughput => Migration::NotApplicable( + "the catalog collections have no validator, so a table or index document carries \ + a warm_throughput field without a schema change", + ), } } diff --git a/crates/storage-mongodb/src/migrations/tests.rs b/crates/storage-mongodb/src/migrations/tests.rs index 651ad34fe..dc595efae 100644 --- a/crates/storage-mongodb/src/migrations/tests.rs +++ b/crates/storage-mongodb/src/migrations/tests.rs @@ -39,7 +39,7 @@ fn every_migration_records_a_row_in_the_catalog_and_names_its_stores() { } #[test] -fn every_body_is_code_and_three_and_four_do_not_apply() { +fn every_body_is_code_and_three_four_and_six_do_not_apply() { for id in MigrationId::ALL { for body in migration(id).bodies() { assert!(matches!(body.body, Body::Code { .. }), "{id}"); @@ -48,6 +48,7 @@ fn every_body_is_code_and_three_and_four_do_not_apply() { for id in [ MigrationId::SortKeyDefinitions, MigrationId::IndexTableBaseKey, + MigrationId::WarmThroughput, ] { let Migration::NotApplicable(reason) = migration(id) else { panic!("{id} applies on MongoDB"); diff --git a/crates/storage-mongodb/tests/migrations_fresh.rs b/crates/storage-mongodb/tests/migrations_fresh.rs index 12cf1838e..338be0b41 100644 --- a/crates/storage-mongodb/tests/migrations_fresh.rs +++ b/crates/storage-mongodb/tests/migrations_fresh.rs @@ -303,7 +303,7 @@ async fn a_fresh_store_gets_every_migration_and_the_expected_definitions() { let Some(uri) = server() else { return }; let d = Deployment::new(&uri, "fresh").await; let report = d.create().await; - assert_eq!((report.from, report.to), (None, Some(5))); + assert_eq!((report.from, report.to), (None, Some(6))); let applied: Vec<(u32, String)> = report .applied .iter() @@ -318,6 +318,7 @@ async fn a_fresh_store_gets_every_migration_and_the_expected_definitions() { (3, d.target.catalog.clone()), (4, d.target.catalog.clone()), (5, d.target.catalog.clone()), + (6, d.target.catalog.clone()), ] ); let not_applicable: Vec = report @@ -326,8 +327,8 @@ async fn a_fresh_store_gets_every_migration_and_the_expected_definitions() { .filter(|s| s.outcome == Outcome::NotApplicable) .map(|s| s.id) .collect(); - assert_eq!(not_applicable, [3, 4], "{report:?}"); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(not_applicable, [3, 4, 6], "{report:?}"); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); let rows = d.catalog_ledger().await; let outcomes: Vec<(u32, Outcome)> = rows.iter().map(|r| (r.id, r.outcome)).collect(); assert_eq!( @@ -338,6 +339,7 @@ async fn a_fresh_store_gets_every_migration_and_the_expected_definitions() { (3, Outcome::NotApplicable), (4, Outcome::NotApplicable), (5, Outcome::Applied), + (6, Outcome::NotApplicable), ] ); assert!( @@ -347,7 +349,7 @@ async fn a_fresh_store_gets_every_migration_and_the_expected_definitions() { let data_rows = Deployment::documents(&d.data(), "extenddb_migrations").await; assert_eq!(data_rows.len(), 1); assert_eq!(data_rows[0].get_i64("_id"), Ok(1)); - assert_eq!(d.setting("catalog_version").await.as_deref(), Some("5")); + assert_eq!(d.setting("catalog_version").await.as_deref(), Some("6")); assert_eq!(d.setting("resource_tag_migration").await, None); let history = Deployment::documents(&d.catalog(), "schema_history").await; assert_eq!(history.len(), 1); @@ -389,11 +391,11 @@ async fn a_second_run_applies_nothing_and_reconverge_keeps_the_rows() { let before = d.catalog_ledger().await; let again = d.migrate().await.expect("migrate"); assert!(again.applied.is_empty()); - assert_eq!((again.from, again.to), (Some(5), Some(5))); + assert_eq!((again.from, again.to), (Some(6), Some(6))); let reconverged = d.reconverge().await.expect("reconverge"); // Migration 1's steps in the catalog and the data database, and migration - // 5's in the catalog: 2 is not convergent, and 3 and 4 do not apply on + // 5's in the catalog: 2 is not convergent, and 3, 4 and 6 do not apply on // MongoDB. let ran: Vec<(u32, String)> = reconverged .applied @@ -421,7 +423,7 @@ async fn a_second_run_applies_nothing_and_reconverge_keeps_the_rows() { before, "reconverge leaves the rows as they are" ); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); d.drop_databases().await; } @@ -456,7 +458,7 @@ async fn forget_clears_an_unreleased_id_and_migrate_applies_it_again() { assert_eq!(pending.iter().map(|p| p.id).collect::>(), [1, 1]); let report = d.migrate().await.expect("migrate"); assert_eq!(report.applied.len(), 2); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); d.drop_databases().await; } @@ -473,19 +475,19 @@ async fn a_changed_checksum_a_newer_catalog_and_an_older_release_are_reported() doc! { "_id": id, "name": "later", "kind": kind, "checksum": "c", "outcome": "applied", "notes": "", "applied_at": bson::DateTime::now(), "applied_by": "9.9.9" } }; - ledger.insert_one(row(6, "breaking")).await.expect("insert"); + ledger.insert_one(row(7, "breaking")).await.expect("insert"); let status = d.status().await; assert_eq!(status.label(), "AheadIncompatible"); assert_eq!(status.exit_code(), exit::AHEAD_INCOMPATIBLE); ledger - .delete_one(doc! { "_id": 6_i64 }) + .delete_one(doc! { "_id": 7_i64 }) .await .expect("delete"); - ledger.insert_one(row(6, "additive")).await.expect("insert"); + ledger.insert_one(row(7, "additive")).await.expect("insert"); assert_eq!(d.status().await.label(), "AheadCompatible"); assert!(d.migrate().await.expect("migrate").applied.is_empty()); ledger - .delete_one(doc! { "_id": 6_i64 }) + .delete_one(doc! { "_id": 7_i64 }) .await .expect("delete"); @@ -500,7 +502,7 @@ async fn a_changed_checksum_a_newer_catalog_and_an_older_release_are_reported() assert_eq!(refused.exit_code(), exit::ATTENTION); d.reconverge().await.expect("reconverge"); assert_eq!(d.setting("resource_tag_migration").await, None); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); settings .update_one( doc! { "_id": "catalog_version" }, @@ -510,7 +512,7 @@ async fn a_changed_checksum_a_newer_catalog_and_an_older_release_are_reported() .expect("version"); assert_eq!(d.status().await.label(), "WrittenByOlderRelease"); d.reconverge().await.expect("reconverge"); - assert_eq!(d.setting("catalog_version").await.as_deref(), Some("5")); + assert_eq!(d.setting("catalog_version").await.as_deref(), Some("6")); // A recorded checksum this binary's body does not have. ledger @@ -558,6 +560,6 @@ async fn a_ledger_document_this_binary_cannot_read_stops_the_status() { .expect_err("an error"); migrations.close().await; assert!(error.to_string().contains("\"sideways\""), "{error}"); - assert_eq!(MigrationId::LATEST.number(), 5); + assert_eq!(MigrationId::LATEST.number(), 6); d.drop_databases().await; } diff --git a/crates/storage-mongodb/tests/migrations_guards.rs b/crates/storage-mongodb/tests/migrations_guards.rs index 793f65361..55284c96f 100644 --- a/crates/storage-mongodb/tests/migrations_guards.rs +++ b/crates/storage-mongodb/tests/migrations_guards.rs @@ -207,7 +207,7 @@ async fn migrate_creates_the_indexes_a_deployment_lacks() { .await .contains(&"table_id_1".to_owned()) ); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); assert_fresh_definitions(&d).await; d.drop_databases().await; } @@ -232,7 +232,7 @@ async fn stop_at_every_request(uri: &str, state: Option<&str>) -> usize { } assert_eq!( d.status().await, - CatalogStatus::Current { version: 5 }, + CatalogStatus::Current { version: 6 }, "{requests}" ); assert_fresh_definitions(&d).await; diff --git a/crates/storage-mongodb/tests/migrations_holders.rs b/crates/storage-mongodb/tests/migrations_holders.rs index ed0cd866c..5eba917e5 100644 --- a/crates/storage-mongodb/tests/migrations_holders.rs +++ b/crates/storage-mongodb/tests/migrations_holders.rs @@ -113,7 +113,7 @@ async fn migration_2_keeps_the_lock_through_a_scan_longer_than_the_idle_limit() applied.expect("every migration applies"); // A shorter run would apply without the progress the scan records. assert!(took > timing.idle_limit * 4, "the run took {took:?}"); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); assert!(d.catalog_ledger().await.iter().any(|row| row.id == 2)); d.drop_databases().await; } @@ -151,7 +151,7 @@ async fn the_status_read_under_the_lock_keeps_it_through_a_long_scan() { "the status read took {took:?}" ); applied.expect("every migration applies"); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); d.drop_databases().await; } diff --git a/crates/storage-mongodb/tests/migrations_locks.rs b/crates/storage-mongodb/tests/migrations_locks.rs index 84727523a..86c2c9064 100644 --- a/crates/storage-mongodb/tests/migrations_locks.rs +++ b/crates/storage-mongodb/tests/migrations_locks.rs @@ -56,7 +56,7 @@ async fn two_migrators_started_together_apply_once_and_the_other_finds_nothing() [0, 6], "one applies every step and the other finds nothing" ); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); d.drop_databases().await; } @@ -204,7 +204,7 @@ async fn migrate_waits_for_a_running_server_and_a_server_refuses_while_a_migrati .is_none() ); // The preview and verify read the status beside the server. - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); serving.release().await; let (held, _) = lock(&d.target, false, Duration::from_secs(2), Timing::PRODUCTION) .await diff --git a/crates/storage-mongodb/tests/migrations_released.rs b/crates/storage-mongodb/tests/migrations_released.rs index c636de7a9..19db70d96 100644 --- a/crates/storage-mongodb/tests/migrations_released.rs +++ b/crates/storage-mongodb/tests/migrations_released.rs @@ -164,12 +164,12 @@ async fn every_released_catalog_upgrades_to_the_definitions_of_a_fresh_install() panic!("{state} is not behind"); }; assert_eq!(version, None, "{state} has no ledger"); - assert_eq!(pending.len(), 6, "{state}"); + assert_eq!(pending.len(), 7, "{state}"); let report = d.migrate().await.expect("migrate"); - assert_eq!((report.from, report.to), (None, Some(5)), "{state}"); + assert_eq!((report.from, report.to), (None, Some(6)), "{state}"); assert_eq!( d.status().await, - CatalogStatus::Current { version: 5 }, + CatalogStatus::Current { version: 6 }, "{state}" ); @@ -401,7 +401,7 @@ async fn a_reconverge_leaves_policies_written_after_migration_2_as_they_are() { .any(|n| n.starts_with("Skipped 2 resource_tag_condition_keys:")), "{report:?}" ); - assert_eq!(d.status().await, CatalogStatus::Current { version: 5 }); + assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); assert_eq!( documents(Deployment::documents(&d.catalog(), "iam_policies").await), before diff --git a/crates/storage-postgres/migrations/0006_warm_throughput.sql b/crates/storage-postgres/migrations/0006_warm_throughput.sql new file mode 100644 index 000000000..d16ae2020 --- /dev/null +++ b/crates/storage-postgres/migrations/0006_warm_throughput.sql @@ -0,0 +1,12 @@ +-- Copyright 2026 ExtendDB contributors +-- SPDX-License-Identifier: Apache-2.0 +-- +-- Migration 6, warm_throughput, PostgreSQL catalog database: a nullable +-- warm_throughput column on tables and on indexes, holding the WarmThroughput +-- a client set (one JSON record, see extenddb_core::types::WarmThroughputRecord). +-- Stored and reported only: nothing pre-warms or scales on it. A row without +-- a value reports the defaults for its billing mode. IF NOT EXISTS makes a +-- second run change nothing. + +ALTER TABLE tables ADD COLUMN IF NOT EXISTS warm_throughput JSONB; +ALTER TABLE indexes ADD COLUMN IF NOT EXISTS warm_throughput JSONB; diff --git a/crates/storage-postgres/src/create_table.rs b/crates/storage-postgres/src/create_table.rs index ed8fdfdf2..32ddc154c 100755 --- a/crates/storage-postgres/src/create_table.rs +++ b/crates/storage-postgres/src/create_table.rs @@ -183,8 +183,8 @@ impl PostgresEngine { } // WarmThroughput as sent, reported back and acted on by nothing (see - // the shape step on `tables.warm_throughput`). UPDATING for the same - // delay the CREATING to ACTIVE flip takes. + // catalog migration 6, `0006_warm_throughput.sql`). UPDATING for the + // same delay the CREATING to ACTIVE flip takes. let delay_secs = Self::control_plane_delay_secs(&mut tx).await?; let warm_record = warm_throughput_record_for_create(input.warm_throughput.as_ref(), delay_secs); diff --git a/crates/storage-postgres/src/migrations/mod.rs b/crates/storage-postgres/src/migrations/mod.rs index 740c3469a..598ad0517 100644 --- a/crates/storage-postgres/src/migrations/mod.rs +++ b/crates/storage-postgres/src/migrations/mod.rs @@ -127,6 +127,15 @@ pub fn migration(id: MigrationId) -> Migration { "0005_drop_continuous_backups.sql" )], }, + MigrationId::WarmThroughput => Migration::Applies { + kind: Kind::Breaking, + convergent: true, + bodies: vec![file_body!( + StoreKind::Catalog, + "migrations", + "0006_warm_throughput.sql" + )], + }, } } @@ -148,7 +157,8 @@ pub(crate) const fn runs_in(id: MigrationId) -> RunsIn { MigrationId::Baseline | MigrationId::ResourceTagConditionKeys | MigrationId::SortKeyDefinitions - | MigrationId::DropContinuousBackups => RunsIn::Transaction, + | MigrationId::DropContinuousBackups + | MigrationId::WarmThroughput => RunsIn::Transaction, MigrationId::IndexTableBaseKey => RunsIn::Autocommit, } } diff --git a/crates/storage-sqlite/src/create_table.rs b/crates/storage-sqlite/src/create_table.rs index bdf015214..92cee5672 100644 --- a/crates/storage-sqlite/src/create_table.rs +++ b/crates/storage-sqlite/src/create_table.rs @@ -123,9 +123,9 @@ impl SqliteEngine { .and_then(|v| v.parse::().ok()) .unwrap_or(crate::DEFAULT_CONTROL_PLANE_DELAY_SECONDS); - // Stored as sent and reported back; nothing acts on it (see the - // schema comment on `tables.warm_throughput`). UPDATING for the same - // delay the CREATING to ACTIVE flip takes. + // Stored as sent and reported back; nothing acts on it (see catalog + // migration 6, `m0006_warm_throughput`). UPDATING for the same delay + // the CREATING to ACTIVE flip takes. let warm_record = warm_throughput_record_for_create(input.warm_throughput.as_ref(), delay_secs); let warm_json = warm_record diff --git a/crates/storage-sqlite/src/migrations/golden_tests.rs b/crates/storage-sqlite/src/migrations/golden_tests.rs index a2e458ebb..def087988 100644 --- a/crates/storage-sqlite/src/migrations/golden_tests.rs +++ b/crates/storage-sqlite/src/migrations/golden_tests.rs @@ -1,9 +1,9 @@ // Copyright 2026 ExtendDB contributors // SPDX-License-Identifier: Apache-2.0 -//! Golden tests for SQLite migrations 2 to 4, each run on a released catalog -//! with the damage or the content its body handles, and migration 1's -//! accepted and converged pre-release shapes. +//! Golden tests for SQLite migrations 2 to 4 and 6, each run on a released +//! catalog with the damage or the content its body handles, and migration +//! 1's accepted and converged pre-release shapes. use std::path::Path; @@ -627,3 +627,53 @@ async fn migration_4_adds_the_base_key_index_to_index_tables_created_before_v0_1 text.push_str(&dump(&db, "SELECT notes FROM extenddb_migrations WHERE id = 4").await); golden("0004_index_table_base_key", &text); } + +/// Migration 6 adds `warm_throughput` to `tables` and `indexes` of a released +/// catalog, leaving every existing row NULL (DescribeTable then reports the +/// defaults), and running it again changes nothing. +#[tokio::test] +async fn migration_6_adds_the_warm_throughput_columns() { + let db = temp_db("m6"); + load_dump(&db, DEV_0_1_14).await; + let columns = "SELECT COUNT(*) FROM pragma_table_info('tables') WHERE name = 'warm_throughput'"; + assert_eq!( + count(&db, columns).await, + 0, + "the 0.1.14 catalog predates it" + ); + run(&db).await.expect("migrate"); + let report = reconverge(&db).await.expect("reconverge"); + assert!(report.applied.iter().any(|s| s.id == 6), "{report:?}"); + assert_eq!( + status(&db).await, + CatalogStatus::Current { version: LATEST } + ); + let mut text = String::new(); + for table in ["tables", "indexes"] { + text.push_str( + &dump( + &db, + &format!( + "SELECT name, type, \"notnull\", dflt_value FROM pragma_table_info('{table}') \ + WHERE name = 'warm_throughput'" + ), + ) + .await, + ); + } + text.push_str( + &dump( + &db, + "SELECT table_name, warm_throughput FROM tables ORDER BY table_name", + ) + .await, + ); + text.push_str( + &dump( + &db, + "SELECT index_name, warm_throughput FROM indexes ORDER BY index_name", + ) + .await, + ); + golden("0006_warm_throughput", &text); +} diff --git a/crates/storage-sqlite/src/migrations/m0006_warm_throughput.rs b/crates/storage-sqlite/src/migrations/m0006_warm_throughput.rs new file mode 100644 index 000000000..0243de8e7 --- /dev/null +++ b/crates/storage-sqlite/src/migrations/m0006_warm_throughput.rs @@ -0,0 +1,62 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Migration 6, `warm_throughput`, SQLite: a nullable `warm_throughput` +//! column on `tables` and on `indexes`. +//! +//! The column holds the `WarmThroughput` a client set on a table or a global +//! secondary index, one JSON record (see +//! `extenddb_core::types::WarmThroughputRecord`). It is stored and reported +//! only: nothing pre-warms or scales on it. A row without a value reports the +//! defaults for its billing mode. +//! +//! SQLite has no `ADD COLUMN IF NOT EXISTS`, so the body reads each table's +//! columns first and adds the column only when it is missing. A second run +//! changes nothing, which makes the migration convergent. + +use async_trait::async_trait; +use extenddb_storage::migrations::{CodeBody, Done, MigrationError}; + +use super::SqliteStores; + +/// This file, whose SHA-256 the checksum test pins. +pub(crate) const SOURCE: &str = include_str!("m0006_warm_throughput.rs"); + +/// Recorded as this migration's checksum. Changes only with a new migration. +const FINGERPRINT: &str = "0006_warm_throughput/sqlite/1"; + +/// The catalog tables that get the column. +const TABLES: [&str; 2] = ["tables", "indexes"]; + +pub(crate) struct WarmThroughput; + +#[async_trait] +impl CodeBody for WarmThroughput { + fn fingerprint(&self) -> &'static str { + FINGERPRINT + } + + /// Nothing this migration finds stops it. + async fn detect(&self, _stores: &SqliteStores) -> Result, MigrationError> { + Ok(Vec::new()) + } + + async fn run(&self, stores: &SqliteStores) -> Result { + for table in TABLES { + let has_column: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM pragma_table_info(?) WHERE name = 'warm_throughput')", + ) + .bind(table) + .fetch_one(&stores.pool) + .await + .map_err(|e| MigrationError::store(format!("read the columns of {table}"), e))?; + if !has_column { + sqlx::query(&format!("ALTER TABLE {table} ADD COLUMN warm_throughput TEXT")) + .execute(&stores.pool) + .await + .map_err(|e| MigrationError::store(format!("add {table}.warm_throughput"), e))?; + } + } + Ok(Done::default()) + } +} diff --git a/crates/storage-sqlite/src/migrations/mod.rs b/crates/storage-sqlite/src/migrations/mod.rs index 06647fbb4..d0bf8d98a 100644 --- a/crates/storage-sqlite/src/migrations/mod.rs +++ b/crates/storage-sqlite/src/migrations/mod.rs @@ -29,6 +29,9 @@ mod m0003_sort_key_definitions; #[rustfmt::skip] #[allow(clippy::all, deprecated)] mod m0004_index_table_base_key; +#[rustfmt::skip] +#[allow(clippy::all, deprecated)] +mod m0006_warm_throughput; mod postcheck; use std::time::Duration; @@ -106,6 +109,15 @@ pub fn migration(id: MigrationId) -> Migration { "0005_drop_continuous_backups.sql" )], }, + MigrationId::WarmThroughput => Migration::Applies { + kind: Kind::Breaking, + convergent: true, + bodies: vec![StoreBody::code( + StoreKind::Catalog, + &m0006_warm_throughput::WarmThroughput, + m0006_warm_throughput::SOURCE, + )], + }, } } diff --git a/crates/storage-sqlite/src/migrations/tests.rs b/crates/storage-sqlite/src/migrations/tests.rs index a837e802e..7cbea1fe6 100644 --- a/crates/storage-sqlite/src/migrations/tests.rs +++ b/crates/storage-sqlite/src/migrations/tests.rs @@ -5,7 +5,7 @@ //! protocol, the copy, the entry points and the in-memory case. The lock //! is in `serve_tests`, beside the server's refusal while a migration //! holds it. -//! Released catalogs are in `released_tests`, migrations 2 to 4 in +//! Released catalogs are in `released_tests`, migrations 2 to 4 and 6 in //! `golden_tests`, refusals and rollbacks in `refusal_tests`, and the killed //! migrator in `crash_tests`. diff --git a/crates/storage-sqlite/src/warm_throughput.rs b/crates/storage-sqlite/src/warm_throughput.rs index 944ce56b4..3e31853c5 100644 --- a/crates/storage-sqlite/src/warm_throughput.rs +++ b/crates/storage-sqlite/src/warm_throughput.rs @@ -61,7 +61,7 @@ impl SqliteEngine { .fetch_one(&mut **tx) .await .map_err(|e| StorageError::Internal(e.to_string()))?; - let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs)?; + let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs); sqlx::query( "UPDATE tables SET warm_throughput = ? WHERE account_id = ? AND table_name = ?", ) @@ -86,7 +86,7 @@ impl SqliteEngine { let Some(existing) = existing else { return Err(StorageError::IndexNotFound(index_name.clone())); }; - let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs)?; + let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs); sqlx::query( "UPDATE indexes SET warm_throughput = ? WHERE table_id = ? AND index_name = ?", ) @@ -112,12 +112,16 @@ mod tests { let engine = SqliteEngine::new(":memory:", 1, "us-east-1", 409_600) .await .expect("engine"); - crate::schema::apply(&engine.pool).await.expect("schema"); - sqlx::query("UPDATE settings SET value = ? WHERE key = 'control_plane_delay_seconds'") - .bind(delay) - .execute(&engine.pool) - .await - .expect("delay"); + crate::migrations::apply_for_tests(&engine.pool).await; + // No migration seeds the setting, so the row is inserted. + sqlx::query( + "INSERT INTO settings (key, value) VALUES ('control_plane_delay_seconds', ?) \ + ON CONFLICT(key) DO UPDATE SET value = excluded.value", + ) + .bind(delay) + .execute(&engine.pool) + .await + .expect("delay"); sqlx::query("INSERT INTO accounts (account_id, account_name) VALUES (?, 'default')") .bind(ACCOUNT) .execute(&engine.pool) diff --git a/crates/storage-sqlite/tests/fixtures/golden/0006_warm_throughput.txt b/crates/storage-sqlite/tests/fixtures/golden/0006_warm_throughput.txt new file mode 100644 index 000000000..d4b692bc2 --- /dev/null +++ b/crates/storage-sqlite/tests/fixtures/golden/0006_warm_throughput.txt @@ -0,0 +1,9 @@ +-- SELECT name, type, "notnull", dflt_value FROM pragma_table_info('tables') WHERE name = 'warm_throughput' +warm_throughput TEXT 0 NULL +-- SELECT name, type, "notnull", dflt_value FROM pragma_table_info('indexes') WHERE name = 'warm_throughput' +warm_throughput TEXT 0 NULL +-- SELECT table_name, warm_throughput FROM tables ORDER BY table_name +cu_orders NULL +-- SELECT index_name, warm_throughput FROM indexes ORDER BY index_name +by_gpk NULL +by_lsk NULL diff --git a/crates/storage/src/migrations/registry.rs b/crates/storage/src/migrations/registry.rs index dc426fd18..667711895 100644 --- a/crates/storage/src/migrations/registry.rs +++ b/crates/storage/src/migrations/registry.rs @@ -67,6 +67,11 @@ migration_ids! { /// it. DropContinuousBackups = 5, "drop_continuous_backups", "drop the continuous_backups table, which no request reads or writes since #335"; + /// Adds a nullable `warm_throughput` column to the `tables` and `indexes` + /// catalog tables, where the `WarmThroughput` a client set on a table or + /// a global secondary index is stored. + WarmThroughput = 6, "warm_throughput", + "add a nullable warm_throughput column to the tables and indexes catalogs"; } /// Highest id that has shipped in a release. Bodies at or below it are frozen: diff --git a/crates/storage/src/snapshot_conformance.rs b/crates/storage/src/snapshot_conformance.rs index 04b58c9b2..29395d6c5 100644 --- a/crates/storage/src/snapshot_conformance.rs +++ b/crates/storage/src/snapshot_conformance.rs @@ -235,6 +235,7 @@ pub fn table_input(name: &str) -> CreateTableInput { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }]), ..Default::default() } diff --git a/devtools/check-postgres-catalog-lock b/devtools/check-postgres-catalog-lock index 81a76f794..ee87974cc 100755 --- a/devtools/check-postgres-catalog-lock +++ b/devtools/check-postgres-catalog-lock @@ -321,9 +321,9 @@ def through_pooler( contains="Ran again 1 baseline", ) report( - all(f"Ran again {n} " in output for n in (1, 3, 4, 5)) + all(f"Ran again {n} " in output for n in (1, 3, 4, 5, 6)) and "Skipped 2 resource_tag_condition_keys:" in output, - "the reconverge through the pooler ran migrations 1, 3, 4 and 5 again and skipped 2", + "the reconverge through the pooler ran migrations 1, 3, 4, 5 and 6 again and skipped 2", output, ) expect_exit(0, "migrate --forget 3 through the pooler", migrate + ["--forget", "3"], env) diff --git a/docs/getting-started.md b/docs/getting-started.md index f4988779a..1388b7b55 100755 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -269,7 +269,7 @@ Config: extenddb.toml Catalog database: extenddb_catalog --- Checking catalog migrations... - OK: Current: the catalog is at migration 5 + OK: Current: the catalog is at migration 6 --- Checking catalog connection... OK: Connected to catalog. --- Checking data database... diff --git a/docs/manuals/04-quickstart-setup-guide.md b/docs/manuals/04-quickstart-setup-guide.md index 6e8339c6b..6575f9d58 100755 --- a/docs/manuals/04-quickstart-setup-guide.md +++ b/docs/manuals/04-quickstart-setup-guide.md @@ -172,7 +172,7 @@ Config: extenddb.toml Catalog database: extenddb_catalog --- Checking catalog migrations... - OK: Current: the catalog is at migration 5 + OK: Current: the catalog is at migration 6 --- Checking catalog connection... OK: Connected to catalog. --- Checking data database... diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 04e3111a6..7df38da26 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -4,7 +4,9 @@ ## Current Status -This release versions the PostgreSQL, SQLite, Cassandra and MongoDB catalogs by a migration ledger and carries catalog migrations 1 to 5, which `extenddb migrate --list` prints. **Every deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema, and migration 5 drops `continuous_backups`, a table and on MongoDB a collection (see [Catalog 0.0.5](#catalog-005)). +This release versions the PostgreSQL, SQLite, Cassandra and MongoDB catalogs by a migration ledger and carries catalog migrations 1 to 6, which `extenddb migrate --list` prints. **Every deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema, and migration 5 drops `continuous_backups`, a table and on MongoDB a collection (see [Catalog 0.0.5](#catalog-005)). + +Migration 6, `warm_throughput`, adds a nullable `warm_throughput` column to the `tables` and `indexes` catalog tables on PostgreSQL, SQLite and Cassandra. It holds the `WarmThroughput` a client sets on a table or a global secondary index, which `DescribeTable` reports; nothing acts on the value (see [Differences from DynamoDB](../differences-from-dynamodb.md)). Tables created before the migration report the defaults for their billing mode. On MongoDB it changes nothing, since the catalog collections have no validator, and the ledger records it as not applicable. Migration 6 is breaking: to go back to a release without it, restore the backup taken before the upgrade (see [Rollback Procedure](#rollback-procedure)). A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. @@ -39,7 +41,7 @@ Exit status 6 means the migration lock, or a running server, was still held when `verify` also checks the connection to the data database. When that check fails, `verify` exits 1 on a Behind catalog instead of 3, because `migrate --yes` cannot apply the pending migrations while the data database cannot be reached. It still exits 4 or 5 for the statuses that need a person. -A migration is additive when the previous release serves the catalog correctly with it applied, and breaking otherwise. Migrations 1 to 5 are breaking: do not run a 0.1.x server against a migrated catalog. A 0.1.x binary that writes to a migrated catalog (its `migrate --yes`, or a SQLite dev server) leaves it WrittenByOlderRelease. `migrate --yes --reconverge` repairs it by running the convergent migrations again. Migration 2, which rewrites stored policies, is not convergent: the repair skips it and prints a line that says so (see the v0.1.14 notes below). +A migration is additive when the previous release serves the catalog correctly with it applied, and breaking otherwise. Migrations 1 to 6 are breaking: do not run a 0.1.x server against a migrated catalog. A 0.1.x binary that writes to a migrated catalog (its `migrate --yes`, or a SQLite dev server) leaves it WrittenByOlderRelease. `migrate --yes --reconverge` repairs it by running the convergent migrations again. Migration 2, which rewrites stored policies, is not convergent: the repair skips it and prints a line that says so (see the v0.1.14 notes below). ## Writing a New Migration diff --git a/docs/manuals/08-install-linux.md b/docs/manuals/08-install-linux.md index 928ba957d..2b868522e 100755 --- a/docs/manuals/08-install-linux.md +++ b/docs/manuals/08-install-linux.md @@ -118,7 +118,7 @@ Expected: ``` === extenddb verify === ... - OK: Current: the catalog is at migration 5 + OK: Current: the catalog is at migration 6 ... === HEALTHY: All checks passed === ``` diff --git a/docs/manuals/09-install-macos.md b/docs/manuals/09-install-macos.md index 85ba0f1e8..b1ca6de2d 100755 --- a/docs/manuals/09-install-macos.md +++ b/docs/manuals/09-install-macos.md @@ -96,7 +96,7 @@ Expected: ``` === extenddb verify === ... - OK: Current: the catalog is at migration 5 + OK: Current: the catalog is at migration 6 ... === HEALTHY: All checks passed === ``` diff --git a/tests/test_resource_tag_migration.py b/tests/test_resource_tag_migration.py index 4a1252ee4..6c23ab98c 100644 --- a/tests/test_resource_tag_migration.py +++ b/tests/test_resource_tag_migration.py @@ -296,11 +296,16 @@ def deployment(tmp_path): def _as_released_catalog(sqlite_path: str) -> None: """Make a catalog this binary created read as one a release before 1.0 left: no ledger rows (the runner reads an empty ledger as it reads a - missing one), a dotted catalog_version, and the marker row every released - baseline seeded. The runner then applies every migration, the - condition-key rewrite included, as it does on a released catalog.""" + missing one), a dotted catalog_version, the marker row every released + baseline seeded, and none of the columns later migrations add, which + migration 1's check of the v1.0 schema refuses. The runner then applies + every migration, the condition-key rewrite included, as it does on a + released catalog.""" conn = sqlite3.connect(sqlite_path) try: + # Migration 6 adds warm_throughput to tables and indexes. + for table in ("tables", "indexes"): + conn.execute(f"ALTER TABLE {table} DROP COLUMN warm_throughput") conn.execute("DELETE FROM extenddb_migrations") conn.execute( "UPDATE settings SET value = '0.0.3' WHERE key = 'catalog_version'" From f709fb1a3fb9a65b5582ad60152bb824af0a93bc Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:49:34 +0000 Subject: [PATCH 03/15] fix(backup): give the manifest its own index entry types The v1 manifest recorded the source table's indexes as GsiInput and LsiInput, the CreateTable request types, so this branch's WarmThroughput member on GsiInput changed the persisted format and failed nested_types_have_the_pinned_members. A backup does not record warm throughput: DescribeBackup's GlobalSecondaryIndexInfo has no warm member, and a restored table starts with the defaults. BackupGsi and BackupLsi in manifest.rs carry the serde names, member order and skip_serializing_if the request types had when format version 1 was frozen, without WarmThroughput. SourceTableSchema uses them, the member pins check them with the same member lists, and the module docs say the index entries are the manifest's own. The fixtures under testdata/extenddb-manifest/ are unchanged, so no format version bump. --- crates/core/src/backup_format/golden_tests.rs | 43 ++++++------ crates/core/src/backup_format/manifest.rs | 68 ++++++++++++++++--- crates/core/src/backup_format/mod.rs | 6 +- 3 files changed, 83 insertions(+), 34 deletions(-) diff --git a/crates/core/src/backup_format/golden_tests.rs b/crates/core/src/backup_format/golden_tests.rs index 068a13804..8887c540b 100644 --- a/crates/core/src/backup_format/golden_tests.rs +++ b/crates/core/src/backup_format/golden_tests.rs @@ -472,15 +472,15 @@ mod generate_golden_manifest { //! writes new ones beside them. use super::super::manifest::manifest_checksum; use super::super::{ - CURRENT_FORMAT_VERSION, DataFileEntry, ExtenddbManifest, ItemOrder, KeyRange, - SnapshotDescriptor, SourceTableSchema, + BackupGsi, BackupLsi, CURRENT_FORMAT_VERSION, DataFileEntry, ExtenddbManifest, ItemOrder, + KeyRange, SnapshotDescriptor, SourceTableSchema, }; use crate::types::{ - AttributeDefinition, AttributeValue, BillingMode, DistanceFunction, GsiInput, Item, - KeySchemaElement, KeyType, LsiInput, OnDemandThroughput, Projection, ProjectionType, - ProvisionedThroughput, ScalarAttributeType, SearchSchemaElement, SearchSchemaElementType, - SseDescription, SseType, StreamSpecification, StreamViewType, TimeToLiveDescription, - TimeToLiveStatus, VectorAttribute, VectorIndexSpecification, + AttributeDefinition, AttributeValue, BillingMode, DistanceFunction, Item, KeySchemaElement, + KeyType, OnDemandThroughput, Projection, ProjectionType, ProvisionedThroughput, + ScalarAttributeType, SearchSchemaElement, SearchSchemaElementType, SseDescription, SseType, + StreamSpecification, StreamViewType, TimeToLiveDescription, TimeToLiveStatus, + VectorAttribute, VectorIndexSpecification, }; fn key(a: &str, t: &str) -> Item { @@ -554,7 +554,7 @@ mod generate_golden_manifest { "arn:aws:kms:us-east-1:123456789012:key/abcd".to_owned(), ), }), - global_secondary_indexes: vec![GsiInput { + global_secondary_indexes: vec![BackupGsi { index_name: "GenreIndex".to_owned(), key_schema: vec![KeySchemaElement { attribute_name: "Genre".to_owned(), @@ -569,7 +569,7 @@ mod generate_golden_manifest { write_capacity_units: 1, }), }], - local_secondary_indexes: vec![LsiInput { + local_secondary_indexes: vec![BackupLsi { index_name: "TitleIndex".to_owned(), key_schema: vec![ KeySchemaElement { @@ -760,7 +760,7 @@ mod generate_golden_manifest { kms_master_key_arn: None, }), // The GSI of an on-demand table has no `ProvisionedThroughput`. - global_secondary_indexes: vec![GsiInput { + global_secondary_indexes: vec![BackupGsi { index_name: "ByKindGlobal".to_owned(), key_schema: vec![element("kind", KeyType::Hash)], projection: Projection { @@ -769,7 +769,7 @@ mod generate_golden_manifest { }, provisioned_throughput: None, }], - local_secondary_indexes: vec![LsiInput { + local_secondary_indexes: vec![BackupLsi { index_name: "ByKind".to_owned(), key_schema: vec![ element("tenant", KeyType::Hash), @@ -925,15 +925,16 @@ mod member_names { use serde::{Serialize, forward_to_deserialize_any}; use super::super::{ - DataFileEntry, ExtenddbManifest, ItemOrder, KeyRange, SnapshotDescriptor, SourceTableSchema, + BackupGsi, BackupLsi, DataFileEntry, ExtenddbManifest, ItemOrder, KeyRange, + SnapshotDescriptor, SourceTableSchema, }; use super::generate_golden_manifest::{golden, unordered}; use crate::types::{ - AttributeDefinition, AttributeValue, BillingMode, DistanceFunction, GsiInput, - KeySchemaElement, KeyType, LsiInput, OnDemandThroughput, Projection, ProjectionType, - ProvisionedThroughput, ScalarAttributeType, SearchSchemaElement, SearchSchemaElementType, - SseDescription, SseType, StreamSpecification, StreamViewType, TimeToLiveDescription, - TimeToLiveStatus, VectorAttribute, VectorIndexSpecification, + AttributeDefinition, AttributeValue, BillingMode, DistanceFunction, KeySchemaElement, + KeyType, OnDemandThroughput, Projection, ProjectionType, ProvisionedThroughput, + ScalarAttributeType, SearchSchemaElement, SearchSchemaElementType, SseDescription, SseType, + StreamSpecification, StreamViewType, TimeToLiveDescription, TimeToLiveStatus, + VectorAttribute, VectorIndexSpecification, }; /// A deserializer that records what a derived `Deserialize` passes to @@ -1355,8 +1356,8 @@ mod member_names { &["Status", "SSEType", "KMSMasterKeyArn"], ); let gsi = &t.global_secondary_indexes[0]; - check::( - "GsiInput", + check::( + "BackupGsi", gsi, &[ "IndexName", @@ -1370,8 +1371,8 @@ mod member_names { &gsi.projection, &["ProjectionType", "NonKeyAttributes"], ); - check::( - "LsiInput", + check::( + "BackupLsi", &other.source_table.local_secondary_indexes[0], &["IndexName", "KeySchema", "Projection"], ); diff --git a/crates/core/src/backup_format/manifest.rs b/crates/core/src/backup_format/manifest.rs index 28658df85..2aaf9e6e4 100644 --- a/crates/core/src/backup_format/manifest.rs +++ b/crates/core/src/backup_format/manifest.rs @@ -39,8 +39,11 @@ //! what `DescribeTable` and `DescribeTimeToLive` reported for the source; //! `DescribeBackup` reports them as `SourceTableFeatureDetails` and a //! restore does not reproduce them (the restored table has no stream and no -//! TTL, as on the service). Nested members reuse the `DynamoDB` wire types, as on the wire. Most -//! of their optional members are omitted when absent; `OnDemandThroughput` +//! TTL, as on the service). Nested members reuse the `DynamoDB` wire types, as on the wire, +//! except the index entries: [`BackupGsi`] and [`BackupLsi`] are the manifest's own types, +//! with the members, names, order and serialization the `CreateTable` request types had when +//! format version 1 was frozen, so a member the request gains later (a GSI's `WarmThroughput`) +//! does not change the format. Most of their optional members are omitted when absent; `OnDemandThroughput` //! is not, and writes `null` for an absent `MaxReadRequestUnits` or //! `MaxWriteRequestUnits` (`{"MaxReadRequestUnits":100, //! "MaxWriteRequestUnits":null}`), which is its wire serialization and part @@ -93,9 +96,9 @@ use super::layout::{ }; use super::strict_item::item_without_duplicates_raw; use crate::types::{ - AttributeDefinition, BillingMode, GsiInput, Item, KeySchemaElement, KeyType, LsiInput, - OnDemandThroughput, ProvisionedThroughput, SseDescription, StreamSpecification, - TimeToLiveDescription, VectorIndexSpecification, + AttributeDefinition, BillingMode, Item, KeySchemaElement, KeyType, OnDemandThroughput, + Projection, ProvisionedThroughput, SseDescription, StreamSpecification, TimeToLiveDescription, + VectorIndexSpecification, }; /// The backup format version this build reads and writes. @@ -158,9 +161,54 @@ pub enum ItemOrder { Unordered, } +/// A global secondary index of the source table, as the manifest records it. +/// +/// The members, serde names, order and `skip_serializing_if` are those of +/// [`crate::types::GsiInput`] when format version 1 was frozen. The manifest +/// has its own type so that a member the `CreateTable` request gains does not +/// change the persisted format: `WarmThroughput`, which a backup does not +/// record (the service's `GlobalSecondaryIndexInfo` in `DescribeBackup` has +/// no warm member, and a restored table starts with the defaults). +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct BackupGsi { + /// Name of the index. + #[serde(rename = "IndexName")] + pub index_name: String, + /// Key schema of the index. + #[serde(rename = "KeySchema")] + pub key_schema: Vec, + /// Attributes projected into the index. + #[serde(rename = "Projection")] + pub projection: Projection, + /// Provisioned throughput of the index; omitted when absent (the index + /// of an on-demand table). + #[serde( + rename = "ProvisionedThroughput", + skip_serializing_if = "Option::is_none" + )] + pub provisioned_throughput: Option, +} + +/// A local secondary index of the source table, as the manifest records it, +/// with the members, serde names and order of [`crate::types::LsiInput`] when +/// format version 1 was frozen (see [`BackupGsi`]). +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct BackupLsi { + /// Name of the index. + #[serde(rename = "IndexName")] + pub index_name: String, + /// Key schema of the index. + #[serde(rename = "KeySchema")] + pub key_schema: Vec, + /// Attributes projected into the index. + #[serde(rename = "Projection")] + pub projection: Projection, +} + /// The source table definition captured at backup time. /// -/// Field types reuse the existing wire types, so the nested JSON matches what +/// Field types reuse the existing wire types, apart from the index entries +/// ([`BackupGsi`], [`BackupLsi`]), so the nested JSON matches what /// the `DynamoDB` API itself carries (`PascalCase` members inside each element). /// Timestamps are `f64` epoch seconds, the same representation /// [`crate::types::BackupDetails`] uses. @@ -195,10 +243,10 @@ pub struct SourceTableSchema { pub sse_description: Option, /// Global secondary indexes defined on the source table; empty when none. /// Required (module docs). - pub global_secondary_indexes: Vec, + pub global_secondary_indexes: Vec, /// Local secondary indexes defined on the source table; empty when none. /// Required (module docs). - pub local_secondary_indexes: Vec, + pub local_secondary_indexes: Vec, /// Vector indexes, as creation specifications. Three states: `null`, the /// source backend has no vector support (a target with vector support /// restores the table without vector indexes); `[]`, the source supports @@ -1534,7 +1582,7 @@ mod tests { "arn:aws:kms:us-east-1:123456789012:key/abcd".to_owned(), ), }), - global_secondary_indexes: vec![GsiInput { + global_secondary_indexes: vec![BackupGsi { index_name: "GenreIndex".to_owned(), key_schema: vec![KeySchemaElement { attribute_name: "Genre".to_owned(), @@ -1549,7 +1597,7 @@ mod tests { write_capacity_units: 1, }), }], - local_secondary_indexes: vec![LsiInput { + local_secondary_indexes: vec![BackupLsi { index_name: "TitleIndex".to_owned(), key_schema: vec![ KeySchemaElement { diff --git a/crates/core/src/backup_format/mod.rs b/crates/core/src/backup_format/mod.rs index 9287a9124..e254f11b7 100644 --- a/crates/core/src/backup_format/mod.rs +++ b/crates/core/src/backup_format/mod.rs @@ -90,7 +90,7 @@ pub use layout::{ trim_backup_arn, }; pub use manifest::{ - CURRENT_FORMAT_VERSION, DataFileEntry, DataFileReader, ExtenddbManifest, ItemOrder, KeyRange, - KeyRangeCheck, KeyRangeTracker, SnapshotDescriptor, SourceTableSchema, check_format_version, - verify_manifest_checksum, + BackupGsi, BackupLsi, CURRENT_FORMAT_VERSION, DataFileEntry, DataFileReader, ExtenddbManifest, + ItemOrder, KeyRange, KeyRangeCheck, KeyRangeTracker, SnapshotDescriptor, SourceTableSchema, + check_format_version, verify_manifest_checksum, }; From 3546b080d3e04e811642959130c34b7367877f33 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:49:34 +0000 Subject: [PATCH 04/15] fix(engine): refuse a GSI Update that carries a throughput beside WarmThroughput UpdateGsiAction deserialized only IndexName and WarmThroughput, so an Update carrying ProvisionedThroughput or OnDemandThroughput next to WarmThroughput answered 200 and dropped the other member. On an on-demand table switched to PROVISIONED in the same request, the index was left at 0/0. It now reads both members as opaque values. Measured on DynamoDB (us-east-1, 2026-11-03): an Update carrying WarmThroughput beside OnDemandThroughput on an on-demand table, beside ProvisionedThroughput or OnDemandThroughput on a provisioned table, and in the request that switches an on-demand table with a GSI to PROVISIONED, is refused with "One or more parameter values were invalid: WarmThroughput must be the only operation in the request for index: gsi". The rule does not depend on the billing mode, so it is checked with the other element rules, before the table is read. An Update without WarmThroughput is refused once the table is read, under the billing mode the table has after the request: the request's BillingMode when it carries one, the stored mode otherwise. Under PAY_PER_REQUEST one carrying ProvisionedThroughput or nothing gets the measured on-demand sentence; one carrying only OnDemandThroughput, and any under PROVISIONED (the index update of a switch to PROVISIONED included), gets "UpdateGlobalSecondaryIndex is not yet supported", as on main. --- crates/core/src/types/table.rs | 20 +++- crates/engine/src/gsi_updates.rs | 172 +++++++++++++++++++++++++----- crates/engine/src/update_table.rs | 13 ++- 3 files changed, 172 insertions(+), 33 deletions(-) diff --git a/crates/core/src/types/table.rs b/crates/core/src/types/table.rs index 2982af087..d812121b7 100755 --- a/crates/core/src/types/table.rs +++ b/crates/core/src/types/table.rs @@ -1141,14 +1141,30 @@ pub struct DeleteGsiAction { /// Update an existing GSI. /// /// `WarmThroughput` is stored and reported (see [`super::WarmThroughputRecord`]). -/// A provisioned-throughput update is recognized by the deserializer but not -/// yet implemented: the engine returns a clear "not yet supported" error. +/// `ProvisionedThroughput` and `OnDemandThroughput` are recognized by the +/// deserializer but an update of either is not yet implemented: they are read +/// as opaque values only so that the engine refuses an `Update` carrying +/// either, with or without `WarmThroughput`, instead of accepting it and +/// dropping the member. #[derive(Debug, Clone, PartialEq, Deserialize)] pub struct UpdateGsiAction { #[serde(rename = "IndexName")] pub index_name: String, #[serde(rename = "WarmThroughput", default)] pub warm_throughput: Option, + #[serde(rename = "ProvisionedThroughput", default)] + pub provisioned_throughput: Option, + #[serde(rename = "OnDemandThroughput", default)] + pub on_demand_throughput: Option, +} + +impl UpdateGsiAction { + /// Whether the action carries `ProvisionedThroughput` or + /// `OnDemandThroughput`, which ExtendDB does not apply to an index. + #[must_use] + pub fn carries_throughput(&self) -> bool { + self.provisioned_throughput.is_some() || self.on_demand_throughput.is_some() + } } /// `UpdateTable` request body. diff --git a/crates/engine/src/gsi_updates.rs b/crates/engine/src/gsi_updates.rs index 6ca3f7caf..955bd788f 100644 --- a/crates/engine/src/gsi_updates.rs +++ b/crates/engine/src/gsi_updates.rs @@ -15,15 +15,22 @@ //! global secondary index update per index is allowed simultaneously. //! Index: " (T3b one element, V3c two). It is checked before the //! one-action rule, so an Update and a Delete of one index answer it. -//! - An `Update` on an on-demand table that carries anything other than -//! `WarmThroughput` (a `ProvisionedThroughput`, or nothing at all): "The -//! only Updates for index: when TableThroughputMode is +//! - An `Update` that carries `WarmThroughput` beside `ProvisionedThroughput` +//! or `OnDemandThroughput`: "WarmThroughput must be the only operation in +//! the request for index: ", on an on-demand table, on a provisioned +//! table, and in the request that switches an on-demand table to +//! PROVISIONED (measured 2026-11-03). It does not depend on the billing +//! mode, so it is checked here, before the table is read. +//! - An `Update` without `WarmThroughput` on a table that is, or that the +//! request makes, on-demand, carrying a `ProvisionedThroughput` or nothing +//! at all: "The only Updates for index: when TableThroughputMode is //! PAY_PER_REQUEST can be to OnDemandThroughput, WarmThroughput" (T3d, -//! T3e). The billing mode is known only once the table is read, so that -//! rule lives in `warm_throughput::validate_warm_throughput_updates`; here -//! an `Update` without `WarmThroughput` on a table whose mode is not yet -//! known is refused as not yet supported (the service's sentence for a -//! provisioned table was not measured). +//! T3e). The stored billing mode is known only once the table is read, so +//! [`refuse_bare_updates`] runs after the read. ExtendDB does not change an +//! index's throughput, so every other `Update` without `WarmThroughput` (one +//! carrying only `OnDemandThroughput`, any on a provisioned table, the +//! index update of a switch to PROVISIONED) is refused as not yet +//! supported, as before this rule existed. //! //! A `Create` has a valid name, a key schema of defined attributes, and (when //! given) a throughput of at least 1 unit each; a `Delete` has a valid name. @@ -53,10 +60,20 @@ pub fn on_demand_update_members(name: &str) -> String { ) } -/// Whether any `Update` element carries nothing but its index name (a -/// `ProvisionedThroughput` on it is not a member of ExtendDB's action, so it -/// reads the same). Such an element needs the table's billing mode for its -/// sentence, so the caller reads the table first. +/// Measured 2026-11-03: an `Update` carrying `WarmThroughput` beside +/// `ProvisionedThroughput` or `OnDemandThroughput`, the sentence for `name`. +#[must_use] +pub fn index_warm_throughput_only_operation(name: &str) -> String { + format!( + "One or more parameter values were invalid: WarmThroughput must be the only operation \ + in the request for index: {name}" + ) +} + +/// Whether any `Update` element carries no `WarmThroughput`: nothing but its +/// index name, or a throughput ExtendDB does not apply to an index. Such an +/// element needs the table's billing mode for its sentence, so the caller +/// reads the table first. #[must_use] pub(crate) fn has_bare_update(input: &UpdateTableInput) -> bool { input @@ -70,10 +87,15 @@ pub(crate) fn has_bare_update(input: &UpdateTableInput) -> bool { }) } -/// Refuse every `Update` element that carries nothing but its index name: -/// on an on-demand table with the measured sentence (rows T3d, T3e); on a -/// provisioned table, where the service's sentence was not measured, with -/// ExtendDB's own "not yet supported". +/// Refuse every `Update` element that carries no `WarmThroughput`. +/// `pay_per_request` is the billing mode the table has after the request: the +/// request's `BillingMode` when it carries one, the stored mode otherwise. +/// +/// Under PAY_PER_REQUEST an element carrying `ProvisionedThroughput` or +/// nothing at all answers the measured sentence (rows T3d, T3e). Every other +/// such element (only `OnDemandThroughput`, which that sentence names as +/// allowed, or any element under PROVISIONED, where the service changes the +/// index's throughput) answers ExtendDB's own "not yet supported". /// /// # Errors /// @@ -89,11 +111,15 @@ pub(crate) fn refuse_bare_updates( .filter_map(|u| u.update.as_ref()) .find(|a| a.warm_throughput.is_none()) { - return Err(DynamoDbError::ValidationException(if pay_per_request { - on_demand_update_members(&upd.index_name) - } else { - "UpdateGlobalSecondaryIndex is not yet supported".to_owned() - })); + let on_demand_rule = + upd.provisioned_throughput.is_some() || upd.on_demand_throughput.is_none(); + return Err(DynamoDbError::ValidationException( + if pay_per_request && on_demand_rule { + on_demand_update_members(&upd.index_name) + } else { + "UpdateGlobalSecondaryIndex is not yet supported".to_owned() + }, + )); } Ok(()) } @@ -148,11 +174,17 @@ pub(crate) fn validate_gsi_updates(input: &UpdateTableInput) -> Result<(), Dynam )); } if let Some(upd) = &update.update { - // An Update carrying WarmThroughput is stored (checked against the - // index's current values once the table is read). One carrying - // nothing else is refused by [`refuse_bare_updates`] once the + // An Update carrying only WarmThroughput is stored (checked against + // the index's current values once the table is read). One carrying + // a throughput beside it is refused here, in every billing mode; + // one without it is refused by [`refuse_bare_updates`] once the // table's billing mode is known. extenddb_core::validation::validate_index_name(&upd.index_name)?; + if upd.warm_throughput.is_some() && upd.carries_throughput() { + return Err(DynamoDbError::ValidationException( + index_warm_throughput_only_operation(&upd.index_name), + )); + } } if let Some(create) = &update.create { extenddb_core::validation::validate_index_name(&create.index_name)?; @@ -287,9 +319,95 @@ mod tests { .expect("different indexes"); } - /// Rows T3d and T3e: an Update with nothing but the index name (or a - /// member ExtendDB's action does not carry) on an on-demand table answers - /// the measured sentence; on a provisioned table ExtendDB's own. + /// Measured 2026-11-03: an Update carrying WarmThroughput beside + /// ProvisionedThroughput, OnDemandThroughput or both answers the + /// only-operation sentence before the table is read, so the answer is + /// the same under either billing mode and the throughput is never + /// dropped with a 200. + #[test] + fn a_throughput_beside_warm_throughput_is_refused_in_either_billing_mode() { + let warm = json!({ "ReadUnitsPerSecond": 12000, "WriteUnitsPerSecond": 4000 }); + let provisioned = json!({ "ReadCapacityUnits": 9, "WriteCapacityUnits": 9 }); + let on_demand = json!({ "MaxReadRequestUnits": 100 }); + for members in [ + json!({ "ProvisionedThroughput": provisioned }), + json!({ "OnDemandThroughput": on_demand }), + json!({ "ProvisionedThroughput": provisioned, "OnDemandThroughput": on_demand }), + ] { + let mut update = json!({ "IndexName": "gsi", "WarmThroughput": warm }); + update + .as_object_mut() + .unwrap() + .extend(members.as_object().unwrap().clone()); + for billing in [None, Some("PROVISIONED"), Some("PAY_PER_REQUEST")] { + let mut request = input(json!([{ "Update": update }])); + request.billing_mode = billing.map(|b| serde_json::from_value(json!(b)).unwrap()); + assert_eq!( + message(validate_gsi_updates(&request)), + "One or more parameter values were invalid: WarmThroughput must be the only \ + operation in the request for index: gsi", + "{update} with BillingMode {billing:?}" + ); + } + } + } + + /// An Update without WarmThroughput: on an on-demand table one carrying + /// ProvisionedThroughput or nothing answers the measured sentence (rows + /// T3d, T3e); one carrying only OnDemandThroughput, and every one on a + /// provisioned table (the index update of a switch to PROVISIONED + /// included), answers ExtendDB's "not yet supported". + #[test] + fn an_update_without_warm_throughput_answers_by_the_billing_mode_after_the_request() { + let provisioned = json!({ "ReadCapacityUnits": 3, "WriteCapacityUnits": 3 }); + let on_demand = json!({ "MaxReadRequestUnits": 100 }); + let on_demand_sentence = "One or more parameter values were invalid: The only Updates \ + for index: gsi when TableThroughputMode is PAY_PER_REQUEST can be to \ + OnDemandThroughput, WarmThroughput"; + let not_yet = "UpdateGlobalSecondaryIndex is not yet supported"; + for (update, pay_per_request, expected) in [ + (json!({ "IndexName": "gsi" }), true, on_demand_sentence), + ( + json!({ "IndexName": "gsi", "ProvisionedThroughput": provisioned }), + true, + on_demand_sentence, + ), + ( + json!({ "IndexName": "gsi", "ProvisionedThroughput": provisioned, "OnDemandThroughput": on_demand }), + true, + on_demand_sentence, + ), + ( + json!({ "IndexName": "gsi", "OnDemandThroughput": on_demand }), + true, + not_yet, + ), + (json!({ "IndexName": "gsi" }), false, not_yet), + ( + json!({ "IndexName": "gsi", "ProvisionedThroughput": provisioned }), + false, + not_yet, + ), + ( + json!({ "IndexName": "gsi", "OnDemandThroughput": on_demand }), + false, + not_yet, + ), + ] { + let request = input(json!([{ "Update": update }])); + validate_gsi_updates(&request).expect("the element rules pass; the mode decides"); + assert!(has_bare_update(&request)); + assert_eq!( + message(refuse_bare_updates(&request, pay_per_request)), + expected, + "{update} pay_per_request={pay_per_request}" + ); + } + } + + /// Rows T3d and T3e: an Update with nothing but the index name on an + /// on-demand table answers the measured sentence; on a provisioned table + /// ExtendDB's own. #[test] fn a_bare_update_is_refused_with_the_billing_mode_sentence() { let bare = input(json!([{ "Update": { "IndexName": "gsi" } }])); diff --git a/crates/engine/src/update_table.rs b/crates/engine/src/update_table.rs index 061fd29de..576610c7a 100755 --- a/crates/engine/src/update_table.rs +++ b/crates/engine/src/update_table.rs @@ -193,10 +193,15 @@ pub async fn handle_update_table( )?; } if bare_update { - let pay_per_request = existing - .billing_mode_summary - .as_ref() - .is_some_and(|s| s.billing_mode == BillingMode::PayPerRequest); + // The mode the table has after the request: a request that leaves + // PAY_PER_REQUEST is not held to the on-demand rule. + let pay_per_request = match input.billing_mode { + Some(mode) => mode == BillingMode::PayPerRequest, + None => existing + .billing_mode_summary + .as_ref() + .is_some_and(|s| s.billing_mode == BillingMode::PayPerRequest), + }; crate::gsi_updates::refuse_bare_updates(&input, pay_per_request)?; } if touches_warm { From 33d4e66c884d1bd36ccae5937895f8912eca6aa8 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:49:34 +0000 Subject: [PATCH 05/15] fix(engine): keep the larger WarmThroughput when an update overlaps one in progress Inside the UPDATING window a request is validated against the values the window started from, so it can be lower than the change in progress. apply_update wrote the requested side, which lowered the value the decrease rule otherwise refuses. Measured on DynamoDB (us-east-1, 2026-11-03): on an on-demand table and on a GSI, 12200/4200 and then 12000/4000 about 50 ms later both answered 200, and each settled at 12200/4200 ACTIVE. apply_update now keeps, per side, the larger of the requested and the stored value. Outside the window a lower request is refused before it reaches the merge, so nothing changes there. The wire test no longer re-sends the higher values after U3, so its settled assertion checks this rule on ExtendDB; the service takes about ten minutes to settle, which the docstring records with the measurement. --- crates/core/src/types/warm_throughput.rs | 37 +++++++++++++------ .../core/src/types/warm_throughput/tests.rs | 19 +++++++++- tests/python/test_warm_throughput.py | 22 ++++++----- 3 files changed, 54 insertions(+), 24 deletions(-) diff --git a/crates/core/src/types/warm_throughput.rs b/crates/core/src/types/warm_throughput.rs index fadbef9c4..52b769114 100644 --- a/crates/core/src/types/warm_throughput.rs +++ b/crates/core/src/types/warm_throughput.rs @@ -192,12 +192,22 @@ impl WarmThroughputRecord { } /// The record an `UpdateTable` writes over `existing` at `now`: each side - /// the request names replaces the stored side, a side it leaves out keeps - /// the stored value (or stays unset and keeps reporting the default), the - /// `UPDATING` deadline restarts, and the previous pair becomes what - /// `existing` reported at `now` (its own previous pair while it was still - /// `UPDATING`, its stored sides otherwise), so a request inside the window - /// keeps reporting the values the window started from. + /// the request names becomes the larger of the requested and the stored + /// side, a side it leaves out keeps the stored value (or stays unset and + /// keeps reporting the default), the `UPDATING` deadline restarts, and the + /// previous pair becomes what `existing` reported at `now` (its own + /// previous pair while it was still `UPDATING`, its stored sides + /// otherwise), so a request inside the window keeps reporting the values + /// the window started from. + /// + /// The larger side is kept because a request validated against the + /// previous pair can be lower than the change still in progress. Measured + /// on 2026-11-03 on an on-demand table and on a GSI: 12200/4200 and then + /// 12000/4000 about 50 ms later both answered 200, and each settled at + /// 12200/4200 `ACTIVE`. Outside the window a lower request is refused as a + /// decrease before it gets here, so the rule changes nothing there; it + /// also keeps two concurrent `UpdateTable`s, each validated against the + /// description read before either wrote, from lowering the value. /// /// A `spec` with neither side (`WarmThroughput: {}`) changes no value and /// restarts the window. The service answers `InternalFailure` with an empty @@ -222,13 +232,16 @@ impl WarmThroughputRecord { existing.write_units_per_second, ) }; + let side = |requested: Option, stored: Option| match (requested, stored) { + (Some(r), Some(s)) => Some(r.max(s)), + (requested, stored) => requested.or(stored), + }; Self { - read_units_per_second: spec - .read_units_per_second - .or(existing.read_units_per_second), - write_units_per_second: spec - .write_units_per_second - .or(existing.write_units_per_second), + read_units_per_second: side(spec.read_units_per_second, existing.read_units_per_second), + write_units_per_second: side( + spec.write_units_per_second, + existing.write_units_per_second, + ), previous_read_units_per_second: previous_read, previous_write_units_per_second: previous_write, updating_until, diff --git a/crates/core/src/types/warm_throughput/tests.rs b/crates/core/src/types/warm_throughput/tests.rs index 62cf714aa..70d63d34e 100644 --- a/crates/core/src/types/warm_throughput/tests.rs +++ b/crates/core/src/types/warm_throughput/tests.rs @@ -108,7 +108,10 @@ fn the_previous_pair_is_reported_and_compared_while_updating() { ); assert_eq!(raised.reported(on_demand(), 10.0), (24_000, 8_000)); // U3: inside the window a request carrying the previous values validates - // against 12000/4000 and, applied, keeps reporting 12000/4000. + // against 12000/4000, keeps reporting 12000/4000 while UPDATING, and + // settles at the higher pair still in progress, as the service does + // (measured 2026-11-03: 12200/4200, then 12000/4000 inside the window, + // settled at 12200/4200 on a table and on a GSI). let inside = WarmThroughputRecord::apply_update( Some(raised), &WarmThroughput { @@ -119,7 +122,19 @@ fn the_previous_pair_is_reported_and_compared_while_updating() { 5.0, ); assert_eq!(inside.reported(on_demand(), 6.0), (12_000, 4_000)); - assert_eq!(inside.reported(on_demand(), 15.0), (12_000, 4_000)); + assert_eq!(inside.reported(on_demand(), 15.0), (24_000, 8_000)); + // One side lower and one higher than the change in progress: each side + // keeps the larger value. + let mixed = WarmThroughputRecord::apply_update( + Some(raised), + &WarmThroughput { + read_units_per_second: Some(12_000), + write_units_per_second: Some(9_000), + }, + 15.0, + 5.0, + ); + assert_eq!(mixed.reported(on_demand(), 15.0), (24_000, 9_000)); // S2c: the same values re-sent on a settled table report themselves UPDATING. let same = WarmThroughputRecord::apply_update( Some(raised), diff --git a/tests/python/test_warm_throughput.py b/tests/python/test_warm_throughput.py index 93ffa31ed..b2024db47 100644 --- a/tests/python/test_warm_throughput.py +++ b/tests/python/test_warm_throughput.py @@ -144,13 +144,17 @@ def test_create_with_warm_throughput_is_present_at_once_and_partial_is_echoed( def test_update_warm_throughput_goes_updating_then_active( self, dynamodb_client, tables, is_extenddb ): - """Rows U1, U2, U3, U5, U6, S2a, S2b and S2c: UpdateTable with WarmThroughput - is accepted and the response reports the PREVIOUS values with Status - UPDATING while TableStatus and IndexStatus stay ACTIVE; inside the window a - request carrying those previous values is accepted (U3); once settled the new - values read ACTIVE and the same request is a decrease (S2b). Only the waits - are gated: ExtendDB returns to ACTIVE within the control plane delay, the - service after minutes, so the settled assertions run on ExtendDB only.""" + """Rows U1, U2, U3, U3b, U5, U6, S2a, S2b and S2c: UpdateTable with + WarmThroughput is accepted and the response reports the PREVIOUS values with + Status UPDATING while TableStatus and IndexStatus stay ACTIVE; inside the + window a request carrying those previous values is accepted (U3) and does not + lower the change in progress: the table settles at the higher values (U3b, + measured 2026-11-03: 12200/4200 and then 12000/4000 about 50 ms later both + answered 200, and the table and a GSI each read 12200/4200 ACTIVE once + settled, about ten minutes later); once settled the same request is a + decrease (S2b). Only the waits are gated: ExtendDB returns to ACTIVE within + the control plane delay, the service after minutes, so the settled + assertions run on ExtendDB only.""" name, _ = _create( dynamodb_client, tables, @@ -180,9 +184,7 @@ def test_update_warm_throughput_goes_updating_then_active( "TableDescription" ] assert t["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} - if is_extenddb: - _wait_warm_active(dynamodb_client, name) - dynamodb_client.update_table(TableName=name, WarmThroughput=higher) + # U3b: the table settles at the higher values, whether or not U3 ran. if is_extenddb: active = _wait_warm_active(dynamodb_client, name) assert active["WarmThroughput"] == {**higher, "Status": "ACTIVE"} From ed0a4811b6ba270c15b88fa8909571d131564b98 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:49:34 +0000 Subject: [PATCH 06/15] fix(storage): merge concurrent WarmThroughput updates without losing a side MongoDB read the record with find_one and wrote the merge back with $set, and Cassandra read the column and wrote it through the table LWT (conditional only on IF EXISTS) and a plain UPDATE of the index row, so two UpdateTables that set different sides could both answer 200 while one side was lost. - MongoDB: the update_one filter includes the field as read; when nothing matches, the document is read again and the merge retried. - Cassandra: with a WarmThroughput, the table LWT is conditional on table_id and on the record read; a GSI's record is written by its own LWT, conditional on index_type and the record read. Both read again and retry when the condition fails. The BillingMode arm that let a switch to PAY_PER_REQUEST skip the column on an unmigrated catalog, and swallowed any read or parse error with it, is gone: migration 6 runs before a server serves the catalog. - PostgreSQL: the restart of a created GSI's UPDATING window after the build is conditional on the record the Create inserted, so an Update of the index accepted while the build ran keeps its value. A stored record that does not parse (a hand edit, or a shape this build does not know) is now logged and read as never set on every backend, as the neighbouring on_demand_throughput column is read. DescribeTable reports the defaults, and UpdateTable writes a record over it, where every description build failed with a 500 before. A wire test sends ten rounds of two UpdateTables together, one per side, on the table and on a GSI, and asserts both sides settle. It runs on ExtendDB only, since the service bills each increase. --- crates/storage-cassandra/src/table_helpers.rs | 6 +- crates/storage-cassandra/src/update_table.rs | 173 +++++++++++------- .../storage-cassandra/src/warm_throughput.rs | 92 ++++++++-- crates/storage-mongodb/src/table_engine.rs | 4 +- crates/storage-mongodb/src/warm_throughput.rs | 153 +++++++++++----- crates/storage-postgres/src/table_helpers.rs | 4 +- crates/storage-postgres/src/update_table.rs | 8 +- .../storage-postgres/src/warm_throughput.rs | 37 ++-- crates/storage-sqlite/src/table_helpers.rs | 4 +- crates/storage/src/util/warm_throughput.rs | 75 ++++---- tests/python/test_warm_throughput_updates.py | 66 +++++++ 11 files changed, 436 insertions(+), 186 deletions(-) diff --git a/crates/storage-cassandra/src/table_helpers.rs b/crates/storage-cassandra/src/table_helpers.rs index b24531e2d..ed2cbc28b 100644 --- a/crates/storage-cassandra/src/table_helpers.rs +++ b/crates/storage-cassandra/src/table_helpers.rs @@ -165,9 +165,9 @@ impl CassandraEngine { .ok() .flatten() .and_then(|s: String| serde_json::from_str(&s).ok()); - // Absent on a catalog that has not run V006 (SELECT * tolerates it). + // Added by catalog migration 6; null when the client never set it. let warm_json: Option = table_row.get_by_name("warm_throughput").ok().flatten(); - let warm_throughput_record = parse_warm_throughput_record(warm_json.as_deref())?; + let warm_throughput_record = parse_warm_throughput_record(warm_json.as_deref()); // A restore target reports its registration for as long as the row // carries it; the engine shows it only while the table is CREATING. @@ -233,7 +233,7 @@ impl CassandraEngine { item_count: 0, index_arn: index_arn(&self.region, account_id, &table_name, &index_name), warm_throughput: None, - warm_throughput_record: parse_warm_throughput_record(warm_json.as_deref())?, + warm_throughput_record: parse_warm_throughput_record(warm_json.as_deref()), }); } else { lsis.push(LsiDescription { diff --git a/crates/storage-cassandra/src/update_table.rs b/crates/storage-cassandra/src/update_table.rs index 187c6fa36..a8cb32633 100644 --- a/crates/storage-cassandra/src/update_table.rs +++ b/crates/storage-cassandra/src/update_table.rs @@ -155,54 +155,25 @@ impl CassandraEngine { // WarmThroughput on the table and on updated indexes: stored as sent, // nothing acts on it. Validated by the engine against the current values. - // - // A request with a BillingMode and a WarmThroughput is the one the - // engine builds for a switch to PAY_PER_REQUEST (a client cannot send - // both). It must not fail on a catalog that has not run V006, where - // the column is absent: the switch is skipped past the warm throughput - // and the reported value still follows the on-demand minimum at read - // time. A request that names WarmThroughput on its own fails there, as - // documented. - let warm_json = match &input.warm_throughput { - Some(spec) => { - match self - .merged_table_warm_throughput(account_id, &input.table_name, spec) - .await - { - Ok(json) => Some(json), - Err(e) if input.billing_mode.is_some() => { - tracing::warn!( - "update_table: warm throughput not stored with the billing mode \ - change (catalog migration V006 not applied?): {e}" - ); - None - } - Err(e) => return Err(e), - } - } - None => None, - }; - if let Some(json) = warm_json.as_deref() { - add_update!("warm_throughput", json); + // The table's record is written by the table LWT below, conditional on + // the record read here; the indexes' by their own LWTs after the + // indexes batch (`store_index_warm_throughput`). Both read again and + // retry when another UpdateTable wrote the record first. + let mut warm_read: Option = None; + if let Some(spec) = &input.warm_throughput { + let merge = self + .merged_table_warm_throughput(account_id, &input.table_name, spec) + .await?; + add_update!("warm_throughput", merge.merged.as_str()); + warm_read = merge.read; } for update in input.global_secondary_index_updates.iter().flatten() { if let Some(action) = &update.update && let Some(spec) = &action.warm_throughput { - let json = self - .merged_index_warm_throughput(&table_id, &action.index_name, spec) + // Refuse a missing index before anything is written. + self.merged_index_warm_throughput(&table_id, &action.index_name, spec) .await?; - add_index_op!( - format!( - "UPDATE {catalog_ks}.indexes SET warm_throughput = ? \ - WHERE table_id = ? AND index_name = ?" - ), - QueryValues::SimpleValues(vec![ - Value::from(json.as_str()), - Value::from(table_id.as_str()), - Value::from(action.index_name.as_str()), - ]) - ); } } @@ -483,6 +454,7 @@ impl CassandraEngine { account_id, &input, table_cols, + warm_read, indexes_batch, indexes_batch_has_statements, needs_shard_init, @@ -517,7 +489,8 @@ impl CassandraEngine { &self, account_id: &str, input: &UpdateTableInput, - table_cols: Vec<(&'static str, Value)>, + mut table_cols: Vec<(&'static str, Value)>, + mut warm_read: Option, indexes_batch: BatchQueryBuilder, indexes_batch_has_statements: bool, needs_shard_init: bool, @@ -574,20 +547,92 @@ impl CassandraEngine { .map(|(col, _)| format!("{col} = ?")) .collect::>() .join(", "); + // With a WarmThroughput the condition is also on the record read, + // and on `table_id`, which a missing row does not have. + let warm_col = table_cols + .iter() + .position(|(col, _)| *col == "warm_throughput"); + let condition = if warm_col.is_some() { + "IF table_id = ? AND warm_throughput = ?" + } else { + "IF EXISTS" + }; let cql = format!( "UPDATE {catalog_ks}.tables SET {set_clause} \ - WHERE account_id = ? AND table_name = ? IF EXISTS" + WHERE account_id = ? AND table_name = ? {condition}" ); - let mut vals: Vec = table_cols.into_iter().map(|(_, v)| v).collect(); - vals.push(Value::from(account_id)); - vals.push(Value::from(input.table_name.as_str())); - let result = crate::cassandra_util::query_lwt( - &self.session, - &cql, - QueryValues::SimpleValues(vals), - ) - .await - .inspect_err(|_e| { + let mut attempts = 0; + let applied = loop { + let mut vals: Vec = table_cols.iter().map(|(_, v)| v.clone()).collect(); + vals.push(Value::from(account_id)); + vals.push(Value::from(input.table_name.as_str())); + if warm_col.is_some() { + vals.push(Value::from(table_id)); + vals.push(crate::warm_throughput::condition_value( + warm_read.as_deref(), + )); + } + let result = crate::cassandra_util::query_lwt( + &self.session, + &cql, + QueryValues::SimpleValues(vals), + ) + .await + .inspect_err(|_e| { + for held_id in &taken_holds { + let session = self.session_arc(); + let account_ks = account_ks.clone(); + let held_id = held_id.clone(); + let table_id = table_id.to_owned(); + tokio::spawn(async move { + let _ = crate::propagation_hold::release_propagation_hold( + &session, + &account_ks, + &table_id, + &held_id, + ) + .await; + }); + } + })?; + if crate::cassandra_util::lwt_applied(&result)? { + break Ok(true); + } + // Not applied: the row is gone, or another UpdateTable wrote the + // warm throughput since it was read. Merge again over what that + // request wrote and retry; a missing row ends the loop. + let (Some(col), Some(spec)) = (warm_col, input.warm_throughput.as_ref()) else { + break Ok(false); + }; + attempts += 1; + if attempts >= crate::warm_throughput::MERGE_ATTEMPTS { + break Err(StorageError::Internal( + "warm throughput: the record changed on every attempt to merge the \ + request" + .to_owned(), + )); + } + match self + .merged_table_warm_throughput(account_id, &input.table_name, spec) + .await + { + Ok(merge) => { + table_cols[col].1 = Value::from(merge.merged.as_str()); + warm_read = merge.read; + } + Err(StorageError::TableNotFound(_)) => break Ok(false), + Err(e) => break Err(e), + } + }; + // A condition that stays false means the row was deleted between + // our earlier SELECT and this UPDATE: treat as TableNotFound. + // NOTE: the tables LWT commits before the indexes batch below runs. + // A failure in the indexes batch leaves the table row updated with + // no corresponding index rows; the recovery worker does not cover + // this gap (it only scans existing CREATING index rows). This is a + // known atomicity gap — conditional batches cannot span tables in + // Cassandra. Tracked in ADR-0021. + let applied = applied.inspect_err(|_e| { for held_id in &taken_holds { let session = self.session_arc(); let account_ks = account_ks.clone(); @@ -604,15 +649,6 @@ impl CassandraEngine { }); } })?; - // IF EXISTS returns false only if the row was deleted between our - // earlier SELECT and this UPDATE — treat as TableNotFound. - // NOTE: the tables LWT commits before the indexes batch below runs. - // A failure in the indexes batch leaves the table row updated with - // no corresponding index rows; the recovery worker does not cover - // this gap (it only scans existing CREATING index rows). This is a - // known atomicity gap — conditional batches cannot span tables in - // Cassandra. Tracked in ADR-0021. - let applied = crate::cassandra_util::lwt_applied(&result)?; if !applied { for held_id in &taken_holds { let session = self.session_arc(); @@ -660,6 +696,17 @@ impl CassandraEngine { return Err(e); } + // Post-batch: each GSI Update's WarmThroughput, as its own conditional + // write on the index row (see `store_index_warm_throughput`). + for update in input.global_secondary_index_updates.iter().flatten() { + if let Some(action) = &update.update + && let Some(spec) = &action.warm_throughput + { + self.store_index_warm_throughput(table_id, &action.index_name, spec) + .await?; + } + } + // Post-batch: shard init (has its own internal batch across two keyspaces). if needs_shard_init { self.init_stream_shards(account_id, &input.table_name, &account_ks, table_id) diff --git a/crates/storage-cassandra/src/warm_throughput.rs b/crates/storage-cassandra/src/warm_throughput.rs index 7581b9d3d..af51c2cfb 100644 --- a/crates/storage-cassandra/src/warm_throughput.rs +++ b/crates/storage-cassandra/src/warm_throughput.rs @@ -5,7 +5,7 @@ //! the `UPDATING` window takes, and the `UpdateTable` merges. //! //! The record is written as sent (one JSON text per table and per index, see -//! catalog migration V006) and read back by DescribeTable; nothing pre-warms +//! catalog migration 6) and read back by DescribeTable; nothing pre-warms //! or scales on it, and it has no effect on throughput, throttling or //! storage. See `extenddb_core::types::WarmThroughputRecord`. @@ -14,8 +14,29 @@ use extenddb_core::types::WarmThroughput; use extenddb_storage::error::StorageError; use extenddb_storage::util::{warm_throughput_record_for_update, warm_throughput_record_json}; +use cdrs_tokio::query::QueryValues; +use cdrs_tokio::types::value::Value; + use crate::CassandraEngine; -use crate::cassandra_util::query_optional; +use crate::cassandra_util::{lwt_applied, query_lwt, query_optional}; + +/// How many times a conditional warm throughput write reads the record again +/// after another `UpdateTable` changed it between the read and the write. +pub(crate) const MERGE_ATTEMPTS: usize = 32; + +/// A stored record as read (`None` when the column is null) and the JSON to +/// write over it. The write is conditional on `read`, so a merge that another +/// request overtook is retried instead of overwriting that request's side. +pub(crate) struct WarmMerge { + pub(crate) read: Option, + pub(crate) merged: String, +} + +/// The bind value of a lightweight transaction condition on a text column +/// that may be null. +pub(crate) fn condition_value(read: Option<&str>) -> Value { + read.map_or(Value::Null, Value::from) +} impl CassandraEngine { /// `control_plane_delay_seconds` from the settings table (0.25 when @@ -41,15 +62,13 @@ impl CassandraEngine { } /// The JSON an `UpdateTable` stores in `tables.warm_throughput`: the - /// request merged over the current record. Read as its own statement so - /// a catalog that has not run V006 only fails a request that names the - /// member. + /// request merged over the current record, with the record as read. pub(crate) async fn merged_table_warm_throughput( &self, account_id: &str, table_name: &str, spec: &WarmThroughput, - ) -> Result { + ) -> Result { let catalog_ks = self.catalog_keyspace(); let row = query_optional::( &self.session, @@ -67,19 +86,23 @@ impl CassandraEngine { existing.as_deref(), spec, self.control_plane_delay_secs().await, - )?; - warm_throughput_record_json(&record) + ); + Ok(WarmMerge { + merged: warm_throughput_record_json(&record)?, + read: existing, + }) } /// The JSON an `UpdateTable` stores in `indexes.warm_throughput` for a /// `GlobalSecondaryIndexUpdates.Update`: the request merged over the - /// index's current record. `IndexNotFound` when the table has no such GSI. + /// index's current record, with the record as read. `IndexNotFound` when + /// the table has no such GSI. pub(crate) async fn merged_index_warm_throughput( &self, table_id: &str, index_name: &str, spec: &WarmThroughput, - ) -> Result { + ) -> Result { let catalog_ks = self.catalog_keyspace(); let row = query_optional::( &self.session, @@ -101,7 +124,52 @@ impl CassandraEngine { existing.as_deref(), spec, self.control_plane_delay_secs().await, - )?; - warm_throughput_record_json(&record) + ); + Ok(WarmMerge { + merged: warm_throughput_record_json(&record)?, + read: existing, + }) + } + + /// Store a `GlobalSecondaryIndexUpdates.Update.WarmThroughput` merged over + /// the index's record, as a lightweight transaction conditional on the + /// record read; when another `UpdateTable` wrote the record first it is + /// read again and merged again, so two requests that set different sides + /// both keep their side. + pub(crate) async fn store_index_warm_throughput( + &self, + table_id: &str, + index_name: &str, + spec: &WarmThroughput, + ) -> Result<(), StorageError> { + let catalog_ks = self.catalog_keyspace(); + let cql = format!( + "UPDATE {catalog_ks}.indexes SET warm_throughput = ? \ + WHERE table_id = ? AND index_name = ? \ + IF index_type = 'GSI' AND warm_throughput = ?" + ); + for _ in 0..MERGE_ATTEMPTS { + let merge = self + .merged_index_warm_throughput(table_id, index_name, spec) + .await?; + let result = query_lwt( + &self.session, + &cql, + QueryValues::SimpleValues(vec![ + Value::from(merge.merged.as_str()), + Value::from(table_id), + Value::from(index_name), + condition_value(merge.read.as_deref()), + ]), + ) + .await?; + if lwt_applied(&result)? { + return Ok(()); + } + } + Err(StorageError::Internal(format!( + "warm throughput of index {index_name}: the record changed on every attempt to \ + merge the request" + ))) } } diff --git a/crates/storage-mongodb/src/table_engine.rs b/crates/storage-mongodb/src/table_engine.rs index dd71485dc..663b2eff6 100644 --- a/crates/storage-mongodb/src/table_engine.rs +++ b/crates/storage-mongodb/src/table_engine.rs @@ -1611,7 +1611,7 @@ impl MongoEngine { warm_throughput: None, warm_throughput_record: crate::warm_throughput::warm_record_from_doc( idx_doc, - )?, + ), }); } "LSI" => { @@ -1708,7 +1708,7 @@ impl MongoEngine { table_class_summary, on_demand_throughput, restore_summary, - warm_throughput_record: crate::warm_throughput::warm_record_from_doc(doc)?, + warm_throughput_record: crate::warm_throughput::warm_record_from_doc(doc), // Fields for features this backend does not implement, vector // indexes today, take their defaults. Adding one to // TableDescription then does not break this build. diff --git a/crates/storage-mongodb/src/warm_throughput.rs b/crates/storage-mongodb/src/warm_throughput.rs index be1605e92..659903e77 100644 --- a/crates/storage-mongodb/src/warm_throughput.rs +++ b/crates/storage-mongodb/src/warm_throughput.rs @@ -14,6 +14,7 @@ use extenddb_core::types::{ }; use extenddb_storage::error::StorageError; use extenddb_storage::util::warm_throughput_record_for_create; +use mongodb::Collection; use crate::MongoEngine; @@ -28,19 +29,25 @@ pub(crate) fn warm_record_bson( } } -/// The record on a table or index document, if any. -/// -/// # Errors -/// -/// `Internal` when the stored field is present but not a record. -pub(crate) fn warm_record_from_doc( - doc: &Document, -) -> Result, StorageError> { - match doc.get("warm_throughput") { - None | Some(Bson::Null) => Ok(None), - Some(b) => bson::from_bson(b.clone()) - .map(Some) - .map_err(|e| StorageError::Internal(format!("warm throughput record: {e}"))), +/// The record on a table or index document, if any. A field that is not a +/// record is logged and read as never set, so the defaults are reported (see +/// `extenddb_storage::util::parse_warm_throughput_record`). +pub(crate) fn warm_record_from_doc(doc: &Document) -> Option { + warm_record_from_bson(doc.get("warm_throughput")) +} + +fn warm_record_from_bson(field: Option<&Bson>) -> Option { + match field { + None | Some(Bson::Null) => None, + Some(b) => match bson::from_bson(b.clone()) { + Ok(record) => Some(record), + Err(e) => { + tracing::warn!( + "stored warm_throughput is not a record, reporting the defaults: {e}" + ); + None + } + }, } } @@ -54,12 +61,61 @@ pub(crate) fn warm_record_bson_for_create( Ok((record, bson)) } +/// How many times a merge reads the record again after another write changed +/// it between the read and the conditional write. +const MERGE_ATTEMPTS: usize = 32; + +/// Merge `spec` into the `warm_throughput` field of the document `filter` +/// selects, starting from `current`, the field as read. The write is +/// conditional on the field still holding what was read; when another +/// `UpdateTable` wrote it first, the document is read again and the merge +/// retried, so two requests that set different sides both keep their side. +async fn merge_warm_throughput( + coll: &Collection, + filter: &Document, + mut current: Bson, + spec: &WarmThroughput, + (updating_until, now): (f64, f64), + not_found: impl Fn() -> StorageError, +) -> Result<(), StorageError> { + for _ in 0..MERGE_ATTEMPTS { + let record = WarmThroughputRecord::apply_update( + warm_record_from_bson(Some(¤t)), + spec, + updating_until, + now, + ); + let mut guarded = filter.clone(); + guarded.insert("warm_throughput", current.clone()); + let result = coll + .update_one( + guarded, + doc! { "$set": { "warm_throughput": warm_record_bson(Some(&record))? } }, + ) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if result.matched_count > 0 { + return Ok(()); + } + let doc = coll + .find_one(filter.clone()) + .await + .map_err(|e| StorageError::Internal(e.to_string()))? + .ok_or_else(¬_found)?; + current = doc.get("warm_throughput").cloned().unwrap_or(Bson::Null); + } + Err(StorageError::Internal( + "warm throughput: the record changed on every attempt to merge the request".to_owned(), + )) +} + impl MongoEngine { /// Store the request's table-level `WarmThroughput` and each /// `GlobalSecondaryIndexUpdates.Update.WarmThroughput`, merged over what - /// is stored. Read-modify-write without a transaction, like the rest of - /// this backend's UpdateTable. The engine has already validated the - /// values against the current ones. + /// is stored. Each merge is a conditional write on the value it read, + /// retried when another request wrote first (see + /// `merge_warm_throughput`). The engine has already validated the values + /// against the current ones. pub(crate) async fn store_warm_throughput_updates( &self, account_id: &str, @@ -87,19 +143,18 @@ impl MongoEngine { .ok_or_else(|| StorageError::TableNotFound(input.table_name.clone()))?; if let Some(spec) = &input.warm_throughput { - let record = WarmThroughputRecord::apply_update( - warm_record_from_doc(&table_doc)?, + merge_warm_throughput( + &tables_coll, + &table_filter, + table_doc + .get("warm_throughput") + .cloned() + .unwrap_or(Bson::Null), spec, - updating_until, - now, - ); - tables_coll - .update_one( - table_filter, - doc! { "$set": { "warm_throughput": warm_record_bson(Some(&record))? } }, - ) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; + (updating_until, now), + || StorageError::TableNotFound(input.table_name.clone()), + ) + .await?; } if gsi_updates.is_empty() { @@ -119,19 +174,18 @@ impl MongoEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))? .ok_or_else(|| StorageError::IndexNotFound(index_name.clone()))?; - let record = WarmThroughputRecord::apply_update( - warm_record_from_doc(&index_doc)?, + merge_warm_throughput( + &indexes_coll, + &filter, + index_doc + .get("warm_throughput") + .cloned() + .unwrap_or(Bson::Null), spec, - updating_until, - now, - ); - indexes_coll - .update_one( - filter, - doc! { "$set": { "warm_throughput": warm_record_bson(Some(&record))? } }, - ) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; + (updating_until, now), + || StorageError::IndexNotFound(index_name.clone()), + ) + .await?; } Ok(()) } @@ -151,19 +205,20 @@ mod tests { updating_until: 1.5, }; let doc = doc! { "warm_throughput": warm_record_bson(Some(&record)).expect("bson") }; - assert_eq!(warm_record_from_doc(&doc).expect("parses"), Some(record)); + assert_eq!(warm_record_from_doc(&doc), Some(record)); let null = doc! { "warm_throughput": Bson::Null }; - assert_eq!(warm_record_from_doc(&null).expect("parses"), None); - assert_eq!(warm_record_from_doc(&doc! {}).expect("parses"), None); + assert_eq!(warm_record_from_doc(&null), None); + assert_eq!(warm_record_from_doc(&doc! {}), None); assert_eq!(warm_record_bson(None).expect("bson"), Bson::Null); } #[test] - fn garbage_in_the_field_is_an_internal_error() { - let bad = doc! { "warm_throughput": "not a record" }; - assert!(matches!( - warm_record_from_doc(&bad), - Err(StorageError::Internal(m)) if m.starts_with("warm throughput record:") - )); + fn garbage_in_the_field_reads_as_never_set() { + for bad in [ + doc! { "warm_throughput": "not a record" }, + doc! { "warm_throughput": { "ReadUnitsPerSecond": "lots" } }, + ] { + assert_eq!(warm_record_from_doc(&bad), None, "{bad}"); + } } } diff --git a/crates/storage-postgres/src/table_helpers.rs b/crates/storage-postgres/src/table_helpers.rs index 31edd9052..6522086b3 100755 --- a/crates/storage-postgres/src/table_helpers.rs +++ b/crates/storage-postgres/src/table_helpers.rs @@ -247,7 +247,7 @@ impl PostgresEngine { &idx.index_name, ), warm_throughput: None, - warm_throughput_record: parse_warm_throughput_record(warm_json.as_deref())?, + warm_throughput_record: parse_warm_throughput_record(warm_json.as_deref()), }); } else { lsis.push(LsiDescription { @@ -367,7 +367,7 @@ impl PostgresEngine { .and_then(|v| serde_json::from_value(v).ok()), warm_throughput_record: parse_warm_throughput_record( row.warm_throughput.map(|v| v.to_string()).as_deref(), - )?, + ), // Vector indexes are read separately and applied by the caller: this // builder takes `index_rows` only, and the two callers differ in // whether they need them. `build_table_description` fills them in; diff --git a/crates/storage-postgres/src/update_table.rs b/crates/storage-postgres/src/update_table.rs index 9a3474414..0f16ffe1e 100755 --- a/crates/storage-postgres/src/update_table.rs +++ b/crates/storage-postgres/src/update_table.rs @@ -437,6 +437,9 @@ impl PostgresEngine { Self::store_warm_throughput_updates(&mut tx, account_id, &table_id, &input).await?; // Apply GSI updates (create/delete). + // The WarmThroughput record each created GSI was inserted with, so the + // restart after the build leaves a later Update's record alone. + let mut created_warm: Vec<(String, serde_json::Value)> = Vec::new(); let mut created_index_ids: Vec = Vec::new(); let mut deleted_index_ids: Vec = Vec::new(); // The merged and pruned attribute definitions persisted by this @@ -496,6 +499,9 @@ impl PostgresEngine { .execute(&mut *tx) .await .map_err(|e| StorageError::Internal(e.to_string()))?; + if let Some(warm) = gsi_warm { + created_warm.push((create.index_name.clone(), warm)); + } created_index_ids.push(index_id); // The data table is created before this transaction commits, and // the index is published CREATING; see the block before the @@ -1104,7 +1110,7 @@ impl PostgresEngine { // The build outlasts a new GSI's WarmThroughput UPDATING window; start // it again so the response reports it as the service does. - self.restart_created_gsi_warm_throughput(&table_id, &input) + self.restart_created_gsi_warm_throughput(&table_id, &created_warm) .await?; self.build_table_description(account_id, &input.table_name) diff --git a/crates/storage-postgres/src/warm_throughput.rs b/crates/storage-postgres/src/warm_throughput.rs index 8d43d398d..09bc53c55 100644 --- a/crates/storage-postgres/src/warm_throughput.rs +++ b/crates/storage-postgres/src/warm_throughput.rs @@ -8,11 +8,9 @@ //! pre-warms or scales on it, and it has no effect on throughput, throttling //! or storage. See `extenddb_core::types::WarmThroughputRecord`. -use extenddb_core::types::UpdateTableInput; +use extenddb_core::types::{UpdateTableInput, WarmThroughputRecord}; use extenddb_storage::error::StorageError; -use extenddb_storage::util::{ - warm_throughput_record_for_create, warm_throughput_record_for_update, -}; +use extenddb_storage::util::{warm_throughput_record_for_update, warm_throughput_updating_until}; use crate::PostgresEngine; @@ -74,7 +72,7 @@ impl PostgresEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))?; let existing = existing.map(|v| v.to_string()); - let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs)?; + let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs); sqlx::query( "UPDATE tables SET warm_throughput = $1 WHERE account_id = $2 AND table_name = $3", ) @@ -100,7 +98,7 @@ impl PostgresEngine { return Err(StorageError::IndexNotFound(index_name.clone())); }; let existing = existing.map(|v| v.to_string()); - let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs)?; + let record = warm_throughput_record_for_update(existing.as_deref(), spec, delay_secs); sqlx::query( "UPDATE indexes SET warm_throughput = $1 WHERE table_id = $2 AND index_name = $3", ) @@ -123,19 +121,16 @@ impl PostgresEngine { /// default 0.25 s window. Without this the response, which is described /// after the build, would report the new index's `WarmThroughput` `ACTIVE` /// where the service reports it `UPDATING`. + /// + /// `created` holds each index's name and the record it was inserted with. + /// The write is conditional on that record, so a `WarmThroughput` Update + /// of the index accepted while the build ran keeps its value. pub(crate) async fn restart_created_gsi_warm_throughput( &self, table_id: &str, - input: &UpdateTableInput, + created: &[(String, serde_json::Value)], ) -> Result<(), StorageError> { - let creates: Vec<_> = input - .global_secondary_index_updates - .iter() - .flatten() - .filter_map(|u| u.create.as_ref()) - .filter(|c| c.warm_throughput.is_some()) - .collect(); - if creates.is_empty() { + if created.is_empty() { return Ok(()); } let mut conn = self @@ -144,21 +139,23 @@ impl PostgresEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))?; let delay_secs = Self::control_plane_delay_secs(&mut conn).await?; - for create in creates { - let Some(record) = - warm_throughput_record_for_create(create.warm_throughput.as_ref(), delay_secs) + for (index_name, inserted) in created { + let Ok(mut record) = serde_json::from_value::(inserted.clone()) else { continue; }; + record.updating_until = warm_throughput_updating_until(delay_secs); let json = serde_json::to_value(record).map_err(|e| StorageError::Internal(e.to_string()))?; sqlx::query( "UPDATE indexes SET warm_throughput = $1 \ - WHERE table_id = $2 AND index_name = $3 AND index_type = 'GSI'", + WHERE table_id = $2 AND index_name = $3 AND index_type = 'GSI' \ + AND warm_throughput = $4", ) .bind(json) .bind(table_id) - .bind(&create.index_name) + .bind(index_name) + .bind(inserted) .execute(&mut *conn) .await .map_err(|e| StorageError::Internal(e.to_string()))?; diff --git a/crates/storage-sqlite/src/table_helpers.rs b/crates/storage-sqlite/src/table_helpers.rs index b2c9123a2..5ca4aabd3 100644 --- a/crates/storage-sqlite/src/table_helpers.rs +++ b/crates/storage-sqlite/src/table_helpers.rs @@ -236,7 +236,7 @@ impl SqliteEngine { warm_throughput: None, warm_throughput_record: parse_warm_throughput_record( idx.warm_throughput.as_deref(), - )?, + ), }); } else { lsis.push(LsiDescription { @@ -350,7 +350,7 @@ impl SqliteEngine { .on_demand_throughput .as_deref() .and_then(|s| serde_json::from_str(s).ok()), - warm_throughput_record: parse_warm_throughput_record(row.warm_throughput.as_deref())?, + warm_throughput_record: parse_warm_throughput_record(row.warm_throughput.as_deref()), // Any core field this backend does not populate takes its default, so // adding one does not break this build. ..Default::default() diff --git a/crates/storage/src/util/warm_throughput.rs b/crates/storage/src/util/warm_throughput.rs index 8050e0985..2a7694895 100644 --- a/crates/storage/src/util/warm_throughput.rs +++ b/crates/storage/src/util/warm_throughput.rs @@ -32,24 +32,16 @@ pub fn warm_throughput_record_for_create( } /// The record an update path stores over the stored JSON (`None` when the -/// resource never had one). -/// -/// # Errors -/// -/// `Internal` when the stored JSON does not parse. +/// resource never had one, or when it does not parse). +#[must_use] pub fn warm_throughput_record_for_update( existing_json: Option<&str>, spec: &WarmThroughput, delay_secs: f64, -) -> Result { - let existing = parse_warm_throughput_record(existing_json)?; +) -> WarmThroughputRecord { + let existing = parse_warm_throughput_record(existing_json); let now = epoch_seconds_now(); - Ok(WarmThroughputRecord::apply_update( - existing, - spec, - now + delay_secs.max(0.0), - now, - )) + WarmThroughputRecord::apply_update(existing, spec, now + delay_secs.max(0.0), now) } /// The stored JSON of a record. @@ -64,17 +56,23 @@ pub fn warm_throughput_record_json(record: &WarmThroughputRecord) -> Result, -) -> Result, StorageError> { - json.map(|text| { - serde_json::from_str(text) - .map_err(|e| StorageError::Internal(format!("warm throughput record: {e}"))) - }) - .transpose() +/// A value that does not parse as a record (a hand edit, or a record shape +/// this build does not know) is logged and read as never set, so the +/// resource reports the defaults for its billing mode, as the neighbouring +/// `on_demand_throughput` column does, and an `UpdateTable` that sets the +/// member writes a record over it. +#[must_use] +pub fn parse_warm_throughput_record(json: Option<&str>) -> Option { + let text = json?; + match serde_json::from_str(text) { + Ok(record) => Some(record), + Err(e) => { + tracing::warn!( + "stored warm_throughput does not parse as a record, reporting the defaults: {e}" + ); + None + } + } } #[cfg(test)] @@ -116,14 +114,12 @@ mod tests { write_units_per_second: Some(75), }, 0.0, - ) - .expect("merged"); + ); assert_eq!(updated.read_units_per_second, Some(100)); assert_eq!(updated.write_units_per_second, Some(75)); let again = parse_warm_throughput_record(Some( &warm_throughput_record_json(&updated).expect("json"), )) - .expect("parses") .expect("present"); assert_eq!( (again.read_units_per_second, again.write_units_per_second), @@ -138,11 +134,26 @@ mod tests { assert!((again.updating_until - updated.updating_until).abs() < 1e-6); } + /// A value that does not parse reads as never set (the defaults are + /// reported), and an update writes a whole record over it. #[test] - fn garbage_in_the_column_is_an_internal_error_not_a_default() { - let err = parse_warm_throughput_record(Some("{\"x\":")).expect_err("refused"); - assert!( - matches!(err, StorageError::Internal(m) if m.starts_with("warm throughput record:")) - ); + fn garbage_in_the_column_reads_as_never_set_and_an_update_replaces_it() { + for garbage in ["{\"x\":", "{\"ReadUnitsPerSecond\": \"lots\"}", "\"text\""] { + assert_eq!( + parse_warm_throughput_record(Some(garbage)), + None, + "{garbage}" + ); + let updated = warm_throughput_record_for_update( + Some(garbage), + &WarmThroughput { + read_units_per_second: Some(13_000), + write_units_per_second: None, + }, + 0.0, + ); + assert_eq!(updated.read_units_per_second, Some(13_000)); + assert_eq!(updated.write_units_per_second, None); + } } } diff --git a/tests/python/test_warm_throughput_updates.py b/tests/python/test_warm_throughput_updates.py index 3592a0bf3..7d9a71bdb 100644 --- a/tests/python/test_warm_throughput_updates.py +++ b/tests/python/test_warm_throughput_updates.py @@ -12,6 +12,8 @@ from __future__ import annotations +import threading + import pytest from botocore.exceptions import ClientError @@ -418,3 +420,67 @@ def test_never_set_warm_throughput_across_capacity_and_billing_changes( _wait_all_active(dynamodb_client, name) active = _wait_warm_active(dynamodb_client, name, timeout=900) assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} + + +class TestWarmThroughputConcurrentUpdates: + def test_two_updates_that_set_different_sides_both_keep_their_side( + self, dynamodb_client, tables, is_extenddb + ): + """Two UpdateTables sent together, one raising ReadUnitsPerSecond and one + WriteUnitsPerSecond, both answer 200, and the settled value carries both + sides, on the table and on a GSI, in each of ten rounds. ExtendDB only: an + increase on the service is billed and settles after minutes. Each backend + merges the request over the stored record; a merge that another request + overtook is retried instead of writing over that request's side.""" + if not is_extenddb: + pytest.skip("raises warm throughput twenty times; the service bills each increase") + name, _ = _create( + dynamodb_client, + tables, + BillingMode="PAY_PER_REQUEST", + GlobalSecondaryIndexes=[GSI], + ) + _wait_all_active(dynamodb_client, name) + index = GSI["IndexName"] + + def on_table(member, value): + return {"WarmThroughput": {member: value}} + + def on_index(member, value): + return { + "GlobalSecondaryIndexUpdates": [ + {"Update": {"IndexName": index, "WarmThroughput": {member: value}}} + ] + } + + for shape, reported in ( + (on_table, lambda t: t["WarmThroughput"]), + (on_index, lambda t: t["GlobalSecondaryIndexes"][0]["WarmThroughput"]), + ): + for i in range(1, 11): + read, write = 12000 + 100 * i, 4000 + 100 * i + barrier = threading.Barrier(2) + errors = [] + + def send(member, value, shape=shape, barrier=barrier, errors=errors): + barrier.wait() + try: + dynamodb_client.update_table(TableName=name, **shape(member, value)) + except ClientError as e: + errors.append(e.response["Error"]) + + senders = [ + threading.Thread(target=send, args=("ReadUnitsPerSecond", read)), + threading.Thread(target=send, args=("WriteUnitsPerSecond", write)), + ] + for sender in senders: + sender.start() + for sender in senders: + sender.join() + assert errors == [], (shape.__name__, i, errors) + settled = _wait_warm_active(dynamodb_client, name) + assert reported(settled) == { + "ReadUnitsPerSecond": read, + "WriteUnitsPerSecond": write, + "Status": "ACTIVE", + }, (shape.__name__, i) From 9fc0a41da299fd5ed553a0a28d2e6c1a5ba074eb Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:49:34 +0000 Subject: [PATCH 07/15] fix(engine): clear an imported index's WarmThroughput ImportTable drops the table-level member, since TableCreationParameters has none, but passed each GSI's WarmThroughput through, and ImportTable does not run CreateTable's validation, so a value CreateTable refuses (-7 read units) was stored and reported. The import now clears it, and the imported table and its indexes report the defaults. Running CreateTable's validation on the import's create input, which also lets a GSI ProvisionedThroughput of -3 through on main, is left to its own issue. --- crates/engine/src/import_export.rs | 49 +++++++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/crates/engine/src/import_export.rs b/crates/engine/src/import_export.rs index 7cf2eb51e..66f2ddb18 100755 --- a/crates/engine/src/import_export.rs +++ b/crates/engine/src/import_export.rs @@ -347,7 +347,18 @@ fn create_table_input_from_params(tcp: &TableCreationParameters) -> CreateTableI // member; nothing to resolve. table_throughput_mode: None, provisioned_throughput: tcp.provisioned_throughput.clone(), - global_secondary_indexes: tcp.global_secondary_indexes.clone(), + // TableCreationParameters has no table-level WarmThroughput, and an + // imported index's is cleared the same way: ImportTable does not run + // CreateTable's validation, so a value kept here would be stored + // unchecked. Imported tables and indexes report the defaults. + global_secondary_indexes: tcp.global_secondary_indexes.clone().map(|gsis| { + gsis.into_iter() + .map(|gsi| extenddb_core::types::GsiInput { + warm_throughput: None, + ..gsi + }) + .collect() + }), local_secondary_indexes: None, vector_indexes: None, stream_specification: None, @@ -459,3 +470,39 @@ mod export_arn_tests { } } } + +#[cfg(test)] +mod import_warm_throughput_tests { + use super::*; + + /// An imported index's WarmThroughput is not stored, as the table's is + /// not: a value CreateTable refuses (-7) cannot reach the catalog. + #[test] + fn an_imported_gsi_warm_throughput_is_cleared() { + let tcp: TableCreationParameters = serde_json::from_value(serde_json::json!({ + "TableName": "t", + "AttributeDefinitions": [ + { "AttributeName": "pk", "AttributeType": "S" }, + { "AttributeName": "g", "AttributeType": "S" } + ], + "KeySchema": [{ "AttributeName": "pk", "KeyType": "HASH" }], + "BillingMode": "PROVISIONED", + "ProvisionedThroughput": { "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 }, + "GlobalSecondaryIndexes": [{ + "IndexName": "gsi", + "KeySchema": [{ "AttributeName": "g", "KeyType": "HASH" }], + "Projection": { "ProjectionType": "ALL" }, + "ProvisionedThroughput": { "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 }, + "WarmThroughput": { "ReadUnitsPerSecond": -7, "WriteUnitsPerSecond": 0 } + }] + })) + .expect("parameters"); + let input = create_table_input_from_params(&tcp); + assert_eq!(input.warm_throughput, None); + let gsis = input.global_secondary_indexes.expect("the index is kept"); + assert_eq!(gsis.len(), 1); + assert_eq!(gsis[0].index_name, "gsi"); + assert_eq!(gsis[0].warm_throughput, None); + assert!(gsis[0].provisioned_throughput.is_some()); + } +} From cce43488c6b0b3b515f8dfc57ea79ca8e776e455 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:49:34 +0000 Subject: [PATCH 08/15] fix(devtools): leave the Terraform suite out of run-tests --all devtools/run-tests --extenddb --all is the documented local command (developer test guide, AGENTS.md, tests/README.md). With --terraform in --all it needed releases.hashicorp.com and registry.terraform.io and failed on macOS and arm64, since install-terraform.sh only fetches linux_amd64. CI passes --terraform explicitly, so --all now leaves it out and the help text says so. --- devtools/run-tests | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/devtools/run-tests b/devtools/run-tests index 17da66a7a..5552e8986 100755 --- a/devtools/run-tests +++ b/devtools/run-tests @@ -26,6 +26,7 @@ # --external External Java test suite # --comprehensive Comprehensive Python suite # --all all of the above +# --terraform Terraform apply of tests/terraform/warm-throughput (not in --all) # --catalog-check post-test catalog integrity check # # Options: @@ -90,8 +91,9 @@ Suites (at least one required): --external External Java test suite --comprehensive Comprehensive Python suite --terraform Terraform apply of tests/terraform/warm-throughput (installs - the pinned Terraform unless TERRAFORM names a binary) - --all all suites + the pinned linux_amd64 Terraform unless TERRAFORM names a + binary, and downloads the AWS provider; not part of --all) + --all all suites above except --terraform --catalog-check post-test catalog integrity check Options: @@ -119,7 +121,7 @@ while [[ $# -gt 0 ]]; do --external) RUN_EXTERNAL=true; shift ;; --comprehensive) RUN_COMPREHENSIVE=true; shift ;; --terraform) RUN_TERRAFORM=true; shift ;; - --all) RUN_RUST=true; RUN_RUST_INTEGRATION=true; RUN_PYTEST=true; RUN_EXTERNAL=true; RUN_COMPREHENSIVE=true; RUN_TERRAFORM=true; shift ;; + --all) RUN_RUST=true; RUN_RUST_INTEGRATION=true; RUN_PYTEST=true; RUN_EXTERNAL=true; RUN_COMPREHENSIVE=true; shift ;; --release) RELEASE=true; shift ;; --filter) FILTER="$2"; shift 2 ;; --catalog-check) RUN_CATALOG_CHECK=true; shift ;; From 5940f95dcd7711ef49fae53ebe78ce71b21ee7f6 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 12:23:54 +0000 Subject: [PATCH 09/15] test(mongodb): count migration 6's step in the two-migrator lock test Migration 6 does not apply on MongoDB, but the runner reports it as a step with the outcome NotApplicable and records it in the ledger, as migrations_fresh.rs expects. A migrator that starts from the 0.1.12 fixture therefore reports seven steps. The migration 6 commit moved this test's catalog version assertion to 6 and left its expected step count at six. --- crates/storage-mongodb/tests/migrations_locks.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/storage-mongodb/tests/migrations_locks.rs b/crates/storage-mongodb/tests/migrations_locks.rs index 86c2c9064..3f249d638 100644 --- a/crates/storage-mongodb/tests/migrations_locks.rs +++ b/crates/storage-mongodb/tests/migrations_locks.rs @@ -53,7 +53,7 @@ async fn two_migrators_started_together_apply_once_and_the_other_finds_nothing() applied.sort_unstable(); assert_eq!( applied, - [0, 6], + [0, 7], "one applies every step and the other finds nothing" ); assert_eq!(d.status().await, CatalogStatus::Current { version: 6 }); From 74cb23d9a5a855c1d93b323116330809de11d43a Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 12:32:22 +0000 Subject: [PATCH 10/15] docs(schema): regenerate the schema files for migration 6 Migration 6 adds a warm_throughput column to the tables and indexes catalogs, and the catalog jobs compare a fresh install with these files. Each file is the schema-dump output of a fresh init by this branch's binary: PostgreSQL lists the two columns as jsonb after the last column of each table, SQLite shows them as TEXT in the CREATE TABLE statements, and Cassandra lists them as text regular columns in name order. The dumps differ from the committed files by those two columns only. MongoDB's file lists collections and indexes, and migration 6 adds neither, so it is unchanged. --- docs/schema/cassandra.txt | 2 ++ docs/schema/postgres.txt | 2 ++ docs/schema/sqlite.txt | 4 ++-- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/schema/cassandra.txt b/docs/schema/cassandra.txt index bc7ee91cc..65cfc33af 100644 --- a/docs/schema/cassandra.txt +++ b/docs/schema/cassandra.txt @@ -144,6 +144,7 @@ column indexes.key_schema: text regular column indexes.projection: text regular column indexes.propagation_delay_ms: int regular column indexes.provisioned_throughput: text regular +column indexes.warm_throughput: text regular column login_attempts.principal: text partition_key 0 column login_attempts.attempted_at: timestamp clustering 0 desc column login_attempts.source_ip: text regular @@ -192,6 +193,7 @@ column tables.ttl_cleanup_generation: uuid regular column tables.ttl_generation: uuid regular column tables.ttl_index_ready: boolean regular column tables.ttl_sweep_owner: uuid regular +column tables.warm_throughput: text regular column tags.resource_arn: text partition_key 0 column tags.tag_key: text clustering 0 asc column tags.tag_value: text regular diff --git a/docs/schema/postgres.txt b/docs/schema/postgres.txt index cee98bf9b..0f86445a0 100644 --- a/docs/schema/postgres.txt +++ b/docs/schema/postgres.txt @@ -127,6 +127,7 @@ column indexes.projection: jsonb NOT NULL column indexes.index_status: text NOT NULL DEFAULT 'ACTIVE'::text column indexes.provisioned_throughput: jsonb column indexes.propagation_delay_ms: integer +column indexes.warm_throughput: jsonb column login_attempts.principal: text NOT NULL column login_attempts.attempted_at: timestamp with time zone NOT NULL DEFAULT now() column login_attempts.success: boolean NOT NULL @@ -181,6 +182,7 @@ column tables.ttl_index_ready: boolean NOT NULL DEFAULT false column tables.table_class: text column tables.sse_specification: jsonb column tables.on_demand_throughput: jsonb +column tables.warm_throughput: jsonb column tags.resource_arn: text NOT NULL column tags.tag_key: text NOT NULL column tags.tag_value: text NOT NULL diff --git a/docs/schema/sqlite.txt b/docs/schema/sqlite.txt index c74145d0f..974461efe 100644 --- a/docs/schema/sqlite.txt +++ b/docs/schema/sqlite.txt @@ -28,7 +28,7 @@ table iam_sessions on iam_sessions: CREATE TABLE iam_sessions ( session_token TE table iam_user_tags on iam_user_tags: CREATE TABLE iam_user_tags ( account_id TEXT NOT NULL, user_name TEXT NOT NULL, tag_key TEXT NOT NULL, tag_value TEXT NOT NULL, PRIMARY KEY (account_id, user_name, tag_key), FOREIGN KEY (account_id, user_name) REFERENCES iam_users(account_id, user_name) ON DELETE CASCADE ) table iam_users on iam_users: CREATE TABLE iam_users ( account_id TEXT NOT NULL REFERENCES accounts(account_id) ON DELETE CASCADE, user_name TEXT NOT NULL, user_arn TEXT NOT NULL UNIQUE, password_hash TEXT, created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), PRIMARY KEY (account_id, user_name) ) table idempotency_tokens on idempotency_tokens: CREATE TABLE idempotency_tokens ( account_id TEXT NOT NULL, token TEXT NOT NULL, fingerprint TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), PRIMARY KEY (account_id, token) ) -table indexes on indexes: CREATE TABLE indexes ( table_id TEXT NOT NULL, index_id TEXT NOT NULL, index_name TEXT NOT NULL, index_type TEXT NOT NULL, key_schema TEXT NOT NULL, projection TEXT NOT NULL, index_status TEXT NOT NULL DEFAULT 'ACTIVE', provisioned_throughput TEXT, propagation_delay_ms INTEGER, PRIMARY KEY (table_id, index_name), CONSTRAINT indexes_table_id_fkey FOREIGN KEY (table_id) REFERENCES tables(table_id) ON DELETE CASCADE, CONSTRAINT chk_propagation_delay_ms_non_negative CHECK (propagation_delay_ms IS NULL OR propagation_delay_ms >= 0) ) +table indexes on indexes: CREATE TABLE indexes ( table_id TEXT NOT NULL, index_id TEXT NOT NULL, index_name TEXT NOT NULL, index_type TEXT NOT NULL, key_schema TEXT NOT NULL, projection TEXT NOT NULL, index_status TEXT NOT NULL DEFAULT 'ACTIVE', provisioned_throughput TEXT, propagation_delay_ms INTEGER, warm_throughput TEXT, PRIMARY KEY (table_id, index_name), CONSTRAINT indexes_table_id_fkey FOREIGN KEY (table_id) REFERENCES tables(table_id) ON DELETE CASCADE, CONSTRAINT chk_propagation_delay_ms_non_negative CHECK (propagation_delay_ms IS NULL OR propagation_delay_ms >= 0) ) table login_attempts on login_attempts: CREATE TABLE login_attempts ( principal TEXT NOT NULL, attempted_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), success INTEGER NOT NULL, source_ip TEXT ) table metrics on metrics: CREATE TABLE metrics ( bucket TEXT NOT NULL, metric TEXT NOT NULL, table_name TEXT NOT NULL DEFAULT '', index_name TEXT NOT NULL DEFAULT '', operation TEXT NOT NULL DEFAULT '', sum REAL NOT NULL DEFAULT 0, count INTEGER NOT NULL DEFAULT 0, min REAL NOT NULL DEFAULT 9e999, max REAL NOT NULL DEFAULT -9e999, PRIMARY KEY (bucket, metric, table_name, index_name, operation) ) table schema_history on schema_history: CREATE TABLE schema_history ( filename TEXT PRIMARY KEY, applied_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ) @@ -37,6 +37,6 @@ table settings on settings: CREATE TABLE settings ( key TEXT PRIMARY KEY, value table stream_records on stream_records: CREATE TABLE stream_records ( shard_id TEXT NOT NULL REFERENCES stream_shards(shard_id) ON DELETE CASCADE, sequence_number TEXT NOT NULL, table_id TEXT NOT NULL, event_name TEXT NOT NULL, record_data TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), PRIMARY KEY (shard_id, sequence_number) ) table stream_shards on stream_shards: CREATE TABLE stream_shards ( shard_id TEXT PRIMARY KEY, table_id TEXT NOT NULL REFERENCES tables(table_id) ON DELETE CASCADE, parent_shard_id TEXT, starting_sequence_number TEXT NOT NULL, ending_sequence_number TEXT, created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')) ) table table_restores on table_restores: CREATE TABLE table_restores ( table_id TEXT PRIMARY KEY REFERENCES tables(table_id) ON DELETE CASCADE, source_backup_arn TEXT NOT NULL, source_table_arn TEXT, restore_date_time TEXT NOT NULL ) -table tables on tables: CREATE TABLE tables ( account_id TEXT NOT NULL REFERENCES accounts(account_id) ON DELETE CASCADE, table_name TEXT NOT NULL, key_schema TEXT NOT NULL, attribute_definitions TEXT NOT NULL, billing_mode TEXT NOT NULL DEFAULT 'PAY_PER_REQUEST', provisioned_throughput TEXT, stream_specification TEXT, table_status TEXT NOT NULL DEFAULT 'CREATING', creation_date_time TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), table_size_bytes INTEGER NOT NULL DEFAULT 0, item_count INTEGER NOT NULL DEFAULT 0, table_arn TEXT NOT NULL, table_id TEXT NOT NULL, ttl_attribute TEXT, deletion_protection_enabled INTEGER NOT NULL DEFAULT 0, status_transition_at TEXT, stream_label TEXT, ttl_index_ready INTEGER NOT NULL DEFAULT 0, table_class TEXT, sse_specification TEXT, on_demand_throughput TEXT, PRIMARY KEY (account_id, table_name), CONSTRAINT tables_table_id_unique UNIQUE (table_id) ) +table tables on tables: CREATE TABLE tables ( account_id TEXT NOT NULL REFERENCES accounts(account_id) ON DELETE CASCADE, table_name TEXT NOT NULL, key_schema TEXT NOT NULL, attribute_definitions TEXT NOT NULL, billing_mode TEXT NOT NULL DEFAULT 'PAY_PER_REQUEST', provisioned_throughput TEXT, stream_specification TEXT, table_status TEXT NOT NULL DEFAULT 'CREATING', creation_date_time TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), table_size_bytes INTEGER NOT NULL DEFAULT 0, item_count INTEGER NOT NULL DEFAULT 0, table_arn TEXT NOT NULL, table_id TEXT NOT NULL, ttl_attribute TEXT, deletion_protection_enabled INTEGER NOT NULL DEFAULT 0, status_transition_at TEXT, stream_label TEXT, ttl_index_ready INTEGER NOT NULL DEFAULT 0, table_class TEXT, sse_specification TEXT, on_demand_throughput TEXT, warm_throughput TEXT, PRIMARY KEY (account_id, table_name), CONSTRAINT tables_table_id_unique UNIQUE (table_id) ) table tags on tags: CREATE TABLE tags ( resource_arn TEXT NOT NULL, tag_key TEXT NOT NULL, tag_value TEXT NOT NULL, PRIMARY KEY (resource_arn, tag_key) ) table vector_indexes on vector_indexes: CREATE TABLE vector_indexes ( table_id TEXT NOT NULL, index_id TEXT NOT NULL, index_name TEXT NOT NULL, dimensions INTEGER NOT NULL, distance_function TEXT NOT NULL, vector_attribute TEXT NOT NULL, search_schema TEXT, projection TEXT NOT NULL, index_status TEXT NOT NULL DEFAULT 'CREATING', backfilling INTEGER, -- Items the backfill skipped because their stored bytes cannot enter the -- index (unparseable row, malformed or wrong-dimension vector). NULL until -- a backfill has completed; 0 afterwards when nothing was skipped. Kept so -- an operator can see that an ACTIVE index deliberately omits rows, rather -- than the build looping forever on them or dying part-way. skipped_item_count INTEGER, PRIMARY KEY (table_id, index_name), CONSTRAINT vector_indexes_table_id_fkey FOREIGN KEY (table_id) REFERENCES tables(table_id) ON DELETE CASCADE, CONSTRAINT chk_vector_dimensions_positive CHECK (dimensions > 0), CONSTRAINT chk_vector_backfilling_bool CHECK (backfilling IS NULL OR backfilling IN (0, 1)), -- An ACTIVE index must not carry the member at all, which is what the -- service does. Enforced here as well as in core, so a bug in the backend -- cannot persist a state the wire contract forbids. CONSTRAINT chk_vector_active_has_no_backfilling CHECK (index_status <> 'ACTIVE' OR backfilling IS NULL) ) From 0b2315aa21a6605341d664429166eb2cd0352b9b Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:47:37 +0000 Subject: [PATCH 11/15] test(engine): pin the billing mode a bare GSI Update is answered by update_table.rs chose the mode for refuse_bare_updates from the request's BillingMode when it carries one, else from the stored mode, but no test covered that choice: with it reverted to the stored mode only, every engine test still passed. The choice moves into gsi_updates::pay_per_request_after, and a unit test sends the switch to PROVISIONED the service requires on a PAY_PER_REQUEST table (BillingMode PROVISIONED, the table's and the index's ProvisionedThroughput 3/3), which gets "UpdateGlobalSecondaryIndex is not yet supported", and the same Update without BillingMode, which gets the on-demand sentence. With the request's mode ignored the test fails. Migration 6 is now WarmThroughput, so the worked example in CONTRIBUTING.md's "Adding a catalog migration" becomes migration 7, and the version and banner samples in the getting-started and quickstart guides read catalog migration 6, as this build prints. --- CONTRIBUTING.md | 40 +++++++-------- crates/engine/src/gsi_updates.rs | 60 ++++++++++++++++++++++- crates/engine/src/update_table.rs | 13 ++--- docs/getting-started.md | 4 +- docs/manuals/04-quickstart-setup-guide.md | 2 +- 5 files changed, 86 insertions(+), 33 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b0d641a98..421a28e08 100755 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -341,7 +341,7 @@ SQLite, PostgreSQL, Cassandra and MongoDB migrate through this framework. ### Example: a nullable column on `tables` -Migration 6, `table_note`, adds a nullable `note TEXT` column to the catalog's +Migration 7, `table_note`, adds a nullable `note TEXT` column to the catalog's `tables` table on SQLite and PostgreSQL. In `crates/storage/src/migrations/registry.rs`, at the end of the @@ -349,18 +349,18 @@ In `crates/storage/src/migrations/registry.rs`, at the end of the ```rust /// Adds a nullable `note` column to the `tables` catalog table. - TableNote = 6, "table_note", + TableNote = 7, "table_note", "add a nullable note column to the tables catalog"; ``` PostgreSQL has `ADD COLUMN IF NOT EXISTS`, so its body is a file, -`crates/storage-postgres/migrations/0006_table_note.sql`: +`crates/storage-postgres/migrations/0007_table_note.sql`: ```sql -- Copyright 2026 ExtendDB contributors -- SPDX-License-Identifier: Apache-2.0 -- --- Migration 6, table_note, PostgreSQL catalog database: a nullable note +-- Migration 7, table_note, PostgreSQL catalog database: a nullable note -- column on tables. IF NOT EXISTS makes a second run change nothing. ALTER TABLE tables ADD COLUMN IF NOT EXISTS note TEXT; @@ -375,7 +375,7 @@ In `crates/storage-postgres/src/migrations/mod.rs`, the arm in `migration()`: bodies: vec![file_body!( StoreKind::Catalog, "migrations", - "0006_table_note.sql" + "0007_table_note.sql" )], }, ``` @@ -393,13 +393,13 @@ and the id in the transactional arm of `runs_in()`: SQLite has no `ADD COLUMN IF NOT EXISTS`, and a file with a plain `ADD COLUMN` fails when it runs a second time, so the SQLite body is code that checks for the column first. -`crates/storage-sqlite/src/migrations/m0006_table_note.rs`: +`crates/storage-sqlite/src/migrations/m0007_table_note.rs`: ```rust // Copyright 2026 ExtendDB contributors // SPDX-License-Identifier: Apache-2.0 -//! Migration 6, `table_note`, SQLite: a nullable `note` column on `tables`. +//! Migration 7, `table_note`, SQLite: a nullable `note` column on `tables`. //! //! SQLite has no `ADD COLUMN IF NOT EXISTS`, so the body reads the table's //! columns first and adds the column only when it is missing. A second run @@ -411,10 +411,10 @@ use extenddb_storage::migrations::{CodeBody, Done, MigrationError}; use super::SqliteStores; /// This file, whose SHA-256 the checksum test pins. -pub(crate) const SOURCE: &str = include_str!("m0006_table_note.rs"); +pub(crate) const SOURCE: &str = include_str!("m0007_table_note.rs"); /// Recorded as this migration's checksum. Changes only with a new migration. -const FINGERPRINT: &str = "0006_table_note/sqlite/1"; +const FINGERPRINT: &str = "0007_table_note/sqlite/1"; pub(crate) struct TableNote; @@ -453,7 +453,7 @@ beside the others: ```rust #[rustfmt::skip] #[allow(clippy::all, deprecated)] -mod m0006_table_note; +mod m0007_table_note; ``` and the arm in `migration()`: @@ -464,8 +464,8 @@ and the arm in `migration()`: convergent: true, bodies: vec![StoreBody::code( StoreKind::Catalog, - &m0006_table_note::TableNote, - m0006_table_note::SOURCE, + &m0007_table_note::TableNote, + m0007_table_note::SOURCE, )], }, ``` @@ -477,8 +477,8 @@ rows of `tables`: ```rust #[tokio::test] -async fn migration_6_adds_the_note_column_to_tables() { - let db = temp_db("m6"); +async fn migration_7_adds_the_note_column_to_tables() { + let db = temp_db("m7"); load_dump(&db, DEV_0_1_14).await; run(&db).await.expect("migrate"); let mut text = dump( @@ -488,12 +488,12 @@ async fn migration_6_adds_the_note_column_to_tables() { ) .await; text.push_str(&dump(&db, "SELECT table_name, note FROM tables ORDER BY table_name").await); - golden("0006_table_note", &text); + golden("0007_table_note", &text); } ``` -`EXTENDDB_BLESS_EXPECTED=1 cargo test -p extenddb-storage-sqlite --lib migration_6` -writes `crates/storage-sqlite/tests/fixtures/golden/0006_table_note.txt`, one +`EXTENDDB_BLESS_EXPECTED=1 cargo test -p extenddb-storage-sqlite --lib migration_7` +writes `crates/storage-sqlite/tests/fixtures/golden/0007_table_note.txt`, one tab-separated line per row under each query: ``` @@ -508,15 +508,15 @@ The PostgreSQL body is a file, so it has no golden test. The example declares `Kind::Breaking`. A nullable column can be additive by the rule in step 4, but declaring it additive needs the old-binary job, which this example does not add. As a breaking migration it makes a 1.x binary -without migration 6 refuse a catalog that has it, and a rollback restores the +without migration 7 refuse a catalog that has it, and a rollback restores the backup. The same PR changes the code that reads and writes `tables.note` in each backend's catalog store, and commits the `docs/schema/sqlite.txt` and `docs/schema/postgres.txt` that the integration jobs regenerate. It changes no per-table DDL, because `tables` is a catalog table, and adds no `RELEASED` -entry. `extenddb migrate --list` then shows migration 6, and on a catalog at -migration 5 `extenddb migrate` lists migration 6 as pending and exits 3. +entry. `extenddb migrate --list` then shows migration 7, and on a catalog at +migration 6 `extenddb migrate` lists migration 7 as pending and exits 3. ### Running the migration tests diff --git a/crates/engine/src/gsi_updates.rs b/crates/engine/src/gsi_updates.rs index 955bd788f..e02d54647 100644 --- a/crates/engine/src/gsi_updates.rs +++ b/crates/engine/src/gsi_updates.rs @@ -36,7 +36,7 @@ //! given) a throughput of at least 1 unit each; a `Delete` has a valid name. use extenddb_core::error::DynamoDbError; -use extenddb_core::types::UpdateTableInput; +use extenddb_core::types::{BillingMode, BillingModeSummary, UpdateTableInput}; /// Row V1a and V1b, whole. pub const ONE_ACTION_PER_ELEMENT: &str = "One or more parameter values were invalid: Only one \ @@ -124,6 +124,20 @@ pub(crate) fn refuse_bare_updates( Ok(()) } +/// Whether the table is PAY_PER_REQUEST after `input`: the request's +/// `BillingMode` when it carries one, else the stored mode. A request that +/// leaves PAY_PER_REQUEST is not held to the on-demand rule of +/// [`refuse_bare_updates`]. +pub(crate) fn pay_per_request_after( + input: &UpdateTableInput, + stored: Option<&BillingModeSummary>, +) -> bool { + match input.billing_mode { + Some(mode) => mode == BillingMode::PayPerRequest, + None => stored.is_some_and(|s| s.billing_mode == BillingMode::PayPerRequest), + } +} + /// Apply the element rules to `input`. /// /// # Errors @@ -405,6 +419,50 @@ mod tests { } } + /// The mode an Update without WarmThroughput is answered by is the one the + /// table has after the request. On a PAY_PER_REQUEST table the switch to + /// PROVISIONED that the service requires (BillingMode PROVISIONED, the + /// table's and the index's ProvisionedThroughput 3/3) gets "not yet + /// supported"; the same Update without BillingMode gets the on-demand + /// sentence. + #[test] + fn the_billing_mode_after_the_request_decides_a_bare_updates_answer() { + let stored = BillingModeSummary { + billing_mode: BillingMode::PayPerRequest, + last_update_to_pay_per_request_date_time: None, + }; + let update = json!([{ "Update": { + "IndexName": "gsi", + "ProvisionedThroughput": { "ReadCapacityUnits": 3, "WriteCapacityUnits": 3 }, + }}]); + let switch: UpdateTableInput = serde_json::from_value(json!({ + "TableName": "t", + "BillingMode": "PROVISIONED", + "ProvisionedThroughput": { "ReadCapacityUnits": 3, "WriteCapacityUnits": 3 }, + "GlobalSecondaryIndexUpdates": update, + })) + .unwrap(); + assert!(!pay_per_request_after(&switch, Some(&stored))); + assert_eq!( + message(refuse_bare_updates( + &switch, + pay_per_request_after(&switch, Some(&stored)) + )), + "UpdateGlobalSecondaryIndex is not yet supported" + ); + let stay = input(update); + assert!(pay_per_request_after(&stay, Some(&stored))); + assert_eq!( + message(refuse_bare_updates( + &stay, + pay_per_request_after(&stay, Some(&stored)) + )), + "One or more parameter values were invalid: The only Updates for index: gsi when \ + TableThroughputMode is PAY_PER_REQUEST can be to OnDemandThroughput, WarmThroughput" + ); + assert!(!pay_per_request_after(&stay, None)); + } + /// Rows T3d and T3e: an Update with nothing but the index name on an /// on-demand table answers the measured sentence; on a provisioned table /// ExtendDB's own. diff --git a/crates/engine/src/update_table.rs b/crates/engine/src/update_table.rs index 576610c7a..0cd84d6e9 100755 --- a/crates/engine/src/update_table.rs +++ b/crates/engine/src/update_table.rs @@ -193,15 +193,10 @@ pub async fn handle_update_table( )?; } if bare_update { - // The mode the table has after the request: a request that leaves - // PAY_PER_REQUEST is not held to the on-demand rule. - let pay_per_request = match input.billing_mode { - Some(mode) => mode == BillingMode::PayPerRequest, - None => existing - .billing_mode_summary - .as_ref() - .is_some_and(|s| s.billing_mode == BillingMode::PayPerRequest), - }; + let pay_per_request = crate::gsi_updates::pay_per_request_after( + &input, + existing.billing_mode_summary.as_ref(), + ); crate::gsi_updates::refuse_bare_updates(&input, pay_per_request)?; } if touches_warm { diff --git a/docs/getting-started.md b/docs/getting-started.md index 1388b7b55..08d2725d3 100755 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -287,7 +287,7 @@ extenddb runs as a daemon (background process) and logs to syslog. On startup it ```bash ./target/release/extenddb serve --config extenddb.toml -# extenddb 0.1.14 (catalog migration 5) starting on 127.0.0.1:18443 +# extenddb 0.1.14 (catalog migration 6) starting on 127.0.0.1:18443 # storage: postgres (postgresql://extenddb:***@localhost:5432/extenddb_catalog) ``` @@ -1370,7 +1370,7 @@ Each runner requires its tools to be installed. The runner checks prerequisites ```bash ./target/release/extenddb version # extenddb 0.1.14 -# catalog migration 5 (postgres) +# catalog migration 6 (postgres) # commit abc1234 # built 2026-04-17T12:00:00Z ``` diff --git a/docs/manuals/04-quickstart-setup-guide.md b/docs/manuals/04-quickstart-setup-guide.md index 6575f9d58..08333d9dd 100755 --- a/docs/manuals/04-quickstart-setup-guide.md +++ b/docs/manuals/04-quickstart-setup-guide.md @@ -130,7 +130,7 @@ Check the version: ```bash ./target/release/extenddb version # extenddb 0.1.14 -# catalog migration 5 (postgres) +# catalog migration 6 (postgres) # commit abc1234 # built 2026-04-17T12:00:00Z ``` From af59c94dc139f0168013ee0096dbb53f2d8aaafe Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Tue, 3 Nov 2026 14:03:44 +0000 Subject: [PATCH 12/15] test(engine): call the billing mode loop variable mode CodeQL's rust/cleartext-logging rule reads a variable named `billing` as billing information and flagged the assertion message that prints it in the WarmThroughput-beside-a-throughput test. The variable holds a BillingMode value such as PROVISIONED; it is renamed to `mode`, and the test is unchanged otherwise. --- crates/engine/src/gsi_updates.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/crates/engine/src/gsi_updates.rs b/crates/engine/src/gsi_updates.rs index e02d54647..cd23914a2 100644 --- a/crates/engine/src/gsi_updates.rs +++ b/crates/engine/src/gsi_updates.rs @@ -353,14 +353,14 @@ mod tests { .as_object_mut() .unwrap() .extend(members.as_object().unwrap().clone()); - for billing in [None, Some("PROVISIONED"), Some("PAY_PER_REQUEST")] { + for mode in [None, Some("PROVISIONED"), Some("PAY_PER_REQUEST")] { let mut request = input(json!([{ "Update": update }])); - request.billing_mode = billing.map(|b| serde_json::from_value(json!(b)).unwrap()); + request.billing_mode = mode.map(|m| serde_json::from_value(json!(m)).unwrap()); assert_eq!( message(validate_gsi_updates(&request)), "One or more parameter values were invalid: WarmThroughput must be the only \ operation in the request for index: gsi", - "{update} with BillingMode {billing:?}" + "{update} with BillingMode {mode:?}" ); } } From 8c35832e0f3d48379b1e9a439469226ea5a23259 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Wed, 4 Nov 2026 22:47:48 +0000 Subject: [PATCH 13/15] test(storage-cassandra): set the warm throughput fields in the delete backfill test The test that main added in #424 builds an UpdateTableInput and a CreateGsiAction, which this branch extends with warm_throughput and sse_specification. Signed-off-by: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> --- crates/storage-cassandra/tests/direct/index.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/crates/storage-cassandra/tests/direct/index.rs b/crates/storage-cassandra/tests/direct/index.rs index c66c9283a..8100bd5b7 100644 --- a/crates/storage-cassandra/tests/direct/index.rs +++ b/crates/storage-cassandra/tests/direct/index.rs @@ -1880,6 +1880,8 @@ async fn test_a_delete_removes_the_index_row_a_backfill_wrote() { .update_table( &table.key_info.account_id, UpdateTableInput { + sse_specification: None, + warm_throughput: None, table_name: table.key_info.table_name.clone(), billing_mode: None, table_throughput_mode: None, @@ -1897,6 +1899,7 @@ async fn test_a_delete_removes_the_index_row_a_backfill_wrote() { non_key_attributes: None, }, provisioned_throughput: None, + warm_throughput: None, }), update: None, delete: None, From d8879f9f0d59554b177f4b42d3145318b5e868b6 Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Fri, 6 Nov 2026 16:37:50 +0000 Subject: [PATCH 14/15] test: skip the warm throughput restore cells on a server with a backup store Since #700, the PostgreSQL and SQLite jobs run the Python suite against a server with a filesystem backup store. CreateBackup there writes to the store, and RestoreTableFromBackup refuses such a backup until the restore job lands. S6c and S6d restore the backup, so they now skip on that server, as the restore tests on main do. S6b still runs on every server. Signed-off-by: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> --- tests/python/test_warm_throughput_backup.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tests/python/test_warm_throughput_backup.py b/tests/python/test_warm_throughput_backup.py index 40ebbe9cd..2fbb33010 100644 --- a/tests/python/test_warm_throughput_backup.py +++ b/tests/python/test_warm_throughput_backup.py @@ -9,6 +9,7 @@ from __future__ import annotations +import os import time import pytest @@ -80,6 +81,20 @@ def test_a_restored_table_reports_the_defaults_and_a_gsi_override_cannot_set_it( for gsi in desc.get("SourceTableFeatureDetails", {}).get("GlobalSecondaryIndexes", []): assert "WarmThroughput" not in gsi + # S6c and S6d restore the backup. Until the restore job lands, a server + # with a backup store refuses that restore with ValidationException. + # Its message is "is in a backup store; this release restores + # in-catalog backups only". So S6c and S6d skip there, and S6b above + # runs on every server. + if ( + os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() in ("filesystem", "s3") + and os.environ.get("EXTENDDB_VALIDATION_MODE", "").lower() != "true" + ): + pytest.skip( + "RestoreTableFromBackup of a backup in the configured store is refused " + "until the restore job lands (PR 9a)" + ) + # S6c: no member while CREATING; the defaults once ACTIVE, on the table # and the index, although the source had set larger values. target = unique_name("wt-restore") From 4a1926fdb3347059aaea32d7a4981fc8b985ae7b Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Fri, 6 Nov 2026 17:14:28 +0000 Subject: [PATCH 15/15] test: accept the end of a 50 ms warm throughput window on ExtendDB devtools/run-tests sets control_plane_delay_seconds to 0.05, so the UPDATING window that a WarmThroughput change opens lasts 50 ms on ExtendDB. The server reads its clock when it writes the change and again when it builds the response. If 50 ms or more pass between the two reads, the response reports the new pair as ACTIVE. The MongoDB job of d8879f9f0 failed this way in U3, which answered 200 with the settled 24000/8000 pair and ACTIVE. Two helpers now check the responses of U1, U2, S2c, U6, S8a and T7, and of the index update beside a table update. Against the service they assert what the test asserted before. On ExtendDB they also accept the new pair with ACTIVE when the call took at least 50 ms. U3 also accepts the S2b refusal and the higher pair with UPDATING, because the window of U2 can end at any point of U3. U3b still checks the settled pair. Signed-off-by: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> --- tests/python/test_warm_throughput.py | 75 ++++++++++++++------ tests/python/test_warm_throughput_updates.py | 16 +++-- tests/python/warm_throughput_helpers.py | 32 +++++++++ 3 files changed, 94 insertions(+), 29 deletions(-) diff --git a/tests/python/test_warm_throughput.py b/tests/python/test_warm_throughput.py index b2024db47..1bf4d6703 100644 --- a/tests/python/test_warm_throughput.py +++ b/tests/python/test_warm_throughput.py @@ -28,8 +28,11 @@ ON_DEMAND_NEEDS_BILLING, ONLY_OPERATION, TABLE_CLASS_ONLY, + _assert_opened, + _assert_opened_status, _create, _sentence, + _update_table, _validation, _wait_all_active, _wait_warm_active, @@ -164,26 +167,49 @@ def test_update_warm_throughput_goes_updating_then_active( _wait_all_active(dynamodb_client, name) # U1: the same values as current, and nothing else in the request. - resp = dynamodb_client.update_table(TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS) - t = resp["TableDescription"] + t, took = _update_table(dynamodb_client, TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS) assert t["TableStatus"] == "ACTIVE" - assert t["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + _assert_opened( + t["WarmThroughput"], ON_DEMAND_DEFAULTS, ON_DEMAND_DEFAULTS, is_extenddb, took + ) if is_extenddb: _wait_warm_active(dynamodb_client, name) # U2 and S2a: higher values; the response carries the previous pair. higher = {"ReadUnitsPerSecond": 24000, "WriteUnitsPerSecond": 8000} - t = dynamodb_client.update_table(TableName=name, WarmThroughput=higher)["TableDescription"] + t, took = _update_table(dynamodb_client, TableName=name, WarmThroughput=higher) assert t["TableStatus"] == "ACTIVE" - assert t["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + _assert_opened(t["WarmThroughput"], ON_DEMAND_DEFAULTS, higher, is_extenddb, took) described = dynamodb_client.describe_table(TableName=name)["Table"]["WarmThroughput"] if described["Status"] == "UPDATING": assert described == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} # U3: inside the window the previous values are not a decrease. - t = dynamodb_client.update_table(TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS)[ - "TableDescription" - ] - assert t["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} + # On ExtendDB the window of U2 lasts EXTENDDB_WINDOW_SECONDS and + # can end at any point of this request. If it ends before the + # check, the request is a decrease (S2b). If it ends between the + # check and the write, the response has the higher pair, UPDATING + # or ACTIVE. If it ends after the write, the response has the + # previous pair UPDATING, or the higher pair ACTIVE once the + # window of U3 ends. U3b checks the settled pair in each case. + try: + t = dynamodb_client.update_table( + TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS + )["TableDescription"] + except ClientError as e: + if not is_extenddb: + raise + err = e.response["Error"] + assert err["Code"] == "ValidationException", err + assert err["Message"] == _sentence("ReadUnitsPerSecond", "table", DECREASE) + else: + warm = t["WarmThroughput"] + if is_extenddb and warm["ReadUnitsPerSecond"] == higher["ReadUnitsPerSecond"]: + assert warm in ( + {**higher, "Status": "UPDATING"}, + {**higher, "Status": "ACTIVE"}, + ) + else: + assert warm == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} # U3b: the table settles at the higher values, whether or not U3 ran. if is_extenddb: active = _wait_warm_active(dynamodb_client, name) @@ -194,10 +220,8 @@ def test_update_warm_throughput_goes_updating_then_active( dynamodb_client.update_table(TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS) _validation(exc, _sentence("ReadUnitsPerSecond", "table", DECREASE)) # S2c: the same values re-sent report themselves UPDATING. - t = dynamodb_client.update_table(TableName=name, WarmThroughput=higher)[ - "TableDescription" - ] - assert t["WarmThroughput"] == {**higher, "Status": "UPDATING"} + t, took = _update_table(dynamodb_client, TableName=name, WarmThroughput=higher) + _assert_opened(t["WarmThroughput"], higher, higher, is_extenddb, took) _wait_warm_active(dynamodb_client, name) # U5: read alone; the write side keeps its value. @@ -211,16 +235,19 @@ def test_update_warm_throughput_goes_updating_then_active( # U6: a GSI through GlobalSecondaryIndexUpdates.Update (same values, so # the previous and new pairs agree on both targets). - resp = dynamodb_client.update_table( + t, took = _update_table( + dynamodb_client, TableName=name, GlobalSecondaryIndexUpdates=[ {"Update": {"IndexName": "gsi", "WarmThroughput": ON_DEMAND_DEFAULTS}} ], ) - gsi = resp["TableDescription"]["GlobalSecondaryIndexes"][0] + gsi = t["GlobalSecondaryIndexes"][0] assert gsi["IndexStatus"] == "ACTIVE" - assert gsi["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} - assert resp["TableDescription"]["TableStatus"] == "ACTIVE" + _assert_opened( + gsi["WarmThroughput"], ON_DEMAND_DEFAULTS, ON_DEMAND_DEFAULTS, is_extenddb, took + ) + assert t["TableStatus"] == "ACTIVE" if is_extenddb: active = _wait_warm_active(dynamodb_client, name) assert active["GlobalSecondaryIndexes"][0]["WarmThroughput"] == { @@ -403,7 +430,9 @@ def test_gsi_update_below_current_names_the_index(self, dynamodb_client, tables) ) _validation(exc, _sentence(member, "index gsi", DECREASE)) - def test_table_warm_throughput_beside_another_change(self, dynamodb_client, tables): + def test_table_warm_throughput_beside_another_change( + self, dynamodb_client, tables, is_extenddb + ): """The per-companion rules, measured on an on-demand table with one GSI (values otherwise valid): TableClass and DeletionProtection companions answer their own "must be the only operation" sentence; a @@ -435,14 +464,16 @@ def test_table_warm_throughput_beside_another_change(self, dynamodb_client, tabl ) _validation(exc, expected) - resp = dynamodb_client.update_table( + t, took = _update_table( + dynamodb_client, TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS, GlobalSecondaryIndexUpdates=[ {"Update": {"IndexName": "gsi", "WarmThroughput": ON_DEMAND_DEFAULTS}} ], ) - t = resp["TableDescription"] assert t["TableStatus"] == "ACTIVE" - assert t["WarmThroughput"]["Status"] == "UPDATING" - assert t["GlobalSecondaryIndexes"][0]["WarmThroughput"]["Status"] == "UPDATING" + _assert_opened_status(t["WarmThroughput"]["Status"], is_extenddb, took) + _assert_opened_status( + t["GlobalSecondaryIndexes"][0]["WarmThroughput"]["Status"], is_extenddb, took + ) diff --git a/tests/python/test_warm_throughput_updates.py b/tests/python/test_warm_throughput_updates.py index 7d9a71bdb..b804e71a1 100644 --- a/tests/python/test_warm_throughput_updates.py +++ b/tests/python/test_warm_throughput_updates.py @@ -28,8 +28,10 @@ ON_DEMAND_NEEDS_BILLING, ONLY_OPERATION, TABLE_CLASS_ONLY, + _assert_opened, _create, _sentence, + _update_table, _validation, _wait_all_active, _wait_warm_active, @@ -284,14 +286,14 @@ def test_gsi_create_with_warm_throughput_in_update_table( immediate), so those two statuses are asserted on the service only.""" name = _on_demand_with_gsi(dynamodb_client, tables) sent = {"ReadUnitsPerSecond": 30000, "WriteUnitsPerSecond": 8000} - resp = dynamodb_client.update_table( + t, took = _update_table( + dynamodb_client, TableName=name, AttributeDefinitions=ATTRS, GlobalSecondaryIndexUpdates=[{"Create": {**GSI_TWO, "WarmThroughput": sent}}], ) - t = resp["TableDescription"] by_name = {g["IndexName"]: g for g in t["GlobalSecondaryIndexes"]} - assert by_name["gsi-two"]["WarmThroughput"] == {**sent, "Status": "UPDATING"} + _assert_opened(by_name["gsi-two"]["WarmThroughput"], sent, sent, is_extenddb, took) assert by_name["gsi"]["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} if not is_extenddb: assert t["TableStatus"] == "UPDATING" @@ -365,10 +367,10 @@ def test_switch_to_on_demand_raises_a_low_warm_throughput( _wait_all_active(dynamodb_client, name) active = _wait_warm_active(dynamodb_client, name, timeout=900) assert active["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "ACTIVE"} - resp = dynamodb_client.update_table(TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS) - assert resp["TableDescription"]["WarmThroughput"] == { - **ON_DEMAND_DEFAULTS, "Status": "UPDATING" - } + t, took = _update_table(dynamodb_client, TableName=name, WarmThroughput=ON_DEMAND_DEFAULTS) + _assert_opened( + t["WarmThroughput"], ON_DEMAND_DEFAULTS, ON_DEMAND_DEFAULTS, is_extenddb, took + ) def test_never_set_warm_throughput_across_capacity_and_billing_changes( self, dynamodb_client, tables, is_extenddb diff --git a/tests/python/warm_throughput_helpers.py b/tests/python/warm_throughput_helpers.py index bd9204b4a..300e7478b 100644 --- a/tests/python/warm_throughput_helpers.py +++ b/tests/python/warm_throughput_helpers.py @@ -95,6 +95,38 @@ def _validation(exc_info, expected: str) -> None: assert err["Message"] == expected, err["Message"] +# devtools/run-tests sets control_plane_delay_seconds to 0.05. On ExtendDB, +# the UPDATING window that a request opens then lasts 50 milliseconds. If the +# server takes longer to build its response, the window is over, and the +# response reports the new pair ACTIVE. A call that returns sooner cannot see +# the window end. +EXTENDDB_WINDOW_SECONDS = 0.05 + + +def _update_table(client, **kwargs) -> tuple[dict, float]: + """UpdateTable: the TableDescription, and the seconds the call took.""" + start = time.monotonic() + resp = client.update_table(**kwargs) + return resp["TableDescription"], time.monotonic() - start + + +def _assert_opened(warm: dict, previous: dict, new: dict, is_extenddb: bool, took: float) -> None: + """The WarmThroughput of the response to a request that opens a window: + the previous pair with UPDATING, as the service answers. On ExtendDB, a + call that took at least EXTENDDB_WINDOW_SECONDS can also answer the new + pair with ACTIVE.""" + if is_extenddb and took >= EXTENDDB_WINDOW_SECONDS and warm == {**new, "Status": "ACTIVE"}: + return + assert warm == {**previous, "Status": "UPDATING"} + + +def _assert_opened_status(status: str, is_extenddb: bool, took: float) -> None: + """The Status that `_assert_opened` allows, for a check of the status alone.""" + if is_extenddb and took >= EXTENDDB_WINDOW_SECONDS and status == "ACTIVE": + return + assert status == "UPDATING" + + @pytest.fixture() def tables(dynamodb_client): """Register created table names for deletion after the test."""