Repository navigation
feat: store and report WarmThroughput on tables and global secondary indexes - #474
LeeroyHannigan wants to merge 16 commits into
Conversation
…indexes Since Terraform AWS provider 6.13.0 the aws_dynamodb_table resource waits for WarmThroughput.Status to read ACTIVE after every table and index create or update and treats a description without the member as not found, so an apply against ExtendDB polled until it timed out (#462). ExtendDB now accepts WarmThroughput on CreateTable, UpdateTable and GlobalSecondaryIndexUpdates (Create and Update), stores what the client sent, and reports the member with the shape, defaults, lifecycle and validation measured on Amazon DynamoDB on 2026-10-23: * Absent while the table or index is CREATING unless the request named it (then echoed as sent with Status UPDATING); 12000 read and 4000 write units on an on-demand table and its indexes, the provisioned units on a provisioned table and index; present with Status on every CreateTable, DescribeTable, UpdateTable and DeleteTable description. * An update reports the previous values with Status UPDATING for the control plane delay, then the new values ACTIVE; a decrease is validated against the reported values. A switch to PAY_PER_REQUEST raises a stored value to the on-demand minimum. A capacity change on a never-set provisioned table moves the reported default with the units. * Refusals carry the service's sentences: below the on-demand minimum, below the provisioned units, below the current value, each naming "table" or "index <name>"; one action per GlobalSecondaryIndexUpdate element; one update per index; on an on-demand table an index Update may only carry OnDemandThroughput or WarmThroughput. * UpdateTable with a table-level WarmThroughput refuses companions in the measured order: Server-Side Encryption, TableClass, DeletionProtection (each with its own sentence), OnDemandThroughput (needs BillingMode), a GlobalSecondaryIndexUpdates Delete, then BillingMode, ProvisionedThroughput, StreamSpecification or vector index updates ("WarmThroughput must be the only operation in the request"). GlobalSecondaryIndexUpdates Update beside it is accepted. A request carrying only WarmThroughput is accepted. SSESpecification is now read on UpdateTableInput so it can be refused rather than dropped. * A restored table starts with the defaults; a GlobalSecondaryIndexOverride carrying WarmThroughput is refused with the service's sentence. Nothing acts on the values. There is no pre-warming and no scaling at the database level, and the stored numbers have no effect on throughput, throttling or storage; they exist so clients that read the member work. The account maximum the service enforces is not enforced here. Storage: one nullable warm_throughput column on tables and indexes on SQLite (schema plus shape convergence), PostgreSQL (001_schema.sql plus shape steps for existing catalogs) and Cassandra (migration V006, catalog version 0.0.5 so extenddb migrate applies it); a document field on MongoDB. Each backend stores one JSON record (the client's sides, the previous sides, and the UPDATING deadline); the wire member is derived in one place, TableDescription::prepare_for_response. On PostgreSQL, UpdateTable builds a new GSI before it answers (#401), which takes longer than the 0.25 s UPDATING window the index row is inserted with. The window of each GSI the request created with WarmThroughput starts again once the build has run, so the response reports it UPDATING as the service does. Tests: unit tests for the record, defaults and every rule; three dual-target wire files (lifecycle and validation, update shapes and billing changes, backup and restore) covering every measured row; a Terraform apply of the configuration from the provider issue plus a table with a GSI on every backend, with Terraform 1.17.0 and provider 6.70.0 pinned and checksum-asserted in CI; a Cassandra test for a catalog that has not run V006. Measured on live DynamoDB: the three wire files pass against the service (12, 13 and 1 tests) and against SQLite, PostgreSQL, MongoDB and Cassandra. Cassandra lib (61) and direct_integration (151) suites pass. Negative controls: with the member removed the Terraform apply fails with the provider's "couldn't find resource (21 retries)" after about three and a half minutes, which is the hang from the issue; with the validation removed the three refusal rows fail; five further built mutants (table member, index member, UPDATING deadline, only-operation rule, swapped defaults) each fail the wire file. Documented in docs/differences-from-dynamodb.md (Capacity and Throttling) and docs/getting-started.md. Closes #462.
3b40060 to
0af920c
Compare
|
Rebased on main at abf8082; conflicts in crates/storage-postgres/src/update_table.rs, crates/storage-sqlite/src/update_table.rs and crates/storage/src/util/mod.rs, resolved by keeping #401's GSI build flow (the new index is inserted CREATING) with this branch's warm_throughput column and bind, and both sides' imports and util exports. The branch's commit also returned six files from #214 to their state before it: it deleted docs/rfcs/0003-backend-acceptance-criteria.md and removed the links to it from CONTRIBUTING.md, README.md, docs/design/04-component-storage.md, docs/rfcs/0002-backend-plugin-policy.md and docs/rfcs/README.md. Each of those files was byte-identical to its version before #214, and neither the commit message nor this PR mentions them, so they now match main and this PR no longer touches them. On PostgreSQL, UpdateTable now builds the new GSI before it answers (#401), which takes longer than the 0.25 s UPDATING window, so test_gsi_create_with_warm_throughput_in_update_table failed with Status ACTIVE in the response. restart_created_gsi_warm_throughput in crates/storage-postgres/src/warm_throughput.rs starts that window again after the build. A DescribeTable during the build reads IndexStatus CREATING with WarmThroughput UPDATING; the response to UpdateTable then starts the window again, so the ACTIVE flip of the index and the UPDATING window of its WarmThroughput are observed in that order. tests/python/test_warm_throughput*.py (26) and tests/test_gsi_online_backfill.py pass on PostgreSQL. |
|
@robinnsc, this one needs your call on the backup format before I rebase it. #474 adds No release tag has #394 yet, so I see three ways: add |
|
Thanks for checking first. I'd go with the second option, a GSI type of the manifest's own, done in this PR as part of the rebase. My reasoning is that WarmThroughput isn't something a backup should record. The service's own record of a backed-up index, The underlying problem is that v1 reuses So in this PR that would be the two types next to |
robinnsc
left a comment
robinnsc
left a comment
There was a problem hiding this comment.
We used three separate reviewers on this one (API behaviour and the engine, storage and upgrades across the four backends, and a verifier that re-ran every finding against the base). For clients that never send WarmThroughput nothing that worked at the base is refused: 46 UpdateTable shapes without the member give the same results on head and base SQLite and on head PostgreSQL apart from the new measured refusal sentences, and the existing UpdateTable, GSI, stream and TTL suites match. The description's rows reproduce on SQLite and PostgreSQL, the three wire files pass (26 each), catalogs from the 0.1.14 SQLite, PostgreSQL and MongoDB images and from a base-binary Cassandra catalog upgrade with the right defaults and their backups restore, nothing can stay UPDATING after a restart since the status is derived at read time, and the Terraform check is real: the base binary fails it with the provider's "couldn't find resource (21 retries)" in 195 s and the head passes in 48 s.
The ones I'd hold this for are the first inline comment and the rebase below: a GSI Update that carries WarmThroughput now drops a ProvisionedThroughput or OnDemandThroughput beside it with a 200, where base and main refuse it. The backup-manifest question is answered in the conversation.
The main thing before this can merge is the rebase onto main's migration framework (#666 and #681), which replaced all three upgrade mechanisms the PR uses. On main serve refuses a catalog with pending migrations, so the PostgreSQL shape steps applied at serve time, SQLite's shape convergence in schema.rs and bootstrapper.rs, and Cassandra's V006 with catalog version 0.0.5 all have to become migration 6 through crates/storage/src/migrations/registry.rs. CONTRIBUTING.md's "Adding a catalog migration" walks through almost exactly this change as its worked example (migration 6, table_note, a nullable column on tables). I think that means a WarmThroughput = 6 entry in the registry with RELEASED_THROUGH left at 0; on PostgreSQL a file body migrations/0006_warm_throughput.sql with ADD COLUMN IF NOT EXISTS on tables and indexes, the arm in migration() and the id in the transactional arm of runs_in(); on SQLite a code body src/migrations/m0006_warm_throughput.rs that checks pragma_table_info first, as lacks_warm_throughput already does, with its golden test and tests/fixtures/golden/0006_warm_throughput.txt; on Cassandra migrations/catalog/0006_warm_throughput.cql with the two ALTER TABLE ... ADD IF NOT EXISTS and no catalog_version insert (step 3 says a body must not name it), dropping V006, the CATALOG_VERSION = "0.0.5" bump and the unmigrated-catalog tolerance, which includes the arm at update_table.rs:173 that today also swallows any read or parse error when the request carries a BillingMode; and on MongoDB a Migration::NotApplicable arm, since the catalog collections have no validator. All of them Kind::Breaking and convergent: true, since additive needs the old-binary job that doesn't exist yet. docs/schema/{sqlite,postgres,cassandra}.txt need regenerating (the integration jobs diff them), and _as_released_catalog in tests/test_resource_tag_migration.py, which clears the ledger of a fresh SQLite catalog, will need to drop the new columns too or migration 1's post-check refuses them.
One trap worth knowing about: merging this branch into main does not report a conflict on the PostgreSQL schema. Rename detection maps the edit of 001_schema.sql onto main's 0001_baseline.sql, and git merge-tree --write-tree origin/main prints "Auto-merging crates/storage-postgres/migrations/0001_baseline.sql" and adds both warm_throughput JSONB columns to migration 1. Migration 1 is a file body checksummed by its contents, so every catalog main has already migrated would report Changed (exit 5), and a fresh init would fail migration 1's post-check with "column tables.warm_throughput is not part of the v1.0 schema". That hunk needs reverting by hand, along with the CREATE TABLE edit in SQLite's schema.rs, which main no longer has.
The upgrade manual's new "Catalog 0.0.5 (Cassandra only)" section also collides with main, where "Catalog 0.0.5" and V006__drop_continuous_backups.cql already mean the continuous-backups drop that became migration 5. I think the section goes away and the "Current Status" line becomes "migrations 1 to 6" with a short note for 6, including that since 6 is Breaking, going back to a release without it means restoring the pre-upgrade backup.
| .iter() | ||
| .flatten() | ||
| .filter_map(|u| u.update.as_ref()) | ||
| .find(|a| a.warm_throughput.is_none()) |
There was a problem hiding this comment.
An Update that carries WarmThroughput next to ProvisionedThroughput or OnDemandThroughput now succeeds and quietly drops the other member, because UpdateGsiAction only deserializes IndexName and WarmThroughput and this check only looks at warm_throughput.is_none(). On the head SQLite and PostgreSQL builds, Update {IndexName: gsi, ProvisionedThroughput: 9/9, WarmThroughput: {ReadUnitsPerSecond: 10}} on a provisioned 5/5 table answers 200 and the index stays at 5/5, while the same Update without WarmThroughput is refused with "UpdateGlobalSecondaryIndex is not yet supported", which is also what the base and main answer to both. On an on-demand table it gets round the rule in on_demand_update_members: ProvisionedThroughput alone is refused, ProvisionedThroughput plus WarmThroughput is accepted, and an OnDemandThroughput beside WarmThroughput is accepted and not stored. The worst case I found is the request DynamoDB needs to switch an on-demand table with a GSI to PROVISIONED (BillingMode, ProvisionedThroughput 3/3 and the GSI's ProvisionedThroughput 3/3) with a warm value added to the GSI Update: it answers 200 and leaves a provisioned 3/3 table whose index is 0/0 and reports WarmThroughput {12000, 0}. I think UpdateGsiAction needs to read ProvisionedThroughput and OnDemandThroughput, even as opaque values, so that an Update carrying either is still refused whether or not WarmThroughput is present, and then the doc comment at table.rs:1144 ("recognized by the deserializer but not yet implemented") is true again. A unit test per billing mode with both members in one Update would pin it.
There was a problem hiding this comment.
Fixed in 3546b08. UpdateGsiAction now reads ProvisionedThroughput and OnDemandThroughput as opaque values.
I measured the combinations on DynamoDB first (us-east-1, 2026-11-03). An Update carrying WarmThroughput beside OnDemandThroughput on an on-demand table, beside ProvisionedThroughput or OnDemandThroughput on a 1/1 provisioned table, and your switch shape (BillingMode PROVISIONED, ProvisionedThroughput 3/3, the GSI Update with ProvisionedThroughput 3/3 and a WarmThroughput) were all refused with "One or more parameter values were invalid: WarmThroughput must be the only operation in the request for index: gsi", and both tables were unchanged afterwards.
ExtendDB now answers that sentence in either billing mode, from validate_gsi_updates, before the table is read. An Update without WarmThroughput keeps main's answers (see the reply on update_table.rs:199). a_throughput_beside_warm_throughput_is_refused_in_either_billing_mode sends ProvisionedThroughput, OnDemandThroughput and both, with no BillingMode, PROVISIONED and PAY_PER_REQUEST. With the check taken out of validate_gsi_updates, the test fails on the first case, which is accepted. The doc comment on UpdateGsiAction now says what the two members are read for.
| assert t["WarmThroughput"] == {**ON_DEMAND_DEFAULTS, "Status": "UPDATING"} | ||
| if is_extenddb: | ||
| _wait_warm_active(dynamodb_client, name) | ||
| dynamodb_client.update_table(TableName=name, WarmThroughput=higher) |
There was a problem hiding this comment.
I think this re-send hides the one thing U3 leaves open, which is what the value settles on. On ExtendDB, raising a settled on-demand table to 30000/9000 and then sending 12000/4000 5 ms later, inside the 0.25 s window, answers 200 twice and the table settles at {12000, 4000, ACTIVE}, on SQLite and PostgreSQL alike, while the same 12000/4000 on the settled 30000/9000 table is refused as a decrease. So U3 together with apply_update (warm_throughput.rs:226) is a way to lower the value that the decrease rule otherwise refuses. Whether the service settles at 12000 or keeps 30000 isn't in the measurements; it might be worth measuring that once and asserting it on both targets instead of re-sending higher here. If the service keeps the higher value, taking the larger of the stored and requested side in apply_update would match it, and would also close the same decrease through two concurrent UpdateTables, which both validate against the description read before the write.
There was a problem hiding this comment.
Fixed in 33d4e66. I measured it on DynamoDB (us-east-1, 2026-11-03): on an on-demand table and on a GSI, 12200/4200 and then 12000/4000 about 50 ms later both answered 200, and both settled at 12200/4200 ACTIVE, the GSI about nine minutes later and the table about eleven.
apply_update now keeps, per side, the larger of the requested and the stored value, so your 30000/9000 then 12000/4000 sequence settles at 30000/9000. Outside the window a lower request is still refused as a decrease before the merge. The test no longer re-sends higher, so its settled assertion checks this rule on ExtendDB. On the service the settle takes about ten minutes, so as with the other settled rows the docstring records the measurement (row U3b) and the wait runs on ExtendDB only. the_previous_pair_is_reported_and_compared_while_updating asserts the settled pair and a case with one side lower and one higher. With the max rule reverted it fails: the pair settles at 12000/4000 where the test expects 24000/8000.
| let json = | ||
| serde_json::to_value(record).map_err(|e| StorageError::Internal(e.to_string()))?; | ||
| sqlx::query( | ||
| "UPDATE indexes SET warm_throughput = $1 \ |
There was a problem hiding this comment.
This write runs after the GSI build and puts the Create's record back unconditionally, so a WarmThroughput Update on the same index that was accepted while the build ran is overwritten. On the head PostgreSQL build with 30000 items, a Create of gidx with {13000, 4100} took 6.9 s; an Update to {ReadUnitsPerSecond: 25000} sent 0.4 s in answered 200 and DescribeTable showed 25000 right after, and once the Create returned the index read {13000, 4100, ACTIVE}. The same script on SQLite ends at 25000, since there the Update waits for the writer. I think adding AND warm_throughput = $4 with the record the Create stored would keep the restart and leave a later Update alone.
There was a problem hiding this comment.
Fixed in ed0a481. update_table keeps the record each created GSI was inserted with, and restart_created_gsi_warm_throughput writes the new deadline with AND warm_throughput = $4 bound to that record, so an Update accepted while the build ran keeps its value. I did not add a test for it: the window needs a build that outlasts the Update's arrival, as in your 30000-item reproduction, which I have not rerun here.
| --comprehensive) RUN_COMPREHENSIVE=true; shift ;; | ||
| --all) RUN_RUST=true; RUN_RUST_INTEGRATION=true; RUN_PYTEST=true; RUN_EXTERNAL=true; RUN_COMPREHENSIVE=true; shift ;; | ||
| --terraform) RUN_TERRAFORM=true; shift ;; | ||
| --all) RUN_RUST=true; RUN_RUST_INTEGRATION=true; RUN_PYTEST=true; RUN_EXTERNAL=true; RUN_COMPREHENSIVE=true; RUN_TERRAFORM=true; shift ;; |
There was a problem hiding this comment.
Putting --terraform into --all means the documented devtools/run-tests --extenddb --all (developer test guide line 129, AGENTS.md line 218, tests/README.md line 37) now needs access to releases.hashicorp.com and registry.terraform.io, and fails on macOS or arm64 since install-terraform.sh only fetches linux_amd64. The binary is cached in $TMPDIR/terraform-1.17.0 after the first run, but the provider lands in a fresh mktemp directory every time. CI passes --terraform explicitly, so it wouldn't lose anything if --all left it out, or skipped it with a message off linux/x86_64. Separately, main.tf only creates, plans and destroys, so the update paths this PR changes (a GSI added, a billing switch) have no provider-level check; a second apply with a change might be worth adding.
There was a problem hiding this comment.
| deletion_protection_enabled: None, | ||
| table_class: None, | ||
| on_demand_throughput: None, | ||
| warm_throughput: None, |
There was a problem hiding this comment.
The table-level member is dropped here, but each GSI's WarmThroughput still comes through global_secondary_indexes: tcp.global_secondary_indexes.clone() and ImportTable never runs CreateTable's validation, so an imported provisioned 5/5 index with WarmThroughput: {ReadUnitsPerSecond: -7, WriteUnitsPerSecond: 0} is stored and reported as -7 while CreateTable refuses the same GSI. It isn't new in kind, since a GSI ProvisionedThroughput of -3 goes through ImportTable on the base build too, so I'd either clear the GSI member here as well for consistency or, probably better as its own issue, run validate_create_table on the import's create input.
There was a problem hiding this comment.
Changed in 9fc0a41: the import clears each GSI's WarmThroughput as it does the table's, so an imported table and its indexes report the defaults. an_imported_gsi_warm_throughput_is_cleared uses your index with -7, and without the clear it fails with the -7 still on the index. Running validate_create_table on the import's create input belongs with #771, where I added the -3 ProvisionedThroughput reproduction from main.
main replaced the three upgrade mechanisms this branch used (#666, #681): serve refuses a catalog with pending migrations, so the PostgreSQL shape steps applied at serve time, SQLite's shape convergence and Cassandra's V006 with catalog version 0.0.5 cannot run any more. The columns are now migration 6, `warm_throughput`, in crates/storage/src/migrations/registry.rs, Breaking and convergent on every backend, with RELEASED_THROUGH left at 0: - PostgreSQL: migrations/0006_warm_throughput.sql, ADD COLUMN IF NOT EXISTS on tables and indexes, run in a transaction. - SQLite: the code body m0006_warm_throughput.rs, which reads pragma_table_info before each ALTER TABLE, with a golden test on the 0.1.14 catalog (tests/fixtures/golden/0006_warm_throughput.txt). - Cassandra: migrations/catalog/0006_warm_throughput.cql with two ALTER TABLE ... ADD IF NOT EXISTS and no catalog_version insert. V006 is gone. - MongoDB: NotApplicable, since the catalog collections have no validator. The merge left migration 1 and SQLite's schema.rs as main has them, so no warm_throughput hunk reached 0001_baseline.sql. The tests that pin the latest migration or a catalog's definitions after every migration now expect 6, and _as_released_catalog in tests/test_resource_tag_migration.py drops the two columns so migration 1's check of the v1.0 schema accepts the catalog. The upgrade manual drops this branch's Catalog 0.0.5 section, which collided with main's continuous-backups drop, and describes migration 6 under Current Status, including that going back to a release without it means restoring the backup taken before the upgrade. The verify examples in the install guides read migration 6, and docs/schema/{sqlite,postgres,cassandra}.txt are regenerated from a fresh init. Literals added on main since the branch point gain the new members: GsiInput in crates/storage/src/snapshot_conformance.rs and in two Cassandra direct tests, and UpdateTableInput and CreateGsiAction in another. The SQLite warm throughput tests build their catalog with migrations::apply_for_tests and insert control_plane_delay_seconds, which no migration seeds.
The v1 manifest recorded the source table's indexes as GsiInput and LsiInput, the CreateTable request types, so this branch's WarmThroughput member on GsiInput changed the persisted format and failed nested_types_have_the_pinned_members. A backup does not record warm throughput: DescribeBackup's GlobalSecondaryIndexInfo has no warm member, and a restored table starts with the defaults. BackupGsi and BackupLsi in manifest.rs carry the serde names, member order and skip_serializing_if the request types had when format version 1 was frozen, without WarmThroughput. SourceTableSchema uses them, the member pins check them with the same member lists, and the module docs say the index entries are the manifest's own. The fixtures under testdata/extenddb-manifest/ are unchanged, so no format version bump.
…mThroughput UpdateGsiAction deserialized only IndexName and WarmThroughput, so an Update carrying ProvisionedThroughput or OnDemandThroughput next to WarmThroughput answered 200 and dropped the other member. On an on-demand table switched to PROVISIONED in the same request, the index was left at 0/0. It now reads both members as opaque values. Measured on DynamoDB (us-east-1, 2026-11-03): an Update carrying WarmThroughput beside OnDemandThroughput on an on-demand table, beside ProvisionedThroughput or OnDemandThroughput on a provisioned table, and in the request that switches an on-demand table with a GSI to PROVISIONED, is refused with "One or more parameter values were invalid: WarmThroughput must be the only operation in the request for index: gsi". The rule does not depend on the billing mode, so it is checked with the other element rules, before the table is read. An Update without WarmThroughput is refused once the table is read, under the billing mode the table has after the request: the request's BillingMode when it carries one, the stored mode otherwise. Under PAY_PER_REQUEST one carrying ProvisionedThroughput or nothing gets the measured on-demand sentence; one carrying only OnDemandThroughput, and any under PROVISIONED (the index update of a switch to PROVISIONED included), gets "UpdateGlobalSecondaryIndex is not yet supported", as on main.
…ne in progress Inside the UPDATING window a request is validated against the values the window started from, so it can be lower than the change in progress. apply_update wrote the requested side, which lowered the value the decrease rule otherwise refuses. Measured on DynamoDB (us-east-1, 2026-11-03): on an on-demand table and on a GSI, 12200/4200 and then 12000/4000 about 50 ms later both answered 200, and each settled at 12200/4200 ACTIVE. apply_update now keeps, per side, the larger of the requested and the stored value. Outside the window a lower request is refused before it reaches the merge, so nothing changes there. The wire test no longer re-sends the higher values after U3, so its settled assertion checks this rule on ExtendDB; the service takes about ten minutes to settle, which the docstring records with the measurement.
…a side MongoDB read the record with find_one and wrote the merge back with $set, and Cassandra read the column and wrote it through the table LWT (conditional only on IF EXISTS) and a plain UPDATE of the index row, so two UpdateTables that set different sides could both answer 200 while one side was lost. - MongoDB: the update_one filter includes the field as read; when nothing matches, the document is read again and the merge retried. - Cassandra: with a WarmThroughput, the table LWT is conditional on table_id and on the record read; a GSI's record is written by its own LWT, conditional on index_type and the record read. Both read again and retry when the condition fails. The BillingMode arm that let a switch to PAY_PER_REQUEST skip the column on an unmigrated catalog, and swallowed any read or parse error with it, is gone: migration 6 runs before a server serves the catalog. - PostgreSQL: the restart of a created GSI's UPDATING window after the build is conditional on the record the Create inserted, so an Update of the index accepted while the build ran keeps its value. A stored record that does not parse (a hand edit, or a shape this build does not know) is now logged and read as never set on every backend, as the neighbouring on_demand_throughput column is read. DescribeTable reports the defaults, and UpdateTable writes a record over it, where every description build failed with a 500 before. A wire test sends ten rounds of two UpdateTables together, one per side, on the table and on a GSI, and asserts both sides settle. It runs on ExtendDB only, since the service bills each increase.
ImportTable drops the table-level member, since TableCreationParameters has none, but passed each GSI's WarmThroughput through, and ImportTable does not run CreateTable's validation, so a value CreateTable refuses (-7 read units) was stored and reported. The import now clears it, and the imported table and its indexes report the defaults. Running CreateTable's validation on the import's create input, which also lets a GSI ProvisionedThroughput of -3 through on main, is left to its own issue.
devtools/run-tests --extenddb --all is the documented local command (developer test guide, AGENTS.md, tests/README.md). With --terraform in --all it needed releases.hashicorp.com and registry.terraform.io and failed on macOS and arm64, since install-terraform.sh only fetches linux_amd64. CI passes --terraform explicitly, so --all now leaves it out and the help text says so.
Migration 6 does not apply on MongoDB, but the runner reports it as a step with the outcome NotApplicable and records it in the ledger, as migrations_fresh.rs expects. A migrator that starts from the 0.1.12 fixture therefore reports seven steps. The migration 6 commit moved this test's catalog version assertion to 6 and left its expected step count at six.
Migration 6 adds a warm_throughput column to the tables and indexes catalogs, and the catalog jobs compare a fresh install with these files. Each file is the schema-dump output of a fresh init by this branch's binary: PostgreSQL lists the two columns as jsonb after the last column of each table, SQLite shows them as TEXT in the CREATE TABLE statements, and Cassandra lists them as text regular columns in name order. The dumps differ from the committed files by those two columns only. MongoDB's file lists collections and indexes, and migration 6 adds neither, so it is unchanged.
update_table.rs chose the mode for refuse_bare_updates from the request's BillingMode when it carries one, else from the stored mode, but no test covered that choice: with it reverted to the stored mode only, every engine test still passed. The choice moves into gsi_updates::pay_per_request_after, and a unit test sends the switch to PROVISIONED the service requires on a PAY_PER_REQUEST table (BillingMode PROVISIONED, the table's and the index's ProvisionedThroughput 3/3), which gets "UpdateGlobalSecondaryIndex is not yet supported", and the same Update without BillingMode, which gets the on-demand sentence. With the request's mode ignored the test fails. Migration 6 is now WarmThroughput, so the worked example in CONTRIBUTING.md's "Adding a catalog migration" becomes migration 7, and the version and banner samples in the getting-started and quickstart guides read catalog migration 6, as this build prints.
|
@robinnsc, thanks for the review. Merged main at 5ff323b (3906cdb) and again at 03baa32 (e531256), and addressed every point; replies are on each inline comment. Merge with main: the branch's upgrade code now runs through main's migration framework (641e663). Literals that main added since the branch point gained the new members (GsiInput in crates/storage/src/snapshot_conformance.rs and two Cassandra direct tests, UpdateTableInput and CreateGsiAction in another), and the SQLite warm throughput tests build their catalog with The GSI Update that dropped ProvisionedThroughput or OnDemandThroughput: fixed in 3546b08. On DynamoDB every combination of WarmThroughput with either member, in both billing modes and in the switch to PROVISIONED, answers "One or more parameter values were invalid: WarmThroughput must be the only operation in the request for index: gsi", and ExtendDB now does the same before the table is read. Migration framework: The 0001_baseline.sql trap: Upgrade manual: this branch's "Catalog 0.0.5 (Cassandra only)" section is gone. Current Status reads "migrations 1 to 6" with a paragraph for 6 that ends on restoring the pre-upgrade backup to go back to a release without it, and "Migrations 1 to 6 are breaking". Backup manifest, from the conversation: BackupGsi and BackupLsi in manifest.rs, with the serde names, order and skip_serializing_if of the request types and no WarmThroughput, used by SourceTableSchema and the member pins (f709fb1). The manifest fixtures are unchanged. Measured on DynamoDB (us-east-1, 2026-11-03): the overlap of two updates inside the UPDATING window on a table and on a GSI (both settle at the higher value, used in 33d4e66), and the four GSI Update shapes above. Run on the merged tree: fmt; clippy 1.97 Wire tests against a SQLite server built from the branch: test_warm_throughput.py, test_warm_throughput_updates.py, test_warm_throughput_backup.py and test_gsi_online_backfill.py, 29 passed. The PostgreSQL, MongoDB and Cassandra jobs in CI run the same files. Each fix's unit test fails with the fix reverted: the per-side maximum in 0b2315a, from a check before this push: the billing mode a bare GSI Update is answered by is now chosen in |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL's rust/cleartext-logging rule reads a variable named `billing` as billing information and flagged the assertion message that prints it in the WarmThroughput-beside-a-throughput test. The variable holds a BillingMode value such as PROVISIONED; it is renamed to `mode`, and the test is unchanged otherwise.
… backfill test The test that main added in #424 builds an UpdateTableInput and a CreateGsiAction, which this branch extends with warm_throughput and sse_specification. Signed-off-by: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com>
|
The merge queue removed this twice with failed checks. #424 added a test to 2d338bf merges main. 8c35832 sets The push dismissed the approval, so the PR needs a new one before it can go back into the queue. |
Closes #462.
Summary
Since Terraform AWS provider 6.13.0 (September 2025),
aws_dynamodb_tablewaits forWarmThroughput.Statusto readACTIVEafter every table and index create or update, and treats aDescribeTablewithout the member as not found. ExtendDB omitted the member, so everyterraform applyagainst it polled untilcouldn't find resource (21 retries). The same hang against DynamoDB Local is hashicorp/terraform-provider-aws#44344, closed by the provider team as an emulator-side fix; the current provider is 6.70.0 and the only workaround was pinning 6.12.0.ExtendDB now stores and reports
WarmThroughputon tables and global secondary indexes on all four backends, with the shape, defaults, lifecycle and validation measured on Amazon DynamoDB.Nothing acts on the values. There is no pre-warming and no scaling at the database level, and the stored numbers have no effect on throughput, throttling or storage. They exist so clients that read the member work. This is stated in
docs/differences-from-dynamodb.md(Capacity and Throttling), indocs/getting-started.md, and at the storage field.What the service does, and what ExtendDB now does
Every rule below was measured on live DynamoDB (us-east-1, 2026-10-23 and 2026-11-03) before it was written; the raw observations are in the dual-target wire tests' docstrings as row ids.
{12000, 4000, ACTIVE}on an on-demand table and its GSIs, the provisioned units on a provisioned table and its GSIs; LSIs carry noneStatus: UPDATINGwhile CREATING; filled once ACTIVE (a read-only{20000}becomes{20000, 4000})UPDATING, then the new valuesACTIVE; a decrease is validated against the reported valuesUPDATINGwindowValidationExceptionwith the service's sentence, namingtableorindex <name>WarmThroughput must be the only operation in the request); a GSI Update beside it is acceptedWarmThroughput must be the only operation in the request for index: <name>); on an on-demand table an index Update may only carry OnDemandThroughput or WarmThroughput; a Create with WarmThroughput reports it on the CREATING indexExtendDB returns to
ACTIVEaftercontrol_plane_delay_secondswhere the service takes minutes, and it does not enforce the account maximum (LimitExceededExceptionon the service). An index Update without WarmThroughput is answered by the billing mode the table has after the request: under PAY_PER_REQUEST one carrying ProvisionedThroughput or nothing gets the service's on-demand sentence, and one carrying only OnDemandThroughput, or any under PROVISIONED, gets "UpdateGlobalSecondaryIndex is not yet supported", as on main. ImportTable reports the defaults for the imported table and its indexes.Storage
One nullable
warm_throughputcolumn ontablesandindexes, added by catalog migration 6 (warm_throughput, Breaking, convergent): a file body on PostgreSQL (migrations/0006_warm_throughput.sql) and Cassandra (migrations/catalog/0006_warm_throughput.cql), a code body on SQLite (m0006_warm_throughput.rs). On MongoDB the record is a document field and the ledger records migration 6 as not applicable.extenddb migrate --yesapplies it; the upgrade manual describes it, including that going back to a release without it means restoring the backup taken before the upgrade. Each backend stores one JSON record; the wire member is derived in one place,TableDescription::prepare_for_response, on every response path. A stored record that does not parse is logged and read as never set, so the defaults are reported.Concurrent UpdateTables that set different sides both keep their side: MongoDB filters the write on the record read and retries, Cassandra writes the record through LWTs conditional on the record read and retries, and SQLite and PostgreSQL merge inside the catalog transaction.
UpdateTableInputnow readsSSESpecificationso the member can be refused instead of silently dropped, andUpdateGsiActionreadsProvisionedThroughputandOnDemandThroughputso an index Update carrying either is refused instead of dropping it.The backup manifest records the source table's indexes as
BackupGsiandBackupLsi, its own types with the members, names and serialization the request types had when format version 1 was frozen, soGsiInput's new member does not change the format.Tests
tests/python/: lifecycle and validation (12), update shapes and billing changes (13), backup and restore (1). They pass against the service and against SQLite, PostgreSQL, MongoDB and Cassandra. The update-shapes file also has an ExtendDB-only test that sends two UpdateTables together, one per side, for ten rounds on a table and on a GSI.tests/terraform/warm-throughput/: the configuration from the provider issue plus a table with a GSI, applied with Terraform 1.17.0 and provider 6.70.0 (both pinned, the Terraform zip checksum-asserted against the published sums) on every backend in CI viadevtools/run-tests --terraform; asserts a clean apply, a second plan with no changes, and destroys.--allleaves it out, since it downloads Terraform and the provider and only installs linux_amd64.Negative controls: with the member removed the Terraform apply fails with the provider's
couldn't find resource (21 retries)after about three and a half minutes, which is the hang from the issue; with the validation removed the refusal rows fail; five further built mutants (table member, index member, UPDATING deadline, only-operation rule, swapped defaults) each fail the wire file.fmt, clippy 1.97
-D warningsand MSRV 1.88 on every crate in the diff; no dependency change.Out of scope, filed separately
Measured on
mainwhile working this issue, not touched here: the service's per-member "must be the only operation" rules for TableClass, DeletionProtection and Server-Side Encryption when WarmThroughput is absent (ExtendDB accepts those pairs); a GSI reportingIndexStatus: ACTIVEwhile its table is CREATING;TableStatus: ACTIVEin the response to a capacity change where the service says UPDATING;UpdateTablewith an emptyWarmThroughput: {}or an Update naming a missing index, where the service answers its ownInternalFailure. ImportTable not running CreateTable's validation is #771; a Terraform apply that changes the tables is #800.