Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
4b7389a
fix: answer requests during a table's DELETING window as the service …
LeeroyHannigan Oct 29, 2026
5b4c9ff
test(cassandra): keep the DELETING-window rows out of the control-pla…
LeeroyHannigan Oct 29, 2026
b060b0e
Merge branch 'main' into fix/505-deleting-window-answers
LeeroyHannigan Nov 3, 2026
e583912
fix(mongodb): finish a DeleteTable that stops after its DELETING mark
LeeroyHannigan Nov 3, 2026
1de1816
test(postgres): describe the concurrent DeleteTable test as the code …
LeeroyHannigan Nov 3, 2026
040be0d
fix(postgres): keep an abandoned restore target not found while it is…
LeeroyHannigan Nov 3, 2026
acb03bf
fix(sqlite): answer only a missing _ddb_ table as a table that is gone
LeeroyHannigan Nov 3, 2026
9f87c96
refactor(sqlite): drop the unreachable no-row arm after the DELETING …
LeeroyHannigan Nov 3, 2026
8156c69
test(sqlite): find the cargo target directory from the test binary's …
LeeroyHannigan Nov 3, 2026
657e1c0
test: run the DELETING window wire tests at any control-plane delay
LeeroyHannigan Nov 3, 2026
3a0ee7d
test: wait for a table being deleted at fixture teardown
LeeroyHannigan Nov 3, 2026
f2a777a
fix(engine): answer an index read on a table that is gone as not found
LeeroyHannigan Nov 3, 2026
97b4bef
fix(engine): refuse TagResource and UntagResource while a table is DE…
LeeroyHannigan Nov 3, 2026
34a9c9a
fix(mongodb): remove a deleted table's tags only with its document
LeeroyHannigan Nov 3, 2026
4e1ebe0
test: check that ListTagsOfResource lists a DELETING table's tag
LeeroyHannigan Nov 3, 2026
d8d6e05
Merge branch 'main' into fix/505-deleting-window-answers
LeeroyHannigan Nov 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/integration-mongodb.yml
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,14 @@ jobs:
EXTENDDB_TEST_MONGODB_STANDALONE_URI: mongodb://127.0.0.1:27019/?directConnection=true
EXTENDDB_TEST_MONGODB_REQUIRED: "1"
run: cargo test -p extenddb-storage-mongodb snapshot_engine
- name: Run DELETING window tests
# What DeleteTable and UpdateTable answer while a table is DELETING,
# and that a delete which stopped after its mark is finished. Each
# test works in databases of its own on the replica set above.
env:
EXTENDDB_TEST_MONGODB_URI: mongodb://127.0.0.1:27018/?directConnection=true
EXTENDDB_TEST_MONGODB_REQUIRED: "1"
run: cargo test -p extenddb-storage-mongodb --lib deleting_window_tests
- name: Stop MongoDB containers
if: always()
run: docker rm -f extenddb-runtests-mongo-27018 extenddb-snapshot-standalone-27019
Expand Down
17 changes: 9 additions & 8 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -407,14 +407,15 @@ jobs:
# storage layer directly against this job's PostgreSQL. The PutItem create
# race tests hold a create open in an outside transaction, the
# write-transaction conflict tests force a deadlock, the stream clock test
# holds a row lock open while a write waits on it, and the backup restore
# cases need catalog rows the current binary would not write; all need
# direct access. The online GSI build tests reorder queue rows by hand and
# pause writers with a test hook. They build their own throwaway databases;
# the connection string is the server, not a database. `--tests` runs
# every target of the crate, so a new test file needs no edit here, and
# EXTENDDB_TEST_PG_REQUIRED turns a test's skip for a missing connection
# string into a failure.
# holds a row lock open while a write waits on it, the backup restore
# cases need catalog rows the current binary would not write, and the
# DELETING window tests hold a table DELETING with no poller to finish it;
# all need direct access. The online GSI build tests reorder queue rows by
# hand and pause writers with a test hook. They build their own throwaway
# databases; the connection string is the server, not a database. `--tests`
# runs every target of the crate, so a new test file needs no edit here,
# and EXTENDDB_TEST_PG_REQUIRED turns a test's skip for a missing
# connection string into a failure.
- name: Run PostgreSQL storage-level tests
# table_snapshot: the SnapshotEngine consistency check; it needs the
# shared page-size hook, compiled only into the crate's own test
Expand Down
10 changes: 10 additions & 0 deletions crates/core/src/error/messages.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@ pub enum ErrorMessageKey {
TableNotFound,
TableAlreadyExists,
TableInUse,
/// A control-plane change refused because the table is `DELETING`.
/// Measured on the service 2026-10-24 for a second `DeleteTable` and for
/// `UpdateTable` during the window.
TableBeingDeleted,
MissingKeySchema,
KeySchemaTooMany,
KeySchemaFirstNotHash,
Expand Down Expand Up @@ -82,6 +86,12 @@ pub fn error_message(key: ErrorMessageKey, args: &[&str]) -> String {
ErrorMessageKey::TableInUse => {
format!("Table is being created, updated, or deleted: {}", args[0])
}
ErrorMessageKey::TableBeingDeleted => {
format!(
"Attempt to change a resource which is still in use: Table is being deleted: {}",
args[0]
)
}
ErrorMessageKey::MissingKeySchema => {
"1 validation error detected: Value null at 'keySchema' failed to satisfy constraint: \
Member must not be null"
Expand Down
3 changes: 3 additions & 0 deletions crates/engine/src/create_table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,9 @@ pub fn storage_err_to_dynamo(e: extenddb_storage::error::StorageError) -> Dynamo
ErrorMessageKey::TableInUse,
&[&name],
)),
StorageError::TableBeingDeleted(name) => DynamoDbError::ResourceInUseException(
error_message(ErrorMessageKey::TableBeingDeleted, &[&name]),
),
// Only RestoreTableFromBackup raises this; its own mapping makes it
// TableInUseException. Elsewhere (a plain CreateTable into a name
// that is a CREATING restore target) the service's text is the one
Expand Down
25 changes: 25 additions & 0 deletions crates/engine/src/delete_table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,28 @@ pub async fn handle_delete_table(
};
serialize_output(&output)
}

#[cfg(test)]
mod tests {
use super::*;
use extenddb_storage::error::StorageError;

/// A second DeleteTable during the DELETING window: the measured sentence
/// (2026-10-24), byte for byte. The other non-ACTIVE statuses keep theirs.
#[test]
fn a_deleting_table_has_its_own_sentence() {
match storage_err_to_dynamo(StorageError::TableBeingDeleted("t".to_owned())) {
DynamoDbError::ResourceInUseException(msg) => assert_eq!(
msg,
"Attempt to change a resource which is still in use: Table is being deleted: t"
),
other => panic!("expected ResourceInUseException, got {other:?}"),
}
match storage_err_to_dynamo(StorageError::TableNotActive("t".to_owned())) {
DynamoDbError::ResourceInUseException(msg) => {
assert_eq!(msg, "Table is being created, updated, or deleted: t");
}
other => panic!("expected ResourceInUseException, got {other:?}"),
}
}
}
32 changes: 32 additions & 0 deletions crates/engine/src/index_helpers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,38 @@ pub fn classify_vector_index_read(
}
}

/// The answer for a GSI or LSI name that `index_info_by_table_id` reported
/// missing (`err`) on the table `key_info` describes.
///
/// The key info can come from the key-info cache, and the cache keeps an entry
/// read while the table was `DELETING` after the control plane removes the
/// table, since that removal does not reach the cache. The index rows go with
/// the table, so a cached entry would turn a read on a table that is gone into
/// `The table does not have the specified index` until the entry expires. The
/// catalog is therefore asked for the table again. A table it no longer has is
/// answered as a read on a missing table (`ResourceNotFoundException`) and its
/// cache entry is dropped; a table it still has keeps `err`. The extra read
/// runs only for an index name that does not resolve.
pub async fn unresolved_index_error(
ctx: &OperationContext,
key_info: &TableKeyInfo,
err: extenddb_storage::error::StorageError,
) -> DynamoDbError {
match ctx
.storage
.table_key_info(&ctx.account_id, &key_info.table_name)
.await
{
Err(gone @ extenddb_storage::error::StorageError::TableNotFound(_)) => {
ctx.auth_cache
.invalidate_table_key_info(&ctx.account_id, &key_info.table_name)
.await;
crate::create_table::storage_err_to_dynamo(gone)
}
_ => crate::create_table::storage_err_to_dynamo(err),
}
}

/// Resolve an index name that `index_info_by_table_id` reported missing
/// against the table's vector index metadata.
///
Expand Down
4 changes: 2 additions & 2 deletions crates/engine/src/query.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ use crate::create_table::storage_err_to_dynamo;
use crate::expression_helpers::{build_expression_maps, parse_optional_filter};
use crate::index_helpers::{
VectorIndexReadRefusal, classify_unresolved_index_read, combined_lek_key_schema,
validate_query_select_index_support,
unresolved_index_error, validate_query_select_index_support,
};
use crate::legacy_filter::{desugar_filter, desugar_key_conditions};
use crate::read_helpers::{PostRead, read_page};
Expand Down Expand Up @@ -101,7 +101,7 @@ pub async fn handle_query(
Err(err @ StorageError::IndexNotFound(_)) => {
match classify_unresolved_index_read(ctx, &key_info, idx_name).await? {
VectorIndexReadRefusal::NotFound => {
return Err(storage_err_to_dynamo(err));
return Err(unresolved_index_error(ctx, &key_info, err).await);
}
// Query refuses a backfilling vector index with the same
// message as an active one (measured 2026-08-20; Scan
Expand Down
5 changes: 4 additions & 1 deletion crates/engine/src/scan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ use crate::create_table::storage_err_to_dynamo;
use crate::expression_helpers::{build_expression_maps, parse_optional_filter};
use crate::index_helpers::{
VectorIndexReadRefusal, classify_unresolved_index_read, combined_lek_key_schema,
unresolved_index_error,
};
use crate::legacy_filter::desugar_filter;
use crate::read_helpers::{PostRead, read_page};
Expand Down Expand Up @@ -259,7 +260,9 @@ pub async fn handle_scan(
Err(err @ StorageError::IndexNotFound(_)) => {
return Err(
match classify_unresolved_index_read(ctx, &key_info, idx_name).await? {
VectorIndexReadRefusal::NotFound => storage_err_to_dynamo(err),
VectorIndexReadRefusal::NotFound => {
unresolved_index_error(ctx, &key_info, err).await
}
VectorIndexReadRefusal::Backfilling => {
DynamoDbError::ValidationException(format!(
"{VECTOR_INDEX_BACKFILLING_SCAN_PREFIX}{idx_name}"
Expand Down
58 changes: 54 additions & 4 deletions crates/engine/src/tagging.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,48 @@ fn extract_table_name_from_arn(arn: &str) -> Option<&str> {
Some(table_name.split('/').next().unwrap_or(table_name))
}

/// What [`validate_resource_arn`] answers for a table that is `DELETING`.
#[derive(Clone, Copy, PartialEq, Eq)]
enum WhileDeleting {
/// `ListTagsOfResource`: the table is listed until it is gone.
Serve,
/// `TagResource` and `UntagResource`: refused as a table still in use.
Refuse,
}

/// Refuse a tag change on a table that is being deleted. The service answers
/// `TagResource` and `UntagResource` on a `DELETING` table with
/// `ResourceInUseException` and the sentence `DeleteTable` and `UpdateTable`
/// give in that window, and serves `ListTagsOfResource` (measured
/// 2026-11-03). The key info carries no status, and the data plane reads a
/// `DELETING` table as found on PostgreSQL and SQLite, so the status is read
/// with `describe_table`. Any other answer from it is left to the existence
/// check that follows.
async fn refuse_while_deleting(
table_name: &str,
ctx: &OperationContext,
) -> Result<(), DynamoDbError> {
if let Ok(desc) = ctx
.storage
.describe_table(
&ctx.account_id,
extenddb_core::types::DescribeTableInput {
table_name: table_name.to_owned(),
},
)
.await
&& desc.table_status == extenddb_core::types::TableStatus::Deleting
{
return Err(DynamoDbError::ResourceInUseException(
extenddb_core::error::error_message(
extenddb_core::error::ErrorMessageKey::TableBeingDeleted,
&[table_name],
),
));
}
Ok(())
}

/// Validate that the ARN refers to an existing table owned by the caller.
///
/// The error classes and messages below match `DynamoDB`, verified against the
Expand All @@ -61,13 +103,18 @@ fn extract_table_name_from_arn(arn: &str) -> Option<&str> {
/// | Account differs from the caller's | `AccessDeniedException` |
/// | Table ARN with a different partition, service, or region | `ValidationException` (invalid `TableArn`) |
/// | Table does not exist | `ResourceNotFoundException` |
/// | Table is `DELETING`, for `TagResource` and `UntagResource` (`while_deleting` is `Refuse`) | `ResourceInUseException` (`Table is being deleted`) |
///
/// Account is checked before the partition, service, and region, matching the
/// service: a cross-account ARN is denied without revealing whether its other
/// fields are valid. For a table ARN, a no-permissions principal is denied by
/// authorization before this runs; the partition, service, region, and
/// existence errors therefore surface only for an authorized principal.
async fn validate_resource_arn(arn: &str, ctx: &OperationContext) -> Result<(), DynamoDbError> {
async fn validate_resource_arn(
arn: &str,
ctx: &OperationContext,
while_deleting: WhileDeleting,
) -> Result<(), DynamoDbError> {
if !arn.starts_with("arn:") {
return Err(DynamoDbError::ValidationException(format!(
"One or more parameter values were invalid: ARNs must start with 'arn:': {arn}"
Expand Down Expand Up @@ -140,6 +187,9 @@ async fn validate_resource_arn(arn: &str, ctx: &OperationContext) -> Result<(),
))
})?;

if while_deleting == WhileDeleting::Refuse {
refuse_while_deleting(table_name, ctx).await?;
}
// Verify the table exists via table_key_info (lightweight check).
ctx.storage
.table_key_info(&ctx.account_id, table_name)
Expand Down Expand Up @@ -175,7 +225,7 @@ pub async fn handle_tag_resource(
));
}

validate_resource_arn(&input.resource_arn, ctx).await?;
validate_resource_arn(&input.resource_arn, ctx, WhileDeleting::Refuse).await?;

ctx.storage
.tag_resource(&input.resource_arn, &input.tags)
Expand Down Expand Up @@ -212,7 +262,7 @@ pub async fn handle_untag_resource(
));
}

validate_resource_arn(&input.resource_arn, ctx).await?;
validate_resource_arn(&input.resource_arn, ctx, WhileDeleting::Refuse).await?;

ctx.storage
.untag_resource(&input.resource_arn, &input.tag_keys)
Expand Down Expand Up @@ -247,7 +297,7 @@ pub async fn handle_list_tags_of_resource(
));
}

validate_resource_arn(&input.resource_arn, ctx).await?;
validate_resource_arn(&input.resource_arn, ctx, WhileDeleting::Serve).await?;

let tags = ctx
.storage
Expand Down
28 changes: 28 additions & 0 deletions crates/engine/src/ttl.rs
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,15 @@ fn storage_to_dynamo(e: StorageError) -> DynamoDbError {
StorageError::TableNotActive(name) => {
DynamoDbError::ResourceInUseException(format!("Table {name} is not in ACTIVE state"))
}
// No TTL path raises this today (the TTL metadata paths answer
// TableNotActive for every non-ACTIVE status), but the variant exists
// and a backend that raises it here must not become a 500.
StorageError::TableBeingDeleted(name) => {
DynamoDbError::ResourceInUseException(extenddb_core::error::error_message(
extenddb_core::error::ErrorMessageKey::TableBeingDeleted,
&[&name],
))
}
other => {
tracing::error!(internal_error = %other, "storage internal error");
DynamoDbError::InternalServerError("Internal server error".to_owned())
Expand All @@ -237,6 +246,25 @@ fn storage_to_dynamo(e: StorageError) -> DynamoDbError {
mod tests {
use super::*;

/// The TTL mapper renders a DELETING table with the service's sentence and
/// keeps the other non-ACTIVE statuses on theirs.
#[test]
fn a_deleting_table_has_its_own_sentence() {
match storage_to_dynamo(StorageError::TableBeingDeleted("t".to_owned())) {
DynamoDbError::ResourceInUseException(msg) => assert_eq!(
msg,
"Attempt to change a resource which is still in use: Table is being deleted: t"
),
other => panic!("expected ResourceInUseException, got {other:?}"),
}
match storage_to_dynamo(StorageError::TableNotActive("t".to_owned())) {
DynamoDbError::ResourceInUseException(msg) => {
assert_eq!(msg, "Table t is not in ACTIVE state");
}
other => panic!("expected ResourceInUseException, got {other:?}"),
}
}

#[test]
fn valid_ttl_attribute_names() {
assert!(validate_ttl_attribute_name("ttl").is_ok());
Expand Down
30 changes: 30 additions & 0 deletions crates/engine/src/update_table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,15 @@ fn update_table_err_to_dynamo(
StorageError::TableNotActive(name) => {
DynamoDbError::ResourceInUseException(format!("Table {name} is not in ACTIVE state"))
}
// The service words a DELETING table differently from the other
// non-ACTIVE statuses (measured 2026-10-24); the sentence lives with
// the engine's other message templates.
StorageError::TableBeingDeleted(name) => {
DynamoDbError::ResourceInUseException(extenddb_core::error::error_message(
extenddb_core::error::ErrorMessageKey::TableBeingDeleted,
&[&name],
))
}
StorageError::IndexAlreadyExists(name) => DynamoDbError::ValidationException(format!(
"One or more parameter values were invalid: Index already exists: {name}"
)),
Expand Down Expand Up @@ -500,4 +509,25 @@ mod tests {
update_table_err_to_dynamo(StorageError::Internal("disk on fire".to_owned()), "t");
assert!(matches!(err, DynamoDbError::InternalServerError(_)));
}

/// UpdateTable on a DELETING table: the measured sentence (2026-10-24),
/// byte for byte, and the other non-ACTIVE statuses keep theirs.
#[test]
fn a_deleting_table_has_its_own_sentence() {
let err = update_table_err_to_dynamo(StorageError::TableBeingDeleted("t".to_owned()), "t");
match err {
DynamoDbError::ResourceInUseException(msg) => assert_eq!(
msg,
"Attempt to change a resource which is still in use: Table is being deleted: t"
),
other => panic!("expected ResourceInUseException, got {other:?}"),
}
let err = update_table_err_to_dynamo(StorageError::TableNotActive("t".to_owned()), "t");
match err {
DynamoDbError::ResourceInUseException(msg) => {
assert_eq!(msg, "Table t is not in ACTIVE state");
}
other => panic!("expected ResourceInUseException, got {other:?}"),
}
}
}
9 changes: 5 additions & 4 deletions crates/server/src/key_info_cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -160,10 +160,11 @@ impl CachedTableKeyInfoStore {
/// cached and is what the caller sees.
///
/// Every backend's `table_key_info` answers `TableNotFound` both for a name
/// the catalog does not have and for one it has in a state other than
/// `ACTIVE` (a `CreateTable` or `RestoreTableFromBackup` target that is still
/// `CREATING`; the data plane reports both as `ResourceNotFoundException`, as
/// the service does). Caching the second as a negative entry made reads on a
/// the catalog does not have and for one it has as `CREATING` (a `CreateTable`
/// or `RestoreTableFromBackup` target; the data plane reports both as
/// `ResourceNotFoundException`, as the service does). PostgreSQL and SQLite
/// answer the key info for a `DELETING` row, since the service serves the data
/// plane until the table is gone. Caching the second as a negative entry made reads on a
/// restored table fail for up to `negative_ttl` after `DescribeTable` reported
/// it `ACTIVE`: a probe during the copy stored the entry, and nothing
/// invalidated it at the transition (the service's first
Expand Down
Loading
Loading