Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions crates/core/src/validation/number.rs
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,9 @@ pub fn validate_and_normalize_number(s: &str) -> Result<String, DynamoDbError> {
// digits (`0E126` is an overflow, `0.0E-130` an underflow). Only a zero
// whose exponent does not fit in an i32 is checked here. A refused
// zero keeps its raw spelling in `AttributeValue`, as every refused
// number does, and Query, Scan (#544, #562) and ImportTable do not
// validate numbers yet: refusing `0E126` here would make them look it
// up and store it as a key of its own instead of `0` (#618).
// number does, and Query and Scan (#544, #562) do not validate
// numbers yet: refusing `0E126` here would make them look it up as a
// key of its own instead of `0` (#618).
if i32::try_from(explicit_exp).is_err() {
check_zero_magnitude(explicit_exp, frac_part)?;
}
Expand Down Expand Up @@ -548,13 +548,12 @@ mod tests {

// DynamoDB refuses each of these zeros: `0E126` and the next four are a
// `Number overflow`, `0.0E-130` and the rest a `Number underflow`. They
// stay accepted as 0 until Query and Scan (#544, #562) and ImportTable
// validate numbers, which #618 still tracks: a zero refused by this
// function keeps its raw spelling, and those paths would then look up and
// store `0E126` as a key of its own. Move these rows to the refused lists
// above when they do.
// stay accepted as 0 until Query and Scan (#544, #562) validate numbers,
// which #618 still tracks: a zero refused by this function keeps its raw
// spelling, and those operations would then look up `0E126` as a key of
// its own. Move these rows to the refused lists above when they do.
#[test]
fn zero_with_i32_exponent_stays_accepted_until_query_scan_and_import_validate() {
fn zero_with_i32_exponent_stays_accepted_until_query_and_scan_validate() {
for n in [
"0E126",
"00E126",
Expand Down
87 changes: 87 additions & 0 deletions crates/core/src/validation/number/operations.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
//! a 32-bit exponent and scale; see [`validate_number_i32_scale`]. An empty
//! value answers the message without the colon.
//!
//! `ImportTable` reads the numbers of a row the second way (measured
//! 2026-11-03; see [`import_row`]).
//!
//! `PutItem`, `UpdateItem` and `DeleteItem` start the message with
//! `1 validation error detected: `, and the other operations do not.
//! `TransactWriteItems` reports a value in `ExpressionAttributeValues` as
Expand Down Expand Up @@ -214,6 +217,90 @@ pub fn transact_write_item(item: &TransactWriteItem) -> Result<(), DynamoDbError
item_numbers(Rules::I32Scale, key_or_item).map_err(bare)
}

/// Why `ImportTable` refuses a row holding a number the service cannot read.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ImportRowFault {
/// A value that is not a number. The service refuses the row and does not
/// process the rows after it ("Expected a valid numeric. Remainder of the
/// file will not be processed.").
Malformed,
/// A number outside the supported range or with more than 38 digits, with
/// the service's message for it. The service refuses the row and imports
/// the rows after it.
OutOfRange(String),
}

/// Refuse an `ImportTable` row holding a number the service cannot read.
///
/// Measured 2026-11-03 (us-east-1), one import from S3 per value, each file a
/// row with a valid number, the row under test, and another valid row:
///
/// - The service reads a row's numbers as `BatchWriteItem` does, with a
/// 32-bit exponent and scale; see [`validate_number_i32_scale`]. `abc`,
/// ` 5`, an empty value, `12e`, `1E2147483648` and `1E-2147483648` are not
/// numbers, and `1E2147483647` is a `Number underflow`.
/// - A row holding a value that is not a number stops the import of the
/// file: the row and the rows after it are not imported, and the row is
/// the last one counted as processed. This holds whatever else the row
/// holds: a number out of range ahead of it in the row, or no key.
/// - A row holding a number out of range, or with more than 38 digits, is
/// not imported and the rows after it are.
///
/// Either way the row counts as one error, and numbers in an `NS`, `L` or `M`
/// count as the row's numbers. A row with no key and the number `1E126` was
/// logged as `Number overflow`, and a row whose key has the wrong type and
/// the number `1E126` as the type mismatch; the caller checks numbers before
/// keys, so only the second row's log line differs.
///
/// Two values are read differently from the service:
///
/// - A zero whose exponent fits in 32 bits reaches this check as `0` (see
/// [`validate_number_i32_scale`]), so a row holding `0E126`, which the
/// service refuses as `Number overflow`, is imported with `0`.
/// - The service reads a Unicode decimal digit as the digit (a row holding
/// `\u{ff15}` was imported with `5`). This check does not, as no other
/// number check does, so such a row ends the import of the file.
///
/// # Errors
///
/// Returns the row's [`ImportRowFault`]: `Malformed` when any value is not a
/// number, and otherwise the first number out of range, in the order of the
/// item's map.
pub fn import_row(item: &Item) -> Result<(), ImportRowFault> {
let mut numbers = Vec::new();
for value in item.values() {
collect_numbers(value, &mut numbers);
}
let mut out_of_range = None;
for n in numbers {
match validate_number_i32_scale(n) {
Ok(()) => {}
// Every message that check gives for a value that is not a number
// starts with this text, and none of its range messages does; the
// tests pin the split for every measured value.
Err(message) if message.starts_with(NUMERIC_VALUE) => {
return Err(ImportRowFault::Malformed);
}
Err(message) => {
out_of_range.get_or_insert(message);
}
}
}
out_of_range.map_or(Ok(()), |message| Err(ImportRowFault::OutOfRange(message)))
}

/// Every number in `value`: an `N`, the members of an `NS`, and the numbers
/// nested in an `L` or `M`.
fn collect_numbers<'a>(value: &'a AttributeValue, numbers: &mut Vec<&'a str>) {
match value {
AttributeValue::N(n) => numbers.push(n),
AttributeValue::NS(set) => numbers.extend(set.iter().map(String::as_str)),
AttributeValue::L(list) => list.iter().for_each(|v| collect_numbers(v, numbers)),
AttributeValue::M(map) => map.values().for_each(|v| collect_numbers(v, numbers)),
_ => {}
}
}

/// The first fault in an `ExpressionAttributeValues` map: a value nested too
/// deep, then a number the service cannot read. That is the order `PutItem`
/// and `DeleteItem` answered before these checks; the service answered the
Expand Down
101 changes: 101 additions & 0 deletions crates/core/src/validation/number/operations/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -587,3 +587,104 @@ fn batch_write_item_visits_tables_in_name_order() {
.unwrap();
assert_eq!(refused(batch_write_item(&input)), format!("{NUMERIC}: abc"));
}

// `ImportTable` rows: measured 2026-11-03 (us-east-1), one import from S3 per
// row. A row is read through `Item`'s deserializer, as the import file
// reader reads it, so a valid number arrives normalized and any other
// keeps its text.

fn import_item(row: &Value) -> Item {
serde_json::from_value(row.clone()).unwrap()
}

#[test]
fn import_row_stops_at_a_value_that_is_not_a_number() {
for row in [
json!({"pk": {"S": "a"}, "n": {"N": "abc"}}),
json!({"pk": {"S": "a"}, "n": {"N": ""}}),
json!({"pk": {"S": "a"}, "n": {"N": " 5"}}),
json!({"pk": {"S": "a"}, "n": {"N": "12e"}}),
json!({"pk": {"S": "a"}, "n": {"N": "1E2147483648"}}),
json!({"pk": {"S": "a"}, "n": {"N": "1E9223372036854775808"}}),
json!({"pk": {"S": "a"}, "n": {"N": "1E-2147483648"}}),
json!({"pk": {"S": "a"}, "n": {"N": "1.5E-2147483647"}}),
json!({"pk": {"S": "a"}, "s": {"NS": ["1", "abc"]}}),
json!({"pk": {"N": "abc"}}),
// A number out of range ahead of it in the row does not change the
// answer, nor does a missing key.
json!({"pk": {"S": "a"}, "a": {"N": "1E126"}, "b": {"N": "abc"}}),
json!({"other": {"S": "x"}, "n": {"N": "abc"}}),
// The service imports a row holding a Unicode decimal digit with the
// digit (`5` here); this check refuses it, as every number check does.
json!({"pk": {"S": "a"}, "n": {"N": "\u{ff15}"}}),
] {
assert_eq!(
import_row(&import_item(&row)),
Err(ImportRowFault::Malformed),
"{row}"
);
}
}

#[test]
fn import_row_refuses_a_number_out_of_range() {
let ones = "1".repeat(39);
for (row, message) in [
(json!({"pk": {"S": "a"}, "n": {"N": "1E126"}}), OVERFLOW),
(
json!({"pk": {"S": "a"}, "n": {"N": "1E999999999"}}),
OVERFLOW,
),
(json!({"pk": {"S": "a"}, "n": {"N": "1E-131"}}), UNDERFLOW),
(json!({"pk": {"S": "a"}, "n": {"N": "-1E-131"}}), UNDERFLOW),
// The magnitude passes the 32-bit maximum and wraps to a negative.
(
json!({"pk": {"S": "a"}, "n": {"N": "1E2147483647"}}),
UNDERFLOW,
),
(json!({"pk": {"S": "a"}, "n": {"N": ones}}), DIGITS),
(
json!({"pk": {"S": "a"}, "s": {"NS": ["1", "1E126"]}}),
OVERFLOW,
),
(
json!({"pk": {"S": "a"}, "m": {"M": {"x": {"N": "1E126"}}}}),
OVERFLOW,
),
(
json!({"pk": {"S": "a"}, "l": {"L": [{"N": "1E-131"}]}}),
UNDERFLOW,
),
(json!({"pk": {"N": "1E126"}}), OVERFLOW),
(json!({"other": {"S": "x"}, "n": {"N": "1E126"}}), OVERFLOW),
] {
assert_eq!(
import_row(&import_item(&row)),
Err(ImportRowFault::OutOfRange(message.to_owned())),
"{row}"
);
}
}

#[test]
fn import_row_accepts_what_the_service_imports() {
for row in [
json!({"pk": {"S": "a"}, "n": {"N": "5"}}),
json!({"pk": {"S": "a"}, "n": {"N": "1.50"}}),
json!({"pk": {"S": "a"}, "n": {"N": "-0.0"}}),
json!({"pk": {"S": "a"}, "n": {"N": "+5"}}),
json!({"pk": {"S": "a"}, "n": {"N": "1.5E2"}}),
json!({"pk": {"S": "a"}, "n": {"N": "0E125"}}),
json!({"pk": {"S": "a"}, "n": {"N": "9.9999999999999999999999999999999999999E125"}}),
json!({"pk": {"S": "a"}, "n": {"N": "1E-130"}}),
json!({"pk": {"S": "a"}, "s": {"NS": ["1.0", "2"]}}),
json!({"pk": {"N": "2.50"}}),
json!({"pk": {"S": "a"}, "b": {"BOOL": true}, "s": {"S": "abc"}}),
] {
assert_eq!(import_row(&import_item(&row)), Ok(()), "{row}");
}
// The service refuses `0E126` as `Number overflow`; the row's value is
// `0` until the 64-bit reader range checks every zero (#618).
let zero = json!({"pk": {"S": "a"}, "n": {"N": "0E126"}});
assert_eq!(import_row(&import_item(&zero)), Ok(()));
}
101 changes: 91 additions & 10 deletions crates/engine/src/import_export.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ use extenddb_core::types::{
CreateTableInput, ExportFormat, ImportStatus, ImportTableDescription, ImportTableInput,
ImportTableOutput, Item, TableCreationParameters,
};
use extenddb_core::validation::number::operations::{ImportRowFault, import_row};
use extenddb_core::validation::{
validate_attribute_name_sizes, validate_item_keys, validate_item_nesting_depth,
validate_item_size, validate_key_sizes,
Expand Down Expand Up @@ -119,33 +120,96 @@ pub async fn handle_import_table(

let mut imported_count: i64 = 0;
let mut error_count: i64 = 0;
let processed_count = i64::try_from(items.len()).unwrap_or(i64::MAX);
let mut processed_count: i64 = 0;
// Created before the rows are read so that each row's log line names the
// import it belongs to; `item_number` counts items, not file lines.
let import_arn = format!("{}:import/{}", table_arn, uuid::Uuid::new_v4());
let table_name = tcp.table_name.as_str();

for item in items {
processed_count += 1;
// Numbers before the other checks: a row holding a value that is not
// a number ends the import of the file whatever else the row holds,
// and a row holding a number out of range is skipped (see
// `import_row`).
match import_row(&item) {
Ok(()) => {}
Err(ImportRowFault::OutOfRange(message)) => {
tracing::warn!(
import_arn = ?import_arn,
table_name = ?table_name,
item_number = processed_count,
error = %message,
"import: skipping item with a number out of range"
);
error_count += 1;
continue;
}
Err(ImportRowFault::Malformed) => {
tracing::warn!(
import_arn = ?import_arn,
table_name = ?table_name,
item_number = processed_count,
"import: item holds a value that is not a number; the rest of the file is not imported"
);
error_count += 1;
break;
}
}
if let Err(e) =
validate_item_keys(&item, &key_info.key_schema, &key_info.attribute_definitions)
{
tracing::warn!(error = %e, "import: skipping item with invalid keys");
tracing::warn!(
import_arn = ?import_arn,
table_name = ?table_name,
item_number = processed_count,
error = ?e.to_string(),
"import: skipping item with invalid keys"
);
error_count += 1;
continue;
}
if let Err(e) = validate_item_nesting_depth(&item) {
tracing::warn!(error = %e, "import: skipping item with excessive nesting");
tracing::warn!(
import_arn = ?import_arn,
table_name = ?table_name,
item_number = processed_count,
error = ?e.to_string(),
"import: skipping item with excessive nesting"
);
error_count += 1;
continue;
}
if let Err(e) = validate_item_size(&item, ctx.limits.max_item_size_bytes) {
tracing::warn!(error = %e, "import: skipping oversized item");
tracing::warn!(
import_arn = ?import_arn,
table_name = ?table_name,
item_number = processed_count,
error = ?e.to_string(),
"import: skipping oversized item"
);
error_count += 1;
continue;
}
if let Err(e) = validate_attribute_name_sizes(&item, &ctx.limits) {
tracing::warn!(error = %e, "import: skipping item with oversized attribute name");
tracing::warn!(
import_arn = ?import_arn,
table_name = ?table_name,
item_number = processed_count,
error = ?e.to_string(),
"import: skipping item with oversized attribute name"
);
error_count += 1;
continue;
}
if let Err(e) = validate_key_sizes(&item, &key_info.key_schema, &ctx.limits) {
tracing::warn!(error = %e, "import: skipping item with oversized key");
tracing::warn!(
import_arn = ?import_arn,
table_name = ?table_name,
item_number = processed_count,
error = ?e.to_string(),
"import: skipping item with oversized key"
);
error_count += 1;
continue;
}
Expand All @@ -159,11 +223,28 @@ pub async fn handle_import_table(
}

let end_time = epoch_seconds();
let import_arn = format!("{}:import/{}", table_arn, uuid::Uuid::new_v4());

// An import that refused any row is FAILED with `ItemValidationError`,
// and the rows it accepted stay in the table (measured 2026-11-03). The
// service's message sends the reader to its error log in CloudWatch;
// ExtendDB writes each refused row to the server log instead.
let (import_status, failure_code, failure_message) = if error_count > 0 {
(
ImportStatus::Failed,
Some("ItemValidationError".to_owned()),
Some(
"Some of the items failed validation checks and were not imported. \
Please check the server log for more details."
.to_owned(),
),
)
} else {
(ImportStatus::Completed, None, None)
};

let description = ImportTableDescription {
import_arn,
import_status: ImportStatus::Completed,
import_status,
table_arn,
table_id: Some(table_id),
file_source: input.file_source,
Expand All @@ -174,8 +255,8 @@ pub async fn handle_import_table(
imported_item_count: imported_count,
start_time: Some(start_time),
end_time: Some(end_time),
failure_code: None,
failure_message: None,
failure_code,
failure_message,
};

serialize_output(&ImportTableOutput {
Expand Down
1 change: 1 addition & 0 deletions docs/differences-from-dynamodb.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ adaptation when switching between ExtendDB and the real service.
| Export formats | DYNAMODB_JSON, ION | DYNAMODB_JSON, ION |
| Import execution | Asynchronous (background job) | Synchronous (completes before returning) |
| Export execution | Point-in-time snapshot | Current snapshot, synchronous |
| Imported rows holding a number the service cannot store | A row holding a value that is not a number ends the import of the file: it and the rows after it are not imported. A row holding a number out of range or with more than 38 digits is skipped, and the rows after it are imported. Each refused row counts as one error and is logged to CloudWatch, and an import with any error is `FAILED` with `ItemValidationError`, keeping the rows it imported (measured 2026-11-03) | The same, with three differences. A zero whose exponent fits in 32 bits but is out of range, such as `0E126`, is imported as `0`, where the service skips the row (`Number overflow` for `0E126`). A Unicode decimal digit (`\uff15`), which the service imports as the digit (`5`), is not a number to ExtendDB, so the row ends the import of the file. Each refused row is logged to the server log, and the `FailureMessage` names the server log instead of CloudWatch |

## Control Plane

Expand Down
Loading
Loading