[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine (stacked on this PR) · next: PITR surface (#335 rebased), catalog index, backup job, restore job.
Closes #627 (part of #624).

What

Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review, eight adversarial review rounds and the maintainer's reviews asked for before v1 freezes. It also changes main's wire deserializer for AttributeValue (crates/core/src/types/attribute_value.rs, below). One commit, squashed with #337's work (Lee is co-author); supersedes #337. Rebased onto main after #384 merged, which moved the backup id and ARN parser it introduced into this module (below). backup_format's module doc gains one sentence: the design overview states the rules the format's writers and readers honour; this module's docs and the testdata README are the record of the format itself.

Data files. DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one serialized line, not the whole file. encode_item_line_into is the one encoding of the {"Item": ...} line, into a buffer the caller keeps; encode_item_line, which the engine's ExportTableToPointInTime writer now calls, and write_item, which reuses one buffer per encoder, both call it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The cap is sized from the measured worst case at the default 400 KB item limit, a list of number sets: 67.6x, a 27.7 MB line, a 1.21x margin (operator-raised item limits above about 496 KB can exceed it). The doc carries the measured table of shapes and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted and finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush and on the decoder's source reads. GZIP_LEVEL = 6 is pinned, and a test pins the length and sha256 of a fixed input's output, so two runs of one build produce the same bytes.

DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed and uncompressed sizes are enforced while reading (a source longer than its recorded compressed size fails with the new CompressedSizeMismatch one byte past it); digests and the item count are checked at end of stream; lines are capped at MAX_LINE_BYTES and read into one buffer reused across items (it keeps the longest line's capacity, at most the cap). Decoding one line peaks at about 76 bytes of heap per line byte for a list of empty strings, and at most about 128 for any shape measured (115 for the worst item, lists of one-member map chains; 127 for nested objects that are refused once built), whatever the nesting depth: about 4 GiB at the cap (measured without serde_json's preserve_order, the default binary and cargo test -p extenddb-core; a build with the MongoDB backend enables it through bson and measures about two thirds, 51 for the flat list). DataFileDecoder's doc states the bound once. Any error is sticky and the Iterator impl is fused on top, so error-dropping consumers terminate. Line-level errors name the 1-based line in the file, and serde's column is labelled as within the line when serde's line is 1; a type descriptor or name ending in at line 3 column 4 sets serde's position, and such a message is shown as serde rendered it, with no column of its own. Items are checked for JSON shape only; the decoder does not apply the target's item limits, which the restore applies to each item before writing it. A line ends in \n, except that the last line's may be absent (that line is an item, counted in uncompressed_bytes without a newline), and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; a repeated type descriptor at the top of an attribute value is refused on this path and stays last-wins on the request path; nested maps keep JSON's last-wins); JSON whitespace, a \r before the newline included, is accepted as JSON accepts it. Lines are UTF-8 without BOM; the writer emits one gzip member, and the reader accepts concatenated members as one stream, a line spanning members included. verified() keeps the slice path that checks digests before the first item and then decodes without hashing again. CompressedHasher is the incremental hasher a restore runs while spooling a download; DataFileEntry::decoder_prehashed and ExtenddbManifest::reader_prehashed then read the spooled copy, or a file in place under an owner-only store root, without hashing it again (about 0.43 µs per 318-byte item, 9% of decoding, on a CPU without SHA extensions) and keep every size, count and line check; the digests are not compared again, so a replaced file of another length is refused by the compressed-size check and a same-length replacement is not detected.

AttributeValue deserializer (crates/core/src/types/attribute_value.rs, main's wire deserializer). The L and M arms now consume the serde_json::Value they own instead of cloning and re-deserializing it at each level. That copy had made an item PutItem accepts, 30 L wrappers around a list of 409,470 empty strings (31 containers, the maximum), cost 8.6 GB and 12.7 s to decode; the request path shares the deserializer, so this also bounds PutItem (measured in-process: 8,581 MB before, 301 MB after, on a 3.69 MB body). Results and every API error text are unchanged: a nested error is rendered once per level through serde_json::Error::custom, as main's per-level from_value did, so a message ending in at line 0 column 0 (an unknown descriptor of that name inside an L or M) loses one such suffix per level as on main; a reviewer-run differential over 600,000 text inputs and the Value path agreed (not committed), and nested_error_text_is_unchanged pins thirteen nested error texts, seven of that class taken from main's build, one with a suffix more than the levels strip. data_file_nested_decode_memory, which is not ignored, bounds a 32-level item's decode at 120 bytes of heap per line byte.

validation/number.rs (main's code, reached by every format parser and the request path): the exponent arithmetic is done in i64. An exponent at either end of i32 combined with the digits overflowed: a debug build panicked on 10E2147483647, 49E2147483647 and .00E-2147483647, and a release build reported overflow as underflow and the reverse, which is the ValidationException text the API returns. It is fixed in this PR because the format's key checks call it, so every format parser reaches it: each N key of a key_range endpoint (from_json_bytes) and each data-file item's N key (the writer's KeyRangeTracker and the reader's per-file check). Tests pin both reasons at the ends, the zero forms, and a key range and a data-file key carrying 10E2147483647 refused with the overflow reason. An invalid N key's InvalidKey message now carries that reason.

Manifest. Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes, replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. verify_manifest_checksum takes the sidecar as the bytes the store returned and accepts one shape: the line the writer stores (BackupDocuments::sidecar), <64 lowercase hex> extenddb-manifest.json, with or without its final \n; bytes that are not UTF-8 and a bare digest, which sha256sum -c refuses too, are refused. from_json_bytes requires a JSON document whose format_version is a u32, gates on it (check_format_version: 1 to CURRENT_FORMAT_VERSION; UnsupportedFormatVersion names the accepted range) before any other member is interpreted, parses, then runs ExtenddbManifest::validate, which BackupDocuments::build also runs so a writer cannot produce bytes the reader refuses: the gate; the source table (item_count and table_size_bytes not negative; table_creation_date_time finite; the key schema has at least one HASH, HASH before RANGE, each key attribute named once and defined exactly once, on every manifest, a multi-part key included, which is backed up unordered); identity: backup_arn is in canonical form and parses, backup_creation_date_time is exactly the f64 its id's instant gives, backup_name is 3–255 of the table-name alphabet, source_table.table_name is the ARN's table, and source_table.table_arn (read with core::arn::parse) is a table ARN in the same partition for the same account and table. Then the data-file list: keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes (entry errors are InvalidDataFileEntry, key errors InvalidDataFileKey), backup_size_bytes equal to the sum without overflow, a key_range per file on a key-ordered manifest (key-only endpoints of the schema's shape, first <= last, ranges ascending without overlap against every earlier file; S/B endpoints non-empty and N endpoints refused unless written normalized) and none on an unordered one; an empty table is a manifest with no files. A key_range endpoint that names an attribute twice, or whose value repeats its type descriptor ({"S":"0","S":"A"}) or names two, is refused at parse time, as a data file line of that shape is. Non-overlapping ranges do not by themselves rule out a key in two files: ExtenddbManifest::reader (and reader_prehashed) runs KeyRangeCheck over each file's items when the manifest is key-ordered, without the caller constructing it, and runs its finish after the decoder's end-of-stream checks. The check refuses a first or last item that is not the endpoint, an item not strictly after the one before, an item past last, an empty file (KeyOrderViolation) and an item whose key KeyOrder::endpoint_key refuses (InvalidKey); with the ranges, that rules duplicates out. KeyRangeCheck is a KeyRangeTracker plus the recorded range. The writer runs one tracker over every file it encodes (next_file closes a file with its KeyRange and keeps its last key, so a boundary that does not ascend is refused at the next file's first item, naming the previous file's last key, with nothing of that file encoded; finish closes the last file), so the endpoint check and strict ascent exist once, within and across files. The tracker checks keys in place with an N range key parsed once, and the recorded endpoints are parsed once: per 318-byte item 0.66 µs and 5 allocations, from 1.07 µs and 11.5 (one partition: 0.69 µs and 5.3, from 1.64 µs and 19.3). Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json holds a JsonError that bounds serde's text (which quotes the input) to 120 bytes in Display and Debug alike, and does not expose the inner error, as source() or otherwise; ChecksumMismatch prints its expected value with {:?}.

key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B standard padded base64) compared as UTF-8 bytes; range key S as UTF-8 bytes, B as raw bytes, N by exact decimal value. The module doc states the N normalization rules, the base64 alphabet, the string encoding and the precision another implementation must reproduce, and a test pins its examples. N endpoints must be valid DynamoDB numbers written in normalized form; the parser reads them as written and refuses 5.0 or 6E0. KeyOrder::resolve refuses multi-part keys, duplicate attribute definitions, a range key listed before the hash key and a range key with the hash key's name; #395 uses it as its orderability test. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings; kind is a free string the reader does not validate and a restore must not branch on. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on PostgreSQL text keys, and states what main's writers do with an invalid N: PutItem refuses it, BatchWriteItem and TransactWriteItems store it as written (#618), UpdateItem on an absent item and ImportTable run no number check on the key. A key-ordered writer that meets such a stored key refuses the backup naming the key; KeyRangeTracker::push is its one call per item and makes the check KeyOrder::endpoint_key makes, and KeyRangeCheck is that tracker plus the recorded range. The N rules state the exponent's range (a signed 32-bit integer).

Optional source_table members serialize as explicit null. global_secondary_indexes, local_secondary_indexes, vector_indexes, stream_specification and time_to_live are required members (the last three may be null); stream_specification and time_to_live record what DescribeTable and DescribeTimeToLive reported, so DescribeBackup can report SourceTableFeatureDetails as #384 does; neither is restored. OnDemandThroughput writes null for an absent member, as on the wire, and the module doc records it; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes. Golden fixtures testdata/extenddb-manifest/v1.json (every optional member set, a stream specification and a TTL description included), v1-minimal.json (every optional member absent, every list empty) and v1-unordered.json ("key_range":null files and the absent state of every nested optional member v1.json sets: OnDemandThroughput with both limits null, an SseDescription without SSEType or KMSMasterKeyArn, a GSI without ProvisionedThroughput, a disabled StreamSpecification without StreamViewType, a disabled TTL without AttributeName; and a vector index with SearchSchema and Projection, an LSI with INCLUDE), each with its .sha256, compared byte for byte; a non-ignored test binds the writer's three constructors to the checked-in bytes, so the two copies cannot drift. They are permanent read tests: the #[ignore]d writer (EXTENDDB_WRITE_GOLDEN) writes v{CURRENT} fixtures with create_new and refuses to overwrite an existing file. The evolution rule, stated on ExtenddbManifest and in the testdata README: any change to the bytes written for the fixtures' inputs is a version bump, a new member included; a new member omitted when absent does not change those bytes, so nested_types_have_the_pinned_members pins the members every manifest type reads and writes; a new or renamed enum value no fixture uses changes no byte either, so enum_values_are_pinned pins every manifest enum's values (the derived ones through serde's variant list; KeyType, ScalarAttributeType, DistanceFunction and AttributeValue's ten type descriptors, read by hand-written code, through tables filled from a match with no wildcard, the descriptors also checked to be the only ones accepted among every string of one to four upper-case letters); a new member gets #[serde(default)] and v{N} fixtures beside the v1 ones; readers accept 1..=CURRENT, so a bump excludes older readers from newer backups whatever they contain. Unknown members are tolerated at every level; one test injects one into every object of two fixtures.

DynamoDB export manifest. ExportFileEntry.etag is Option<String>; dataFileS3Key is the full store key, <prefix>/data/<file> (for a full export to a prefix the service writes <s3Prefix>/AWSDynamoDB/<export id>/data/<file>, measured; the documentation's full-export example omits the prefix; the module doc states both). DataFileEntry::from_checksums builds each entry from the encoder's result, and BackupDocuments::build(header, data_files, store_key_prefix, bucket), each data file paired with the ETag its put returned, assembles the manifest, its sidecar, manifest-files.json and manifest-summary.json from one input. It derives backup_size_bytes, backup_creation_date_time and the key prefix (the store's key prefix, every leading and trailing / dropped, then BackupArn::store_prefix; the store normalizes its configured prefix itself and refuses an empty component, so the two agree on every prefix it accepts) once and validates once. Each manifest-files.json line comes from the manifest's DataFileEntry, so the two manifests carry one md5 and one item_count, and the module doc tables every summary member's value: exportArn the backup ARN; the three instants the snapshot instant in RFC 3339 UTC with milliseconds; s3Prefix the backup prefix with every leading and trailing / dropped once (a// and /a/ are a); manifestFilesS3Key <s3Prefix>/manifest-files.json, built by the function that builds dataFileS3Key, so an empty prefix gives manifest-files.json; s3SseAlgorithm null; billedSizeBytes the compressed size; itemCount the sum. s3SseAlgorithm is Option<String>. The type documents the write order: documents_before_manifest() (the two export manifests and the sidecar), the job's re-check of its claim, then manifest_document(); documents() is the four in that order. for_backup, from_data_file, write_files_manifest, to_json_bytes and manifest_checksum are private to backup_format.

Layout: one backup id and ARN implementation. #384 added a backup id and ARN parser in extenddb_storage::util::backup_id; this PR moves it into extenddb_core::backup_format::layout unchanged in behaviour (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError), on top of core::arn::fields (the split authorization uses) and validation::validate_table_name_chars (the alphabet CreateTable uses; the engine's validate_name_member and validate_index_name still spell it themselves). format_backup_arn is the one backup-ARN formatter (storage's backup_arn(), BackupArn and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (the parser and both authorization sites call it); it trims both ends as main did, and the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn and is_backup_resource_arn. BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots). The 255-byte bound is narrower than max_table_name_length when an operator raises it; the layout doc says so, and the design has CreateBackup refuse such tables.

Why

Design review of #348 against #337 (#396, section 1): the encoder buffered each whole compressed file and hashed it in a second pass; the manifest checksum had an undocumented canonical form; etag was required but the store cannot supply one; optional fields were omitted rather than null; vector indexes were serialized as runtime descriptions; the ARN parser hardcoded arn:aws. The review rounds on this PR then found the sticky decoder made the Iterator impl non-terminating, a dropped poisoned encoder still wrote into its sink, non-DynamoDB numbers passed as range endpoints, backup_arn and source_table were never cross-checked, the encoder could write a line the decoder refuses, the streaming decoder's digest path and the base64 shape of md5 were untested, backup_creation_date_time had two definitions (the id's instant and a separate field), and the crate carried a second copy of #384's ARN parser. The fourth round found the cap's sizing claim wrong (9x, not 7.5x), a dropped open encoder completing a valid file in its sink, Interrupted fatal on flush and on the decoder's source, the time equality a half-millisecond window, unbounded untrusted fields in error messages, zero-item files accepted, duplicate attributes in a key_range kept last-wins, and a dozen validation rules no test distinguished from a weaker one. An earlier review by the maintainer then found: the cap refused items at the default limit (a list of N of 1E-130 is 47x; number sets are worse); the compressed size was unchecked until end of stream, so an oversized source was read in full; error messages measured raw length rather than the escaped form, and serde's messages were unbounded; the md5 and sidecar shape rules accepted strings that are not what they claim; overlap was checked only between neighbours and size sums could wrap; a repeated top-level attribute was kept last-wins; and several tests could not tell the rule from a weaker one. The fifth round found: the decode-memory bound was false by a factor of the nesting depth, because the AttributeValue deserializer copied each nested value at every level, so a 32-level item PutItem accepts at the default limits needed 8.6 GB to decode, in a restore and on the request path; non-overlapping key ranges did not rule out a key in two files, and nothing tied a file's items to its range; un-normalized N endpoints were read silently as normalized, so the verified bytes had two readings; KeyOrder::resolve accepted one attribute as both hash and range key; the golden fixtures missed shapes the docs said they pin, and OnDemandThroughput's null contradicted the module doc; the version rule and the fixture-regeneration rule could not both hold after a version bump; ChecksumMismatch printed its untrusted expected unescaped and FormatError::Json's Debug printed serde's whole message; data-line errors named "line 1" for every line; and several facts had more than one copy. The sixth round found that the format's evolution rules contradicted each other (the design allowed an additive member without a bump while the fixtures refused any byte change), that the writer's fixture constructors were an unbound second copy of the frozen files, that unknown-member tolerance was tested at the top level only, that the writer could serialize a manifest the reader refuses, that the N normalization rules, base64 alphabet, string encoding and comparison precision a non-Rust reader needs were stated nowhere, that the summary manifest's values for a backup were undefined, that md5 and item_count existed in two manifests with no binding, that multi-member gzip acceptance was unstated, and that the manifest could not carry the stream and TTL descriptions #384's DescribeBackup reports (#396 M2). The maintainer's review of 2026-10-28 found that the key-order doc named BatchWriteItem and TransactWriteItems among the writers that refuse an invalid N key, which they store (#618), and stated no rule for a key-ordered writer that meets one; that the doc did not state the measured dataFileS3Key shape; that one class of nested error text differed from main; that KeyRangeCheck's equal case was not equal; that an endpoint repeating its type descriptor was read last-wins; that a non-finite timestamp passed the writer and failed the reader; that no fixture held the absent state of four nested members and a new omitted member changed no bytes; that manifestFilesS3Key had a leading / for an empty prefix; and that the N rules did not state the exponent's range. The seventh round found that no test pinned the manifest's enum values, so a new TimeToLiveStatus or a renamed AES256 changed the format unnoticed; that main's number.rs overflowed on exponents at the ends of i32, reachable from every format parser; that the reader contract did not state the unterminated last line the reader accepts; that the over-strip direction of nested error texts was untested; that a// gave the summary a different prefix from the data keys; that a descriptor name could forge a data-line error's column; that the writer had no ascent check, so a snapshot out of order would publish a backup every restore refuses; that KeyRangeCheck cost 11 to 19 allocations per item; that a restore hashed every compressed byte twice; that the caller assembled four derived facts and two prefixes the format then re-checked; and that nothing made a restore run the key range check. The eighth round found that the format and the store disagreed on a leading / in the configured prefix; that AttributeValue's descriptors were the one manifest enum no test pinned; that reader_prehashed's not hashing, the summary's itemCount rule and build's own size check had no test that told them from a weaker rule; that a key-order violation at a file boundary was found only at build, after every file was uploaded; that decoder_prehashed's safety condition excluded the design's in-place read; that the decoder allocated a line buffer per item; and that the sidecar check took &str, build took a parallel etag slice and documents() did not bind the write order's split around the claim re-check. All of these are format decisions that are expensive to change after v1 ships.

Testing done

cargo test -p extenddb-core: 695 pass, 1 ignored (the fixture writer); 175 of them under backup_format (174 run). The same 695 pass in cargo test --workspace --lib, where bson turns preserve_order on. Three integration binaries in crates/core/tests: data_file_line_bomb (a 256 MiB single-line bomb stops within twice the cap's memory), data_file_nested_decode_memory (a 32-level item decodes within 120 bytes of heap per line byte; it measures about 1,200 if either the L or the M arm copies its subtree; 76 in the default build, 51 in CI's --workspace run where bson turns preserve_order on) and data_file_decode_peak (ignored; measures the decode RSS at the cap: 2.44 GiB in release). cargo test -p extenddb-storage 130, -p extenddb-engine 178, -p extenddb-server 202 pass (the re-exports, pk_to_text delegation, encode_item_line in the export writer, the authorization sites' trim_backup_arn, and the request path through the changed deserializer); cargo test -p extenddb-storage-sqlite --lib 144 pass, 1 ignored, including the test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers. The same on cargo +1.88.0 test --locked -p extenddb-core. 32 hand mutants of the rules fixed in round 4 and 39 of the round-5 fixes were each killed by the suite; 20 of the round-6 fixes were killed, one was equivalent (a dead branch in number.rs) and was replaced by three that were killed. 13 mutants of the fixes for the maintainer's review of 2026-10-28 were each killed, among them the five that review found surviving (KeyRangeCheck's equal case, a repeated endpoint descriptor, and the fixtures' absent states, the MaxReadRequestUnits case included) and a new optional member with skip_serializing_if. 15 mutants of the round-7 fixes were each killed, among them the three enum mutants that survived before that round, the per-level strip applied twice, a one-slash store_key, the tracker without its ascent check and the reader without its check. 18 mutants of the round-8 fixes were each killed at the test it targets, among them the four that survived that round (an AttributeValue descriptor alias, reader_prehashed hashing anyway, the summary's itemCount from source_table, build's size sum wrapping). The request-path memory figure is from an in-process run of the handler's two parse steps, not from the server binary.

Tests in the module cover: the streaming encoder through a short-write sink matches the in-memory encoder byte for byte; determinism and the pinned output digest; a sink error poisons the encoder and dropping it writes nothing more; unfinished-encoder drops write nothing; try_finish failure poisons; an Ok(0) sink is an error within a deadline; Interrupted on every sink and source call; the line cap refuses with the encoder usable, and the worst-case item at the default limit fits it; incremental CompressedHasher equals one-shot; the streaming decoder's end-of-stream digest check; DataFileEntry::decoder carrying every expectation; the uncompressed and compressed size caps firing early; wrong item counts in both directions; a tampered stream never completes cleanly; trailing bytes; invalid UTF-8; an unterminated final line; sticky errors and the fused iterator; a repeated top-level attribute; verified() rejects before the first item; sidecar round trip, tampering and 27 refused shapes, the bare digest and four non-UTF-8 byte strings among them; the version gate runs first and accepts 1 to CURRENT_FORMAT_VERSION only and names the range in its message; the writer's constructors equal the fixtures; unknown members at every level of two fixtures (48 injections); key-schema shape rules on an unordered manifest and a multi-part key accepted; to_json_bytes refusing an un-normalized endpoint, an invalid stored N key as an endpoint, a size off by one, a 13-digit id, a later version and a non-finite timestamp; from_checksums and from_data_file agreeing with the encoder and the entry; for_backup's members and the millisecond timestamp; the key-order doc's normalization examples; identity rules (ARN form, creation time, backup_name, source_table table, account and partition); negative source counts; required index members; key_range rules (unordered manifests, overlap against any earlier file, unsorted, non-key, un-normalized, empty S/B or wrong-type endpoints in first and last, duplicate attributes, a repeated or second type descriptor); the per-file key-range check's seven refusals (a repeated key and an invalid N key among them; a first item that is not first through reader and a file with no items through reader added) and two files with non-overlapping ranges each holding every key; the same through ExtenddbManifest::reader, sticky and fused; the writer's KeyRangeTracker refusing an out-of-order or repeated key and agreeing with KeyOrder::compare on S, N and B keys and, across files through next_file, refusing a boundary that does not ascend at the next file's first item with nothing encoded; the prehashed decoder keeping every check but the digest, its uncompressed count one too high caught at the end of the stream, and reader_prehashed reading a wrong-digest file reader refuses; BackupDocuments' four documents agreeing on one prefix and reproducing v1.json and its sidecar, each etag on its file, the three-then-one write order, a leading / dropped, the summary's itemCount as the sum when source_table.item_count differs; the pinned values of every manifest enum, AttributeValue's ten descriptors included; N exponents at both ends of i32; a forged position in a data-line error; a reused encoder line buffer; resolve refusing RANGE before HASH and a shared name; B hash keys in base64 order; bytewise-versus-length-first range-key pairs; N range keys differing beyond 17 digits; md5 shape cells; size-sum overflow; zero-item files; bounded messages at field lengths from 81 bytes to 2 MB, for keys, names, ARNs and serde's text; bounded Debug of FormatError::Json; an escaped ChecksumMismatch; line numbers in data-line errors; explicit nulls (v1-minimal.json for the source table's optional members, v1-unordered.json for "key_range":null and OnDemandThroughput's two absent members; v1.json has none); every optional nested member set in v1.json and absent in v1-unordered.json; the pinned member lists of the 18 manifest types; three-state vector_indexes; the golden manifests (three fixtures) re-serialize byte for byte; nested deserializer error texts pinned; the layout parser's id-shape grid and its two divergences from BackupArn; a differential run of the moved parser against #384's over 5.9M inputs found no difference (reviewer-run, not committed). The DynamoDB-layout fixture is constructed from the documented export shape, not captured from the service; the module doc says so.

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo clippy -p extenddb-core --all-targets -- -D warnings -D clippy::pedantic   # 0 findings under backup_format/, crates/core/tests/ and the changed lines of attribute_value.rs and validation/number.rs (clippy 1.98; 1.97's doc_markdown flags backend names in some unchanged prose)
cargo test -p extenddb-core                                                      # 695 passed, 1 ignored
cargo test -p extenddb-storage -p extenddb-engine -p extenddb-server
cargo test -p extenddb-storage-sqlite --lib
cargo +1.88.0 test --locked -p extenddb-core

Not changed here

• No writer or reader of the format exists yet outside tests and the export writer's shared line encoder; the backup job and restore PRs are the first consumers.
• The item validator (number normalization, size limits, key types against the manifest schema) is a PR 8 deliverable in extenddb-core that the restore binds (docs(backup): design revision for off-host backup and restore [backup/restore 1/4] #396); the DataFileDecoder doc states what the decoder checks and that it does not apply the target's item limits.
• The engine's account check on backup ARNs still goes through check_backup_arn (now in core) rather than the owned BackupArn; the catalog PR adopts BackupArn in the engine.
• KeyOrder::compare parses N range keys on every call; it serves validate_data_files and feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395. The per-item checks (KeyRangeTracker, KeyRangeCheck) parse each item's N once and the recorded endpoints once.
• A live DynamoDB ImportTable compatibility check of data/ is Phase 2 in the design.
• The engine's ImportTable reader (read_dynamodb_json) is a second, laxer reader of the data-file line; the data_file module doc lists the differences, and binding it is a follow-up.
• Copies of format facts in the engine and server (the name alphabet in validate_name_member and validate_index_name, the partition constants, the table-ARN prefix in Rust and SQL, is_backup_resource_arn's own split) are main's code and stay.
• LineTooLong carries no line number; the variant is shared with the encoder.
• The deserializer still builds a serde_json::Value of each top-level attribute's value before converting it; that is the remaining 76 bytes per line byte.
• ExtenddbManifest::check_files_manifest (a cross-check of manifest-files.json against the manifest) is not added; BackupDocuments::build writes both manifests from one input and nothing in ExtendDB reads manifest-files.json.
• The backup job must gather the stream specification and the TTL description at snapshot time (feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395's TableSnapshot.description says streams and TTL may be absent); that is the job PR's work.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic) for the files this PR adds
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, section 1. On-disk format: this is format version 1 before any backup has been written with it; the golden fixtures (three) are the frozen shape and the writer's constructors are bound to them by a test.

Breaking changes

None for users; no backup has been written in this format, and the moved ARN parser accepts exactly what #384's did. Relative to #337's draft API: DataFileDecoder::new takes any Read and defers verification to end of stream (use verified() for the old behavior); ExtenddbManifest loses manifest_sha256, seal, verify_manifest_sha256; BackupArnParts / parse_backup_arn are the moved #384 functions and BackupArn is the owned form; sse_specification is renamed sse_description; FormatError::Json holds a JsonError instead of a serde_json::Error, and FormatError gains KeyOrderViolation and InvalidDataFileEntry (the entry-shape reasons move there from InvalidDataFileKey); KeyRangeCheck, KeyRangeTracker, DataFileReader, BackupDocuments, ManifestHeader, encode_item_line_into, JsonError, check_format_version and KeyOrder::endpoint_key are new exports, with ExtenddbManifest::reader/reader_prehashed and DataFileEntry::decoder_prehashed; global_secondary_indexes, local_secondary_indexes and vector_indexes are required manifest members; the sidecar has a single accepted form (no bare digest); the version gate accepts 1 to CURRENT_FORMAT_VERSION; a third golden fixture, v1-unordered.json, joins the frozen shape; SourceTableSchema gains the required members stream_specification and time_to_live (both nullable; a v1 document without them is refused); ExportSummaryManifest.s3_sse_algorithm is Option<String>; validate_source_table refuses a key schema with no HASH, a HASH after a RANGE, a repeated key attribute or one without exactly one definition; UnsupportedFormatVersion's text names the range; ExtenddbManifest::validate and DataFileEntry::from_checksums are new; to_json_bytes, manifest_checksum, write_files_manifest, ExportFileEntry::from_data_file and ExportSummaryManifest::for_backup are private to backup_format, and BackupDocuments::build, which runs validate, is the writer's one entry point; KeyOrderViolation's text is "data file items break key order"; the three golden fixtures are regenerated (new bytes, new sidecars) because format version 1 is not yet written by any release. On main, crates/core/src/types/attribute_value.rs's wire deserializer is restructured (the L and M arms consume the owned Value) with identical results and error text, so no API response changes. Relative to the round-7 API: BackupDocuments::build takes Vec<(DataFileEntry, Option<String>)> and no etag slice; verify_manifest_checksum takes the sidecar as &[u8]; KeyRangeTracker is one per backup with next_file; BackupDocuments::documents_before_manifest and manifest_document are new.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

[LeeroyHannigan]

I ran the core, storage, engine and server suites on stable and 1.88.0 (633, 130, 178 and 202, as the description says), 80 mutants of the guarded rules in backup_format/, and a hostile-input harness against the decoder and the manifest parser. I compared the moved backup id and ARN code with main's over 7.8 million generated inputs and 289 wire requests and found no difference. The key order matches what PostgreSQL's ORDER BY returns with the documented recipe and what a DynamoDB Query returns, and an ExportTableToPointInTime output captured from the service parses and round-trips.

Seven items are inline. Smaller items:

• A data line that names one attribute twice is kept last-wins. ImportTable refuses the same bytes with ItemValidationError (measured), and this PR refuses duplicates in key_range and in the manifest.
• is_md5_base64 accepts 24 = characters. The sidecar check accepts a line that names a different file, a second line and upper-case hex, where the manifest requires lower-case.
• ddb_export.rs:17: the service writes the export prefix first, <s3Prefix>/AWSDynamoDB/<export id>/data/<file>.
• layout.rs:192 says the service tolerates surrounding whitespace. Measured: a trailing newline or CRLF resolves, and a leading space or tab is refused with Invalid Backup ARN. Main's parser trims both, so only the sentence is this PR's.
• KeyOrder::compare takes a stored N hash key as written, while validate_key_item refuses one that is not normalized. Say which writer guarantees the normal form, or normalize in hash_text.
• No test pins GZIP_LEVEL or the bytes it produces: level 9 and a timestamp in the gzip header both pass. A fixed input with its expected sha256 would.
• The bomb test reads the process-wide VmHWM, so an earlier test that allocates 1 GiB hides the mutant that materializes the line.

validation/number.rs overflows on an exponent such as 10E2147483647 (a panic in test builds), and the manifest parser now reaches it. That code is unchanged from main, where PutItem reaches it too, so I will file it against main.

Once the inline items are fixed, I will approve.

[LeeroyHannigan]

I re-ran the nine items from my last review at 0132dc4. PutItem refuses 1E126, abc and 1E-131 as N partition keys on this branch and on main while BatchWriteItem, TransactWriteItems and UpdateItem on an absent item store them, as the corrected paragraph says, and KeyRangeTracker and KeyRangeCheck refuse such a key with an error naming it. The deserializer gives main's values and error texts for 1,002,175 generated inputs (1,237 differed before) and on 328 wire cells; the repeated endpoint descriptor, a non-finite creation time and 0E2147483648 are refused, and an empty prefix gives manifest-files.json. I rebuilt the mutants that passed at the last head and nine more against the new fixes (15 built, 15 fail a test), and the suites pass at the description's counts on stable and 1.88.0.

[LeeroyHannigan]

LGTM 🚀