fix(encryption) [18/N] wire in ags1 file length for tamper proofing - #3236
xanderbailey wants to merge 15 commits into
Conversation
| "AGS1 key metadata is missing the encrypted file length", | ||
| ) | ||
| })?; | ||
| if length < u64::from(MIN_STREAM_LENGTH) { |
There was a problem hiding this comment.
| let length = self.key_metadata.file_length().ok_or_else(|| { | ||
| Error::new( | ||
| ErrorKind::DataInvalid, | ||
| "AGS1 key metadata is missing the encrypted file length", |
There was a problem hiding this comment.
| // `Metadata::content_length()` silently returns 0 when the service did not report a | ||
| // size, and most object stores don't: S3 only populates it from the `x-amz-object-size` | ||
| // response header, which general-purpose buckets never send. A bogus 0 here would be | ||
| // written into `manifest_length` and into the AGS1 `file_length` used for truncation | ||
| // protection, making the file permanently unreadable, so trust our own byte count and | ||
| // only use the service value to detect a genuine mismatch. |
There was a problem hiding this comment.
I will admit this was not my find, claude found that S3 openDal will return 0 here so we need to keep track of the bytes written ourselves.
| manifest_list_writer.add_manifests(new_manifests.into_iter())?; | ||
| let writer_next_row_id = manifest_list_writer.next_row_id(); | ||
| manifest_list_writer.close().await?; | ||
| let file_metadata = manifest_list_writer.close().await?; |
There was a problem hiding this comment.
Now need to close the writer before we encrypt_manifest_list_key_metadata
laskoviymishka
left a comment
There was a problem hiding this comment.
The direction here is right — wiring the ciphertext length into the key metadata so reads can detect truncation without a stat is the correct fix, and switching manifest_length to the on-disk (ciphertext) size is actually more spec-correct than the old plaintext value, not just a behavior change.
I'd hold this before it lands, though, because the read-path contract and the write API shape are what the rest of this series (and cross-client interop) build on, and I'd like both nailed down before they propagate.
The one I care most about: encrypted_length() now makes file_length mandatory on every read — metadata(), reader(), and read() all fail hard without it, and there's no stat fallback. That means every encrypted file the earlier PRs in this series already wrote (without file_length) becomes permanently unreadable by iceberg-rust, not just by Java. I don't think we can fall back silently — trusting the declared length is the point — but I'd want us to pick explicitly: a warned fallback that keeps old tables readable, or an accepted hard break called out in the changelog. Right now it's silent either way.
Second, EncryptedOutputFile::write() returns the size but leaves self.key_metadata() stale, so a caller doing the old key_metadata().encode() silently writes key metadata with no length and only finds out at read time. All four call sites happen to do the dance correctly today, but that's a sharp edge for a public API.
Things I'd like to settle in this PR before merge:
- decide fallback vs. documented hard break for missing
file_lengthon read - make the
write()/key_metadata()contract hard to misuse (return the updated key metadata, or update it in place) - drop the
.expect()in the snapshot producer for a propagated error - a changelog entry covering all four
close()/write()signature changes - an oversized-
file_lengthtest so the truncation check is proven in both directions
The rest are smaller and inline. Once the read contract and the write API are settled, happy to take another pass.
| Ok(Box::new(decrypting)) | ||
| } | ||
|
|
||
| fn encrypted_length(&self) -> Result<u64> { |
There was a problem hiding this comment.
This makes file_length mandatory on every read path — metadata(), reader(), and read() all route through here now, and there's no stat fallback. So every encrypted manifest, manifest list, and puffin file the earlier PRs in this series already wrote (without file_length in the key metadata) becomes permanently unreadable by iceberg-rust, not just by Java.
I get that trusting the declared length is the whole point of truncation proofing, so I don't think we can fall back silently. But I'd want us to pick one explicitly: either fall back to self.inner.metadata() when file_length is None and warn that truncation protection is degraded (matches PyIceberg, keeps old tables readable), or accept the hard break and call it out prominently in the changelog as a one-time migration for the encryption feature.
If the feature is explicitly no-stability-yet, the second is fine — I just don't want it to be a silent break. wdyt?
There was a problem hiding this comment.
Posted below but I think the hard break is fine yet since we haven't had an iceberg-rust release with this feature enabled. WDYT? I'd like to be consistent with the Java client here if possible
| /// Write bytes to file (transparently encrypted). | ||
| pub async fn write(&self, bs: Bytes) -> Result<()> { | ||
| /// Write bytes to the file and return its encrypted size. | ||
| pub async fn write(&self, bs: Bytes) -> Result<FileMetadata> { |
There was a problem hiding this comment.
write() returns the ciphertext size, but self.key_metadata is never updated with it. So the caller has to remember to .clone().with_file_length(file_metadata.size) before encoding — all four current call sites do, but the pre-PR idiom output.key_metadata().encode() still compiles and silently produces key metadata with file_length = None, which then fails at read time with a DataInvalid that gives no hint the write path forgot the length.
That's a sharp edge for a public API. I'd either have write() hand back the updated StandardKeyMetadata alongside the FileMetadata (a small WrittenFile { file_metadata, key_metadata }), or update self.key_metadata in place so key_metadata() reflects the length after the write. Either removes the boilerplate and the footgun.
wdyt?
There was a problem hiding this comment.
Played around with a couple of approaches.
My preference is key_metadata(&self) -> key_metadata_with_length(&self, length: u64), so you can't get key metadata without supplying a length — output.key_metadata().encode() becomes impossible instead of silently emitting file_length: None. Draft: https://github.com/xanderbailey/iceberg-rust/pull/4/changes
Two honest limitations: it enforces that you supplied a length, not that it's the length of what you just wrote; and it only guards the EncryptedOutputFile route, so a bare StandardKeyMetadata::encode() is unaffected. I think this is okay since StandardKeyMetadata is really a wire type so I think it's reasonable to assume that people should be responsible for putting the correct length in it.
It does pick up two things for free — hoisting the output file past close() is the same restructure the .expect() in your other comment needed, and it lets WriterFuture and the key_metadata field go away, so it's net a little simpler. It also covers PuffinWriter::new_from_encrypted, which retains no key metadata and so already relies on the caller holding the output file.
On the two options you suggested: both target the one-shot write(), which has no production callers — every one is a test. Production goes through writer() -> Box<dyn FileWrite>. Most FileWrite impls are plain storage backends and shouldn't become encryption-aware, so a WrittenFile from write() doesn't reach the paths that matter.
That's also why the length is hard to verify: the only designs that guarantee it are ones where the writer produces the metadata, which means it stops being an opaque Box<dyn FileWrite>. Interior mutability keeps the trait object by handing an Arc<OnceLock<u64>> to AesGcmFileWrite to post back after close — but key_metadata() then returns a Result, it's a runtime check, it needs the same lifetime restructure anyway, and it couples the stream codec to the output-file abstraction. I find it harder to reason about and I'd rather have the compile error.
I also tried forcing the length at encode time: https://github.com/xanderbailey/iceberg-rust/pull/3/changes That's the broader fix — encode() is the one choke point all key metadata passes through, so it covers Parquet and any future write path. But it started as a mandatory encode(file_length: u64) and had to soften to Option<u64>: bindings/python/tests/test_encryption.py pins the wire format in both directions including the null-union tags for absent optionals, and PyIceberg's key_metadata.py declares file_length: int | None, so length-less encoding has to stay expressible. Python work is very new (last few days) so we could break that if we believe it's the right path forward.
So: xanderbailey#4 is smaller and can't express omission at all, xanderbailey#3 covers more surface but is larger and softer. I lean towards the first.
There was a problem hiding this comment.
Thinking about this more, I think xanderbailey#4 is the right way forward, I left some comments inline on that PR to explain why
There was a problem hiding this comment.
Agree here, the xanderbailey#4 looks sane to me.
@xanderbailey - shall we integrate this in this PR?
There was a problem hiding this comment.
Happy to, would like to get @blackmwk's take here too since we had some discussion about this specific architecture when we merged it initially.
There was a problem hiding this comment.
While I agree the key_metadata_with_file_length is the right direction, could we have a more meaningful name? For example key_metadta_with_saved_file_metadata(&self, file_metadata: &FileMetadata)
There was a problem hiding this comment.
Happy to do that! I'll merge that branch in here and address the remaining comments
|
|
||
| async fn close(&mut self) -> Result<()> { | ||
| Ok(()) | ||
| async fn close(&mut self) -> Result<FileMetadata> { |
There was a problem hiding this comment.
SharedMemoryWrite::close now returns the buffer length, but the write_through_ags1 helper still does writer.close().await.unwrap() and drops it, so none of the ~dozen tests using it assert that AesGcmFileWrite::close's reported size matches the actual ciphertext length. That's the primary AGS1 close path, and it's the value the whole PR hangs on.
I'd have the helper return (Vec<u8>, FileMetadata) and assert metadata.size == encrypted.len() in at least one caller. Cheap, and it pins the streaming path directly.
| /// | ||
| /// Calling close on closed file will generate an error. | ||
| async fn close(&mut self) -> Result<()>; | ||
| async fn close(&mut self) -> Result<FileMetadata>; |
There was a problem hiding this comment.
Flipping close() from Result<()> to Result<FileMetadata> on the public FileWrite trait (plus EncryptedOutputFile::write, ManifestListWriter::close, PuffinWriter::close) is a breaking change for any downstream implementor or Ok(()) matcher. That's fine for this series — I'd just make sure the changelog lists all four signature changes so it isn't a surprise.
While we're here, "stored size" is ambiguous for the encrypting wrappers, where it's the ciphertext size (larger than what the caller wrote). Worth one line saying that explicitly, since that distinction is the whole point of the PR.
| Ok(ManifestFile { | ||
| manifest_path: self.location, | ||
| manifest_length: length as i64, | ||
| manifest_length: file_metadata.size.try_into()?, |
There was a problem hiding this comment.
Good catch switching this to the on-disk size — for encrypted manifests the old content.len() was the plaintext Avro length, which is actually a spec violation (field 501 is the on-disk file length), so this is more correct, not just different.
Since it's a deliberate behavior change that's invisible for unencrypted files and only shows up encrypted, I'd add a one-line comment noting the value is now the ciphertext/on-disk length, plus a small regression test for an unencrypted manifest asserting manifest_length == plaintext_avro_len so nobody "fixes" it back later.
There was a problem hiding this comment.
iceberg-rust/crates/iceberg/src/transaction/append.rs
Lines 468 to 469 in 45925b7
| Some( | ||
| self.table | ||
| .encryption_manager() | ||
| .expect("Encryption manager must be present when key metadata exists") |
There was a problem hiding this comment.
This .expect() panics if the invariant ever breaks, and it's only sound because key_metadata.is_some() implies the manager is present — an invariant enforced by construction up in the match, not by the types. A future reorder here turns into an unrecoverable panic in library code.
Since we already matched on encryption_manager() above, I'd capture the Arc in that first arm and carry it down alongside key_metadata, so there's no second lookup and no expect. wdyt?
There was a problem hiding this comment.
I actually wrote it both ways and didn't like the way it read but I'll try again...
| // protection, making the file permanently unreadable, so trust our own byte count and | ||
| // only use the service value to detect a genuine mismatch. | ||
| let reported_size = metadata.content_length(); | ||
| if reported_size != 0 && reported_size != self.bytes_written { |
There was a problem hiding this comment.
The comment above this is great — it explains exactly why we trust bytes_written over content_length(). Given that reasoning, though, this mismatch branch never actually fires on the common production path: S3/GCS/Azure return 0, so reported_size != 0 is false and we skip it. It only triggers for in-memory operators, where the two are equal by construction.
So it reads like a cross-validation guarantee but protects nothing where it'd matter. I'd either demote it to a debug_assert! or reword the intent to "only validates when the store reports a size." Not blocking — I just don't want a future reader to trust it as a real integrity check. wdyt?
|
Thanks for the review:
iceberg-rust hasn't yet shipped a version with encryption supported so this isn't a break and it's consistent with Java. I was surprised to see that Java client hard throws in this case also I have say |
laskoviymishka
left a comment
There was a problem hiding this comment.
Almost there, the read contract and write API are settled the way I hoped, and the coverage around them is excellent. Two things from round one are still open, and once they're in I think this lands.
The one I still care most about is the .expect() in the snapshot producer. It's the same one from round one: after deferring the KMS call past close(), we re-fetch encryption_manager() and .expect() it. The invariant holds today, but it's a panic on the commit path that a future refactor can quietly turn live, and it's cheap to make it a propagated error — capturing the manager in the match arm removes both the second lookup and the .expect(). Left an inline with the shape.
The other is documenting the breaks. This PR changes four public signatures (FileWrite::close, EncryptedOutputFile::write, ManifestListWriter::close, PuffinWriter::close) and — because file_length is now mandatory on read with no stat fallback — permanently retires any encrypted file written by the earlier PRs in this series. The hard-fail itself is correct and matches Java; I'm not asking for a fallback. But both the API break and the format break need a CHANGELOG Breaking Changes entry (and a line in the PR description for the format break), so downstream and anyone on a dev build knows those tables have to be rewritten.
Two smaller things that are new this pass, neither blocking: AesGcmFileRead::new() is public but doesn't enforce the MIN_STREAM_LENGTH guard that encrypted_length() does, so a direct caller can still build a reader that skips GCM verification; and the Parquet data-file path drops the size from close(), so encrypted data files never get file_length — probably a follow-up, but worth a TODO or a tracking issue. Both inline.
Everything else I asked for in round one is in:
write()now returns the ciphertext size (thekey_metadata()sharp edge is narrowed to a design nit, inline)- the read path hard-requires
file_lengthwith no silent fallback, exactly as we discussed - the oversized-
file_lengthtest landed, and then some:test_oversized_file_length_is_rejectedandtest_oversized_declared_length_is_rejectedacross both error-on-OOB and clamping storage, plustest_truncated_file_is_rejected/test_truncated_empty_file_is_rejectedand the end-to-endappend.rscheck thatfile_lengthequals the on-disk size
Drop the .expect() and add the CHANGELOG entry and I'm happy to approve.
| Some( | ||
| self.table | ||
| .encryption_manager() | ||
| .expect("Encryption manager must be present when key metadata exists") |
There was a problem hiding this comment.
This is the .expect() from round one — still a panic on the commit path. The invariant holds today (we only set key_metadata in the Some(em) arm), but a second encryption_manager() lookup after the await is exactly the kind of thing a later refactor quietly breaks, and then a commit panics instead of erroring.
Rather than re-look-it-up, I'd capture the manager in the match arm up top and reuse it here — hold Some((em.clone(), encrypted_output.key_metadata().clone())) — then em.encrypt_manifest_list_key_metadata(&key_metadata.with_file_length(file_metadata.size)).await?. That drops the .expect() and the redundant second call in one go.
| /// Write bytes to file (transparently encrypted). | ||
| pub async fn write(&self, bs: Bytes) -> Result<()> { | ||
| /// Write bytes to the file and return its encrypted size. | ||
| pub async fn write(&self, bs: Bytes) -> Result<FileMetadata> { |
There was a problem hiding this comment.
Thanks for having write() return the size — that's the piece I wanted. The sharp edge from last round is still here though: key_metadata() keeps returning length-less metadata, so every call site has to remember the output.key_metadata().clone().with_file_length(file_metadata.size).encode() dance, and the one that forgets writes a file that only fails at read time.
All four call sites do it right today, so this isn't blocking. But I'd like the type to make it hard to get wrong — either write() returns (FileMetadata, StandardKeyMetadata) with the length already stamped, or a small WriteSummary that vends into_key_metadata(). wdyt?
| /// Minimum valid AGS1 stream length (header + one empty block). | ||
| #[cfg(test)] | ||
| pub const MIN_STREAM_LENGTH: u32 = GCM_STREAM_HEADER_LENGTH + NONCE_LENGTH + GCM_TAG_LENGTH; | ||
| pub(crate) const MIN_STREAM_LENGTH: u32 = GCM_STREAM_HEADER_LENGTH + NONCE_LENGTH + GCM_TAG_LENGTH; |
There was a problem hiding this comment.
Now that this is pub(crate), one gap worth closing while we're here: EncryptedInputFile::encrypted_length() rejects anything below MIN_STREAM_LENGTH, but AesGcmFileRead::new() is public and takes the length directly with no such check. Construct it with stream_length == GCM_STREAM_HEADER_LENGTH and num_blocks is 0, so reads return empty bytes with no GCM verification at all.
I'd push the < MIN_STREAM_LENGTH check down into new() so num_blocks >= 1 holds by construction — then the guard inside read() becomes unreachable rather than a live bypass for direct callers. Not blocking, but it closes the gap for the public entry point.
| self.0 | ||
| .close() | ||
| .await | ||
| .map(|_| ()) |
There was a problem hiding this comment.
This .map(|_| ()) is where encrypted data files fall out of the new scheme: close() now hands back the ciphertext size, but the Arrow adaptor drops it, so a data file's StandardKeyMetadata never gets file_length and the truncation check can't fire on read the way it now does for manifests.
I don't think we need to solve it here — the ArrowAsyncFileWriter surface makes it awkward — but I'd leave a // TODO(encryption): wire file_length once ArrowAsyncFileWriter can surface it or open a tracking issue so the obligation doesn't get lost. wdyt?
There was a problem hiding this comment.
For what it's worth, Java doesn't use the standard key metadata for parquet truncation. I think PME does this internally, Java does still write the parquet length into the StandardKeyMetadata.fileLength, it's just not read.
| } | ||
|
|
||
| /// Returns the optional file length. | ||
| /// Returns the encrypted file length in bytes, required for AGS1 files. |
There was a problem hiding this comment.
Small thing: the doc now says "required for AGS1 files" but the return is still Option<u64>, which reads as a contradiction. The field is genuinely optional in the spec schema and absent on anything written before this PR — it's the AGS1 read path that now hard-requires it.
Maybe "Returns the encrypted file length. Must be set before opening an AGS1 reader; absent on files written before this change." Keeps the Option honest.
| #[async_trait::async_trait] | ||
| impl FileRead for ShortRead { | ||
| async fn read(&self, range: Range<u64>) -> Result<Bytes> { | ||
| Ok(Bytes::from(vec![0; (range.end - range.start - 1) as usize])) |
There was a problem hiding this comment.
Tiny thing in the helper: range.end - range.start - 1 is unsaturated u64, so a zero-length range panics in debug and wraps to a giant allocation in release. It doesn't fire with today's ranges, but since it's shaped like a real FileRead, let len = (range.end - range.start).saturating_sub(1) as usize; avoids the trap if it's ever reused.
| pub async fn metadata(&self) -> Result<FileMetadata> { | ||
| let raw_meta = self.inner.metadata().await?; | ||
| let plaintext_size = AesGcmFileRead::calculate_plaintext_length(raw_meta.size)?; | ||
| let plaintext_size = AesGcmFileRead::calculate_plaintext_length(self.encrypted_length()?)?; |
There was a problem hiding this comment.
nit: Should we removve the async now?
| Ok(Box::new(decrypting)) | ||
| } | ||
|
|
||
| fn encrypted_length(&self) -> Result<u64> { |
There was a problem hiding this comment.
Please add comments to this method to explain why this change.
| /// Write bytes to file (transparently encrypted). | ||
| pub async fn write(&self, bs: Bytes) -> Result<()> { | ||
| /// Write bytes to the file and return its encrypted size. | ||
| pub async fn write(&self, bs: Bytes) -> Result<FileMetadata> { |
There was a problem hiding this comment.
While I agree the key_metadata_with_file_length is the right direction, could we have a more meaningful name? For example key_metadta_with_saved_file_metadata(&self, file_metadata: &FileMetadata)
laskoviymishka
left a comment
There was a problem hiding this comment.
this is looking good to me to merge.
…/key-metadata-lengths
Which issue does this PR close?
What changes are included in this PR?
I missed this in the first implementation of encryption. We have tamper proofing built into the encryption spec in the form of the file length on the
StandardKeyMetadatawhich we previously were neither reading or writing.This actually makes manifest files and lists un-readable from Java https://github.com/apache/iceberg/blob/b3756cd3876bd0ee13383c52ebd05c75df1c3dfe/core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java#L51
The perhaps controversial part of this PR is changing
FileWrite::closeto returnFileMetadata, this makes it very easy to get the number of written bytes out of the file writer and into theStandardKeyMetadata. This is a public API break but I think it's actually a nice improvement.Compatibility and API changes
AGS1 reads now require
file_lengthin key metadata and do not fall back to a storage stat. Encrypted manifests, manifest lists, and Puffin files written by earlier development builds without this field must be rewritten using a build that can still read them before upgrading. The hard-fail matches the Java client.FileWrite::close,EncryptedOutputFile::write,ManifestListWriter::close, andPuffinWriter::closenow returnResult<io::FileMetadata>containing the on-disk size.EncryptedOutputFile::key_metadata()is replaced bykey_metadata_with_saved_file_metadata(&FileMetadata), taking the result ofwrite()orclose().EncryptedInputFile::metadata()is now synchronous because it derives the plaintext size from key metadata without I/O. These changes are recorded in the changelog.Are these changes tested?
All 1,755
icebergunit tests passed, including the encrypted snapshot, manifest, manifest-list, Puffin, truncation, and oversized-length coverage. The direct AGS1 reader test now rejects every length below the minimum authenticated stream size, and an unencrypted manifest test asserts its recorded length equals the stored Avro bytes. Clippy passed foricebergandiceberg-storage-opendalwith all targets and features; formatting and the generatedicebergpublic API snapshot also pass.AI Disclosure