feat: support zstd metadata compression by wirybeaver · Pull Request #2894 · apache/iceberg-rust

wirybeaver · 2026-07-25T23:06:29Z

Which issue does this PR close?

Part of #2411. This implements the "Support zstd metadata compression codec" tracking item without closing the broader v3 epic.

What changes are included in this PR?

Accept write.metadata.compression-codec=zstd case-insensitively, using the existing default zstd level.
Always write canonical .zstd.metadata.json names (.gz.metadata.json for gzip). Trailing .metadata.json.zstd and .metadata.json.gz forms are accepted only when parsing existing locations for interoperability, and subsequent versions are canonicalized.
Detect gzip and standard zstd frames from content when reading metadata, independent of filenames.
Compress metadata with zstd when writing while preserving codec/location consistency checks.
Add an iceberg-go zstd metadata fixture and interoperability, error-path, canonicalization, transition, and round-trip coverage.

No catalog-specific, dependency, lockfile, or format-version changes are included.

Are these changes tested?

Yes:

cargo test -p iceberg --lib — 1473 passed
cargo fmt --all -- --check
cargo clippy -p iceberg --all-targets --all-features -- -D warnings

xanderbailey

Nice PR, thanks for working on this! Left a comment on style, let me know what you think

xanderbailey · 2026-07-27T21:31:23Z

+        let zstd_suffix = CompressionCodec::zstd_default().suffix()?;
+        let metadata_suffix = ".metadata.json";
+
+        let (stripped, compression_codec) = if let Some(stripped) =
+            file_name.strip_suffix(&format!("{metadata_suffix}{zstd_suffix}"))
+        {
+            (stripped, CompressionCodec::zstd_default())
+        } else if let Some(stripped) =
+            file_name.strip_suffix(&format!("{metadata_suffix}{gzip_suffix}"))
+        {
+            (stripped, CompressionCodec::gzip_default())
+        } else if let Some(stripped) = file_name.strip_suffix(metadata_suffix) {
+            if let Some(stripped) = stripped.strip_suffix(zstd_suffix) {
+                (stripped, CompressionCodec::zstd_default())
+            } else if let Some(stripped) = stripped.strip_suffix(gzip_suffix) {
+                (stripped, CompressionCodec::gzip_default())
+            } else {
+                (stripped, CompressionCodec::None)
+            }
        } else {
-            (stripped, CompressionCodec::None)
+            return Err(Error::new(
+                ErrorKind::Unexpected,
+                format!("Invalid metadata file ending: {file_name}"),
+            ));


Suggested change

let zstd_suffix = CompressionCodec::zstd_default().suffix()?;

let metadata_suffix = ".metadata.json";

let (stripped, compression_codec) = if let Some(stripped) =

file_name.strip_suffix(&format!("{metadata_suffix}{zstd_suffix}"))

{

(stripped, CompressionCodec::zstd_default())

} else if let Some(stripped) =

file_name.strip_suffix(&format!("{metadata_suffix}{gzip_suffix}"))

{

(stripped, CompressionCodec::gzip_default())

} else if let Some(stripped) = file_name.strip_suffix(metadata_suffix) {

if let Some(stripped) = stripped.strip_suffix(zstd_suffix) {

(stripped, CompressionCodec::zstd_default())

} else if let Some(stripped) = stripped.strip_suffix(gzip_suffix) {

(stripped, CompressionCodec::gzip_default())

} else {

(stripped, CompressionCodec::None)

}

} else {

(stripped, CompressionCodec::None)

return Err(Error::new(

ErrorKind::Unexpected,

format!("Invalid metadata file ending: {file_name}"),

));

let codecs = [

CompressionCodec::zstd_default(),

CompressionCodec::gzip_default(),

];

let (stripped, compression_codec) = codecs

.into_iter()

.find_map(|codec| {

strip_metadata_suffix(file_name, codec.suffix().ok()?).map(|s| (s, codec))

})

.or_else(|| {

file_name

.strip_suffix(METADATA_SUFFIX)

.map(|s| (s, CompressionCodec::None))

})

.ok_or_else(|| {

Error::new(

ErrorKind::Unexpected,

format!("Invalid metadata file ending: {file_name}"),

)

})?;

fn strip_metadata_suffix<'a>(file_name: &'a str, codec_suffix: &str) -> Option<&'a str> { file_name .strip_suffix(METADATA_SUFFIX) .and_then(|s| s.strip_suffix(codec_suffix)) .or_else(|| { file_name .strip_suffix(codec_suffix) .and_then(|s| s.strip_suffix(METADATA_SUFFIX)) }) }

I wonder if this kind of thing reads a little better, this is very nested and hard to follow as it currently stands

Done in fdc645c. I added the suggested strip_metadata_suffix helper and iterator-based parsing, using the shared suffix-to-codec mapping so the canonical and trailing forms stay easy to extend.

xanderbailey · 2026-07-27T21:38:28Z

On closer look at the Java docs https://iceberg.apache.org/docs/latest/configuration/#write-properties, I don't actually see zstd here... am I missing something?

xanderbailey · 2026-07-27T21:49:06Z

Go has it... apache/iceberg-go#1020

They claim in that pr that py-iceberg and java have it but I can't find it...

wirybeaver · 2026-07-28T02:15:15Z

Go has it... apache/iceberg-go#1020

They claim in that pr that py-iceberg and java have it but I can't find it...

Yeah, I refer to the iceberg-go. And #2411 also mentioned this zstd support for metadata

blackmwk · 2026-07-28T09:37:39Z

+        let zstd_suffix = CompressionCodec::zstd_default().suffix()?;
+        let metadata_suffix = ".metadata.json";
+
+        let (stripped, compression_codec) = if let Some(stripped) =
+            file_name.strip_suffix(&format!("{metadata_suffix}{zstd_suffix}"))
+        {
+            (stripped, CompressionCodec::zstd_default())
+        } else if let Some(stripped) =
+            file_name.strip_suffix(&format!("{metadata_suffix}{gzip_suffix}"))
+        {
+            (stripped, CompressionCodec::gzip_default())
+        } else if let Some(stripped) = file_name.strip_suffix(metadata_suffix) {
+            if let Some(stripped) = stripped.strip_suffix(zstd_suffix) {
+                (stripped, CompressionCodec::zstd_default())
+            } else if let Some(stripped) = stripped.strip_suffix(gzip_suffix) {
+                (stripped, CompressionCodec::gzip_default())
+            } else {
+                (stripped, CompressionCodec::None)
+            }
        } else {
-            (stripped, CompressionCodec::None)
+            return Err(Error::new(
+                ErrorKind::Unexpected,
+                format!("Invalid metadata file ending: {file_name}"),
+            ));


blackmwk · 2026-07-28T09:40:19Z

                CompressionCodec::None.name(),
-                CompressionCodec::gzip_default().name()
+                CompressionCodec::gzip_default().name(),
+                CompressionCodec::zstd_default().name()


I'm not a big fan of doing it this way, it's hard to extend and maitain. Ideally, we should have a static map in compression file, like TABLE_METADATA_SUFFIX_TO_COMPRESSION, TABLE_METADATA_MAGIC_TO_COMPRESSION, TABLE_METADATA_SUPPORTED_COMPreSSION, and reuse them in all places.

Done in fdc645c. I centralized the supported metadata codecs, suffix mappings, and magic mappings in compression.rs, then reused them for property validation, location parsing, and content-based detection. The write path now relies on the validated codec directly as well.

xanderbailey · 2026-08-09T02:13:06Z

Small heads up just to make sure there aren’t any semantic merge conflicts here, it might be worth rebasing on main since #2887 Adds some codecs.

blackmwk · 2026-09-07T09:36:50Z

+pub(crate) const TABLE_METADATA_SUPPORTED_COMPRESSION: &[CompressionCodec] = &[
+    CompressionCodec::None,
+    CompressionCodec::Gzip(GZIP_DEFAULT_LEVEL),
+    CompressionCodec::Zstd(ZSTD_DEFAULT_LEVEL),
+];
+
+pub(crate) const TABLE_METADATA_SUFFIX_TO_COMPRESSION: &[(&str, CompressionCodec)] = &[
+    (ZSTD_SUFFIX, CompressionCodec::Zstd(ZSTD_DEFAULT_LEVEL)),
+    (GZIP_SUFFIX, CompressionCodec::Gzip(GZIP_DEFAULT_LEVEL)),
+];
+
+pub(crate) const TABLE_METADATA_MAGIC_TO_COMPRESSION: &[(&[u8], CompressionCodec)] = &[
+    (GZIP_MAGIC, CompressionCodec::Gzip(GZIP_DEFAULT_LEVEL)),
+    (ZSTD_MAGIC, CompressionCodec::Zstd(ZSTD_DEFAULT_LEVEL)),
+];


I think they should be const fn functions of CompressionCodec?

Replaced the static metadata codec maps with associated const helpers on CompressionCodec in 89da77d.

The codec list, suffix mapping, and magic mapping now live behind CompressionCodec methods and are reused by property validation, location parsing, and content-based reads.

blackmwk · 2026-09-07T09:38:16Z

+        serde_json::from_value(serde_json::Value::String(lowercase_value))
+            .map_err(|_| invalid_metadata_compression_codec(value))?;
+
+    if TABLE_METADATA_SUPPORTED_COMPRESSION.contains(&codec) {


This is obviously incorrect, you missed compression level, see https://github.com/apache/iceberg-rust/pull/2894/changes#r3948429401

Made metadata compression support validation level-aware in 89da77d.

The check now matches Gzip() and Zstd() instead of comparing against default-level enum values. Tests cover non-default Gzip and Zstd levels to prevent this regression.

Add zstd metadata codec validation, canonical filename generation, alternate suffix parsing, and content-based frame detection for reads. Cover interoperability, codec transitions, truncated frames, and semantic round trips. Part of apache#2411.

Reuse shared supported-codec, suffix, and magic mappings across metadata property validation, location parsing, and I/O. Simplify alternate suffix parsing with a dedicated helper.

wirybeaver mentioned this pull request Jul 25, 2026

EPIC: v3 Support Tracking #2411

Open

xanderbailey reviewed Jul 27, 2026

View reviewed changes

blackmwk reviewed Jul 28, 2026

View reviewed changes

wirybeaver force-pushed the iceberg-rs-zstd branch from a2b816e to c8cb597 Compare September 4, 2026 17:52

blackmwk reviewed Sep 7, 2026

View reviewed changes

wirybeaver added 3 commits September 15, 2026 14:48

feat: support zstd metadata compression

33c2e55

Add zstd metadata codec validation, canonical filename generation, alternate suffix parsing, and content-based frame detection for reads. Cover interoperability, codec transitions, truncated frames, and semantic round trips. Part of apache#2411.

refactor: centralize metadata compression mappings

7f1e95e

Reuse shared supported-codec, suffix, and magic mappings across metadata property validation, location parsing, and I/O. Simplify alternate suffix parsing with a dedicated helper.

fix: make metadata codec helpers level-aware

89da77d

wirybeaver force-pushed the iceberg-rs-zstd branch from c8cb597 to 89da77d Compare September 15, 2026 14:59

Uh oh!

Conversation

wirybeaver commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

xanderbailey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xanderbailey commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xanderbailey commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wirybeaver commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xanderbailey commented Aug 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wirybeaver Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wirybeaver Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wirybeaver commented Jul 25, 2026 •

edited

Loading

xanderbailey commented Jul 27, 2026 •

edited

Loading

xanderbailey commented Jul 27, 2026 •

edited

Loading

wirybeaver commented Jul 28, 2026 •

edited

Loading

wirybeaver Sep 15, 2026 •

edited

Loading

wirybeaver Sep 15, 2026 •

edited

Loading