feat(python) add standard key metadata python bindings

[xanderbailey]

Which issue does this PR close?

• Closes #.

As discussed in apache/iceberg-python#3948. We think it might be a good idea to push as much of the core logic for encryption in iceberg-python into iceberg-rust. This PR starts with adding encode / decode for standard key metadata. I don't want the iceberg rust side to block progress on the python side but I will track moving things over to the rust implementation as versions of iceberg-rust are published.

What changes are included in this PR?

Registers new encryption python module, exposing encode_standard_key_metadata and decode_standard_key_metadata.

decode_standard_key_metadata returns a frozen StandardKeyMetadata class with named encryption_key / aad_prefix / file_length getters, rather than a bare 3-tuple, so callers can't confuse the two bytes fields positionally. Its __repr__ redacts the key, matching pyiceberg's field(repr=False) and the core crate's SecureKey Debug impl.

Are these changes tested?

Yes — unit tests in bindings/python/tests/test_encryption.py.

The wire format is pinned in both directions across the same four cases as apache/iceberg-python#3948, including the null union tags written for absent optional fields and the empty-but-present aad_prefix, so the byte-level interop with Java and PyIceberg is covered rather than just the round trip. Invalid key lengths are rejected on both the encode and decode paths.

Upstream Avro follow-up

apache-avro 0.21 (pinned here) and 0.22 decode a missing or truncated union tag as null instead of erroring, so a truncated payload decodes as though its optional fields were simply absent. This is pre-existing behaviour in the core crate rather than something introduced here, and it is already fixed upstream by apache/avro-rs#664 (merged, milestone 0.23.0), so no Iceberg-specific workaround is needed. Thanks @kevinjqliu for digging out the upstream fix.

test_decode_rejects_truncated_union_tags pins the current behaviour as a strict=True xfail. Once apache-avro is bumped to 0.23 the decode will start erroring, the test will XPASS, and the strict marker will fail CI as a prompt to drop it.

AI Disclosure

[kevinjqliu]

LGTM

I pushed a small change to align the error message with python's

I also built the binding and tested locally against pyiceberg, all the existing tests from apache/iceberg-python#3948 passed

[kevinjqliu]

lets see if others have any feedback on this PR

[kevinjqliu]

apache-avro 0.21 and 0.22 can decode missing or truncated union tags as null. This pre-existing decoder issue will be addressed upstream rather than with an Iceberg-specific workaround. Upstream issue: TODO: add the apache/avro-rs issue link once filed.

BTW this issue was also flagged by my agent and I looked into it. There's already a fix in the main branch: apache/avro-rs#664

[laskoviymishka]

This is clean and I'd merge it — the submodule mirrors the manifest/transform registration pattern, the wrappers are thin, and the pinned wire-format bytes check out against the V1 schema and Java's null-first union ordering, so the interop story holds up.

kevinjqliu's already approved, aligned the error message with pyiceberg, and confirmed #3948's tests pass against the binding, so everything here is a second opinion rather than a gate — none of it blocks merge.

The one I'd most want a look at is decode robustness. The underlying decoder silently accepts trailing bytes after a valid datum, and (per your own TODO) decodes truncated union tags as None instead of erroring — both pre-existing in the core crate, but this PR is the first public surface over them, and silently accepting corrupted key material is worth pinning with a test now even if the fix is upstream. I'd add a strict xfail for the truncated-union case and file the issue the TODO references so it has somewhere to point.

Smaller things: the bare 3-tuple return is an easy footgun for an encryption API (positional encryption_key vs aad_prefix) — I'd at least document the positions, ideally a named type; FeatureUnsupported collapsing to ValueError loses a distinction Python callers could use; and the wire-format test only pins the encode direction with both optionals present, so the decode direction and the null-union cases (the interop-critical ones) aren't covered.

[laskoviymishka]

The bare 3-tuple leaves Python callers with positional access only, and for an encryption API the confusion between encryption_key and aad_prefix (both bytes) is a real footgun — decoded[0] vs decoded[1] is easy to get wrong, and switching to a named type later is a breaking change.

I know manifest/transform expose named classes (PyManifest, PyTransform) while this stays a tuple, so there's precedent both ways. I'd lean toward a small #[pyclass] with named getters here given what these bytes are, but at minimum I'd document the positions on the type alias and in the decode_standard_key_metadata docstring: (encryption_key, aad_prefix, file_length). wdyt?

[xanderbailey]

2634e97

[xanderbailey]

This does come with a small allocation cost I think but I'm happy with the change

[laskoviymishka]

to_py_err collapses everything to PyValueError, so the unsupported-version path (which is FeatureUnsupported) surfaces as a ValueError a caller can't distinguish from malformed data. It's the one error kind here with a natural Python counterpart, so I'd map it through a small local helper:

fn key_metadata_err(err: iceberg::Error) -> PyErr {
    match err.kind() {
        ErrorKind::FeatureUnsupported => PyNotImplementedError::new_err(err.message().to_string()),
        _ => to_py_err(err),
    }
}

While we're here — err.to_string() prepends the kind, so str(exc) reads FeatureUnsupported => Unsupported key metadata version: …, which diverges from pyiceberg's plain message; err.message() drops the prefix. No regression for the pyiceberg migration since it also raises ValueError, so this is forward-looking, not blocking.

[xanderbailey]

#3226 has landed so I don't think a change is needed here

[laskoviymishka]

The error-string assertions are pulling in two directions here — this one pins nearly the whole sentence (now owned in three places: the Rust format!, the Rust assert_eq!, and this match=), while test_decode_rejects_empty_buffer and test_encode_rejects_invalid_key_length pin nothing and pass on any ValueError.

I'd match a minimal stable fragment throughout — match=r"version: 2" here, and add match="Empty key metadata buffer" / match="key length" to the two that currently assert type only. That gets you a message check on all three without the full-sentence coupling.

[laskoviymishka]

This pins only the encode direction with both optionals present, but the decode direction against known-good bytes is the actual interop guarantee (us reading Java-produced bytes), and the null-union encoding for absent fields — the \x00 byte a Java/PyIceberg reader expects — is never exercised. A null-branch bug would round-trip fine here but break cross-client.

pyiceberg #3948 pins four cases; I'd mirror them:

assert encryption.decode_standard_key_metadata(
    b"\x01\x20" + AES128_KEY + b"\x02\x04ad\x02\x80\x10"
) == (AES128_KEY, b"ad", 1024)
assert encryption.encode_standard_key_metadata(AES128_KEY) == b"\x01\x20" + AES128_KEY + b"\x00\x00"
assert encryption.encode_standard_key_metadata(AES128_KEY, b"ad") == b"\x01\x20" + AES128_KEY + b"\x02\x04ad\x00"

The last one also covers the aad-present/file-length-absent combination that isn't tested today.

[laskoviymishka]

Your PR body notes apache-avro 0.21 decodes a missing/truncated union tag as null rather than erroring, so b"\x01\x20" + AES128_KEY (version + valid key, union tags omitted) likely decodes as (key, None, None) — indistinguishable from a legit key-only payload. Without a test anchoring that current behavior, there's no regression signal when the upstream fix lands.

I'd add it as a strict xfail alongside these rejection tests so it flips to passing once fixed:

@pytest.mark.xfail(reason="apache-avro#NNN: truncated union tags decode as null", strict=True)
def test_decode_rejects_truncated_union_tags():
    with pytest.raises(ValueError):
        encryption.decode_standard_key_metadata(b"\x01\x20" + AES128_KEY)

The PR-body TODO says the upstream issue isn't filed yet — worth filing it so this reason can link somewhere real. Pre-existing core behavior, not something to hold merge on.

[xanderbailey]

I have another PR up to fix the trailing bytes issue #3225

[xanderbailey]

Also resolving the unsupported error issue here https://github.com/apache/iceberg-rust/pull/3226/changes. I think that's just the wrong error to be throwing here

[kevinjqliu]

very excited for the next release of pyiceberg-core 🥳