feat(storage): Add HDFS support via opendal services-hdfs-native

[mixermt]

Which issue does this PR close?

• Closes Support tables which live on HDFS #2440.
• Part of HDFS support in iceberg-rust #1130.

This revives #1131's successor #2441 by @jordepic (closed by the stale bot after a first review round), rebased onto the current Storage-trait layout and updated for opendal 0.58, where services-hdfs-native changed behavior in ways that required design changes (details below).

What changes are included in this PR?

Adds an opendal-hdfs-native cargo feature to iceberg-storage-opendal, with OpenDalStorageFactory::HdfsNative / OpenDalStorage::HdfsNative variants and hdfs:// routing in OpenDalResolvingStorage. The backend uses OpenDAL's services-hdfs-native (pure-Rust HDFS RPC via hdfs-native — no JNI/libhdfs). The feature is experimental and not part of opendal-all, matching opendal-oss/opendal-azdls.

NameNode resolution (differs from #2441, forced by opendal 0.58 where name_node is mandatory and its comma-split list is the HA mechanism):

1. hdfs.name-node property when set — a single endpoint or a comma-separated list for HA failover (new HDFS_NAME_NODE / HDFS_HADOOP_CONF_PREFIX constants in iceberg::io).
2. Otherwise the path authority (hdfs://host:port/path).
3. Authority-less paths without the property are rejected with a pointed error (opendal 0.58 can no longer defer to fs.defaultFS).

hadoop.-prefixed properties pass through to the HDFS client config, overriding $HADOOP_CONF_DIR values (mirroring the hadoop. catalog-property convention of the Java integrations); hdfs-native still loads core-site.xml/hdfs-site.xml from $HADOOP_CONF_DIR/$HADOOP_HOME for everything else, and Kerberos works via libgssapi_krb5 (runtime dlopen). Operators are cached per effective NameNode since each holds live RPC connections.

Relative paths are returned opendal-style without a leading / — opendal::Deleter::delete (used by delete_stream) rejects leading slashes, which an integration test caught.

Test infrastructure: single-node HDFS docker fixture (apache/hadoop:3.5.0, host networking — required because hdfs-native dials DataNodes by their registered IP, unroutable on a bridge). The DataNode healthcheck gates on NameNode registration so --wait means writable. The fixture is opt-in (profiles: [hdfs], started with COMPOSE_PROFILES=hdfs make docker-up) and the tests self-skip unless ICEBERG_TEST_HDFS_ENDPOINT is set, like the HF tests; the Linux CI job sets both.

Are these changes tested?

• 23 unit tests across config parsing and normalization, path parsing (authority/port/authority-less/non-hierarchical), NameNode precedence, operator caching, batch keys, relativize, and scheme resolution.
• 12 integration tests against the docker fixture covering exists/read/write/overwrite/metadata/range reads/streaming writes/delete/delete_prefix/delete_stream (+empty), resolving storage over hdfs://, and the HA flow (logical authority in the path + hdfs.name-node property). All 12 verified green from a cold cluster locally and skip cleanly when the endpoint is not configured; the suite runs in CI on Linux.
• make check, full-workspace --all-features lib tests, and the existing s3/gcs/resolving integration suites pass; public-api.txt regenerated for both crates.

AI Disclosure

Developed with AI assistance (Claude Code): drafting code/tests/fixtures starting from #2441, and cross-checking the design against the vendored opendal 0.58.1 / hdfs-native 0.14.5 sources. I reviewed the implementation and ran all verification locally. Areas worth reviewer attention: the NameNode-resolution semantics above (opendal's synthetic-nameservice behavior constrains what hdfs://<nameservice> paths can do without the property), and the Windows --all-features build of hdfs-native, which I could only verify via CI.

🤖 Generated with Claude Code

[blackmwk]

I found two correctness issues that need to be addressed before merge: malformed hdfs: URLs can panic, and bulk deletion can conflate NameNodes running on different ports. Details and suggested regression coverage are inline. Please also resolve the two existing review threads.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Rust maintainer. After you've addressed the points above and pushed an update, an Apache Iceberg Rust maintainer — a real person — will take the next look at the PR. The findings cite the project's review criteria; if you think one of them is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Iceberg Rust handles maintainer review: CONTRIBUTING.md.

[blackmwk]

Thanks @mixermt for this pr. I think the variant should be named to Hdfs-native rather than hdfs. Hdfs should be reserved for non native hdfs support.

[mixermt]

@blackmwk
I've addressed all the recent feedback.
Could you please give this one last review? 🙏

[comphead]

Reviewed the backend against the locked opendal-service-hdfs-native 0.58.1 and hdfs-native 0.14.5 sources, and against the existing hf/oss/azdls backends in this crate.

The overall shape fits the crate well: *_config_parse / *_batch_key / (Operator, &str) mirror hf, and the dedicated batch_key arm is genuinely needed, since host-only keying would merge NameNodes that differ only by port. The operator cache is justified too, given each client re-reads the Hadoop XML and opens RPC connections.

Leaving 5 inline comments: 2 blockers (test gating versus what the description claims, and blocking I/O under the cache write lock) and 3 majors (the cache leaking into the public API, duplicated NameNode-precedence logic, and empty-name-node handling).

I also have a handful of minor simplification notes and two questions (operator caching pins the Client to the tokio runtime that first built it, since hdfs-native captures Handle::try_current() eagerly; and the hadoop. prefix scope). Happy to add those if useful, but they seemed like noise next to the above.

[comphead]

Blocker: these tests are not #[ignore]d, and there is no CI opt-in.

The PR description says "Tests are #[ignore]d (host networking is Linux-only); CI opts in via cargo nextest --run-ignored=only -E 'test(file_io_hdfs)'". Neither is on the branch:

• no #[ignore] on any of the 12 tests in this file;
• git diff main...HEAD -- .github/ Makefile is empty, so there is no workflow step and no --run-ignored=only;
• dev/docker-compose.yaml adds hdfs-namenode/hdfs-datanode with no profiles: key, so make docker-up starts them unconditionally.

Since the Makefile has test: docker-up and ci.yml runs --all-targets --all-features, opendal-hdfs-native is enabled and all 12 tests run by default. On macOS/Windows, network_mode: "host" does not put the NameNode on the host loopback, so make test will fail for every contributor on those platforms.

Either add #[ignore] plus a Linux-only CI step as described, or put the two compose services behind a profiles: key so they are opt-in.

While here, the description's test counts look stale too: 16 unit tests in hdfs_native.rs, 3 in lib.rs, 2 in resolving.rs (not 24), and crates/iceberg/src/io/storage/config/hdfs.rs has no tests (not 3).

[mixermt]

You're right — that paragraph was left over from before the #[ignore] step was removed at @blackmwk's request, and the counts were stale. Fixed in d17ae38 by following the HF tests' approach: the HDFS tests skip unless ICEBERG_TEST_HDFS_ENDPOINT is set, the compose services sit behind a hdfs profile, and the Linux CI job opts in via env. make test now skips them on any platform. Description updated.

[comphead]

Blocker: blocking I/O runs while the cache write lock is held.

hdfs_native_operator_build -> Operator::from_config -> HdfsNativeBuilder::build() -> hdfs_native::ClientBuilder::build() -> Configuration::new(..), which does synchronous fs::read_to_string on core-site.xml/hdfs-site.xml (hdfs-native 0.14.5, src/common/config.rs:228), followed by NameServiceProxy::new. All of that happens on a tokio worker thread, under operators.write().

Two effects: the worker thread is blocked on file I/O, and every concurrent caller is serialized behind this lock, including callers that would have hit the cache for a different, already-built NameNode.

The double-checked-locking shape mirrors OpenDalResolvingStorage::resolve, but there the build is pure CPU. The cost profile is different here.

Suggested: build outside the lock and insert with entry(..).or_insert_with(..), accepting the occasional duplicate build, or move the build into spawn_blocking.

[mixermt]

Agreed, fixed in d17ae38: the operator is built outside the lock and inserted with entry().or_insert, so a racing first caller may build a duplicate that gets dropped before any I/O. One note on scope: NameServiceProxy::new doesn't connect (connections are lazy on first RPC), so the work under the old lock was two local XML reads per NameNode — still not worth holding a write lock for.

[comphead]

Major: the operator cache becomes part of the public API.

public-api.txt gains:

pub iceberg_storage_opendal::OpenDalStorage::HdfsNative::operators: alloc::sync::Arc<std::sync::poison::rwlock::RwLock<std::collections::hash::map::HashMap<alloc::string::String, opendal_core::types::operator::operator::Operator>>>

No other variant exposes internal state; they all carry only config. This pins the cache representation (including the choice of std::sync::RwLock) as a semver-relevant detail, so swapping in a different cache later becomes a breaking change.

Suggested: wrap it in a newtype with private fields, e.g. pub struct OperatorCache(RwLock<HashMap<String, Operator>>), so only an opaque type appears in the public API.

[mixermt]

Fair point. d17ae38 wraps the cache in HdfsNativeOperatorCache (private field, Default), so the public API only names that type and the internal representation can change freely.

[comphead]

Major: the NameNode-precedence rule is implemented twice.

hdfs_native_batch_key here and hdfs_native_create_operator (L99) each independently compute "configured name_node, else path authority". These two have to agree: delete_stream uses batch_key_for_path to pick the deleter and create_operator to build it, so if the rules ever drift, paths get grouped onto a deleter built against a different NameNode and deletes silently go to the wrong cluster.

Suggested: extract a single fn hdfs_native_effective_name_node(config: &HdfsNativeConfig, path: &str) -> Result<String> and have both call it. That also makes test_hdfs_native_batch_key_configured_name_node_wins and test_hdfs_native_create_operator_configured_name_node_wins redundant, since they currently assert the same rule twice.

Separately, unwrap_or_default() maps both "unparsable path" and "authority-less with no property" to "", so those share a grouping key. Harmless today because create_operator then errors on the same path, but it is a silent coupling worth not relying on.

[mixermt]

Done in d17ae38: both create_operator and the batch key go through one hdfs_native_effective_name_node. Unresolvable paths now key on the full path (same as hf_batch_key) instead of "". Dropped the duplicate precedence test.

[comphead]

Major: an empty hdfs.name-node silently defeats the path-authority fallback.

hdfs_native_config_parse (L35-37) accepts "" as-is, so config.name_node becomes Some("") and .or(authority_name_node) short-circuits, meaning the authority is never consulted. init_hdfs_config("") then filters out the empty entry and emits dfs.ha.namenodes.nameservice = "", producing a confusing failure well downstream instead of the pointed error this function is otherwise careful to give.

Worth noting that opendal's own HdfsNativeBuilder::name_node() guards this with if !name_node.is_empty(), but Operator::from_config sets the config struct directly and bypasses the setter, so that guard does not apply here.

Suggested: in hdfs_native_config_parse, .filter(|s| !s.trim().is_empty()) before assigning. Trimming a trailing / there as well would stop hdfs://nn:8020/ and a hdfs://nn:8020 path authority from producing two cache entries for the same cluster.

[mixermt]

Good catch — from_config does bypass the builder's empty-string guard. d17ae38 trims the property, strips a trailing /, and treats an empty value as unset, with tests for both cases.

[comphead]

interesting can we test on multi datanodes

[mixermt]

It's possible: with host networking, each extra DataNode needs its own dfs.datanode.{address,http.address,ipc.address} ports and data dir. I kept the fixture single-node to match opendal's own HDFS fixture — DataNode count doesn't change the FileIO adapter's behavior (failover and erasure-coded reads are hdfs-native's domain), and it's already the slowest service in make docker-up. Happy to add a two-DataNode variant as a follow-up if there's a scenario you'd like covered.

[mixermt]

Thanks @comphead for the thorough review — all five items are addressed in d17ae38 and answered inline. The security_audit failure was RUSTSEC-2026-0285 in rustls 0.23.41, which is main's current pin (AWS SDK chain, unrelated to hdfs-native; tracked in #3222). Bumped to 0.23.45 in b8cc860, the same fix #3165 applied. The minor simplification notes you held back are welcome — please send them.

"Bumped to 0.23.45 in b8cc860, the same fix #3165 applied" if we keep it without it CI will stay red, tell me if we want to revert this under this PR