What is the problem the feature request solves?
Comet's native Iceberg scan cannot read a table whose data lives on HDFS. hdfs:// is not in the scheme allowlist that mirrors storage_factory_for, so any Iceberg table on HDFS falls back to the JVM reader and gets no native acceleration.
This is a gap relative to the plain-Parquet native scan, which does read HDFS (through libhdfs/JNI, via fs.comet.libhdfs.schemes). On-premise Iceberg deployments are commonly HDFS-backed, so the Iceberg acceleration Comet offers is unavailable to them today.
Describe the potential solution
Route hdfs:// to iceberg-rust's hdfs-native OpenDAL backend, a pure-Rust HDFS RPC client that needs no JNI or libhdfs. That backend is being added upstream in apache/iceberg-rust#3111; this issue tracks the Comet-side work once it lands.
Comet-side work:
- Add an
hdfs arm to storage_factory_for and admit the scheme in the JVM gates (CometScanRule.icebergReadableSchemes, and CometIcebergNativeWrite.SupportedStorageSchemes for the write path).
- Forward the
hdfs./hadoop. property prefixes to the native FileIO.
- Resolve the NameNode endpoints for HA clusters. opendal's
HdfsNativeBuilder never dials the authority written in the path: it builds one client against a synthetic authority and synthesizes the HA config from the comma-separated hdfs.name-node value. iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a real host:port. An HA location reads hdfs://<nameservice>/..., and a nameservice is not a routable host, so the endpoints have to be derived from dfs.ha.namenodes.<ns> / dfs.namenode.rpc-address.<ns>.<nn> in the session Hadoop configuration.
- Decline a scan whose data/delete files span more than one HDFS authority: one
hdfs.name-node per scan overrides the authority of every path the FileIO opens, so a second nameservice would be read from the first one's NameNode at the same relative path.
Additional context
Note that this is a second, independent HDFS client in the same process: the plain-Parquet path reaches HDFS through libhdfs/JNI while an Iceberg table would be opened over pure-Rust RPC. The two read the same $HADOOP_CONF_DIR XML but hold separate connections and separate Kerberos state.
Blocked on apache/iceberg-rust#3111.
What is the problem the feature request solves?
Comet's native Iceberg scan cannot read a table whose data lives on HDFS.
hdfs://is not in the scheme allowlist that mirrorsstorage_factory_for, so any Iceberg table on HDFS falls back to the JVM reader and gets no native acceleration.This is a gap relative to the plain-Parquet native scan, which does read HDFS (through libhdfs/JNI, via
fs.comet.libhdfs.schemes). On-premise Iceberg deployments are commonly HDFS-backed, so the Iceberg acceleration Comet offers is unavailable to them today.Describe the potential solution
Route
hdfs://to iceberg-rust'shdfs-nativeOpenDAL backend, a pure-Rust HDFS RPC client that needs no JNI or libhdfs. That backend is being added upstream in apache/iceberg-rust#3111; this issue tracks the Comet-side work once it lands.Comet-side work:
hdfsarm tostorage_factory_forand admit the scheme in the JVM gates (CometScanRule.icebergReadableSchemes, andCometIcebergNativeWrite.SupportedStorageSchemesfor the write path).hdfs./hadoop.property prefixes to the nativeFileIO.HdfsNativeBuildernever dials the authority written in the path: it builds one client against a synthetic authority and synthesizes the HA config from the comma-separatedhdfs.name-nodevalue. iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a realhost:port. An HA location readshdfs://<nameservice>/..., and a nameservice is not a routable host, so the endpoints have to be derived fromdfs.ha.namenodes.<ns>/dfs.namenode.rpc-address.<ns>.<nn>in the session Hadoop configuration.hdfs.name-nodeper scan overrides the authority of every path the FileIO opens, so a second nameservice would be read from the first one's NameNode at the same relative path.Additional context
Note that this is a second, independent HDFS client in the same process: the plain-Parquet path reaches HDFS through libhdfs/JNI while an Iceberg table would be opened over pure-Rust RPC. The two read the same
$HADOOP_CONF_DIRXML but hold separate connections and separate Kerberos state.Blocked on apache/iceberg-rust#3111.