Skip to content

Reuse planned file metrics for eager Iceberg column statistics (no double IO) - #32

Draft
alexanderbianchi wants to merge 1 commit into
alex.bianchi/dd-source-iceberg-stats-20261029-df56from
alex.bianchi/dd-source-iceberg-stats-single-pass
Draft

alexanderbianchi wants to merge 1 commit into
alex.bianchi/dd-source-iceberg-stats-20261029-df56from
alex.bianchi/dd-source-iceberg-stats-single-pass

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Nov 5, 2026 •

Copy link
Copy Markdown

Draft. Stacked on DataFusion 56 (datafusion-contrib#851) plus the dd-source stats hooks, rebased from the DataFusion 55 line (old head kept at alex.bianchi/dd-source-iceberg-stats-single-pass-df55).

Eager column statistics used to re-read the snapshot's manifests after planning. Now the scan requests with_column_stats for the statistics columns, and statistics are summarized from each planned FileScanTask::column_stats() (apache/iceberg-rust#3424). Lazy path unchanged.

Dependencies: iceberg-rust bumped to apache/iceberg-rust 13f84f9 (Arrow 60 plus #3424), fetched via DataDog/iceberg-rust, which mirrors it. A root [patch] points datafusion-iceberg (apache/datafusion-iceberg#99 at 5398be1) at the same revision so FileScanTask is one type. Adapted to its API (UnboundPartitionField builder, FileScanTaskDeleteFile accessors).

Notes:

  • Upstream has no take_column_stats, so the retained metrics now stay on the tasks and are serialized to workers with them.
  • join_selection_can_build_derived_keys... test: DataFusion 56 already builds the derived keys without the provider, so only the provider-enabled outcome is asserted.

Tests: cargo test -p datafusion-distributed-iceberg and -p datafusion-distributed --lib pass; clippy on the iceberg crate is clean. Core crate clippy fails on an unused PRODUCER_SALT_DEFAULT import that is not from this change.

Request column statistics from the scan (iceberg-rust with_column_stats,
apache/iceberg-rust#3424) and summarize them from the planned FileScanTasks
instead of re-reading manifests, avoiding double IO.

Bump iceberg-rust to 13f84f9 (Arrow 60 plus #3424), patching datafusion-iceberg
to the same revision so FileScanTask is one type, and adapt to its API
(UnboundPartitionField builder, FileScanTaskDeleteFile accessors). Adapt the
column byte statistics tests to DataFusion 56.
@alexanderbianchi
alexanderbianchi changed the base branch from alex.bianchi/dd-source-iceberg-stats-20261029 to alex.bianchi/dd-source-iceberg-stats-20261029-df56 November 17, 2026 02:57
@alexanderbianchi
alexanderbianchi force-pushed the alex.bianchi/dd-source-iceberg-stats-single-pass branch from 239ffcd to 42e904b Compare November 17, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant