Repository navigation
Description
Problem
#67 plans FileScanTasks before execution and distributes them across target_partitions. Each task is a whole file: collect_into in read/file_planning.rs round-robins tasks as plan_files() yields them, with start/length covering the full file.
A table with fewer large files than partitions leaves partitions idle, and a file with many row groups is read end to end by one partition. Files written by Spark or Flink are commonly 128 MB to 1 GB with many row groups, so this is the usual case.
The reader already handles byte ranges: when task.start() != 0 || task.length() != 0, ArrowReader keeps only the row groups starting inside [start, start + length) (filter_row_groups_by_byte_range). The missing piece was knowing the row-group boundaries at planning time. apache/iceberg-rust#3704 adds FileScanTask::split_offsets(), populated from DataFile.split_offsets (spec field 132), so a planner can cut at those offsets without opening the file.
Cutting at arbitrary byte positions is not an option: a cut inside a row group makes two tasks open the file while one of them yields nothing.
Proposal
In eager planning, after plan_files() and before partition assignment:
- Pick a target split size: a session setting such as
iceberg.target_split_size_bytes(Iceberg Java'sread.split.target-sizedefaults to 128 MB), or total selected bytes /target_partitionswith a floor. - For each task with
length()above the target and non-emptysplit_offsets(), cut it at offsets chosen to keep each piece near the target. Each piece is a clone of the task with its ownstart/length; delete files and the residual predicate stay on every piece. - Tasks with no offsets, or below the target, stay whole.
- Assign the resulting tasks to partitions. Round-robin is enough for a first version; largest-first byte balancing as in feat: balance whole-file Iceberg work across tasks datafusion-contrib/datafusion-distributed#788 is a natural follow-up.
Expose the splitting step as a public function on the task list so with_planned_files(groups) consumers (DFD, Ballista, Comet) can reuse it.
Lazy planning is out of scope; #84 covers that path.
Open questions
- Position deletes: each piece loads the same delete vectors. feat: plan Iceberg file tasks before execution #67 shares the delete loader across partitions within an execution, so this should be one load per file rather than per piece; worth confirming.
- Whether
EXPLAINshould show the split count next to the file count.
Dependencies
- feat: plan Iceberg file tasks before execution #67 merged.
- iceberg-rust revision bumped past feat(scan): carry the data file's split offsets on FileScanTask iceberg-rust#3704.
Validation
- One file with 4 row groups and
target_partitions=4yields 4 tasks, each partition reads exactly one row group, and results match the single-partition scan, including with position deletes and a predicate. - A file with
split_offsetsNoneor empty stays one task. - A file below the target stays one task regardless of offsets.
instasnapshot of the plan showing the split count.
Activity
alexanderbianchi commentedon Nov 17, 2026
Phenomenal, thanks for making the iceberg-rust PR. I will look deeper into this after 67 is out, but we have seen issues with skew when 1 file is large and the others small this will help with that.