Skip to content

Split large data files at their split_offsets during eager planning #108

Open

Description

@CTTY

Problem

#67 plans FileScanTasks before execution and distributes them across target_partitions. Each task is a whole file: collect_into in read/file_planning.rs round-robins tasks as plan_files() yields them, with start/length covering the full file.

A table with fewer large files than partitions leaves partitions idle, and a file with many row groups is read end to end by one partition. Files written by Spark or Flink are commonly 128 MB to 1 GB with many row groups, so this is the usual case.

The reader already handles byte ranges: when task.start() != 0 || task.length() != 0, ArrowReader keeps only the row groups starting inside [start, start + length) (filter_row_groups_by_byte_range). The missing piece was knowing the row-group boundaries at planning time. apache/iceberg-rust#3704 adds FileScanTask::split_offsets(), populated from DataFile.split_offsets (spec field 132), so a planner can cut at those offsets without opening the file.

Cutting at arbitrary byte positions is not an option: a cut inside a row group makes two tasks open the file while one of them yields nothing.

Proposal

In eager planning, after plan_files() and before partition assignment:

  1. Pick a target split size: a session setting such as iceberg.target_split_size_bytes (Iceberg Java's read.split.target-size defaults to 128 MB), or total selected bytes / target_partitions with a floor.
  2. For each task with length() above the target and non-empty split_offsets(), cut it at offsets chosen to keep each piece near the target. Each piece is a clone of the task with its own start/length; delete files and the residual predicate stay on every piece.
  3. Tasks with no offsets, or below the target, stay whole.
  4. Assign the resulting tasks to partitions. Round-robin is enough for a first version; largest-first byte balancing as in feat: balance whole-file Iceberg work across tasks datafusion-contrib/datafusion-distributed#788 is a natural follow-up.

Expose the splitting step as a public function on the task list so with_planned_files(groups) consumers (DFD, Ballista, Comet) can reuse it.

Lazy planning is out of scope; #84 covers that path.

Open questions

  • Position deletes: each piece loads the same delete vectors. feat: plan Iceberg file tasks before execution #67 shares the delete loader across partitions within an execution, so this should be one load per file rather than per piece; worth confirming.
  • Whether EXPLAIN should show the split count next to the file count.

Dependencies

Validation

  • One file with 4 row groups and target_partitions=4 yields 4 tasks, each partition reads exactly one row group, and results match the single-partition scan, including with position deletes and a predicate.
  • A file with split_offsets None or empty stays one task.
  • A file below the target stays one task regardless of offsets.
  • insta snapshot of the plan showing the split count.

Activity

  1. alexanderbianchi commented on Nov 17, 2026

    @alexanderbianchi

    Phenomenal, thanks for making the iceberg-rust PR. I will look deeper into this after 67 is out, but we have seen issues with skew when 1 file is large and the others small this will help with that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions