Skip to content

feat: give the Iceberg reader dynamic filters to skip row groups - #22

Draft
Perseus wants to merge 3 commits into
anirudh.singh/iceberg-dyn-filtering-3from
anirudh.singh/iceberg-dyn-filtering-4
Draft

Perseus wants to merge 3 commits into
anirudh.singh/iceberg-dyn-filtering-3from
anirudh.singh/iceberg-dyn-filtering-4

Conversation

@Perseus

@Perseus Perseus commented Nov 10, 2026

Copy link
Copy Markdown
Owner

As the reader starts each file, the scan now ANDs the newest dynamic filters onto the file's own filter, so the reader can skip row groups the filters rule out. This covers joins on columns that aren't partition columns, and top-N.

The reader gets some columns wrong, so filters on them are left out: identity partition columns (the partition skipper checks those), columns with a default value, and nested columns. Above 200 listed values, IN lists are dropped and their bounds kept. The reader only gets the filters when row group filtering is on and page skipping is off, because page skipping gets widened columns wrong. Any error leaves the file as it is.

@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch from 0bfae70 to c7461f9 Compare November 11, 2026 14:35
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch from 77858c6 to c59f43e Compare November 11, 2026 14:35
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch 6 times, most recently from 1b0cda0 to 67e59e2 Compare November 15, 2026 17:05
The next commits give the Iceberg reader the scan's dynamic filters, so it
can skip row groups. The reader only uses the filter stored on each
FileScanTask, and iceberg-rust has no way to change that filter.

and_predicate works around this. It turns the task's filter back into an
unbound predicate, ANDs the new predicate onto it, binds the result and
builds a copy of the task with it. Nothing calls it yet.

The copy sets each of the task's fields by hand. A snapshot test lists the
task's fields, so it fails if iceberg-rust adds a field the copy would
leave out.
reader_filter keeps the parts of the dynamic filters the reader can check
for a file, leaving out the parts it can get wrong. Nothing calls it yet.

It drops comparisons on these columns:
- Identity partition columns. A data file doesn't have to store them, and
  the reader treats a missing column as NULL. A comparison like x = 5 on
  it then drops every row of the file. The partition check already covers
  these columns.
- Columns with a default value. Files written before the column was added
  don't store it.
- Nested columns. The reader's row filter fails on them.

A dropped comparison becomes "matches everything", so the filter only
matches more rows. A NOT is dropped as a whole, because dropping a part
inside it would skip rows that match.

The reader checks every row once per IN list value. When the IN lists it
gets hold more than 200 values in total, they are dropped and the min/max
bounds are kept.
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch 2 times, most recently from 69b0379 to 2ba5b50 Compare November 16, 2026 14:54
With iceberg.dynamic_filtering_enabled on, the scan now gives the reader
the newest dynamic filters as it starts each file. The reader uses them to
skip row groups, and to drop the rows that don't match. This helps joins on
columns that aren't partition columns, and top-N.

Any error leaves the file's task as it was.
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch from 2ba5b50 to 7849838 Compare November 16, 2026 21:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant