Skip to content

[AURON #2431] Prune Iceberg changelog tasks by metadata predicates - #2456

Open
goutamadwant wants to merge 1 commit into
apache:masterfrom
goutamadwant:feature-2431-changelog-metadata-pruning
Open

[AURON #2431] Prune Iceberg changelog tasks by metadata predicates#2456
goutamadwant wants to merge 1 commit into
apache:masterfrom
goutamadwant:feature-2431-changelog-metadata-pruning

Conversation

@goutamadwant

@goutamadwant goutamadwant commented Aug 8, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #2431

Rationale for this change

Iceberg changelog metadata values are constant for each changelog task. Auron currently plans every supported task in the changelog range even when a predicate on _change_type, _commit_snapshot_id, or _change_ordinal proves that a task cannot match.

What changes are included in this PR?

  • Associate supported metadata-only filters with an unambiguous Iceberg changelog scan before adaptive query stages are created.
  • Prune validated changelog tasks for = and IN predicates combined with AND.
  • Keep OR, NOT, complex expressions, mixed data-column predicates, and ambiguous scan matches on the existing post-scan filter path.
  • Preserve existing fallback validation for unsupported changelog tasks and mixed file formats.
  • Add regression coverage for matching, nonmatching, conjunction, and unsupported-expression cases.

Are there any user-facing changes?

Yes. Eligible Iceberg changelog queries plan and read fewer native scan files. There are no public API or configuration changes.

How was this patch tested?

  • AuronIcebergIntegrationSuite: 43 tests passed.
  • Focused changelog metadata pruning regression: passed.
  • ./dev/reformat --check: passed across the configured Spark 3.0-3.5 and 4.0-4.1 profiles.
  • ./build/mvn -B spotless:check test-compile -pl thirdparty/auron-iceberg -am -Ppre -Pscala-2.12 -Pspark-3.5 -Piceberg-1.10.1 -DskipBuildNative -DskipTests: passed.

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Codex was used to Review and understand the codebase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Push down simple Iceberg changelog metadata predicates

1 participant