Skip to content

fix: do not negate partially converted predicates - #59

Merged
gabotechs merged 1 commit into
apache:mainfrom
quantscious:fix/cast-pushdown
Nov 17, 2026
Merged

gabotechs merged 1 commit into
apache:mainfrom
quantscious:fix/cast-pushdown

Conversation

@terbed

@terbed terbed commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Closes #44

When one side of an AND can't be converted, to_iceberg_and_predicate keeps only the other side. That is sound for Inexact pushdown because the result is implied by the filter. NOT then negated it, producing a predicate that drops matching rows: NOT (a = 1 AND upper(b) = 'X') was pushed as NOT (a = 1).

This adds a TransformedResult::PartialPredicate variant for predicates that are implied by the expression but not equivalent to it:

  • AND returns it when it drops an operand.
  • AND and OR propagate it from their operands.
  • NOT does not convert it, so the negated filter is not pushed down.

A partial predicate that is not under a NOT is pushed down as before.

This mainly affects direct callers of convert_filters_to_predicate and filters that reach it unsimplified; DataFusion's optimizer usually rewrites NOT (A AND B) before pushdown.

The cast and bind changes originally in this PR were removed: #91 already fixed the cast handling on main (#36), and the bind check is handled separately in #105.

No public API or dependency changes.


AI assistance: OpenAI Codex assisted with implementation and tests.

@debugmiller

debugmiller commented Nov 11, 2026 •

Copy link
Copy Markdown

Not a reviewer/committer but I also ran into this issue and decided to take a look. Some (hopefully) helpful observations to move this PR forward:

GIven the above you can probably trim this down to a narrower PR

@terbed

terbed commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Thank you @debugmiller for the suggestions.
I've updated the PR accordingly, minimizing the changes and focusing this pr solely on issue #44.
Could you please review @mbutrovich ?

@gabotechs

Copy link
Copy Markdown
Contributor

Hi @terbed, I see the PR description does not match the content, does it need to be updated?

@terbed terbed changed the title fix: preserve filter semantics for casts and negation fix: do not negate partially converted predicates Nov 16, 2026
@terbed

terbed commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Hello @gabotechs,
Thanks for the remark! The PR is drifted and narrowed compared to the initial description. I've updated it, now it is current.

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks @terbed!

@gabotechs
gabotechs merged commit dfcc19a into apache:main Nov 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

convert_filters_to_predicate negates a weakened AND, producing a predicate that drops matching rows

3 participants