Skip to content

feat(workflow-operator): let Filter combine predicates with AND - #8484

Open
gupta-sahil01 wants to merge 1 commit into
apache:mainfrom
gupta-sahil01:feat/filter-and-or
Open

gupta-sahil01 wants to merge 1 commit into
apache:mainfrom
gupta-sahil01:feat/filter-and-or

Conversation

@gupta-sahil01

@gupta-sahil01 gupta-sahil01 commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The Filter operator hard-codes OR semantics — a tuple passes as soon as any one
predicate matches. Users who need AND (e.g. age > 30 AND country = "US") have to
chain multiple Filter operators, which adds workflow clutter and an extra hop per
condition.

This PR makes the combination configurable:

  • New PredicateCombinator enum (OR → "any (OR)", AND → "all (AND)"),
    following the @JsonValue / @JsonCreator pattern already used by
    ComparisonType in the same package.
  • New optional predicateCombinator property on SpecializedFilterOpDesc,
    defaulting to OR and declared above predicates so the dropdown renders above the
    predicate rows. The UI renders it from the JSON schema, so there are no frontend
    changes.
  • SpecializedFilterOpExec picks the filter function once: forall for AND,
    exists for OR, with both cases named so a future combinator gets a compiler warning.
  • Python export (generateStandaloneCode) joins conditions with & under AND and
    | under OR.
  • Docs and descriptions updated to stop claiming OR-only behavior.

Backward compatibility: the property is not in the schema's required list, so
Filters saved before this PR (which lack the key) still pass the frontend validator.
A missing value deserializes to OR, and so does an explicit null
(@JsonSetter(nulls = Nulls.SKIP)), so the executor never sees a null combinator.
Existing workflows behave exactly as before and no migration is needed.

One behavioral note for reviewers: an empty predicate list continues to drop every
tuple in both modes, in the engine and in the export. A bare forall returns true
on an empty list, which would have silently flipped an empty Filter from "drop
everything" to "pass everything", so the AND branch is guarded with
predicates.nonEmpty &&. The docs call this out.

Nested predicate trees (e.g. (A AND B) OR C) are intentionally out of scope — they
need more extensive UI work and are noted as a follow-up in the issue.

Any related issues, documentation, discussions?

Closes #6939

Operator documentation updated in the same PR:
docs/reference/operators/data-cleaning/filter.md and
docs/reference/operators/data-cleaning/_index.md.

How was this PR tested?

sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.filter.*": 7 suites,
76 passed, 0 failed. 4 pandas parity tests (tagged IntegrationTest) were canceled
locally because the local Python has no pandas; they are unchanged by this PR.

  • PredicateCombinatorSpec (new): wire names via getName, case-insensitive
    fromString, rejection of an unknown name, and the Jackson round-trip.
  • SpecializedFilterOpDescSpec:
    • the property defaults to OR; descriptor JSON with no predicateCombinator
      deserializes to OR; an explicit AND survives the polymorphic LogicalOp
      round-trip;
    • bad input: "xor" fails with a ValueInstantiationException naming the field, and
      an executor built from "predicateCombinator": null behaves as OR;
    • the generated operator schema leaves predicateCombinator out of required, lists
      it before predicates, and defaults it to any (OR);
    • export: two predicates join with | under OR and & under AND; no predicates keeps
      no rows under either combinator.
  • SpecializedFilterOpExecSpec: with two predicates where only one matches, OR
    keeps the tuple and AND drops it, with the failing predicate in either position; AND
    keeps it when both match; AND with an empty predicate list drops everything; AND and
    OR agree on a single predicate. A before block resets the shared descriptor between
    tests.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

@github-actions github-actions Bot added feature docs Changes related to documentations common labels Sep 10, 2026
@github-actions

github-actions Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Committers with relevant context: @parshimers
    You can request their reviews formally with /request-review @parshimers.

  • Contributors with relevant context: @kz930, @Yicong-Huang
    You can notify them by mentioning @kz930, @Yicong-Huang in a comment.

@codecov-commenter

codecov-commenter commented Sep 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.80%. Comparing base (03a5bb8) to head (d7a2586).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...mber/operator/filter/SpecializedFilterOpDesc.scala 83.33% 0 Missing and 1 partial ⚠️
...mber/operator/filter/SpecializedFilterOpExec.scala 80.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8484   +/-   ##
=========================================
  Coverage     91.80%   91.80%           
- Complexity     5024     5033    +9     
=========================================
  Files          1279     1280    +1     
  Lines         56582    56601   +19     
  Branches       7425     7430    +5     
=========================================
+ Hits          51945    51963   +18     
+ Misses         2770     2769    -1     
- Partials       1867     1869    +2     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.53% <ø> (ø) Carriedforward from 03a5bb8
amber 85.79% <90.90%> (+<0.01%) ⬆️
computing-unit-managing-service 64.05% <ø> (ø)
config-service 87.73% <ø> (ø)
file-service 82.31% <ø> (ø)
frontend 96.50% <ø> (ø) Carriedforward from 03a5bb8
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.50% <ø> (ø) Carriedforward from 03a5bb8
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main cc1f201 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 377 0.23 24,865/38,081/38,081 us 🔴 -11.5% / 🔴 +153.1%
🟢 bs=100 sw=10 sl=64 825 0.503 120,474/134,886/134,886 us 🟢 -13.5% / 🔴 +29.9%
⚪ bs=1000 sw=10 sl=64 973 0.594 1,026,805/1,066,257/1,066,257 us ⚪ within ±5% / 🔴 -10.9%
Baseline details

Latest main cc1f201 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 377 tuples/sec 425 tuples/sec 812.43 tuples/sec -11.3% -53.6%
bs=10 sw=10 sl=64 MB/s 0.23 MB/s 0.26 MB/s 0.496 MB/s -11.5% -53.6%
bs=10 sw=10 sl=64 p50 24,865 us 22,768 us 12,322 us +9.2% +101.8%
bs=10 sw=10 sl=64 p95 38,081 us 35,931 us 15,047 us +6.0% +153.1%
bs=10 sw=10 sl=64 p99 38,081 us 35,931 us 18,205 us +6.0% +109.2%
bs=100 sw=10 sl=64 throughput 825 tuples/sec 827 tuples/sec 1,058 tuples/sec -0.2% -22.0%
bs=100 sw=10 sl=64 MB/s 0.503 MB/s 0.505 MB/s 0.646 MB/s -0.4% -22.1%
bs=100 sw=10 sl=64 p50 120,474 us 116,358 us 96,752 us +3.5% +24.5%
bs=100 sw=10 sl=64 p95 134,886 us 155,933 us 103,874 us -13.5% +29.9%
bs=100 sw=10 sl=64 p99 134,886 us 155,933 us 119,052 us -13.5% +13.3%
bs=1000 sw=10 sl=64 throughput 973 tuples/sec 987 tuples/sec 1,092 tuples/sec -1.4% -10.9%
bs=1000 sw=10 sl=64 MB/s 0.594 MB/s 0.603 MB/s 0.666 MB/s -1.5% -10.8%
bs=1000 sw=10 sl=64 p50 1,026,805 us 1,020,361 us 939,475 us +0.6% +9.3%
bs=1000 sw=10 sl=64 p95 1,066,257 us 1,055,219 us 982,954 us +1.0% +8.5%
bs=1000 sw=10 sl=64 p99 1,066,257 us 1,055,219 us 1,020,177 us +1.0% +4.5%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,530.05,200,128000,377,0.230,24864.87,38081.35,38081.35
1,100,10,64,20,2425.35,2000,1280000,825,0.503,120474.49,134885.57,134885.57
2,1000,10,64,20,20562.38,20000,12800000,973,0.594,1026804.60,1066257.11,1066257.11

@gupta-sahil01

Copy link
Copy Markdown
Contributor Author

/request-review @parshimers

@github-actions
github-actions Bot requested a review from parshimers September 10, 2026 18:26
@gupta-sahil01

Copy link
Copy Markdown
Contributor Author

/request-review @xuang7

@github-actions
github-actions Bot requested a review from xuang7 September 25, 2026 04:43
renovate-bot pushed a commit to renovate-bot/apache-_-texera that referenced this pull request Oct 7, 2026
### What changes were proposed in this PR?

Adds a check to `pr-assignment.yml` that flags a PR closing an issue
someone else has already claimed. It also stops the merge-time credit
step from unassigning a claimant whose own PR is still open.

```
Before:  apache#8267 open (claimant) ── apache#8339 "Closes apache#8149" ── (silence) ── apache#8339 merges
         ── claimant unassigned from apache#8149, apache#8267 left with 0 changed files
After:   apache#8267 open (claimant) ── apache#8339 "Closes apache#8149" ── comment on apache#8339 names the claimant + apache#8267
                                                     ── apache#8339 merges ── claimant stays assigned while apache#8267 is open
```

| Where | Change |
| --- | --- |
| New step "Flag linked issues someone else has claimed" (PR `opened` /
`edited`) | For each open same-repo issue in `closingIssuesReferences`,
flags it when **(a)** it has an assignee other than the PR opener, or
**(b)** its `closedByPullRequestsReferences` lists another `OPEN` PR by
a different author. Posts or updates one `<!--
texera:issue-claim-conflict -->` comment with an issue / claimant / PR
table, and deletes the comment once nothing matches. If overlapping runs
each post a copy, every run keeps the oldest and deletes the others, so
they end up keeping the same comment (`claimCommentChanges`). A run can
also read the PR before an edit and write after a newer run has
finished, so after any write it reads the PR again and repeats until a
pass changes nothing (`syncClaimComment`, at most 3 passes). The last
run to write is then also the last to read, and it corrects a stale
write. |
| "Credit issue assignees on PR merge" | Keeps an assignee who has their
own open PR on the issue. If that PR later closes without merging, the
existing close-without-merge step still unassigns them. |
| `.github/scripts/issue-claims.js` | The decision logic, which both
steps `require`. It is checked out from the base branch, which is safe
under `pull_request_target`. The workflow now declares `contents: read`
for that checkout. |

**(b) is needed on its own.** GitHub silently drops the opener
self-assign for an outside contributor who never commented on the issue.
For example, apache#6674 shows no assignee but has the open PR apache#8603.

**It posts a comment and does not fail a check:**

| Reason | Detail |
| --- | --- |
| Both PRs match | The claimant's own PR matches the rule too: apache#8267
would flag apache#8339 back. A failing check would also land on the person the
rule protects. |
| Some takeovers are agreed | Only a person can tell a duplicate from an
agreed takeover. One of the 2 PRs it would flag today (below), apache#8484, is
an agreed takeover, and a failing check would block it. |
| It would go stale | The check runs again only when this PR is opened
or edited. Closing the other PR or unassigning someone doesn't re-run
it, so a failed check would stay red until someone edits the PR. |

On its first post, the comment @-mentions the claimant, so they get a
notification. The apache#8149 claimant never got one. Closed issues are
skipped, because the risk is two PRs in progress at once. Both queries
read up to 100 linked PRs per issue. That is the GraphQL maximum; no
issue in this repo has more than 5. There is no concurrency group: with
the default queue, a group cancels a waiting run, and the body-edit
unassign and the merge credit need every event. One known limitation:
the comment is refreshed only when this PR is edited, not when the other
PR closes.

### Any related issues, documentation, discussions?

Closes apache#8676

Complements apache#8675, which adds an AGENTS.md rule to check claims before
coding. This PR catches the cases that rule misses.

### How was this PR tested?

Unit tests: 32 pass. Two of them replay the apache#8149 and apache#6674 incidents as
GraphQL fixtures.

```bash
bash .github/scripts/test_issue_claims.sh   # tests 42, pass 42, fail 0
```

The `infra` job picks these up through its `test_*.sh` discovery.
Mutation check: reverting the credit fix fails the apache#8149 merge test, and
dropping (b) fails the apache#6674 test. For `claimCommentChanges`, removing
any one of the id sort, the Bot-only filter, the deletion of extra
copies or the unchanged-body check fails at least one test. For
`syncClaimComment`, limiting it to one pass fails both delayed-run
tests.

`pull_request_target` runs the workflow from `main`, so this PR can't
run its own steps. Instead, I ran both edited `github-script` steps,
extracted from the YAML, in a local harness. GraphQL calls went to the
live API (read-only). Every REST write was recorded, not sent.

| Scenario | Result |
| --- | --- |
| apache#8339 opened, fixture = apache#8149 as of 2026-09-02 | posts 1 comment:
`apache#8149 · <claimant> (assignee) · apache#8267` |
| edited, with an older bot comment and a human comment quoting the
marker | updates only the bot comment |
| edited again, comment already current | no write |
| edited after `Closes apache#8149` is removed | deletes the bot comment; the
human comment is left alone |
| live: apache#8603 opened (apache#6674) | no conflict, no writes |
| live: apache#8484 edited | posts 1 comment naming the apache#6939 assignee |
| credit step, fixture = apache#8339's merge on 2026-09-25 | apache#8149 keeps its
claimant (`kept (own open PR)=[…]`); an assignee without a PR on apache#8338
is still removed |
| credit step, live query on apache#8339 | query valid, no writes |
| race: this run posts, then its re-list shows an older copy from
another run | deletes its own copy (and first updates the older one if
it is stale) |
| re-list after posting fails, or a delete returns 404 | logs a warning
and stops; a 404 skips only that one delete |
| edited, with two copies left by an earlier race | updates the oldest,
deletes the other |
| delayed run: reads the claim, a newer run removes it, then the delayed
run posts | its re-read finds nothing to flag and deletes its own post |
| delayed run: writes old text over a newer run's comment | its re-read
restores the newer text |
| live: guard and credit queries with `first: 100` | both valid against
the schema |

I also modelled two racing runs against one shared comment list and
tried every ordering of their API calls, using the real
`claimCommentChanges`. Whenever both runs see the same state, exactly
one up-to-date comment is left; 104 orderings for an open followed by a
quick edit. When both runs find nothing to flag, no comment is left.

Dry run over every open PR: 218 open, 188 with closing references, **2
flagged**. Both are real claims:

| PR | Issue | Claim |
| --- | --- | --- |
| apache#5558 (draft) | apache#5422 | the issue's author assigned themselves |
| apache#8484 | apache#6939 | a `/take` that was never released; a maintainer later
told apache#8484's author to go ahead |

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@gupta-sahil01

Copy link
Copy Markdown
Contributor Author

Any updates here @aglinxinyuan ?

@aglinxinyuan aglinxinyuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up. The enum, the executor change and the empty-list guard look right, and defaulting to OR is the right call. Two things need fixing before this merges; smaller notes are inline.

  1. After the rebase, Python export ignores AND. Since this branched, #8502 added generateStandaloneCode() to SpecializedFilterOpDesc. It joins the predicates with " | ". That file merges cleanly with your change; only SpecializedFilterOpDescSpec conflicts. So once rebased, an AND Filter still exports as OR: age > 30 AND country = "US" comes out as (…age > 30) | (…country == "US"), which keeps rows matching either predicate. Please:

    • join on the combinator:
      val joiner = if (predicateCombinator == PredicateCombinator.AND) " & " else " | "
      s"out1df = in1df[${conditions.mkString(joiner)}].reset_index(drop=True)"
    • fix the two comments that say the executor uses predicates.exists, one in the descriptor and one in its spec;
    • add a two-predicate AND export test.

    When you resolve the spec conflict, keep both sides.

  2. required = true makes saved Filters invalid. See the inline comment. "No migration needed" holds for the backend, but not for the frontend validator.

Also, docs/reference/operators/data-cleaning/_index.md:42 still says Filter uses OR only.

The Filter operator hard-coded OR semantics, so users needing AND had to
chain multiple Filter operators. Add a PredicateCombinator enum exposed as
a descriptor property, defaulting to OR so existing workflows are
unaffected — a stored desc without the field deserializes to OR, so no
migration is needed.

Empty predicate lists keep dropping every tuple in both modes; a bare
forall would have flipped that to passing everything.
@gupta-sahil01

Copy link
Copy Markdown
Contributor Author

Rebased onto main and addressed the review. The Python export now joins with & under AND, the exists comments are reworded, and there's a two-predicate AND export test. predicateCombinator is now optional, so saved Filters stay valid. _index.md is updated. Filter tests pass locally (76 passed; 4 pandas parity tests skipped locally because pandas isn't installed).

@gupta-sahil01

Copy link
Copy Markdown
Contributor Author

/request-review @aglinxinyuan

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common docs Changes related to documentations feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add option to combine Filter operator predicates with AND instead of only OR

3 participants