Repository navigation
feat(workflow-operator): let Filter combine predicates with AND - #8484
gupta-sahil01 wants to merge 1 commit into
Conversation
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8484 +/- ##
=========================================
Coverage 91.80% 91.80%
- Complexity 5024 5033 +9
=========================================
Files 1279 1280 +1
Lines 56582 56601 +19
Branches 7425 7430 +5
=========================================
+ Hits 51945 51963 +18
+ Misses 2770 2769 -1
- Partials 1867 1869 +2
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 377 | 0.23 | 24,865/38,081/38,081 us | 🔴 -11.5% / 🔴 +153.1% |
| 🟢 | bs=100 sw=10 sl=64 | 825 | 0.503 | 120,474/134,886/134,886 us | 🟢 -13.5% / 🔴 +29.9% |
| ⚪ | bs=1000 sw=10 sl=64 | 973 | 0.594 | 1,026,805/1,066,257/1,066,257 us | ⚪ within ±5% / 🔴 -10.9% |
Baseline details
Latest main cc1f201 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 377 tuples/sec | 425 tuples/sec | 812.43 tuples/sec | -11.3% | -53.6% |
| bs=10 sw=10 sl=64 | MB/s | 0.23 MB/s | 0.26 MB/s | 0.496 MB/s | -11.5% | -53.6% |
| bs=10 sw=10 sl=64 | p50 | 24,865 us | 22,768 us | 12,322 us | +9.2% | +101.8% |
| bs=10 sw=10 sl=64 | p95 | 38,081 us | 35,931 us | 15,047 us | +6.0% | +153.1% |
| bs=10 sw=10 sl=64 | p99 | 38,081 us | 35,931 us | 18,205 us | +6.0% | +109.2% |
| bs=100 sw=10 sl=64 | throughput | 825 tuples/sec | 827 tuples/sec | 1,058 tuples/sec | -0.2% | -22.0% |
| bs=100 sw=10 sl=64 | MB/s | 0.503 MB/s | 0.505 MB/s | 0.646 MB/s | -0.4% | -22.1% |
| bs=100 sw=10 sl=64 | p50 | 120,474 us | 116,358 us | 96,752 us | +3.5% | +24.5% |
| bs=100 sw=10 sl=64 | p95 | 134,886 us | 155,933 us | 103,874 us | -13.5% | +29.9% |
| bs=100 sw=10 sl=64 | p99 | 134,886 us | 155,933 us | 119,052 us | -13.5% | +13.3% |
| bs=1000 sw=10 sl=64 | throughput | 973 tuples/sec | 987 tuples/sec | 1,092 tuples/sec | -1.4% | -10.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.594 MB/s | 0.603 MB/s | 0.666 MB/s | -1.5% | -10.8% |
| bs=1000 sw=10 sl=64 | p50 | 1,026,805 us | 1,020,361 us | 939,475 us | +0.6% | +9.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,066,257 us | 1,055,219 us | 982,954 us | +1.0% | +8.5% |
| bs=1000 sw=10 sl=64 | p99 | 1,066,257 us | 1,055,219 us | 1,020,177 us | +1.0% | +4.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,530.05,200,128000,377,0.230,24864.87,38081.35,38081.35
1,100,10,64,20,2425.35,2000,1280000,825,0.503,120474.49,134885.57,134885.57
2,1000,10,64,20,20562.38,20000,12800000,973,0.594,1026804.60,1066257.11,1066257.11|
/request-review @parshimers |
|
/request-review @xuang7 |
### What changes were proposed in this PR? Adds a check to `pr-assignment.yml` that flags a PR closing an issue someone else has already claimed. It also stops the merge-time credit step from unassigning a claimant whose own PR is still open. ``` Before: apache#8267 open (claimant) ── apache#8339 "Closes apache#8149" ── (silence) ── apache#8339 merges ── claimant unassigned from apache#8149, apache#8267 left with 0 changed files After: apache#8267 open (claimant) ── apache#8339 "Closes apache#8149" ── comment on apache#8339 names the claimant + apache#8267 ── apache#8339 merges ── claimant stays assigned while apache#8267 is open ``` | Where | Change | | --- | --- | | New step "Flag linked issues someone else has claimed" (PR `opened` / `edited`) | For each open same-repo issue in `closingIssuesReferences`, flags it when **(a)** it has an assignee other than the PR opener, or **(b)** its `closedByPullRequestsReferences` lists another `OPEN` PR by a different author. Posts or updates one `<!-- texera:issue-claim-conflict -->` comment with an issue / claimant / PR table, and deletes the comment once nothing matches. If overlapping runs each post a copy, every run keeps the oldest and deletes the others, so they end up keeping the same comment (`claimCommentChanges`). A run can also read the PR before an edit and write after a newer run has finished, so after any write it reads the PR again and repeats until a pass changes nothing (`syncClaimComment`, at most 3 passes). The last run to write is then also the last to read, and it corrects a stale write. | | "Credit issue assignees on PR merge" | Keeps an assignee who has their own open PR on the issue. If that PR later closes without merging, the existing close-without-merge step still unassigns them. | | `.github/scripts/issue-claims.js` | The decision logic, which both steps `require`. It is checked out from the base branch, which is safe under `pull_request_target`. The workflow now declares `contents: read` for that checkout. | **(b) is needed on its own.** GitHub silently drops the opener self-assign for an outside contributor who never commented on the issue. For example, apache#6674 shows no assignee but has the open PR apache#8603. **It posts a comment and does not fail a check:** | Reason | Detail | | --- | --- | | Both PRs match | The claimant's own PR matches the rule too: apache#8267 would flag apache#8339 back. A failing check would also land on the person the rule protects. | | Some takeovers are agreed | Only a person can tell a duplicate from an agreed takeover. One of the 2 PRs it would flag today (below), apache#8484, is an agreed takeover, and a failing check would block it. | | It would go stale | The check runs again only when this PR is opened or edited. Closing the other PR or unassigning someone doesn't re-run it, so a failed check would stay red until someone edits the PR. | On its first post, the comment @-mentions the claimant, so they get a notification. The apache#8149 claimant never got one. Closed issues are skipped, because the risk is two PRs in progress at once. Both queries read up to 100 linked PRs per issue. That is the GraphQL maximum; no issue in this repo has more than 5. There is no concurrency group: with the default queue, a group cancels a waiting run, and the body-edit unassign and the merge credit need every event. One known limitation: the comment is refreshed only when this PR is edited, not when the other PR closes. ### Any related issues, documentation, discussions? Closes apache#8676 Complements apache#8675, which adds an AGENTS.md rule to check claims before coding. This PR catches the cases that rule misses. ### How was this PR tested? Unit tests: 32 pass. Two of them replay the apache#8149 and apache#6674 incidents as GraphQL fixtures. ```bash bash .github/scripts/test_issue_claims.sh # tests 42, pass 42, fail 0 ``` The `infra` job picks these up through its `test_*.sh` discovery. Mutation check: reverting the credit fix fails the apache#8149 merge test, and dropping (b) fails the apache#6674 test. For `claimCommentChanges`, removing any one of the id sort, the Bot-only filter, the deletion of extra copies or the unchanged-body check fails at least one test. For `syncClaimComment`, limiting it to one pass fails both delayed-run tests. `pull_request_target` runs the workflow from `main`, so this PR can't run its own steps. Instead, I ran both edited `github-script` steps, extracted from the YAML, in a local harness. GraphQL calls went to the live API (read-only). Every REST write was recorded, not sent. | Scenario | Result | | --- | --- | | apache#8339 opened, fixture = apache#8149 as of 2026-09-02 | posts 1 comment: `apache#8149 · <claimant> (assignee) · apache#8267` | | edited, with an older bot comment and a human comment quoting the marker | updates only the bot comment | | edited again, comment already current | no write | | edited after `Closes apache#8149` is removed | deletes the bot comment; the human comment is left alone | | live: apache#8603 opened (apache#6674) | no conflict, no writes | | live: apache#8484 edited | posts 1 comment naming the apache#6939 assignee | | credit step, fixture = apache#8339's merge on 2026-09-25 | apache#8149 keeps its claimant (`kept (own open PR)=[…]`); an assignee without a PR on apache#8338 is still removed | | credit step, live query on apache#8339 | query valid, no writes | | race: this run posts, then its re-list shows an older copy from another run | deletes its own copy (and first updates the older one if it is stale) | | re-list after posting fails, or a delete returns 404 | logs a warning and stops; a 404 skips only that one delete | | edited, with two copies left by an earlier race | updates the oldest, deletes the other | | delayed run: reads the claim, a newer run removes it, then the delayed run posts | its re-read finds nothing to flag and deletes its own post | | delayed run: writes old text over a newer run's comment | its re-read restores the newer text | | live: guard and credit queries with `first: 100` | both valid against the schema | I also modelled two racing runs against one shared comment list and tried every ordering of their API calls, using the real `claimCommentChanges`. Whenever both runs see the same state, exactly one up-to-date comment is left; 104 orderings for an open followed by a quick edit. When both runs find nothing to flag, no comment is left. Dry run over every open PR: 218 open, 188 with closing references, **2 flagged**. Both are real claims: | PR | Issue | Claim | | --- | --- | --- | | apache#5558 (draft) | apache#5422 | the issue's author assigned themselves | | apache#8484 | apache#6939 | a `/take` that was never released; a maintainer later told apache#8484's author to go ahead | ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
Any updates here @aglinxinyuan ? |
aglinxinyuan
left a comment
There was a problem hiding this comment.
Thanks for picking this up. The enum, the executor change and the empty-list guard look right, and defaulting to OR is the right call. Two things need fixing before this merges; smaller notes are inline.
-
After the rebase, Python export ignores AND. Since this branched, #8502 added
generateStandaloneCode()toSpecializedFilterOpDesc. It joins the predicates with" | ". That file merges cleanly with your change; onlySpecializedFilterOpDescSpecconflicts. So once rebased, an AND Filter still exports as OR:age > 30ANDcountry = "US"comes out as(…age > 30) | (…country == "US"), which keeps rows matching either predicate. Please:- join on the combinator:
val joiner = if (predicateCombinator == PredicateCombinator.AND) " & " else " | " s"out1df = in1df[${conditions.mkString(joiner)}].reset_index(drop=True)"
- fix the two comments that say the executor uses
predicates.exists, one in the descriptor and one in its spec; - add a two-predicate AND export test.
When you resolve the spec conflict, keep both sides.
- join on the combinator:
-
required = truemakes saved Filters invalid. See the inline comment. "No migration needed" holds for the backend, but not for the frontend validator.
Also, docs/reference/operators/data-cleaning/_index.md:42 still says Filter uses OR only.
The Filter operator hard-coded OR semantics, so users needing AND had to chain multiple Filter operators. Add a PredicateCombinator enum exposed as a descriptor property, defaulting to OR so existing workflows are unaffected — a stored desc without the field deserializes to OR, so no migration is needed. Empty predicate lists keep dropping every tuple in both modes; a bare forall would have flipped that to passing everything.
8425ccd to
d7a2586
Compare
|
Rebased onto main and addressed the review. The Python export now joins with & under AND, the exists comments are reworded, and there's a two-predicate AND export test. predicateCombinator is now optional, so saved Filters stay valid. _index.md is updated. Filter tests pass locally (76 passed; 4 pandas parity tests skipped locally because pandas isn't installed). |
|
/request-review @aglinxinyuan |
What changes were proposed in this PR?
The
Filteroperator hard-codes OR semantics — a tuple passes as soon as any onepredicate matches. Users who need AND (e.g.
age > 30 AND country = "US") have tochain multiple
Filteroperators, which adds workflow clutter and an extra hop percondition.
This PR makes the combination configurable:
PredicateCombinatorenum (OR→"any (OR)",AND→"all (AND)"),following the
@JsonValue/@JsonCreatorpattern already used byComparisonTypein the same package.predicateCombinatorproperty onSpecializedFilterOpDesc,defaulting to
ORand declared abovepredicatesso the dropdown renders above thepredicate rows. The UI renders it from the JSON schema, so there are no frontend
changes.
SpecializedFilterOpExecpicks the filter function once:forallfor AND,existsfor OR, with both cases named so a future combinator gets a compiler warning.generateStandaloneCode) joins conditions with&under AND and|under OR.Backward compatibility: the property is not in the schema's
requiredlist, soFilters saved before this PR (which lack the key) still pass the frontend validator.
A missing value deserializes to
OR, and so does an explicitnull(
@JsonSetter(nulls = Nulls.SKIP)), so the executor never sees a null combinator.Existing workflows behave exactly as before and no migration is needed.
One behavioral note for reviewers: an empty predicate list continues to drop every
tuple in both modes, in the engine and in the export. A bare
forallreturnstrueon an empty list, which would have silently flipped an empty Filter from "drop
everything" to "pass everything", so the AND branch is guarded with
predicates.nonEmpty &&. The docs call this out.Nested predicate trees (e.g.
(A AND B) OR C) are intentionally out of scope — theyneed more extensive UI work and are noted as a follow-up in the issue.
Any related issues, documentation, discussions?
Closes #6939
Operator documentation updated in the same PR:
docs/reference/operators/data-cleaning/filter.mdanddocs/reference/operators/data-cleaning/_index.md.How was this PR tested?
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.filter.*": 7 suites,76 passed, 0 failed. 4 pandas parity tests (tagged
IntegrationTest) were canceledlocally because the local Python has no pandas; they are unchanged by this PR.
PredicateCombinatorSpec(new): wire names viagetName, case-insensitivefromString, rejection of an unknown name, and the Jackson round-trip.SpecializedFilterOpDescSpec:OR; descriptor JSON with nopredicateCombinatordeserializes to
OR; an explicitANDsurvives the polymorphicLogicalOpround-trip;
"xor"fails with aValueInstantiationExceptionnaming the field, andan executor built from
"predicateCombinator": nullbehaves as OR;predicateCombinatorout ofrequired, listsit before
predicates, and defaults it toany (OR);|under OR and&under AND; no predicates keepsno rows under either combinator.
SpecializedFilterOpExecSpec: with two predicates where only one matches, ORkeeps the tuple and AND drops it, with the failing predicate in either position; AND
keeps it when both match; AND with an empty predicate list drops everything; AND and
OR agree on a single predicate. A
beforeblock resets the shared descriptor betweentests.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)