Skip to content

test(workflow-operator): share Python test discovery - #9406

Merged
aglinxinyuan merged 4 commits into
apache:mainfrom
aglinxinyuan:codex/test/python-test-support
Nov 15, 2026
Merged

aglinxinyuan merged 4 commits into
apache:mainfrom
aglinxinyuan:codex/test/python-test-support

Conversation

@aglinxinyuan

@aglinxinyuan aglinxinyuan commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Share Python discovery and package probes across 34 operator specs, removing 892 net lines. PythonTestSupport.configuredPython tries the trimmed UdfConfig.pythonPath, which UDF_PYTHON_PATH sets, and then python3, python and py; PythonWorkerPoolSpec keeps its PATH-only lookup. Probe deadlines, lazy callers and integration tags are unchanged. Each spec keeps its existing failure or cancellation policy and assertions. The shared probe discards the child's output instead of leaving it in a pipe nothing reads, and PythonCodeRawInvalidTextSpec's failure message lists what the lookup tried.

Any related issues, documentation, discussions?

Closes #9401.

How was this PR tested?

With JDK 17 and Python 3.12 (pandas, numpy and plotly installed) first on the PATH:

sbt "WorkflowOperator/testOnly *SamplingHelpersSpec *DistinctOpDescSpec *HashJoinOpDescSpec *HuggingFaceBinaryValueBehaviorSpec *IfOpDescSpec *IntervalJoinOpDescSpec *MachineLearningScorerOpDescSpec *RegexOpDescSpec *ReservoirSamplingOpDescSpec *DifferenceOpDescSpec *IntersectOpDescSpec *SymmetricDifferenceOpDescSpec *StableMergeSortOpDescSpec *SortPartitionsOpDescSpec *ArrowSourceOpDescSpec *CSVScanSourceOpDescSpec *CSVOldScanSourceOpDescSpec *JSONLScanSourceOpDescSpec *TextInputSourceOpDescSpec *SubstringSearchOpDescSpec *TypeCastingOpDescSpec *ImageVisualizerOpDescSpec *CarpetPlotOpDescSpec *FigureFactoryTableOpDescSpec *FilledAreaPlotOpDescSpec *GaugeChartOpDescSpec *Histogram2DOpDescSpec *PolarChartOpDescSpec *TablesPlotOpDescSpec *TreePlotOpDescSpec *UrlVizOpDescSpec *WaterfallChartOpDescSpec *PythonCodeRawInvalidTextSpec *PythonWorkerPoolSpec"
sbt "WorkflowOperator/Test/compile" "WorkflowOperator/scalafmtCheckAll" "WorkflowOperator/scalafixAll --check"

The 34 suites ran 334 cases: 333 passed, and the existing Windows URL HTML line-ending assertion in UrlVizOpDescSpec failed as it does on main. Running main's versions of the same 34 specs gives the same test names, order and outcomes in every suite, and the specs keep the same 61 integration-tag registrations.

Apart from PythonWorkerPoolSpec, every replaced lookup tried the configured interpreter and then python3, python and py, as configuredPython does. The three visualization specs that read UDF_PYTHON_PATH directly now get the same value through udf.conf, trimmed. Every replaced probe imports the same modules within the same deadline. Formatting and lint checks pass.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-6), Claude Code (Claude Opus 5.5, Claude Fable 5.2).

Keep interpreter order, package deadlines, integration tags and each caller’s fail-or-cancel policy in one test utility.
@aglinxinyuan aglinxinyuan added refactor Refactor the code common labels Nov 11, 2026
Copilot AI balanced review requested due to automatic review settings November 11, 2026 00:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions github-actions Bot removed the refactor Refactor the code label Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @kz930, @Xiao-zhen-Liu, @PG1204
    You can notify them by mentioning @kz930, @Xiao-zhen-Liu, @PG1204 in a comment.

@codecov-commenter

codecov-commenter commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.77%. Comparing base (b83963a) to head (e9cbf96).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9406      +/-   ##
============================================
- Coverage     91.79%   91.77%   -0.02%     
- Complexity     5022     5024       +2     
============================================
  Files          1283     1279       -4     
  Lines         56874    56215     -659     
  Branches       7440     7383      -57     
============================================
- Hits          52205    51590     -615     
+ Misses         2800     2770      -30     
+ Partials       1869     1855      -14     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.50% <ø> (-0.06%) ⬇️ Carriedforward from 6db4a64
amber 85.76% <ø> (+0.01%) ⬆️
computing-unit-managing-service 64.05% <ø> (ø)
config-service 87.73% <ø> (ø)
file-service 82.31% <ø> (ø)
frontend 96.55% <ø> (+0.11%) ⬆️ Carriedforward from 6db4a64
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.50% <ø> (-0.01%) ⬇️ Carriedforward from 6db4a64
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 10 worse · ⚪ 3 noise (<±5%) · 0 without baseline

Compared against main b83963a benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 564 0.344 17,540/24,994/24,994 us 🔴 +28.5% / 🔴 +76.3%
🔴 bs=100 sw=10 sl=64 1,178 0.719 82,677/112,443/112,443 us 🔴 +10.5% / 🔴 +20.6%
🟢 bs=1000 sw=10 sl=64 1,429 0.872 699,833/751,484/751,484 us 🟢 -6.0% / 🟢 -18.4%
Baseline details

Latest main b83963a from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 564 tuples/sec 657 tuples/sec 924.59 tuples/sec -14.2% -39.0%
bs=10 sw=10 sl=64 MB/s 0.344 MB/s 0.401 MB/s 0.564 MB/s -14.2% -39.0%
bs=10 sw=10 sl=64 p50 17,540 us 16,225 us 11,019 us +8.1% +59.2%
bs=10 sw=10 sl=64 p95 24,994 us 19,456 us 14,177 us +28.5% +76.3%
bs=10 sw=10 sl=64 p99 24,994 us 19,456 us 17,317 us +28.5% +44.3%
bs=100 sw=10 sl=64 throughput 1,178 tuples/sec 1,279 tuples/sec 1,238 tuples/sec -7.9% -4.9%
bs=100 sw=10 sl=64 MB/s 0.719 MB/s 0.781 MB/s 0.756 MB/s -7.9% -4.9%
bs=100 sw=10 sl=64 p50 82,677 us 74,835 us 85,949 us +10.5% -3.8%
bs=100 sw=10 sl=64 p95 112,443 us 102,053 us 93,224 us +10.2% +20.6%
bs=100 sw=10 sl=64 p99 112,443 us 102,053 us 115,841 us +10.2% -2.9%
bs=1000 sw=10 sl=64 throughput 1,429 tuples/sec 1,370 tuples/sec 1,278 tuples/sec +4.3% +11.8%
bs=1000 sw=10 sl=64 MB/s 0.872 MB/s 0.836 MB/s 0.78 MB/s +4.3% +11.8%
bs=1000 sw=10 sl=64 p50 699,833 us 735,860 us 836,081 us -4.9% -16.3%
bs=1000 sw=10 sl=64 p95 751,484 us 799,696 us 878,656 us -6.0% -14.5%
bs=1000 sw=10 sl=64 p99 751,484 us 799,696 us 921,222 us -6.0% -18.4%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,354.71,200,128000,564,0.344,17539.82,24994.44,24994.44
1,100,10,64,20,1697.71,2000,1280000,1178,0.719,82677.19,112443.00,112443.00
2,1000,10,64,20,14000.37,20000,12800000,1429,0.872,699833.11,751483.74,751483.74

@kunwp1
kunwp1 requested a lite review from Copilot November 12, 2026 23:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@kunwp1 kunwp1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Move the hash-join, interval-join, difference, intersect, symmetric
difference and Hugging Face binary-value specs onto PythonTestSupport;
each probe keeps its modules. PythonCodeRawInvalidTextSpec uses the
shared tag and import probe and keeps its fail-or-cancel policy.

configuredPython reads UdfConfig.pythonPath, which UDF_PYTHON_PATH
already overrides, so environmentPython goes and the image, 2D
histogram and URL visualization specs use configuredPython.

Restore the case-insensitive column comments in the regex and
substring-search specs, and drop the two resolution comments that no
longer describe any code.
@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Nov 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 13, 2026

@bobbai00 bobbai00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 0 must-fix · 2 advisory · 2 polish: a clean, behavior-preserving test refactor. The notes below are optional.

Correctness (1)

  • PythonTestSupport.scala:50: the probe never reads the child's output, so heavy output reads as a failure (advisory, see inline)

Simplifications (1)

  • PythonTestSupport.scala:29: main gained a fourth copy in DistinctOpDescSpec after this branch was cut (advisory, see inline)

Polish: 2 quick touch-ups (see inline comments).

Verification trace

Checked that the refactor preserves behavior. Each of the 29 import probes passes the module list and deadline its removed copy used (60 s, and 120 s in Histogram2DOpDescSpec), and the command string is built the same way. A script comparing base and head for the 33 specs found the same test names in every file and 60 tag registrations on both sides. UdfConfig.pythonPath reads the same key from the same resource the copies parsed. An in-memory merge with main at ed53531 is clean and leaves no call to a removed helper. The pipe finding was reproduced with the same ProcessBuilder calls on JDK 17.

DistinctOpDescSpec gained its own interpreter lookup and import probe on
main (apache#8975). It now uses configuredPython and canImport, like the other
converted specs, and drops its local NeedsPythonPackages.

The shared probe discards the child's output instead of leaving it in a
pipe that nothing reads, since a child blocks once that pipe fills.
configuredPython documents its lookup order, and the failure message in
PythonCodeRawInvalidTextSpec names what was tried.
@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Nov 15, 2026
Merged via the queue into apache:main with commit 13aa747 Nov 15, 2026
39 checks passed
@aglinxinyuan
aglinxinyuan deleted the codex/test/python-test-support branch November 15, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Share Python discovery and package probes in operator tests

5 participants