Repository navigation
Conversation
…port The standalone helpers are packaged as `texera-support`, generated from the operators' helper blocks. An exported script imports the helpers it calls from it instead of defining them, and the export downloads as a zip with the module as `texera_support.py` next to `script.py`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9240 +/- ##
============================================
- Coverage 91.80% 91.59% -0.22%
- Complexity 5025 5034 +9
============================================
Files 1279 1282 +3
Lines 56557 56880 +323
Branches 7419 7401 -18
============================================
+ Hits 51924 52100 +176
- Misses 2768 2918 +150
+ Partials 1865 1862 -3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 364 | 0.222 | 26,109/34,915/34,915 us | 🟢 -20.9% / 🔴 +154.3% |
| ⚪ | bs=100 sw=10 sl=64 | 824 | 0.503 | 119,636/135,300/135,300 us | ⚪ within ±5% / 🔴 +52.8% |
| ⚪ | bs=1000 sw=10 sl=64 | 935 | 0.571 | 1,063,566/1,140,034/1,140,034 us | ⚪ within ±5% / 🔴 +37.9% |
Baseline details
Latest main 6db4a64 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 364 tuples/sec | 367 tuples/sec | 1,005 tuples/sec | -0.8% | -63.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.222 MB/s | 0.224 MB/s | 0.614 MB/s | -0.9% | -63.8% |
| bs=10 sw=10 sl=64 | p50 | 26,109 us | 23,762 us | 10,325 us | +9.9% | +152.9% |
| bs=10 sw=10 sl=64 | p95 | 34,915 us | 44,137 us | 13,728 us | -20.9% | +154.3% |
| bs=10 sw=10 sl=64 | p99 | 34,915 us | 44,137 us | 16,323 us | -20.9% | +113.9% |
| bs=100 sw=10 sl=64 | throughput | 824 tuples/sec | 837 tuples/sec | 1,347 tuples/sec | -1.6% | -38.8% |
| bs=100 sw=10 sl=64 | MB/s | 0.503 MB/s | 0.511 MB/s | 0.822 MB/s | -1.6% | -38.8% |
| bs=100 sw=10 sl=64 | p50 | 119,636 us | 116,709 us | 80,846 us | +2.5% | +48.0% |
| bs=100 sw=10 sl=64 | p95 | 135,300 us | 141,320 us | 88,523 us | -4.3% | +52.8% |
| bs=100 sw=10 sl=64 | p99 | 135,300 us | 141,320 us | 112,010 us | -4.3% | +20.8% |
| bs=1000 sw=10 sl=64 | throughput | 935 tuples/sec | 940 tuples/sec | 1,394 tuples/sec | -0.5% | -32.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.571 MB/s | 0.574 MB/s | 0.851 MB/s | -0.5% | -32.9% |
| bs=1000 sw=10 sl=64 | p50 | 1,063,566 us | 1,058,761 us | 787,853 us | +0.5% | +35.0% |
| bs=1000 sw=10 sl=64 | p95 | 1,140,034 us | 1,116,611 us | 826,583 us | +2.1% | +37.9% |
| bs=1000 sw=10 sl=64 | p99 | 1,140,034 us | 1,116,611 us | 872,816 us | +2.1% | +30.6% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,548.77,200,128000,364,0.222,26108.93,34914.99,34914.99
1,100,10,64,20,2426.07,2000,1280000,824,0.503,119636.10,135299.66,135299.66
2,1000,10,64,20,21387.33,20000,12800000,935,0.571,1063565.62,1140034.18,1140034.18Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # workflow-compiling-service/src/main/scala/org/apache/texera/amber/translator/WorkflowToPythonTranslator.scala # workflow-compiling-service/src/main/scala/org/apache/texera/service/resource/WorkflowToPythonResource.scala # workflow-compiling-service/src/test/scala/org/apache/texera/amber/translator/WorkflowToPythonTranslatorSpec.scala
The discussion on where the helpers go asked for a clearer name than texera_support. The module, its directory, the regenerate variable, the response field and the frontend property now all say helpers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A copied script that imports its helpers does not run on its own. The copy notice now says the module comes with the zip download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
left a comment
There was a problem hiding this comment.
The exported zip still has a filename collision.
The zip unzips script.py and texera_helpers.py beside the sources, so a source by either name was overwritten by the exported code. Both names now count as taken when sources are named. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What changes were proposed in this PR?
An exported Python script no longer defines the helpers it calls. It imports them from a new module,
texera_helpers, and the export downloads as a zip that holds the module next to the script.texera-helpers/holds the module. Itstexera_helpers/__init__.pyis generated from the helper blocks the operators already declare, listed inStandaloneHelpers.moduleHelpers.StandaloneHelpersSpecfails when the file is stale, and rewrites it withTEXERA_HELPERS_WRITE=1.from texera_helpers import ...with only the names the script calls. It refuses an operator whose helper block is not in the module, so a new helper cannot be left out.script.pyandtexera_helpers.py, so the unzipped folder runs withpython script.pyand nothing installed.texera_helpers.pyin the zip download next to it.Why a zip and not a PyPI package
The proposal was to publish the helpers as a PyPI package, so a script could start with
pip install texera-helpers. That needs an Apache release to publish from, a PPMC vote on it, and a decision on the package name, and every exported script would need the release that matches its Texera version.The zip needs none of that. The module comes from the same Texera that wrote the script, so the two always match, and a user installs nothing for it. If the project later publishes the package, the script keeps the same
from texera_helpers import ...line and only the zip step goes away.What it looks like
The export modal on a workflow that reads a CSV, filters, sorts, limits and draws a bar chart. The script imports its helpers from the module, and the button left of copy downloads the zip.
Any related issues, documentation, discussions?
Related to #8325. The layout is discussed in #9368, where the zip (option B) was preferred. The module was named
texera_supportuntil that discussion asked for a clearer name.The verify harness from #8325 runs the exported scripts. Once it lands, its
StandaloneRunnerandTranslatedPlanRunSpecneedPYTHONPATHset totexera-helpers, and that change will be added here.How was this PR tested?
On this branch, merged with main:
StandaloneHelpersSpecchecks the generated module against the helper blocks.WorkflowToPythonTranslatorSpecchecks that a helper is imported rather than defined, that a name in a comment or a string is not imported, and that an operator whose helper the module lacks is refused.WorkflowToPythonResourceSpecchecks that the module comes back with a script that imports from it, and not with one that does not.menu.component.spec.tschecks the zip holds the script and the module, and only the script when nothing is imported, and that the copy notice names the module when the script imports one.Before the rename, on the fork's integration branch, which has the verify harness:
OperatorBehaviorSpecran every operator's exported script with the module onPYTHONPATH, 151 passed and 2 ignored. By hand, a downloaded zip, unzipped, ran withpython script.py, also from another directory.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with Claude Code