Skip to content

feat: export Python scripts that import their helpers from texera-helpers - #9240

Open
kz930 wants to merge 8 commits into
apache:mainfrom
kz930:feat/texera-support
Open

kz930 wants to merge 8 commits into
apache:mainfrom
kz930:feat/texera-support

Conversation

@kz930

@kz930 kz930 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

An exported Python script no longer defines the helpers it calls. It imports them from a new module, texera_helpers, and the export downloads as a zip that holds the module next to the script.

  • texera-helpers/ holds the module. Its texera_helpers/__init__.py is generated from the helper blocks the operators already declare, listed in StandaloneHelpers.moduleHelpers. StandaloneHelpersSpec fails when the file is stale, and rewrites it with TEXERA_HELPERS_WRITE=1.
  • The translator writes from texera_helpers import ... with only the names the script calls. It refuses an operator whose helper block is not in the module, so a new helper cannot be left out.
  • The export endpoint returns the module with the script when the script imports from it.
  • The export modal gets a download button. The zip holds script.py and texera_helpers.py, so the unzipped folder runs with python script.py and nothing installed.
  • Copying the script still works. When the script imports from the module, the copy notice says it needs the texera_helpers.py in the zip download next to it.

Why a zip and not a PyPI package

The proposal was to publish the helpers as a PyPI package, so a script could start with pip install texera-helpers. That needs an Apache release to publish from, a PPMC vote on it, and a decision on the package name, and every exported script would need the release that matches its Texera version.

The zip needs none of that. The module comes from the same Texera that wrote the script, so the two always match, and a user installs nothing for it. If the project later publishes the package, the script keeps the same from texera_helpers import ... line and only the zip step goes away.

What it looks like

The export modal on a workflow that reads a CSV, filters, sorts, limits and draws a bar chart. The script imports its helpers from the module, and the button left of copy downloads the zip.

pr9240-export-modal-texera-helpers

Any related issues, documentation, discussions?

Related to #8325. The layout is discussed in #9368, where the zip (option B) was preferred. The module was named texera_support until that discussion asked for a clearer name.

The verify harness from #8325 runs the exported scripts. Once it lands, its StandaloneRunner and TranslatedPlanRunSpec need PYTHONPATH set to texera-helpers, and that change will be added here.

How was this PR tested?

On this branch, merged with main:

  • StandaloneHelpersSpec checks the generated module against the helper blocks.
  • WorkflowToPythonTranslatorSpec checks that a helper is imported rather than defined, that a name in a comment or a string is not imported, and that an operator whose helper the module lacks is refused.
  • WorkflowToPythonResourceSpec checks that the module comes back with a script that imports from it, and not with one that does not.
  • menu.component.spec.ts checks the zip holds the script and the module, and only the script when nothing is imported, and that the copy notice names the module when the script imports one.

Before the rename, on the fork's integration branch, which has the verify harness: OperatorBehaviorSpec ran every operator's exported script with the module on PYTHONPATH, 151 passed and 2 ignored. By hand, a downloaded zip, unzipped, ran with python script.py, also from another directory.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

…port

The standalone helpers are packaged as `texera-support`, generated from the
operators' helper blocks. An exported script imports the helpers it calls from
it instead of defining them, and the export downloads as a zip with the module
as `texera_support.py` next to `script.py`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added feature frontend Changes related to the frontend GUI docs Changes related to documentations common platform Non-amber Scala service paths labels Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @sshiv012, @Xiao-zhen-Liu
    You can notify them by mentioning @sshiv012, @Xiao-zhen-Liu in a comment.

@codecov-commenter

codecov-commenter commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.92857% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.59%. Comparing base (6db4a64) to head (39ff088).
⚠️ Report is 47 commits behind head on main.

Files with missing lines Patch % Lines
...c/app/workspace/component/menu/menu.component.html 61.53% 5 Missing ⚠️
...src/app/workspace/component/menu/menu.component.ts 81.81% 1 Missing and 1 partial ⚠️
.../amber/translator/WorkflowToPythonTranslator.scala 93.33% 0 Missing and 1 partial ⚠️
...ra/service/resource/WorkflowToPythonResource.scala 83.33% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9240      +/-   ##
============================================
- Coverage     91.80%   91.59%   -0.22%     
- Complexity     5025     5034       +9     
============================================
  Files          1279     1282       +3     
  Lines         56557    56880     +323     
  Branches       7419     7401      -18     
============================================
+ Hits          51924    52100     +176     
- Misses         2768     2918     +150     
+ Partials       1865     1862       -3     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.53% <ø> (ø) Carriedforward from 50af782
amber 85.36% <100.00%> (-0.43%) ⬇️
computing-unit-managing-service 64.02% <ø> (-0.03%) ⬇️
config-service 87.73% <ø> (ø)
file-service 82.16% <ø> (-0.15%) ⬇️
frontend 96.40% <70.83%> (-0.11%) ⬇️
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.50% <ø> (ø) Carriedforward from 50af782
workflow-compiling-service 80.42% <90.47%> (+2.11%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 1 worse · ⚪ 12 noise (<±5%) · 0 without baseline

Compared against main 6db4a64 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 364 0.222 26,109/34,915/34,915 us 🟢 -20.9% / 🔴 +154.3%
⚪ bs=100 sw=10 sl=64 824 0.503 119,636/135,300/135,300 us ⚪ within ±5% / 🔴 +52.8%
⚪ bs=1000 sw=10 sl=64 935 0.571 1,063,566/1,140,034/1,140,034 us ⚪ within ±5% / 🔴 +37.9%
Baseline details

Latest main 6db4a64 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 364 tuples/sec 367 tuples/sec 1,005 tuples/sec -0.8% -63.8%
bs=10 sw=10 sl=64 MB/s 0.222 MB/s 0.224 MB/s 0.614 MB/s -0.9% -63.8%
bs=10 sw=10 sl=64 p50 26,109 us 23,762 us 10,325 us +9.9% +152.9%
bs=10 sw=10 sl=64 p95 34,915 us 44,137 us 13,728 us -20.9% +154.3%
bs=10 sw=10 sl=64 p99 34,915 us 44,137 us 16,323 us -20.9% +113.9%
bs=100 sw=10 sl=64 throughput 824 tuples/sec 837 tuples/sec 1,347 tuples/sec -1.6% -38.8%
bs=100 sw=10 sl=64 MB/s 0.503 MB/s 0.511 MB/s 0.822 MB/s -1.6% -38.8%
bs=100 sw=10 sl=64 p50 119,636 us 116,709 us 80,846 us +2.5% +48.0%
bs=100 sw=10 sl=64 p95 135,300 us 141,320 us 88,523 us -4.3% +52.8%
bs=100 sw=10 sl=64 p99 135,300 us 141,320 us 112,010 us -4.3% +20.8%
bs=1000 sw=10 sl=64 throughput 935 tuples/sec 940 tuples/sec 1,394 tuples/sec -0.5% -32.9%
bs=1000 sw=10 sl=64 MB/s 0.571 MB/s 0.574 MB/s 0.851 MB/s -0.5% -32.9%
bs=1000 sw=10 sl=64 p50 1,063,566 us 1,058,761 us 787,853 us +0.5% +35.0%
bs=1000 sw=10 sl=64 p95 1,140,034 us 1,116,611 us 826,583 us +2.1% +37.9%
bs=1000 sw=10 sl=64 p99 1,140,034 us 1,116,611 us 872,816 us +2.1% +30.6%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,548.77,200,128000,364,0.222,26108.93,34914.99,34914.99
1,100,10,64,20,2426.07,2000,1280000,824,0.503,119636.10,135299.66,135299.66
2,1000,10,64,20,21387.33,20000,12800000,935,0.571,1063565.62,1140034.18,1140034.18

kz930 and others added 2 commits October 31, 2026 21:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930

kz930 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 This is a draft until #8325's PRs land, but the design is ready for a look: the export now imports its helpers from a generated texera_support module and downloads as a zip with it. Feedback on the approach is welcome.

kz930 and others added 3 commits November 10, 2026 20:56
# Conflicts:
#	workflow-compiling-service/src/main/scala/org/apache/texera/amber/translator/WorkflowToPythonTranslator.scala
#	workflow-compiling-service/src/main/scala/org/apache/texera/service/resource/WorkflowToPythonResource.scala
#	workflow-compiling-service/src/test/scala/org/apache/texera/amber/translator/WorkflowToPythonTranslatorSpec.scala
The discussion on where the helpers go asked for a clearer name than
texera_support. The module, its directory, the regenerate variable, the
response field and the frontend property now all say helpers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A copied script that imports its helpers does not run on its own. The
copy notice now says the module comes with the zip download.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930 kz930 changed the title feat: export Python scripts that import their helpers from texera-support feat: export Python scripts that import their helpers from texera-helpers Nov 11, 2026
@kz930
kz930 marked this pull request as ready for review November 11, 2026 17:20
@kz930

kz930 commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 Ready for review. The export imports its helpers from a generated texera_helpers module and downloads as a zip with it, the layout #9368 preferred; the module was renamed from texera_support there, and copying the script now says it needs that module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exported zip still has a filename collision.

The zip unzips script.py and texera_helpers.py beside the sources, so a
source by either name was overwritten by the exported code. Both names now
count as taken when sources are named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common docs Changes related to documentations feature frontend Changes related to the frontend GUI platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants