Skip to content

feat(workflow-operator): make Python UDF templates work after uncommenting one class - #8770

Merged
Xiao-zhen-Liu merged 4 commits into
apache:mainfrom
sshiv012:sshiv012/tex-86-udf-templates
Nov 7, 2026
Merged

Xiao-zhen-Liu merged 4 commits into
apache:mainfrom
sshiv012:sshiv012/tex-86-udf-templates

Conversation

@sshiv012

@sshiv012 sshiv012 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The Python UDF, 2-in Python UDF and 1-out Python UDF templates were fully commented out, including from pytexera import *. They also never said to keep only one class. So uncommenting a class failed with NameError, and keeping several failed the one-Operator check.

  • from pytexera import * is now active.
  • A short header explains the rules:
# Keep exactly ONE class below: uncomment it and delete the others.
# Its method must match its base class:
#   UDFOperatorV2 -> process_tuple, UDFBatchOperator -> process_batch,
#   UDFTableOperator -> process_table
# Always `yield` results (never `return` a value). Put setup code in open();
# if you override __init__, call super().__init__() first.
  • The 2-in template also says that port 0 (model) is read before port 1 (tuples).
  • The UiParameter lines are kept.
  • The template copies in docs/reference/operators/user-defined-functions/python/ are synced.

Saved workflows keep their own code and are not affected.

Before, every line started with #:

# Choose from the following templates:
# ...
# from pytexera import *
#
# class ProcessTupleOperator(UDFOperatorV2):
# ...

After, in the code editor:

template-after

Any related issues, documentation, discussions?

Part of #8736. Split out of #8737.

How was this PR tested?

  • The specs for all three descriptors read the defaultValue. They check that the import is the only active line and the header is there, and that uncommenting any one class leaves exactly one active class.
  • New Python tests load the template shape and check that uncommenting each class loads without adding an import.
  • The template copy in the notebook-migration prompt is updated to match.
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.udf.python.*" "WorkflowOperator/scalafmtCheck" "WorkflowOperator/Test/scalafmtCheck"
cd amber && pytest -m "not integration" src/test/python/core/architecture/managers/test_executor_manager.py
cd frontend && yarn eslint src/app/workspace/service/notebook-migration/migration-prompts.ts

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

…nting one class

The Python UDF, 2-in Python UDF and 1-out Python UDF templates were
fully commented out, including `from pytexera import *`, and did not
say to keep a single class. Uncommenting a class then failed with
NameError, and keeping several failed the one-Operator check.

- Keep `from pytexera import *` active in all three templates
- Add a header: keep exactly one class, the base class to method
  pairing, `yield` not `return`, open() and super().__init__()
- 2-in template: say port 0 (model) is read before port 1 (tuples)
- Keep the UiParameter guidance lines
- Sync the template copies in the operator reference docs

Saved workflows keep their own code and are unaffected.

Refs apache#8736

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aicam, @carloea2, @zyratlo
    You can notify them by mentioning @aicam, @carloea2, @zyratlo in a comment.

@codecov-commenter

codecov-commenter commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.32%. Comparing base (57faa6b) to head (3ca3c9e).

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8770   +/-   ##
=========================================
  Coverage     92.32%   92.32%           
- Complexity     5045     5046    +1     
=========================================
  Files          1260     1260           
  Lines         54026    54026           
  Branches       6744     6744           
=========================================
+ Hits          49879    49880    +1     
+ Misses         2487     2486    -1     
  Partials       1660     1660           
Flag Coverage Δ *Carryforward flag
access-control-service 77.38% <ø> (ø)
agent-service 99.16% <ø> (ø) Carriedforward from d9c50f2
amber 87.32% <ø> (+0.01%) ⬆️
computing-unit-managing-service 60.48% <ø> (ø)
config-service 87.37% <ø> (ø)
file-service 82.00% <ø> (ø)
frontend 96.54% <ø> (-0.01%) ⬇️
notebook-migration-service 83.73% <ø> (ø)
pyamber 98.57% <ø> (ø)
workflow-compiling-service 72.43% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 7 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main 57faa6b benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 656 0.4 15,355/21,358/21,358 us 🔴 +33.4% / 🔴 +46.3%
🔴 bs=100 sw=10 sl=64 1,600 0.976 59,938/89,190/89,190 us 🔴 +14.0% / 🟢 +34.4%
🟢 bs=1000 sw=10 sl=64 1,898 1.158 520,473/586,243/586,243 us 🟢 -5.1% / 🟢 +50.7%
Baseline details

Latest main 57faa6b from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 656 tuples/sec 765 tuples/sec 929.17 tuples/sec -14.2% -29.4%
bs=10 sw=10 sl=64 MB/s 0.4 MB/s 0.467 MB/s 0.567 MB/s -14.3% -29.5%
bs=10 sw=10 sl=64 p50 15,355 us 13,857 us 11,110 us +10.8% +38.2%
bs=10 sw=10 sl=64 p95 21,358 us 16,012 us 14,603 us +33.4% +46.3%
bs=10 sw=10 sl=64 p99 21,358 us 16,012 us 17,357 us +33.4% +23.1%
bs=100 sw=10 sl=64 throughput 1,600 tuples/sec 1,608 tuples/sec 1,191 tuples/sec -0.5% +34.4%
bs=100 sw=10 sl=64 MB/s 0.976 MB/s 0.982 MB/s 0.727 MB/s -0.6% +34.3%
bs=100 sw=10 sl=64 p50 59,938 us 60,137 us 88,770 us -0.3% -32.5%
bs=100 sw=10 sl=64 p95 89,190 us 78,251 us 97,256 us +14.0% -8.3%
bs=100 sw=10 sl=64 p99 89,190 us 78,251 us 115,734 us +14.0% -22.9%
bs=1000 sw=10 sl=64 throughput 1,898 tuples/sec 1,930 tuples/sec 1,259 tuples/sec -1.7% +50.7%
bs=1000 sw=10 sl=64 MB/s 1.158 MB/s 1.178 MB/s 0.769 MB/s -1.7% +50.7%
bs=1000 sw=10 sl=64 p50 520,473 us 506,181 us 857,339 us +2.8% -39.3%
bs=1000 sw=10 sl=64 p95 586,243 us 617,977 us 893,239 us -5.1% -34.4%
bs=1000 sw=10 sl=64 p99 586,243 us 617,977 us 923,509 us -5.1% -36.5%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,304.99,200,128000,656,0.400,15354.72,21358.12,21358.12
1,100,10,64,20,1250.09,2000,1280000,1600,0.976,59938.28,89189.82,89189.82
2,1000,10,64,20,10539.34,20000,12800000,1898,1.158,520472.84,586243.02,586243.02

@sshiv012

sshiv012 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

/request-review @Xiao-zhen-Liu

@github-actions
github-actions Bot requested a review from Xiao-zhen-Liu October 2, 2026 20:27
…tion prompt

- A bare `return` to end a generator early is fine, so the templates
  now say "never `return` a value" (reference docs synced).
- The notebook-migration prompt still embedded the old, fully
  commented template; update its header and import line.
- The Python template test covers the Batch class too and says it is a
  reduced copy (the Scala specs check the real text).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sshiv012

sshiv012 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Fixed: templates now say "never return a value", and the notebook-migration prompt matches the new template.

@github-actions github-actions Bot added the frontend Changes related to the frontend GUI label Oct 8, 2026
defaultCode, activeLines and uncommentClass were copied into the three
template specs. Move them into PythonUdfTemplateTestUtils, which takes
the descriptor class, so a fix lands in one place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Xiao-zhen-Liu
Xiao-zhen-Liu added this pull request to the merge queue Oct 15, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 15, 2026
@Xiao-zhen-Liu
Xiao-zhen-Liu added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 21, 2026
@Xiao-zhen-Liu
Xiao-zhen-Liu added this pull request to the merge queue Oct 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 22, 2026
@Xiao-zhen-Liu
Xiao-zhen-Liu added this pull request to the merge queue Nov 6, 2026
Merged via the queue into apache:main with commit 53e3c0a Nov 7, 2026
60 of 62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common docs Changes related to documentations frontend Changes related to the frontend GUI pyamber

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants