Repository navigation
feat(workflow-operator): make Python UDF templates work after uncommenting one class - #8770
Conversation
…nting one class The Python UDF, 2-in Python UDF and 1-out Python UDF templates were fully commented out, including `from pytexera import *`, and did not say to keep a single class. Uncommenting a class then failed with NameError, and keeping several failed the one-Operator check. - Keep `from pytexera import *` active in all three templates - Add a header: keep exactly one class, the base class to method pairing, `yield` not `return`, open() and super().__init__() - 2-in template: say port 0 (model) is read before port 1 (tuples) - Keep the UiParameter guidance lines - Sync the template copies in the operator reference docs Saved workflows keep their own code and are unaffected. Refs apache#8736 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8770 +/- ##
=========================================
Coverage 92.32% 92.32%
- Complexity 5045 5046 +1
=========================================
Files 1260 1260
Lines 54026 54026
Branches 6744 6744
=========================================
+ Hits 49879 49880 +1
+ Misses 2487 2486 -1
Partials 1660 1660
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 656 | 0.4 | 15,355/21,358/21,358 us | 🔴 +33.4% / 🔴 +46.3% |
| 🔴 | bs=100 sw=10 sl=64 | 1,600 | 0.976 | 59,938/89,190/89,190 us | 🔴 +14.0% / 🟢 +34.4% |
| 🟢 | bs=1000 sw=10 sl=64 | 1,898 | 1.158 | 520,473/586,243/586,243 us | 🟢 -5.1% / 🟢 +50.7% |
Baseline details
Latest main 57faa6b from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 656 tuples/sec | 765 tuples/sec | 929.17 tuples/sec | -14.2% | -29.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.4 MB/s | 0.467 MB/s | 0.567 MB/s | -14.3% | -29.5% |
| bs=10 sw=10 sl=64 | p50 | 15,355 us | 13,857 us | 11,110 us | +10.8% | +38.2% |
| bs=10 sw=10 sl=64 | p95 | 21,358 us | 16,012 us | 14,603 us | +33.4% | +46.3% |
| bs=10 sw=10 sl=64 | p99 | 21,358 us | 16,012 us | 17,357 us | +33.4% | +23.1% |
| bs=100 sw=10 sl=64 | throughput | 1,600 tuples/sec | 1,608 tuples/sec | 1,191 tuples/sec | -0.5% | +34.4% |
| bs=100 sw=10 sl=64 | MB/s | 0.976 MB/s | 0.982 MB/s | 0.727 MB/s | -0.6% | +34.3% |
| bs=100 sw=10 sl=64 | p50 | 59,938 us | 60,137 us | 88,770 us | -0.3% | -32.5% |
| bs=100 sw=10 sl=64 | p95 | 89,190 us | 78,251 us | 97,256 us | +14.0% | -8.3% |
| bs=100 sw=10 sl=64 | p99 | 89,190 us | 78,251 us | 115,734 us | +14.0% | -22.9% |
| bs=1000 sw=10 sl=64 | throughput | 1,898 tuples/sec | 1,930 tuples/sec | 1,259 tuples/sec | -1.7% | +50.7% |
| bs=1000 sw=10 sl=64 | MB/s | 1.158 MB/s | 1.178 MB/s | 0.769 MB/s | -1.7% | +50.7% |
| bs=1000 sw=10 sl=64 | p50 | 520,473 us | 506,181 us | 857,339 us | +2.8% | -39.3% |
| bs=1000 sw=10 sl=64 | p95 | 586,243 us | 617,977 us | 893,239 us | -5.1% | -34.4% |
| bs=1000 sw=10 sl=64 | p99 | 586,243 us | 617,977 us | 923,509 us | -5.1% | -36.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,304.99,200,128000,656,0.400,15354.72,21358.12,21358.12
1,100,10,64,20,1250.09,2000,1280000,1600,0.976,59938.28,89189.82,89189.82
2,1000,10,64,20,10539.34,20000,12800000,1898,1.158,520472.84,586243.02,586243.02|
/request-review @Xiao-zhen-Liu |
…tion prompt - A bare `return` to end a generator early is fine, so the templates now say "never `return` a value" (reference docs synced). - The notebook-migration prompt still embedded the old, fully commented template; update its header and import line. - The Python template test covers the Batch class too and says it is a reduced copy (the Scala specs check the real text). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Fixed: templates now say "never |
defaultCode, activeLines and uncommentClass were copied into the three template specs. Move them into PythonUdfTemplateTestUtils, which takes the descriptor class, so a fix lands in one place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What changes were proposed in this PR?
The Python UDF, 2-in Python UDF and 1-out Python UDF templates were fully commented out, including
from pytexera import *. They also never said to keep only one class. So uncommenting a class failed withNameError, and keeping several failed the one-Operator check.from pytexera import *is now active.docs/reference/operators/user-defined-functions/python/are synced.Saved workflows keep their own code and are not affected.
Before, every line started with
#:After, in the code editor:
Any related issues, documentation, discussions?
Part of #8736. Split out of #8737.
How was this PR tested?
defaultValue. They check that the import is the only active line and the header is there, and that uncommenting any one class leaves exactly one active class.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)