Repository navigation
feat(pyamber): explain Python UDF load errors with cause and fix - #8768
Conversation
When a UDF fails to load, the message now says what to change: - no Operator class: says to uncomment or write exactly one class - abstract class: names the class, its base, the missing method, and the base that matches the method it did write - several classes: names them and says to keep one - NameError for a pytexera symbol: says to add `from pytexera import *` - @OVERRIDES mismatch: lists the base class to method pairing - any other load error: adds the line of the user's code that failed The existing "There should be one and only one Operator defined" wording stays as the first sentence. Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1c32b97)
… hint - An error with no UDF line and no hint is returned unchanged - An error that can't be rebuilt from one message (UnicodeDecodeError) is kept as is Refs apache#8736 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9a6cf94)
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8768 +/- ##
============================================
+ Coverage 92.30% 92.32% +0.01%
Complexity 5046 5046
============================================
Files 1260 1260
Lines 53887 53985 +98
Branches 6711 6732 +21
============================================
+ Hits 49741 49839 +98
Misses 2486 2486
Partials 1660 1660
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 360 | 0.22 | 27,111/38,764/38,764 us | 🔴 +34.6% / 🔴 +150.6% |
| 🔴 | bs=100 sw=10 sl=64 | 764 | 0.466 | 124,150/194,784/194,784 us | 🔴 +29.3% / 🔴 +87.0% |
| ⚪ | bs=1000 sw=10 sl=64 | 921 | 0.562 | 1,086,552/1,151,092/1,151,092 us | ⚪ within ±5% / 🔴 +16.8% |
Baseline details
Latest main 988c655 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 360 tuples/sec | 439 tuples/sec | 822.11 tuples/sec | -18.0% | -56.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.22 MB/s | 0.268 MB/s | 0.502 MB/s | -17.9% | -56.2% |
| bs=10 sw=10 sl=64 | p50 | 27,111 us | 21,747 us | 12,129 us | +24.7% | +123.5% |
| bs=10 sw=10 sl=64 | p95 | 38,764 us | 28,807 us | 15,469 us | +34.6% | +150.6% |
| bs=10 sw=10 sl=64 | p99 | 38,764 us | 28,807 us | 18,357 us | +34.6% | +111.2% |
| bs=100 sw=10 sl=64 | throughput | 764 tuples/sec | 826 tuples/sec | 1,058 tuples/sec | -7.5% | -27.8% |
| bs=100 sw=10 sl=64 | MB/s | 0.466 MB/s | 0.504 MB/s | 0.646 MB/s | -7.5% | -27.8% |
| bs=100 sw=10 sl=64 | p50 | 124,150 us | 118,862 us | 97,399 us | +4.4% | +27.5% |
| bs=100 sw=10 sl=64 | p95 | 194,784 us | 150,682 us | 104,184 us | +29.3% | +87.0% |
| bs=100 sw=10 sl=64 | p99 | 194,784 us | 150,682 us | 117,615 us | +29.3% | +65.6% |
| bs=1000 sw=10 sl=64 | throughput | 921 tuples/sec | 924 tuples/sec | 1,099 tuples/sec | -0.3% | -16.2% |
| bs=1000 sw=10 sl=64 | MB/s | 0.562 MB/s | 0.564 MB/s | 0.671 MB/s | -0.4% | -16.2% |
| bs=1000 sw=10 sl=64 | p50 | 1,086,552 us | 1,086,599 us | 946,743 us | -0.0% | +14.8% |
| bs=1000 sw=10 sl=64 | p95 | 1,151,092 us | 1,123,539 us | 985,437 us | +2.5% | +16.8% |
| bs=1000 sw=10 sl=64 | p99 | 1,151,092 us | 1,123,539 us | 1,012,622 us | +2.5% | +13.7% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,555.22,200,128000,360,0.220,27110.92,38763.71,38763.71
1,100,10,64,20,2617.62,2000,1280000,764,0.466,124149.65,194783.57,194783.57
2,1000,10,64,20,21712.94,20000,12800000,921,0.562,1086552.22,1151091.71,1151091.71|
/request-review @Xiao-zhen-Liu |
- Never rebuild the user's exception. The line number and fix are attached as a note, so the type, message and attributes (e.g. NameError.name, OSError.errno, a custom exception's fields) stay as raised, and the error is re-raised as-is. - The RPC server includes exception notes in ControlError's message, so the hint reaches the Static Error tab for every error type, including KeyError and custom exceptions. - A class with two pytexera bases is told to use one; classes with no UDF base (e.g. a LoopEndOperator subclass) get no UDF base/method list. - A class alias (`Y = X`) is no longer counted as a second Operator. - Drop the inaccurate "helper classes must not subclass an Operator"; say to keep the import "at the top". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Fixed: hints are now attached as a note to the original exception (no rebuilding), and |
Xiao-zhen-Liu
left a comment
There was a problem hiding this comment.
Attaching the hint as a note fixes the garbled messages. Six comments are still open: the table check, the no-class message (a class without a base, and the 1-out example), the opening sentence, the missing error type, line numbers with UI parameters, and @overrides on a helper.
- Keep "There should be one and only one Operator defined" only when there are zero or several classes, not when one class misses its method. - A class without a pytexera base is named and told which bases to use; a 1-out UDF gets a UDFSourceOperator example, and a base/UDF type mismatch says what to subclass. - The hint includes the error type, e.g. "(AssertionError at line 3 of the UDF code)", since the UI shows only the error text. - Line numbers skip the UI-parameter hook that the injector appends to the first class. - For "No super class method found", also say to remove @OVERRIDES from a helper method. - A test checks the base class/method table against the pytexera classes' abstract methods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A SyntaxError after the injected UI-parameter hook now reports the user's line (lineno and end_lineno shifted); the error is otherwise raised unchanged. - An error located inside the hook is not given a user line number; the hint says "(<Error> in the generated UI-parameter code)". - Document the limits: the hook is found by its reserved name with the shape the injector generates today, and runtime tracebacks are not remapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Done, please check and let me know if more changes are required. Thank you for the review |
- Result panel: the empty states now say "No operator is selected" or "This operator has no results yet"; clicking a failed box shows its Static Error tab (apache#8771). - UDF guide: new rows for "does not implement <method>" and "does not subclass a pytexera base class", @OVERRIDES on a helper, the "(NameError at line 2 of the UDF code)" format, and "Columns" for 1-out UDFs (apache#8768, apache#8769). - Retake errors-all-operators.png with the final message, on a Texera built from all five code PRs on current main. Refs apache#8736 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What changes were proposed in this PR?
When a Python UDF fails to load, the error now says what to fix. The existing "There should be one and only one Operator defined" sentence stays first, so anything matching on it still works.
The line number and fix are attached as a note to the original exception, so its type, message and attributes stay as raised. The RPC server now includes exception notes in the
ControlErrormessage, which is what the Static Error tab shows.No Operator class was found. Uncomment or write exactly one class...WrapperClass subclasses UDFOperatorV2 but does not implement process_tuple. If you meant process_table, subclass UDFTableOperator instead.Found 2: FirstOperator, SecondOperator. Keep one and delete or comment out the others.from pytexera import *name 'UDFOperatorV2' is not defined (line 2 of the UDF code). Add from pytexera import * at the top of the UDF.@overrideson the wrong methodX uses more than one base class (UDFOperatorV2, UDFTableOperator). Subclass exactly one.Before and after, in the Static Error tab. Before is the untouched template; after is a class with the wrong method:
Missing import, after:
Any related issues, documentation, discussions?
Part of #8736. Split out of #8737 as the reviewer asked.
How was this PR tested?
New tests in
test_executor_manager.pycover each case above, plus a syntax error, aKeyErrorand a custom exception at load time (both keep their own message and get the hint as a note), a class alias, and aLoopEndOperatorsubclass.test_async_rpc_server.pycovers notes inControlError, and loads a broken UDF inside a handler to check the message that gets sent back.The screenshots are from a local Texera built from this branch. The displayed text for those cases is the same after the follow-up commit.
Known limitations: UI-parameter source-line mapping relies on the generated hook's reserved name and structure. Runtime traceback locations (e.g., errors inside process_tuple) are not remapped.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)