Repository navigation
Conversation
…hots - Python UDF guide: working first example, the 5 rules (import, one class, matching method, yield, output columns), API table, setup, multiple inputs, and a troubleshooting table built from real errors - How to use Texera: dashboard menu, computing units, operator state colors, where results and errors appear, pause/stop, and a "workflow looks stuck" checklist - Dataset tutorial: current UI (Versions & Files, file picker), note that file paths pin a dataset version, bundled sample CSV (old link 404) - Link the guide from the Python UDF reference pages; update UI overview menu; replace dead links in Examples and Concepts Pages with images become Hugo leaf bundles (same URLs) so images sit next to their page. Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a UDF fails to load, the message now says what to change: - no Operator class: says to uncomment or write exactly one class - abstract class: names the class, its base, the missing method, and the base that matches the method it did write - several classes: names them and says to keep one - NameError for a pytexera symbol: says to add `from pytexera import *` - @OVERRIDES mismatch: lists the base class to method pairing - any other load error: adds the line of the user's code that failed The existing "There should be one and only one Operator defined" wording stays as the first sentence. Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Computing Units: creating one (fields per type), status dots, reuse vs create, what terminating deletes, common error messages - Sharing and Permissions: share dialog, read/write/owner, that datasets must be shared separately, public items and Downloadable - How Texera Runs Your Workflow (Concepts): pipelining, blocking operators, regions and the Layers > Regions view, workers, and execution mode - Link them from the Concepts index and the "how to use" guide Result reuse is not documented: its controls are disabled on main and each run clears the previous results. Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nting one class The Python UDF, 2-in Python UDF and 1-out Python UDF templates were fully commented out, including `from pytexera import *`, and did not say to keep a single class. Uncommenting a class then failed with NameError, and keeping several failed the one-Operator check. - Keep `from pytexera import *` active in all three templates - Add a header: keep exactly one class, the base class to method pairing, `yield` not `return`, open() and super().__init__() - 2-in template: say port 0 (model) is read before port 1 (tuples) - Keep the UiParameter guidance lines - Sync the reference docs copies and the Python UDF guide wording Saved workflows keep their own code and are unaffected. The untouched template still has no class, which now reports "No Operator class was found" with what to do. Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…smatches - `return` instead of `yield` in process_tuple, process_batch, process_table or produce now raises "<method> must `yield` results, not `return` them" instead of MatchError / 'NoneType' not iterable. Returning a list still works. - UDFTableOperator / UDFBatchOperator create their buffers lazily, so an __init__ that skips super().__init__() no longer crashes with '_TableOperator__table_data' (same approach as the lazy template decoder in Operator). - Yielding a dict of DataFrames says to yield the DataFrame itself. - Output column mismatches point to 'Retain input columns' and 'Extra output column(s)'. - Python UDF guide lists the new messages next to the older ones. Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… errors - Kill button: tooltip "Stop this run. A stopped run can't be resumed." - Result panel empty state: say to click an operator for its results and Console, or an empty spot on the canvas for all operators' errors (the Static Error tab only appears when nothing is selected) Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e CU quota is reached The quota error only said "You may only have N computing-unit(s) running at the same time". It now adds: "Reuse one of your running computing units, or terminate one you no longer need." Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A different unit can change memory, CPU, GPUs, the software image and installed Python packages, so it can help with resource or environment problems. Say that it usually won't fix code or configuration errors instead of saying a workflow fails the same way everywhere. Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Say how to open the Operators panel and the result panel, and that search-added operators stack on top of each other - Filter: click + next to Predicates; field is Attribute - Python UDF: under User-defined Functions > Python; open Result tab - Static Error only appears when something failed - Layers button can be hidden on narrow windows - Admin menu contents; local units show NaN limits Refs TEX-86 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8737 +/- ##
============================================
- Coverage 92.73% 92.59% -0.14%
- Complexity 4947 4962 +15
============================================
Files 1241 1242 +1
Lines 52608 52956 +348
Branches 6507 6559 +52
============================================
+ Hits 48785 49035 +250
- Misses 2206 2292 +86
- Partials 1617 1629 +12
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 479 | 0.292 | 21,194/25,049/25,049 us | 🔴 +17.1% / 🔴 +68.5% |
| 🔴 | bs=100 sw=10 sl=64 | 1,163 | 0.71 | 82,955/120,188/120,188 us | 🔴 -5.3% / 🟢 -17.9% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,405 | 0.857 | 711,726/803,824/803,824 us | ⚪ within ±5% / 🟢 +31.9% |
Baseline details
Latest main acc6dd1 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 479 tuples/sec | 530 tuples/sec | 807.63 tuples/sec | -9.6% | -40.7% |
| bs=10 sw=10 sl=64 | MB/s | 0.292 MB/s | 0.323 MB/s | 0.493 MB/s | -9.6% | -40.8% |
| bs=10 sw=10 sl=64 | p50 | 21,194 us | 18,104 us | 12,578 us | +17.1% | +68.5% |
| bs=10 sw=10 sl=64 | p95 | 25,049 us | 27,663 us | 16,088 us | -9.4% | +55.7% |
| bs=10 sw=10 sl=64 | p99 | 25,049 us | 27,663 us | 19,009 us | -9.4% | +31.8% |
| bs=100 sw=10 sl=64 | throughput | 1,163 tuples/sec | 1,228 tuples/sec | 1,039 tuples/sec | -5.3% | +11.9% |
| bs=100 sw=10 sl=64 | MB/s | 0.71 MB/s | 0.749 MB/s | 0.634 MB/s | -5.2% | +12.0% |
| bs=100 sw=10 sl=64 | p50 | 82,955 us | 80,121 us | 101,098 us | +3.5% | -17.9% |
| bs=100 sw=10 sl=64 | p95 | 120,188 us | 121,895 us | 107,890 us | -1.4% | +11.4% |
| bs=100 sw=10 sl=64 | p99 | 120,188 us | 121,895 us | 119,853 us | -1.4% | +0.3% |
| bs=1000 sw=10 sl=64 | throughput | 1,405 tuples/sec | 1,411 tuples/sec | 1,065 tuples/sec | -0.4% | +31.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.857 MB/s | 0.861 MB/s | 0.65 MB/s | -0.5% | +31.8% |
| bs=1000 sw=10 sl=64 | p50 | 711,726 us | 710,438 us | 992,940 us | +0.2% | -28.3% |
| bs=1000 sw=10 sl=64 | p95 | 803,824 us | 782,025 us | 1,035,756 us | +2.8% | -22.4% |
| bs=1000 sw=10 sl=64 | p99 | 803,824 us | 782,025 us | 1,062,329 us | +2.8% | -24.3% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,417.77,200,128000,479,0.292,21193.63,25049.31,25049.31
1,100,10,64,20,1719.75,2000,1280000,1163,0.710,82955.29,120188.27,120188.27
2,1000,10,64,20,14236.75,20000,12800000,1405,0.857,711726.03,803824.40,803824.40… hint - An error with no UDF line and no hint is returned unchanged - An error that can't be rebuilt from one message (UnicodeDecodeError) is kept as is Refs apache#8736 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Xiao-zhen-Liu
left a comment
There was a problem hiding this comment.
Before going through the code, two things about the PR as a whole.
1. Could you split it? It bundles six changes that could each be reviewed and reverted on their own, and your commits already follow them:
- pyamber: load error messages (
executor_manager.py) - pyamber: runtime error messages and the lazy table and batch buffers (
operator.py,data_processor.py) - workflow-operator: the three Python UDF templates
- frontend: the Kill button tooltip and the empty Result panel text
- computing-unit-managing-service: the quota message
- docs: 31 of the 49 files
The code changes don't depend on each other and sit in four modules with different reviewers. PRs here are squash-merged, so reverting one part later, for example the change in operator.py, would also revert the docs. #8736 already lists these as separate changes. The docs PR should go last, since the UDF guide quotes the new messages and tells readers to uncomment a class in the new template.
2. Please add before/after screenshots of what users will see: the new template in the code editor, a few of the new messages in the Static Error tab and the Console, the Kill button tooltip, and the empty Result panel. The 15 images in the PR are docs screenshots taken without these changes, so they don't show any of this. After the split, each PR can carry its own.
| - `print(...)` in your code shows up in the **Console** tab. Click the UDF to see it. | ||
| - If a row fails, the UDF turns **magenta** (paused) and the error appears in its **Console**. Fix the code, then stop the run (red ⊗ next to *Pause*) and run again. | ||
|
|
||
|  |
There was a problem hiding this comment.
This screenshot shows pampy.MatchError: '_' not provided, which the table below lists as the older error for return instead of yield. With this PR a reader who makes that mistake sees the new message instead, so could you retake it with the PR's changes?
There was a problem hiding this comment.
Thank you for reviewing this, some images were outdated and wrongly pushed. I will check and update it in the revised PR
|
|
||
| - **Click an empty spot on the canvas.** If any operator failed, the result panel shows a **Static Error** tab with errors from every operator, including code that failed to start. A run that stops right away with every box at 0 rows usually means this. | ||
|
|
||
|  |
There was a problem hiding this comment.
This shows the old message, "There should be one and only one Operator defined", with nothing after it. With this PR the message also says what to fix, so please retake it with the PR's changes.
There was a problem hiding this comment.
Thank you for reviewing this, some images were outdated and wrongly pushed. I will check and update it in the revised PR
|
Thanks @Xiao-zhen-Liu, both points make sense. I split this into six PRs, one per area. Each targets
The code PRs have before and after screenshots taken on a local Texera built from those changes. The quota one has none, since that dialog only shows on Kubernetes. In the docs PR I retook the two images you flagged (the Console error and the Static Error) with the new messages. Closing this one in favor of those. |
…smatches (apache#8769) ### What changes were proposed in this PR? Clear messages for the common Python UDF mistakes that show up while the workflow runs: - **`return` instead of `yield`** in `process_tuple`, `process_batch`, `process_table` or `produce` now says `process_table must yield results, not return them. Replace return x with yield x.` Before, users saw `pampy.MatchError: '_' not provided` or `'NoneType' object is not iterable`. Returning a list still works. - **`__init__` without `super().__init__()`** on a table or batch UDF no longer crashes with `'_TableOperator__table_data'`. The buffers are created on first use, the same way `Operator` already creates its template decoder. - **A DataFrame in a non-binary column** now says which column, and to yield the DataFrame itself or make the column binary. A DataFrame in a BINARY column is pickled as before. - **Output columns that don't match** keep the original message and point to "Retain input columns" and "Extra output column(s)" (or "Columns" for a 1-out Python UDF). Before and after, in the Console: | Before | After | |---|---| |<img width="800" height="288" alt="runtime-before" src="https://github.com/user-attachments/assets/7f854ffe-f5eb-4013-8fe4-9eb3a9e740ff" /> | <img width="800" height="288" alt="runtime-after" src="https://github.com/user-attachments/assets/fbaec933-683f-4dd2-b603-5d988b4f338d" /> | ### Any related issues, documentation, discussions? Part of apache#8736. Split out of apache#8737. ### How was this PR tested? New tests in `test_operator.py` (each `return` case, a returned list still working, table and batch UDFs with and without `super().__init__()`) and `test_data_processor.py` (`process_tuple` and `on_finish` returning a value, a DataFrame in BINARY, LARGE_BINARY and STRING columns, a missing or misspelled column next to a DataFrame, a wrong column type). ``` cd amber pytest -m "not integration" src/test/python/core/models/test_operator.py src/test/python/core/runnables/test_data_processor.py pytest -m "not integration" ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python ``` The screenshots are from a local Texera built from this branch. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) --------- Co-authored-by: suryaacharan <suryaacharan@suryaacharans-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…che#8768) ### What changes were proposed in this PR? When a Python UDF fails to load, the error now says what to fix. The existing "There should be one and only one Operator defined" sentence stays first, so anything matching on it still works. The line number and fix are attached as a note to the original exception, so its type, message and attributes stay as raised. The RPC server now includes exception notes in the `ControlError` message, which is what the Static Error tab shows. | Mistake | Message now adds | |---|---| | No class (e.g. the whole template left commented out) | `No Operator class was found. Uncomment or write exactly one class...` | | Method doesn't match the base class | `WrapperClass subclasses UDFOperatorV2 but does not implement process_tuple. If you meant process_table, subclass UDFTableOperator instead.` | | Two classes | `Found 2: FirstOperator, SecondOperator. Keep one and delete or comment out the others.` | | Missing `from pytexera import *` | `name 'UDFOperatorV2' is not defined (line 2 of the UDF code). Add from pytexera import * at the top of the UDF.` | | `@overrides` on the wrong method | the base class to method pairing | | A class with two pytexera bases | `X uses more than one base class (UDFOperatorV2, UDFTableOperator). Subclass exactly one.` | | Any other load error | the line number in the user's code | Before and after, in the Static Error tab. Before is the untouched template; after is a class with the wrong method: | Before | After | |---|---| | <img width="800" height="288" alt="load-before" src="https://github.com/user-attachments/assets/bd2ac22c-1953-435c-b3d5-0cf2e6f0ddda" />| <img width="800" height="348" alt="load-after" src="https://github.com/user-attachments/assets/53985c6f-9ad3-4d93-aed1-0974d14e85ad" /> | Missing import, after: <img width="800" height="303" alt="load-after-import" src="https://github.com/user-attachments/assets/c1ed1322-f7d0-4937-8aa2-848917a115ba" /> ### Any related issues, documentation, discussions? Part of apache#8736. Split out of apache#8737 as the reviewer asked. ### How was this PR tested? New tests in `test_executor_manager.py` cover each case above, plus a syntax error, a `KeyError` and a custom exception at load time (both keep their own message and get the hint as a note), a class alias, and a `LoopEndOperator` subclass. `test_async_rpc_server.py` covers notes in `ControlError`, and loads a broken UDF inside a handler to check the message that gets sent back. ``` cd amber pytest -m "not integration" src/test/python/core/architecture/managers/test_executor_manager.py src/test/python/core/architecture/rpc/test_async_rpc_server.py pytest -m "not integration" ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python ``` The screenshots are from a local Texera built from this branch. The displayed text for those cases is the same after the follow-up commit. Known limitations: UI-parameter source-line mapping relies on the generated hook's reserved name and structure. Runtime traceback locations (e.g., errors inside process_tuple) are not remapped. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) --------- Co-authored-by: suryaacharan <suryaacharan@suryaacharans-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e CU quota is reached (apache#8772) ### What changes were proposed in this PR? When a user hits the computing unit limit, the error only said "You may only have N computing-unit(s) running at the same time". It now adds what to do: "Reuse a running computing unit, terminate a cloud unit, or disconnect a local unit you no longer need." Local units count toward the limit too, and the UI removes them with "Disconnect", which calls the same terminate endpoint and frees the slot. ### Any related issues, documentation, discussions? Part of apache#8736. Split out of apache#8737. ### How was this PR tested? Updated `ComputingUnitManagingServiceExceptionSpec`. ``` sbt "ComputingUnitManagingService/testOnly org.apache.texera.service.util.ComputingUnitManagingServiceExceptionSpec" "ComputingUnitManagingService/scalafmtCheck" "ComputingUnitManagingService/Test/scalafmtCheck" ``` There's no screenshot because this error only appears on a Kubernetes deployment, and I tested on a local install. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) --------- Co-authored-by: suryaacharan <suryaacharan@suryaacharans-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…nting one class (apache#8770) ### What changes were proposed in this PR? The Python UDF, 2-in Python UDF and 1-out Python UDF templates were fully commented out, including `from pytexera import *`. They also never said to keep only one class. So uncommenting a class failed with `NameError`, and keeping several failed the one-Operator check. - `from pytexera import *` is now active. - A short header explains the rules: ```python # Keep exactly ONE class below: uncomment it and delete the others. # Its method must match its base class: # UDFOperatorV2 -> process_tuple, UDFBatchOperator -> process_batch, # UDFTableOperator -> process_table # Always `yield` results (never `return` a value). Put setup code in open(); # if you override __init__, call super().__init__() first. ``` - The 2-in template also says that port 0 (model) is read before port 1 (tuples). - The UiParameter lines are kept. - The template copies in `docs/reference/operators/user-defined-functions/python/` are synced. Saved workflows keep their own code and are not affected. Before, every line started with `#`: ```python # Choose from the following templates: # ... # from pytexera import * # # class ProcessTupleOperator(UDFOperatorV2): # ... ``` After, in the code editor: <img width="630" height="388" alt="template-after" src="https://github.com/user-attachments/assets/d376eb72-a40e-4493-9018-21c2b089ca9e" /> ### Any related issues, documentation, discussions? Part of apache#8736. Split out of apache#8737. ### How was this PR tested? - The specs for all three descriptors read the `defaultValue`. They check that the import is the only active line and the header is there, and that uncommenting any one class leaves exactly one active class. - New Python tests load the template shape and check that uncommenting each class loads without adding an import. - The template copy in the notebook-migration prompt is updated to match. ``` sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.udf.python.*" "WorkflowOperator/scalafmtCheck" "WorkflowOperator/Test/scalafmtCheck" cd amber && pytest -m "not integration" src/test/python/core/architecture/managers/test_executor_manager.py cd frontend && yarn eslint src/app/workspace/service/notebook-migration/migration-prompts.ts ``` ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) --------- Co-authored-by: suryaacharan <suryaacharan@suryaacharans-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… errors (apache#8771) ### What changes were proposed in this PR? Two small hints in the workspace: - **Kill button tooltip:** "Stop this run. A stopped run can't be resumed." It was an icon-only button with no explanation. - **Empty Result panel:** the heading now says why it is empty: "No operator is selected." or "This operator has no results yet." With nothing selected it also says "Click an operator to see its results. If a run fails, its errors show here." It always adds "Errors an operator hits while running show in that operator's Console." A failed run's errors only appear in the Static Error tab when nothing is selected, and errors during a run pause it and go to the operator's Console. Neither hint shows while previewing a workflow version. Before and after, with nothing selected (top) and an operator selected (middle). The bottom row (a failed UDF showing its Static Error tab when clicked) is no longer part of this PR: on main that error still arrives as "unknown operator", because `ControlError` drops the worker that apache#8668 attaches. That will be fixed in the backend in a separate PR. <img width="1648" height="854" alt="texera-8771-before-after-v2" src="https://github.com/user-attachments/assets/b1034c28-ce7d-4baf-b2f2-cfd935bc503e" /> The tooltip is a native `title`, which browsers don't draw in screenshots. I checked it in the running app: hovering the red button shows "Stop this run. A stopped run can't be resumed." ### Any related issues, documentation, discussions? Part of apache#8736. Split out of apache#8737. ### How was this PR tested? New specs in `result-panel.component.spec.ts` (nothing selected, an operator selected, version preview) and `menu.component.spec.ts` (the tooltip). ``` cd frontend yarn test --include src/app/workspace/component/result-panel/result-panel.component.spec.ts --include src/app/workspace/component/menu/menu.component.spec.ts yarn test yarn format:ci ``` ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) --------- Co-authored-by: suryaacharan <suryaacharan@suryaacharans-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What changes were proposed in this PR?
This makes Python UDF mistakes explain themselves, and rewrites the user docs for people who aren't programmers. The motivating case, from the UMich team, is in #8736.
UDF templates (Python UDF, 2-in, 1-out)
from pytexera import *is no longer commented out, so uncommenting one class gives a working UDF.yieldnotreturn, andopen()/super().__init__().Load errors (
executor_manager.py)The old "There should be one and only one Operator defined" sentence stays first, then the message says what to fix:
No Operator class was found. Uncomment or write exactly one class...WrapperClass subclasses UDFOperatorV2 but does not implement process_tuple. If you meant process_table, subclass UDFTableOperator instead.Found 2: FirstOperator, SecondOperator. Keep one...name 'UDFOperatorV2' is not defined (line 2 of the UDF code). Add from pytexera import * at the top of the UDF.Runtime errors (
operator.py,data_processor.py)returninstead ofyieldnow saysprocess_table must yield results, not return theminstead ofMatchError/'NoneType' object is not iterable. Returning a list still works.super().__init__()no longer crashes with'_TableOperator__table_data'.Operatoralready does this for its template decoder.Frontend
Computing unit quota error now adds: "Reuse one of your running computing units, or terminate one you no longer need."
Docs
environmenttab that no longer exists, and its sample CSV link returned 404.page/index.mdwith the images next to it), so the URLs stay the same.The commits are split by area if you'd rather review them one at a time.
Any related issues, documentation, discussions?
Closes #8736
Related: #8667 / #8668 fix the silent hang on UDF setup failures. Once that lands, users will see these messages.
How was this PR tested?
Every behavior change has a test written first.
test_executor_manager.py,test_operator.pyandtest_data_processor.py. Full suite:cd amber && pytest -m "not integration"gives 1349 passed. ruff check and format are clean.defaultValueand check that uncommenting any one class leaves exactly one active class. The quota message spec is updated. Checked withWorkflowOperator/testOnly ...udf.python.*,ComputingUnitManagingService/testOnly ...ComputingUnitManagingServiceExceptionSpec, and scalafmtCheck.result-panel.component.spec.tsandmenu.component.spec.ts(197 passed), plus prettier and eslint.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with Claude Code