Repository navigation
Conversation
apache#8668 tags a failed worker RPC with the worker (WorkflowRuntimeException with relatedWorkerId), but ControlError had no field for it. When the coordinator returned the error to the web service, reconstructThrowable rebuilt a plain Throwable, so WorkflowService recorded "unknown operator" and the UI could not show the error under the failed operator (e.g. a Python UDF that fails while starting). - Add relatedWorkerId to ControlError. - mkControlError fills it from a WorkflowRuntimeException. - reconstructThrowable rebuilds a WorkflowRuntimeException when it is set, so the worker reaches WorkflowService's error handler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
Backport auto-label reportThis
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 519 | 0.317 | 17,707/29,442/29,442 us | 🔴 +18.4% / 🔴 +129.8% |
| 🔴 | bs=100 sw=10 sl=64 | 1,083 | 0.661 | 90,093/120,188/120,188 us | 🔴 +14.2% / 🔴 +46.9% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,281 | 0.782 | 779,283/832,083/832,083 us | ⚪ within ±5% / 🔴 -15.6% |
Baseline details
Latest main f020980 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 519 tuples/sec | 577 tuples/sec | 1,084 tuples/sec | -10.1% | -52.1% |
| bs=10 sw=10 sl=64 | MB/s | 0.317 MB/s | 0.352 MB/s | 0.662 MB/s | -9.9% | -52.1% |
| bs=10 sw=10 sl=64 | p50 | 17,707 us | 16,891 us | 9,548 us | +4.8% | +85.5% |
| bs=10 sw=10 sl=64 | p95 | 29,442 us | 24,863 us | 12,811 us | +18.4% | +129.8% |
| bs=10 sw=10 sl=64 | p99 | 29,442 us | 24,863 us | 15,432 us | +18.4% | +90.8% |
| bs=100 sw=10 sl=64 | throughput | 1,083 tuples/sec | 1,136 tuples/sec | 1,467 tuples/sec | -4.7% | -26.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.661 MB/s | 0.693 MB/s | 0.895 MB/s | -4.6% | -26.2% |
| bs=100 sw=10 sl=64 | p50 | 90,093 us | 87,160 us | 74,441 us | +3.4% | +21.0% |
| bs=100 sw=10 sl=64 | p95 | 120,188 us | 105,197 us | 81,800 us | +14.2% | +46.9% |
| bs=100 sw=10 sl=64 | p99 | 120,188 us | 105,197 us | 103,712 us | +14.2% | +15.9% |
| bs=1000 sw=10 sl=64 | throughput | 1,281 tuples/sec | 1,272 tuples/sec | 1,517 tuples/sec | +0.7% | -15.6% |
| bs=1000 sw=10 sl=64 | MB/s | 0.782 MB/s | 0.776 MB/s | 0.926 MB/s | +0.8% | -15.5% |
| bs=1000 sw=10 sl=64 | p50 | 779,283 us | 778,791 us | 725,911 us | +0.1% | +7.4% |
| bs=1000 sw=10 sl=64 | p95 | 832,083 us | 838,839 us | 764,152 us | -0.8% | +8.9% |
| bs=1000 sw=10 sl=64 | p99 | 832,083 us | 838,839 us | 808,901 us | -0.8% | +2.9% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,384.99,200,128000,519,0.317,17707.17,29441.52,29441.52
1,100,10,64,20,1847.32,2000,1280000,1083,0.661,90093.08,120187.63,120187.63
2,1000,10,64,20,15606.86,20000,12800000,1281,0.782,779283.38,832083.23,832083.23
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9015 +/- ##
============================================
- Coverage 91.56% 91.54% -0.02%
+ Complexity 5157 5135 -22
============================================
Files 1300 1296 -4
Lines 58092 57781 -311
Branches 7615 7572 -43
============================================
- Hits 53194 52898 -296
+ Misses 2959 2940 -19
- Partials 1939 1943 +4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
When a Python UDF fails while starting, its error loses track of which operator failed on the way back to the UI, so it gets recorded as "unknown operator" and clicking the failed UDF shows an empty Console. This PR carries the failing worker through ControlError so the error lands on the right operator, and since v1.2 already has #8668 but not this fix, v1.2 and v1.3 users would hit the same confusing empty Console. @mengw15 @yangzhang75 please check if this needs to be backported (it is a small fix). |
|
Removing release/v1.2: diagnostics only, and it changes the control protocol. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/request-review @Xiao-zhen-Liu |
What changes were proposed in this PR?
#8668 tags a failed worker RPC with the worker it targeted (
WorkflowRuntimeExceptionwithrelatedWorkerId), so the surfaced error can point at that worker. But the worker is dropped one hop later: the coordinator returns the error to the web service as aControlError, which has no field for it, andErrorUtils.reconstructThrowablerebuilds a plainThrowable.WorkflowService's error handler then finds no worker and records the error as "unknown operator".You can see this with a Python UDF that fails while starting (e.g. the default template, which has no class). The coordinator log shows
WorkflowRuntimeException … Worker:WF23-PythonUDFV2-operator-udfdemo-main-0, but the UI getsoperatorId: "unknown operator". So clicking the failed UDF shows only an empty Console and no error.This PR carries the worker the rest of the way:
ControlErrorgets an optionalrelatedWorkerId(core.ActorVirtualIdentity).mkControlErrorfills it from aWorkflowRuntimeException.reconstructThrowablerebuilds aWorkflowRuntimeExceptionwith that worker when the field is set, soWorkflowServicemaps the error to the failing operator.Errors without a worker are unchanged.
Any related issues, documentation, discussions?
Follow-up to #8668 (issue #8667). Part of #8736. With this, #8771 no longer needs a frontend workaround to show a failed UDF's startup error when the UDF is clicked.
How was this PR tested?
ErrorUtilsSpec:mkControlErrorcarries the worker of aWorkflowRuntimeException, and leaves it empty otherwise;reconstructThrowablerebuilds aWorkflowRuntimeExceptionwith the worker, message and cause;getOperatorFromActorIdOpt;Throwable.ControlErrordirectly pass the new field (None).DataProcessingSpec: a source feeding a Java UDF that doesn't compile. The UDF is in the first region, so its error comes back as the reply tostartWorkflow, throughControlError. The test expects aWorkflowRuntimeExceptionnaming the UDF worker. It fails on main and passes with this PR.End to end, on a local Texera built from current main plus this PR and #8771 (which no longer has a frontend filter), I ran a Python UDF that fails while starting:
operatorIdthe UI receives"unknown operator"(workerIdempty)"PythonUDFV2-operator-udfdemo"(workerIdWorker:WF23-PythonUDFV2-operator-udfdemo-main-0)The message also loses its
java.lang.Throwable:prefix, since the rebuiltWorkflowRuntimeExceptionprints just its message.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)