Repository navigation
Replies: 3 comments
|
Update: how I plan to land multi-port output results I've rebuilt this on top of #9054 (thanks @kz930) and split it into small PRs, so each one is easy to review on its own. Showing results for every output port
Python UDF output ports Order: 1 → 2 → 3 / 4, with 5 and 6 running alongside. 1 and 5 don't depend on anything. Questions for maintainers:
Not included: R UDF multi-port output. It needs per-port schemas in RUDFOpDesc and changes in the separate texera-rudf plugin, sO I'd suggest it as a follow-up. |
|
Thanks for splitting the result-handling changes into small PRs, @sshiv012. Following up on our meeting on Friday, let us take this in two phases:
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Why
A common step in an analysis produces more than one thing: a cleaned table and a chart of it, a train split and a test split, matched rows and rejects. Today one operator shows one result. To get a table and a plot you chain a second operator off the same link and look at two result tabs. That works, but it adds another operator for each additional output and a Python UDF that already has both in hand has no way to hand them out separately.
The port model already supports this. Split and If emit on two ports today. What is missing is the plumbing between a Python UDF and the result panel. This proposal is to finish that plumbing, in small PRs, and to agree on the API decisions up front.
Current architecture
Ports are first-class from the descriptor down to storage:
The Scala worker routes by port. An executor returns
(tuple, Some(port))fromprocessTupleMultiPort, andOutputManager.passTupleToDownstreamsends it only on links whosefromPortIdmatches. Split and If use this.Two paths stop honoring the port.
Python worker. The UDF yields tuples with no port. pyamber finalizes each tuple against the first port's schema, then hands it to every partitioner, so it goes out on every link of every port and into every port's storage.
Result delivery. Result delivery, pagination, export, and sync execution then select port 0. The websocket result event is keyed by operator, and the result panel builds one Result tab per operator. The output mode (table vs HTML) is taken from the first external port encountered, so one operator is either all table or all HTML.
Summary of what exists and what does not:
Related: #8274 and #8276 (pyamber
get_portignores the id; fixes the lookup only, nothing calls it with a port yet), #2078 (UDF port validation).What would change
Four incremental PRs: the first delivers multi-port results for Scala operators; the remaining PRs build toward multi-port Python UDF output.
1. Results per port
Key the result store by operator plus port. Send the websocket update per port, add a
portId(default 0) to the pagination, export and sync requests, and read the output mode from the port being served. The result panel shows one tab per external output port when an operator has more than one, labeled by the port's display name.Single port workflows retain their behavior and presentation, so existing workflows should see no difference. This alone makes both ports of Split and If viewable.
2. pyamber honors a port
tuple_to_batch(tuple, port)filters partitioners by the link's source port.data_processorfinalizes against that port's schema. Statistics and storage writes take the port. Lands on top of #8276. Arrow batch construction uses that port’s schema, including buffered batches flushed at completion.3. pytexera: naming the port
Add an explicit PortOutput wrapper that carries a value and its destination port. Bare yields continue to target port 0. The SDK will introduce a return type covering both bare and wrapped outputs.
A bare yield targets port 0. all_output_to_tuple learns to unwrap PortOutput, normalize its value into tuples, and preserve the destination port for each tuple.
4. Per-port output schema and mode on the Python UDF
The descriptor propagates a schema for port 0 only, and the saved port description carries no schema or mode. Add a per-port
kind:table(default)visualizationhtml-content: STRINGWorkflows saved before this default every port to
table. The port editor in the property panel gets the kind field for output ports; today it edits input ports only.Nothing changes in the scheduler, storage layout, Arrow data path (maybe Arrow serialization code?), or the
PythonDataHeaderproto: partitioning happens on the Python side, so Scala never needs the port.Where I want feedback
kindvs. per-port column list.kindcovers the table+chart case with one field. A per-port extra-columns list is more general but a bigger form change. Start withkindand add columns later?Map[opId, Map[portId, update]]vs. a composite key"opId:portId". The composite key is the smaller frontend diff; the nested map keeps one shape for every operator and avoids key parsing in every consumerAcceptance Criteria
Validate a UDF with at least three outputs, including different table/HTML schemas, an empty output port, and buffered output flushed at completion. Verify independent results, pagination, export, invalid-port errors, and compatibility with existing single-port workflows.
Note - This supports any number of output ports; table plus visualization is the initial use case. Independently configurable table schemas are future work.
All reactions