You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Function is called multiple times when using multi ports #4074
When there are multiple ports, each port is called multiple times. For example, using this code in the multiport tests node:
class ProcessTableOperator(UDFTableOperator):
@overrides
def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]:
df0 = pd.DataFrame(table)
print(f'port is {port}')
df1 = pd.DataFrame(table)
print(f'port is {port}')
yield None
The f'port is {port} is printed twice
This signals to me that the function is called as many times as the number of ports, which should not be a normal behavior in my opinion.
Additionally, port 1 is called before port 0 because port 1 data is smaller and is generated upstream first, which Chris and Jiadong mentioned is what's intended. However, if a user is unaware, they may write a code to read in the data from port 0 first because that's the intended order, which will be an error as data is not loaded. Currently I write my own wrapper so inputs from each port can be recognized correctly, but do you have any example of how you often handle this case? I can make a new issue for this question if that's easier.
How to reproduce?
Dataset: samn16081314-downsampled-5k; workflow: PanKbase_example_5k_multi-port - I have shared these via email to Meng, Jiadong, Chris and Chen. Please let me know if I should share them with any other team members!
To run this workflow, you'll need to install some packages yourself. In Python:
@hhvu0102 Thank you for the issue. You are trying to use some advanced features of the UDF. What you described all look like intended behavior to me.
For the first "print" problem:
process_table(self, table: Table, port: int) is invoked once for each port, and the port parameter is a single value for each invocation. In your case, the process_table for port 1 is invoked first, and since you code is printing it twice in the process_table function, it will be printed twice. The behavior of your code is:
port 1's logic, which is processed first: df0 = port 1's table print port 1 df1 = port 1's table print port 1
port 1's logic, which is processed first: df0 = port 0's table print port 0 df1 = port 0's table print port 0
The correct way to write these kind of logic should be (similar to what we mentioned in the wiki):
@overrides
def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]:
if port == 0:
# port 0's processing logic
elif port == 1:
# port 1's processing logic
For the second problem about input order:
You can specify the order of the two inputs by specifying a dependency from port 1 to port 0 (see the below gif). This will tell the engine to process port 0 before port 1.
I'd like to reopen this thread with some follow-up questions:
If I want to use the data from both ports at the same time, what is your recommendation to write my function? Pseudo-code that I imagine is:
# some code to make sure that data from both ports are present and have finished loading -- not sure what your recommendation is
def function_to_called_once_only_when_all_args_available(a, b):
print(f"function_to_call Called Once with: {a}, {b}")
return {"a": a, "b": b}
The example you supplied on the wiki may break if self.model in the else statement does not exist yet. To me, this example implicitly assumed there's a model in self which is not the case in all instances.
I'm curious what the motivation for this design choice is. Would be great if you can explain, even if it involves complicated technical details.
Xinyuan answered this in #8683: with the dependency from port 1 to port 0, call your function in the port 1 branch. If you'd rather not rely on the order, keep each table and call the function when both have arrived. process_table is called once for each port, and only after all of that port's data has arrived, so you don't need any code to check that the data has finished loading:
frompytexeraimport*classProcessTableOperator(UDFTableOperator):
@overridesdefopen(self) ->None:
self.tables= {}
@overridesdefprocess_table(self, table: Table, port: int) ->Iterator[Optional[TableLike]]:
self.tables[port] =tableiflen(self.tables) ==2:
# both ports have all their data; this runs onceyieldyour_function(self.tables[0], self.tables[1])
Both ways assume "Worker count" is 1, which is the default. With more workers, each port's data is split among the workers, and each worker would call your_function with only its part of the data.
2. The wiki example
You're right: the example only works because port 0 finishes before port 1, and the wiki doesn't point that out. It is written for the "2-in Python UDF" operator, where port 1 (tuples) always waits until port 0 (model) has finished, so self.model is set before any tuple arrives on port 1. In a regular Python UDF with two ports there is no such wait unless you add the dependency, so the example fails when port 1's data comes first. It also fails if port 0 gets no tuples at all. The Python UDF guide is being rewritten in #8737, and the new version says that inputs arrive in no fixed order and shows the pattern above.
3. Why it works this way
Texera uses pipelined execution: operators run at the same time, and data is passed to the next operator while it is being produced, not after the previous operator finishes. This has three benefits: results start to appear sooner, operators work in parallel instead of waiting for each other, and intermediate results don't have to be stored in full between operators. To make this work, each operator, built-in or UDF, receives its input one tuple at a time, together with the port the tuple came in on. That is the Tuple API (process_tuple). The Table API is built on top of it: it collects each port's tuples and calls process_table when that port has no more data, so it is called once per port.
For the same reasons, the engine doesn't wait for all ports by default: an operator can start on one input while the operators feeding its other input are still running. When an operator does need one input to be complete before another, it declares a dependency between the two ports, and the engine finishes the first port before it sends any data to the second. The built-in Hash Join works this way: it builds a hash table from the whole left input, then looks up each right tuple in it. The 2-in Python UDF has this dependency built in, and in the regular Python UDF you can add it yourself, as in the GIF above. Without a dependency, the order depends on which upstream operator finishes first, which is why port 1 came first in your workflow.
Activity
chenlica commentedon Nov 22, 2025
@hhvu0102 Thank you.
@Xiao-zhen-Liu Can you take a look when you have a chance?
Xiao-zhen-Liu commentedon Nov 24, 2025
@hhvu0102 Thank you for the issue. You are trying to use some advanced features of the UDF. What you described all look like intended behavior to me.
For the first "print" problem:
process_table(self, table: Table, port: int)isinvoked once for each port, and theportparameter is a single value for each invocation. In your case, theprocess_tablefor port 1 is invoked first, and since you code is printing it twice in theprocess_tablefunction, it will be printed twice. The behavior of your code is:df0=port 1's tableprint port 1df1=port 1's tableprint port 1df0=port 0's tableprint port 0df1=port 0's tableprint port 0The correct way to write these kind of logic should be (similar to what we mentioned in the wiki):
For the second problem about input order:
You can specify the order of the two inputs by specifying a dependency from port 1 to port 0 (see the below gif). This will tell the engine to process port 0 before port 1.
Yicong-Huang commentedon Nov 26, 2025
Closing as it is not a bug. @hhvu0102 please feel free to reopen this if you have more questions.
hhvu0102 commentedon Sep 25, 2026
Hi all,
I'd like to reopen this thread with some follow-up questions:
self.modelin theelsestatement does not exist yet. To me, this example implicitly assumed there's amodelinselfwhich is not the case in all instances.Thanks team!
Xiao-zhen-Liu commentedon Sep 29, 2026
@hhvu0102 Thanks for the follow-up questions.
1. Using the data from both ports at once
Xinyuan answered this in #8683: with the dependency from port 1 to port 0, call your function in the port 1 branch. If you'd rather not rely on the order, keep each table and call the function when both have arrived.
process_tableis called once for each port, and only after all of that port's data has arrived, so you don't need any code to check that the data has finished loading:Both ways assume "Worker count" is 1, which is the default. With more workers, each port's data is split among the workers, and each worker would call
your_functionwith only its part of the data.2. The wiki example
You're right: the example only works because port 0 finishes before port 1, and the wiki doesn't point that out. It is written for the "2-in Python UDF" operator, where port 1 (
tuples) always waits until port 0 (model) has finished, soself.modelis set before any tuple arrives on port 1. In a regular Python UDF with two ports there is no such wait unless you add the dependency, so the example fails when port 1's data comes first. It also fails if port 0 gets no tuples at all. The Python UDF guide is being rewritten in #8737, and the new version says that inputs arrive in no fixed order and shows the pattern above.3. Why it works this way
Texera uses pipelined execution: operators run at the same time, and data is passed to the next operator while it is being produced, not after the previous operator finishes. This has three benefits: results start to appear sooner, operators work in parallel instead of waiting for each other, and intermediate results don't have to be stored in full between operators. To make this work, each operator, built-in or UDF, receives its input one tuple at a time, together with the port the tuple came in on. That is the Tuple API (
process_tuple). The Table API is built on top of it: it collects each port's tuples and callsprocess_tablewhen that port has no more data, so it is called once per port.For the same reasons, the engine doesn't wait for all ports by default: an operator can start on one input while the operators feeding its other input are still running. When an operator does need one input to be complete before another, it declares a dependency between the two ports, and the engine finishes the first port before it sends any data to the second. The built-in Hash Join works this way: it builds a hash table from the whole left input, then looks up each right tuple in it. The 2-in Python UDF has this dependency built in, and in the regular Python UDF you can add it yourself, as in the GIF above. Without a dependency, the order depends on which upstream operator finishes first, which is why port 1 came first in your workflow.