Skip to content

feat(amber): columnar output without round-trip, per-edge format - #9551

Draft
Ma77Ball wants to merge 7 commits into
apache:mainfrom
Ma77Ball:columnar/02-output-planner
Draft

Ma77Ball wants to merge 7 commits into
apache:mainfrom
Ma77Ball:columnar/02-output-planner

Conversation

@Ma77Ball

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

  • Carry an operator's Arrow output to the next edge without the tuplesOfRoot round-trip on one-to-one edges, so a columnar region stays columnar end to end.
  • Decide the wire format per edge from the downstream operator's isColumnarCapable, replacing the single global on/off switch as the decision point.
  • Thread the per-edge columnar flag through ResourceAllocator, LinkConfig, the AddPartitioning control command, and OutputManager / Partitioner.

Any related issues, documentation, discussions?

Closes: #9548
Part of: #9542
Stacked on Part 1 (#9547); isolated diff: Ma77Ball/texera@columnar/01-foundation...columnar/02-output-planner

How was this PR tested?

  • Run sbt "WorkflowExecutionService/testOnly *OutputManagerSpec *NetworkOutputBufferColumnarSeamSpec", expect the no-round-trip output and per-edge decision cases to pass.
  • Run sbt "WorkflowCore/testOnly *PhysicalOpSpec", expect the isColumnarCapable cases to pass.
  • Manual: with enableColumnarWire=true, run a scan into a filter on separate workers at http://texera.localhost and confirm results are identical to the flag-off run.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy.

The foundation for a columnar Arrow wire: an operator-side contract for consuming and producing Arrow batches directly, the Arrow helpers it needs, and the first two operators that implement it (the specialized filter and the Arrow file scan). Nothing is wired into the messaging layer yet, so no operator receives columnar batches at runtime; these are the capabilities the wire will drive.

Contract (ColumnarOperatorExecutor):
- ColumnarOperatorExecutor is a trait with processColumnarBatch(arrowIpcBytes, port) that returns a ColumnarResult. An operator mixes it in to declare it can consume an Arrow batch without decoding each row to a Tuple.
- ColumnarResult is the sealed outcome of consuming one batch: Emit(root) (emit this Arrow result, caller serializes and closes the root), EmitRows(iterator) (emit row-shaped output, e.g. a join), Consumed (consumed with nothing to emit now, a blocking operator accumulating state), and Unsupported (not handled, caller falls back to the row path).
- SourceOperatorExecutor gains produceColumnarBatch(), defaulting to None, so a source whose data is already columnar can hand out Arrow roots instead of going through produceTuple; the caller owns and closes each root.

ArrowUtils helpers:
- serializeRoot writes a VectorSchemaRoot to Arrow IPC stream bytes without closing it; deserializeRootFold reads one IPC batch, applies a function to the live root, and closes the reader (the function must copy out what it keeps).
- selectRows copies the rows a boolean keep-mask selects into a new compacted root; selectionOf returns just the surviving row indices (a selection vector) without copying any cell, and gather materializes the rows a selection vector names into a new root, so a consumer that only needs a count or an aggregate reads indices and never materializes.
- projectionIndices resolves column names to their Arrow field indices by exact name; getCell reads one cell as the Texera value for an attribute type, returning null on a conversion failure to match the row path's lenient decode.

SpecializedFilterOpExec columnar path:
- The operator now also mixes in ColumnarOperatorExecutor while keeping its row path (the per-tuple OR over predicates) unchanged. columnarMask builds a boolean survivor mask for a single numeric predicate (Float8Vector or IntVector) read straight from the Arrow vector, and returns None for any other predicate or column shape so the caller falls back to rows.
- processColumnarBatch deserializes the batch and returns Emit(selectRows(mask)) when the mask applies, else Unsupported. columnarSelection returns the survivor indices without compacting, and columnarMatchCount is the fused filter-then-count that reports the survivor count with no batch materialized. A per-operator RootAllocator backs these and is freed in close().

ArrowSourceOpExec columnar produce:
- produceColumnarBatch hands each already-Arrow file batch out as a VectorSchemaRoot, moving the loaded batch's vectors into an independent root via a zero-copy TransferPair so the caller can serialize and close it while the reader loads the next batch. It returns None (the row path) when offset or limit is set.
Adds the wire that carries a data batch between workers as Arrow IPC bytes (a ColumnarFrame) instead of decoded Tuples (a DataFrame), behind an off-by-default flag. When enabled, every operator's output goes out as Arrow and is decoded back at the receiver, so the format is transparent to the row path; the payoff is on cross-worker links, where one Arrow IPC block replaces per-tuple serialization. Consuming the Arrow batch without decoding (for a ColumnarOperatorExecutor) is a later optimization; this lands the transport and keeps results identical with the flag on or off.

Flag:
- ApplicationConfig.enableColumnarWire reads network-buffering.enable-columnar-wire, defaulting to false (override NETWORK_BUFFERING_ENABLE_COLUMNAR_WIRE). Off by default, so the default wire is DataFrame and unchanged.

Payload:
- ColumnarFrame(arrowIpcBytes, rowCount, schema) is a new DataPayload. It carries a batch as an Arrow IPC stream plus the row count and Texera schema that describe it without decoding. inMemSize is the byte length.

Send side (NetworkOutputBuffer):
- The buffer takes a `columnar` flag (defaulting to ApplicationConfig.enableColumnarWire). On flush, when columnar is on it serializes the buffered tuples to Arrow once (all tuples on a link share the schema) and sends a ColumnarFrame; otherwise it sends a DataFrame as before. An empty flush still sends nothing.

Receive side:
- DataProcessor.processDataPayload handles ColumnarFrame by decoding the Arrow bytes back to tuples and running the identical batch path as DataFrame; the shared startInputBatch helper carries the common state-transition, initBatch, and first-tuple step for both. A columnar batch bound for a Python worker is decoded back to tuples in PythonProxyClient.sendData and handed over as a Data batch, so the Python path is unaffected.

ArrowUtils:
- serializeTuples(schema, tuples) writes a tuple batch to Arrow IPC stream bytes; deserializeTuples(bytes) reads them back to tuples. These are the ColumnarFrame encode/decode endpoints.

Tests:
- ArrowUtilsSpec covers the serializeTuples/deserializeTuples round-trip over a multi-row batch of every field type (including a null row and an empty batch), the payload contract the wire depends on.
- NetworkOutputBufferColumnarSeamSpec checks the buffer emits a ColumnarFrame (carrying the same rows, row count, and schema) when the flag is on and a DataFrame when off, on both the manual and the size-triggered flush, and nothing on an empty flush.
- SelectionVectorFilterSpec checks the specialized filter's native-Arrow path (selection vector, fused filter-count, and the compacting Emit path) against the row path as the source of truth.
AggregateOpExec can now consume an Arrow columnar batch directly, accumulating each group's partial result by reading the group-key and aggregated cells straight from their Arrow vectors instead of decoding a Tuple per row. The accumulation itself is shared with the row path, so a grouped aggregate produces the identical result whether its input arrives as rows or as a columnar batch.

Value-based accumulation (DistributedAggregation):
- DistributedAggregation now accumulates one already-extracted value via iterateValue(partial, value), and carries an extract(tuple) that pulls the single cell an aggregation reads from a tuple (null for COUNT(*)). The row path's iterate(partial, tuple) is derived as iterateValue(partial, extract(tuple)), so accumulation is defined once and both paths use it. init, merge, and finalAgg are unchanged.
- AggregationOperation's six functions (sum, count, average, min, max, concat) are rewritten to this shape: each supplies iterateValue (the per-value logic) and extract (tuple.getField of its attribute, or null for COUNT(*)). count keeps its captured countAllRows flag, so COUNT(*) counts every row and COUNT(column) counts non-null values. average reads a value numerically via a value-based helper (a timestamp counts as its epoch milliseconds, any other value through its string form, null skipped), matching the previous tuple-based reading.

Columnar accumulate (AggregateOpExec):
- The operator mixes in ColumnarOperatorExecutor. processColumnarBatch deserializes the batch and runs accumulateRoot, which resolves once per batch the source-column index and Texera type of each group-by key and each aggregation's input column (index -1 for COUNT(*)), then for each row builds the group key and feeds each aggregation its cell value through iterateValue. It is a blocking operator, so it emits nothing per batch (ColumnarResult.Consumed); results are produced at onFinish via the existing path. A per-operator RootAllocator backs the decode and is freed in close().

Tests:
- AggregateOpSpec adds a columnar-vs-row check: a grouped SUM/COUNT/AVERAGE/MIN/MAX over rows including a null value gives the same per-group result through processColumnarBatch as through processTuple.
- DistributedAggregationSpec is updated to the new constructor (iterateValue + extract) and asserts iterate is derived from them (the row path equals the value path).
Adds a benchmark that measures the columnar wire against the row wire on the real Pekko/Kryo stack, and speeds up the Arrow-to-tuple decode it exercises.

Decode (ArrowUtils):
- getTexeraTuple gains an overload that takes the already-resolved Texera schema, and the no-schema overload delegates to it. deserializeTuples now resolves the Texera schema, the field-vector array, and each column's attribute type once per stream and builds each row with a plain index loop (getCell per column into a fresh Array, then the Tuple constructor), instead of rebuilding the schema and allocating a Scala collection per row.

Benchmark (ColumnarWireBenchSpec):
- Serializes batches through the configured Pekko serialization (SerializationExtension on an ActorSystem built from AmberRuntime.pekkoConfig), so the row path is a real Kryo DataFrame round-trip and the columnar path is ArrowUtils.serializeTuples wrapped in a ColumnarFrame. For narrow-numeric, wide-mixed, and string-heavy workloads at 2k/20k/100k rows it reports sender encode time, receiver decode time, and serialized payload size for each path.
- A second scenario measures the decisive case: a vectorized receiver (a SpecializedFilterOpExec) that reads the Arrow columns directly via processColumnarBatch, against the row path that deserializes the DataFrame and runs the predicate per tuple.
- It is skipped unless run with -Dcolumnar.bench=true, so it does not run in CI.

Findings (this machine, JDK 17, median of 7): the columnar wire has a smaller payload (about 0.45x-0.91x) and a cheaper encode (about 0.5x at 100k rows), but decoding Arrow back to tuples is far more expensive than Kryo rehydrating a DataFrame (roughly 40x-68x), so a receiver that decodes to rows is a net loss. A receiver that vectorizes instead wins clearly: the filter's processColumnarBatch is about 0.13x the row path's receive+process at 100k rows (around 7x faster). So the columnar wire pays off only when the receiver consumes columns natively; routing a ColumnarFrame to a ColumnarOperatorExecutor's processColumnarBatch in the worker (instead of always decoding) is the next step.
Routes an incoming Arrow columnar batch to a ColumnarOperatorExecutor's processColumnarBatch in the worker, so a columnar-capable operator consumes the Arrow columns directly instead of the batch always being decoded back to tuples first. The benchmark showed decoding Arrow to tuples is the columnar wire's dominant cost, far more than Kryo rehydrating a DataFrame, so the wire only pays off when the receiver reads columns natively; this is the path that realizes that win.

Worker routing (DataProcessor):
- A ColumnarFrame now goes through processColumnarFrame. When the executor is a ColumnarOperatorExecutor, the worker transitions to RUNNING, binds the executor's state references, and hands it the Arrow IPC bytes via processColumnarBatch, then routes the ColumnarResult: Emit(root) is materialized to output tuples (the existing output path is tuple-based) and the root is freed; EmitRows installs the returned rows as the output; Consumed emits nothing now (a blocking operator that accumulated the batch); Unsupported, or any non-columnar executor, falls back to decoding the batch to tuples and processing it as a DataFrame. The READY->RUNNING transition is factored out of startInputBatch into transitToRunning, shared by both paths. For a natively consumed batch the per-tuple input manager is left empty (there is no per-tuple input to drain), so the DP loop drains only the output iterator.
- StatisticsManager gains a batched increaseInputStatistics(portId, count, size) so the columnar path reports a whole batch's row count and serialized size in one call rather than once per tuple; the single-tuple overload delegates to it.

Root-to-tuples decode (ArrowUtils):
- tuplesOfRoot decodes an already-open VectorSchemaRoot to tuples (the Emit output path's root-to-rows step). Like deserializeTuples it resolves the Texera schema, the field vectors, and each column's type once and reads each row with a plain index loop, but it works over a live root without closing it.

Tests:
- DataProcessorSpec adds the four routing outcomes plus the non-columnar case, driving real ColumnarFrames through processDataPayload with fake executors: Emit (a pass-through filter) emits the batch's rows and never calls processTuple; EmitRows emits the returned rows; Consumed emits nothing and still counts the batch's input statistics; Unsupported and a plain row-only executor both decode the frame and run it through processTuple per row.
- ArrowUtilsSpec asserts tuplesOfRoot decodes a live root to the same tuples deserializeTuples produces from the IPC bytes.
When a ColumnarOperatorExecutor emits an Arrow result batch, the worker can now ship it straight downstream as a ColumnarFrame instead of decoding it to tuples and letting the network buffer re-encode those tuples back to Arrow. This removes the columns-to-rows-to-columns round-trip that otherwise gave back the vectorized-receive win at the output boundary. It is the first step of the per-edge columnar design (phase 1): eligibility is limited to the edges where shipping the whole batch to a single receiver is correct, and every other case falls back to the existing row output path unchanged.

Output fast path (OutputManager):
- tryEmitColumnarBatch(root, outputPortId) resolves the output port, checks eligibility, and on success serializes the root once and sends the resulting ColumnarFrame through each receiver's NetworkOutputBuffer, returning (portId, serializedByteCount) so the caller records output statistics; it returns None (caller falls back) when not eligible. It does not close the root (the caller owns it). An edge is eligible only when the port needs no storage, every partitioner on the port is one-to-one (a single receiver gets the whole batch), and the receiver buffers are in columnar mode. Hash/range/round-robin shuffles need per-row partitioning (a later phase) and broadcast/storage are deferred, so those return None. The emitted Arrow schema must equal the declared output-port schema, otherwise it falls back so the row path's enforceSchema can adjust it.
- setColumnarOutput(enabled) flips the columnar flag on every current edge. This is a provisional phase-1 switch; phase 2 replaces it with a per-edge decision from the planner (local vs cross-node, and whether the receiver consumes columns).

Network buffer (NetworkOutputBuffer):
- sendColumnarFrame(frame) ships an already-serialized Arrow batch straight to the receiver, flushing any buffered tuples first so the batch keeps its place in the FIFO stream. Its columnar flag becomes a var so the output manager can set it per edge.

Worker routing (DataProcessor):
- The Emit(root) case now tries tryEmitColumnarBatch first and records the batch's output statistics in one call; only when the edge is not eligible does it materialize the root to tuples through the row output path. The root is freed in a finally either way.

Batched statistics (StatisticsManager):
- increaseOutputStatistics(portId, count, size) mirrors the batched input overload, so the columnar output path counts a whole batch's rows and serialized bytes in one call rather than once per tuple; the single-tuple overload delegates to it.

Tests:
- OutputManagerSpec covers tryEmitColumnarBatch: a one-to-one columnar edge ships a single ColumnarFrame that decodes back to the input rows; a hash-shuffle edge is declined (returns None, sends nothing); and an edge with columnar output off is declined.
- NetworkOutputBufferColumnarSeamSpec covers sendColumnarFrame flushing buffered rows before the prebuilt frame, in order.
- DataProcessorSpec adds the eligible-edge case (Emit goes out as one ColumnarFrame, no tuple round-trip, processTuple never runs, output statistics count the batch) and keeps the not-columnar case as the row-output fallback.
…perator

The columnar wire is no longer a single global flag that flips every edge at once. The planner now decides it per link, and an edge ships Arrow only when the downstream operator can consume columns natively. This stops the worst case of the blanket switch, shipping Arrow to a row-only operator that then pays a per-tuple decode, and is the capability half of the per-edge design (phase 2). Locality (a same-JVM link gains nothing from serializing) is a later refinement; worker placement is not modeled at planning time and actor refs resolve lazily, so this change keeps that axis out.

Operator capability (PhysicalOp):
- isColumnarCapable resolves, by name and without loading or instantiating the class, whether the operator's executor mixes in ColumnarOperatorExecutor. Any lookup miss (a code-based or unknown executor) is not capable. This auto-discovers every columnar operator (filter, aggregate, and future ones) with no per-operator registration.

Planner decision (ResourceAllocator, LinkConfig):
- allocate sets LinkConfig.columnar = ApplicationConfig.enableColumnarWire && the downstream operator isColumnarCapable, for each link. The global flag is now a master switch folded into the per-edge decision rather than a second source of truth.

Threading to the sender (proto + handlers):
- AddPartitioningRequest gains a columnar bool. LinkWorkersHandler fills it from the link's LinkConfig; the worker's AddPartitioningHandler passes it to OutputManager.addPartitionerWithPartitioning, which constructs that edge's NetworkOutputBuffers in columnar mode. NetworkOutputBuffer.columnar now defaults to false (the planner is the authority) instead of reading ApplicationConfig.enableColumnarWire itself.

Tests:
- PhysicalOpSpec: isColumnarCapable holds for an executor that implements ColumnarOperatorExecutor, and not for a plain executor, a code-based executor, or an unresolvable class name.
- OutputManagerSpec: addPartitionerWithPartitioning builds columnar buffers when told to (a flushed batch goes out as a ColumnarFrame) and row buffers by default (a DataFrame).
- The AddPartitioningRequest constructions in WorkerSpec, LoggingSpec, and ArrowFlightActorBench pass columnar = false for the new field.
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @Yicong-Huang, @aicam
    You can notify them by mentioning @aglinxinyuan, @Yicong-Huang, @aicam in a comment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(amber): columnar output without the round-trip, decided per edge

1 participant