Skip to content

feat(amber): columnar foundation, contract and opt-in Arrow wire - #9549

Draft
Ma77Ball wants to merge 5 commits into
apache:mainfrom
Ma77Ball:columnar/01-foundation
Draft

Ma77Ball wants to merge 5 commits into
apache:mainfrom
Ma77Ball:columnar/01-foundation

Conversation

@Ma77Ball

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

  • Add the ColumnarOperatorExecutor / ColumnarResult contract so an operator can opt into reading Arrow columns directly; when an operator cannot, the engine decodes to rows, so results never differ.
  • Add the opt-in ColumnarFrame wire (a batch of Arrow columns alongside the row DataFrame), gated by ApplicationConfig.enableColumnarWire and off by default.
  • Speed up the Arrow-to-tuple decode by resolving the Texera schema once per batch instead of rebuilding it per row.
  • Add vectorized receive in DataProcessor so a capable filter or aggregate reads the Arrow columns directly instead of decoding to tuples.

Any related issues, documentation, discussions?

Closes: #9547
Part of: #9542
Discussion: #8571

How was this PR tested?

  • Run sbt "WorkflowCore/testOnly *ArrowUtilsSpec", expect the Arrow round-trip and faster-decode specs to pass with values identical to the row decode.
  • Run sbt "WorkflowExecutionService/testOnly *DataProcessorSpec *OutputManagerSpec", expect the columnar-frame routing and vectorized-receive cases to pass.
  • Manual: set enableColumnarWire=true in common/config/src/main/resources/application.conf, run a CSV scan into a filter at http://texera.localhost, and confirm the output rows are identical to a run with the flag off.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy.

The foundation for a columnar Arrow wire: an operator-side contract for consuming and producing Arrow batches directly, the Arrow helpers it needs, and the first two operators that implement it (the specialized filter and the Arrow file scan). Nothing is wired into the messaging layer yet, so no operator receives columnar batches at runtime; these are the capabilities the wire will drive.

Contract (ColumnarOperatorExecutor):
- ColumnarOperatorExecutor is a trait with processColumnarBatch(arrowIpcBytes, port) that returns a ColumnarResult. An operator mixes it in to declare it can consume an Arrow batch without decoding each row to a Tuple.
- ColumnarResult is the sealed outcome of consuming one batch: Emit(root) (emit this Arrow result, caller serializes and closes the root), EmitRows(iterator) (emit row-shaped output, e.g. a join), Consumed (consumed with nothing to emit now, a blocking operator accumulating state), and Unsupported (not handled, caller falls back to the row path).
- SourceOperatorExecutor gains produceColumnarBatch(), defaulting to None, so a source whose data is already columnar can hand out Arrow roots instead of going through produceTuple; the caller owns and closes each root.

ArrowUtils helpers:
- serializeRoot writes a VectorSchemaRoot to Arrow IPC stream bytes without closing it; deserializeRootFold reads one IPC batch, applies a function to the live root, and closes the reader (the function must copy out what it keeps).
- selectRows copies the rows a boolean keep-mask selects into a new compacted root; selectionOf returns just the surviving row indices (a selection vector) without copying any cell, and gather materializes the rows a selection vector names into a new root, so a consumer that only needs a count or an aggregate reads indices and never materializes.
- projectionIndices resolves column names to their Arrow field indices by exact name; getCell reads one cell as the Texera value for an attribute type, returning null on a conversion failure to match the row path's lenient decode.

SpecializedFilterOpExec columnar path:
- The operator now also mixes in ColumnarOperatorExecutor while keeping its row path (the per-tuple OR over predicates) unchanged. columnarMask builds a boolean survivor mask for a single numeric predicate (Float8Vector or IntVector) read straight from the Arrow vector, and returns None for any other predicate or column shape so the caller falls back to rows.
- processColumnarBatch deserializes the batch and returns Emit(selectRows(mask)) when the mask applies, else Unsupported. columnarSelection returns the survivor indices without compacting, and columnarMatchCount is the fused filter-then-count that reports the survivor count with no batch materialized. A per-operator RootAllocator backs these and is freed in close().

ArrowSourceOpExec columnar produce:
- produceColumnarBatch hands each already-Arrow file batch out as a VectorSchemaRoot, moving the loaded batch's vectors into an independent root via a zero-copy TransferPair so the caller can serialize and close it while the reader loads the next batch. It returns None (the row path) when offset or limit is set.
Adds the wire that carries a data batch between workers as Arrow IPC bytes (a ColumnarFrame) instead of decoded Tuples (a DataFrame), behind an off-by-default flag. When enabled, every operator's output goes out as Arrow and is decoded back at the receiver, so the format is transparent to the row path; the payoff is on cross-worker links, where one Arrow IPC block replaces per-tuple serialization. Consuming the Arrow batch without decoding (for a ColumnarOperatorExecutor) is a later optimization; this lands the transport and keeps results identical with the flag on or off.

Flag:
- ApplicationConfig.enableColumnarWire reads network-buffering.enable-columnar-wire, defaulting to false (override NETWORK_BUFFERING_ENABLE_COLUMNAR_WIRE). Off by default, so the default wire is DataFrame and unchanged.

Payload:
- ColumnarFrame(arrowIpcBytes, rowCount, schema) is a new DataPayload. It carries a batch as an Arrow IPC stream plus the row count and Texera schema that describe it without decoding. inMemSize is the byte length.

Send side (NetworkOutputBuffer):
- The buffer takes a `columnar` flag (defaulting to ApplicationConfig.enableColumnarWire). On flush, when columnar is on it serializes the buffered tuples to Arrow once (all tuples on a link share the schema) and sends a ColumnarFrame; otherwise it sends a DataFrame as before. An empty flush still sends nothing.

Receive side:
- DataProcessor.processDataPayload handles ColumnarFrame by decoding the Arrow bytes back to tuples and running the identical batch path as DataFrame; the shared startInputBatch helper carries the common state-transition, initBatch, and first-tuple step for both. A columnar batch bound for a Python worker is decoded back to tuples in PythonProxyClient.sendData and handed over as a Data batch, so the Python path is unaffected.

ArrowUtils:
- serializeTuples(schema, tuples) writes a tuple batch to Arrow IPC stream bytes; deserializeTuples(bytes) reads them back to tuples. These are the ColumnarFrame encode/decode endpoints.

Tests:
- ArrowUtilsSpec covers the serializeTuples/deserializeTuples round-trip over a multi-row batch of every field type (including a null row and an empty batch), the payload contract the wire depends on.
- NetworkOutputBufferColumnarSeamSpec checks the buffer emits a ColumnarFrame (carrying the same rows, row count, and schema) when the flag is on and a DataFrame when off, on both the manual and the size-triggered flush, and nothing on an empty flush.
- SelectionVectorFilterSpec checks the specialized filter's native-Arrow path (selection vector, fused filter-count, and the compacting Emit path) against the row path as the source of truth.
AggregateOpExec can now consume an Arrow columnar batch directly, accumulating each group's partial result by reading the group-key and aggregated cells straight from their Arrow vectors instead of decoding a Tuple per row. The accumulation itself is shared with the row path, so a grouped aggregate produces the identical result whether its input arrives as rows or as a columnar batch.

Value-based accumulation (DistributedAggregation):
- DistributedAggregation now accumulates one already-extracted value via iterateValue(partial, value), and carries an extract(tuple) that pulls the single cell an aggregation reads from a tuple (null for COUNT(*)). The row path's iterate(partial, tuple) is derived as iterateValue(partial, extract(tuple)), so accumulation is defined once and both paths use it. init, merge, and finalAgg are unchanged.
- AggregationOperation's six functions (sum, count, average, min, max, concat) are rewritten to this shape: each supplies iterateValue (the per-value logic) and extract (tuple.getField of its attribute, or null for COUNT(*)). count keeps its captured countAllRows flag, so COUNT(*) counts every row and COUNT(column) counts non-null values. average reads a value numerically via a value-based helper (a timestamp counts as its epoch milliseconds, any other value through its string form, null skipped), matching the previous tuple-based reading.

Columnar accumulate (AggregateOpExec):
- The operator mixes in ColumnarOperatorExecutor. processColumnarBatch deserializes the batch and runs accumulateRoot, which resolves once per batch the source-column index and Texera type of each group-by key and each aggregation's input column (index -1 for COUNT(*)), then for each row builds the group key and feeds each aggregation its cell value through iterateValue. It is a blocking operator, so it emits nothing per batch (ColumnarResult.Consumed); results are produced at onFinish via the existing path. A per-operator RootAllocator backs the decode and is freed in close().

Tests:
- AggregateOpSpec adds a columnar-vs-row check: a grouped SUM/COUNT/AVERAGE/MIN/MAX over rows including a null value gives the same per-group result through processColumnarBatch as through processTuple.
- DistributedAggregationSpec is updated to the new constructor (iterateValue + extract) and asserts iterate is derived from them (the row path equals the value path).
Adds a benchmark that measures the columnar wire against the row wire on the real Pekko/Kryo stack, and speeds up the Arrow-to-tuple decode it exercises.

Decode (ArrowUtils):
- getTexeraTuple gains an overload that takes the already-resolved Texera schema, and the no-schema overload delegates to it. deserializeTuples now resolves the Texera schema, the field-vector array, and each column's attribute type once per stream and builds each row with a plain index loop (getCell per column into a fresh Array, then the Tuple constructor), instead of rebuilding the schema and allocating a Scala collection per row.

Benchmark (ColumnarWireBenchSpec):
- Serializes batches through the configured Pekko serialization (SerializationExtension on an ActorSystem built from AmberRuntime.pekkoConfig), so the row path is a real Kryo DataFrame round-trip and the columnar path is ArrowUtils.serializeTuples wrapped in a ColumnarFrame. For narrow-numeric, wide-mixed, and string-heavy workloads at 2k/20k/100k rows it reports sender encode time, receiver decode time, and serialized payload size for each path.
- A second scenario measures the decisive case: a vectorized receiver (a SpecializedFilterOpExec) that reads the Arrow columns directly via processColumnarBatch, against the row path that deserializes the DataFrame and runs the predicate per tuple.
- It is skipped unless run with -Dcolumnar.bench=true, so it does not run in CI.

Findings (this machine, JDK 17, median of 7): the columnar wire has a smaller payload (about 0.45x-0.91x) and a cheaper encode (about 0.5x at 100k rows), but decoding Arrow back to tuples is far more expensive than Kryo rehydrating a DataFrame (roughly 40x-68x), so a receiver that decodes to rows is a net loss. A receiver that vectorizes instead wins clearly: the filter's processColumnarBatch is about 0.13x the row path's receive+process at 100k rows (around 7x faster). So the columnar wire pays off only when the receiver consumes columns natively; routing a ColumnarFrame to a ColumnarOperatorExecutor's processColumnarBatch in the worker (instead of always decoding) is the next step.
Routes an incoming Arrow columnar batch to a ColumnarOperatorExecutor's processColumnarBatch in the worker, so a columnar-capable operator consumes the Arrow columns directly instead of the batch always being decoded back to tuples first. The benchmark showed decoding Arrow to tuples is the columnar wire's dominant cost, far more than Kryo rehydrating a DataFrame, so the wire only pays off when the receiver reads columns natively; this is the path that realizes that win.

Worker routing (DataProcessor):
- A ColumnarFrame now goes through processColumnarFrame. When the executor is a ColumnarOperatorExecutor, the worker transitions to RUNNING, binds the executor's state references, and hands it the Arrow IPC bytes via processColumnarBatch, then routes the ColumnarResult: Emit(root) is materialized to output tuples (the existing output path is tuple-based) and the root is freed; EmitRows installs the returned rows as the output; Consumed emits nothing now (a blocking operator that accumulated the batch); Unsupported, or any non-columnar executor, falls back to decoding the batch to tuples and processing it as a DataFrame. The READY->RUNNING transition is factored out of startInputBatch into transitToRunning, shared by both paths. For a natively consumed batch the per-tuple input manager is left empty (there is no per-tuple input to drain), so the DP loop drains only the output iterator.
- StatisticsManager gains a batched increaseInputStatistics(portId, count, size) so the columnar path reports a whole batch's row count and serialized size in one call rather than once per tuple; the single-tuple overload delegates to it.

Root-to-tuples decode (ArrowUtils):
- tuplesOfRoot decodes an already-open VectorSchemaRoot to tuples (the Emit output path's root-to-rows step). Like deserializeTuples it resolves the Texera schema, the field vectors, and each column's type once and reads each row with a plain index loop, but it works over a live root without closing it.

Tests:
- DataProcessorSpec adds the four routing outcomes plus the non-columnar case, driving real ColumnarFrames through processDataPayload with fake executors: Emit (a pass-through filter) emits the batch's rows and never calls processTuple; EmitRows emits the returned rows; Consumed emits nothing and still counts the batch's input statistics; Unsupported and a plain row-only executor both decode the frame and run it through processTuple per row.
- ArrowUtilsSpec asserts tuplesOfRoot decodes a live root to the same tuples deserializeTuples produces from the IPC bytes.
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @Yicong-Huang, @eugenegujing
    You can notify them by mentioning @aglinxinyuan, @Yicong-Huang, @eugenegujing in a comment.

@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.43972% with 89 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.45%. Comparing base (4db10cf) to head (9b7735c).

Files with missing lines Patch % Lines
...cala/org/apache/texera/amber/util/ArrowUtils.scala 64.64% 34 Missing and 1 partial ⚠️
...mber/operator/filter/SpecializedFilterOpExec.scala 54.54% 8 Missing and 12 partials ⚠️
...operator/source/scan/arrow/ArrowSourceOpExec.scala 0.00% 13 Missing ⚠️
...amber/core/executor/ColumnarOperatorExecutor.scala 0.00% 6 Missing ⚠️
.../architecture/pythonworker/PythonProxyClient.scala 0.00% 4 Missing and 1 partial ⚠️
...era/amber/operator/aggregate/AggregateOpExec.scala 85.71% 0 Missing and 5 partials ⚠️
...ber/engine/architecture/worker/DataProcessor.scala 94.73% 2 Missing ⚠️
...chitecture/worker/managers/StatisticsManager.scala 50.00% 0 Missing and 1 partial ⚠️
...amber/engine/common/ambermessage/DataPayload.scala 66.66% 1 Missing ⚠️
...a/amber/core/executor/SourceOperatorExecutor.scala 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9549      +/-   ##
============================================
- Coverage     91.57%   91.45%   -0.12%     
- Complexity     5159     5192      +33     
============================================
  Files          1300     1301       +1     
  Lines         58092    58345     +253     
  Branches       7615     7666      +51     
============================================
+ Hits          53198    53360     +162     
- Misses         2958     3028      +70     
- Partials       1936     1957      +21     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.56% <ø> (ø) Carriedforward from 4db10cf
amber 85.18% <68.43%> (-0.26%) ⬇️
computing-unit-managing-service 63.88% <ø> (ø)
config-service 87.50% <ø> (ø)
file-service 82.16% <ø> (ø)
frontend 96.45% <ø> (ø) Carriedforward from 4db10cf
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.38% <ø> (ø) Carriedforward from 4db10cf
result-service 85.71% <ø> (ø)
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 5 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main 4db10cf benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 396 0.242 23,951/36,619/36,619 us 🔴 +11.3% / 🔴 +185.8%
🔴 bs=100 sw=10 sl=64 811 0.495 118,669/151,766/151,766 us 🟢 -10.1% / 🔴 +85.5%
⚪ bs=1000 sw=10 sl=64 970 0.592 1,027,168/1,096,428/1,096,428 us ⚪ within ±5% / 🔴 +43.5%
Baseline details

Latest main 4db10cf from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 396 tuples/sec 446 tuples/sec 1,084 tuples/sec -11.2% -63.5%
bs=10 sw=10 sl=64 MB/s 0.242 MB/s 0.272 MB/s 0.662 MB/s -11.0% -63.4%
bs=10 sw=10 sl=64 p50 23,951 us 21,528 us 9,548 us +11.3% +150.9%
bs=10 sw=10 sl=64 p95 36,619 us 40,478 us 12,811 us -9.5% +185.8%
bs=10 sw=10 sl=64 p99 36,619 us 40,478 us 15,432 us -9.5% +137.3%
bs=100 sw=10 sl=64 throughput 811 tuples/sec 859 tuples/sec 1,467 tuples/sec -5.6% -44.7%
bs=100 sw=10 sl=64 MB/s 0.495 MB/s 0.525 MB/s 0.895 MB/s -5.7% -44.7%
bs=100 sw=10 sl=64 p50 118,669 us 113,479 us 74,441 us +4.6% +59.4%
bs=100 sw=10 sl=64 p95 151,766 us 168,823 us 81,800 us -10.1% +85.5%
bs=100 sw=10 sl=64 p99 151,766 us 168,823 us 103,712 us -10.1% +46.3%
bs=1000 sw=10 sl=64 throughput 970 tuples/sec 972 tuples/sec 1,517 tuples/sec -0.2% -36.1%
bs=1000 sw=10 sl=64 MB/s 0.592 MB/s 0.593 MB/s 0.926 MB/s -0.2% -36.1%
bs=1000 sw=10 sl=64 p50 1,027,168 us 1,028,890 us 725,911 us -0.2% +41.5%
bs=1000 sw=10 sl=64 p95 1,096,428 us 1,061,154 us 764,152 us +3.3% +43.5%
bs=1000 sw=10 sl=64 p99 1,096,428 us 1,061,154 us 808,901 us +3.3% +35.5%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,505.13,200,128000,396,0.242,23950.97,36619.29,36619.29
1,100,10,64,20,2464.83,2000,1280000,811,0.495,118668.51,151765.57,151765.57
2,1000,10,64,20,20620.07,20000,12800000,970,0.592,1027167.56,1096428.03,1096428.03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(amber): columnar foundation, operator contract and opt-in Arrow wire

2 participants