Skip to content

refactor(amber, common): simplify tuple, Arrow, operator allocs - #9515

Merged
aglinxinyuan merged 8 commits into
apache:mainfrom
aglinxinyuan:refactor/common-tuple-operator-allocs
Nov 17, 2026
Merged

aglinxinyuan merged 8 commits into
apache:mainfrom
aglinxinyuan:refactor/common-tuple-operator-allocs

Conversation

@aglinxinyuan

@aglinxinyuan aglinxinyuan commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Simplifies transient collection allocations, repeated traversals, and redundant metadata/regex builds across common/workflow-core, common/workflow-operator, and amber (+316/-238 production lines; +611/-249 total including characterization tests), with no change to observable behavior or generated code:

  • Tuple & Arrow conversion: Checks Schema add/remove names against the lowercase attributeIndex keys, hashes and compares Tuple fields in place, walks attributes with one iterator in Tuple.Builder and SeqTupleLike.enforceSchema instead of zipping or copying arrays, and populates Arrow field arrays and schemas without extra temporary collections in ArrowUtils. A getTexeraTuple(rowIndex, vectorSchemaRoot, schema) overload takes an already converted schema, so PythonProxyServer converts a root's Arrow schema once per Data put and ArrowSourceOpExec once per file instead of once per row.
  • UI UDF parameter injection: Validates and binds UI parameter resources in single passes (UiUdfParameterContract, UdfUiParameterSupport), replaces incremental String.patch edits and duplicate AST field scans in JavaUdfUiParameterInjector, avoids line splitting and pre-compiles the reserved hook regex in PythonUdfUiParameterInjector, and formats sorted manifest entries via HexFormat and a shared formatEntries helper in RUdfUiParameterInjector.
  • Text & ML scorer operators: Hoists dictionary stop-word sets and skips unused literal rendering in conjunction mode (DictionaryMatcherOpDesc, DictionaryMatcherOpExec), removes intermediate wrappers in CaseSensitiveAnalyzer, KeywordSearchOpDesc, and KeywordSearchOpExec, avoids leading regex flag list allocations in RegexOpDesc, and streamlines column validation in MachineLearningScorerOpDesc.

Any related issues, documentation, discussions?

Closes #9514.
Closes #9521.

How was this PR tested?

Characterization tests came first (a347f3d946, 1370fab8c2), then the cleanups and helper extractions (eeb039181d, 9efd195ade, bcaa8684e5, 73d6f66b1e). On final HEAD (73d6f66b1e):

sbt "WorkflowCore/scalafmtCheck" "WorkflowCore/Test/scalafmtCheck" "WorkflowOperator/scalafmtCheck" "WorkflowOperator/Test/scalafmtCheck" "WorkflowExecutionService/scalafmtCheck" "WorkflowExecutionService/Test/scalafmtCheck" "WorkflowCore/scalafix --check" "WorkflowCore/Test/scalafix --check" "WorkflowOperator/scalafix --check" "WorkflowOperator/Test/scalafix --check" "WorkflowExecutionService/scalafix --check" "WorkflowExecutionService/Test/scalafix --check"
STORAGE_ICEBERG_CATALOG_TYPE=hadoop sbt "WorkflowCore/testOnly * -- -l org.apache.texera.common.tags.NonParallelTest"
UDF_PYTHON_PATH="<python3.12>" sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.dictionary.* org.apache.texera.amber.operator.keywordSearch.* org.apache.texera.amber.operator.machineLearning.Scorer.* org.apache.texera.amber.operator.regex.* org.apache.texera.amber.operator.udf.* org.apache.texera.amber.operator.source.scan.arrow.* org.apache.texera.amber.util.ArrowUtilsSpec"
STORAGE_ICEBERG_CATALOG_TYPE=hadoop UDF_PYTHON_PATH="<python3.12>" sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.architecture.sendsemantics.partitioners.PartitionersSpec org.apache.texera.amber.engine.common.storage.EmptyRecordStorageSpec org.apache.texera.amber.engine.common.storage.SequentialRecordStorageSpec org.apache.texera.amber.engine.common.storage.VFSRecordStorageSpec org.apache.texera.amber.engine.architecture.pythonworker.*"
  • Format & Scalafix checks: all 6 scalafmtCheck and 6 scalafix --check targets passed.
  • WorkflowCore non-Docker suites (-l org.apache.texera.common.tags.NonParallelTest): 54 suites completed, 712 succeeded, 0 failed, 1 pre-existing pending (PortIdentityKeyDeserializerSpec's pendingUntilFixed case).
  • WorkflowOperator affected packages: 29 suites completed, 412 succeeded, 0 failed (including live Python 3.12 pandas/sklearn scoring and regex checks).
  • WorkflowExecutionService tuple/storage and Python proxy suites: 9 suites completed, 92 succeeded, 0 failed.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Antigravity IDE (Gemini 3.8 Flash), Claude Code (Claude Opus 5.5)

Add characterization tests in zero-overlap common/workflow-core and common/workflow-operator specs before allocation and helper cleanups.
Combine verified BA-952 (4-file core), BA-946, BA-621, and BA-953 allocation and traversal cleanups plus shared helpers in Schema and RUdfUiParameterInjector.
Copilot AI balanced review requested due to automatic review settings November 17, 2026 05:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@aglinxinyuan aglinxinyuan added the refactor Refactor the code label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @kz930, @carloea2, @Yicong-Huang
    You can notify them by mentioning @kz930, @carloea2, @Yicong-Huang in a comment.

@aglinxinyuan
aglinxinyuan requested a review from kunwp1 November 17, 2026 05:57
@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.30435% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.60%. Comparing base (82224ea) to head (73d6f66).

Files with missing lines Patch % Lines
...operator/udf/java/JavaUdfUiParameterInjector.scala 57.14% 0 Missing and 3 partials ⚠️
...ator/udf/python/PythonUdfUiParameterInjector.scala 75.00% 1 Missing and 2 partials ⚠️
...ala/org/apache/texera/amber/core/tuple/Tuple.scala 96.66% 0 Missing and 1 partial ⚠️
...org/apache/texera/amber/core/tuple/TupleLike.scala 85.71% 0 Missing and 1 partial ⚠️
...cala/org/apache/texera/amber/util/ArrowUtils.scala 96.66% 0 Missing and 1 partial ⚠️
.../operator/dictionary/DictionaryMatcherOpDesc.scala 90.90% 0 Missing and 1 partial ⚠️
...operator/source/scan/arrow/ArrowSourceOpExec.scala 0.00% 0 Missing and 1 partial ⚠️
...era/amber/operator/udf/UdfUiParameterSupport.scala 83.33% 0 Missing and 1 partial ⚠️
...ra/amber/operator/udf/UiUdfParameterContract.scala 50.00% 0 Missing and 1 partial ⚠️
...amber/operator/udf/r/RUdfUiParameterInjector.scala 93.33% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main    #9515   +/-   ##
=========================================
  Coverage     91.60%   91.60%           
- Complexity     5043     5055   +12     
=========================================
  Files          1284     1284           
  Lines         57020    57081   +61     
  Branches       7414     7430   +16     
=========================================
+ Hits          52232    52288   +56     
  Misses         2915     2915           
- Partials       1873     1878    +5     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.56% <ø> (ø) Carriedforward from 82224ea
amber 85.34% <91.30%> (+0.01%) ⬆️
computing-unit-managing-service 64.02% <ø> (ø)
config-service 87.73% <ø> (ø)
file-service 82.16% <ø> (ø)
frontend 96.42% <ø> (ø) Carriedforward from 82224ea
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.50% <ø> (ø) Carriedforward from 82224ea
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aglinxinyuan
aglinxinyuan requested a review from mengw15 November 17, 2026 06:01
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ No material benchmark regressions detected

🟢 4 better · 🔴 0 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main 82224ea benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🟢 bs=10 sw=10 sl=64 566 0.345 16,410/24,664/24,664 us 🟢 -13.1% / 🔴 +79.7%
🟢 bs=100 sw=10 sl=64 1,254 0.765 80,892/86,471/86,471 us 🟢 -16.6% / 🟢 -22.8%
⚪ bs=1000 sw=10 sl=64 1,450 0.885 691,668/734,607/734,607 us ⚪ within ±5% / 🟢 -15.8%
Baseline details

Latest main 82224ea from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 566 tuples/sec 548 tuples/sec 1,005 tuples/sec +3.3% -43.7%
bs=10 sw=10 sl=64 MB/s 0.345 MB/s 0.334 MB/s 0.614 MB/s +3.3% -43.8%
bs=10 sw=10 sl=64 p50 16,410 us 16,180 us 10,325 us +1.4% +58.9%
bs=10 sw=10 sl=64 p95 24,664 us 28,374 us 13,728 us -13.1% +79.7%
bs=10 sw=10 sl=64 p99 24,664 us 28,374 us 16,323 us -13.1% +51.1%
bs=100 sw=10 sl=64 throughput 1,254 tuples/sec 1,218 tuples/sec 1,347 tuples/sec +3.0% -6.9%
bs=100 sw=10 sl=64 MB/s 0.765 MB/s 0.743 MB/s 0.822 MB/s +3.0% -7.0%
bs=100 sw=10 sl=64 p50 80,892 us 79,186 us 80,846 us +2.2% +0.1%
bs=100 sw=10 sl=64 p95 86,471 us 103,668 us 88,523 us -16.6% -2.3%
bs=100 sw=10 sl=64 p99 86,471 us 103,668 us 112,010 us -16.6% -22.8%
bs=1000 sw=10 sl=64 throughput 1,450 tuples/sec 1,430 tuples/sec 1,394 tuples/sec +1.4% +4.1%
bs=1000 sw=10 sl=64 MB/s 0.885 MB/s 0.873 MB/s 0.851 MB/s +1.4% +4.0%
bs=1000 sw=10 sl=64 p50 691,668 us 705,188 us 787,853 us -1.9% -12.2%
bs=1000 sw=10 sl=64 p95 734,607 us 737,836 us 826,583 us -0.4% -11.1%
bs=1000 sw=10 sl=64 p99 734,607 us 737,836 us 872,816 us -0.4% -15.8%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,353.63,200,128000,566,0.345,16409.74,24664.32,24664.32
1,100,10,64,20,1595.48,2000,1280000,1254,0.765,80892.11,86471.06,86471.06
2,1000,10,64,20,13792.85,20000,12800000,1450,0.885,691667.54,734606.60,734606.60

@kunwp1 kunwp1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments.

- Drop the `@SerialVersionUID` pins on `Schema`, `Tuple`,
  `Tuple.Builder` and `MachineLearningScorerOpDesc`: Pekko routes
  `Serializable` through Kryo, which ignores them.
- `Schema.add` and `remove` read the lowercase names from
  `attributeIndex.keySet`, so `lowerCaseAttributeNames` is gone.
- `Tuple.hashCode` and `equals` read the fields in place instead of
  cloning or copying them. `Builder.add(tuple)`, `addSequentially` and
  `SeqTupleLike.enforceSchema` walk the attributes with one iterator
  and no field copy.
- `ArrowUtils.getTexeraTuple` takes each attribute from the iterator,
  since the root's schema has one attribute per vector.
- `MachineLearningScorerOpDesc` keeps its scaladoc on
  `validateScoredColumns`, filters the per-class metrics only for
  timestamps, and builds the output schema with the `Schema.add` fold.
- Name the Java UDF injector case after what it checks.
@aglinxinyuan
aglinxinyuan requested a review from kunwp1 November 17, 2026 07:39

@kunwp1 kunwp1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM~!!

Pin `ArrowSourceOpExec` reading several record batches (row order, the
file's schema on every row, offset and limit spanning batches) and a
file with none, and `PythonProxyServer` turning an empty Data put into
an empty `DataFrame`, before each root's schema is converted once.
`ArrowUtils.getTexeraTuple(rowIndex, root)` converted the root's Arrow
schema on every call, so `PythonProxyServer` and `ArrowSourceOpExec`
rebuilt the same Texera `Schema` for every row they read. A
`getTexeraTuple(rowIndex, root, schema)` overload takes the converted
schema: `PythonProxyServer` converts once per Data put and
`ArrowSourceOpExec` once per file, on its first row. The two-argument
form converts and delegates.
@aglinxinyuan aglinxinyuan changed the title refactor(common): simplify tuple, Arrow, and operator allocs refactor(amber, common): simplify tuple, Arrow, operator allocs Nov 17, 2026
`(left, right) match` still built a `Tuple2` for every field
`fieldEquals` compared, and `(0 until n).forall` built a `Range` and a
capturing lambda on every `equals` call (seen with `javap`). Distinct
and the set operators call `equals` per row, so it now walks the
fields with an index loop and plain type tests.
@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Nov 17, 2026
Merged via the queue into apache:main with commit 55e179a Nov 17, 2026
36 checks passed
@aglinxinyuan
aglinxinyuan deleted the refactor/common-tuple-operator-allocs branch November 17, 2026 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Convert an Arrow batch's schema once instead of once per row Simplify tuple, Arrow, UDF parameter, and text operator allocations

4 participants