Repository navigation
refactor(amber, common): simplify tuple, Arrow, operator allocs - #9515
Merged
aglinxinyuan merged 8 commits intoNov 17, 2026
Merged
aglinxinyuan merged 8 commits into
aglinxinyuan merged 8 commits into
Conversation
Add characterization tests in zero-overlap common/workflow-core and common/workflow-operator specs before allocation and helper cleanups.
Combine verified BA-952 (4-file core), BA-946, BA-621, and BA-953 allocation and traversal cleanups plus shared helpers in Schema and RUdfUiParameterInjector.
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9515 +/- ##
=========================================
Coverage 91.60% 91.60%
- Complexity 5043 5055 +12
=========================================
Files 1284 1284
Lines 57020 57081 +61
Branches 7414 7430 +16
=========================================
+ Hits 52232 52288 +56
Misses 2915 2915
- Partials 1873 1878 +5
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
✅ No material benchmark regressions detected🟢 4 better · 🔴 0 worse · ⚪ 11 noise (<±5%) · 0 without baseline
Baseline detailsLatest main
Raw CSVconfig_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,353.63,200,128000,566,0.345,16409.74,24664.32,24664.32
1,100,10,64,20,1595.48,2000,1280000,1254,0.765,80892.11,86471.06,86471.06
2,1000,10,64,20,13792.85,20000,12800000,1450,0.885,691667.54,734606.60,734606.60 |
kunwp1
reviewed
Nov 17, 2026
- Drop the `@SerialVersionUID` pins on `Schema`, `Tuple`, `Tuple.Builder` and `MachineLearningScorerOpDesc`: Pekko routes `Serializable` through Kryo, which ignores them. - `Schema.add` and `remove` read the lowercase names from `attributeIndex.keySet`, so `lowerCaseAttributeNames` is gone. - `Tuple.hashCode` and `equals` read the fields in place instead of cloning or copying them. `Builder.add(tuple)`, `addSequentially` and `SeqTupleLike.enforceSchema` walk the attributes with one iterator and no field copy. - `ArrowUtils.getTexeraTuple` takes each attribute from the iterator, since the root's schema has one attribute per vector. - `MachineLearningScorerOpDesc` keeps its scaladoc on `validateScoredColumns`, filters the per-class metrics only for timestamps, and builds the output schema with the `Schema.add` fold. - Name the Java UDF injector case after what it checks.
kunwp1
approved these changes
Nov 17, 2026
Pin `ArrowSourceOpExec` reading several record batches (row order, the file's schema on every row, offset and limit spanning batches) and a file with none, and `PythonProxyServer` turning an empty Data put into an empty `DataFrame`, before each root's schema is converted once.
`ArrowUtils.getTexeraTuple(rowIndex, root)` converted the root's Arrow schema on every call, so `PythonProxyServer` and `ArrowSourceOpExec` rebuilt the same Texera `Schema` for every row they read. A `getTexeraTuple(rowIndex, root, schema)` overload takes the converted schema: `PythonProxyServer` converts once per Data put and `ArrowSourceOpExec` once per file, on its first row. The two-argument form converts and delegates.
2 tasks done
`(left, right) match` still built a `Tuple2` for every field `fieldEquals` compared, and `(0 until n).forall` built a `Range` and a capturing lambda on every `equals` call (seen with `javap`). Distinct and the set operators call `equals` per row, so it now walks the fields with an index loop and plain type tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Simplifies transient collection allocations, repeated traversals, and redundant metadata/regex builds across
common/workflow-core,common/workflow-operator, andamber(+316/-238 production lines; +611/-249 total including characterization tests), with no change to observable behavior or generated code:Schemaadd/remove names against the lowercaseattributeIndexkeys, hashes and comparesTuplefields in place, walks attributes with one iterator inTuple.BuilderandSeqTupleLike.enforceSchemainstead of zipping or copying arrays, and populates Arrow field arrays and schemas without extra temporary collections inArrowUtils. AgetTexeraTuple(rowIndex, vectorSchemaRoot, schema)overload takes an already converted schema, soPythonProxyServerconverts a root's Arrow schema once per Data put andArrowSourceOpExeconce per file instead of once per row.UiUdfParameterContract,UdfUiParameterSupport), replaces incrementalString.patchedits and duplicate AST field scans inJavaUdfUiParameterInjector, avoids line splitting and pre-compiles the reserved hook regex inPythonUdfUiParameterInjector, and formats sorted manifest entries viaHexFormatand a sharedformatEntrieshelper inRUdfUiParameterInjector.DictionaryMatcherOpDesc,DictionaryMatcherOpExec), removes intermediate wrappers inCaseSensitiveAnalyzer,KeywordSearchOpDesc, andKeywordSearchOpExec, avoids leading regex flag list allocations inRegexOpDesc, and streamlines column validation inMachineLearningScorerOpDesc.Any related issues, documentation, discussions?
Closes #9514.
Closes #9521.
How was this PR tested?
Characterization tests came first (
a347f3d946,1370fab8c2), then the cleanups and helper extractions (eeb039181d,9efd195ade,bcaa8684e5,73d6f66b1e). On finalHEAD(73d6f66b1e):scalafmtCheckand 6scalafix --checktargets passed.WorkflowCorenon-Docker suites (-l org.apache.texera.common.tags.NonParallelTest): 54 suites completed, 712 succeeded, 0 failed, 1 pre-existing pending (PortIdentityKeyDeserializerSpec'spendingUntilFixedcase).WorkflowOperatoraffected packages: 29 suites completed, 412 succeeded, 0 failed (including live Python 3.12 pandas/sklearn scoring and regex checks).WorkflowExecutionServicetuple/storage and Python proxy suites: 9 suites completed, 92 succeeded, 0 failed.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Antigravity IDE (Gemini 3.8 Flash), Claude Code (Claude Opus 5.5)