Repository navigation
Conversation
Create reusable nested macros from selected operators, pin their definition versions, expose ports and properties, and expand them before compilation, execution, and paused reconfiguration. Bundle dependencies for portable JSON exports and keep their graphs intact when cloning workflows. Closes apache#8913 Discussion: apache#5248 (comment) Validation: - node node_modules/@angular/cli/bin/ng.js test --watch=false --progress=false (6608 passed, 1 skipped) - sbt "WorkflowExecutionService/testOnly org.apache.texera.web.resource.dashboard.user.workflow.WorkflowMacroCloneSpec" (2 passed) - node .yarn/releases/yarn-4.14.1.cjs build - node .yarn/releases/yarn-4.14.1.cjs format:ci - sbt "scalafixAll --check" - sbt scalafmtCheckAll - Playwright browser checks with mocked API: selection collapse, exposed properties, nested inspection/back navigation, and independent editable copy.
Preserve pinned definitions through agent edits and reuse graph expansion, validation and logical plans. Normalize portable bundles and reuse palette controls and search, keeping deep macro inspection responsive. Related to apache#8913
Resolve each occurrence in the compiler, preserve nested revision snapshots and portable imports, and expose run statistics consistently in the canvas, Form View and agent workflows. Closes apache#8913
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is
Additional details and impacted files
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 490 | 0.299 | 19,987/24,682/24,682 us | 🟢 -22.6% / 🔴 +109.3% |
| 🟢 | bs=100 sw=10 sl=64 | 1,019 | 0.622 | 97,012/106,127/106,127 us | 🟢 -12.8% / 🔴 -30.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,130 | 0.689 | 879,954/979,140/979,140 us | ⚪ within ±5% / 🔴 +28.1% |
Baseline details
Latest main f020980 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 490 tuples/sec | 520 tuples/sec | 1,084 tuples/sec | -5.8% | -54.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.299 MB/s | 0.317 MB/s | 0.662 MB/s | -5.7% | -54.8% |
| bs=10 sw=10 sl=64 | p50 | 19,987 us | 17,682 us | 9,548 us | +13.0% | +109.3% |
| bs=10 sw=10 sl=64 | p95 | 24,682 us | 31,901 us | 12,811 us | -22.6% | +92.7% |
| bs=10 sw=10 sl=64 | p99 | 24,682 us | 31,901 us | 15,432 us | -22.6% | +59.9% |
| bs=100 sw=10 sl=64 | throughput | 1,019 tuples/sec | 1,036 tuples/sec | 1,467 tuples/sec | -1.6% | -30.5% |
| bs=100 sw=10 sl=64 | MB/s | 0.622 MB/s | 0.632 MB/s | 0.895 MB/s | -1.6% | -30.5% |
| bs=100 sw=10 sl=64 | p50 | 97,012 us | 93,453 us | 74,441 us | +3.8% | +30.3% |
| bs=100 sw=10 sl=64 | p95 | 106,127 us | 121,763 us | 81,800 us | -12.8% | +29.7% |
| bs=100 sw=10 sl=64 | p99 | 106,127 us | 121,763 us | 103,712 us | -12.8% | +2.3% |
| bs=1000 sw=10 sl=64 | throughput | 1,130 tuples/sec | 1,145 tuples/sec | 1,517 tuples/sec | -1.3% | -25.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.689 MB/s | 0.699 MB/s | 0.926 MB/s | -1.4% | -25.6% |
| bs=1000 sw=10 sl=64 | p50 | 879,954 us | 858,710 us | 725,911 us | +2.5% | +21.2% |
| bs=1000 sw=10 sl=64 | p95 | 979,140 us | 1,006,526 us | 764,152 us | -2.7% | +28.1% |
| bs=1000 sw=10 sl=64 | p99 | 979,140 us | 1,006,526 us | 808,901 us | -2.7% | +21.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,407.98,200,128000,490,0.299,19987.22,24681.65,24681.65
1,100,10,64,20,1962.86,2000,1280000,1019,0.622,97012.19,106126.86,106126.86
2,1000,10,64,20,17706.28,20000,12800000,1130,0.689,879953.57,979139.93,979139.93
What changes were proposed in this PR?
Add versioned workflow macros. Users can select operators on the canvas, expose inputs, outputs and Form View fields, and reuse the resulting macro in another workflow. Breadcrumbs open a nested occurrence on the same canvas.
Each occurrence pins a saved revision. Saving an edit through a parent updates that path. Other usages and earlier workflow snapshots keep their pins. Export includes the transitive macro definitions and revisions needed to reopen the graph on another deployment.
The Scala compiler resolves the pins, validates exposed properties, and removes macro boundaries before submitting a native graph to Amber. It retains the occurrence paths used to display errors, schemas, results and live or historical statistics. Agent edits use the same compiler validation.
Draft status. Merged and tested against
apache:mainatf020980c3e. The PR remains open for design feedback.The nested canvas shows explicit graph boundaries. These screenshots compare an earlier iteration and the current terminal icons.
Any related issues, documentation, discussions?
Closes #8913. The draft moves expansion into the compiler, updating the frontend-expansion proposal recorded in that issue. The lifecycle and portability questions are open for community feedback. Design discussion · Xiaozhen’s original UI demo
How was this PR tested?
On merge commit
aa6bee1c38, 419 Scala tests and 363 agent tests passed. The full Angular suite passed 7,659 tests with one skip. Chromium passed 28 standard browser tests and all 187 tests in the focused macro and result suites. All 22 compiler cases matched their original input, expected output and error trees.A real Amber execution ran five occurrences of the same macro at three depths, processing 10, 20, 30, 40 and 50 rows. The test reopened the saved execution from Iceberg and checked the macro, inner Split, and both terminals against their respective live per-port counts. Other regressions cover nested saves, Undo/Redo and Canvas/Form View import/export.
To try it in the UI, select connected operators, create a macro, then drag a second instance onto the canvas. Run the workflow and open each occurrence using its breadcrumb path. Reopen the saved execution to compare the same per-port counts.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)