Repository navigation
feat(visualization): export the charts drawn over a field or a sequence - #8504
Conversation
Ten charts implement the trait: Filled Area, Contour, Dumbbell, Time Series, Quiver, Choropleth Map, 3D Scatter, Volcano, Gantt and Carpet. What they share is the question they answer, which is how a value varies across a domain rather than where a row sits on an axis. Split out of the coordinate-system change on review, which had grown past what one reading can hold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (17.02%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #8504 +/- ##
============================================
- Coverage 92.60% 92.40% -0.21%
- Complexity 5046 5047 +1
============================================
Files 1258 1258
Lines 53663 53794 +131
Branches 6683 6698 +15
============================================
+ Hits 49697 49706 +9
- Misses 2312 2430 +118
- Partials 1654 1658 +4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 579 | 0.354 | 17,059/21,010/21,010 us | 🟢 -7.7% / 🔴 +40.6% |
| 🔴 | bs=100 sw=10 sl=64 | 1,153 | 0.704 | 84,646/119,034/119,034 us | 🔴 +27.5% / 🔴 +14.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,347 | 0.822 | 741,676/809,234/809,234 us | ⚪ within ±5% / 🟢 +22.6% |
Baseline details
Latest main 4184858 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 579 tuples/sec | 597 tuples/sec | 822.11 tuples/sec | -3.0% | -29.6% |
| bs=10 sw=10 sl=64 | MB/s | 0.354 MB/s | 0.364 MB/s | 0.502 MB/s | -2.7% | -29.5% |
| bs=10 sw=10 sl=64 | p50 | 17,059 us | 16,699 us | 12,129 us | +2.2% | +40.6% |
| bs=10 sw=10 sl=64 | p95 | 21,010 us | 22,755 us | 15,469 us | -7.7% | +35.8% |
| bs=10 sw=10 sl=64 | p99 | 21,010 us | 22,755 us | 18,357 us | -7.7% | +14.4% |
| bs=100 sw=10 sl=64 | throughput | 1,153 tuples/sec | 1,206 tuples/sec | 1,058 tuples/sec | -4.4% | +9.0% |
| bs=100 sw=10 sl=64 | MB/s | 0.704 MB/s | 0.736 MB/s | 0.646 MB/s | -4.3% | +9.0% |
| bs=100 sw=10 sl=64 | p50 | 84,646 us | 81,791 us | 97,399 us | +3.5% | -13.1% |
| bs=100 sw=10 sl=64 | p95 | 119,034 us | 93,328 us | 104,184 us | +27.5% | +14.3% |
| bs=100 sw=10 sl=64 | p99 | 119,034 us | 93,328 us | 117,615 us | +27.5% | +1.2% |
| bs=1000 sw=10 sl=64 | throughput | 1,347 tuples/sec | 1,331 tuples/sec | 1,099 tuples/sec | +1.2% | +22.6% |
| bs=1000 sw=10 sl=64 | MB/s | 0.822 MB/s | 0.812 MB/s | 0.671 MB/s | +1.2% | +22.6% |
| bs=1000 sw=10 sl=64 | p50 | 741,676 us | 745,248 us | 946,743 us | -0.5% | -21.7% |
| bs=1000 sw=10 sl=64 | p95 | 809,234 us | 836,653 us | 985,437 us | -3.3% | -17.9% |
| bs=1000 sw=10 sl=64 | p99 | 809,234 us | 836,653 us | 1,012,622 us | -3.3% | -20.1% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,345.15,200,128000,579,0.354,17059.08,21009.51,21009.51
1,100,10,64,20,1734.69,2000,1280000,1153,0.704,84646.41,119033.93,119033.93
2,1000,10,64,20,14851.35,20000,12800000,1347,0.822,741675.78,809233.54,809233.54
carloea2
left a comment
There was a problem hiding this comment.
The chart outputs use separate files, and contour errors no longer stop the workflow. Looks good.
carloea2
left a comment
There was a problem hiding this comment.
Filled Area calculates tolerance differently in standalone mode. The native path uses len times 5 divided by 100. The standalone path divides len by 100 first. For 150 groups this gives 7 versus 5. Please use the same formula and add a boundary test.
carloea2
left a comment
There was a problem hiding this comment.
Carpet Plot still differs on bad input. The native path returns an error page for missing columns or values that cannot be converted to numbers. Standalone calls dropna and astype first and crashes. Please keep the same error behavior and add both tests.
The exported block for the filled area plot rounded its tolerance the other way round: five percent of the groups is (n * 5) // 100, not (n // 100) * 5, and at 150 groups that is 7 rather than 5. The script refused three tables the operator draws. The loop around it is now the operator's own, down to the break that ends it. The carpet plot dropped nulls before it looked for the columns to drop them from, so a column that is not in the table reached pandas as a KeyError and a value that is not a number reached astype as a ValueError. Both ended the whole exported run, where the operator answers each of them with a page saying what is wrong. The block now checks in the operator's order and writes the same pages. Both are checked by running the exported code rather than by reading it: the filled area plot at the boundary the tolerance sets, 7 disjoint groups of 150 drawn and 8 refused, and the carpet plot over a missing column, a column of words, an empty table, a table of nulls, and a table it can draw. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Both are real and both are fixed: Filled Area now uses the operator's |
carloea2
left a comment
There was a problem hiding this comment.
The Filled Area and Carpet changes address the earlier result and error handling differences. Looks good.
…-integration Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
carloea2
left a comment
There was a problem hiding this comment.
The current delta only tags the Python tests; the chart implementations are unchanged from the previous review. I retested with the shared helpers from merged #8327, #8502, #8340 and #8344 supplied locally.
Command: sbt -batch "WorkflowOperator/testOnly *ContourPlotOpDescSpec *FilledAreaPlotOpDescSpec *CarpetPlotOpDescSpec -- -oDF" "scalafmtCheckAll". On Windows, 40 tests pass and two fail while reading the generated HTML. With PYTHONUTF8=1, all 42 tests and formatting pass.
The unmodified branch also fails compilation in CI on those missing shared helpers. Please refresh it from main before rerunning CI.
The module and the Plotly page are both UTF-8, but the Carpet Plot and Filled Area drivers read them in the machine's default encoding, so on Windows the page failed with a UnicodeDecodeError at byte 0x8f. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
left a comment
There was a problem hiding this comment.
The encoding fix passes on Windows. With current upstream main merged locally, all 99 runnable tests across 11 suites and formatting checks pass. The one existing pending test is unchanged. The remaining CI failure occurs while downloading sbt, before compilation. Code looks good; CI still needs a successful run. Leaving the draft unchanged.
…pache#8346) ### What changes were proposed in this PR? Eight visualizations implement `StandaloneCodeGenerator`: the polar family (polar, radar in its two forms, wind rose), the ternary pair, and the gauge and bullet indicators. What they share is the question they answer, which is where on an angle, a radius or a simplex a row lands. The ten drawn over a field or a sequence are in apache#8504. With those, apache#8344 and this one, every visualization Texera offers can be exported. Each is compared as a figure rather than as a frame, after both paths have drawn. A plot's meaning is in the numbers behind it, so that is what the comparison reads; comparing the rendered picture instead would fail over a layout detail that carries no meaning. ### Any related issues, documentation, discussions? Part of apache#8325, 16 of 27; that issue lists the set in order. It needs apache#8327 for the trait, so it does not compile until that lands, and the rows these operators add to the verification runner follow with the harness rather than as whole new files here. Closes apache#8419, the task this change is the whole of. ### How was this PR tested? Each operator asserts the block it emits in its own spec. Once the verification lands it is also run through the engine and through its generated script, on every configuration its schema offers, and the two answers compared; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ython (apache#9154) ### What changes were proposed in this PR? The Python that "Export to Python" writes carried notes meant for whoever maintains the generator: why a frame is copied, why a row is dropped before scoring, why an empty cell keeps its row. Someone running the exported script does not need them, and they made the script longer to read. This PR takes those comments out of the exported code in 24 operators, and keeps a one or two line note in Scala above the string where the reason is worth keeping. - Machine learning: the sklearn estimators' shared training, testing and fit-and-score code, Linear Regression, and the Machine Learning Scorer. - Hugging Face: Iris Logistic Regression, Sentiment Analysis, Spam SMS Detection, Text Summarization. - Visualization: Bar, Box/Violin, Bubble, Bullet, Choropleth, Contour, Dendrogram, Gauge, Hierarchy, Histogram 2D, Icicle, Radar, Tree Plot, URL Visualizer. - Unnest String. Only comments go. The translator's own markers stay: the `# [Operator]` heading over each section, `# --- Output ---`, and the `# TODO` it writes for an operator it cannot export yet. The Python that operators run inside the engine is not touched. ### Any related issues, documentation, discussions? The comments came in with the export PRs apache#8365, apache#8367, apache#8368, apache#8504 and apache#8340. apache#9035 also changes Unnest String's export. Whichever of the two merges second keeps the other's change. ### How was this PR tested? No generated statement changes, only comment lines. Running the verification suite before and after, the 1,098 scripts it generates for 142 operators are identical once comment lines and temporary paths are set aside, and none of them has an explanatory comment left. All of `WorkflowOperator/test` passes (2,912 tests), as do `scalafmtCheck` and `scalafixAll --check`. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) ### What changes were proposed in this PR? Root cause: `VolcanoPlotOpDesc.generatePythonCode` hard-codes the y column as the literal `"-log10(pvalue)"` and plots it under that name, so the y axis always says `-log10(pvalue)` even when the selected column is `padj`. The title is the literal `"Volcano Plot"` and the x axis and color bar show the raw effect column name. None of this can be changed from the property panel. ``` Before: P-Value Column = padj -> y column "-log10(pvalue)" -> y axis "-log10(pvalue)" After: P-Value Column = padj -> y column "-log10(padj)" -> y axis "-log10(padj)", or the Y Label the user typed ``` The y column is now named after the selected p-value column, so the axis and the hover text say which column was transformed. Three optional label properties are added, empty by default: | Property | Empty | Set | |---|---|---| | Title | `Volcano Plot` | the text typed | | X Label | effect column name | the text typed | | Y Label | `-log10(<p-value column>)` | the text typed | The color bar has no label of its own: the points are colored by the effect column, the same column that is on the x axis, so the color bar carries the X Label text (or the effect column name when X Label is empty). If a later change lets the color encode a different column, a label for it would come with that change. The labels go through `fig.update_layout(xaxis_title, yaxis_title, coloraxis_colorbar_title_text)`, same as Line Chart's X Label / Y Label. Each label is an `EncodableString`, so quotes and non-ASCII text are fine. A whitespace-only field counts as empty. The hover box still shows raw column names (plotly behaviour, same as Line Chart). Behavior change for saved workflows: a workflow with `padj` as the p-value column now renders `-log10(padj)` instead of `-log10(pvalue)`. That is the fix. Since apache#8504 merged, the operator also has an export path, `generateStandaloneCode()`, which writes the same figure as a standalone script. It carried the same hard-coded `"-log10(pvalue)"` and `title="Volcano Plot"`, so this PR applies the same change there: the y column is named after the selected p-value column and the three labels go through `fig.update_layout` as escaped Python literals (the export has no decoder, so `pyStringLiteral` escapes quotes and backslashes). The engine template and the exported script now label the figure the same way. Rebase note: `coloraxis_colorbar_title_text` is emitted unconditionally, in both templates. Once the PR for the significance coloring (apache#9039) lands, the categorical mode has no `coloraxis`, so whichever branch rebases second needs to make that entry conditional. Figures below are the operator's generated Python from `main` and from this branch, run on the 430-gene example workflow from the issue; the third one has the title and both axis labels filled in. | Before | After (defaults) | After (labels set) | |---|---|---| | <img width="1398" height="849" alt="before-main" src="https://github.com/user-attachments/assets/0c0dd5ef-3675-44e2-98f6-236b54bd631b" /> | <img width="1398" height="849" alt="labels-after-default" src="https://github.com/user-attachments/assets/ee8dc430-2142-47ac-947f-59c535c2b308" /> | <img width="1398" height="849" alt="labels-after-custom" src="https://github.com/user-attachments/assets/8299992d-d84d-48fe-9556-e66a8579e859" /> | ### Any related issues, documentation, discussions? Closes apache#9037 PR apache#8504 (merged) added the `generateStandaloneCode()` export path with the same hard-coded `"-log10(pvalue)"`; the mirroring described above is the follow-up it needed. ### How was this PR tested? New tests in `VolcanoPlotOpDescSpec`, written before the source change: - y column and axis are named after the selected p-value column; the old literal is gone - every label falls back to its default when empty or whitespace-only - a set label replaces its default and reaches the code only as a decode site (sentinels with quotes, backslash, non-ASCII) - the color bar title follows X Label when it is set and the effect column name otherwise - one set label leaves the other two at their defaults - a workflow JSON saved before this change (only the two column keys) generates byte-for-byte the same code as a default-constructed operator - `generateStandaloneCode`: the y column and axis are named after the p-value column (including a name with a quote, `p"adj`), the labels fall back to the column-derived defaults, a set label is emitted as an escaped literal and the default title is gone, the color bar carries the X Label text, and whitespace-only keeps the default ``` sbt -Dsbt.server.forcestart=true "WorkflowOperator/testOnly *VolcanoPlotOpDescSpec" # Tests: succeeded 15, failed 0, canceled 0, ignored 0, pending 1 (the pending one is pre-existing) sbt -Dsbt.server.forcestart=true "WorkflowOperator/scalafmtAll" "WorkflowOperator/Test/scalafmtAll" sbt -Dsbt.server.forcestart=true "WorkflowOperator/scalafixAll --check" # success ``` Smoke run: the generated code was executed outside Texera (pytexera stubbed, pandas + plotly 5.24.1) on the 430-gene mock table. Defaults give title `Volcano Plot`, x `lfc_shrunk`, y `-log10(padj)`, color bar `lfc_shrunk`; with the labels set the figure shows the typed text; whitespace-only title and y label fall back to the defaults. The exported script for the default and the all-labels configuration parses with `ast.parse`. ### Was this PR authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code (claude-fable-5-1). Co-authored with Claude; the author reviewed the change before submission.
### What changes were proposed in this PR? The Volcano Plot uses `hover_name=table.columns[0]`, so the tooltip label is whatever column comes first in the input. That is fine for a DESeq2 table starting with `gene` and useless for one starting with a composite id or a number, and there is no way to change it. This PR copies Scatter Plot's `Hover column` property over, one column only. | Hover column | Result | |---|---| | empty or whitespace-only (default) | `hover_name=table.columns[0]`, unchanged for saved workflows | | set, present in the input | `hover_name=<that column>` | | set, missing from the input | the operator's error page `Missing hover column in table.`, like the existing check for the two required columns (Scatter Plot lets plotly raise a `ValueError` here) | Since apache#8504 merged, the operator also exports a standalone script through `generateStandaloneCode()`, which had `hover_name=table.columns[0]` hard-coded. This PR mirrors the property there: the hover column goes in as an escaped Python literal, and a set column that the input lacks adds one more `elif` to the script's column checks, writing the same error page. Whether the field counts as empty is decided by one helper for both paths, so the exported figure and the engine's cannot disagree. The label fix (apache#9037) has landed and this branch is merged with it. The hover column is independent of it: the merged operator differs from `main` only by the hover property, its checks and the two `hover_name` arguments. The PR for the significance thresholds (apache#9039) touches neighbouring lines of `VolcanoPlotOpDesc.scala`, so whichever of the two lands second needs a small merge. Figures below are the operator's generated Python from `main` and from this branch (Hover column = `gene`), run on the 430-gene example workflow from the issue whose first column is `transcript|gene|chr|strand`; the same point is hovered in both. | Before | After (Hover column = gene) | |---|---| | <img width="1398" height="849" alt="hover-before-first-column" src="https://github.com/user-attachments/assets/57df5276-196c-45b4-946d-098d77f739cb" /> | <img width="1398" height="849" alt="hover-after-gene" src="https://github.com/user-attachments/assets/2aca08f2-e817-4ea2-940f-ce90eeb447e4" /> | ### Any related issues, documentation, discussions? Closes apache#9038 PR apache#8504 (merged) added the `generateStandaloneCode()` export path with the same `hover_name=table.columns[0]`; the mirroring described above is the follow-up it needed. ### How was this PR tested? New tests in `VolcanoPlotOpDescSpec`, written before the source change: - empty or whitespace-only field keeps `hover_name=table.columns[0]` and emits no hover check - a set field emits the decoded column as `hover_name` plus the presence check, and no `table.columns[0]` - a column name with quotes and non-ASCII text reaches the code only as a decode site - a workflow JSON saved before this change generates byte-for-byte the same code as a default-constructed operator - `generateStandaloneCode`: an empty field keeps `hover_name=table.columns[0]` and adds no check; a set field is emitted as a literal plus an exact five-line `elif` branch between the required-column check and the drawing branch; a name with quotes, non-ASCII text or a backslash comes out escaped rather than decoded; whitespace-only equals empty ``` sbt -Dsbt.server.forcestart=true "WorkflowOperator/testOnly *VolcanoPlotOpDescSpec" # Tests: succeeded 24, failed 0, canceled 0, ignored 0, pending 1 (the pending one is pre-existing; the count includes the label tests now on main) sbt -Dsbt.server.forcestart=true "WorkflowOperator/scalafmtAll" "WorkflowOperator/Test/scalafmtAll" sbt -Dsbt.server.forcestart=true "WorkflowOperator/scalafixAll --check" # success ``` Smoke run outside Texera (pytexera stubbed, pandas + plotly 5.24.1) on the 430-gene mock table: defaults hover-label the first point as `ENST00000000000|GENE_0000|chr2|+`; Hover column = `gene` gives `GENE_0000`; Hover column = `no_such_column` yields `<h1>Volcano Plot failed</h1><p>Missing hover column in table.</p>`. The exported script for the default and the `gene` configuration parses with `ast.parse`. ### Was this PR authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code (claude-fable-5-1). Co-authored with Claude; the author reviewed the change before submission.
What changes were proposed in this PR?
Ten charts implement
StandaloneCodeGenerator: Filled Area, Contour, Dumbbell, Time Series, Quiver, Choropleth Map, 3D Scatter, Volcano, Gantt and Carpet. What they share is the question they answer, which is how a value varies across a domain rather than where a row sits on an axis.Each emits the plotly its executor emits and writes the page to the file the translator names for it, so a plan holding two charts writes two files rather than one overwriting the other. A chart that cannot be drawn writes the reason to that page rather than printing it: the engine's operator yields a page either way, so an export that printed instead would leave the run with no file where the workflow had one.
#8346 keeps the charts drawn on a coordinate system.
Any related issues, documentation, discussions?
Part of #8325, 17 of 27; that issue lists the set in order.
Two behaviour fixes travel with these charts, both on operators that moved here: the Filled Area Plot requires its line group once the switch that reads it is on, and the Contour Plot says what it cannot contour instead of ending the run. Closes #8283, closes #8076.
Closes #8503, the task this change is the whole of.
How was this PR tested?
Each chart asserts the block it emits in its own spec. Contour, Filled Area and Carpet additionally run the generated Python, since what they are worth checking is that the page comes out at all on the inputs that cannot be drawn: collinear points, a group with no line to draw, a table emptied by dropping nulls.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
🤖 Generated with Claude Code