Repository navigation
fix(cubejs): stream csv/arrow exports from the source db, not Cube Store - #98
Merged
Merged
Conversation
Cube's query queue refuses to stream a query whose compiled plan targets
Cube Store ("Streaming queries to Cube Store aren't supported"), so
arrow/csv /load requests that hit an external rollup failed with a 500.
The plan already flags pre-aggregations to skip the native ClickHouse
path, but still sent them to semantic streaming. When the compiled query
is external, fall through to the buffered /load pipeline instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the buffered fallback: that path cut Arrow at 100k rows, skipped the Dataset Exported audit and held the whole result in memory. Compile the export with disableExternalPreAggregations, as Cube's own SQL API does for streaming, so the query never targets Cube Store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
doktorhelgi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(cubejs): stream csv/arrow exports from the source db, not Cube Store
Problem
cxs2's dashboard proxy (
/api/data-viz-demo/cube→dbx.fraios.dev/api/v1/load, arrow format) got HTTP 500Error: Streaming queries to Cube Store aren't supported(4× on 2026-10-14, cuberoad_segments_driven). The export handler compiles the query normally, so a matching Cube Store rollup makes itexternal, and Cube's queue refuses to stream external queries.Fix
buildLoadExportPlancompiles the export withdisableExternalPreAggregations: true, the same thing Cube's own SQL API does before streaming (gateway.jssqlApiLoad). The query then runs against the source database (ClickHouse), and pre-aggregations stored there still apply.The first version of this PR sent these queries to the buffered
/loadpath instead. A review found that path cuts Arrow at 100k rows without saying so, skips theDataset Exportedaudit and holds the whole result in memory, so it was replaced.Trade-off: an export that a Cube Store rollup could have served now scans the source tables. That is slower, but the result is complete, streamed and audited.
Tests
compiles without Cube Store pre-aggregations so the export can stream. It checks the compile flag and that the export streams. It fails on main and passes with the fix.node --experimental-test-module-mocks --test test/loadExport.test.js test/loadExportCapabilities.test.js→ 13 pass, 0 fail.🤖 Generated with Claude Code