Repository navigation
fix(cubejs): BigQuery read-only rollups typed from the result schema - #90
Merged
Merged
Conversation
In read-only mode Cube 1.7.30's BigQuery driver falls back to the base
downloadQueryResults, which guesses column types from the rows. A decimal
after whole numbers then fails the Cube Store load ("Can't parse int"),
and an empty partition cannot be typed at all. This is why read-only
regressed abler-web-prod on 2026-10-07 and was reverted (#89).
A postinstall patch gives the driver its own downloadQueryResults: run
the query job, then read the column types from BigQuery's result schema
(getQueryResults with maxResults 0), mapped exactly as tableColumnTypes
maps them for the staging path. Neither 1.7.50 nor 1.8.1 fixes this
upstream.
Tested on 84 real Abler partitions in-cluster: all 69 that failed in
read-only now build, and 27 prod partitions are identical (row count and
every numeric sum) to their staging-built Cube Store tables.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This reverts commit 7792062, now that read-only downloads take their column types from BigQuery's result schema. Rollups load straight into Cube Store; nothing is written to the customer's BigQuery project. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The first read-only attempt (#88, reverted in #89) regressed abler-web-prod. In read-only mode, Cube 1.7.30's BigQuery driver falls back to the base
downloadQueryResults, which guesses column types from the rows:Can't parse int).Neither Cube 1.7.50 nor 1.8.1 changes this, so upgrading would not help.
What
BigQueryDriver.downloadQueryResults. It runs the query job, then reads the column types from BigQuery's result schema (getQueryResults({maxResults: 0, autoPaginate: false})), mapped exactly astableColumnTypesmaps them for the staging path. A drift guard refuses to apply it to changed Cube source.readOnly: truefor BigQuery datasources (reverts Revert BigQuery read-only rollups (regressed abler-web-prod) #89).Tests
Unit (
cube17Regression):tableColumnTypesfor the same schema, with whole numbers first, so row guessing would get them wrong.Real data, in-cluster (84 Abler partitions taken from the failed read-only run, with Synmetrix's own datasource loader and the patched driver):
Can't parse intpartitions.Full suite, locally: only the 2 known stale-local-install dependency-hardening failures. The image build runs the full suite from the lockfile.
Rollout
Same as #88: once
latestis built, delete the cubejs pods (API one at a time, then the worker); Argo selfHeal is on.The datasource hash changes again, so Cube cleans up stale table versions. Watch the partition count per schema, not the table count. Rollback is a revert PR.
🤖 Generated with Claude Code