Skip to content

fix(cubejs): BigQuery rollups read-only; failed rollup downloads no longer crash the refresh worker - #88

Merged
acmeguy merged 3 commits into
mainfrom
fix/bigquery-readonly-preaggs
Oct 7, 2026
Merged

acmeguy merged 3 commits into
mainfrom
fix/bigquery-readonly-preaggs

Conversation

@acmeguy

@acmeguy acmeguy commented Oct 7, 2026

Copy link
Copy Markdown

Why

The refresh worker restarted 33 times in under four days. Each crash stopped rollup builds for every team. The cause was one tenant's BigQuery rollups (Abler):

  • abler-web (BQ QA): Cube stages every rollup partition as a table in the customer's BigQuery project before loading it into Cube Store. Its service account may not create that dataset (bigquery.datasets.create), so none of its rollups ever built.
  • abler-web-prod (BQ Prod): 2,045 rollups are in Cube Store, but some builds lost their staging table mid-load. Cube 1.7.30 pipes BigQuery's stream through two pipe() calls that never forward errors, so a "Not found: Table" became an uncaughtException, and the process guard exits on those.

What

  1. BigQuery datasources are read-only (prepareDbParams). Cube runs the rollup query and loads the result straight into Cube Store, with no staging tables in the customer's project.
    • Rollup table versions don't depend on readOnly, so existing Cube Store tables stay current and nothing mass-rebuilds.
    • Each partition's result is held in worker memory. Abler's largest is 54,796 rows.
  2. Stream errors are forwarded (postinstall patches, alongside the existing YAML and ClickHouse ones). One patch covers the BigQuery driver's stream(), the other the loader's LargeStreamWarning wrap at both of its sites. A failed download now fails only that build.
  3. Docs: cube principles on SQL wrapping, ClickHouse map keys, refresh cadence and the Cube Store Nullable(Bool) limit.

Tests

  • Loader path (cube17Regression): the test runs the real BigQuery driver through PreAggregationLoader.getTableDataWithTempTable. Before the patches it failed with uncaughtException; with them it is a handled rejection.
  • Drift guards: both new patches refuse to apply if Cube's source changes.
  • Read-only (prepareDbParams): a BigQuery driver built from the prepared settings reports readOnly().
  • Full suite, locally: the only failures are the 2 dependency-hardening checks against a stale local install. This PR's image build runs the full suite from the lockfile.

Rollout

After merge, restart synmetrix-cubejs and synmetrix-cubejs-refresh-worker together, so the API and the worker hash the BigQuery datasource the same way. Then watch:

  • the worker's restart count;
  • Abler's rollup tables in Cube Store;
  • other tenants' build errors.

🤖 Generated with Claude Code

acmeguy and others added 3 commits October 7, 2026 16:39
Cube already wraps a cube's sql as a subquery, so an extra SELECT * FROM
(...) only adds a layer. On ClickHouse, a map key read in a member over a
SELECT * subquery reads the whole map (293 MiB vs 28 MiB measured on
cst.semantic_events); extract keys in the SELECT that reads the table.
Refresh cadence should follow data arrival; Cube 1.7 rejects
incremental/update_window next to refresh_key.sql. Cube Store rejects
Nullable(Bool) rollup columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…worker

Cube 1.7.30 pipes the BigQuery query stream into its row stream, and the
pre-aggregation loader pipes that into a LargeStreamWarning; pipe() never
forwards errors. A "Not found: Table" from one tenant's staging table
became an uncaughtException, the process guard exited, and every team's
in-flight rollup builds died with it (33 restarts in under four days).

Two postinstall patches forward the errors so they reach the upload
pipeline, which already fails just that build. The regression test runs
the loader's real download path and failed with the uncaughtException
before the patches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without readOnly, Cube stages every rollup partition as a table in the
customer's BigQuery project before loading it into Cube Store. abler-web's
service account may not create that dataset, so none of its rollups ever
built; abler-web-prod's staging tables went missing mid-build. Read-only
runs the rollup query and loads the result straight into Cube Store.
Table versions do not depend on readOnly, so existing rollups stay
current. Each partition's result is held in worker memory; Abler's
largest is 54,796 rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@acmeguy
acmeguy force-pushed the fix/bigquery-readonly-preaggs branch from eb3ad17 to 87e08c9 Compare October 7, 2026 14:39
@acmeguy
acmeguy merged commit 6de9cd2 into main Oct 7, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant