[acmeguy]

Problem

On dev the refresh worker froze on 2026-10-08 03:41 UTC and stopped refreshing pre-aggregations for every tenant for 4.5 days. There were no errors and no restarts. It was reported as abler's BigQuery pre-aggs not updating.

Root cause

At the start of every interval the scheduler fetches its contexts with fetchGraphQL → getDataSources(), and it runs contexts serially. fetchGraphQL used node-fetch 2.7.0, which pipes the body through PassThrough → Gunzip with pipe(). pipe() does not forward an aborted response. So when Hasura's gzip + Content-Length response was cut off mid-body, res.json() stayed pending forever. The whole await chain was then garbage-collected and the interval never finished.

Evidence from the live process (inspector + heap snapshot):

• the once-per-interval "context quarantined" log stopped after 03:39
• the CubeStore queue was empty and no websocket messages were pending
• no Hasura, BigQuery or ClickHouse sockets were open
• compilers, driver promises and queue reconciles were all settled
• the interval's execution promise had no async frame retaining it

Fix

Use Node's built-in fetch (undici) in fetchGraphQL. It rejects with terminated on a cut-off body and bounds headers and body at 300s by default. eventEmitter.js still uses node-fetch, but it is fire-and-forget and not on the scheduler path.

Test

src/utils/__tests__/graphql.test.js serves half of a gzip response and then drops the connection. It stayed pending before this change and is rejected after. The full cubejs suite passes 701/703. The 2 failures are the existing runtimeDependencyHardening checks of locally installed packages (Docker postinstall only).

Dev was unblocked by restarting synmetrix-cubejs-refresh-worker. A stall monitor follows separately.

🤖 Generated with Claude Code