Repository navigation
perf(cubejs): scheduler reads model code once per version, not every interval - #95
Merged
Merged
Conversation
…interval Every refresh interval fetched the full source of every active model for all 75 datasources (5.1 MB from Hasura, then YAML-parsed on the main thread) only to re-check the legacy FILTER_PARAMS-partition quarantine. Versions are immutable, so that verdict never changes for a version. The interval query now carries model ids only (0.06 MB on dev). Model code is read once per newly active version (largest 702 KB) and the verdict cached; versions that are no longer active are dropped from the cache. The smaller per-interval response also shrinks the window for the in-transit cut-offs seen between Hasura and the worker. Verified against dev Hasura: identical quarantine verdicts (2 datasources / 4 models) and identical model ids for every datasource. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every refresh-scheduler interval (≈1/min) fetched the full source of every active model for all 75 datasources: 5.1 MB from Hasura, then YAML-parsed on the worker's main thread. It did this only to re-check the legacy FILTER_PARAMS-partition quarantine. Versions are immutable, so that verdict never changes for a given version.
This large per-minute response is also the one that has been cut off in transit (2026-10-08, and again on 2026-10-12 at 18:17). #93 turned that from a multi-day hang into a single failed interval; this change shrinks the response itself.
What
getDataSources(scheduler-only) carries model ids only.getVersionModelCode(versionId)reads one version's model code.scheduledRefreshContextscaches the unsafe-model count per active version id and reads code only for versions it hasn't seen. Inactive versions are evicted each interval.buildScheduledRefreshContextstakes an injectableunsafeSchemaCount. It defaults to the old code-based check, so existing callers and tests are unchanged.Verified against dev Hasura (read-only)
schemaVersion/ cache isolation)Test
scheduledRefreshQuarantineCache.test.jschecks three things: two intervals read each version's code once; a newly published version is read and its datasource un-quarantined; quarantined datasources stay excluded. The existing scheduler tests pass. Cubejs suite: 703/705; the 2 failures are the existingruntimeDependencyHardeninglocal-install checks.🤖 Generated with Claude Code