Publish end-of-support DBR runtimes still requested by clusters (12.2, 16.1) - #28
Merged
Conversation
rugpanov
marked this pull request as ready for review
August 31, 2026 10:46
…, 16.1) EoS runtime lines are dropped from the release-notes index and skipped by dbr_point_releases' is_eos check, so discover_dbr never sees them and there is no way to enumerate them. VPEX E_ENV_UNSUPPORTED telemetry shows a few EoS env_keys still requested by live clusters with no environment published: dbr/12.2.x-scala2.12 (8 events / 2 workspaces) and dbr/16.1.x-scala2.12 (1 / 1). Add a narrow, telemetry-curated allowlist (sync.DBR_EOS_PUBLISH) of EoS page slugs to publish anyway. Factor sync_dbr's per-slug body into _sync_dbr_page and add sync_dbr_eos, which runs the allowlist through it directly (bypassing the EoS skip); each is a pre-18-style page so it publishes only its own <key_ver>.x folder. 16.1 is Python 3.12 on pandas 1.5.3 (no cp312 wheel), so it joins DROP_BY_ENV like 16.4/serverless-v3; 12.2 is Python 3.9 and keeps pandas. Both generated envs pass uv sync. Standard (non-ML) only: no EoS ML line has traffic. Co-authored-by: Isaac <no-reply@databricks.com>
Review round (Isaac) surfaced two latent gaps in sync_dbr_eos, neither triggered by today's allowlist: - A bare-major 18+ slug would emit a <major>.x umbrella folder colliding with the live umbrella sync_dbr builds. Reject such slugs (via _umbrella_major) with a clear message; specific EoS point-release slugs still publish normally. - The per-Scala DROP_BY_ENV coupling (a dual-Scala py3.12 EoS line would emit an undropped scala2.13 folder) is a manual-maintenance property shared with the existing 16.4 handling and caught by verify_resolve on add — documented in the DBR_EOS_PUBLISH comment rather than duplicated in code. Also update README: the "EoS point releases are skipped" note now records the DBR_EOS_PUBLISH exception. Generated artifacts unchanged; 46 unit tests pass. Co-authored-by: Isaac <no-reply@databricks.com>
rugpanov
force-pushed
the
fix/dbr-publish-eos-runtimes
branch
from
August 31, 2026 11:16
aedaff4 to
c7a4fdd
Compare
misha-db
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
VPEX
E_ENV_UNSUPPORTEDtelemetry (an owned target with no environment published) shows a few end-of-support DBR runtime lines still requested by live clusters:dbr/12.2.x-scala2.12dbr/16.1.x-scala2.12(The louder
E_ENV_UNSUPPORTEDenv_keys —dbr/18.x-scala2.13,dbr/16.4.x-scala2.13,dbr/16.4.x-cpu-ml-scala2.13— are already fixed by #26 / #25.)These EoS lines can't be published today because they are un-discoverable: they're dropped from the release-notes index (so
discover_dbrnever sees them) and skipped bydbr_point_releases'is_eoscheck, and theeosindex page links none of them. There is no page that enumerates EoS runtimes.Change
sync.py— addDBR_EOS_PUBLISH(["12.2", "16.1"]): a narrow, telemetry-curated allowlist of EoS page slugs to publish anyway. Its comment carries the telemetry counts and the rule: add a slug only when telemetry shows its exact env_key gettingE_ENV_UNSUPPORTED— a deliberate, scoped exception to the EoS-not-published policy.sync_dbr's per-slug body into_sync_dbr_page(slug, point_release, umbrella_major)(mirrors the existing_sync_dbr_ml_pagesignature), and addsync_dbr_eos()that runs the allowlist through it directly — fetched by slug, so the EoS skip never applies. Each is a pre-18-style page (real minor in its title), so it publishes only its own<key_ver>.xfolder, no bare-major umbrella. Wired intomain().envgen.py— add16.1.x-scala2.12toDROP_BY_ENV. DBR 16.1 is Python 3.12 pinned to pandas 1.5.3 (no cp312 wheel), the same case as 16.4 / serverless-v3 (Drop pandas on DBR 16.4 + serverless-v3 (no cp312 wheel) #23). DBR 12.2 is Python 3.9 and keepspandas~=1.4.2.Scope: standard (non-ML) only — no EoS ML line has any
E_ENV_UNSUPPORTEDtraffic. The allowlist pattern extends tosync_dbr_mllater if that changes.Generated artifacts
python/dbr/12.2.x-scala2.12/(Python 3.9.21, 126 packages)python/dbr/16.1.x-scala2.12/(Python 3.12.3, 164 packages)Testing
SyncDbrEosTest: publishes an allowlisted line despite its(EoS)title; skips an allowlisted slug whose page is gone (fetch failure). Extended theenvgenscoped-drop test to cover16.1.x-scala2.12. Full suite: 45 tests pass.uv sync(verify_resolve.py, via the Databricks pypi proxy).Relates to DECO-28292 (bug 3).
This pull request and its description were written by Isaac.