Skip to content

Publish end-of-support DBR runtimes still requested by clusters (12.2, 16.1) - #28

Merged
rugpanov merged 2 commits into
mainfrom
fix/dbr-publish-eos-runtimes
Aug 31, 2026
Merged

Publish end-of-support DBR runtimes still requested by clusters (12.2, 16.1)#28
rugpanov merged 2 commits into
mainfrom
fix/dbr-publish-eos-runtimes

Conversation

@rugpanov

Copy link
Copy Markdown
Collaborator

Problem

VPEX E_ENV_UNSUPPORTED telemetry (an owned target with no environment published) shows a few end-of-support DBR runtime lines still requested by live clusters:

env_key events workspaces
dbr/12.2.x-scala2.12 8 2
dbr/16.1.x-scala2.12 1 1

(The louder E_ENV_UNSUPPORTED env_keys — dbr/18.x-scala2.13, dbr/16.4.x-scala2.13, dbr/16.4.x-cpu-ml-scala2.13 — are already fixed by #26 / #25.)

These EoS lines can't be published today because they are un-discoverable: they're dropped from the release-notes index (so discover_dbr never sees them) and skipped by dbr_point_releases' is_eos check, and the eos index page links none of them. There is no page that enumerates EoS runtimes.

Change

  • sync.py — add DBR_EOS_PUBLISH (["12.2", "16.1"]): a narrow, telemetry-curated allowlist of EoS page slugs to publish anyway. Its comment carries the telemetry counts and the rule: add a slug only when telemetry shows its exact env_key getting E_ENV_UNSUPPORTED — a deliberate, scoped exception to the EoS-not-published policy.
  • Factor sync_dbr's per-slug body into _sync_dbr_page(slug, point_release, umbrella_major) (mirrors the existing _sync_dbr_ml_page signature), and add sync_dbr_eos() that runs the allowlist through it directly — fetched by slug, so the EoS skip never applies. Each is a pre-18-style page (real minor in its title), so it publishes only its own <key_ver>.x folder, no bare-major umbrella. Wired into main().
  • envgen.py — add 16.1.x-scala2.12 to DROP_BY_ENV. DBR 16.1 is Python 3.12 pinned to pandas 1.5.3 (no cp312 wheel), the same case as 16.4 / serverless-v3 (Drop pandas on DBR 16.4 + serverless-v3 (no cp312 wheel) #23). DBR 12.2 is Python 3.9 and keeps pandas~=1.4.2.

Scope: standard (non-ML) only — no EoS ML line has any E_ENV_UNSUPPORTED traffic. The allowlist pattern extends to sync_dbr_ml later if that changes.

Generated artifacts

  • python/dbr/12.2.x-scala2.12/ (Python 3.9.21, 126 packages)
  • python/dbr/16.1.x-scala2.12/ (Python 3.12.3, 164 packages)

Testing

  • New SyncDbrEosTest: publishes an allowlisted line despite its (EoS) title; skips an allowlisted slug whose page is gone (fetch failure). Extended the envgen scoped-drop test to cover 16.1.x-scala2.12. Full suite: 45 tests pass.
  • Both generated environments pass uv sync (verify_resolve.py, via the Databricks pypi proxy).

Relates to DECO-28292 (bug 3).

This pull request and its description were written by Isaac.

@rugpanov
rugpanov marked this pull request as ready for review August 31, 2026 10:46
rugpanov and others added 2 commits August 31, 2026 13:14
…, 16.1)

EoS runtime lines are dropped from the release-notes index and skipped by
dbr_point_releases' is_eos check, so discover_dbr never sees them and there is no
way to enumerate them. VPEX E_ENV_UNSUPPORTED telemetry shows a few EoS env_keys
still requested by live clusters with no environment published:
dbr/12.2.x-scala2.12 (8 events / 2 workspaces) and dbr/16.1.x-scala2.12 (1 / 1).

Add a narrow, telemetry-curated allowlist (sync.DBR_EOS_PUBLISH) of EoS page slugs
to publish anyway. Factor sync_dbr's per-slug body into _sync_dbr_page and add
sync_dbr_eos, which runs the allowlist through it directly (bypassing the EoS
skip); each is a pre-18-style page so it publishes only its own <key_ver>.x folder.
16.1 is Python 3.12 on pandas 1.5.3 (no cp312 wheel), so it joins DROP_BY_ENV like
16.4/serverless-v3; 12.2 is Python 3.9 and keeps pandas. Both generated envs pass
uv sync. Standard (non-ML) only: no EoS ML line has traffic.

Co-authored-by: Isaac <no-reply@databricks.com>
Review round (Isaac) surfaced two latent gaps in sync_dbr_eos, neither triggered
by today's allowlist:

- A bare-major 18+ slug would emit a <major>.x umbrella folder colliding with the
  live umbrella sync_dbr builds. Reject such slugs (via _umbrella_major) with a
  clear message; specific EoS point-release slugs still publish normally.
- The per-Scala DROP_BY_ENV coupling (a dual-Scala py3.12 EoS line would emit an
  undropped scala2.13 folder) is a manual-maintenance property shared with the
  existing 16.4 handling and caught by verify_resolve on add — documented in the
  DBR_EOS_PUBLISH comment rather than duplicated in code.

Also update README: the "EoS point releases are skipped" note now records the
DBR_EOS_PUBLISH exception. Generated artifacts unchanged; 46 unit tests pass.

Co-authored-by: Isaac <no-reply@databricks.com>
@rugpanov
rugpanov force-pushed the fix/dbr-publish-eos-runtimes branch from aedaff4 to c7a4fdd Compare August 31, 2026 11:16
@rugpanov
rugpanov merged commit 597793b into main Aug 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants