Repository navigation
ci(drift): fail fast on a retired drift probe model - #585
Merged
Merged
Conversation
text-embedding-004 answers 404 on the Gemini Developer API, so the canary logged a 404 every night and checked nothing. gemini-embedding-001 answers embedContent with a 3072-value embedding (checked live 2026-10-26).
Before the nightly drift run, scripts/drift-model-preflight.ts asks each provider about every model the live probes pin, listed in one place in src/__tests__/drift/probe-models.ts. A gone or past-shutdown model fails the step with "probe model X retired on DATE — update <constant>", and notify reports it as a retirement, not as API drift. A shutdown date within 30 days is a warning in the job summary and in Slack. Missing keys, providers without retirement data, and transient errors skip. Retirement fields, each checked live: OpenAI shutdown_date, Anthropic lifecycle/retires_at, OpenRouter expiration_date, Cohere is_deprecated, and a 404 from Gemini, Anthropic, OpenAI or Cohere when a model is gone.
…be models The retirement preflight reads probe-models.ts without loading helpers.ts, which starts servers. helpers.ts re-exports the constant, so the K9 probes are unchanged.
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
OpenAI retired
o4-mini(shutdown_date2026-10-23). The K9 drift probe still pinned it, so the nightly Drift Tests failed for three nights (run 40279023362).notifyreported each failure as "HTTP API drift detected — providers changed response formats".fix-drift.ymlsyncs only model-list churn, so nothing caught the retirement. #584 moved the probe togpt-5.4-mini. This PR catches the next retirement weeks ahead and names it plainly.What
src/__tests__/drift/probe-models.ts: one side-effect-free list of every provider model the live drift probes pin.OPENAI_K9_PROBE_MODEL(from fix(drift): probe K9 exhaustion on gpt-5.4-mini after o4-mini retirement #584) now lives here, andhelpers.tsre-exports it. Legs that choose their model from the live listing (resolveLiveModel) heal themselves and are not listed. A unit test checks that each entry is still pinned in the source file it names.scripts/drift-model-preflight.ts: a new step,Preflight — probe model retirement, in the dailydriftjob, after the key-freshness preflight.::errorwithprobe model X retired on DATE — update <constant>, and the step fails before any drift test runs.::warning, a row in the step summary, and theretirement_alertoutput.notify: a retired probe model posts "Drift probe model retired … (not API drift)" instead of "HTTP API drift". A soon-to-retire model posts a warning on an otherwise quiet day, and is added to any other alert. It reuses the existingSLACK_WEBHOOK. No new secrets; no new actions or permissions.gemini.drift.ts: the embeddings canary calledtext-embedding-004, which answers 404 (the canary logged it and passed). It now callsgemini-embedding-001, which was checked live and returns a 3072-value embedding. Without this change, the new preflight would fail the first nightly run.Retirement data per provider (each checked with one live request, 2026-10-26)
GET /v1/models/{id}shutdown_date(o4-mini→ 2026-10-23,gpt-5-mini→ 2026-12-11)GET /v1/models/{id}lifecycle,deprecated_at,retires_atclaude-3-haiku-20240307)GET /v1beta/models/{id}text-embedding-004)GET /v1/models/{id}is_deprecatedGET /api/v1/modelsexpiration_dateGetFoundationModelmodelLifecycle.status(ACTIVE/LEGACY), needs AWS creds: skippedLocal red-green (real provider APIs, real drift keys)
RED. In a scratch copy with
OPENAI_K9_PROBE_MODEL = "o4-mini", exit 1:Warning path. In a scratch copy with
gpt-5-miniadded andDRIFT_PREFLIGHT_WARN_DAYS=60, exit 0. The step outputretirement_alertcarries the same line:GREEN. This branch with the real probe models, exit 0:
The date logic has unit tests with fixed dates. A model counts as retired on its shutdown date, and the warning window includes its last day. A mutation from
<= 0to< 0makes the shutdown-day test fail. The executednotifytests fail against the unmodified workflow (4 of 5; the negative control passes).Gates (run one at a time, exit codes)
typecheck 0 · lint 0 · format:check 0 · new tests 0 (51) · actionlint on
test-drift.yml0 (the 5 repo-wide info findings are inchangelog-radar.ymlandtest-pytest.yml; main has the same 5) · zizmor--min-severity medium0 · build 0 ·pnpm test0 (484 files, 19690 passed)Worth knowing
gpt-4o-miniandgpt-4.1-minishut down on 2027-01-14, andgpt-realtime-minion 2027-01-20. The preflight starts warning about them in mid-December.anthropic.claude-sonnet-4-5-20250929-v1:0already reportsLEGACYin us-west-2. This preflight cannot see that, because it holds no AWS credentials.