Repository navigation
fix(drift): probe K9 exhaustion on gpt-5.4-mini after o4-mini retirement - #584
Merged
Merged
Conversation
OpenAI shut down o4-mini on 2026-10-23 (its /v1/models shutdown_date); the K9 HTTP and WS probes have answered 404 model_not_found since. Move the probe model into one OPENAI_K9_PROBE_MODEL constant, grade the retained o4-mini replays against OPENAI_K9_REPLAY_MODEL, and accept output_tokens equal to reasoning_tokens instead of exactly 0.
commit: |
jpr5
added a commit
that referenced
this pull request
Oct 27, 2026
## Why OpenAI retired `o4-mini` (`shutdown_date` 2026-10-23). The K9 drift probe still pinned it, so the nightly **Drift Tests** failed for three nights ([run 40279023362](https://github.com/CopilotKit/aimock/actions/runs/40279023362)). `notify` reported each failure as "HTTP API drift detected — providers changed response formats". `fix-drift.yml` syncs only model-list churn, so nothing caught the retirement. #584 moved the probe to `gpt-5.4-mini`. This PR catches the next retirement weeks ahead and names it plainly. ## What - `src/__tests__/drift/probe-models.ts`: one side-effect-free list of every provider model the live drift probes **pin**. `OPENAI_K9_PROBE_MODEL` (from #584) now lives here, and `helpers.ts` re-exports it. Legs that choose their model from the live listing (`resolveLiveModel`) heal themselves and are not listed. A unit test checks that each entry is still pinned in the source file it names. - `scripts/drift-model-preflight.ts`: a new step, `Preflight — probe model retirement`, in the daily `drift` job, after the key-freshness preflight. - Gone, or past its shutdown date: `::error` with `probe model X retired on DATE — update <constant>`, and the step fails before any drift test runs. - Shutdown date within 30 days, or deprecated: `::warning`, a row in the step summary, and the `retirement_alert` output. - No key, no retirement API, 401/403, 429/5xx or network error: informational skip, never a failure. - `notify`: a retired probe model posts "Drift probe model retired … (not API drift)" instead of "HTTP API drift". A soon-to-retire model posts a warning on an otherwise quiet day, and is added to any other alert. It reuses the existing `SLACK_WEBHOOK`. No new secrets; no new actions or permissions. - `gemini.drift.ts`: the embeddings canary called `text-embedding-004`, which answers 404 (the canary logged it and passed). It now calls `gemini-embedding-001`, which was checked live and returns a 3072-value embedding. Without this change, the new preflight would fail the first nightly run. ## Retirement data per provider (each checked with one live request, 2026-10-26) | Provider | Field | Gone | |---|---|---| | OpenAI `GET /v1/models/{id}` | `shutdown_date` (`o4-mini` → 2026-10-23, `gpt-5-mini` → 2026-12-11) | 404 | | Anthropic `GET /v1/models/{id}` | `lifecycle`, `deprecated_at`, `retires_at` | 404 (`claude-3-haiku-20240307`) | | Gemini `GET /v1beta/models/{id}` | no date field | 404 (`text-embedding-004`) | | Cohere `GET /v1/models/{id}` | `is_deprecated` | 404 ("removed on September 15, 2025") | | OpenRouter `GET /api/v1/models` | `expiration_date` | missing from the listing | | Bedrock `GetFoundationModel` | only `modelLifecycle.status` (ACTIVE/LEGACY), needs AWS creds: **skipped** | ResourceNotFound | | Vertex / Ollama / fal | none: **skipped** | — | ## Local red-green (real provider APIs, real drift keys) **RED.** In a scratch copy with `OPENAI_K9_PROBE_MODEL = "o4-mini"`, exit 1: ``` ::error title=probe-model-retired::probe model o4-mini (openai) retired on 2026-10-23 — update OPENAI_K9_PROBE_MODEL in src/__tests__/drift/probe-models.ts ::error::Probe-model preflight failed: a drift probe model is retired. Drift was not run — this is not API drift. ``` **Warning path.** In a scratch copy with `gpt-5-mini` added and `DRIFT_PREFLIGHT_WARN_DAYS=60`, exit 0. The step output `retirement_alert` carries the same line: ``` ::warning title=probe-model-retiring::probe model gpt-5-mini (openai) retires on 2026-12-11 (in 46 days) — update the pin in src/__tests__/drift/providers.ts and its entry in src/__tests__/drift/probe-models.ts ``` **GREEN.** This branch with the real probe models, exit 0: ``` preflight: probe model gpt-4o-mini (openai): served (shutdown 2027-01-14) preflight: probe model text-embedding-3-small (openai): served (no shutdown date) preflight: probe model whisper-1 (openai): served (shutdown 2027-02-26) preflight: probe model gpt-realtime-mini (openai): served (shutdown 2027-01-20) preflight: probe model gpt-4.1-mini (openai): served (shutdown 2027-01-14) preflight: probe model gpt-5.4-mini (openai): served (no shutdown date) preflight: probe model claude-haiku-4-5-20251001 (anthropic): served (no shutdown date) preflight: probe model claude-sonnet-4-5-20250929 (anthropic): served (no shutdown date) preflight: probe model claude-sonnet-4-5 (anthropic): served (no shutdown date) preflight: probe model gemini-2.5-flash (gemini): served (Gemini publishes no shutdown date) preflight: probe model gemini-embedding-001 (gemini): served (Gemini publishes no shutdown date) preflight: probe model rerank-v3.5 (cohere): served (Cohere publishes no shutdown date) preflight: probe model qwen/qwen3-32b (openrouter): served (no shutdown date) preflight: probe model us.anthropic.claude-sonnet-4-5-20250929-v1:0 (bedrock): skipped — no retirement API reachable here (GetFoundationModel gives only ACTIVE/LEGACY and needs AWS credentials) preflight: probe model gemini-2.5-flash (vertex): skipped — no retirement API reachable here preflight: probe model qwen2:0.5b (ollama): skipped — local daemon model; nothing to retire upstream preflight: probe model qwen3:0.6b (ollama): skipped — local daemon model; nothing to retire upstream preflight: probe model fal-ai/flux/schnell (fal): skipped — fal exposes no model retirement API preflight: no drift probe model is retired ``` The date logic has unit tests with fixed dates. A model counts as retired on its shutdown date, and the warning window includes its last day. A mutation from `<= 0` to `< 0` makes the shutdown-day test fail. The executed `notify` tests fail against the unmodified workflow (4 of 5; the negative control passes). ## Gates (run one at a time, exit codes) typecheck 0 · lint 0 · format:check 0 · new tests 0 (51) · actionlint on `test-drift.yml` 0 (the 5 repo-wide info findings are in `changelog-radar.yml` and `test-pytest.yml`; main has the same 5) · zizmor `--min-severity medium` 0 · build 0 · `pnpm test` 0 (484 files, 19690 passed) ## Worth knowing - The OpenAI probe models `gpt-4o-mini` and `gpt-4.1-mini` shut down on 2027-01-14, and `gpt-realtime-mini` on 2027-01-20. The preflight starts warning about them in mid-December. - On Bedrock, `anthropic.claude-sonnet-4-5-20250929-v1:0` already reports `LEGACY` in us-west-2. This preflight cannot see that, because it holds no AWS credentials.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
OpenAI retired
o4-minion 2026-10-23.GET /v1/models/o4-ministill lists it, with"shutdown_date": "2026-10-23". The same date is ono4-mini-2025-04-16ando3-mini. Since that date, the K9 reasoning-exhaustion probe has failed every night: Oct 23–25, runs 40069285652, 40168287724 and 40279023362.The real API, with the exact K9 request:
{"type":"error","status":404,"error":{"type":"invalid_request_error","code":"model_not_found","message":"The modelo4-minidoes not exist or you do not have access to it.","param":null}}404with the samemodel_not_foundbody.The exhaustion contract itself did not change, so the mock is unchanged.
Fix (drift harness only)
OPENAI_K9_PROBE_MODEL = "gpt-5.4-mini"insrc/__tests__/drift/helpers.ts. It has noshutdown_date. Both drift files and the K9 notes use this one constant, so the next retirement is a one-line change.OPENAI_K9_REPLAY_MODEL = "o4-mini": the retained 2026-10-10 replay fixtures stay frozen and are still graded against the model they were captured from.output_tokens === output_tokens_details.reasoning_tokens. It no longer requiresoutput_tokens === 0.output_tokens: 32, all of themreasoning_tokens.response.created,response.in_progress,response.output_item.added,reasoning_summary_part.added,reasoning_summary_text.delta…,reasoning_summary_text.done,reasoning_summary_part.done,response.output_item.done,response.incomplete(max_output_tokens).Red / green (local, real OpenAI API)
RED (origin/main, live WS K9):
HTTP K9 on the same request:
404 model_not_found(body above).GREEN (this branch, live):
Full
pnpm test:drift(AIMOCK_NATIVE_PROVIDER_DRIFT=1, OpenAI key only): 593 passed, 3 failed. All three failures are outside this PR:/modelsfamily canary:gpt-6.1-terraappeared today and is not yet classified. The deterministic drift-sync job handles new model families.Gates (each run on its own, exit code 0):
typecheck,lint,format:check,build,api:reportwithgit diff --exit-code etc/clean, andpnpm test(483 files, 19662 passed). The offline K9 replay tests pass with no key.