fix(platform): back off terminal Codex model capacity

[yannickmonney]

Summary

Codex’s exact terminal “Selected model is at capacity” failure now becomes model_capacity. Its normal counted retry waits one minute through the existing durable queue, and the prior broker account stays eligible. The original error, configured model, retry limit and three-start hourly circuit remain intact. The typed result survives checkpoint restoration, and an early capacity refusal on a resumed turn no longer immediately launches fresh; the announced conversation handle remains available for its counted retry. The delay is sampled after the winning terminal write and ledger; later queue/commit latency or wall-clock changes can consume part of it.

This backoff applies to project task-agent runs. Automation workflow-agent capacity handling is unchanged and tracked separately in #4530.

The backend email integration fixture now uses nonce recipients and counts every owned delivery, including a forbidden preference-off delivery. A deliberately unrelated real notification cannot pollute its exact counts. This fixes an incidental fixture isolation defect without changing production mail delivery.

Checklist

• Existing model-capacity implementation passed the normal 49-task gate, Knip and SAST before publication.
• Tested head fd552f5d on main a14fa1fb passed the normal 49-task gate, Knip and SAST (0 findings), then all seven hosted workflows; backend integration passed 1692/1692 and browser tests had no retries.
• Current head 39094816 normally merges main 7efc00a5: all 22 feature files and all 66 upstream files are byte-identical to their respective parents, with no overlapping paths or lock/workflow change. Scoped format, conflict, link and manual guards pass; the prior full local gate is retained as prior-head evidence.
• Manual test register and type-aware scoped lint pass.
• Existing localized model-failure labels are reused; no message keys changed.
• EN/DE/FR task-automation docs were rendered at desktop and mobile widths; the checkpoint/resume and integration-fixture corrections change no product documentation or UI.

Test plan

• 128 focused parser/host/retry/account-selection tests and 22 real-Postgres model-capacity records cover terminal-only classification, exec fencing, durable delay, account eligibility and the unchanged hourly circuit. Fixture kick/read/cancel shares one transaction, so ordinary workers cannot launch the fixture run.
• Original refreshed-head backend run: 1690/1691 checks across all 236 lanes; all five model-capacity assertions passed. The sole failure counted three notification emails instead of two. The third message’s identity was not retained, so its producer remains unknown.
• Controlled fixture regression: 19/20 before recipient isolation, 20/20 afterward across two real-Postgres lanes. The injected unrelated delivery is observed separately; exact owned counts remain one and two, with no deduplication or content-based inclusion. Read, undo, preference, subject and overdue deep-link assertions remain enforced.
• Checkpoint/resume regressions failed before the fix (2 failures) and passed afterward in the 76-test focused suite: restoration consumes the saved terminal event without replaying stdout, and a resumed capacity failure launches exactly once without penalizing the account.
• Fresh complete hosted checks remain required before merge. Live provider recovery is a separate deployment observation.

[yannickmonney]

Independent review of PR #4529 at 5d42ffd5a279e07a0f299573b54cd6524a3ea6f9 (base e7d255bf5a6364d0dd324de91ecdad4fc0fb7689), both parts, API-only.

I am Tale agent #6 (53fe2a33-e8f6-4034-8025-e91d39fdc466), working on card TALE-1234 (6ea24eba-c6d9-4c41-826b-843e56b02fd4) in run 37decf6f-6c3e-4123-a99e-c51bd35de510. I am not the PR's author. I read the source at the exact head through the contents API. I did not clone, install, build or run tests locally.

Verdict: REQUEST CHANGES

There is one small blocking gap (B1), in the recovery path. The rest of the change is sound: classification, the backoff, the budgets, the account handling, the integration lane and the docs.

Blocking

B1: the new providerErrorKind is lost when a drain window restores its checkpoint, so a recovered turn settles as harness_error

How the kind travels:

• The parser now sets providerErrorKind: 'model_capacity' on turn-ended (lib/harnesses/parsers/codex-jsonl.ts:266-268, lib/harnesses/types.ts:319).
• The host reads it back from window.ended (backend/core/tasks/agent_run_host.ts:1911, :1971).

Where it is dropped:

• window.ended is carried across windows in the exec checkpoint (backend/core/chat/external_turn_shared.ts:586-599).
• It is restored with turnCheckpointSchema.parse (:533, :541).
• turnEndSchema (:349-364) is a plain zod 4.3.6 z.object. It lists apiErrorStatus but not providerErrorKind, so the parse strips the new key.

When it happens:

1. A window parses turn.failed.
2. A checkpoint is flushed. The 5 s throttle in maybeCheckpoint (:579-602) allows it, and the cursor already covers that chunk (session_client.ts:1933-1937).
3. The settle does not complete in that window: the worker dies or the settle throws.
4. A later window resumes the exec from the checkpoint. That can be a job retry, the stalled-turn re-attach (domains/tasks/reattach.ts) or the replay-gap adoption (external_turn_shared.ts:744-763).

That window restores ended without the kind. The turn.failed line is already consumed: it is not replayed, and parser.end() only flushes an unterminated line (codex-jsonl.ts:124-126). Nothing re-derives the kind.

Effect: continueOrSettle stamps harness_error. The retry is counted and immediate, and kick-plan.ts:138-144 excludes the healthy account. That is exactly the pre-PR behaviour this PR removes, here on the recovery path. It is fail-safe and not a regression, but the typed contract does not survive a hand-off.

Fix (small): add providerErrorKind: z.literal('model_capacity').optional() to turnEndSchema. Add an old-fails/new-passes regression: restore a checkpoint whose ended carries the kind, and expect failureCode: 'model_capacity'.

Non-blocking

N1: workflow agent steps keep the old behaviour, so the two lanes now diverge.

• The shared parser emits the kind on both lanes.
• backend/core/automations/agent_host.ts:2613-2621 still maps every errored end to harness_error.
• agent_retry.ts:386-395 then counts that failure, burns parked.brokerTokenHash and re-kicks at once.
• agent_retry.ts:321-323 documents "task-lane parity with resolveAutoRetryBudget".

This is not a regression: automation behaviour is unchanged, and the docs only describe task runs. Please either extend the workflow lane, or state the task-only scope in the PR body and open a follow-up issue.

N2: the wording in backend/domains/tasks/spec.md:152-153. "it neither spends a free account-cooldown wait" can be read as if a free wait were kept in reserve. In the code, a capacity failure never qualifies for the free wait: freeCooldownWaits needs a 429 predecessor (task_auto_retry.ts:216-230, and delegated-start.ts:202-211 for the circuit). Suggested wording: "it does not qualify for the free account-cooldown wait and does not exclude the account."

N3: no test covers a resumed capacity failure. On a resumed Codex turn, which is the usual place for a second failure, a capacity error trips isResumeLaunchFailure (agent_run_host.ts:1797-1811): the turn has no text and no timeline, and its thread id echoes the resume handle. So the run row gets no agentSessionId. I traced that the conversation still survives:

• the start's op-row write carries the resume handle (agent_run_host.ts:2682);
• upsertSessionOp keeps it through coalesce (domains/tasks/agent-turn-shim.ts:654-655);
• kick-plan.ts:155-168 falls back to the op row.

The new host test only covers a fresh turn that has assistant text. A resumed variant would lock this path in.

Verified at the head (PASS from source)

Classification

• The parser matches only an exact, trimmed sentence on the turn.failed terminal channel (codex-jsonl.ts:251-271, :307-309).
• error events, assistant text and tool output never classify (parser test).
• No 429 is invented, so the run records no broker cooldown (agent_run_host.ts:2155-2164) and no rotation (:2171-2178 only rewrites harness_error).
• The Codex pin agrees: services/sandbox-runtime/Dockerfile:471 sets CODEX_VERSION=0.160.0.
• Any other wording falls back to the old harness_error behaviour.

Backoff against the budget, retryAfter and TASK-R12

• model_capacity is retryable (task_auto_retry.ts:85-97).
• resolveAutoRetryBudget counts it with no free skip: three retries. AUTO_RETRY_HISTORY_LIMIT is unchanged.
• The floor is sampled after the winning flip and the ledger entry (agent-runs.ts:593-598).
• The task.agent_retry job itself is not delayed. It re-derives every guard at once, admits the automated start (backend/jobs/task-list.ts:507-514) and kicks a queued run right away. Only the task.agent_turn job waits (agent-runs.ts:345-352).
• Capacity rows never match freeCooldownWaits, so every retry is an ordinary automated start.
• The outcome per chain:
  • an automated chain gets its original start plus two retries before task_circuit_breaker;
  • a person's run gets three retries.

Starvation, duplicates and orphans

• The once-only flip arms one retry per failed run (agent-runs.ts:563-579). A redelivered job reads superseded (task-list.ts:342-346).
• One live run per task (partial unique index plus ON CONFLICT, agent-runs.ts:312-344). A manual Retry or a mention during the hold reuses the queued run.
• The 60 s hold sits inside the 4-minute stranded-queued sweep (domains/sandbox/recovery.ts:19-24, reattach.ts:242-245), so the run is not re-kicked early.
• A busy-agent recheck carries a startAfterMs that has already passed by then, so it is ignored (task-list.ts:487-504).
• The held queued run marks its workspace busy (delegated-start.ts:283-290). The existing cooldown hold does the same, and so does the running retry it replaces. This adds no new kind of starvation.

Credential rotation

• A capacity row ends a rotation streak like any other failure.
• This opens no loophole: a dead grant cannot produce capacity errors.
• The host test asserts that the run records no recordBrokerFailureInternal call and no apiErrorStatus.

Data and rolling deploys

• failure_code is free text with no CHECK constraint (migration 0133).
• Old readers count the code and exclude the account, which is the old behaviour.
• The retry payload schema, unchanged in task-list.ts:274-284, already accepts startAfterMs.
• An old UI shows the unknown class.

Docs

• The EN, DE and FR paragraphs say the same thing, and each sits under "the cases below".
• Column names match the UI catalogues (In progress / In Bearbeitung / En cours, in messages/*/tasks.yml).
• FR uses no-break spaces before : and typographic apostrophes.
• The manual reference row matches the integration records.

Freshness

• main is two commits ahead of the base: 04b00cc4fe (#4326) and 85360da575 (#4201).
• The only file both sides change, tests/manual/reference/automation.md, merges cleanly: git merge-file over contents-API copies gives 0 conflicts.
• main touches none of the related backend files.

The head moved during this review

At 08:18:26Z root merged main (85360da575) into the branch. The new head is 7090d27dde7cbd8c7b53f1f5e04e2d5a77713dd0, with parents 5d42ffd5 and 85360da575. This review stays bound to 5d42ffd5, as briefed.

No evil merge. I compared 5d42ffd5...7090d27d with e7d255bf...85360da575:

• They touch the same 9 main-side files with identical patches.
• The blobs are identical too, except automation.md, which equals the clean 3-way merge.
• The PR's own 20 files are unchanged.

So B1 and N1–N3 apply verbatim at 7090d27d.

CI at 7090d27d (read once at 08:28Z, no reruns): 78 runs, of which 53 success, 19 skipped, 1 neutral, 2 failure and 3 in progress.

• Backend integration failed: job 113217505248 (run 37748977594) passed 1690 of 1691 checks.
  • The one failure: FAIL overdue notification email opens its task — emails=3 (want 2).
  • The same check passed at 5d42ffd5 (emails=2), and main 85360da575 passed Backend integration (job 113208743469).
  • Neither this PR nor main's two commits touch overdue notifications or email. A timing flake is likely but unproven.
  • The five model capacity: records passed again.
• CI ready (Checks) failed as a result.
• Before merge, the owner (root) needs a green result at the current head, through CI admission.

CI at the exact head (5d42ffd5)

At 5d42ffd5 there are 83 check runs: 60 success, 22 skipped, 1 neutral (Trivy) and 0 failed. All seven CI ready (…) aggregators succeeded.

Backend integration, job 113203157355 (run 37744632883, attempt 1), at 07:45:27Z:

• All five model capacity: records passed:
  • the 60 s floor (settled at 1791445527725, startAfter 1791445587728);
  • excluded=undefined;
  • the pg-boss startAfter equals the floor;
  • the cancellation happens before the jobs become visible;
  • the circuit answers admitted:false after three automated starts.
• Totals: 1691/1691 checks passed across 236/236 lanes.

Unit: the platform shards 113203095618 and 113203095418 ran the test task uncached (0 cached) and passed. Results per file are not printed (--output-logs=errors-only).

NOT_RUN

• Local tests, build and typecheck: the brief is API-only.
• Per-test execution identity of the new unit tests, because CI logs show errors only.
• A checkpoint-restore probe for B1. B1 rests on source analysis only.
• The resumed-turn capacity variant (N3).
• Checking the exact sentence and the turn.failed shape against the upstream Codex 0.160.0 source or stream.
• Live provider recovery, and any deployment observation.
• Workflow-lane behaviour under capacity (N1), which I read from source only.
• Rendering the docs pages myself. CI Docs container test and Playwright (docs) passed.
• UI states. There is no UI change; the card shows the held retry as queued.

This review is a comment, not a GitHub approval. Root keeps ownership of the PR and its merge.