diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 40486d1b240..fde48c033ad 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -121,8 +121,27 @@ labels local presets separately; those normally omit both `authMode` and `apiKey The [`retryOn429`](/reference/configuration/) same-key 429 replay applies only to API-key providers (`authMode: "key"`). OAuth, forward, and local presets are excluded — their credentials must never be replayed on the same token, and local runtimes have no remote key to -preserve. It is opt-in: when the option is absent the feature is off; object presence enables -it unless `enabled: false`. +preserve. It is opt-in except for documented destination-specific defaults. Canonical Mistral +Chat uses a patient retry policy (6 replays, 10-second fallback interval), within the +existing request-wide send allowance. Explicit `retryOn429` settings, including +`enabled: false`, take precedence. The Mistral preset also forwards supplied prompt-cache +keys by default; `promptCacheKey: false` disables forwarding. Cache affinity improves the +chance of reuse, but does not guarantee a cache hit or remove token-throughput limits. + +Mistral uses one API-key provider for both Studio and Vibe keys. Key types and plan allowances +are described in [Mistral's API-key documentation](https://docs.mistral.ai/admin/identity-access/api-keys); +the proxy does not infer a subscription tier from the key. For every model, explicit upstream +`Retry-After` deadlines are shared between calls on the same credential and model. A 429 +without a retry time can activate a conservative 60-second recovery wait when valid minute +counters show exhausted capacity, or the refused query's measured cost exceeds the remaining +tokens while fitting the full advertised allowance. This is a local recovery policy, not an +exact server-reset prediction. Healthy traffic never predicts the next query's cache or cost. +After the wait, one recovery probe runs; success restores ordinary traffic. Missing, malformed, +zero-entitlement, exhausted-monthly, or above-full-allowance evidence does not invent a reset or compact history. +`requestPacing: { "enabled": false }` or `retryOn429: { "respectRetryAfter": false }` +disables shared pacing. If Mistral's +retry time exceeds `retryOn429.maxIntervalMs`, the proxy returns the 429 and retry hint rather +than replaying early. Context windows and per-model compaction preferences remain independent. When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This applies to native Chat, Messages, Responses and compact. Waiting for replacement-key capacity does not consume the upstream response-header timeout. Per-key pacing and round-robin use guarded HTTP/SSE even when upstream WebSockets are enabled; WebSocket-only steering and injection require a route without those key policies. This does not change the saved active key. An established upstream turn keeps its credential; later sends are refused if that key cools. Pre-output key rejection can recover to a healthy key within the existing request limit. Direct Images and compact requests record genuine 401/429 cooldowns without replaying that paid POST. diff --git a/docs-site/src/content/docs/ja/guides/providers.md b/docs-site/src/content/docs/ja/guides/providers.md index f539c96b981..d09073acf35 100644 --- a/docs-site/src/content/docs/ja/guides/providers.md +++ b/docs-site/src/content/docs/ja/guides/providers.md @@ -57,7 +57,10 @@ Codex login を Pool モードで使うと、Providers の概要には任意の [`retryOn429`](/ja/reference/configuration/)（同一キーでの 429 リトライ）は API キー プロバイダー （`authMode: "key"`）のみに適用されます。OAuth・forward・ローカル プリセットは除外されます — 同じトークンを再送すべきではなく、ローカルランタイムには保存すべきリモートキーがありません。 -オプトインです: オプションが無ければ無効、オブジェクトがあれば `enabled: false` でない限り有効です。 +通常はオプトインです。正規の Mistral `openai-chat`（`https://api.mistral.ai/v1`）では、 +オプションを省略すると同一キーで最大 6 回再送し、待機間隔の既定値は 10 秒です。 +明示的な設定が優先され、`enabled: false` でこの既定の再試行を無効にできます。 +他の対象では、オブジェクトを指定すると `enabled: false` でない限り有効になります。 ## 1. ChatGPT ログイン(forward / パススルー) diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index bd1221c396e..cda4d7e0b7a 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -145,7 +145,7 @@ account を削除しても mapping は保持され、同じ id を再追加す | `parallelToolCalls?` | `boolean` |並列ツール呼び出しを切り替えます。 OpenAI Chat はデフォルトでオンになっています。非チャット アダプターは明示的な `true` でのみアドバタイズします。 | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` |正確なプレースホルダー ID、欠落している端末 ID、および（`repairInvalidIds` で）正規の `msg_`/`rs_` 接頭辞を欠く message/reasoning ID に対するダウンストリーム SSE 修復はデフォルトで無効になっています。関数呼び出し ID は決して書き換えられません。組み込み DeepSeek は最後の 2 つをデフォルトで有効にします。 | | `responsesSnapshotRepair?` | `boolean` | デフォルトで無効のクライアント向け修復です。SSE と JSON の Responses ライフサイクルで欠落した status、output、ツールメタデータを補完し、raw 検査と永続化は変更しません。 | -| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key プロバイダーのみ(`authMode: "key"`)。オプトインの同一ターゲット 429 リトライ: `retryOn429` が無ければ無効で、オブジェクトがあれば `enabled: false` でない限り有効になります。429 時に待機(上流の `Retry-After` または固定間隔)してから、キー フェイルオーバーの前に同一キーで同一リクエストを再送します — メインのテキストターン回復ループ、Responses passthrough、画像/動画ブリッジ、web-search サイドカー、ターミナル継続要求をすべてカバーします。再送の対象はプリストリームの HTTP 429 応答のみで、カスタム `runTurn` トランスポートは HTTP リトライループの対象外です。`attempts` は最初の 429 以降の同一キー再送回数(合計送信数 = `attempts` + 1)で、メインの回復ループ・ターミナルガード継続・ブリッジ再試行で共有されるリクエスト単位の予算です。`attempts` を使い切っても同一キーでの再送が止まるだけで、通常のキー フェイルオーバーまたは最終エラー処理が利用可能なターゲットに応じて続きます — キー認証の passthrough ワイヤにはフェイルオーバーがないため、使い切った 429 はそのまま返ります。Codex 自体は 429 をリトライしないため、単一キーのプロバイダーでは唯一の防御です。デフォルト: `enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(1回の待機は `maxIntervalMs` で上限、その上限は 600000)、`respectRetryAfter: true`。 | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key プロバイダーのみ(`authMode: "key"`)。同一ターゲットの 429 リトライ。正規の Mistral `openai-chat`（`https://api.mistral.ai/v1`）では、設定を省略すると同一キーで最大 6 回再送し、待機間隔の既定値は 10 秒です。明示的な設定が優先され、`enabled: false` でこの既定の再試行を無効にできます。通常、他の対象はオプトインで、オブジェクトを指定すると `enabled: false` でない限り有効になります。429 時に待機(上流の `Retry-After` または固定間隔)してから、キー フェイルオーバーの前に同一キーで同一リクエストを再送します — メインのテキストターン回復ループ、Responses passthrough、画像/動画ブリッジ、web-search サイドカー、ターミナル継続要求をすべてカバーします。再送の対象はプリストリームの HTTP 429 応答のみで、カスタム `runTurn` トランスポートは HTTP リトライループの対象外です。`attempts` は最初の 429 以降の同一キー再送回数(合計送信数 = `attempts` + 1)で、メインの回復ループ・ターミナルガード継続・ブリッジ再試行で共有されるリクエスト単位の予算です。`attempts` を使い切っても同一キーでの再送が止まるだけで、通常のキー フェイルオーバーまたは最終エラー処理が利用可能なターゲットに応じて続きます — キー認証の passthrough ワイヤにはフェイルオーバーがないため、使い切った 429 はそのまま返ります。Codex 自体は 429 をリトライしないため、単一キーのプロバイダーでは唯一の防御です。オブジェクト指定時のデフォルト: `enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(1回の待機は `maxIntervalMs` で上限、その上限は 600000)、`respectRetryAfter: true`。 | | `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | キー認証の `openai-chat` および `openai-responses` プロバイダーのみ。`authMode: "forward"` のプロバイダー（ChatGPT アカウントプール）はこのオプションを読まず、既定の再試行段数を維持します。ストリーム開始前に上流から返される一時的なステータス（500、502、503、504、520、521、522）に対するオプトインの再試行です。設定がなければ無効で、オブジェクトを指定すると `enabled: false` でない限り有効になります。最初の Responses リクエスト、ターミナルガード継続、ネイティブの `/v1/chat/completions`、および 429／アカウント回復時の再取得が対象です。`attempts` は最初の送信を含め、1 回のリクエストで許可される上流への送信総数です（1～10、デフォルトは 3）。接続リセット回復と共有するリクエスト単位の単一予算であるため、`3` を指定した場合、プロバイダーに到達する実リクエストは最大 3 回です。待機には 400 ms を基準とする固定式の指数バックオフを使用し、上限は 5 秒で、`Retry-After` に従います。レート制限を扱う `retryOn429` とは別の機能であり、ストリーム開始後の失敗は再送されません。 各 Combo ターゲットでは、事前予約した初回送信もそのターゲットの設定総数に含まれます。`attempts: 1` は 1 回だけ送信し、予約を返却して余裕を作ったり、共有上限を広げたりしません。送信前のローカル拒否やキャンセルでは未使用の予約を解放します。 | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | ネイティブ `openai-responses` プロバイダー（`authMode: "forward"` を含む）と、上流ヘッダー受信前の汎用 Responses 変換経路に適用されます。呼び出し側が何も観測しないまま失敗した送信を、オプトインで置き換えます。設定がなければ無効で、オブジェクトを指定すると `enabled: false` でない限り有効になります。レスポンスヘッダーが届く前に接続が切れた場合と、ヘッダー後に SSE 本文が制御イベントだけを運んだまま切れた場合の両方が対象です。canonical ChatGPT upstream WebSocket で create フレームの送信後、Responses イベントが届く前にソケットが閉じたかエラーになった場合も同じく対象で、その置き換えは HTTP で送信します。置き換えるのは自己完結したリクエストだけで、`store: false`、完全な `input`、`previous_response_id` / `conversation` / `stream_id` がないこと、クライアントが実行するツールのみ、が条件です。`replacements` は、すべてのレッグとすべてのコンボ子リクエストを合わせて 1 つの論理リクエストが行える置き換え送信の回数です（1..2、デフォルトは 1）。レッグ単位の再試行回数でも送信予算でもないため、置き換え送信もそのレッグがすでに持つ送信許容量に収まる必要があります。すでに出力やツール呼び出しを送ったリクエストは、この値に関わらず置き換えません。元の送信がすでに開始されていた場合は置き換えた推論も課金される可能性があるため、既定では無効です。 汎用変換経路では、初回送信と再構築後の送信のどちらも、同じ許可と送信予算でヘッダー受信前の接続リセットを置き換えます。アダプター独自のトランスポートと、上流ヘッダー受信後の変換ストリーム障害は対象外です。 | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` が `auto` または `none` のみを受け入れるモデル。強制的な選択は格下げされます。 | diff --git a/docs-site/src/content/docs/ko/guides/providers.md b/docs-site/src/content/docs/ko/guides/providers.md index 90e504c9206..37c06b8fbd3 100644 --- a/docs-site/src/content/docs/ko/guides/providers.md +++ b/docs-site/src/content/docs/ko/guides/providers.md @@ -55,8 +55,10 @@ shipped v1 config는 marker 2의 단일 옵션 행으로 자동 이관됩니다. [`retryOn429`](/ko/reference/configuration/)（동일 키 429 재시도）는 API 키 프로바이더 （`authMode: "key"`）에만 적용됩니다. OAuth·forward·로컬 프리셋은 제외됩니다 — 같은 토큰을 -재전송해서는 안 되며, 로컬 런타임에는 보존할 원격 키가 없습니다. 옵트인입니다: 옵션이 없으면 -꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. +재전송해서는 안 되며, 로컬 런타임에는 보존할 원격 키가 없습니다. 일반적으로 명시적 설정이 필요합니다. 정식 Mistral `openai-chat` 엔드포인트 +(`https://api.mistral.ai/v1`)는 옵션이 없으면 동일 키로 최대 6회 재전송하며 기본 대기 간격은 10초입니다. +명시적 설정이 우선하고, `enabled: false`는 이 기본 재시도를 끕니다. +다른 대상은 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. ## 1. ChatGPT 로그인 (forward / 패스스루) diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index 010b1df4ccd..b223cf2ccae 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -145,7 +145,7 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | `parallelToolCalls?` | `boolean` | 병렬 도구 호출을 켜거나 끕니다. OpenAI Chat은 기본으로 켜져 있고, 비-chat 어댑터는 명시적으로 `true`일 때만 이를 노출합니다. | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | 기본값이 꺼진 downstream SSE 복구입니다. 정확한 자리표시자 id, 누락된 종료 id, 그리고(`repairInvalidIds`) 정규 `msg_`/`rs_` 접두사가 없는 message/reasoning id를 복구합니다. function-call id는 다시 쓰지 않습니다. 내장 DeepSeek은 마지막 두 가지를 기본으로 켭니다. | | `responsesSnapshotRepair?` | `boolean` | 기본값이 꺼진 클라이언트용 복구입니다. SSE와 JSON의 Responses 수명 주기에서 누락된 status, output, 도구 메타데이터를 채우며 raw 검사와 영속화는 변경하지 않습니다. | -| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key 프로바이더 전용(`authMode: "key"`). 동일 대상 429 재시도: `retryOn429`가 없으면 기능이 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 429 시 대기(업스트림 `Retry-After` 또는 고정 간격) 후 키 장애 조치 전에 동일 키로 동일 요청을 재전송합니다 — 일반 텍스트 턴 복구 루프, Responses passthrough, 이미지/비디오 브리지, web-search 사이드카, 터미널 연속 요청을 모두 포함합니다. 재전송 대상은 프리스트림 HTTP 429 응답뿐이며, 커스텀 `runTurn` 전송은 HTTP 재시도 루프에서 제외됩니다. `attempts`는 첫 429 이후의 동일 키 재전송 횟수(총 전송 = `attempts` + 1)이며, 메인 복구 루프·터미널 가드 연속 요청·브리지 재시도가 공유하는 요청 단위 예산입니다. `attempts`를 모두 소진해도 동일 키 재전송만 중단되며, 이후에는 일반 키 장애 조치 또는 최종 오류 처리가 사용 가능한 대상에 따라 진행됩니다 — 키 인증 passthrough 와이어에는 장애 조치가 없으므로 소진된 429가 그대로 반환됩니다. Codex 자체는 429를 재시도하지 않으므로 단일 키 프로바이더의 유일한 방어선입니다. 기본값: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000`(단일 대기는 `maxIntervalMs`로 상한, 그 자체는 600000으로 상한), `respectRetryAfter: true`. | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key 프로바이더 전용(`authMode: "key"`). 동일 대상 429 재시도입니다. 정식 Mistral `openai-chat` 엔드포인트(`https://api.mistral.ai/v1`)는 옵션이 없으면 동일 키로 최대 6회 재전송하며 기본 대기 간격은 10초입니다. 명시적 설정이 우선하고, `enabled: false`는 이 기본 재시도를 끕니다. 다른 대상은 일반적으로 명시적 설정이 필요하며, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 429 시 대기(업스트림 `Retry-After` 또는 고정 간격) 후 키 장애 조치 전에 동일 키로 동일 요청을 재전송합니다 — 일반 텍스트 턴 복구 루프, Responses passthrough, 이미지/비디오 브리지, web-search 사이드카, 터미널 연속 요청을 모두 포함합니다. 재전송 대상은 프리스트림 HTTP 429 응답뿐이며, 커스텀 `runTurn` 전송은 HTTP 재시도 루프에서 제외됩니다. `attempts`는 첫 429 이후의 동일 키 재전송 횟수(총 전송 = `attempts` + 1)이며, 메인 복구 루프·터미널 가드 연속 요청·브리지 재시도가 공유하는 요청 단위 예산입니다. `attempts`를 모두 소진해도 동일 키 재전송만 중단되며, 이후에는 일반 키 장애 조치 또는 최종 오류 처리가 사용 가능한 대상에 따라 진행됩니다 — 키 인증 passthrough 와이어에는 장애 조치가 없으므로 소진된 429가 그대로 반환됩니다. Codex 자체는 429를 재시도하지 않으므로 단일 키 프로바이더의 유일한 방어선입니다. 객체를 지정했을 때의 기본값: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000`(단일 대기는 `maxIntervalMs`로 상한, 그 자체는 600000으로 상한), `respectRetryAfter: true`. | | `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | 키 인증 `openai-chat` 및 `openai-responses` 프로바이더 전용입니다. `authMode: "forward"` 프로바이더(ChatGPT 계정 풀)는 이 옵션을 읽지 않고 기본 재시도 단계를 유지합니다. 스트림 시작 전의 일시적인 업스트림 상태(500, 502, 503, 504, 520, 521, 522)를 선택적으로 재시도합니다. 이 옵션이 없으면 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 최초 Responses 요청, 터미널 가드 연속 요청, 네이티브 `/v1/chat/completions`, 429/계정 복구 재조회를 포함합니다. `attempts`는 최초 전송을 포함하여 요청 하나에 허용되는 업스트림 전송의 총횟수(1..10, 기본값 3)입니다. 연결 재설정 복구와 요청 단위 예산 하나를 공유하므로 `3`이면 실제로 프로바이더에 도달하는 요청은 최대 세 번입니다. 대기에는 400ms로 고정된 지수 백오프를 사용하고 상한은 5초이며 `Retry-After`를 따릅니다. 속도 제한을 처리하는 `retryOn429`와는 별개이며, 스트림 도중의 실패는 절대 재전송하지 않습니다. 각 Combo 대상은 선예약한 첫 전송을 해당 대상의 설정 총횟수에 포함합니다. `attempts: 1`은 한 번만 전송하며, 예약을 환불해 여유를 만들거나 공유 상한을 늘리지 않습니다. 전송 전 로컬 거부나 취소는 미사용 예약을 해제합니다. | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | 네이티브 `openai-responses` 프로바이더(`authMode: "forward"` 포함)와 업스트림 헤더 수신 전의 일반 Responses 변환 전송 경로에 적용됩니다. 호출자가 아무것도 관측하지 못한 채 실패한 전송을 선택적으로 대체합니다. 이 옵션이 없으면 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 응답 헤더가 오기 전에 연결이 끊어진 경우와, 헤더 이후 SSE 본문이 제어 이벤트만 실은 채 끊어진 경우를 모두 다룹니다. canonical ChatGPT 업스트림 WebSocket에서 create 프레임을 보낸 뒤 Responses 이벤트가 오기 전에 소켓이 닫히거나 오류가 난 경우도 같은 방식으로 다루며, 이때 대체 전송은 HTTP로 보냅니다. 자체 완결된 요청만 대체합니다. `store: false`, 완전한 `input`, `previous_response_id`·`conversation`·`stream_id` 없음, 클라이언트가 실행하는 도구만 해당합니다. `replacements`는 모든 구간과 모든 콤보 자식을 합쳐 논리 요청 하나가 만들 수 있는 대체 전송 횟수입니다(1..2, 기본값 1). 구간별 재시도 횟수도 전송 예산도 아니므로, 대체 전송도 해당 구간이 이미 가진 전송 허용량 안에 들어가야 합니다. 이미 출력이나 도구 호출을 내보낸 요청은 이 값과 무관하게 대체하지 않습니다. 원본 전송이 이미 시작됐다면 대체한 추론도 과금될 수 있어서 기본값은 꺼짐입니다. 일반 변환 전송 경로에서는 최초 전송과 재구축한 전송 모두 같은 대체 권한과 전송 예산으로 헤더 수신 전 연결 리셋을 대체합니다. 어댑터가 자체 관리하는 전송 경로와 업스트림 헤더 수신 후 변환 스트림 장애는 제외됩니다. | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice`가 `auto` 또는 `none`만 받는 모델입니다. 강제 선택은 낮은 수준으로 바뀝니다. | diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 3752356b52e..83faaae76d1 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -281,7 +281,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `fastEnabled?` | `boolean` | Operator switch for the provider's Fast lane. `false` turns Fast off (no Fast toggle, no `--fast` row, no fast wire field) and overrides `supportsServiceTier`. `true` enables a lane the registry marks opt-in. Absent keeps the registry default: off for `anthropic` and `anthropic-apikey`, whose fast mode spends usage credits at 2x price, and unchanged for every other provider. The dashboard Models page shows an Off/On row for opt-in providers. | | `modelSupportsServiceTier?` | `Record` | Exact upstream model capability overrides. Exact `true` enables canonical Fast for that model; exact `false` narrows provider defaults. An explicit provider-level `supportsServiceTier: false` remains fail-closed and cannot be reopened. Exact `true` does not authorize foreign caller-tier forwarding on Chat. Undeclared models fall back to provider-wide behavior. Management `PATCH /api/providers` merges entries and accepts `null` to clear one. | | `chatServiceTier?` | `boolean` | Provider-wide Chat-wire opt-in for forwarding caller `service_tier` values. On a classified route it governs foreign values such as `flex`, not proxy-owned canonical Fast after capability validation; on an unclassified route it governs every caller value because no Fast capability has been validated. Exact model capability does not authorize foreign forwarding. Responses routes retain their capability-based caller forwarding behavior. | -| `promptCacheKey?` | `boolean` | Provider-wide `openai-chat` opt-in for forwarding a `prompt_cache_key`. The adapter forwards the key it is given and never invents one, but the key is not always the caller's: Claude Messages translation derives one from `metadata.user_id`, or from a model/system/tools cohort when no metadata is sent. Default off. Enable only when the upstream documents support, because strict gateways may reject the unknown field with HTTP 400. | +| `promptCacheKey?` | `boolean` | Provider-wide `openai-chat` opt-in for forwarding a `prompt_cache_key`. The adapter forwards the key it is given and never invents one, but the key is not always the caller's: Claude Messages translation derives one from `metadata.user_id`, or from a model/system/tools cohort when no metadata is sent. Default off unless declared by the provider preset; Mistral enables it per its [prompt-caching contract](https://docs.mistral.ai/studio/conversations/advanced/prompt-caching), and explicit `false` disables it. Enable only when the upstream documents support, because strict gateways may reject the unknown field with HTTP 400. | | `preserveResponsesReasoningContent?` | `boolean` | Default off for custom providers. In **Providers → Settings**, enable **Replay plaintext reasoning** for a custom Responses endpoint only if it accepts this history. This sends prior plaintext reasoning to that endpoint and may increase token use. Keep plaintext reasoning content on replayed Responses reasoning items instead of blanking it, including custom native v1 compact input (blanking is the ChatGPT backend's rule). Enable for upstreams whose contract accepts reasoning replay, such as DeepSeek or a self-hosted Kimi on vLLM. Proxy-minted `ocxr1` envelopes are always stripped. A hand-configured `openai-responses` provider without it logs one warning per host the first time plaintext reasoning is dropped. `preserveReasoningContentModels` does not affect this wire. | | `preserveResponsesInputItemIds?` | `boolean` | Default off. Preserve valid `input[].id` when `store: false` for explicitly trusted launcher relays that require turn identity. Ordinary Responses providers keep stripping IDs; xAI custom-call ID repair remains active. | | `preserveResponsesMessageMetadata?` | `boolean` | Default off. Preserve private `internal_chat_message_metadata_passthrough` on input items for explicitly trusted launcher relays, including custom native v1 compact requests. This forwards private turn/session metadata to that destination; do not enable for ordinary gateways. | @@ -349,7 +349,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | Disabled-by-default downstream SSE repair for exact placeholder ids, missing terminal ids, and (with `repairInvalidIds`) message/reasoning ids missing the canonical `msg_`/`rs_` prefix. Function-call ids are never rewritten. Built-in DeepSeek enables the last two by default. | | `responsesSnapshotRepair?` | `boolean` | Disabled-by-default client-facing repair for sparse Responses lifecycle snapshots in SSE and JSON. Fills missing canonical status, output, and tool metadata while raw inspection and persistence remain unchanged. | | `webSearchBridge?` | `{ enabled?: boolean; backend?: "ollama" \| "openai" \| "anthropic" \| "xai" \| "gemini" \| "exa"; maxSearches?: number; timeoutMs?: number; endpoint?: string }` | Key-auth `openai-responses` passthrough providers only. Off by default. Codex always declares the hosted `web_search` tool, and the passthrough relays it on the assumption the destination executes it. A gateway that does not run hosted search answers with a `function_call` named `web_search` that nothing runs, and the undeclared-tool guard ends the turn. With `enabled: true` and an explicit `backend` OpenCodex intercepts that call, runs the search itself, feeds the result back to the same upstream, and shows Codex a hosted `web_search_call` cell. Never armed for `authMode: "forward"` (ChatGPT already searches) or for a provider that executes hosted search upstream. `backend` is required; there is no implicit default and a missing credential for the named backend leaves the bridge disarmed rather than falling through to another paid search. `ollama` reuses this provider's own API key on `POST /api/web_search`, so the origin must be `https://ollama.com` unless the operator names `endpoint` explicitly. `openai` / `anthropic` / `xai` / `gemini` / `exa` reuse the matching sidecar executor and that executor's own credential (`webSearchSidecar.exaApiKey` for Exa). The search model comes from `webSearchSidecar.model` only when `webSearchSidecar.backend` resolves to the same backend this bridge names; otherwise the bridge runs that backend's own default, because a model chosen for one vendor is rejected by another. An unset `webSearchSidecar.backend` resolves to `openai`, so an unset-backend model reaches an `openai` bridge and no other. There is no per-provider bridge model override. Streaming turns only. A turn that mixes `web_search` with another client tool call still fails closed rather than dropping the client's call. Assistant text such as XML-like `` prose is not executed. Defaults: `maxSearches: 3` (1..10), `timeoutMs: 60000` (1000..600000). | -| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key providers only (`authMode: "key"`). Opt-in same-target 429 retry: when `retryOn429` is absent the feature is off, except that key-auth OpenCode Go and Command Code (canonical endpoints) fall back to a patient policy (6 replays, 10 s interval, 60 s cap); object presence enables it unless `enabled: false`, which also turns that fallback off. On 429 the proxy waits (upstream `Retry-After` or the fixed interval) and replays the identical request on the same key before any key failover — across the main text-turn recovery loop, the Responses passthrough wire, the image/video bridge, the web-search sidecar, and terminal continuations. Only pre-stream HTTP 429 responses are eligible for replay; custom `runTurn` transports are outside the HTTP retry loop. `attempts` counts same-key replays after the first 429 (total sends = `attempts` + 1) and is one request-wide budget shared by the main recovery loop, the terminal-guard continuation, and bridge retries. Exhausting `attempts` only stops further same-key replays: normal key failover or final-error handling then applies per the available targets — on the key-auth passthrough wire there is no failover, so the exhausted 429 surfaces as-is. Codex itself never retries 429, so this is the only defense for single-key providers. Defaults: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (any single wait is capped at `maxIntervalMs`, itself capped at 600000), `respectRetryAfter: true`. | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key providers only (`authMode: "key"`). Opt-in same-target 429 retry: when `retryOn429` is absent the feature is off, except that key-auth OpenCode Go and Command Code (canonical endpoints) fall back to a patient policy (6 replays, 10 s interval, 60 s cap), and canonical Mistral `openai-chat` falls back to 6 replays with a 10 s interval and 60 s cap; object presence enables it unless `enabled: false`, which also turns that fallback off. On 429 the proxy waits (upstream `Retry-After` or the fixed interval) and replays the identical request on the same key before any key failover — across the main text-turn recovery loop, the Responses passthrough wire, the image/video bridge, the web-search sidecar, and terminal continuations. Only pre-stream HTTP 429 responses are eligible for replay; custom `runTurn` transports are outside the HTTP retry loop. `attempts` counts same-key replays after the first 429 (total sends = `attempts` + 1) and is one request-wide budget shared by the main recovery loop, the terminal-guard continuation, and bridge retries. Exhausting `attempts` only stops further same-key replays: normal key failover or final-error handling then applies per the available targets — on the key-auth passthrough wire there is no failover, so the exhausted 429 surfaces as-is. For canonical Mistral Chat, an explicit upstream delay above the allowed wait stops replay and preserves the retry hint; other providers retain the capped-delay behavior. Mistral shares retry deadlines per credential/model unless `requestPacing.enabled: false` or `retryOn429.respectRetryAfter: false`. A 429 with validated minute-budget pressure and no retry time activates a conservative 60 s recovery wait, followed by one probe; success restores ordinary traffic. Healthy traffic never predicts the next token charge. This policy uses advertised allowances, excludes known exhausted monthly allowances, does not claim an exact reset time, and never triggers compaction. Codex itself never retries 429, so this is the only defense for single-key providers. Defaults: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (any single wait is capped at `maxIntervalMs`, itself capped at 600000), `respectRetryAfter: true`. | | `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Key-auth `openai-chat` and `openai-responses` providers only. `authMode: "forward"` providers (the ChatGPT account pool) never read this option and keep the default ladder. Opt-in retry for pre-stream transient upstream statuses (500, 502, 503, 504, 520, 521, 522): absent means off, object presence enables it unless `enabled: false`. Covers the initial Responses request, the Responses passthrough lane and each of its recovery legs (OAuth-401 replay, same-target 429 replay, validated rebuild), the terminal-guard continuation, and native `/v1/chat/completions`. `attempts` is the TOTAL number of upstream sends allowed for one request including the first (1..10, default 3) — it is one budget shared with connection-reset recovery, so `3` means at most three real requests reach the provider. On the Responses passthrough lane the configured value is additionally intersected with the request-wide send allowance, so a value below that allowance narrows the ladder exactly while a value above it does not raise the bound. Each Combo target includes its prepaid first send in that target's configured total; `attempts: 1` sends once, without refunding the reservation or widening the shared Combo ceiling. A local refusal or cancellation before dispatch releases an unsent booking. Waits use a fixed 400 ms exponential backoff capped at 5 s and honor `Retry-After`. Separate from `retryOn429`, which handles rate limiting; mid-stream failures are never replayed. | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | Native `openai-responses` providers, including `authMode: "forward"`, plus generic translated Responses dispatch before upstream headers. Opt-in replacement of a send that failed while the caller had observed nothing: absent means off, object presence enables it unless `enabled: false`. Covers both ambiguous stages — a connection that died before any response header, and an SSE body that died after the header while carrying only control events. A canonical ChatGPT upstream WebSocket that closed or errored after its create frame left, before any Responses event, is covered the same way, and its replacement is sent over HTTP. Only a self-contained request is ever replaced: `store: false`, complete `input`, no `previous_response_id`, `conversation` or `stream_id`, and only client-executed tools. `replacements` is the number of replacement sends ONE logical request may make across every leg and every combo child (1..2, default 1) — not a per-leg retry count and not a send budget, so a replacement still has to fit inside the send allowance the leg already had. A request that already emitted output or a tool call is never replaced, whatever this is set to. The replacement inference may still be billed if the origin had already started the first one, which is why this is off by default. Generic translated dispatch supports pre-header reset replacement for initial and rebuilt sends under the same grant and send budget. Adapter-owned transports and translated stream failures after upstream headers are excluded. | | `autoToolChoiceOnlyModels?` | `string[]` | Models whose `tool_choice` accepts only `auto` or `none`; forced choices are downgraded. | diff --git a/docs-site/src/content/docs/ru/guides/providers.md b/docs-site/src/content/docs/ru/guides/providers.md index 6234f0f3e46..d78a02b0ae5 100644 --- a/docs-site/src/content/docs/ru/guides/providers.md +++ b/docs-site/src/content/docs/ru/guides/providers.md @@ -65,8 +65,10 @@ description: Все способы, которыми opencodex аутентиф Повтор при 429 на том же ключе ([`retryOn429`](/ru/reference/configuration/)) применим только к провайдерам с API-ключом (`authMode: "key"`). Пресеты OAuth, forward и local исключены — их учётные данные нельзя повторно отправлять по тому же токену, а у локальных сред выполнения нет -удалённого ключа. Это opt-in: при отсутствии опции функция выключена; наличие объекта включает -её, если только `enabled: false`. +удалённого ключа. Обычно требуется явная настройка. Для канонического Mistral `openai-chat` +(`https://api.mistral.ai/v1`) отсутствие опции разрешает до шести повторов с тем же ключом. +Интервал по умолчанию — 10 секунд. Явная настройка имеет приоритет; `enabled: false` +отключает эти повторы. Для остальных целей объект включает функцию, если не задано `enabled: false`. ## 1. Вход через ChatGPT (forward / passthrough) diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index fbca4a5d274..ee1dda01f3e 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -158,7 +158,7 @@ cross-route credential fallback не существует. Строки API GPT- | `parallelToolCalls?` | `boolean` | Переключатель parallel tool call'ов. Для OpenAI Chat по умолчанию включено; не-chat adapter'ы рекламируют это только при явном `true`. | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | По умолчанию выключенная downstream SSE-repair для exact placeholder-id, отсутствующих terminal-id и (с `repairInvalidIds`) message/reasoning id без канонического префикса `msg_`/`rs_`. Function-call id никогда не переписываются. Встроенный DeepSeek включает последние два по умолчанию. | | `responsesSnapshotRepair?` | `boolean` | По умолчанию выключенная клиентская repair для неполных lifecycle snapshot'ов Responses в SSE и JSON. Добавляет отсутствующие status, output и tool metadata, не меняя raw inspection и persistence. | -| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Только для провайдеров с API-ключом (`authMode: "key"`). Опциональный повтор при 429 на том же таргете: если `retryOn429` отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. При 429: ожидание (`Retry-After` апстрима или фиксированный интервал) и повтор идентичного запроса на том же ключе до любого фейловера ключей — покрывает основной цикл восстановления текстовых ходов, passthrough-канал Responses, мост изображений/видео, sidecar web-search и терминальные продолжения. Повтор допустим только для HTTP 429, полученных до начала потока; пользовательские транспорты `runTurn` не входят в цикл HTTP-повторов. `attempts` — это число повторов на том же ключе после первого 429 (всего отправок = `attempts` + 1) и единый бюджет на запрос, общий для основного цикла восстановления, терминального продолжения и повторов моста. Исчерпание `attempts` лишь останавливает дальнейшие повторы на том же ключе; далее применяется обычный фейловер ключей или финальная обработка ошибки в зависимости от доступных таргетов — на passthrough-канале с ключевой аутентификацией фейловера нет, поэтому исчерпанный 429 возвращается как есть. Codex сам никогда не повторяет 429, поэтому это единственная защита для провайдеров с одним ключом. По умолчанию: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (любое ожидание ограничено `maxIntervalMs`, который сам ограничен 600000), `respectRetryAfter: true`. | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Только для провайдеров с API-ключом (`authMode: "key"`). Повтор при 429 на том же таргете. Для канонического Mistral `openai-chat` (`https://api.mistral.ai/v1`) отсутствие опции разрешает до шести повторов с тем же ключом с интервалом по умолчанию 10 секунд. Явная настройка имеет приоритет; `enabled: false` отключает эти повторы. Для остальных целей обычно требуется явная настройка: объект включает функцию, если не задано `enabled: false`. При 429: ожидание (`Retry-After` апстрима или фиксированный интервал) и повтор идентичного запроса на том же ключе до любого фейловера ключей — покрывает основной цикл восстановления текстовых ходов, passthrough-канал Responses, мост изображений/видео, sidecar web-search и терминальные продолжения. Повтор допустим только для HTTP 429, полученных до начала потока; пользовательские транспорты `runTurn` не входят в цикл HTTP-повторов. `attempts` — это число повторов на том же ключе после первого 429 (всего отправок = `attempts` + 1) и единый бюджет на запрос, общий для основного цикла восстановления, терминального продолжения и повторов моста. Исчерпание `attempts` лишь останавливает дальнейшие повторы на том же ключе; далее применяется обычный фейловер ключей или финальная обработка ошибки в зависимости от доступных таргетов — на passthrough-канале с ключевой аутентификацией фейловера нет, поэтому исчерпанный 429 возвращается как есть. Codex сам никогда не повторяет 429, поэтому это единственная защита для провайдеров с одним ключом. Значения по умолчанию при явном объекте: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (любое ожидание ограничено `maxIntervalMs`, который сам ограничен 600000), `respectRetryAfter: true`. | | `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Только для провайдеров `openai-chat` и `openai-responses` с аутентификацией по ключу. Провайдеры с `authMode: "forward"` (пул аккаунтов ChatGPT) никогда не читают эту настройку и сохраняют число повторов по умолчанию. Опциональный повтор при временных статусах апстрима до начала потока (500, 502, 503, 504, 520, 521, 522): если параметр отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. Покрывает исходный запрос `Responses`, продолжение терминального предохранителя, нативный `/v1/chat/completions`, а также повторные запросы при восстановлении после 429 или ошибки учётной записи. `attempts` — ОБЩЕЕ число разрешённых отправок в апстрим для одного запроса, включая первую (1..10, по умолчанию 3). Это единый бюджет на запрос, общий с восстановлением после сброса соединения, поэтому `3` означает, что до провайдера дойдут не более трёх реальных запросов. Ожидание использует экспоненциальную задержку с фиксированной начальной величиной 400 мс, ограниченную 5 с, и учитывает `Retry-After`. Параметр не связан с `retryOn429`, который обрабатывает ограничение частоты запросов; сбои после начала потока никогда не воспроизводятся. Для каждой цели Combo первая заранее зарезервированная отправка входит в настроенный итог этой цели: `attempts: 1` допускает одну отправку, без возврата резерва ради дополнительного лимита и без расширения общего потолка. Локальный отказ или отмена до отправки освобождает неиспользованный резерв. | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | Для нативных провайдеров `openai-responses`, включая `authMode: "forward"`, а также для общего пути отправки переведённых запросов Responses до получения заголовков от upstream. Необязательная замена отправки, которая завершилась неудачей, когда вызывающая сторона ещё ничего не наблюдала: если параметр отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. Покрывает обе неоднозначные стадии — обрыв соединения до любого заголовка ответа и обрыв тела SSE после заголовка, когда оно несло только управляющие события. Так же покрывается upstream WebSocket canonical ChatGPT, который закрылся или завершился ошибкой после отправки кадра create и до любого события Responses; замена в этом случае отправляется по HTTP. Заменяется только самодостаточный запрос: `store: false`, полный `input`, отсутствие `previous_response_id`, `conversation` и `stream_id`, и только инструменты, исполняемые клиентом. `replacements` — число замещающих отправок, которые ОДИН логический запрос может сделать по всем участкам и всем дочерним запросам комбо (1..2, по умолчанию 1). Это не число повторов на участок и не бюджет отправок, поэтому замена всё равно должна поместиться в уже имеющийся у участка лимит отправок. Запрос, который уже выдал вывод или вызов инструмента, не заменяется никогда, каким бы ни было это значение. Замещающий вывод модели всё равно может быть оплачен, если источник уже начал первый, поэтому параметр выключен по умолчанию. Общий путь перевода поддерживает замену при сбросе соединения до заголовков для первоначальных и заново сформированных отправок, используя то же разрешение и бюджет отправок. Транспорты, управляемые самими адаптерами, и сбои переведённого потока после получения заголовков от upstream исключены. | | `autoToolChoiceOnlyModels?` | `string[]` | Модели, у которых `tool_choice` принимает только `auto` или `none`; forced choice понижается. | diff --git a/docs-site/src/content/docs/zh-cn/guides/providers.md b/docs-site/src/content/docs/zh-cn/guides/providers.md index c6ef72b88a3..df3200ac69c 100644 --- a/docs-site/src/content/docs/zh-cn/guides/providers.md +++ b/docs-site/src/content/docs/zh-cn/guides/providers.md @@ -52,7 +52,9 @@ shipped v1 配置自动迁移到 marker 2 的单一选项行。原配置只保 [`retryOn429`](/zh-cn/reference/configuration/)（同 key 的 429 重试）仅适用于 API-key 提供商 （`authMode: "key"`）。OAuth、forward 与本地预设均被排除——同一 token 绝不可重放，本地运行时 -也没有需要保留的远程 key。仅在配置后启用，默认关闭；配置了对象即启用，除非 `enabled: false`。 +也没有需要保留的远程 key。通常需要显式配置。对于标准 Mistral `openai-chat` 端点（`https://api.mistral.ai/v1`）， +未配置此选项时会使用同一 key 最多重放 6 次，默认等待间隔为 10 秒。 +显式配置优先；`enabled: false` 会禁用此默认重试。其他目标配置对象后即启用，除非 `enabled: false`。 ## 1. ChatGPT 登录（forward / 透传） diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index de7d9c61871..ccca6067848 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -147,7 +147,7 @@ selector，而不是分配一个新名称。 | `parallelToolCalls?` | `boolean` | 切换并行工具调用。OpenAI Chat 默认开启；非 chat 适配器只有显式 `true` 时才会声明支持。 | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | 默认关闭的下游 SSE 修复，用于精确占位 id、缺失的终止 id，以及（`repairInvalidIds`）缺少规范 `msg_`/`rs_` 前缀的 message/reasoning id。function-call id 永远不会被重写。内置 DeepSeek 默认启用后两项。 | | `responsesSnapshotRepair?` | `boolean` | 默认关闭的客户端修复，用于补全 SSE 与 JSON 中稀疏 Responses 生命周期快照缺失的 status、output 和工具元数据；原始检查与持久化保持不变。 | -| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | 仅限 API-key 提供商（`authMode: "key"`）。可选的同目标 429 重试：未配置 `retryOn429` 时功能关闭；对象存在即启用，除非 `enabled: false`。收到 429 时等待（上游 `Retry-After` 或固定间隔）后在相同 key 上重放完全相同请求，再进入任何 key 故障转移——覆盖主文本恢复循环、Responses passthrough、图像/视频桥、web-search 侧车与终结续接。重放仅适用于流开始前的 HTTP 429 响应；自定义 `runTurn` 传输不在 HTTP 重试循环范围内。`attempts` 是首个 429 之后的同 key 重放次数（总发送次数 = `attempts` + 1），是主恢复循环、终结守卫续接与桥接重试共享的按请求统一预算；`attempts` 耗尽只会停止进一步的同 key 重放：随后按可用目标进行正常的 key 故障转移或最终错误处理——key 认证的 passthrough 线路上没有故障转移，因此耗尽的 429 会原样透出。Codex 自身从不重试 429，因此这是单 key 提供商唯一的防线。默认值：`enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`（单次等待以 `maxIntervalMs` 为上限，其本身上限 600000）、`respectRetryAfter: true`。 | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | 仅限 API-key 提供商（`authMode: "key"`）。同目标 429 重试：对于标准 Mistral `openai-chat` 端点（`https://api.mistral.ai/v1`），未配置此选项时会使用同一 key 最多重放 6 次，默认等待间隔为 10 秒。显式配置优先；`enabled: false` 会禁用此默认重试。其他目标通常需要显式配置；对象存在即启用，除非 `enabled: false`。收到 429 时等待（上游 `Retry-After` 或固定间隔）后在相同 key 上重放完全相同请求，再进入任何 key 故障转移——覆盖主文本恢复循环、Responses passthrough、图像/视频桥、web-search 侧车与终结续接。重放仅适用于流开始前的 HTTP 429 响应；自定义 `runTurn` 传输不在 HTTP 重试循环范围内。`attempts` 是首个 429 之后的同 key 重放次数（总发送次数 = `attempts` + 1），是主恢复循环、终结守卫续接与桥接重试共享的按请求统一预算；`attempts` 耗尽只会停止进一步的同 key 重放：随后按可用目标进行正常的 key 故障转移或最终错误处理——key 认证的 passthrough 线路上没有故障转移，因此耗尽的 429 会原样透出。Codex 自身从不重试 429，因此这是单 key 提供商唯一的防线。显式配置对象的默认值：`enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`（单次等待以 `maxIntervalMs` 为上限，其本身上限 600000）、`respectRetryAfter: true`。 | | `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | 仅限使用 key 认证的 `openai-chat` 与 `openai-responses` 提供商。`authMode: "forward"` 的提供商（ChatGPT 账号池）从不读取此选项，保持默认重试次数。可选的流开始前上游瞬态状态码（500、502、503、504、520、521、522）重试：未配置时关闭；对象存在即启用，除非 `enabled: false`。覆盖初始 Responses 请求、终结守卫续接、原生 `/v1/chat/completions`，以及 429/账户恢复重新获取。`attempts` 是单个请求允许向上游发送的总次数，包含首次发送（1..10，默认 3）；它是与连接重置恢复共享的按请求预算，因此 `3` 表示最多只有三个实际请求到达提供商。等待采用固定 400 毫秒的指数退避，上限为 5 秒，并遵循 `Retry-After`。此选项独立于处理速率限制的 `retryOn429`；流开始后的故障绝不会重放。 每个 Combo 目标都把预先预约的首次发送计入该目标配置的总次数；`attempts: 1` 只发送一次，不会退还预约来增加额度，也不会提高共享上限。发送前的本地拒绝或取消会释放未使用的预约。 | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | 适用于原生 `openai-responses` 提供商（包含 `authMode: "forward"`），以及收到上游响应头之前的通用 Responses 转换发送路径。可选地替换一次在调用方尚未观察到任何内容时就失败的发送：未配置时关闭；对象存在即启用，除非 `enabled: false`。涵盖两个不确定阶段——响应头到达前连接断开，以及响应头之后 SSE 正文只承载控制事件时断开。canonical ChatGPT 上游 WebSocket 在 create 帧发出之后、任何 Responses 事件到达之前关闭或出错时，也按同样方式处理，其替换发送走 HTTP。只有自包含的请求才会被替换：`store: false`、完整的 `input`、没有 `previous_response_id`／`conversation`／`stream_id`，且只使用由客户端执行的工具。`replacements` 是单个逻辑请求在所有环节和所有组合子请求中可以进行的替换发送次数（1..2，默认 1）；它既不是按环节的重试次数，也不是发送预算，因此替换发送仍必须落在该环节已有的发送额度之内。已经产生输出或工具调用的请求，无论此值为何都不会被替换。如果上游已经开始了第一次推理，被替换的推理仍可能计费，因此该选项默认关闭。 通用转换发送路径的首次发送和重建后的发送都支持响应头之前连接重置后的替换，使用同一授权和发送预算。适配器自行管理的传输，以及收到上游响应头之后的转换流故障，均不在此选项范围内。 | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` 只接受 `auto` 或 `none` 的模型；强制选择会被降级。 | diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index d9502640531..ad75b889082 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1460,6 +1460,7 @@ "qwen38-preserve-reasoning.test.ts": "providers", "rate-limit-reset-credits.test.ts": "gui", "rate-limit-retry.test.ts": "providers", + "mistral-recovery.test.ts": "providers", "reasoning-effort-summary-default.test.ts": "responses", "reasoning-effort.test.ts": "codex-integration", "reasoning-envelope.test.ts": "responses", diff --git a/src/lib/provider-admission-wait.ts b/src/lib/provider-admission-wait.ts new file mode 100644 index 00000000000..238282856b3 --- /dev/null +++ b/src/lib/provider-admission-wait.ts @@ -0,0 +1,14 @@ +/** Internal RequestInit metadata; never serialized into headers or upstream JSON. */ +export const PROVIDER_HEADER_DEADLINE = Symbol("provider-header-deadline"); +export type ProviderHeaderDeadline = { clientSignal: AbortSignal; pause(): void; resume(): void }; + +/** A local admission queue waits on client cancellation, outside the upstream header timer. */ +export async function waitOutsideProviderHeaderDeadline( + init: RequestInit | undefined, + wait: (signal?: AbortSignal) => Promise, +): Promise { + const deadline = (init as (RequestInit & { [PROVIDER_HEADER_DEADLINE]?: ProviderHeaderDeadline }) | undefined)?.[PROVIDER_HEADER_DEADLINE]; + deadline?.pause(); + try { return await wait(deadline?.clientSignal ?? init?.signal ?? undefined); } + finally { deadline?.resume(); } +} diff --git a/src/lib/upstream-retry.ts b/src/lib/upstream-retry.ts index e03e4ce6846..e97e754e8a7 100644 --- a/src/lib/upstream-retry.ts +++ b/src/lib/upstream-retry.ts @@ -29,9 +29,17 @@ import { redactSecretString } from "./redact"; * marker that survives body re-wrapping (combo failure consumption re-parses the JSON). */ const nonReplayableResponses = new WeakSet(); +const nonReplayableRetryAfter = new WeakMap(); -export function markResponseNonReplayable(response: Response): void { +/** Prevent another send and retain an optional server delay on this response. */ +export function markResponseNonReplayable(response: Response, retryAfter?: string): void { nonReplayableResponses.add(response); + if (retryAfter !== undefined) nonReplayableRetryAfter.set(response, retryAfter); +} + +/** Return the server delay retained for a response that must not be replayed. */ +export function nonReplayableRetryAfterFor(response: Response): string | undefined { + return nonReplayableRetryAfter.get(response); } export function isNonReplayableResponse(response: Response): boolean { @@ -144,6 +152,8 @@ export function retainReplayRefusal(response: T): T { /** Carry the verdict from a response onto the one that replaces it. */ export function carryReplayRefusal(source: Response, rewrapped: T): T { + const retryAfter = nonReplayableRetryAfterFor(source); + if (retryAfter !== undefined) markResponseNonReplayable(rewrapped, retryAfter); return isReplayRefusalResponse(source) ? retainReplayRefusal(rewrapped) : rewrapped; } diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index 2c02f9a0ca8..c3965c23977 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -12,6 +12,7 @@ import { isKeyInCooldown, getKeyCooldownUntil, setKeyCooldown } from "./api-key- export { clearKeyCooldowns, getKeyCooldownUntil, sweepExpiredApiKeyCooldowns } from "./api-key-cooldowns"; import { commitProviderApiKeySelection, providerApiKeySelectionIsCurrent } from "./api-key-selection"; import { RequestPacingQueueOverloadError } from "./request-pacing"; +import { isCanonicalMistralChat } from "./mistral-transport"; import type { ProviderApiKeySelection } from "../types/provider"; import { routedProviderConfig } from "./routed-config"; import { getProviderRegistryEntry } from "./registry"; @@ -585,7 +586,7 @@ function allocateDistributedKey(providerName: string, provider: OcxProviderConfi * (subscription traffic such as Muse Spark) falls back to a patient same-key policy so a * burst 429 waits and replays instead of surfacing to the client and aborting a long * session; key-auth Command Code at its canonical endpoints gets the same fallback (#5180), and - * every other provider without the knob keeps today's fail-fast behavior. + * canonical Mistral Chat uses a slower fallback; other destinations stay fail-fast. * OAuth/forward/local credentials are never replayed on the same token. The returned * policy is fully defaulted so callers never re-check fields. */ @@ -609,8 +610,15 @@ export function rateLimitRetryPolicyFor( respectRetryAfter: policy.respectRetryAfter ?? DEFAULT_RATE_LIMIT_RETRY.respectRetryAfter, }; } - // No explicit knob: patient fallback for OpenCode Go and canonical Command Code only. + // No explicit knob: destination-specific fallbacks, never OAuth or local credentials. if (provider.authMode !== undefined && provider.authMode !== "key") return null; + // Mistral's token-throughput and temporary capacity refusals both use HTTP 429. + // Keep relays and other wire adapters opt-in, and reuse the existing send budget. + if (typeof provider.baseUrl === "string" && typeof provider.adapter === "string" && isCanonicalMistralChat({ + adapter: provider.adapter, authMode: provider.authMode, baseUrl: provider.baseUrl, + })) { + return { ...DEFAULT_RATE_LIMIT_RETRY, attempts: 6, intervalMs: 10_000 }; + } if (isOpenCodeGoDestination(provider)) return { ...OPENCODE_GO_RATE_LIMIT_RETRY }; // Command Code's Provider API rate-limits long muse-spark turns the same way (#5180): a single // key cannot fail over, and the Codex client does not retry a 429, so the turn aborts. The key diff --git a/src/providers/mistral-transport.ts b/src/providers/mistral-transport.ts new file mode 100644 index 00000000000..906167b7ae7 --- /dev/null +++ b/src/providers/mistral-transport.ts @@ -0,0 +1,200 @@ +import { createHash } from "node:crypto"; +import type { OcxProviderConfig } from "../types"; +import { isEgressTransparentExecutor, markEgressTransparentExecutor } from "../lib/provider-egress"; +import { configuredOutboundFetch } from "../lib/proxy-env"; +import { waitOutsideProviderHeaderDeadline } from "../lib/provider-admission-wait"; +import { markResponseNonReplayable, sleepWithAbort } from "../lib/upstream-retry"; +import { RequestPacingQueueOverloadError } from "./request-pacing"; +import type { OcxProviderTransport } from "./xai-transport"; + +const MAX_IDENTITIES = 128; +const DEFAULT_MAX_WAIT_MS = 60_000; // Existing retry wait budget, not an inferred rate window. +const MINUTE_MS = 60 * 1000; // Unit named by the observed headers, never an account quota. +type RecoveryState = { generation: number; probe?: { done: Promise; finish(): void } }; + +/** Match the supported key-auth Chat destination, including one optional trailing slash. */ +export function isCanonicalMistralChat( + provider: Pick, +): boolean { + if (provider.adapter !== "openai-chat" || (provider.authMode !== undefined && provider.authMode !== "key")) return false; + try { + const url = new URL(provider.baseUrl); + return url.origin === "https://api.mistral.ai" && /^\/v1\/?$/.test(url.pathname) + && !url.username && !url.password && !url.search && !url.hash; + } catch { return false; } +} + +/** Read a nonnegative integer counter; reject missing or malformed values. */ +function count(headers: Headers, name: string): number | undefined { + const raw = headers.get(name)?.trim(); + if (!raw || !/^\d+$/.test(raw)) return undefined; + const value = Number(raw); + return Number.isSafeInteger(value) ? value : undefined; +} + +/** Read the server delay as seconds or an HTTP date; ignore invalid hints. */ +function retryDelay(headers: Headers, now: number): number | undefined { + const raw = headers.get("retry-after")?.trim(); + if (!raw) return undefined; + const value = /^\d+(?:\.\d+)?$/.test(raw) ? Number(raw) * 1000 : Date.parse(raw) - now; + return Number.isFinite(value) ? Math.max(0, value) : undefined; +} + +/** Classify measured refusal evidence only; never predict a future cache hit or token charge. */ +export function mistralMinuteRefusal(headers: Headers): "temporary" | "charge-above-allowance" | "unknown" { + const month = count(headers, "x-ratelimit-limit-tokens-month"); + if (month !== undefined && count(headers, "x-ratelimit-remaining-tokens-month") === 0) return "unknown"; + const limit = count(headers, "x-ratelimit-limit-tokens-minute"); + const remaining = count(headers, "x-ratelimit-remaining-tokens-minute"); + const cost = count(headers, "x-ratelimit-tokens-query-cost"); + if (limit !== undefined && limit > 0 && cost !== undefined && cost > limit) return "charge-above-allowance"; + if (limit !== undefined && limit > 0 && remaining !== undefined && remaining <= limit + && (remaining === 0 || (cost !== undefined && cost > remaining))) return "temporary"; + const requests = count(headers, "x-ratelimit-limit-req-minute"); + if (requests !== undefined && requests > 0 && count(headers, "x-ratelimit-remaining-req-minute") === 0) return "temporary"; + return "unknown"; +} + +/** Process-local observations, never a fixed account/plan quota or a persisted credential. */ +export class MistralBudgetPacer { + private readonly deadlines = new Map(); + private readonly recoveries = new Map(); + private waiting = 0; + + constructor( + private readonly now: () => number = Date.now, + private readonly sleep: (ms: number, signal?: AbortSignal) => Promise = sleepWithAbort, + ) {} + + /** Record server delays and confirmed minute pressure for this credential and model. */ + observe(identity: string, headers: Headers, status = 200): void { + const tokens = count(headers, "x-ratelimit-remaining-tokens-minute"); + const requests = count(headers, "x-ratelimit-remaining-req-minute"); + // Healthy remaining counters never predict the next request's cache/charge. + if (status !== 429 && tokens !== 0 && requests !== 0) return; + // With no explicit time, activate a conservative minute hold only AFTER a + // refusal establishes pressure. Healthy responses never predict the next cost. + const delay = retryDelay(headers, this.now()) + ?? (status === 429 && mistralMinuteRefusal(headers) === "temporary" ? MINUTE_MS : undefined); + if (delay === undefined || delay === 0) return; + for (const [key, deadline] of this.deadlines) { + if (deadline <= this.now()) this.deadlines.delete(key); + } + if (!this.deadlines.has(identity) && this.deadlines.size >= MAX_IDENTITIES) { + this.deadlines.delete(this.deadlines.keys().next().value!); + } + this.deadlines.set(identity, Math.max(this.deadlines.get(identity) ?? 0, this.now() + delay)); + if (status === 429) { + if (!this.recoveries.has(identity) && this.recoveries.size >= MAX_IDENTITIES) { + const idle = [...this.recoveries].find(([, state]) => !state.probe); + if (!idle) return; + this.recoveries.delete(idle[0]); + } + const state = this.recoveries.get(identity) ?? { generation: 0 }; + state.generation++; + this.recoveries.set(identity, state); + } + } + + /** Wait within the admission budget and return a release callback for one recovery probe. */ + async acquire(identity: string, signal?: AbortSignal, maxWaitMs = DEFAULT_MAX_WAIT_MS): Promise<(success?: boolean) => void> { + signal?.throwIfAborted(); + if (this.waiting >= MAX_IDENTITIES) throw new RequestPacingQueueOverloadError("mistral", "queue_full", 60); + this.waiting++; + const deadline = this.now() + maxWaitMs; + try { + for (;;) { + signal?.throwIfAborted(); + const readyAt = this.deadlines.get(identity); + if (readyAt !== undefined && readyAt > deadline) { + const delay = readyAt - this.now(); + throw new RequestPacingQueueOverloadError("mistral", "queue_expired", Math.ceil(delay / 1000)); + } + if (readyAt !== undefined && readyAt > this.now()) { + await this.sleep(readyAt - this.now(), signal); + continue; + } + const state = this.recoveries.get(identity); + if (!state) { this.deadlines.delete(identity); return () => {}; } + if (state.probe) { + const remaining = deadline - this.now(); + if (remaining <= 0) throw new RequestPacingQueueOverloadError("mistral", "queue_expired", 1); + const local = new AbortController(); + try { + await Promise.race([state.probe.done, this.sleep(remaining, AbortSignal.any([local.signal, ...(signal ? [signal] : [])]))]); + } finally { local.abort(); } + continue; + } + let finish!: () => void; + const probe = { done: new Promise(resolve => { finish = resolve; }), finish: () => finish() }; + state.probe = probe; + const generation = state.generation; + let released = false; + return success => { + if (released) return; + released = true; + if (state.probe === probe) delete state.probe; + // Only this probe can settle recovery; an older concurrent success must + // not erase a more recent refusal or the server's minimum wait. + if (success && state.generation === generation) { + if (this.recoveries.get(identity) === state) this.recoveries.delete(identity); + if ((this.deadlines.get(identity) ?? 0) <= this.now()) this.deadlines.delete(identity); + } + probe.finish(); + }; + } + } finally { this.waiting--; } + } +} + +const pacer = new MistralBudgetPacer(); +const wrapped = new WeakSet(); + +/** Wrap canonical Mistral Chat sends with bounded admission and server-delay protection. */ +export function resolveMistralTransport( + provider: OcxProviderTransport, + budgetPacer = pacer, +): OcxProviderTransport { + if (!isCanonicalMistralChat(provider) || (provider.fetch && wrapped.has(provider.fetch))) return provider; + const baseFetch = provider.fetch + ?? markEgressTransparentExecutor(((input, init) => configuredOutboundFetch(input, init)) as typeof globalThis.fetch); + const attemptFetch = (async (input, init) => { + const url = new URL(input instanceof Request ? input.url : String(input)); + if (url.origin !== "https://api.mistral.ai" || url.pathname !== "/v1/chat/completions" + || url.search || url.hash || init?.method?.toUpperCase() !== "POST" || typeof init.body !== "string") { + return baseFetch(input, init); + } + // The Chat adapter serializes model first. Read only that bounded prefix rather + // than parsing/copying a potentially million-token conversation a second time. + const encodedModel = /^\s*\{\s*"model"\s*:\s*("(?:[^"\\]|\\.)*")\s*[,}]/.exec(init.body.slice(0, 4096))?.[1]; + let model: unknown; + try { model = encodedModel === undefined ? undefined : JSON.parse(encodedModel); } catch { return baseFetch(input, init); } + const authorization = new Headers(init.headers).get("authorization"); + if (!authorization || typeof model !== "string" || !model || model.length > 256) return baseFetch(input, init); + // Bind to the actual send credential, including after pool rotation; keep no raw key. + const identity = createHash("sha256").update(JSON.stringify([authorization, model])).digest("hex"); + const pacing = provider.requestPacing?.enabled !== false && provider.retryOn429?.respectRetryAfter !== false; + const release = pacing ? await waitOutsideProviderHeaderDeadline(init, + signal => budgetPacer.acquire(identity, signal, provider.retryOn429?.maxIntervalMs)) : undefined; + let response: Response; + try { + init.signal?.throwIfAborted(); + response = await baseFetch(input, init); + if (pacing) budgetPacer.observe(identity, response.headers, response.status); + } catch (error) { release?.(); throw error; } + release?.(response.ok); + const retry = provider.retryOn429; + if (response.status === 429 && retry?.respectRetryAfter !== false) { + const delay = retryDelay(response.headers, Date.now()); + if (delay !== undefined && delay > (retry?.maxIntervalMs ?? DEFAULT_MAX_WAIT_MS)) { + // The shared retry helper caps waits. Preserve this refusal instead of replaying + // sooner than Mistral permits; the original status/body/Retry-After reach the caller. + markResponseNonReplayable(response, String(Math.ceil(delay / 1000))); + } + } + return response; + }) as typeof globalThis.fetch; + if (isEgressTransparentExecutor(baseFetch)) markEgressTransparentExecutor(attemptFetch); + wrapped.add(attemptFetch); + return { ...provider, fetch: attemptFetch }; +} diff --git a/src/providers/registry/entries-extended.ts b/src/providers/registry/entries-extended.ts index 3756e01d285..1424f6ab8dc 100644 --- a/src/providers/registry/entries-extended.ts +++ b/src/providers/registry/entries-extended.ts @@ -1020,7 +1020,7 @@ export const PROVIDER_REGISTRY_EXTENDED: readonly ProviderRegistryEntry[] = [ }, }, // FREEZE 2026-07-10: codestral-latest is unconfirmed behind auth. Evidence: devlog/_plan/260710_provider_hardening/003_research_aggregators.md. - { id: "mistral", label: "Mistral", baseUrl: "https://api.mistral.ai/v1", adapter: "openai-chat", authKind: "key", dashboardUrl: "https://console.mistral.ai/api-keys", defaultModel: "codestral-latest" }, + { id: "mistral", label: "Mistral", baseUrl: "https://api.mistral.ai/v1", adapter: "openai-chat", authKind: "key", dashboardUrl: "https://console.mistral.ai/api-keys", defaultModel: "codestral-latest", promptCacheKey: true }, { id: "minimax", label: "MiniMax — Coding Plan", baseUrl: "https://api.minimax.io/v1", adapter: "openai-chat", authKind: "key", dashboardUrl: "https://platform.minimax.io", defaultModel: "MiniMax-M3", models: MINIMAX_MODELS, diff --git a/src/providers/xai-transport.ts b/src/providers/xai-transport.ts index b58519b1ccd..6cc62375b2d 100644 --- a/src/providers/xai-transport.ts +++ b/src/providers/xai-transport.ts @@ -3,6 +3,7 @@ import type { OcxProviderConfig } from "../types"; import { isEgressTransparentExecutor, markEgressTransparentExecutor } from "../lib/provider-egress"; import { configuredOutboundFetch } from "../lib/proxy-env"; import { resolveGithubCopilotTransport } from "./github-copilot-transport"; +import { resolveMistralTransport } from "./mistral-transport"; export const XAI_GROK_CLI_BASE_URL = "https://cli-chat-proxy.grok.com/v1"; @@ -121,7 +122,7 @@ export function resolveProviderTransport( if (providerName === "github-copilot") { return resolveGithubCopilotTransport(provider, apiBaseUrl); } - if (providerName !== "xai") return provider; + if (providerName !== "xai") return resolveMistralTransport(provider); const cacheKey = promptCacheKey?.trim(); const affinity = cacheKey ? deriveXaiConvId(cacheKey) : undefined; diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index 008bbc9ec8b..46c3e571f7a 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -1,4 +1,5 @@ import type { NativeResponseControl } from "./native-response-control"; +import { PROVIDER_HEADER_DEADLINE as HEADER_DEADLINE, type ProviderHeaderDeadline as HeaderDeadline } from "../../lib/provider-admission-wait"; import type { CodexWsPreOutputRefusalHold } from "./codex-ws-wire"; import type { Server } from "bun"; import { @@ -62,9 +63,6 @@ export function physicalSendObserverInit(observer: AccountAutoPauseObserver | un } // One WS attempt and its HTTP fallback are one logical physical-send receipt. const WS_FALLBACK_RECEIPT = Symbol.for("opencodex.ws-fallback-receipt"); -// Replacement admission waits on the client signal, outside the response-header deadline. -const HEADER_DEADLINE = Symbol("header-deadline"); -type HeaderDeadline = { clientSignal: AbortSignal; pause(): void; resume(): void }; /** * Announce once, per provider, that an explicit egress route moved this provider off the diff --git a/src/server/responses/non-replayable-error.ts b/src/server/responses/non-replayable-error.ts index 53a21a42008..cb4d78adda8 100644 --- a/src/server/responses/non-replayable-error.ts +++ b/src/server/responses/non-replayable-error.ts @@ -1,4 +1,4 @@ -import { isNonReplayableResponse, isReplayRefusalResponse, markResponseNonReplayable } from "../../lib/upstream-retry"; +import { isNonReplayableResponse, isReplayRefusalResponse, markResponseNonReplayable, nonReplayableRetryAfterFor } from "../../lib/upstream-retry"; import { clientCancelledResponse, readDisplaySafeErrorText } from "./core-errors"; const UPSTREAM_ERROR_TYPES = new Set([ @@ -39,10 +39,12 @@ export async function sanitizeNonReplayableUpstreamError( message: `Provider error ${response.status}: upstream diagnostic withheld for a non-replayable failure`, } }); const headers = new Headers({ "content-type": "application/json" }); + const retryAfter = nonReplayableRetryAfterFor(response); + if (retryAfter !== undefined) headers.set("retry-after", retryAfter); if (response.headers.get("x-should-retry") === "false") headers.set("x-should-retry", "false"); const safe = new Response(response.status === 304 ? null : body, { status: response.status, headers, }); - markResponseNonReplayable(safe); + markResponseNonReplayable(safe, retryAfter); return safe; } diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index a7526944ee6..26f5eed1408 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -1,4 +1,5 @@ # Providers And Adapters +Canonical key-auth Mistral Chat at `https://api.mistral.ai/v1` defaults to a patient bounded same-key 429 policy (six replays, ten-second fallback) in `src/providers/key-failover.ts`; explicit retry settings/opt-out win. The Mistral preset forwards supplied prompt-cache keys unless explicitly disabled. `src/providers/key-failover.ts` and transport selection use the same canonical-destination check, including custom provider names. `src/providers/mistral-transport.ts` shares explicit Retry-After deadlines across calls by actual credential digest and exact model, with bounded memory/waiters and cancellation. Without Retry-After, only a real 429 with validated exhausted minute capacity or a measured query charge above remaining but within the full token allowance activates a conservative 60-second recovery hold. This local policy does not claim the server reset instant. One probe gates concurrent recovery callers; a successful probe clears recovery unless a newer refusal intervened. Healthy traffic never predicts future query/cache cost; unknown, zero-entitlement, exhausted-monthly and above-full-allowance evidence does not invent a minute reset. Oversized Retry-After stops replay rather than shortening the server delay; the non-replayable error projection preserves its normalized numeric hint through optional response metadata. Either `requestPacing.enabled: false` or `retryOn429.respectRetryAfter: false` disables the shared hold. `tests/providers/mistral-recovery.test.ts` covers GLM/Large 4, destination/key/model isolation, absent headers, cache affinity, streamed/buffered replay, cancellation and send ceilings. There is no mid-stream replay or automatic compaction change. GCP JSON storage in `src/providers/api-keys.ts` and durable-row snapshot/post-commit cleanup in `src/providers/gcp-credential-cleanup.ts` follow [Vertex credential markers](providers/google.md#vertex-credential-markers); malformed blank marker accounts do not block cleanup of valid accounts. Combo target `metered` metadata and configured-key opt-in follow the [runtime consent contract](runtime.md#request-local-target-compatibility). Subscription-to-Cursor-to-key regression coverage in `tests/adapters/anthropic/anthropic-subscription-chain.test.ts` checks unchanged healthy subscription wire bodies, effort, key authentication and cache controls, and structured runTurn 429 propagation. OpenCode Go's exact Gemini pin and credential destination guard follow the [Go Gemini transport contract](providers/google.md#opencode-go-gemini-transport). Configured spend ceilings bind adapter inference executors to the [seed and physical-start contract](transports/responses-spend.md#historical-pool-continuity-and-rollback). Zed completion and auth-recovery replays claim separately; token acquisition stays outside inference accounting. Claude CLI, CodeBuddy and Qoder each count one invocation toward the frozen request limit, seed before spawn and retain a producer through completion. Child-internal retries/tool turns are not separately counted; reported actual usage settles in full, including amounts above the seed estimate. Devin combines only [consecutive same-ID tool results](adapters/registry.md#devin-consecutive-tool-results), using linear accumulation while preserving intervening message slots, image parts and error markers without mutating the parsed request. diff --git a/structure/transports/inventory.md b/structure/transports/inventory.md index f0c9870e6e7..77ffb9c6a9a 100644 --- a/structure/transports/inventory.md +++ b/structure/transports/inventory.md @@ -45,6 +45,7 @@ surface is listed here so a maintainer can find the owner without grepping: | Hosted search relay | `src/server/search.ts`, `src/web-search/devin-executor.ts` | A resolved Devin route uses Cognition's bounded non-inference `GetWebSearchResults` RPC with the active account's credential and allowlisted tenant; other routes use the verbatim ChatGPT relay, or an explicitly configured web-search sidecar backend when no forward provider exists. Distinct from the web-search sidecar loop below. | | Image/video generation loop | `src/images/loop.ts`, `src/images/plan.ts`, `src/images/fulfill.ts`, `src/images/xai-client.ts`, `src/images/xai-video-client.ts`, `src/images/artifacts.ts` | A provider-returned image URL is downloaded into a local artifact once, then served locally; warnings stay URL-free because provider CDN URLs may embed credentials. Artifact downloads go through the pinned-IP transport with a 10 s connect deadline (`DOWNLOAD_CONNECT_TIMEOUT_MS`) that bounds TCP/TLS setup on its own, in addition to the 60 s idle timer, and `pinnedHttpsGet` accepts a per-call `connectTimeoutMs`. | | GitHub Copilot | `src/providers/xai-transport.ts` (`resolveProviderTransport`), `src/providers/github-copilot-transport.ts` | `resolveProviderTransport` selects the Copilot transport when the routed provider name is `github-copilot`; the Copilot module then resolves its headers and base URL, and the registry seeds the provider row and model fallback. Auto session and intent requests stay on the credential-captured origin; adapter construction follows the [Copilot Auto contract](../providers-and-adapters.md#github-copilot-auto-selection). | +| Mistral | `src/providers/mistral-transport.ts` | Canonical key-auth Chat is selected by destination, including custom provider names, and shares Retry-After holds by credential and exact model. A real 429 with validated minute pressure and no retry time activates a conservative minute recovery hold and one probe; healthy traffic has no predicted-charge pacing. Local admission uses `src/lib/provider-admission-wait.ts` to pause only the upstream header deadline while retaining client cancellation; the deadline restarts before the send. No fixed account quota, exact server-reset claim or automatic compaction; other providers/destinations retain their transports. See [provider recovery](../providers-and-adapters.md). | | API-key pools | `src/providers/api-key-selection.ts`, `src/providers/key-failover.ts` | A configured `apiKeyPoolStrategy` plus a cooling committed key rotates before the first send (`selectProactiveApiKeyTransport`); a 429 still rotates after the send and records a cooldown. `provider.apiKey` keeps mirroring the active entry so routing stays single-key. The pick is inert without a strategy or while the committed key is healthy. | | OAuth account failover | `src/oauth/generic-account-failover.ts`, `src/oauth/anthropic-routing.ts`, `src/server/responses/antigravity-validation-refusal.ts`, `src/adapters/antigravity-refusal.ts` | Reactive pre-output 429 recovery is presence-driven with 2+ eligible accounts. Anthropic also handles exact revoked-token 401s under its [reauthentication contract](../providers/anthropic-account-pool.md#revoked-oauth-access-token-recovery) and recovers classified pre-output account-entitlement 403s under its [account-pool contract](../providers/anthropic-account-pool.md#account-entitlement-refusal-recovery), and records the refused account's cooldown even when the request has no retry sends left. Primary Antigravity inference through the Google adapter main dispatch may send once on a live sibling after a replayed 401, a terminal credential-refresh failure that marks the account for reauthentication, or a normalized structured `VALIDATION_REQUIRED` 403, within the existing budget. A bounded message-worded verification 403 persists a generation-fenced `verify_account` cause in `src/oauth/store.ts` and `src/oauth/health.ts`, including singleton and last-account refusals. It reads as `needs-reauth(verify)` in `ocx account list`; silent refresh preserves quarantine, while explicit re-login clears it and retires its failover cooldown. Quarantine does not require a retry slot; if persisting it fails, dispatch exits this adapter exchange’s recovery loop and preserves the original bounded 403 without another recovery send in that exchange; enclosing combo and policy fallback rules remain unchanged. A preserved original 403 after sibling build or pre-send admission failure also exits recovery through common bounded error formatting and the combo failure callback; newly constructed failure responses keep their own delivery. Successful sibling replay uses the captured pool activation and the existing send budget. The 401 and structured 403 sibling paths share one request guard; unrelated 403s and transient refresh failures do not rotate. The fetch web-search sidecar accepts only complete bounded structured `VALIDATION_REQUIRED` 403s through the Google normalizer, carries the sibling token/project without a durable reauthentication mark, and uses adapter-owned physical-send accounting; its contract is [Account refusal and rotation boundaries](responses-failover.md#account-refusal-and-rotation-boundaries). Continuations, native passthrough and image sidecars keep their existing behavior. Generic OAuth accounts can be operator-paused; paused accounts are excluded from selection, failover, and proactive Token Guardian refresh. If every account is paused, the request fails with 403 rather than as a login failure. Pool and `oauthAccountFailover` flags govern proactive routing, not the reactive retry: a disabled Anthropic pool recovers through quota ordering rather than its dormant strategy, a per-provider `enabled` beats the global default in either direction, and a non-positive fill-first threshold disables proactive usage-based rotation. Kiro's `kiroAutoSelection` projects the same candidate eligibility into routing and account-list status; unknown evidence stays eligible, while pause, reauth, suspension, cooldown, and confirmed exhaustion have closed reasons. An active singleton or a pool excluded only from automatic Kiro selection may still attempt its active account; an explicitly paused active account is never dispatched, and an all-paused pool fails with 403. A configured Kiro capacity cap can also time out. Kiro's optional process-local account lease is released at response completion or cancellation; other providers retain their admission path. | | OAuth login callback (inbound) | `src/oauth/callback-server.ts` | Every response, including non-callback 404s, closes its connection so a pooled socket cannot deliver a later login to a retired flow on the same callback port. | diff --git a/structure/transports/responses.md b/structure/transports/responses.md index d11c13a36d8..693f74d3c95 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -40,7 +40,7 @@ redirect following; providers requiring a redirect must be configured with their `src/server/responses/fetch-helpers.ts` is a transport leaf shared by Responses, compact, and native Chat. Its runtime imports are limited to the Codex WebSocket transport, provider request pacing, and -the upstream HTTP-version helper. Server, provider, and WebSocket data types remain type-only edges. Opt-in `codexWsReuseAcrossTurns` (default off) in `src/server/responses/codex-ws-pool.ts` keeps a socket for an account and thread across turns, waits at most 750ms for a busy socket before dialing another, and leaves `x-codex-turn-state` / `x-codex-turn-metadata` on each frame. +the upstream HTTP-version helper, and core-owned admission-deadline metadata. Provider-local recovery queues share `src/lib/provider-admission-wait.ts`: local waiting retains client cancellation while pausing the upstream header timer, which restarts before dispatch. Server, provider, and WebSocket data types remain type-only edges. Opt-in `codexWsReuseAcrossTurns` (default off) in `src/server/responses/codex-ws-pool.ts` keeps a socket for an account and thread across turns, waits at most 750ms for a busy socket before dialing another, and leaves `x-codex-turn-state` / `x-codex-turn-metadata` on each frame. Account refusal observers are injected by the calling transport owner and captured before the physical send; their wrapper forwards replacement pacing admission to the physical dispatch owner. It must not import routing, combos, OAuth, adapters, sidecars, response parsing, logging, or relay modules merely because those imports existed in the pre-split `responses.ts` monolith. diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 3a82672e452..bf82f4b65b8 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1018,7 +1018,8 @@ "quota-reset-observation.test.ts": "usage", "quota-reset-seen-store.test.ts": "usage", "quota-scoring.test.ts": "usage", "qwen-cloud-endpoints.test.ts": "gui", "qwen38-preserve-reasoning.test.ts": "providers", "rate-limit-reset-credits.test.ts": "gui", - "rate-limit-retry.test.ts": "providers", "raycast-client.test.ts": "clients", + "rate-limit-retry.test.ts": "providers", + "mistral-recovery.test.ts": "providers", "raycast-client.test.ts": "clients", "raycast-detect.test.ts": "clients", "reasoning-effort-summary-default.test.ts": "responses", "reasoning-effort.test.ts": "codex-integration", "reasoning-envelope.test.ts": "responses", "reasoning-retention.test.ts": "responses", "reasoning-retention-routing.test.ts": "responses", "reasoning-retention-v2.test.ts": "responses", diff --git a/tests/providers/mistral-recovery.test.ts b/tests/providers/mistral-recovery.test.ts new file mode 100644 index 00000000000..7e4a64ffea1 --- /dev/null +++ b/tests/providers/mistral-recovery.test.ts @@ -0,0 +1,561 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { createOpenAIChatAdapter } from "../../src/adapters/openai-chat"; +import { providerConfigSeed } from "../../src/providers/derive"; +import { rateLimitRetryPolicyFor } from "../../src/providers/key-failover"; +import { isCanonicalMistralChat, MistralBudgetPacer, mistralMinuteRefusal, resolveMistralTransport } from "../../src/providers/mistral-transport"; +import { isNonReplayableResponse, sleepWithAbort } from "../../src/lib/upstream-retry"; +import { resolveProviderTransport } from "../../src/providers/xai-transport"; +import { PROVIDER_REGISTRY } from "../../src/providers/registry"; +import { routedProviderConfig } from "../../src/providers/routed-config"; +import { parseRequest } from "../../src/responses/parser"; +import { handleResponses } from "../../src/server/responses"; +import { fetchWithHeaderTimeout } from "../../src/server/responses/fetch-helpers"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; + +const baseUrl = "https://api.mistral.ai/v1"; +const model = "zai-glm-5-3"; +const provider: OcxProviderConfig = { adapter: "openai-chat", baseUrl, authMode: "key", apiKey: "synthetic-mistral-key" }; +const policy = { enabled: true, attempts: 6, intervalMs: 10_000, maxIntervalMs: 60_000, respectRetryAfter: true }; +const originalFetch = globalThis.fetch; +let releaseSpendHome: (() => void) | undefined; +let dispatchId = 0; + +afterEach(() => { + globalThis.fetch = originalFetch; + releaseSpendHome?.(); + releaseSpendHome = undefined; +}); + +describe("Mistral recovery defaults", () => { + test.each([baseUrl, `${baseUrl}/`, "https://API.MISTRAL.AI:443/v1"])("canonical key destination %s waits on 429", url => { + const row = { ...provider, baseUrl: url }; + expect(rateLimitRetryPolicyFor(row)).toEqual(policy); + expect(isCanonicalMistralChat(row)).toBe(true); + expect(resolveMistralTransport(row)).not.toBe(row); + }); + + test("explicit settings and opt-out remain authoritative", () => { + expect(rateLimitRetryPolicyFor({ ...provider, authMode: undefined })).toEqual(policy); + expect(rateLimitRetryPolicyFor({ ...provider, retryOn429: { enabled: false } })).toBeNull(); + expect(rateLimitRetryPolicyFor({ ...provider, retryOn429: { attempts: 1, intervalMs: 123 } })) + .toMatchObject({ attempts: 1, intervalMs: 123 }); + expect(rateLimitRetryPolicyFor({ ...provider, retryOn429: {} })) + .toMatchObject({ attempts: 3, intervalMs: 5_000 }); + }); + + test.each(["oauth", "forward", "local"] as const)("%s authentication never enables same-key retries", authMode => { + expect(rateLimitRetryPolicyFor({ ...provider, authMode })).toBeNull(); + }); + + test.each([ + "https://api.openai.com/v1", "https://api.mistral.ai.example.test/v1", + "https://example.test/api.mistral.ai/v1", "http://api.mistral.ai/v1", + "https://api.mistral.ai:8443/v1", `${baseUrl}?route=custom`, `${baseUrl}#custom`, + "https://api.mistral.ai/other/v1", `${baseUrl}//`, + baseUrl.replace("://", "://synthetic-user:synthetic-password@"), "not a URL", + ])("other destinations retain fail-fast behavior: %s", url => { + const row = { ...provider, baseUrl: url }; + expect(rateLimitRetryPolicyFor(row)).toBeNull(); + expect(isCanonicalMistralChat(row)).toBe(false); + expect(resolveMistralTransport(row)).toBe(row); + }); + + test("a non-Chat adapter at the same host retains its previous behavior", () => { + expect(rateLimitRetryPolicyFor({ ...provider, adapter: "openai-responses" })).toBeNull(); + }); + + test("Mistral registry defaults forward stable caller cache keys, with an explicit opt-out", async () => { + const entry = PROVIDER_REGISTRY.find(row => row.id === "mistral")!; + expect(providerConfigSeed(entry).promptCacheKey).toBe(true); + for (const enabled of [undefined, false]) { + const routed = routedProviderConfig("mistral", { ...provider, ...(enabled === false ? { promptCacheKey: false } : {}) }); + const adapter = createOpenAIChatAdapter(routed); + const parsed = parseRequest({ model, input: "Synthetic fixture", prompt_cache_key: "synthetic-conversation" }); + const request = await adapter.buildRequest(parsed); + expect(JSON.parse(request.body!).prompt_cache_key).toBe(enabled === false ? undefined : "synthetic-conversation"); + } + }); +}); + +describe("Mistral Responses recovery", () => { + async function dispatch(options: { failures: number; stream?: boolean; disabled?: boolean; abort?: AbortController; retryAfter?: string; leased?: boolean; minuteBudget?: boolean; providerName?: string; maxIntervalMs?: number }) { + const bodies: string[] = []; + const authorizations: (string | null)[] = []; + const apiKey = `synthetic-mistral-key-${++dispatchId}`; + globalThis.fetch = (async (_input, init) => { + bodies.push(String(init?.body)); + authorizations.push(new Headers(init?.headers).get("authorization")); + if (options.abort) { + setTimeout(() => options.abort!.abort(), 10); + } + if (bodies.length <= options.failures) return Response.json({ message: "Not enough capacity available for this request, please retry later." }, { + status: 429, headers: options.minuteBudget ? { + "x-ratelimit-limit-tokens-minute": "1000", "x-ratelimit-remaining-tokens-minute": "400", + "x-ratelimit-tokens-query-cost": "600", + } : { "retry-after": options.retryAfter ?? (options.abort ? "30" : "0") }, + }); + const completion = { id: "synthetic-completion", choices: [{ index: 0, message: { role: "assistant", content: "OK" }, finish_reason: "stop" }], usage: { prompt_tokens: 3, completion_tokens: 1, total_tokens: 4 } }; + return options.stream + ? new Response(`data: ${JSON.stringify({ ...completion, choices: [{ index: 0, delta: { content: "OK" }, finish_reason: null }] })}\n\ndata: ${JSON.stringify({ choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage: completion.usage })}\n\ndata: [DONE]\n\n`, { headers: { "content-type": "text/event-stream" } }) + : Response.json(completion); + }) as typeof fetch; + releaseSpendHome = acquireOwnedSpendHome(); + let now = 0; + const waits: number[] = []; + const providerName = options.providerName ?? "mistral"; + const row = { ...provider, apiKey, ...(options.maxIntervalMs === undefined ? {} : { retryOn429: { maxIntervalMs: options.maxIntervalMs } }), ...(options.disabled ? { retryOn429: { enabled: false } } : {}), + ...(options.leased ? { requestPacing: { enabled: true, maxConcurrentRequests: 1 } } : {}) }; + const transport = options.minuteBudget ? resolveMistralTransport({ ...row, fetch: globalThis.fetch, + retryOn429: { intervalMs: 0 } }, new MistralBudgetPacer(() => now, async (ms, signal) => { + signal?.throwIfAborted(); waits.push(ms); now += ms; + })) : row; + const config = { port: 0, defaultProvider: providerName, providers: { [providerName]: transport } } as OcxConfig; + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: `${providerName}/${model}`, input: "Synthetic fixture", stream: options.stream ?? false, prompt_cache_key: "synthetic-conversation" }), + }), config, { model: `${providerName}/${model}`, provider: providerName }, { abortSignal: options.abort?.signal }); + return { response, bodies, authorizations, apiKey, waits }; + } + + test.each([false, true])("pre-output 429 recovers on the same request and key (stream=%s)", async stream => { + const { response, bodies, authorizations, apiKey } = await dispatch({ failures: 1, stream }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("OK"); + expect(bodies).toHaveLength(2); + expect(bodies[1]).toBe(bodies[0]); + expect(JSON.parse(bodies[0]!).prompt_cache_key).toBe("synthetic-conversation"); + expect(authorizations).toEqual([`Bearer ${apiKey}`, `Bearer ${apiKey}`]); + }); + + test("persistent 429 stops after the six configured replays", async () => { + const { response, bodies } = await dispatch({ failures: 10 }); + expect(response.status).toBe(429); + expect(bodies).toHaveLength(7); + await response.text(); + }); + + test.each([false, true])("measured minute pressure recovers through the Responses pipeline (stream=%s)", async stream => { + const { response, bodies, waits } = await dispatch({ failures: 1, stream, minuteBudget: true, leased: true }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("OK"); + expect(bodies).toHaveLength(2); + expect(bodies[1]).toBe(bodies[0]); + expect(JSON.parse(bodies[0]!).prompt_cache_key).toBe("synthetic-conversation"); + expect(waits).toEqual([60000]); + }); + + test("a custom-named Mistral provider returns long server delays without early replay", async () => { + const { response, bodies } = await dispatch({ failures: 1, providerName: "mistral-vibe", retryAfter: "120", maxIntervalMs: 1, leased: true }); + expect(response.status).toBe(429); + expect(response.headers.get("retry-after")).toBe("120"); + expect(bodies).toHaveLength(1); + await response.text(); + }); + + test("explicit opt-out surfaces the first 429", async () => { + const { response, bodies } = await dispatch({ failures: 1, disabled: true }); + expect(response.status).toBe(429); + expect(bodies).toHaveLength(1); + await response.text(); + }); + + test("cancellation during backoff stops without another send", async () => { + const { response, bodies } = await dispatch({ failures: 1, abort: new AbortController() }); + expect(response.status).toBe(499); + expect(bodies).toHaveLength(1); + await response.text(); + }); + + test.each([false, true])("a server wait beyond the budget surfaces without an early replay (leased=%s)", async leased => { + const { response, bodies } = await dispatch({ failures: 1, retryAfter: "120", leased }); + expect(response.status).toBe(429); + expect(response.headers.get("retry-after")).toBe("120"); + expect(bodies).toHaveLength(1); + await response.text(); + }); +}); + +describe("Mistral observed budget pacing", () => { + function fixture() { + let now = 0; + const waits: number[] = []; + const pacing = new MistralBudgetPacer(() => now, async (ms, signal) => { + signal?.throwIfAborted(); + waits.push(ms); + now += ms; + }); + return { pacing, waits }; + } + + const budget = (tokens: string, cost: string, requests = "10", retryAfter?: string) => new Headers({ + "x-ratelimit-remaining-tokens-minute": tokens, + "x-ratelimit-tokens-query-cost": cost, + "x-ratelimit-remaining-req-minute": requests, + ...(retryAfter === undefined ? {} : { "retry-after": retryAfter }), + }); + + test("uses each account's advertised budget without a fixed million-token ceiling", async () => { + const { pacing, waits } = fixture(); + pacing.observe("higher-tier", budget("6000000", "1200000")); + await pacing.acquire("higher-tier"); + expect(waits).toEqual([]); + pacing.observe("lower-tier", budget("0", "600", "10", "7")); + await pacing.acquire("lower-tier"); + expect(waits).toEqual([7000]); + }); + + test("waits for explicit retry times after token or request exhaustion", async () => { + const { pacing, waits } = fixture(); + pacing.observe("tokens", budget("800", "600", "10", "9")); + await pacing.acquire("tokens"); + expect(waits).toEqual([]); + pacing.observe("tokens", budget("0", "600", "10", "9")); + await pacing.acquire("tokens"); + expect(waits).toEqual([9000]); + pacing.observe("requests", budget("9000", "1", "0", "3")); + await pacing.acquire("requests"); + await pacing.acquire("requests"); + expect(waits).toEqual([9000, 3000]); + }); + + test("missing, ambiguous, malformed and stale measurements do not impose quotas", async () => { + const { pacing, waits } = fixture(); + for (const headers of [new Headers(), new Headers({ "x-ratelimit-remaining": "0" }), + new Headers({ "x-ratelimit-remaining-tokens-minute": "NaN" }), + new Headers({ "x-ratelimit-remaining-req-minute": "-1" }), + new Headers({ "x-ratelimit-remaining-tokens-minute": "9007199254740992" }), + budget("0", "600"), budget("400", "600"), budget("0", "600", "10", "invalid")]) { + pacing.observe("unknown", headers); + await pacing.acquire("unknown"); + } + expect(waits).toEqual([]); + pacing.observe("expired", budget("0", "100", "10", "2")); + await pacing.acquire("expired"); + await pacing.acquire("expired"); + expect(waits).toEqual([2000]); + }); + + test.each(["zai-glm-5-3", "mistral-large-4"])("actual credential and model isolate %s budgets", async modelId => { + const { pacing, waits } = fixture(); + const replies: Response[] = []; + const transport = resolveMistralTransport({ ...provider, fetch: (async () => { + const response = new Response("unchanged", { headers: budget("0", "600", "10", "4") }); + replies.push(response); + return response; + }) as typeof fetch }, pacing); + const send = (key: string, model = modelId) => transport.fetch!(new URL(`${baseUrl}/chat/completions`), { + method: "POST", headers: { authorization: `Bearer ${key}` }, body: JSON.stringify({ model }), + }); + expect(await send("synthetic-key-a")).toBe(replies[0]!); + await send("synthetic-key-b"); + await send("synthetic-key-a", "another-model"); + expect(waits).toEqual([]); + await send("synthetic-key-a"); + expect(waits).toEqual([4000]); + }); + + test("pacing opt-out and unrelated providers retain their transport behavior", async () => { + const { pacing, waits } = fixture(); + const fetch = (async () => new Response("OK", { headers: budget("0", "100", "10", "4") })) as typeof globalThis.fetch; + const disabled = { ...provider, fetch, requestPacing: { enabled: false } }; + const transport = resolveMistralTransport(disabled, pacing); + const init = { method: "POST", headers: { authorization: "Bearer synthetic-key" }, body: JSON.stringify({ model }) }; + await transport.fetch!(`${baseUrl}/chat/completions`, init); + await transport.fetch!(`${baseUrl}/chat/completions`, init); + expect(waits).toEqual([]); + expect(resolveProviderTransport("another-provider", { ...provider, fetch, baseUrl: "https://example.test/v1" }).fetch).toBe(fetch); + expect(resolveMistralTransport({ ...provider, fetch, baseUrl: "https://example.test/v1" }).fetch).toBe(fetch); + expect(resolveMistralTransport(transport)).toBe(transport); + }); + + test("long Retry-After is preserved for both numeric and date hints", async () => { + for (const hint of ["120", new Date(Date.now() + 120000).toUTCString()]) { + const response = new Response("original refusal", { status: 429, headers: { "retry-after": hint } }); + const transport = resolveMistralTransport({ ...provider, fetch: (async () => response) as typeof fetch }, new MistralBudgetPacer()); + const result = await transport.fetch!(`${baseUrl}/chat/completions`, { + method: "POST", headers: { authorization: "Bearer synthetic-key" }, body: JSON.stringify({ model }), + }); + expect(result).toBe(response); + expect(isNonReplayableResponse(result)).toBe(true); + expect(await result.text()).toBe("original refusal"); + } + }); + + test("cancelling a budget wait promptly releases it without sending", async () => { + const pacing = new MistralBudgetPacer(); + pacing.observe("cancelled", budget("0", "100", "10", "30")); + const abort = new AbortController(); + const waiting = pacing.acquire("cancelled", abort.signal); + abort.abort(); + await expect(waiting).rejects.toThrow(); + }); + + test("a dated refusal without remaining counters holds later calls within their configured wait budget", async () => { + const { pacing, waits } = fixture(); + pacing.observe("refused", new Headers({ "retry-after": "90" }), 429); + await expect(pacing.acquire("refused", undefined, 10000)).rejects.toMatchObject({ retryAfterSeconds: 90 }); + expect(waits).toEqual([]); + await pacing.acquire("refused", undefined, 90000); + expect(waits).toEqual([90000]); + }); + + test("queue pressure is bounded and cancelled waiters release their capacity", async () => { + const pacing = new MistralBudgetPacer(); + pacing.observe("pressure", budget("0", "1", "10", "30")); + const abort = new AbortController(); + const waiting = Array.from({ length: 128 }, () => pacing.acquire("pressure", abort.signal)); + const settled = Promise.allSettled(waiting); + await expect(pacing.acquire("pressure")).rejects.toMatchObject({ reason: "queue_full" }); + abort.abort(); + expect((await settled).every(result => result.status === "rejected")).toBe(true); + const cancelled = new AbortController(); + const next = pacing.acquire("pressure", cancelled.signal); + cancelled.abort(); + await expect(next).rejects.toThrow(); + }); +}); + +describe("Mistral error-triggered minute recovery", () => { + const headers = (limit: number, remaining: number, cost: number, requests = 100, reqRemaining = requests) => new Headers({ + "x-ratelimit-limit-tokens-minute": String(limit), "x-ratelimit-remaining-tokens-minute": String(remaining), + "x-ratelimit-tokens-query-cost": String(cost), "x-ratelimit-limit-req-minute": String(requests), + "x-ratelimit-remaining-req-minute": String(reqRemaining), + }); + function fixture() { + let now = 0; + const waits: number[] = []; + const pacing = new MistralBudgetPacer(() => now, async (ms, signal) => { + signal?.throwIfAborted(); waits.push(ms); now += ms; + }); + return { pacing, waits }; + } + + test.each([1000, 1000000, 6000000])("only a refused measured charge activates the %s-token minute recovery", async limit => { + const { pacing, waits } = fixture(); + const evidence = headers(limit, limit * 0.4, limit * 0.6); + pacing.observe("key/model", evidence, 200); + (await pacing.acquire("key/model"))(true); + expect(waits).toEqual([]); + pacing.observe("key/model", evidence, 429); + (await pacing.acquire("key/model"))(true); + (await pacing.acquire("key/model"))(true); + expect(waits).toEqual([60000]); + }); + + test("request-minute exhaustion is independent of available tokens", async () => { + const { pacing, waits } = fixture(); + pacing.observe("glm", headers(1000, 990, 10, 1, 0), 429); + (await pacing.acquire("glm"))(true); + expect(waits).toEqual([60000]); + }); + + test("known exhausted monthly allowance never becomes a minute reset or compaction signal", async () => { + const { pacing, waits } = fixture(); + for (const cost of [600, 1200]) { + const evidence = headers(1000, 0, cost); + evidence.set("x-ratelimit-limit-tokens-month", "10000000"); + evidence.set("x-ratelimit-remaining-tokens-month", "0"); + expect(mistralMinuteRefusal(evidence)).toBe("unknown"); + pacing.observe("glm", evidence, 429); + (await pacing.acquire("glm"))(true); + } + expect(waits).toEqual([]); + }); + + test("unknown, malformed, zero-entitlement and above-allowance refusals never invent a reset", async () => { + const { pacing, waits } = fixture(); + for (const evidence of [new Headers(), headers(0, 0, 10), headers(1000, 1001, 10), + headers(1000, 500, 10), headers(1000, 0, 1200), + new Headers({ "x-ratelimit-remaining-tokens-minute": "0" })]) { + pacing.observe("glm", evidence, 429); + (await pacing.acquire("glm"))(true); + } + expect(waits).toEqual([]); + expect(mistralMinuteRefusal(headers(1000, 0, 1200))).toBe("charge-above-allowance"); + expect(mistralMinuteRefusal(headers(0, 0, 1200))).toBe("unknown"); + }); + + test("healthy cache hits and cold requests remain unpaced before a refusal", async () => { + const { pacing, waits } = fixture(); + for (const evidence of [headers(1000, 400, 600), headers(1000, 0, 600), headers(1000, 390, 10)]) { + pacing.observe("glm", evidence, 200); + (await pacing.acquire("glm"))(true); + } + expect(waits).toEqual([]); + }); + + test("an explicit Retry-After including zero takes precedence over minute fallback", async () => { + const { pacing, waits } = fixture(); + for (const hint of ["2", "0"]) { + const evidence = headers(1000, 400, 600); evidence.set("retry-after", hint); + pacing.observe("glm", evidence, 429); + (await pacing.acquire("glm"))(true); + } + expect(waits).toEqual([2000]); + }); + + test("a minute hold exceeding the caller's wait budget never starts a send", async () => { + const { pacing, waits } = fixture(); + pacing.observe("glm", headers(1000, 400, 600), 429); + await expect(pacing.acquire("glm", undefined, 5000)).rejects.toMatchObject({ reason: "queue_expired", retryAfterSeconds: 60 }); + expect(waits).toEqual([]); + }); + + test("one recovery probe gates concurrent callers, then a success restores ordinary traffic", async () => { + let now = 0; + let sleepers = 0; + const pacing = new MistralBudgetPacer(() => now, (_ms, signal) => new Promise((_resolve, reject) => { + sleepers++; + const abort = () => { sleepers--; reject(signal?.reason); }; + signal?.addEventListener("abort", abort, { once: true }); + })); + pacing.observe("glm", headers(1000, 400, 600), 429); + now = 60000; + const release = await pacing.acquire("glm"); + let admitted = false; + const second = pacing.acquire("glm").then(value => { admitted = true; return value; }); + await Promise.resolve(); + expect(admitted).toBe(false); + release(true); + (await second)(true); + expect(admitted).toBe(true); + expect(sleepers).toBe(0); + (await pacing.acquire("glm"))(true); + }); + + test("an intervening refusal survives an older probe's successful release", async () => { + const { pacing, waits } = fixture(); + pacing.observe("glm", headers(1000, 400, 600), 429); + const release = await pacing.acquire("glm"); + pacing.observe("glm", headers(1000, 0, 600), 429); + release(true); + (await pacing.acquire("glm"))(true); + expect(waits).toEqual([60000, 60000]); + }); + + test("transport recovery preserves the exact body and cache key, with no lasting delay after success", async () => { + const { pacing, waits } = fixture(); + const bodies: string[] = []; + const transport = resolveMistralTransport({ ...provider, fetch: (async (_input, init) => { + bodies.push(String(init?.body)); + return new Response("fixture", { status: bodies.length === 1 ? 429 : 200, + headers: headers(1000, bodies.length === 1 ? 400 : 390, bodies.length === 1 ? 600 : 10) }); + }) as typeof fetch }, pacing); + const body = JSON.stringify({ model, prompt_cache_key: "stable-fixture-conversation", messages: [{ role: "user", content: "fixture" }] }); + const send = () => transport.fetch!(`${baseUrl}/chat/completions`, { method: "POST", headers: { authorization: "Bearer synthetic-key" }, body }); + expect((await send()).status).toBe(429); + expect((await send()).status).toBe(200); + expect((await send()).status).toBe(200); + expect(waits).toEqual([60000]); + expect(bodies).toEqual([body, body, body]); + }); + + test("failed network probe releases its slot for a later caller", async () => { + const { pacing, waits } = fixture(); + pacing.observe("glm", headers(1000, 400, 600), 429); + const failed = await pacing.acquire("glm"); + failed(); + (await pacing.acquire("glm"))(true); + expect(waits).toEqual([60000]); + }); + + test("cancelling behind the recovery probe releases the waiter without affecting the probe", async () => { + let now = 0; + let sleepers = 0; + const pacing = new MistralBudgetPacer(() => now, (_ms, signal) => new Promise((_resolve, reject) => { + sleepers++; + signal?.addEventListener("abort", () => { sleepers--; reject(signal.reason); }, { once: true }); + })); + pacing.observe("glm", headers(1000, 400, 600), 429); + now = 60000; + const release = await pacing.acquire("glm"); + const abort = new AbortController(); + const pending = pacing.acquire("glm", abort.signal); + abort.abort(); + await expect(pending).rejects.toThrow(); + expect(sleepers).toBe(0); + release(true); + (await pacing.acquire("glm"))(true); + }); + + test("repeated variable-cost traffic recovers within one retry without predicting cache behavior", async () => { + let now = 0; + let window = 0; + let remaining = 1000; + let failures = 0; + let sends = 0; + let waits = 0; + const pacing = new MistralBudgetPacer(() => now, async (ms, signal) => { + signal?.throwIfAborted(); now += ms; waits++; + }); + for (let index = 0; index < 300; index++) { + // Cheap cached-shaped calls alternate with expensive cold-shaped calls. + const cost = index % 3 === 0 ? 1 : 100 + ((index * 197) % 900); + let completed = false; + for (let attempt = 0; attempt < 2; attempt++) { + const release = await pacing.acquire("glm"); + if (Math.floor(now / 60000) !== window) { window = Math.floor(now / 60000); remaining = 1000; } + const status = cost <= remaining ? 200 : 429; + sends++; + if (status === 200) remaining -= cost; + else failures++; + pacing.observe("glm", headers(1000, remaining, cost), status); + release(status === 200); + if (status === 200) { completed = true; break; } + } + expect(completed).toBe(true); + } + expect(failures).toBeGreaterThan(0); // The first refusal is reactive evidence. + expect(waits).toBe(failures); + expect(sends).toBe(300 + failures); + }); + + test("recovery waiting is outside the upstream response-header deadline", async () => { + let now = 0; + const pacing = new MistralBudgetPacer(() => now, async (ms, signal) => { + await new Promise(resolve => setTimeout(resolve, 30)); + signal?.throwIfAborted(); now += ms; + }); + let sends = 0; + const transport = resolveMistralTransport({ ...provider, fetch: (async (_input, init) => { + if (sends++ === 0) return new Response(null, { status: 429, headers: headers(1000, 400, 600) }); + init?.signal?.throwIfAborted(); + return new Response("OK"); + }) as typeof fetch }, pacing); + const body = JSON.stringify({ model, prompt_cache_key: "deadline-fixture" }); + const init = { method: "POST", body, headers: { authorization: "Bearer synthetic-key" } }; + await transport.fetch!(`${baseUrl}/chat/completions`, init); + const response = await fetchWithHeaderTimeout(`${baseUrl}/chat/completions`, init, + new AbortController().signal, 5, false, transport.fetch!); + expect(await response.text()).toBe("OK"); + expect(sends).toBe(2); + }); + + test("pausing the header deadline never disables client cancellation during admission", async () => { + let now = 0; + let sends = 0; + const pacing = new MistralBudgetPacer(() => now, async (ms, signal) => { + await sleepWithAbort(1000, signal); now += ms; + }); + const transport = resolveMistralTransport({ ...provider, fetch: (async () => { + sends++; return new Response(null, { status: 429, headers: headers(1000, 400, 600) }); + }) as typeof fetch }, pacing); + const init = { method: "POST", body: JSON.stringify({ model }), headers: { authorization: "Bearer synthetic-key" } }; + await transport.fetch!(`${baseUrl}/chat/completions`, init); + const abort = new AbortController(); + const pending = fetchWithHeaderTimeout(`${baseUrl}/chat/completions`, init, abort.signal, 5, false, transport.fetch!); + setTimeout(() => abort.abort(new Error("client stopped")), 15); + await expect(pending).rejects.toThrow("client stopped"); + expect(sends).toBe(1); + }); + + test("the response-header timeout still bounds the actual Mistral send", async () => { + const transport = resolveMistralTransport({ ...provider, fetch: (async (_input, init) => { + await sleepWithAbort(1000, init?.signal ?? undefined); return new Response("too late"); + }) as typeof fetch }, new MistralBudgetPacer()); + const init = { method: "POST", body: JSON.stringify({ model }), headers: { authorization: "Bearer synthetic-key" } }; + await expect(fetchWithHeaderTimeout(`${baseUrl}/chat/completions`, init, + new AbortController().signal, 5, false, transport.fetch!)).rejects.toThrow("Timeout elapsed"); + }); +}); diff --git a/tests/responses/responses-fetch-helpers-boundary.test.ts b/tests/responses/responses-fetch-helpers-boundary.test.ts index c048cf6063d..ce11c1b4dff 100644 --- a/tests/responses/responses-fetch-helpers-boundary.test.ts +++ b/tests/responses/responses-fetch-helpers-boundary.test.ts @@ -182,6 +182,7 @@ describe("Responses fetch-helper import boundary", () => { expect(expectRuntimeImportBoundary(readFileSync(helperPath, "utf8"))).toEqual([ // Import-free sent-header registry read by the request log (#6911). "../../lib/outbound-credential-registry", + "../../lib/provider-admission-wait", "../../lib/provider-egress", "../../lib/provider-tls-profile", "../../lib/proxy-env",