Repository navigation
fix(combos): hold the turn lease across the failover ladder - #7329
vadymhimself wants to merge 1 commit into
Conversation
Each combo child relay binds the turn admission lease to its own controller and unbinds it when its stream ends; the lease releases once nothing is bound. When target 1 returned 200 SSE and then failed before output, the combo cancelled that relay, its unbind released the parent's lease, and target 2 was aborted on bind with "turn already settled" (502). handleComboResponses now binds its own controller for the whole ladder and unbinds it on return. The accepted child's stream is bound before that, so it keeps the lease until it ends. Co-Authored-By: Claude Code <noreply@anthropic.com>
|
✅ Deterministic PR hygiene checks passed. |
⏳ DRAFT
What to do
Review readiness checklist
0/4 boxes ticked. This PR stays in draft until every box above is ticked. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
|
Intake/scoped review at Holding the parent lease across the failover ladder addresses a different lifetime from a child response stream. Preserve cancellation and the handoff to the accepted child, with no leaked lease when all children fail. src/server/responses/core.ts is owner-only under CODEOWNERS; keep Draft pending that owner review and CI. Current-head functional workflows are awaiting fork approval; that is pending execution, not a passing test result. No local suite was run for this branch in this intake. No workflow approval, new scan, live configuration change or merge was performed. Required review/readiness gates remain in force. |
|
Thank you @vadymhimself! This fix is absorbed into #7355 with your commit and authorship intact. Your regression reproduced the 502 on current |
) (#7355) * fix(combo): retry zero-output response.incomplete with no reason A combo target that ends with `response.incomplete` before any output and without `incomplete_details.reason` was accepted as the turn's answer, so the caller got an empty incomplete turn instead of failover to the next target. Treat a missing reason like the transport reasons already listed; semantic reasons such as max_output_tokens still commit the target. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(combos): hold the turn lease across the failover ladder Each combo child relay binds the turn admission lease to its own controller and unbinds it when its stream ends; the lease releases once nothing is bound. When target 1 returned 200 SSE and then failed before output, the combo cancelled that relay, its unbind released the parent's lease, and target 2 was aborted on bind with "turn already settled" (502). handleComboResponses now binds its own controller for the whole ladder and unbinds it on return. The accepted child's stream is bound before that, so it keeps the lease until it ends. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(responses): count Combo recovery sends in the shared send budget * fix(responses): preserve reserved Anthropic pool recovery sends * test(adapters): assert exact combo cooldown rejection and single refund * fix(combo): log the upstream refusal reason on combo target failure The combo hop warning printed only `<target> failed with 401 after Nms`, so a revoked credential or plan downgrade logged hundreds of identical lines that read like a provider outage. Append the upstream error code and parsed message (falling back to the normalized body text), collapsed to one line, redacted and capped at 200 characters via sanitizeLogMetadataString. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(combo): log only the parsed upstream code and message on a hop failure Drop the classificationText fallback so an unparsed upstream body never reaches the warning line; the log boundary is now the two parsed fields. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(combo): redact failure fields before flattening log metadata Preserve credential framing until the shared redactor has removed credential values. Cover multipart framing in a parsed upstream code at the actual combo hop warning boundary. Co-authored-by: Vadym O <bolein95@gmail.com> * fix(combo): retain output carried by reasonless incomplete snapshots A terminal-only upstream can include text or tool items in its final snapshot. Keep that response committed, and cover explicit semantic/unknown reasons and output-first streams as well. Co-authored-by: Vadym O <bolein95@gmail.com> * fix(combo): unbind only registered turn lease controllers Match registerTurn capability checks for plain admission leases so the ladder does not report a spurious release miss. Add accepted fallback stream completion, cancellation and caller-abort coverage with exactly-once attached lease settlement. Co-authored-by: Vadym O <bolein95@gmail.com> * docs(combo): describe incomplete, lease and recovery-budget boundaries * docs(structure): split combo streaming boundaries within the line budget --------- Co-authored-by: Vadym O <bolein95@gmail.com> Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: Epinephrine <luvs01@hanmail.net>
Summary
registerTurn/trackStreamLifetime) and unbinds it when its stream ends;ActiveTurnLease.unbindAbortControllerreleases the lease once no controller is bound (src/server/lifecycle.ts). When target 1 answered HTTP 200 SSE and then failed before output (response.failed), the combo cancelled that relay, its unbind released the parent's lease, and target 2 was aborted as soon as it bound the settled lease ("turn already settled"), so the client got a 502 although target 2 was healthy.handleComboResponses(src/server/responses/core.ts) now binds its own controller to the lease for the whole ladder and unbinds it on return. The accepted child's stream is bound before the ladder returns, so it keeps the lease until the stream ends; a ladder that ends without a bound stream releases it on return, as before. Without a lease nothing is bound or unbound.executeComboResponsesonly throughhandleComboResponses, and the child delivery sites keep binding the lease, so drain andmaxActiveTurnsstill see the accepted child's stream.Verification
a refused child relay does not settle the parent turn's admission leaseintests/server/server-combo-zero-output-failover.test.ts: target 1 streamsresponse.createdthenresponse.failed, target 2 completes, with a realtryAdmitTurn()lease. On unmodifiedupstream/dev8640c2f: fails (Expected: 200, Received: 502; log showsb/m2 failed with 502 ... upstream_resetright aftera/m1). The same test without the lease reaches target 2, which isolates the lease as the cause. With this PR: 9 pass, 0 fail, and the test also assertsgetActiveTurnCount() === 0after the body is read (the lease is released when the accepted stream ends).bun test tests/server/server-combo*.test.ts tests/codex-integration/bearer-admission-routed-provider.test.ts(Bun 1.4.3) -> 327 pass, 0 fail.bun x tsc --noEmit-> exit 0.bun scripts/file-size-ratchet.ts-> passed.tests/ci-workflows/structure-ssot.test.ts-> 51 pass.bun run test:changed(Bun 1.4.3 viaOCX_TEST_RUNNER_BUN) -> 15816 pass, 34 skip, 1 fail across 631 files. The single failure,tests/cli/cli-connect-readiness.test.ts"a rejected preferred runtime falls back without rewriting the saved selection", also fails on an unmodifiedupstream/devcheckout on this machine (local-environment dependent) and is unrelated.bun run testwas not run locally; the remainder is left to CI.Checklist
structure/transports/responses-failover.mdis at its 600-line budget.)Source: carried from the vadymhimself fork build (commits c93bb49, cdd54a8, 23c91b4). The fork suppressed the lease at each child delivery site and re-attached it to the accepted response; on
upstream/devthe lease already supports several bound controllers, so holding it at the ladder is the smaller equivalent.🤖 Generated with Claude Code
Review readiness checklist
This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:
Required local validation passed; commands, results, and any full-suite exception are documented.
I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
I resolved all correct Codex and CodeRabbit findings.
My PR is ready for review.
Summary by CodeRabbit