Skip to content

fix(combos): hold the turn lease across the failover ladder - #7329

Closed
vadymhimself wants to merge 1 commit into
lidge-jun:devfrom
vadymhimself:fix/combo-child-parent-lease
Closed

vadymhimself wants to merge 1 commit into
lidge-jun:devfrom
vadymhimself:fix/combo-child-parent-lease

Conversation

@vadymhimself

@vadymhimself vadymhimself commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • A combo child relay binds the turn admission lease to its own controller (registerTurn / trackStreamLifetime) and unbinds it when its stream ends; ActiveTurnLease.unbindAbortController releases the lease once no controller is bound (src/server/lifecycle.ts). When target 1 answered HTTP 200 SSE and then failed before output (response.failed), the combo cancelled that relay, its unbind released the parent's lease, and target 2 was aborted as soon as it bound the settled lease ("turn already settled"), so the client got a 502 although target 2 was healthy.
  • handleComboResponses (src/server/responses/core.ts) now binds its own controller to the lease for the whole ladder and unbinds it on return. The accepted child's stream is bound before the ladder returns, so it keeps the lease until the stream ends; a ladder that ends without a bound stream releases it on return, as before. Without a lease nothing is bound or unbound.
  • One site rather than a per-delivery-site guard: every combo protocol reaches executeComboResponses only through handleComboResponses, and the child delivery sites keep binding the lease, so drain and maxActiveTurns still see the accepted child's stream.

Verification

  • New test a refused child relay does not settle the parent turn's admission lease in tests/server/server-combo-zero-output-failover.test.ts: target 1 streams response.created then response.failed, target 2 completes, with a real tryAdmitTurn() lease. On unmodified upstream/dev 8640c2f: fails (Expected: 200, Received: 502; log shows b/m2 failed with 502 ... upstream_reset right after a/m1). The same test without the lease reaches target 2, which isolates the lease as the cause. With this PR: 9 pass, 0 fail, and the test also asserts getActiveTurnCount() === 0 after the body is read (the lease is released when the accepted stream ends).
  • bun test tests/server/server-combo*.test.ts tests/codex-integration/bearer-admission-routed-provider.test.ts (Bun 1.4.3) -> 327 pass, 0 fail.
  • bun x tsc --noEmit -> exit 0. bun scripts/file-size-ratchet.ts -> passed. tests/ci-workflows/structure-ssot.test.ts -> 51 pass.
  • bun run test:changed (Bun 1.4.3 via OCX_TEST_RUNNER_BUN) -> 15816 pass, 34 skip, 1 fail across 631 files. The single failure, tests/cli/cli-connect-readiness.test.ts "a rejected preferred runtime falls back without rewriting the saved selection", also fails on an unmodified upstream/dev checkout on this machine (local-environment dependent) and is unrelated.
  • Full bun run test was not run locally; the remainder is left to CI.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed. (No doc change: the lease contract is described at the change site; structure/transports/responses-failover.md is at its 600-line budget.)
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. (Admission accounting only; no credentials, auth, or defaults touched.)

Source: carried from the vadymhimself fork build (commits c93bb49, cdd54a8, 23c91b4). The fork suppressed the lease at each child delivery site and re-attached it to the accepted response; on upstream/dev the lease already supports several bound controllers, so holding it at the ladder is the smaller equivalent.

🤖 Generated with Claude Code

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

  • Required local validation passed; commands, results, and any full-suite exception are documented.

  • I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).

  • I resolved all correct Codex and CodeRabbit findings.

  • My PR is ready for review.

Summary by CodeRabbit

  • Bug Fixes
    • Improved streaming response recovery when an upstream response fails. The request can now succeed using a backup response, while the completed turn is properly cleared afterward. This helps prevent failed attempts from interfering with subsequent requests and ensures response handling remains consistent during failover.

Each combo child relay binds the turn admission lease to its own controller
and unbinds it when its stream ends; the lease releases once nothing is
bound. When target 1 returned 200 SSE and then failed before output, the
combo cancelled that relay, its unbind released the parent's lease, and
target 2 was aborted on bind with "turn already settled" (502).

handleComboResponses now binds its own controller for the whole ladder and
unbinds it on return. The accepted child's stream is bound before that, so
it keeps the lease until it ends.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

⏳ DRAFT

  • review readiness checklist open (0/4 boxes ticked).

What to do

  • Tick all four boxes in the PR description once you're done (currently 0/4).

Review readiness checklist

  • ⬜ Required local validation passed; commands, results, and any full-suite exception are documented.
  • ⬜ I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
  • ⬜ I resolved all correct Codex and CodeRabbit findings.
  • ⬜ My PR is ready for review.

0/4 boxes ticked.

This PR stays in draft until every box above is ticked.

@github-actions
github-actions Bot marked this pull request as draft October 18, 2026 17:39
@coderabbitai

coderabbitai Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: d1116ee0-2ed8-43de-8f65-cb2de7c6d574
📥 Commits

Reviewing files that changed from the base of the PR and between 8640c2f and ebda44c.

📒 Files selected for processing: 2
  • src/server/responses/core.ts
  • tests/server/server-combo-zero-output-failover.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

handleComboResponses now registers and unregisters a turn controller when a lease is provided. A streaming failover test verifies that the backup response succeeds and that no active turns remain.

Changes: Combo turn lifecycle

Layer / File(s) Summary
Turn lifecycle and failover coverage
src/server/responses/core.ts (lines 15, 71–81), tests/server/server-combo-zero-output-failover.test.ts (lines 15, 224–250)
handleComboResponses registers an abort controller with the turn admission lease, awaits combo execution, and unregisters the controller in finally when a lease exists. The streaming failover test checks the backup response and confirms that the active-turn count returns to zero.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to ebda4

The backup response can complete without the failed child ending its turn lease prematurely. No merge-blocking issue is established.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: preserving the turn admission lease across the combo failover ladder.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Ingwannu

Copy link
Copy Markdown
Owner

Intake/scoped review at ebda44c2a982c2196c6591174fb54a449e5a284d.

Holding the parent lease across the failover ladder addresses a different lifetime from a child response stream. Preserve cancellation and the handoff to the accepted child, with no leaked lease when all children fail. src/server/responses/core.ts is owner-only under CODEOWNERS; keep Draft pending that owner review and CI.

Current-head functional workflows are awaiting fork approval; that is pending execution, not a passing test result. No local suite was run for this branch in this intake. No workflow approval, new scan, live configuration change or merge was performed. Required review/readiness gates remain in force.

@lidge-jun

Copy link
Copy Markdown
Owner

Thank you @vadymhimself! This fix is absorbed into #7355 with your commit and authorship intact. Your regression reproduced the 502 on current dev. One follow-up commit there unbinds only a lease that actually registered a controller, which avoids an unmatched unbind on plain admission leases. The maintainer will close this PR when #7355 lands.

lidge-jun added a commit that referenced this pull request Oct 19, 2026
) (#7355)

* fix(combo): retry zero-output response.incomplete with no reason

A combo target that ends with `response.incomplete` before any output and
without `incomplete_details.reason` was accepted as the turn's answer, so
the caller got an empty incomplete turn instead of failover to the next
target. Treat a missing reason like the transport reasons already listed;
semantic reasons such as max_output_tokens still commit the target.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(combos): hold the turn lease across the failover ladder

Each combo child relay binds the turn admission lease to its own controller
and unbinds it when its stream ends; the lease releases once nothing is
bound. When target 1 returned 200 SSE and then failed before output, the
combo cancelled that relay, its unbind released the parent's lease, and
target 2 was aborted on bind with "turn already settled" (502).

handleComboResponses now binds its own controller for the whole ladder and
unbinds it on return. The accepted child's stream is bound before that, so
it keeps the lease until it ends.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(responses): count Combo recovery sends in the shared send budget

* fix(responses): preserve reserved Anthropic pool recovery sends

* test(adapters): assert exact combo cooldown rejection and single refund

* fix(combo): log the upstream refusal reason on combo target failure

The combo hop warning printed only `<target> failed with 401 after Nms`,
so a revoked credential or plan downgrade logged hundreds of identical
lines that read like a provider outage. Append the upstream error code and
parsed message (falling back to the normalized body text), collapsed to one
line, redacted and capped at 200 characters via sanitizeLogMetadataString.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(combo): log only the parsed upstream code and message on a hop failure

Drop the classificationText fallback so an unparsed upstream body never
reaches the warning line; the log boundary is now the two parsed fields.

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(combo): redact failure fields before flattening log metadata

Preserve credential framing until the shared redactor has removed credential values. Cover multipart framing in a parsed upstream code at the actual combo hop warning boundary.

Co-authored-by: Vadym O <bolein95@gmail.com>

* fix(combo): retain output carried by reasonless incomplete snapshots

A terminal-only upstream can include text or tool items in its final snapshot. Keep that response committed, and cover explicit semantic/unknown reasons and output-first streams as well.

Co-authored-by: Vadym O <bolein95@gmail.com>

* fix(combo): unbind only registered turn lease controllers

Match registerTurn capability checks for plain admission leases so the ladder does not report a spurious release miss. Add accepted fallback stream completion, cancellation and caller-abort coverage with exactly-once attached lease settlement.

Co-authored-by: Vadym O <bolein95@gmail.com>

* docs(combo): describe incomplete, lease and recovery-budget boundaries

* docs(structure): split combo streaming boundaries within the line budget

---------

Co-authored-by: Vadym O <bolein95@gmail.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Epinephrine <luvs01@hanmail.net>
@lidge-jun

Copy link
Copy Markdown
Owner

Thank you for this fix! It was absorbed into #7355 (merged into dev as 718692b) with your original commits/Co-authored-by credit. Closing in favor of that PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants