Skip to content

Spread soonest-reset picks across accounts whose weekly resets fall within an hour (HOUSE-84) - #27

Merged
mergify[bot] merged 1 commit into
mainfrom
claude/HOUSE-84-soonest-reset-spread
Oct 28, 2026
Merged

mergify[bot] merged 1 commit into
mainfrom
claude/HOUSE-84-soonest-reset-spread

Conversation

@ericlitman

@ericlitman ericlitman commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Linear: HOUSE-84. The soonest-reset selector rotated nothing: every new session went to the one account whose weekly window resets soonest. Our three Codex accounts reset within 18 seconds of each other, so batch jobs (each call a new session) put their whole load on one account until it answered 429 (HOUSE-83). New sessions now rotate across the accounts that reset within an hour of the soonest one.

 SoonestResetSelector.Pick(auths)
   rank available auths: ungated first, then soonest known weekly reset
-  return winner
+  if winner is gated, or its reset is unknown
+    return winner                                (unchanged: unknown order, all gated, no signals)
+  group = same-provider ungated auths with reset <= winner reset + 1 h   (anchored, not chained)
+  return group[cursor[provider]++ % len(group)]  (ID order, cursor per provider, mutex)

Established sessions keep their account: SessionAffinitySelector returns an available binding before it calls this selector.

Evidence

  • Before (HOUSE-83, proxy log 2026-10-27 11:08Z to 11:22Z): picks came in runs of up to 199 on one account and moved only after a 429. After: TestSoonestResetSelectorPick_RotatesWithinResetHour picks a, b, c, a for three accounts within the hour, and TestSoonestResetSelector_AffinityKeepsSessionWhileNewSessionsRotate shows a bound session staying put while new sessions cover all three.
  • Boundaries: an account exactly 1 h after the soonest joins the group, 1 h 1 s does not (ResetGroupBoundary); resets at 0, 50 and 100 min do not chain (ResetGroupDoesNotChain); an exhausted account is skipped (ResetGroupSkipsExhausted); Claude accounts with unknown resets keep today's first-wins order (UnknownClaudeResetsKeepOrder); the group holds only the winner's provider (ResetGroupHoldsWinnerProvider).

Merge Danger

Door: two-way

Reverting restores single-account picks. No stored state changes; the cursor lives in memory.

Blast Radius: routing

Every new session on a provider whose accounts reset within an hour of each other now rotates. Today that is the three Codex accounts. Claude accounts reset days apart and keep today's order. Quota reserves still filter before selection. Deploying restarts the proxy, which drops session-affinity bindings once.

Acceptance criteria

  • "Among available credentials of one provider, those whose known weekly reset is within one hour of the soonest known weekly reset are picked in turn for new sessions. Gated (exhausted) credentials, quota reserves, unknown-reset ordering and session affinity keep today's behaviour." Unit tests above.
  • "Unit tests: three credentials with resets within an hour get picks spread across all three; a credential resetting more than an hour later is not picked while the near group is available; an exhausted credential in the group is skipped; affinity still keeps an established session on its account." Added in sdk/cliproxy/auth/selector_soonest_reset_test.go.
  • "A load test at 16 calls at a time through the deployed proxy … shows first picks spread across the three Codex accounts and records upstream and client 429 counts beside HOUSE-83's arms." After the deploy; not yet run.
  • "Released as a fork tag and deployed on studio2 outside the Wandermuse alpha release check (2026-10-28 06:00Z to 12:00Z)." After merge.

Gate results

My run on the PR head 78271882ebdfddefb91b49c37be8dd6101851ca3, clean tree, on studio2 (logs in the HOUSE-84 records folder, gate-78271882/):

Check Exit Result
gofmt -l on the changed Go files 0 no changes
go build -o test-output ./cmd/server 0 built
go test -race ./sdk/cliproxy/auth/... 0 ok
go test ./... 0 102 packages ok

Not verified

  • The deployed load test and the account spread on live traffic: after merge and deploy.
  • Quota effect: the soonest account's quota may now expire less fully used when the group's resets differ by up to an hour. With today's 18-second spread the difference is nil.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…ithin an hour (HOUSE-84)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@unfret

unfret Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

✅ Accepted risk

Accepted by ericlitman: HOUSE-84 (authority 48fa77b)

🔴 Critical · Signed rotation cursor can overflow and crash selection

🔴 High · Provider-wide cursor can starve an independent model pool

🟡 Medium · Retry exclusions bias rotation toward later credential IDs

Details

Run: GET /unfret/run/panel:78271882ebdfddefb91b49c37be8dd6101851ca3:FqHw3ChuKgs:tbHkqrJ8DKA.

@unfret review · @unfret status · @unfret help

@unfret unfret Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 3 blocking findings
🔴 Critical · Signed rotation cursor can overflow and crash selection
🔴 High · Provider-wide cursor can starve an independent model pool
🟡 Medium · Retry exclusions bias rotation toward later credential IDs
Check

Comment on lines +86 to +87
cursor := s.cursors[groupProvider]
s.cursors[groupProvider] = cursor + 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Provider-wide cursor can starve an independent model pool

The manager filters candidates by model, but all models consume the same provider cursor. With Codex model X served by [a,b] and model Y served by [c], all with equal known resets, alternating fresh sessions gives X cursor values 0,2,4,… and always selects a; b never receives a session. This reproduces through the manager with session affinity enabled, so unrelated model traffic can completely defeat the new load spreading. Also reported for this defect: - Shared provider cursor can starve independent model pools (sdk/cliproxy/auth/selector_soonest_reset.go:87-88): The manager filters candidates by model before calling this shared selector. If Codex accounts [a,b] serve one model and [c,d] serve another, all with equal known future resets, alternating fresh sessions between the models produces a,d,a,d indefinitely. One pool receives only even cursor values and the other only odd values, so b and c are never selected. This reproduces through Manager with session affinity enabled and defeats spreading on normal multi-model traffic.

Also affects:

Comment on lines +87 to +88
s.cursors[groupProvider] = cursor + 1
return group[cursor%len(group)], nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical · Signed rotation cursor can overflow and crash selection

The per-provider cursor is an unbounded int. On a 32-bit build, a long-lived process serving 2^31 picks for a provider wraps this value negative; with a rotation group containing at least two credentials, one of the next two calls uses a negative cursor % len(group) as a slice index and panics, crashing the proxy. The cursor must remain bounded and non-negative.

}
cursor := s.cursors[groupProvider]
s.cursors[groupProvider] = cursor + 1
return group[cursor%len(group)], nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Retry exclusions bias rotation toward later credential IDs

Retries remove already-tried credentials before calling the selector, so applying the accumulated cursor to the shortened group changes the rotation position. For [a,b,c], selecting a then retrying [b,c] selects c and skips b. With five equal-reset credentials and two non-cooling retries per session, a reproduced 50-session sequence yields attempt counts a=20, b=21, c=25, d=34, e=50. This concentrates retry load on later IDs rather than continuing the ID-order rotation, independently of the cross-model cursor issue. Also reported for this defect: - Numeric indexing biases rotation when retries shrink the group (sdk/cliproxy/auth/selector_soonest_reset.go:89): Retry selection excludes already-tried credentials, but the cursor is applied to the newly shortened slice. With one model and five eligible accounts [a,b,c,d,e], selecting a followed by a retry over [b,c,d,e] selects c, skipping b. With transient cooldowns disabled, repeating 50 three-attempt requests yields selection counts a:20, b:21, c:25, d:34, e:50. The tail account receives traffic from every request rather than the eligible accounts being picked in turn. Rotation must preserve its position across retry exclusions.

Also affects:

@ericlitman

Copy link
Copy Markdown
Contributor Author

Waiver for head 7827188 (HOUSE-84). Every blocking finding on this head is classed not material by babysit-pr's consequence gate; none is verdict-changing.

  • v1:213e75da Provider-wide cursor can starve an independent model pool: pre-accepted. The trigger needs fresh sessions that strictly alternate between models served by disjoint account pools. On 2026-10-27, 6,821 of 6,836 new Codex sessions in the proxy log (99.8%) were gpt-6-luna, served by the same three accounts. The consequence is uneven spread, not security, secret exposure, data loss or gate integrity.
  • v1:4647a0c7 Signed rotation cursor can overflow and crash selection: pre-accepted. The trigger needs a 32-bit build. studio2 runs a 64-bit arm64 binary (file: Mach-O 64-bit executable arm64), where int is 64 bits and 2^63 picks never happen.
  • v1:8836b56e Retry exclusions bias rotation toward later credential IDs: inert. Retries ran on 7.7% of requests under the heaviest load we have (485 of 6,339, 2026-10-27 11:08Z to 11:22Z). With three accounts they only shift which of the two remaining accounts takes the retry; first picks still rotate across all three, which is the ticket's outcome.

If any of these occurs in production, it becomes a ticket fixed with a class rule (for example a last-picked-ID cursor per provider and model).

@mergify
mergify Bot merged commit ed0d889 into main Oct 28, 2026
9 checks passed
@mergify

mergify Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

✅ Merged into main as ed0d889 at 01:26 UTC

Less than a minute in the queue, with no time running CI.

So far

  • ✅ Entered the queue Wed 28 Oct, 01:26 · rule default · triggered by merge protections
  • ✅ Checks skipped · PR is already up-to-date
  • ✅ Merged 01:26 · ed0d889
Merge conditions: 10 of 10 met

This pull request

Condition Required by
✅ github-review-approved GitHub repository ruleset rule Mergify merge protections
✅ -draft Merge Protections rule Required checks
✅ -label = hold Merge Protections rule Required checks

The batch, tested in place

Condition Required by
✅ check-success = Analyze (actions) Merge Protections rule Required checks
✅ check-success = Analyze (go) Merge Protections rule Required checks
✅ check-success = Analyze (javascript-typescript) Merge Protections rule Required checks
✅ check-success = e2e-hermetic Merge Protections rule Required checks
✅ check-success = go Merge Protections rule Required checks
✅ check-success = @unfret/Unfret Merge Protections rule Unfret on risky paths
✅ any of: check-success = @mergify/Mergify Merge Protections, check-neutral = @mergify/Mergify Merge Protections, check-skipped = @mergify/Mergify Merge Protections GitHub repository ruleset rule Mergify merge protections

Times in UTC · dashboard · how the merge queue works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant