Repository navigation
Spread soonest-reset picks across accounts whose weekly resets fall within an hour (HOUSE-84) - #27
Conversation
…ithin an hour (HOUSE-84) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
✅ Accepted riskAccepted by ericlitman: HOUSE-84 (authority 48fa77b) 🔴 Critical · Signed rotation cursor can overflow and crash selection 🔴 High · Provider-wide cursor can starve an independent model pool 🟡 Medium · Retry exclusions bias rotation toward later credential IDs DetailsRun: @unfret review · @unfret status · @unfret help |
There was a problem hiding this comment.
🛑 3 blocking findings
🔴 Critical · Signed rotation cursor can overflow and crash selection
🔴 High · Provider-wide cursor can starve an independent model pool
🟡 Medium · Retry exclusions bias rotation toward later credential IDs
Check
| cursor := s.cursors[groupProvider] | ||
| s.cursors[groupProvider] = cursor + 1 |
There was a problem hiding this comment.
🔴 High · Provider-wide cursor can starve an independent model pool
The manager filters candidates by model, but all models consume the same provider cursor. With Codex model X served by [a,b] and model Y served by [c], all with equal known resets, alternating fresh sessions gives X cursor values 0,2,4,… and always selects a; b never receives a session. This reproduces through the manager with session affinity enabled, so unrelated model traffic can completely defeat the new load spreading. Also reported for this defect: - Shared provider cursor can starve independent model pools (sdk/cliproxy/auth/selector_soonest_reset.go:87-88): The manager filters candidates by model before calling this shared selector. If Codex accounts [a,b] serve one model and [c,d] serve another, all with equal known future resets, alternating fresh sessions between the models produces a,d,a,d indefinitely. One pool receives only even cursor values and the other only odd values, so b and c are never selected. This reproduces through Manager with session affinity enabled and defeats spreading on normal multi-model traffic.
Also affects:
| s.cursors[groupProvider] = cursor + 1 | ||
| return group[cursor%len(group)], nil |
There was a problem hiding this comment.
🔴 Critical · Signed rotation cursor can overflow and crash selection
The per-provider cursor is an unbounded int. On a 32-bit build, a long-lived process serving 2^31 picks for a provider wraps this value negative; with a rotation group containing at least two credentials, one of the next two calls uses a negative cursor % len(group) as a slice index and panics, crashing the proxy. The cursor must remain bounded and non-negative.
| } | ||
| cursor := s.cursors[groupProvider] | ||
| s.cursors[groupProvider] = cursor + 1 | ||
| return group[cursor%len(group)], nil |
There was a problem hiding this comment.
🟡 Medium · Retry exclusions bias rotation toward later credential IDs
Retries remove already-tried credentials before calling the selector, so applying the accumulated cursor to the shortened group changes the rotation position. For [a,b,c], selecting a then retrying [b,c] selects c and skips b. With five equal-reset credentials and two non-cooling retries per session, a reproduced 50-session sequence yields attempt counts a=20, b=21, c=25, d=34, e=50. This concentrates retry load on later IDs rather than continuing the ID-order rotation, independently of the cross-model cursor issue. Also reported for this defect: - Numeric indexing biases rotation when retries shrink the group (sdk/cliproxy/auth/selector_soonest_reset.go:89): Retry selection excludes already-tried credentials, but the cursor is applied to the newly shortened slice. With one model and five eligible accounts [a,b,c,d,e], selecting a followed by a retry over [b,c,d,e] selects c, skipping b. With transient cooldowns disabled, repeating 50 three-attempt requests yields selection counts a:20, b:21, c:25, d:34, e:50. The tail account receives traffic from every request rather than the eligible accounts being picked in turn. Rotation must preserve its position across retry exclusions.
Also affects:
|
Waiver for head 7827188 (HOUSE-84). Every blocking finding on this head is classed not material by babysit-pr's consequence gate; none is verdict-changing.
If any of these occurs in production, it becomes a ticket fixed with a class rule (for example a last-picked-ID cursor per provider and model). |
✅ Merged into
|
| Condition | Required by | |
|---|---|---|
| ✅ | github-review-approved |
GitHub repository ruleset rule Mergify merge protections |
| ✅ | -draft |
Merge Protections rule Required checks |
| ✅ | -label = hold |
Merge Protections rule Required checks |
The batch, tested in place
| Condition | Required by | |
|---|---|---|
| ✅ | check-success = Analyze (actions) |
Merge Protections rule Required checks |
| ✅ | check-success = Analyze (go) |
Merge Protections rule Required checks |
| ✅ | check-success = Analyze (javascript-typescript) |
Merge Protections rule Required checks |
| ✅ | check-success = e2e-hermetic |
Merge Protections rule Required checks |
| ✅ | check-success = go |
Merge Protections rule Required checks |
| ✅ | check-success = @unfret/Unfret |
Merge Protections rule Unfret on risky paths |
| ✅ | any of: check-success = @mergify/Mergify Merge Protections, check-neutral = @mergify/Mergify Merge Protections, check-skipped = @mergify/Mergify Merge Protections |
GitHub repository ruleset rule Mergify merge protections |
Times in UTC · dashboard · how the merge queue works
Summary
Linear: HOUSE-84. The
soonest-resetselector rotated nothing: every new session went to the one account whose weekly window resets soonest. Our three Codex accounts reset within 18 seconds of each other, so batch jobs (each call a new session) put their whole load on one account until it answered 429 (HOUSE-83). New sessions now rotate across the accounts that reset within an hour of the soonest one.Established sessions keep their account:
SessionAffinitySelectorreturns an available binding before it calls this selector.Evidence
TestSoonestResetSelectorPick_RotatesWithinResetHourpicks a, b, c, a for three accounts within the hour, andTestSoonestResetSelector_AffinityKeepsSessionWhileNewSessionsRotateshows a bound session staying put while new sessions cover all three.ResetGroupBoundary); resets at 0, 50 and 100 min do not chain (ResetGroupDoesNotChain); an exhausted account is skipped (ResetGroupSkipsExhausted); Claude accounts with unknown resets keep today's first-wins order (UnknownClaudeResetsKeepOrder); the group holds only the winner's provider (ResetGroupHoldsWinnerProvider).Merge Danger
Door: two-way
Reverting restores single-account picks. No stored state changes; the cursor lives in memory.
Blast Radius: routing
Every new session on a provider whose accounts reset within an hour of each other now rotates. Today that is the three Codex accounts. Claude accounts reset days apart and keep today's order. Quota reserves still filter before selection. Deploying restarts the proxy, which drops session-affinity bindings once.
Acceptance criteria
sdk/cliproxy/auth/selector_soonest_reset_test.go.Gate results
My run on the PR head
78271882ebdfddefb91b49c37be8dd6101851ca3, clean tree, on studio2 (logs in the HOUSE-84 records folder,gate-78271882/):gofmt -lon the changed Go filesgo build -o test-output ./cmd/servergo test -race ./sdk/cliproxy/auth/...go test ./...Not verified
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.