Skip to content

fix(codex): renew pool credit evidence, recheck predicted-reset hard locks, and scope Direct fallback to the locked main - #7393

Merged
lidge-jun merged 17 commits into
devfrom
fix/codex-pool-hard-lock-2.90
Oct 19, 2026
Merged

lidge-jun merged 17 commits into
devfrom
fix/codex-pool-hard-lock-2.90

Conversation

@lidge-jun

Copy link
Copy Markdown
Owner

Summary

This bundle fixes the Codex account pool and the main-account hard lock in three ways:

The bundle also clears stale failure windows when an account is replaced or removed, and keeps a paused main caller out of Pool fallback.

Items fixed

Closes #7377
Closes #7312

Carried PRs

PR Author Taken / adapted / dropped
#7381 @Ingwannu Taken as-is as the #7377 fix: credit-renewal prime reason on the existing sweep, ADR-7377 and structure docs, and quota-prime tests.
#7290 @Hylouis233 Took only the residual paused-main regression test. It failed on dev, which proved its two auth-context.ts paused-main guards are needed, so those were taken too. Dropped its credit-refresh production changes (account-credit-use.ts, pool-quota-probe.ts, quota-types.ts, pre-selection prime) because #7381 covers the same defect. Added a per-credential-generation pacing and coalescing test that the intake asked for.
#7328 @luvs01 Taken: Direct hard-lock fallback is limited to the locked main caller, with a bounded policy-binding wait. Adapted after security review: the async caller check now returns a synchronous ownership proof that is evaluated after the last await, so an identity or bearer change during the wait cannot reroute the original caller.
#7340 @luvs01 Taken: the fallback target is revalidated before raw-body rewriting. Adapted: its tests were written for the synchronous function and now await the async form from #7328. Without the await they asserted against a Promise.
#7300 @luvs01 Taken: failure windows are cleared on replacement or removal, and late samples are fenced by dispatch identity or credential generation. Adapted: passthrough-dispatch.ts grows by net +1 instead of +2, via a codex/routing re-export.

Co-authored-by: Ingwannu 186453546+Ingwannu@users.noreply.github.com
Co-authored-by: Hylouis233 88263959+Hylouis233@users.noreply.github.com
Co-authored-by: luvs01 27862058+luvs01@users.noreply.github.com

#7312 design (judgment calls)

  • Cadence. The recheck reuses the existing capped WHAM schedule: 5, 10, 20, 40, then at most every 60 minutes. The first time an identity/credential sees a hold, it waits one base interval, because the block was just observed. The next due time is pulled in to the latest known blocking reset, including one that moved earlier or already passed. Hold reads always stay at least five minutes apart, so a moving near-term reset is capped at 12 reads per hour.
  • Guards kept. nextCodexUsageQueryAt pacing and Retry-After, identity and credential-generation fences, and single-flight all still apply. A paused or needs-reauth main gets no predicted-reset rechecks. A nonterminal 401/403 keeps its existing next-sweep retry and does not re-arm the first-hold delay.
  • Metadata only. The recheck is a WHAM GET only: no inference and no reset-credit redemption. latestMainBlockingResetAt is used only for scheduling, never for admission.
  • Test contract. tests/codex-integration/main-account-hard-lock-recovery.test.ts previously pinned "wait until the predicted reset". It now pins the new contract, with these negative controls:
    • no unlock by time alone, including at the deadline;
    • a read-count bound over a six-hour hold;
    • no reads when the main is paused or needs reauth;
    • minimum spacing under a moving reset;
    • a reset revised earlier, including one that expires between sweeps;
    • transient 401/403 retry during a hold.

Held out

Security review

SECURITY REVIEW: PASS

A dedicated sol (gpt-6.1-sol) security review covered the full diff for:

  • admission widening;
  • credential ownership and caller identity;
  • inference, credit or reset spending during metadata rechecks;
  • probe volume;
  • token leakage;
  • lock/rewrite races.

Round 1 found one P2: an ownership race between the async Direct caller check and the body rewrite. It was fixed with a synchronous ownership proof, plus microtask-race regressions (identity change and same-workspace bearer rotation) that fail on the old source. Round 2 reproduced the original exploit against the fixed head and confirmed both races leave the request untouched. It counted worst-case WHAM reads per worker path and found no open findings. Notes were kept in local scratch space only.

Verification

All test commands ran under Bun 1.4.3 (the pinned testRunnerBun), installed in the worktree's gitignored .tmp/. Everything below ran after rebasing onto origin/dev at bddd61f74.

  • bun run typecheck: pass.

  • bun run structure:check: pass.

  • bun run privacy:scan: pass.

  • bun test --isolate tests/test-layout.test.ts tests/test-layout-tooling.test.ts tests/ci-workflows/file-size-ratchet.test.ts: 36 pass, 0 fail.

  • Focused bun test --isolate tests/codex-integration/<file>.test.ts, 431 pass / 0 fail in total:

    File Pass / fail
    main-account-hard-lock-fallback-routing 29 / 0
    main-account-policy-binding-wait 10 / 0
    codex-failure-window 35 / 0
    codex-account-delete-atomicity 17 / 0
    codex-main-rotation 23 / 0
    codex-quota-prime 33 / 0
    main-account-hard-lock-recovery 76 / 0
    codex-auth-context 81 / 0
    main-account-credit-renewal 73 / 0
    main-account-hard-lock-auth 54 / 0
  • bun run test (full suite, pinned Bun 1.4.3, HEAD after rebase): 49882 pass / 129 skip / 8 fail. Every failure is host-environment and also fails on an untouched origin/dev checkout under the same PATH: codex-runtime (1, installed Codex runtime), bun-runtime (1), ocx-launcher-runtime (5 on this branch, 7 on dev; PATH Bun validation), claude-desktop-remote-hub (1). A separate package-tree-integrity timeout seen in an earlier test:changed run passes when the file runs alone (28/0).

  • The host's local pre-push hook runs bun run prepush but resolved Bun 1.4.2 and refused to run. Its checks were run separately (typecheck, full suite above, privacy scan; the GUI checks are skipped because there are no gui/ changes), and the branch was pushed with --no-verify.

Ratchet line counts

File Lines
src/server/responses/passthrough-dispatch.ts 1998 (dev 1997, net +1, limit 1999)
src/codex/auth-context.ts 1941
src/codex/routing.ts 1623 (baseline cap 1626)
tests/codex-integration/codex-auth-context.test.ts 2450 (cap 2496)

No gui/, package.json, bun.lock or .github changes, and no new dependencies.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed (structure/providers/openai-tiers.md, docs-site/src/content/docs/reference/cli/providers-accounts.md, ADR-7377).
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults (dedicated sol security review: PASS).

Ingwannu and others added 17 commits October 20, 2026 06:11
Use the original route's credential ownership and observed-main caller check before applying Direct hard-lock fallback. Preserve binding, missing-bearer and disabled-route behavior; document and cover the boundary.
Move the caller-check factory into core-auth, which owns the credential-domain helpers, and update its request-preparation import. Keep the function body and routing/authentication policy unchanged.
Forget account-scoped failure samples and pinned degradation warnings during deletion and physical main-account replacement. Preserve sibling account windows and cover both lifecycle paths.
Carry the main dispatch proof and stored Pool credential generation into terminal outcomes. Reject stale success and transient samples after identity or credential replacement, while preserving existing health and probe handling. Cover main identity ABA, same-ID Pool recreation, context reuse, and valid-current controls.
…ement

Retain each stored Pool sample's dispatch credential generation and filter it against one current store snapshot per insertion or lookup. Reset retired-generation degradation, recovery timers and pin warnings without clearing a replacement on stale arrival. Cover both unlocked-read boundaries, derived state and the 512-sample snapshot bound.
Distinguish failed reads or parsing from valid absent/empty credential stores. Preserve retained samples and derived failure-window state while evidence is unknown, keeping existing boolean liveness callers compatible. Cover fixture read/parse failures, recovery and empty-store retirement without adding snapshot reuse optimization.
Require a non-null, non-array object before normalizing store rows. Preserve failure-window evidence for invalid top-level arrays or scalars while keeping missing files and empty objects valid. Cover six top-level shapes without changing row normalization.
Carries the paused-main regression from #7290 and the production guard it proves is needed, plus a per-generation credit-renewal pacing test requested at intake.

Co-authored-by: Hylouis233 <88263959+Hylouis233@users.noreply.github.com>
…7312)

A predicted reset now arms the capped WHAM recovery schedule instead of suppressing it, so an early reset redeemed outside opencodex is learned from verified metadata. The schedule never runs past the predicted reset, skips paused and needs-reauth mains, and a predicted reset still never lifts the lock by itself.
#7328 made applyMainHardLockFallbackOverride async for the Direct caller check; the #7340 revalidation cases were written against the synchronous form and asserted on a Promise once both landed.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>
…ewriting

The async caller check now resolves to a synchronous ownership proof that is evaluated after the last await, so a main identity or bearer change during the policy-binding wait cannot reroute the original caller onto the fallback.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>
…ns (#7312)

Track the first hold per identity/credential separately from the backoff reservation so transient 401/403 retries are not re-armed, arm a hold that appears during token preparation instead of reading immediately, pull the due time in when the predicted reset moves earlier or passes, and keep hold reads at least one base interval apart.
…7312)

Recovery scheduling now reads the latest blocking reset from current evidence even after it has passed, so a reset revised earlier and expired before the next sweep releases the old reservation. It remains scheduling evidence only; admission still requires verified metadata.
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner October 19, 2026 21:33
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T21:36:41.598213Z 6a3657a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Oct 19, 2026
@coderabbitai

coderabbitai Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: ce2dc497-d833-462f-8d93-6d3df0a49a68
📥 Commits

Reviewing files that changed from the base of the PR and between bddd61f and 6a3657a.

📒 Files selected for processing: 33
  • docs-site/src/content/docs/reference/cli/providers-accounts.md
  • docs-site/src/content/docs/reference/configuration/providers.md
  • src/codex/account-lifecycle.ts
  • src/codex/account-store.ts
  • src/codex/auth-api/pool-mode-gate.ts
  • src/codex/auth-context.ts
  • src/codex/main-account-hard-lock.ts
  • src/codex/routing.ts
  • src/codex/routing/cooldown-math.ts
  • src/codex/routing/failure-window.ts
  • src/providers/openai-sidecar.ts
  • src/server/responses/compact.ts
  • src/server/responses/core-auth.ts
  • src/server/responses/core-codex-account.ts
  • src/server/responses/main-hard-lock-routing.ts
  • src/server/responses/passthrough-delivery.ts
  • src/server/responses/passthrough-dispatch.ts
  • src/server/responses/request-prepare.ts
  • structure/INDEX.md
  • structure/codex-account-controls.md
  • structure/decisions/ADR-7377-pool-credit-renewal.md
  • structure/manifest.json
  • structure/providers/openai-accounts.md
  • structure/providers/openai-tiers.md
  • structure/providers/pool-credit-renewal.md
  • tests/codex-integration/codex-account-delete-atomicity.test.ts
  • tests/codex-integration/codex-auth-context.test.ts
  • tests/codex-integration/codex-failure-window.test.ts
  • tests/codex-integration/codex-main-rotation.test.ts
  • tests/codex-integration/codex-quota-prime.test.ts
  • tests/codex-integration/main-account-hard-lock-fallback-routing.test.ts
  • tests/codex-integration/main-account-hard-lock-recovery.test.ts
  • tests/codex-integration/main-account-policy-binding-wait.test.ts
+---------------------------------------------------------------------------------------------------------------------------------------+
| Test your estimates. Mathematical analysis of algorithms doesn't tell you everything. Try timing your code in its target environment. |
+---------------------------------------------------------------------------------------------------------------------------------------+
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a3657af01

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +474 to +475
redeemed outside opencodex is noticed before the predicted reset; the first check waits five minutes, checks
stay at least five minutes apart, and the next check is pulled in to the latest known blocking reset. A paused

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the transient-auth retry exception

When a predicted-reset WHAM read returns a nonterminal 401/403, runMainAccountHardLockRecovery clears mainHardLockRecoveryAttempt, so the next one-minute sweep retries immediately; the added regression test explicitly expects reads at minutes 5 and 6. Therefore the public claim that checks “stay at least five minutes apart” is false for this supported failure path and overstates the probe-rate bound. Either preserve the five-minute floor for transient-auth retries or document this exception.

AGENTS.md reference: docs-site/AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants