Skip to content

DRU-1065 -- A provider or account switch wakes a call waiting out a usage limit - #1121

Open
dserbano wants to merge 1 commit into
mainfrom
issues/1065
Open

dserbano wants to merge 1 commit into
mainfrom
issues/1065

Conversation

@dserbano

@dserbano dserbano commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

An agent call that hit a usage limit slept until the window reset. Switching the provider or the account during that sleep changed nothing: the build kept waiting, and it retried only once the old window turned. Now a switch wakes the wait, and the call retries at once on the new provider or account.

  • The quota wait is a Workflow.wait on the agents.quota topic, with the reset as its timeout. The reset is read off the latest usage scrape, as before. The step that reads the reset also records which credential hit the limit: the agent's model, its billing, and the email of the subscription it runs on.
  • Agent.wake_quota_waits wakes every run that waits on that topic. A change to the execution defaults, an agent's harness, model, or billing override, a connected subscription, or a pasted token calls it after the change commits, so the woken run reads the new settings.
  • A woken call reads its credential again. If the credential is the same, the call keeps waiting until the original reset. If it differs, the call retries now.
  • A call waits once per credential instead of once in all. If the new account is limited too, its own quota error starts one wait for that account's reset. A second limit on a credential that already waited fails the call, as before.

The retry on another credential does not consult that credential's scrape first. The scrape is keyed by the Druks account, so after a Claude account switch on the same Druks account it still describes the old Claude account. The provider's own rejection is the limit that applies.

DRU-1013's chat notice that the run continues said the usage limit reset. After a switch it has not, so QUOTA_RESUMED_MESSAGE now says the run continues after its usage-limit wait.

A pasted setup token names no email, so swapping one pasted token for another reads as the same account, and the call waits out the reset.

The quota wait records different steps, so a run waiting out a quota when this deploys fails on replay and needs a new start.

@mintlify

mintlify Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
druks 🟢 Ready View Preview Nov 10, 2026, 9:56 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@dserbano

Copy link
Copy Markdown
Collaborator Author

DRU-1065: the fix and the tests

cba71c88 is this change rebased on current main.

The fix

A build whose agent call hit a usage limit slept until the window reset. A switch to another provider or account during that sleep changed nothing. Now a switch wakes the wait, and the call retries at once.

  • The quota wait is Workflow.wait on the agents.quota topic, and the reset is its timeout. The retry_wait step reads the reset off the latest usage scrape, as before. It also records the call's quota_holder: the model, the billing, and the email of the subscription. AgentConfig.quota_holder builds that value.
  • Agent.wake_quota_waits wakes every run that waits on that topic. It runs after the change commits, so a woken run reads the new settings. These changes call it:
    • the execution defaults;
    • an agent's harness, model, or billing override;
    • a connected subscription;
    • a pasted token.
  • A woken call runs the quota_check step, which reads its quota holder again. With the same holder, the call waits until the original reset. With another holder, it retries now.
  • A call waits once per quota holder instead of once in all. If the new account is limited too, its own limit starts one wait for its own reset. A second limit on a holder that already waited fails the call, as before.

What the rebase changed

DRU-1013 (#1140) posts two chat notices from this wait: one when the wait starts and one when the run continues. The rebase keeps both inside the new wait. The second notice said the run continues "now that the provider's usage limit reset". After a switch, the limit has not reset, so QUOTA_RESUMED_MESSAGE now says the run continues after its usage-limit wait. The docstring of report_quota_resume says the same.

The rebase also names the holder quota_holder throughout. It replaces credential in the step's result, the _quota_wait_left parameter, and the comments.

Automated checks

  • ruff check, ruff format --check, and druks-house-rules check --base origin/main pass.
  • pytest backend/ passed 2646 tests. Three tests fail, and each also fails on main on my Mac:
    • test_upload_dir_tar_survives_more_warnings_than_a_pipe_holds, which is macOS-only;
    • the two test_druks_sandbox_script tests, where the script exits with 127 locally.
  • test_agents.py covers:
    • one wait until the reset, with both chat notices;
    • a wake with no switch, which keeps waiting until the reset;
    • a switched account that retries at once and waits once for its own limit.
  • test_durable_sdk.py wakes only the runs that wait on agents.quota, on real DBOS.
  • test_api_settings.py and test_api_providers.py check that a provider change, a connected subscription, and a pasted token wake the waits, and that an effort-only change does not.

Live tests

Every build ran on a local install against dserbano/druks-test-stub. I filed each board ticket and approved each plan in the dashboard. I switched agents and accounts in the Settings UI.

A real usage limit and an account switch. This ran on this commit's image. The image also held a separate change, not part of this PR, that reads a new subscription's usage when it connects.

  1. I connected the Claude subscription of one account and started six large builds.
  2. At 20:05 UTC, that account's 5-hour window was at 0%. Three implement calls failed with the CLI's real limit message, and Druks classified each as rate_limited. Each build's retry_wait step recorded that account as its holder, with a wait until 00:51 UTC.
  3. At 20:07:32, I connected the Claude subscription of a second account.
  4. In the same second, all three runs woke. Each quota_check step returned 0, because the holder had changed. All three started new implement calls on the second account at 20:07:37, almost five hours before the old reset.

Earlier runs, with a simulated limit. These ran before the rebase. A file flag made the harness return a limit error, while the usage scrapes, the waits, the switches, and the restarts were real.

Case Result
Change only the effort while a call waits ✅ The run woke, found the same holder, and kept waiting until the original reset.
Switch the agent from Claude to Codex while it waits ✅ quota_check returned 0, and the call retried on Codex at once.
Switch to Codex while Codex is limited too ✅ The Codex call failed on its own limit and waited for the OpenAI reset.
Restart the web container during that wait ✅ The run recovered and kept waiting, with no replay error.
Switch back to Claude ✅ The call retried within 2 seconds and succeeded.

Two things to know

  • A pasted setup token names no email, so one pasted token swapped for another reads as the same holder. That call waits out the reset.
  • The quota wait records different steps, so a run waiting out a quota when this deploys fails on replay and needs a new start.

@dserbano
dserbano marked this pull request as ready for review November 10, 2026 22:07
@dserbano
dserbano requested a review from czpython as a code owner November 10, 2026 22:07

This branch was successfully deployed

1 active deployment
staging - docs — cba71c88 Deployed Nov 10, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant