Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs-site/src/content/docs/guides/codex-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1154,7 +1154,8 @@ Pool quota responses can authorize a pause only with valid usage and a captured
that is still current. A response without that proof may refresh displayed usage without pausing an account.
Check this server’s authenticated `GET /api/codex-auth/low-quota-events` history for `logged`
alerts or save failures. Manual resume remains in force for the current quota episode. The
default alert reaches only the log and API; it does not produce a desktop or OS notification.
default alert reaches only the log and API. Add `actions.desktop: true` to also show a macOS or
Linux notification on the machine running the proxy; Windows stays log-only.

When the ChatGPT 5-hour quota is exhausted, Codex may offer a reserve fallback model
(`gpt-reserve` / Luna Reserve). While that state is active, the Codex model picker can make
Expand Down
19 changes: 15 additions & 4 deletions docs-site/src/content/docs/reference/configuration/routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ alias would produce an unparseable synthetic Fast selector. Codex's own model ca
account keeps its separate 98% hard lock. The optional shape is:

```json
{"codexPool":{"lowQuotaProtection":{"enabled":true,"threshold":80,"actions":{"pause":true,"notify":true},"windows":{"short":true,"weekly":true}}}}
{"codexPool":{"lowQuotaProtection":{"enabled":true,"threshold":80,"actions":{"pause":true,"notify":true,"desktop":true},"windows":{"short":true,"weekly":true}}}}
```

Absence or `enabled: false` disables it. `threshold` is a finite inclusive percentage from 1
Expand All @@ -52,9 +52,20 @@ With `notify`, the server writes a local log line with window and percentage but
and records a bounded event. Authenticated `GET /api/codex-auth/low-quota-events` exposes only
that server’s events, including account id, window, usage percentage, known reset time, timestamp,
and status. The default status is `logged`: the alert reached the log and event history only.
There is no desktop or OS notification. Each account/window logs once per server-local episode.
If an injected notification sink fails, its failure is recorded and a later eligible observation
can retry it; only a successful sink is marked `delivered`.
Each account/window logs once per server-local episode.

Optional `actions.desktop: true` (it requires `notify: true`) also hands each notice to the
operating-system notifier of the machine running the proxy: `/usr/bin/osascript` on macOS, or
`notify-send` from `/usr/bin`, `/usr/local/bin` or `/bin` on Linux. The notification names the account
only by its pseudonymous log label (for example `p1a2b3c`), never its email, alias or id, and
includes the window, usage, threshold and known reset time. Windows and hosts without a supported
notifier keep the alert `logged`. `delivered` means the OS notifier accepted the notice; it cannot
see whether the system's notification settings then hide it. A failed or timed-out notifier is
recorded as `failed`; an eligible observation at least five minutes later can retry it, without
repeating the log line. Turning `desktop` on or off
applies to the next observation without a restart. A notice still in flight is cancelled at shutdown, and
when the next quota observation detects that its episode ended (reset boundary, recovery,
account removal or policy change); until then the notifier is bounded by a five-second timeout.

## Model resolution order

Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/reference/management-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -776,7 +776,7 @@ manager. Its routes are:
| `PUT, PATCH /api/codex-auth/pool-strategy` | Update Codex account-pool selection strategy | 400 invalid strategy/config |
| `PUT /api/codex-auth/failover` | Set the account failover threshold | 400 invalid threshold |
| `GET /api/codex-auth/quota` | Read cached quota state by account | — |
| `GET /api/codex-auth/low-quota-events?limit=20` | Read only this server’s last 0–100 low-quota log/notice and pause-save events (default 20); includes account id and status (`logged` for the default log-only alert; `delivered` for a successful injected notice sink; `succeeded` for a completed pause save) | 400 invalid limit; management authentication required |
| `GET /api/codex-auth/low-quota-events?limit=20` | Read only this server’s last 0–100 low-quota log/notice and pause-save events (default 20); includes account id and status (`logged` for the default log-only alert or a host without a supported notifier; `delivered` when the opted-in OS notifier accepted the notice; `failed` when it failed or timed out; `cancelled` when its episode ended first; `succeeded` for a completed pause save) | 400 invalid limit; management authentication required |
| `GET /api/codex-auth/reset-credits` | Inspect reset-credit eligibility for an account | 400 missing account id; upstream status passthrough; 500 lookup failure |
| `POST /api/codex-auth/reset-credits/consume` | Consume an eligible reset credit. Optional `operationId` (UUIDv4) makes the redemption idempotent: the same id replays one durable outcome instead of spending a second credit. | 400 missing account id or invalid `operationId`; 409 `identity_mismatch` when the id belongs to another account; upstream status passthrough; 503 `server_busy`, `capacity`, or `unavailable`; 500 consume failure |
| `POST /api/codex-auth/login` | Start Codex login or reauthentication | 400 invalid request; conflict/busy login states |
Expand Down
1 change: 1 addition & 0 deletions scripts/test-layout/layout.json
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@
"cursor-local-installer.test.ts": "providers/cursor",
"codex-quota-query-backoff.test.ts": "codex-integration",
"pnpm-command-isolation.test.ts": "update", "provider-antigravity-quota-retry.test.ts": "providers", "project-config-warning-snapshot.test.ts": "codex-integration", "codex-quota-auto-refresh-generation.test.ts": "codex-integration", "codex-account-clear-paused.test.ts": "codex-integration",
"low-quota-desktop-notifier.test.ts": "codex-integration",
"low-quota-protection.test.ts": "codex-integration",
"bun-runtime-preflight.test.ts": "lib",
"responses-compaction-recovery.test.ts": "responses", "responses-canonical-nonstream.test.ts": "responses", "compaction-recovery-settings.test.ts": "config", "responses-compaction-recovery-policy.test.ts": "responses", "plugin-loader.test.ts": "lib", "plugin-upstream-hooks.test.ts": "lib",
Expand Down
125 changes: 125 additions & 0 deletions src/codex/low-quota-desktop-notifier.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
import { existsSync } from "node:fs";
import { isTestHomeGuardArmed } from "../lib/test-home-guard";

/** What a low-quota notice may tell the operating system. Never an email, raw id or credential. */
export type LowQuotaNotice = {
/** Pseudonymous `p<hex6>` log label of the pool account. */
accountLabel: string;
window: "short" | "weekly";
percentUsed: number;
threshold: number;
/** Epoch milliseconds of the window reset, when known. */
resetAt: number | null;
};

/** Resolves when the OS notifier accepted the notice; rejects on failure, timeout or abort. */
export type LowQuotaNoticeSink = (notice: LowQuotaNotice, signal: AbortSignal) => Promise<void>;

type SpawnedNotifier = { exited: Promise<number | null>; kill(): void };
export type LowQuotaNotifierIo = {
platform?: NodeJS.Platform;
/** Returns true when the absolute path is an existing executable candidate. */
exists?: (path: string) => boolean;
spawn?: (argv: string[]) => SpawnedNotifier;
timeoutMs?: number;
/** Formats the reset time for display; defaults to the host locale. */
formatTime?: (epochMs: number) => string;
};

const OSASCRIPT = "/usr/bin/osascript";
/** Literal system locations only: a PATH lookup would let any earlier PATH entry run as the proxy. */
const NOTIFY_SEND_CANDIDATES = ["/usr/bin/notify-send", "/usr/local/bin/notify-send", "/bin/notify-send"];
const DEFAULT_TIMEOUT_MS = 5_000;
const TITLE = "OpenCodex quota";

export function formatLowQuotaNotice(notice: LowQuotaNotice, formatTime: (epochMs: number) => string = defaultFormatTime): { title: string; body: string } {
const window = notice.window === "short" ? "5-hour" : "weekly";
const reset = notice.resetAt === null ? "reset time unknown" : `resets ${formatTime(notice.resetAt)}`;
return {
title: TITLE,
body: `Account ${notice.accountLabel}: ${window} quota at ${Math.round(notice.percentUsed)}% (threshold ${notice.threshold}%), ${reset}.`,
};
}

function defaultFormatTime(epochMs: number): string {
return new Date(epochMs).toLocaleString();
}

/** notify-send bodies may be parsed as markup by the notification server. */
function escapeMarkup(text: string): string {
return text.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
}

/**
* Builds the fixed argv for the host notifier. The notice text is always a separate argv item:
* the AppleScript source is constant and reads its arguments, and notify-send stops option
* parsing at `--`, so no account or quota text is ever interpreted as script or flags.
*/
export function lowQuotaNotifierArgv(
platform: NodeJS.Platform,
exists: (path: string) => boolean,
title: string,
body: string,
): string[] | undefined {
if (platform === "darwin") {
if (!exists(OSASCRIPT)) return undefined;
return [OSASCRIPT, "-e", "on run argv", "-e", "display notification (item 2 of argv) with title (item 1 of argv)", "-e", "end run", title, body];
}
if (platform === "linux") {
const notifySend = NOTIFY_SEND_CANDIDATES.find(exists);
if (!notifySend) return undefined;
return [notifySend, "--app-name=OpenCodex", "--", title, escapeMarkup(body)];
}
return undefined;
}

function defaultSpawn(argv: string[]): SpawnedNotifier {
const proc = Bun.spawn(argv, { stdin: "ignore", stdout: "ignore", stderr: "ignore", windowsHide: true });
return { exited: proc.exited, kill: () => { try { proc.kill(); } catch { /* already gone */ } } };
}

/**
* Returns a sink that hands notices to this host's notification service, or undefined when the
* platform has none we support (Windows and others stay log-only). An armed test process never
* reaches a real notifier unless a spawn is injected.
*/
export function resolveLowQuotaDesktopSink(io: LowQuotaNotifierIo = {}): LowQuotaNoticeSink | undefined {
if (!io.spawn && isTestHomeGuardArmed()) return undefined;
const platform = io.platform ?? process.platform;
const exists = io.exists ?? existsSync;
const spawn = io.spawn ?? defaultSpawn;
const timeoutMs = io.timeoutMs ?? DEFAULT_TIMEOUT_MS;
const probe = lowQuotaNotifierArgv(platform, exists, TITLE, "");
if (!probe) return undefined;
return (notice, signal) => {
if (signal.aborted) return Promise.reject(new Error("low-quota notice cancelled"));
const { title, body } = formatLowQuotaNotice(notice, io.formatTime);
const argv = lowQuotaNotifierArgv(platform, exists, title, body);
if (!argv) return Promise.reject(new Error("low-quota notifier unavailable"));
let child: SpawnedNotifier;
try {
child = spawn(argv);
} catch (error) {
return Promise.reject(error);
}
return new Promise<void>((resolve, reject) => {
let settled = false;
const finish = (error?: Error) => {
if (settled) return;
settled = true;
clearTimeout(timer);
signal.removeEventListener("abort", onAbort);
if (error) reject(error);
else resolve();
};
const onAbort = () => { child.kill(); finish(new Error("low-quota notice cancelled")); };
const timer = setTimeout(() => { child.kill(); finish(new Error("low-quota notifier timed out")); }, timeoutMs);
(timer as { unref?: () => void }).unref?.();
signal.addEventListener("abort", onAbort, { once: true });
child.exited.then(
code => finish(code === 0 ? undefined : new Error(`low-quota notifier exited ${code}`)),
error => finish(error instanceof Error ? error : new Error("low-quota notifier failed")),
);
});
};
}
88 changes: 57 additions & 31 deletions src/codex/low-quota-protection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,36 @@ import { isCodexAccountPaused, setCodexAccountPaused } from "./account-pause";
import { createLowQuotaEventLedger, type LowQuotaEvent } from "./low-quota-events";
import { registerLowQuotaObserver } from "./low-quota-observer";
import { resetAtToMs } from "./quota-types";
import { codexAccountLogLabel, fallbackCodexAccountLogLabel } from "./account-label";
import type { LowQuotaNotice, LowQuotaNoticeSink } from "./low-quota-desktop-notifier";

type Window = "short" | "weekly";
type Notice = { window: Window; percentUsed: number; threshold: number };
type Dependencies = {
persist?: (config: OcxConfig) => void | Promise<void>;
notify?: (notice: Notice) => void | Promise<void>;
/** Injected sink; takes precedence over desktop delivery. */
notify?: (notice: LowQuotaNotice, signal: AbortSignal) => void | Promise<void>;
/** Resolved per notice, and only while the live policy opts into actions.desktop. */
desktopSink?: () => LowQuotaNoticeSink | undefined;
};
type EventBase = Omit<LowQuotaEvent, "timestamp" | "status" | "delivery">;
type Episode = { reset: string; notice: "in-flight" | "logged" | "delivered" | "failed" | undefined; pausedByUs: boolean; noticeBase?: EventBase };
type Episode = {
reset: string;
notice: "in-flight" | "logged" | "delivered" | "failed" | undefined;
pausedByUs: boolean;
noticeBase?: EventBase;
noticeAbort?: AbortController;
/** A failed notice may retry only after this time, so a broken notifier is not respawned per request. */
noticeRetryAt?: number;
/** The episode's log line was written; retries do not repeat it. */
warned?: boolean;
};
export type LowQuotaRegistration = (() => void) & {
hasPendingSave(): boolean;
flush(): Promise<void>;
listEvents(limit?: number): LowQuotaEvent[];
};

const RETRY_DELAYS_MS = [100, 250];
const NOTICE_RETRY_MS = 5 * 60_000;
const FLUSH_DEADLINE_MS = 500;

/** Each server owns its own policy, episode state and deferred writer. */
Expand All @@ -44,6 +58,12 @@ export function registerCodexLowQuotaProtection(config: OcxConfig, deps: Depende
if (delivery === "pause-save") pauseStatus.set(base, status);
ledger.publish({ ...base, delivery, status, timestamp: Date.now() });
}
/** Publishes one cancellation for an in-flight notice and stops its delivery. */
function cancelNotice(episode: Episode): void {
if (episode.notice === "in-flight" && episode.noticeBase) event(episode.noticeBase, "notice", "cancelled");
episode.noticeAbort?.abort();
episode.noticeAbort = undefined;
}
function cancelTimer(): void {
if (saveTimer) clearTimeout(saveTimer);
saveTimer = null;
Expand Down Expand Up @@ -132,9 +152,7 @@ export function registerCodexLowQuotaProtection(config: OcxConfig, deps: Depende
if (pauseStatus.get(base) === "pending") event(base, "pause-save", "cancelled");
pendingPauseEvents.delete(key);
}
for (const episode of episodes.values()) {
if (episode.notice === "in-flight" && episode.noticeBase) event(episode.noticeBase, "notice", "cancelled");
}
for (const episode of episodes.values()) cancelNotice(episode);
dirty = false;
autoPausedAccounts.clear();
unregister();
Expand Down Expand Up @@ -175,9 +193,7 @@ export function registerCodexLowQuotaProtection(config: OcxConfig, deps: Depende
const policy = config.codexPool?.lowQuotaProtection;
const nextKey = JSON.stringify(policy);
if (policyKey !== nextKey) {
for (const episode of episodes.values()) {
if (episode.notice === "in-flight" && episode.noticeBase) event(episode.noticeBase, "notice", "cancelled");
}
for (const episode of episodes.values()) cancelNotice(episode);
Comment thread
lidge-jun marked this conversation as resolved.
episodes.clear();
autoPausedAccounts.clear();
policyKey = nextKey;
Expand All @@ -187,7 +203,7 @@ export function registerCodexLowQuotaProtection(config: OcxConfig, deps: Depende
for (const id of autoPausedAccounts) if (!liveIds.has(id)) autoPausedAccounts.delete(id);
for (const [key, episode] of episodes) {
if (liveIds.has(key.split("\u0000")[0]!)) continue;
if (episode.notice === "in-flight" && episode.noticeBase) event(episode.noticeBase, "notice", "cancelled");
cancelNotice(episode);
episodes.delete(key);
}
if (!liveIds.has(accountId)) return;
Expand All @@ -203,14 +219,15 @@ export function registerCodexLowQuotaProtection(config: OcxConfig, deps: Depende
const key = `${accountId}\u0000${window}`;
if (percentUsed < policy.threshold) {
const prior = episodes.get(key);
if (prior?.notice === "in-flight" && prior.noticeBase) event(prior.noticeBase, "notice", "cancelled");
if (prior) cancelNotice(prior);
episodes.delete(key);
continue;
}
const resetAt = rawReset === undefined ? null : resetAtToMs(rawReset);
const reset = resetAt === null ? "unknown" : String(resetAt);
let episode = episodes.get(key);
if (!episode || episode.reset !== reset) {
if (episode) cancelNotice(episode);
episode = { reset, notice: undefined, pausedByUs: false };
episodes.set(key, episode);
}
Expand All @@ -234,32 +251,41 @@ export function registerCodexLowQuotaProtection(config: OcxConfig, deps: Depende
scheduleSave();
}
// A manual resume leaves pausedByUs set until recovery or a new reset episode.
if (policy.actions.notify && episode.notice !== "in-flight" && episode.notice !== "logged" && episode.notice !== "delivered") {
console.warn(`[codex-low-quota] ${window === "short" ? "5-hour" : "weekly"} quota reached ${percentUsed}% used (threshold ${policy.threshold}%)`);
if (!deps.notify) {
if (policy.actions.notify && episode.notice !== "in-flight" && episode.notice !== "logged" && episode.notice !== "delivered"
&& !(episode.notice === "failed" && (episode.noticeRetryAt ?? 0) > Date.now())) {
if (!episode.warned) {
episode.warned = true;
console.warn(`[codex-low-quota] ${window === "short" ? "5-hour" : "weekly"} quota reached ${percentUsed}% used (threshold ${policy.threshold}%)`);
}
const sink = deps.notify ?? (policy.actions.desktop === true ? deps.desktopSink?.() : undefined);
if (!sink) {
event(base, "notice", "logged");
episode.notice = "logged";
continue;
}
episode.notice = "in-flight";
episode.noticeBase = base;
const account = config.codexAccounts?.find(candidate => candidate.id === accountId);
const notice: LowQuotaNotice = {
accountLabel: account ? codexAccountLogLabel(account) : fallbackCodexAccountLogLabel(accountId),
window, percentUsed, threshold: policy.threshold, resetAt,
};
const abort = new AbortController();
const current = episode;
current.notice = "in-flight";
current.noticeBase = base;
current.noticeAbort = abort;
event(base, "notice", "pending");
const settle = (status: "delivered" | "failed") => {
if (closed || abort.signal.aborted || episodes.get(key) !== current) return;
event(base, "notice", status);
current.notice = status;
current.noticeRetryAt = status === "failed" ? Date.now() + NOTICE_RETRY_MS : undefined;
current.noticeBase = undefined;
current.noticeAbort = undefined;
};
Comment thread
lidge-jun marked this conversation as resolved.
try {
void Promise.resolve(deps.notify({ window, percentUsed, threshold: policy.threshold })).then(() => {
if (closed || episodes.get(key) !== episode) return;
event(base, "notice", "delivered");
episode.notice = "delivered";
episode.noticeBase = undefined;
}, () => {
if (closed || episodes.get(key) !== episode) return;
event(base, "notice", "failed");
episode.notice = "failed";
episode.noticeBase = undefined;
});
void Promise.resolve(sink(notice, abort.signal)).then(() => settle("delivered"), () => settle("failed"));
} catch {
event(base, "notice", "failed");
episode.notice = "failed";
episode.noticeBase = undefined;
settle("failed");
}
}
}
Expand Down
Loading
Loading