You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
upto sessions: over-cap request cost is dropped instead of saturated → free inference beyond the signed cap #2
In the Flask "upto" session (deferred-settlement) mode, the per-request cost is computed and charged after the response has already been streamed to the client, and if the charge would exceed the session's signed spend cap it is discarded entirely rather than saturated to the remaining budget. A single oversized request therefore delivers arbitrarily expensive inference that is never settled.
The request gate only checks session.is_exhausted (i.e. accumulated_cost >= max_amount from prior requests). The cost of the current request is known only after generation and is added in StreamingSessionResponse.close(). When it would overflow the cap, add_cost drops the whole charge:
# session.pyifsession.accumulated_cost+cost>session.max_amount:
returnFalse# entire cost discarded — response already served
_accumulate_session_cost merely logs "Could not add cost ...". Because _settle_session early-returns on accumulated_cost <= 0, a session whose only request overflowed the cap is never settled at all.
Attack
Open a session with a small cap (e.g. enough for one cheap call).
Send one request with maximum context and max_tokens. Gate sees accumulated_cost=0 < max → allowed. Operator pays the upstream provider for the full generation.
At close, cost >> max_amount → add_cost returns False → accumulated_cost stays 0 → nothing is ever settled.
Result: arbitrarily expensive inference for free. (The Redis path behaves identically — the atomic Lua script keeps the accounting correct but the work is already delivered.)
Impact
Direct, unbounded operator loss (real provider spend not recovered on-chain). High severity for any deployment running session mode with a live facilitator.
Suggested fixes
Minimum: saturate the charge to the cap instead of dropping it — accumulated_cost = min(accumulated_cost + cost, max_amount) — so the operator always collects up to the full signed amount, and mark the session exhausted/settle-due.
Better: cap the effective max_tokens of the request to what the remaining budget affords before serving, so cost cannot exceed budget.
Consider treating a False return from add_cost in _accumulate_session_cost as a signal to close/settle the session rather than a warning to ignore.
Notes
Pre-existing architectural issue in the upto session model, not introduced by #1 (which improves the settlement-loss side). Filing separately as a high-priority follow-up. Closely related to the concurrency cap-bypass issue (filed alongside).
Activity