Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/api/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8484,6 +8484,15 @@ Dollar accounting is known unless explicitly false. A false value must not be tr

> **usd**: `number`

##### usdEstimated?

> `optional` **usdEstimated?**: `number`

The part of `usd` priced from a model catalog because no provider receipt covered the work.
`usd - usdEstimated` is what a provider is known to have billed. Present only with
`usdKnown: false`; absence means nothing here was catalog-priced, not that `usd` is
measured.

##### ms

> **ms**: `number`
Expand Down
15 changes: 13 additions & 2 deletions docs/api/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -19938,7 +19938,7 @@ Resolve an external harness for one exact Runtime-owned manager identity.

### UsageEvent

> **UsageEvent** = \{ `kind`: `"tokens"`; `tokensKnown?`: `false`; `input`: `number`; `output`: `number`; `freshInput?`: `number`; `cacheRead?`: `number`; `cacheWrite?`: `number`; `cacheBreakdownKnown?`: `false`; \} \| \{ `kind`: `"cost"`; `usdKnown?`: `false`; `usd`: `number`; \} \| \{ `kind`: `"iteration"`; \}
> **UsageEvent** = \{ `kind`: `"tokens"`; `tokensKnown?`: `false`; `input`: `number`; `output`: `number`; `freshInput?`: `number`; `cacheRead?`: `number`; `cacheWrite?`: `number`; `cacheBreakdownKnown?`: `false`; \} \| \{ `kind`: `"cost"`; `usdKnown?`: `false`; `usd`: `number`; `usdEstimated?`: `number`; \} \| \{ `kind`: `"iteration"`; \}

Normalized usage event — the single channel every executor reports through, so the
conserved pool meters all runtimes identically. `tokens` carries `LoopTokenUsage`'s
Expand Down Expand Up @@ -20004,7 +20004,7 @@ them is an upper bound. A counter the provider did not report is absent, never z

##### Type Literal

\{ `kind`: `"cost"`; `usdKnown?`: `false`; `usd`: `number`; \}
\{ `kind`: `"cost"`; `usdKnown?`: `false`; `usd`: `number`; `usdEstimated?`: `number`; \}

###### kind

Expand All @@ -20020,6 +20020,17 @@ Known dollar subtotal. When false, `usd` must not be treated as total cost.

> **usd**: `number`

###### usdEstimated?

> `optional` **usdEstimated?**: `number`

The part of `usd` this runtime priced from a model catalog because no provider receipt
covered the work. Requires `usdKnown: false` — a catalog price approximates what a
provider would bill and never measures what it did.

Absence means this runtime priced nothing here, NOT that `usd` is a receipt. `usdKnown`
is what says whether a dollar figure is measured.

***

##### Type Literal
Expand Down
9 changes: 9 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,15 @@ It accepts every wire spelling the sandbox, cli-bridge, Anthropic, OpenAI, and D
A counter the provider did not report stays absent.
A zero would assert the provider measured no cache, which is a different fact from a provider that reported nothing.

The dollar channel keeps a receipt and a price apart.
A provider receipt is a dollar figure the provider billed; cli-bridge sends one only with `cost_known: true` and `provider-receipt` or `billing-receipt` provenance, and the claude harness reports one for the whole `claude -p` invocation.
A cli-bridge turn that carries NO receipt is priced from the model catalog (`estimateCost` in agent-eval) against that turn's own token counts, because a zero there reads as a measured free turn and made a fleet-wide dollar total report `$0` on runs that certainly spent money.
The priced part is carried in `Spend.usdEstimated`, so `usd - usdEstimated` is what a provider is known to have billed, and it is admitted only with `usdKnown: false` — a catalog price approximates what a provider would bill and never measures what it did.
An unpriced model contributes no dollars and leaves the turn unknown rather than free.
The catalog holds one input rate and one output rate per model and no cache-read rate, so a prompt prefix the provider served from cache is priced at the full input rate.
That overstates a cache-heavy turn, which is the correct direction: a discount the catalog cannot support would be invented, and an invented discount understates spend.
A dollar cap is unaffected — `observe` and `reconcile` still refuse unknown dollars under a `maxUsd` root, and an estimate rides `usdKnown: false`.

Two facts make this the whole game:
- `spawn` **reserves** from one root total and refunds the unspent remainder on settle.
A nested driver partitions only its reserved allocation, then reconciles the whole subtree once, so `Σk(treatment) ≡ Σk(blind)` by construction — no arm can buy more compute (`supervise/budget.ts`).
Expand Down
4 changes: 3 additions & 1 deletion src/durable/spawn-journal.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ import type {
TreeView,
} from '../runtime/supervise/types'
import type { PendingWait } from '../runtime/supervise/wait'
import { addTokenUsage, cloneTokenUsage, zeroTokenUsage } from '../runtime/util'
import { addTokenUsage, cloneTokenUsage, usdEstimatedOf, zeroTokenUsage } from '../runtime/util'
import { contentAddress } from './content-address'
import { parseCommittedJsonLines, prepareJsonlAppend, writeAllBytes } from './jsonl-file'

Expand Down Expand Up @@ -1136,6 +1136,7 @@ function cloneJournalSpend(spend: Spend): Spend {
...(spend.tokensKnown === false ? { tokensKnown: false } : {}),
usd: spend.usd,
...(spend.usdKnown === false ? { usdKnown: false } : {}),
...(spend.usdEstimated !== undefined ? { usdEstimated: spend.usdEstimated } : {}),
ms: spend.ms,
}
}
Expand Down Expand Up @@ -1176,6 +1177,7 @@ function addJournalSpend(a: Spend, b: Spend): Spend {
...(a.tokensKnown === false || b.tokensKnown === false ? { tokensKnown: false } : {}),
usd: a.usd + b.usd,
...(a.usdKnown === false || b.usdKnown === false ? { usdKnown: false } : {}),
...usdEstimatedOf(a, b),
ms: a.ms + b.ms,
}
}
Expand Down
12 changes: 11 additions & 1 deletion src/runtime/personify/trajectory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,13 @@ import type {
SpawnJournal,
Spend,
} from '../supervise/types'
import { addTokenUsage, chargedTokens, cloneTokenUsage, zeroTokenUsage } from '../util'
import {
addTokenUsage,
chargedTokens,
cloneTokenUsage,
usdEstimatedOf,
zeroTokenUsage,
} from '../util'
import type {
EqualKArm,
EqualKOnCostOptions,
Expand Down Expand Up @@ -306,6 +312,7 @@ function addNodeSpend(a: Spend, b: Spend): Spend {
...(a.tokensKnown === false || b.tokensKnown === false ? { tokensKnown: false } : {}),
usd: a.usd + b.usd,
...(a.usdKnown === false || b.usdKnown === false ? { usdKnown: false } : {}),
...usdEstimatedOf(a, b),
ms: a.ms + b.ms,
}
}
Expand All @@ -317,6 +324,7 @@ function cloneSpend(spend: Spend): Spend {
...(spend.tokensKnown === false ? { tokensKnown: false } : {}),
usd: spend.usd,
...(spend.usdKnown === false ? { usdKnown: false } : {}),
...(spend.usdEstimated !== undefined ? { usdEstimated: spend.usdEstimated } : {}),
ms: spend.ms,
}
}
Expand All @@ -328,6 +336,8 @@ function addSpend(acc: Spend, delta: Spend): void {
if (delta.tokensKnown === false) acc.tokensKnown = false
acc.usd += delta.usd
if (delta.usdKnown === false) acc.usdKnown = false
if (delta.usdEstimated !== undefined)
acc.usdEstimated = (acc.usdEstimated ?? 0) + delta.usdEstimated
acc.ms += delta.ms
}

Expand Down
96 changes: 90 additions & 6 deletions src/runtime/supervise/bridge-executor.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'
import type { AddressInfo } from 'node:net'
import { estimateCost } from '@tangle-network/agent-eval'
import {
type AgentProfile,
canonicalAgentProfileDigest,
Expand Down Expand Up @@ -862,7 +863,7 @@ describe('bridgeExecutor upstream-error propagation', () => {
expect(requestBody?.messages).toEqual([{ role: 'user', content: 'design the experiment' }])
})

it('marks dollar cost unknown when the bridge reports no price', async () => {
it('prices a turn the bridge reported no price for, and keeps the dollars unknown', async () => {
const stub = await startBridgeStub(
`data: ${JSON.stringify({ usage: { prompt_tokens: 3, completion_tokens: 2 } })}\n\ndata: [DONE]\n\n`,
)
Expand All @@ -871,15 +872,57 @@ describe('bridgeExecutor upstream-error propagation', () => {
const events = await drain(
executor.execute('do the task', new AbortController().signal) as AsyncIterable<UsageEvent>,
)
expect(events).toContainEqual({ kind: 'cost', usd: 0, usdKnown: false })
expect(spendFromUsageEvents(events).usdKnown).toBe(false)
// glm rates over this turn's own 3 in / 2 out. The dollars reach the channel instead of a
// zero, and they carry the marker that says the catalog priced them.
const priced = estimateCost(3, 2, 'pi/tangle-router/glm-5.2')
expect(priced).toBeGreaterThan(0)
expect(events).toContainEqual({
kind: 'cost',
usd: priced,
usdKnown: false,
usdEstimated: priced,
})
const spend = spendFromUsageEvents(events)
expect(spend.usdKnown).toBe(false)
expect(spend.usd).toBe(priced)
expect(spend.usdEstimated).toBe(priced)
expect(executor.resultArtifact().spent).toMatchObject({
tokens: { input: 3, output: 2 },
usd: 0,
usd: priced,
usdEstimated: priced,
usdKnown: false,
})
})

it('reports no dollars for an unpriced model rather than inventing a rate', async () => {
const stub = await startBridgeStub(
`data: ${JSON.stringify({ usage: { prompt_tokens: 3, completion_tokens: 2 } })}\n\ndata: [DONE]\n\n`,
)
server = stub.server
const executor = bridgeExecutor(
{
profile: {
name: 'bridge-test-worker',
harness: 'pi',
model: { provider: 'in-house', default: 'no-such-model-family' },
},
harness: null,
},
{
signal: new AbortController().signal,
seams: { bridge: { bridgeUrl: stub.url, bridgeBearer: 'test-bearer' } },
},
)
const events = await drain(
executor.execute('do the task', new AbortController().signal) as AsyncIterable<UsageEvent>,
)
expect(events).toContainEqual({ kind: 'cost', usd: 0, usdKnown: false })
const spend = spendFromUsageEvents(events)
expect(spend.usd).toBe(0)
expect(spend.usdKnown).toBe(false)
expect(spend.usdEstimated).toBeUndefined()
})

it('lets a trusted terminal total supersede an earlier incomplete cost chunk', async () => {
const body = [
`data: ${JSON.stringify({
Expand Down Expand Up @@ -913,6 +956,39 @@ describe('bridgeExecutor upstream-error propagation', () => {
expect(executor.resultArtifact().spent.usdKnown).not.toBe(false)
})

it('takes a claude invocation receipt as measured dollars, with no estimated part', async () => {
// The EXACT usage object cli-bridge emits for a claude turn carrying `total_cost_usd`
// (drewstone/cli-bridge#159), captured off `deltaToOpenAIChunk`. Tokens and the receipt
// arrive in ONE frame, which is the shape that must not be re-priced.
const body = `data: ${JSON.stringify({
choices: [{ delta: {}, finish_reason: 'stop' }],
usage: {
prompt_tokens: 12_000,
completion_tokens: 900,
total_tokens: 12_900,
cost: 0.0731,
cost_known: true,
cost_provenance: 'provider-receipt',
cost_scope: 'total',
},
})}\n\ndata: [DONE]\n\n`
const stub = await startBridgeStub(body)
server = stub.server
const executor = makeExecutor(stub.url)

const events = await drain(
executor.execute('do the task', new AbortController().signal) as AsyncIterable<UsageEvent>,
)

expect(events).toContainEqual({ kind: 'cost', usd: 0.0731 })
const spent = executor.resultArtifact().spent
expect(spent).toMatchObject({ tokens: { input: 12_000, output: 900 }, usd: 0.0731 })
// A receipt is a measurement, and a turn holding one is never catalog-priced on top.
expect(spent.usdKnown).not.toBe(false)
expect(spent.usdEstimated).toBeUndefined()
expect(spendFromUsageEvents(events).usdEstimated).toBeUndefined()
})

it('preserves absent prompt-cache fields instead of inventing zeroes', async () => {
const body = `data: ${JSON.stringify({
usage: {
Expand Down Expand Up @@ -1030,10 +1106,14 @@ describe('bridgeExecutor upstream-error propagation', () => {
)

expect(requests).toBe(2)
// Turn 1 billed a real $0.01. Turn 2 sent no receipt, so only turn 2's own 4 in / 1 out is
// priced — the receipt and the estimate stay separable in the settled spend.
const turn2 = estimateCost(4, 1, 'pi/tangle-router/glm-5.2')
expect(executor.resultArtifact().spent).toMatchObject({
iterations: 2,
tokens: { input: 7, output: 3 },
usd: 0.01,
usd: 0.01 + turn2,
usdEstimated: turn2,
usdKnown: false,
})
})
Expand Down Expand Up @@ -1280,12 +1360,16 @@ describe('bridgeExecutor upstream-error propagation', () => {
content: expect.stringContaining('stop and use the corrected method'),
},
])
// The interrupted turn presented 5 in / 2 out and billed nothing, so it is priced. The
// resumed turn carried a real $0.01 receipt and is not priced on top of it.
const interruptedTurn = estimateCost(5, 2, 'pi/tangle-router/glm-5.2')
expect(executor.resultArtifact()).toMatchObject({
out: { content: 'corrected answer' },
spent: {
iterations: 2,
tokens: { input: 8, output: 3 },
usd: 0.01,
usd: 0.01 + interruptedTurn,
usdEstimated: interruptedTurn,
usdKnown: false,
},
})
Expand Down
32 changes: 28 additions & 4 deletions src/runtime/supervise/budget.ts
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,21 @@ function assertValidSpend(spend: Spend, label: string): void {
throw new Error(`${label}.${field} must be a non-negative finite number`)
}
}
if (spend.usdEstimated !== undefined) {
if (!Number.isFinite(spend.usdEstimated) || spend.usdEstimated < 0) {
throw new Error(`${label}.usdEstimated must be a non-negative finite number`)
}
// The catalog-priced part is a part OF `usd`, not an addition to it. A larger value would let
// `usd - usdEstimated` report negative provider-billed dollars.
if (spend.usdEstimated > spend.usd) {
throw new Error(`${label}.usdEstimated must not exceed ${label}.usd`)
}
// A catalog price is never a measurement. Admitting one under `usdKnown: true` would let an
// estimate be read as billed spend.
if (spend.usdKnown !== false) {
throw new Error(`${label}.usdEstimated requires ${label}.usdKnown false`)
}
}
}

export interface BudgetPool {
Expand Down Expand Up @@ -240,6 +255,7 @@ export function spendFromUsageEvents(events: UsageEvent[]): Spend {
const tokens = zeroTokenUsage()
let tokensKnown = true
let usd = 0
let usdEstimated = 0
let usdKnown = true
let iterations = 0
for (const ev of events) {
Expand All @@ -248,6 +264,7 @@ export function spendFromUsageEvents(events: UsageEvent[]): Spend {
if (ev.tokensKnown === false) tokensKnown = false
} else if (ev.kind === 'cost') {
usd += ev.usd
usdEstimated += ev.usdEstimated ?? 0
if (ev.usdKnown === false) usdKnown = false
} else {
iterations += 1
Expand All @@ -259,6 +276,7 @@ export function spendFromUsageEvents(events: UsageEvent[]): Spend {
...(tokensKnown ? {} : { tokensKnown: false }),
usd,
...(usdKnown ? {} : { usdKnown: false }),
...(usdEstimated > 0 ? { usdEstimated } : {}),
ms: 0,
}
}
Expand All @@ -268,6 +286,7 @@ async function foldUsage(events: AsyncIterable<UsageEvent> | UsageEvent[]): Prom
const tokens = zeroTokenUsage()
let tokensKnown = true
let usd = 0
let usdEstimated = 0
let usdKnown = true
let iterations = 0
for await (const ev of events) {
Expand All @@ -276,6 +295,7 @@ async function foldUsage(events: AsyncIterable<UsageEvent> | UsageEvent[]): Prom
if (ev.tokensKnown === false) tokensKnown = false
} else if (ev.kind === 'cost') {
usd += ev.usd
usdEstimated += ev.usdEstimated ?? 0
if (ev.usdKnown === false) usdKnown = false
} else {
iterations += 1
Expand All @@ -287,6 +307,7 @@ async function foldUsage(events: AsyncIterable<UsageEvent> | UsageEvent[]): Prom
...(tokensKnown ? {} : { tokensKnown: false }),
usd,
...(usdKnown ? {} : { usdKnown: false }),
...(usdEstimated > 0 ? { usdEstimated } : {}),
ms: 0,
}
}
Expand Down Expand Up @@ -422,16 +443,19 @@ export function createBudgetPool(
violation = `ticket ${ticket.id} spent ${spentTokens} tokens > reserved ${rTokens}`
} else if (spent.iterations > rIterations) {
violation = `ticket ${ticket.id} spent ${spent.iterations} iterations > reserved ${rIterations}`
} else if (unknownUnderCap) {
// Decided BEFORE the dollar comparison below. Dollars that are not measured may not be
// compared against a reservation as if they were billed: a catalog-priced turn can exceed
// the ceiling and would then be reported as an overspend the child never made. The known
// channels still settle, then the dollar channel is permanently tainted and admission
// closes.
violation = `ticket ${ticket.id} reported unknown dollar cost under a dollar-capped budget`
} else if (usdCapped && usdBudgeted && spent.usd > rUsd) {
// USD is conserved ONLY when the root declared a ceiling AND the child declared one to
// be measured against. `maxUsd` is optional on both: when either is unset, usd is an
// OBSERVED quantity (committed for accounting), never a budgeted constraint — an unset
// ceiling must not behave as a hard $0 limit that fail-closes a real priced spend.
violation = `ticket ${ticket.id} spent $${spent.usd} > reserved $${rUsd}`
} else if (unknownUnderCap) {
// The known channels still settle, then the dollar channel is permanently tainted and
// admission closes.
violation = `ticket ${ticket.id} reported unknown dollar cost under a dollar-capped budget`
}

// ── Settlement: unconditional, and the only place the ticket closes ───────────────
Expand Down
Loading