docs(specs): record the GPT-5.6 caching verification — it works, 10.6x on warm turns - #962
Merged
Merged
Conversation
…x on warm turns
Four turns through the SPA on us.openai.gpt-5.6-sol, read back from
GET /admin/costs/sessions/{id}/calls. This is the end-to-end proof the whole
plan was waiting on, and every prediction holds.
turn cacheStatus input cacheRead cacheWrite output cost
1 first_write 2 0 3,679 5 $0.02038
2 hit 2 3,679 21 6 $0.00190
3 hit 2 3,700 22 6 $0.00192
4 hit 2 3,722 22 6 $0.00193
Each PR is confirmed by a specific column:
PR-1 — inputTokens is 2, not ~3,681. The buckets are disjoint and sum exactly
to the prefix, matching the SPA's context readout to the token. cacheWrite is
populated at all only because we recover cache_write_tokens, which Strands
drops; turn 1 would otherwise double-bill 3,679 tokens at input rate PLUS the
1.25x premium — exactly the correction #947 made to this spec.
PR-2 — reached a live model; base URL, per-request bearer token and
inference-profile id all correct.
PR-5 — gaps of 30s/78s/18s classified `hit`, wastedUsd $0.00 throughout.
#956 — the implicit shape the probe predicted: cacheRead GROWS with the
conversation while cacheWrite is just the appended delta. Under explicit mode
cacheRead would be flat and uncached input would climb. The 57%-worse finding
is now confirmed in the agent loop, not only at the transport.
#959 — bedrock:CallWithBearerToken, without which none of it runs.
Fingerprints stable where they should be: toolConfigHash and systemPromptHash
identical across all four calls, historyHash changing each turn. The cost math
reproduces to the cent from the disjoint buckets and the catalog rates, so
CostCalculator is verified against live usage too.
Dollar amounts are provisional — they use the model-card rates seeded on the
dev row, and the output rate is derived from the GovCloud 1:6 ratio rather
than published. The ratios are real; absolute dollars wait on PR-3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The end-to-end proof the plan was waiting on
Four turns through the SPA on
us.openai.gpt-5.6-sol, read back fromGET /admin/costs/sessions/{id}/calls(sessionf76ab27a…):first_writehithithitWarm turns cost 10.6× less than the cold turn. Every prediction the plan made holds, and each PR is confirmed by a specific column rather than by assertion.
What each column proves
PR-1 (#945) — usage normalization.
inputTokensis 2 on every turn, not ~3,681. OpenAI reportsinput_tokensinclusive of both cache buckets; here they're disjoint and sum exactly to the prefix (2+0+3,679 = 3,681; 2+3,679+21 = 3,702), matching the SPA's own context-window readout to the token.cacheWriteInputTokensis populated at all only because we recovercache_write_tokens— Strands drops it. Turn 1 alone would otherwise have double-billed 3,679 tokens at the input rate plus the 1.25× premium. That is precisely the correction #947 made to this spec, now demonstrated on real data rather than argued from a doc.PR-2 (#949) — transport. Reached a live model: base URL, per-request bearer token, inference-profile id all correct.
PR-5 (#951) — model-derived TTL. Gaps of 30s / 78s / 18s classified
hit,wastedUsd = $0.00throughout, no avoidable re-writes.#956 — explicit caching OFF. This is the implicit shape the probe predicted:
cacheReadgrows with the conversation (3,679 → 3,700 → 3,722) whilecacheWriteis just the appended delta (~21). Under explicit modecacheReadwould be flat and uncached input would climb every turn. The 57%-worse finding is now confirmed in the agent loop, not just at the transport — so shipping it off was right.#959 — IAM.
bedrock:CallWithBearerToken, without which none of the above runs.Two extra confirmations
Prefix fingerprints behaved exactly as the cache contract requires —
toolConfigHash8eafb0765ed810a2andsystemPromptHash5a71749b3bee37abidentical across all four calls,historyHashchanging each turn. Nothing nondeterministic leaked into the cacheable prefix.The cost math reproduces to the cent. Deriving each row from the disjoint buckets and the catalog rates matches the reported
costexactly on all four calls, soCostCalculatoris verified against live usage too.They use the model-card rates seeded on the dev row (4.40 in / 26.40 out / 0.44 cache read / 5.50 cache write), and 26.40 is derived from the GovCloud 1:6 input:output ratio, not published. The ratios above are real and independent of that; the absolute dollars wait on PR-3, which is still blocked on AWS not publishing commercial rates.
Docs only — no code.
🤖 Generated with Claude Code