Skip to content

docs(specs): record the GPT-5.6 caching verification — it works, 10.6x on warm turns - #962

Merged
philmerrell merged 1 commit into
developfrom
docs/gpt56-live-verification
Sep 5, 2026
Merged

docs(specs): record the GPT-5.6 caching verification — it works, 10.6x on warm turns#962
philmerrell merged 1 commit into
developfrom
docs/gpt56-live-verification

Conversation

@philmerrell

Copy link
Copy Markdown
Contributor

The end-to-end proof the plan was waiting on

Four turns through the SPA on us.openai.gpt-5.6-sol, read back from GET /admin/costs/sessions/{id}/calls (session f76ab27a…):

turn cacheStatus input cacheRead cacheWrite output cost
1 first_write 2 0 3,679 5 $0.02038
2 hit 2 3,679 21 6 $0.00190
3 hit 2 3,700 22 6 $0.00192
4 hit 2 3,722 22 6 $0.00193

Warm turns cost 10.6× less than the cold turn. Every prediction the plan made holds, and each PR is confirmed by a specific column rather than by assertion.

What each column proves

PR-1 (#945) — usage normalization. inputTokens is 2 on every turn, not ~3,681. OpenAI reports input_tokens inclusive of both cache buckets; here they're disjoint and sum exactly to the prefix (2+0+3,679 = 3,681; 2+3,679+21 = 3,702), matching the SPA's own context-window readout to the token.

cacheWriteInputTokens is populated at all only because we recover cache_write_tokens — Strands drops it. Turn 1 alone would otherwise have double-billed 3,679 tokens at the input rate plus the 1.25× premium. That is precisely the correction #947 made to this spec, now demonstrated on real data rather than argued from a doc.

PR-2 (#949) — transport. Reached a live model: base URL, per-request bearer token, inference-profile id all correct.

PR-5 (#951) — model-derived TTL. Gaps of 30s / 78s / 18s classified hit, wastedUsd = $0.00 throughout, no avoidable re-writes.

#956 — explicit caching OFF. This is the implicit shape the probe predicted: cacheRead grows with the conversation (3,679 → 3,700 → 3,722) while cacheWrite is just the appended delta (~21). Under explicit mode cacheRead would be flat and uncached input would climb every turn. The 57%-worse finding is now confirmed in the agent loop, not just at the transport — so shipping it off was right.

#959 — IAM. bedrock:CallWithBearerToken, without which none of the above runs.

Two extra confirmations

Prefix fingerprints behaved exactly as the cache contract requirestoolConfigHash 8eafb0765ed810a2 and systemPromptHash 5a71749b3bee37ab identical across all four calls, historyHash changing each turn. Nothing nondeterministic leaked into the cacheable prefix.

The cost math reproduces to the cent. Deriving each row from the disjoint buckets and the catalog rates matches the reported cost exactly on all four calls, so CostCalculator is verified against live usage too.

⚠️ The dollars are provisional

They use the model-card rates seeded on the dev row (4.40 in / 26.40 out / 0.44 cache read / 5.50 cache write), and 26.40 is derived from the GovCloud 1:6 input:output ratio, not published. The ratios above are real and independent of that; the absolute dollars wait on PR-3, which is still blocked on AWS not publishing commercial rates.

Docs only — no code.

🤖 Generated with Claude Code

…x on warm turns

Four turns through the SPA on us.openai.gpt-5.6-sol, read back from
GET /admin/costs/sessions/{id}/calls. This is the end-to-end proof the whole
plan was waiting on, and every prediction holds.

  turn  cacheStatus   input  cacheRead  cacheWrite  output      cost
  1     first_write       2          0       3,679       5   $0.02038
  2     hit               2      3,679          21       6   $0.00190
  3     hit               2      3,700          22       6   $0.00192
  4     hit               2      3,722          22       6   $0.00193

Each PR is confirmed by a specific column:

PR-1 — inputTokens is 2, not ~3,681. The buckets are disjoint and sum exactly
to the prefix, matching the SPA's context readout to the token. cacheWrite is
populated at all only because we recover cache_write_tokens, which Strands
drops; turn 1 would otherwise double-bill 3,679 tokens at input rate PLUS the
1.25x premium — exactly the correction #947 made to this spec.

PR-2 — reached a live model; base URL, per-request bearer token and
inference-profile id all correct.

PR-5 — gaps of 30s/78s/18s classified `hit`, wastedUsd $0.00 throughout.

#956 — the implicit shape the probe predicted: cacheRead GROWS with the
conversation while cacheWrite is just the appended delta. Under explicit mode
cacheRead would be flat and uncached input would climb. The 57%-worse finding
is now confirmed in the agent loop, not only at the transport.

#959 — bedrock:CallWithBearerToken, without which none of it runs.

Fingerprints stable where they should be: toolConfigHash and systemPromptHash
identical across all four calls, historyHash changing each turn. The cost math
reproduces to the cent from the disjoint buckets and the catalog rates, so
CostCalculator is verified against live usage too.

Dollar amounts are provisional — they use the model-card rates seeded on the
dev row, and the output rate is derived from the GovCloud 1:6 ratio rather
than published. The ratios are real; absolute dollars wait on PR-3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@philmerrell
philmerrell merged commit 602079a into develop Sep 5, 2026
4 checks passed
@philmerrell
philmerrell deleted the docs/gpt56-live-verification branch September 5, 2026 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant