fix(models): ship explicit prompt caching OFF — it measured 57% more expensive

[philmerrell]

Follow-up to #954 — that PR shipped a measured pessimization

#954 merged at 06:26 today, before I got a live model in front of it. I did, and the premise it was built on does not hold. This flips it off.

No production impact today: there is no GPT-5.6 catalog row (PR-3 is blocked on missing Price List rates), so nothing routes through the transport yet. Which is exactly why it's worth fixing now, before something does.

What the measurement showed

The plan assumed a change in conversation history forces a full cache re-write, which a breakpoint after the static prefix would downgrade to a read. Measured, GPT-5.6's default implicit caching does not re-write history when it grows — it appends the delta (1,521 cacheWrite/turn). So the breakpoint saves no re-write. It only stops history being cached at all, and that cost grows linearly with conversation length.

us.openai.gpt-5.6-sol, dev-ai us-west-2, 8k static prefix, 5 turns, ~1.5k tokens of history growth per turn, priced at the Price List ratios this investigation confirmed (input 1×, cache read 0.1×, cache write 1.25×):

	uncached input	cacheRead	cacheWrite	input-equivalents
explicit	22,790	23,228	5,807	32,372
implicit	10	38,410	13,405	20,607

~57% more. Per turn, explicit's uncached input grew 1,516 → 7,600 while its cacheRead stayed flat at 5,807; implicit held uncached input at 2/turn and let cacheRead grow with the conversation.

The change

BEDROCK_RESPONSES_EXPLICIT_CACHE_ENABLED now defaults OFF and only the literal "true" opts in, so the production request is byte-identical to stock Strands — pinned by a test that asserts no breakpoint, no prompt_cache_key, no extra_body, and instructions intact.

The code is kept rather than deleted because the placement failed, not the mechanism: the API allows 4 breakpoints, and a scheme that also marks the end of history might beat implicit. Re-enabling is gated on re-running the probe and beating the implicit arm.

prompt_cache_key stays tied to the explicit path, so it's off too. Applying it under implicit is plausibly free — but that's a fleet-level routing effect a single-session probe can't measure, and shipping it unmeasured would repeat the mistake this PR fixes. Recorded in the spec as its own measurement.

scripts/probe_gpt56_cache_rates.py

The reproducible artifact behind all of this, and the gate on ever re-enabling explicit mode. It drives the transport directly — no catalog row, no RBAC, no agent loop — so it runs while PR-3 is blocked, and it produces the token denominator for deriving rates from Cost Explorer.

AWS_PROFILE=dev-ai uv run python scripts/probe_gpt56_cache_rates.py --mode both --grow-history

Bonus: this was the first live verification of the rest of the epic

• PR-1 (fix(costs): normalize OpenAI-family token usage to disjoint buckets #945) — buckets disjoint on every turn of every run. Turn 1 reported inputTokens=2 with cacheWriteInputTokens=1,446, where raw OpenAI reports input_tokens=1448 inclusive. That is direct proof the cache-write subtraction (the correction in docs(specs): subtract the cache-write bucket too in GPT-5.6 PR-1 #947) was necessary — without it that turn double-bills 1,446 tokens at the input rate plus the 1.25× premium — and that recovering cache_write_tokens works at all, since Strands drops it and the bucket would otherwise read 0.
• PR-2 (feat(models): add the bedrock-runtime OpenAI Responses transport #949) — the transport reached a live model: base URL, per-request bearer token and inference-profile id all correct.
• PR-5 (fix(observability): derive the prompt-cache TTL from the serving model #951) — warm turns read the prefix, so the 30-minute window is the one that matters.

Rates

The runs consumed ~190k tokens on usage types nothing else in dev uses, so Cost Explorer attribution is unambiguous. Once it settles (~24h):

AWS_PROFILE=dev-ai uv run python scripts/probe_gpt56_cache_rates.py --rates-only --since 2026-09-05

⚠️ Confirm the usage unit before trusting the derived $/MTok — Bedrock token usage types are reported in 1K-token units and the script's conversion assumes that. Cross-check one row against the token totals the run printed.

Testing

cd backend && uv run python -m pytest tests/ -q → 7432 passed, 3 skipped, 0 failed

🤖 Generated with Claude Code