Lookback: count positions the way Bedrock does now (0.4.1)

README.md

@@ -68,7 +68,7 @@ Exit code 2 means it couldn't run at all (bad file, no model access, no AWS cred
 - Dates, times, UUIDs or user-specific text before the cache checkpoint.
 - A cached prefix shorter than the model's minimum (512 to 4,096 tokens depending on the model).
 - Tools listed in a different order, or a tool definition that changed. Tools come first, so this throws away the whole cache.
-- More than 21 new content blocks between the old checkpoint and the new one, which is common after many parallel tool calls. Bedrock only looks back that far: measured live, 21 added blocks still hit and 22 always missed (`scripts/live_lookback_boundary.py`).
+- More than 21 lookback positions between the old checkpoint and the new one, common in long chains of single tool calls or with many separate text blocks. Each text block is one position, a group of tool calls or tool results side by side is one, and thinking blocks don't count. Measured live on October 30 2026: 21 positions still hit and 22 always missed (`scripts/live_lookback_positions.py`). On October 7 2026 Bedrock still counted every tool call and result, so many parallel tool calls lost the cache; by October 28 they no longer did.
 - Model IDs or application inference profile ARNs that your library may not recognize, so it quietly stops sending cache markers. Legacy models (Sonnet 4, Opus 4 and 4.1, 3.5 Haiku) are checked against Anthropic's documented minimums, since AWS's caching table doesn't list them.
 - A switch between the Converse and InvokeModel APIs, which builds a different prompt.
 - More than 4 checkpoints, an unknown cache lifetime (TTL), or a 1-hour checkpoint after a 5-minute one.
@@ -166,7 +166,7 @@ Only model calls (generations) are counted. Agent and chain spans can carry the
 
 ## How it's tested
 
-The checks follow AWS's documented prompt-caching rules. The core behaviour was verified against live Amazon Bedrock: Converse and InvokeModel, streaming, both cache lifetimes (5 minutes and 1 hour), model minimums, prompt changes and the exact lookback limit (21 added blocks hit, 22 miss). Every check added in 0.3 was measured live first, and `scripts/live_v03_checks.py` re-runs those cases against Bedrock and compares CacheCanary's verdict with what Bedrock did (15 of 15 agreed in October 2026). Every check also has unit tests. The log reader is tested against real Bedrock invocation logs (redacted copies are in [tests/fixtures](https://github.com/Haarris/cachecanary/tree/main/tests/fixtures)). The Langfuse reader is tested against records that Strands, LiteLLM and LangChain wrote into a real Langfuse 4.64 server, and against a production Langfuse Cloud project, where its numbers matched a separate recount for every model.
+The checks follow AWS's documented prompt-caching rules. The core behaviour was verified against live Amazon Bedrock: Converse and InvokeModel, streaming, both cache lifetimes (5 minutes and 1 hour), model minimums, prompt changes and the exact lookback limit (21 positions hit, 22 miss, re-measured October 30 2026 on Sonnet 4.6 and 5.5 after Bedrock changed how it counts tool calls). Every check added in 0.3 was measured live first, and `scripts/live_v03_checks.py` re-runs those cases against Bedrock and compares CacheCanary's verdict with what Bedrock did (15 of 15 agreed in October 2026). Every check also has unit tests. The log reader is tested against real Bedrock invocation logs (redacted copies are in [tests/fixtures](https://github.com/Haarris/cachecanary/tree/main/tests/fixtures)). The Langfuse reader is tested against records that Strands, LiteLLM and LangChain wrote into a real Langfuse 4.64 server, and against a production Langfuse Cloud project, where its numbers matched a separate recount for every model.
 
 To run the live checks yourself, use an AWS Region without production traffic: `python scripts/live_test.py --region <region>`.
 

scripts/capture_langchain_requests.py

@@ -52,7 +52,8 @@ def run(script, model, prompts, middleware, system=SYSTEM):
     def save(name, body):
         (out / name).write_text(json.dumps(body, indent=2) + "\n")
 
-    # Finding 2: ChatBedrock (InvokeModel) with the caching middleware, 11 parallel tool calls.
+    # Finding 2: ChatBedrock (InvokeModel) with the caching middleware, 11 parallel tool calls. On Oct 7 2026 this lost the cache; Bedrock now
+    # counts a group of tool calls as one lookback position, so cachecanary diff reports it as kept.
     chat_bedrock = ChatBedrock(model=MODEL, region_name="us-west-2", endpoint_url=fake.url, model_kwargs={"max_tokens": 50})
     bodies = run(["tools:11", "text"], chat_bedrock, ["Check my orders."], [BedrockPromptCachingMiddleware()])
     save("invoke-turn1.json", bodies[0])

scripts/capture_litellm_requests.py

@@ -83,7 +83,8 @@ def capture(path: Path, **kwargs) -> None:
     out.mkdir(parents=True, exist_ok=True)
     last_message = [{"location": "message", "role": "system"}, {"location": "message", "index": -1}]
 
-    # Finding 2: an agent step with 12 parallel tool calls.
+    # Finding 2: an agent step with 12 parallel tool calls. On Oct 7 2026 this lost the cache; Bedrock now
+    # counts a group of tool calls as one lookback position, so cachecanary diff reports it as kept.
     turn1 = [{"role": "system", "content": SYSTEM},
              {"role": "user", "content": "Check orders 1 to 12 and tell me which shipped."}]
     capture(out / "agent-turn1.json", model=SONNET_46, tools=TOOLS, messages=turn1,

scripts/capture_strands_requests.py

@@ -49,7 +49,8 @@ def save(name, body):
         (out / name).write_text(json.dumps(body, indent=2) + "\n")
 
     auto = CacheConfig(strategy="auto")
-    # Finding 2: one agent step with 11 parallel tool calls.
+    # Finding 2: one agent step with 11 parallel tool calls. On Oct 7 2026 this lost the cache; Bedrock now
+    # counts a group of tool calls as one lookback position, so cachecanary diff reports it as kept.
     bodies = run(["tools:11", "text"], ["Check my orders."], auto)
     save("agent-turn1.json", bodies[0])
     save("agent-turn2.json", bodies[1])

scripts/live_langchain_checks.py

@@ -19,7 +19,6 @@
 
 import argparse
 import json
-import os
 import sys
 import time
 import uuid
@@ -128,16 +127,17 @@ def twice(case, middleware, **model_kwargs):
         checks += [looked_up[1]["read"] > 1000, named[1]["read"] > 1000]
 
     # Agent loops: LangChain's real loop on the stand-in, then the exact requests replayed to Bedrock.
-    os.environ.setdefault("AWS_ACCESS_KEY_ID", "unused-by-the-stand-in")
     fake = Fake()
 
     def capture(script, *, invoke_route=False, middleware=None, manual_point=False):
         system, history = tag_prompts()
         fake.reset(script)
         if invoke_route:
-            model = ChatBedrock(model=MODEL, region_name=args.region, endpoint_url=fake.url, model_kwargs={"max_tokens": 50})
+            model = ChatBedrock(model=MODEL, region_name=args.region, endpoint_url=fake.url, model_kwargs={"max_tokens": 50},
+                                aws_access_key_id="unused-by-the-stand-in", aws_secret_access_key="unused-by-the-stand-in")
         else:
-            model = ChatBedrockConverse(model=MODEL, region_name=args.region, endpoint_url=fake.url, max_tokens=50)
+            model = ChatBedrockConverse(model=MODEL, region_name=args.region, endpoint_url=fake.url, max_tokens=50,
+                                        aws_access_key_id="unused-by-the-stand-in", aws_secret_access_key="unused-by-the-stand-in")
         if manual_point:
             system = SystemMessage(content=[{"type": "text", "text": system}, ChatBedrockConverse.create_cache_point()])
         agent = create_agent(model=model, tools=[lookup_order], system_prompt=system,
@@ -172,14 +172,15 @@ def kept_with_retry(case, script, **kwargs):
                 return kept(rows)
         return False
 
+    # Since Oct 2026 (by Oct 28) Bedrock counts a group of tool calls (and their results) as one lookback
+    # position, so every route keeps the whole conversation after a wide step.
     checks.append(kept_with_retry("ChatBedrockConverse, 21 parallel calls", ["tools:21", "text"]))
-    checks.append(not kept(replay("ChatBedrockConverse, 22 parallel calls", capture(["tools:22", "text"]))))
+    checks.append(kept_with_retry("ChatBedrockConverse, 40 parallel calls", ["tools:40", "text"]))
     checks.append(kept_with_retry("ChatBedrock, 10 parallel calls", ["tools:10", "text"], invoke_route=True))
-    invoke_11 = replay("ChatBedrock, 11 parallel calls", capture(["tools:11", "text"], invoke_route=True))
-    checks.append(invoke_11[1]["read"] == 0)  # nothing read: no separate system prompt marker on this route
+    checks.append(kept_with_retry("ChatBedrock, 40 parallel calls", ["tools:40", "text"], invoke_route=True))
     checks.append(kept_with_retry("middleware only, 11 parallel calls", ["tools:11", "text"]))
-    checks.append(not kept(replay("manual system cache point + middleware, 11 parallel calls",
-                                  capture(["tools:11", "text"], manual_point=True))))
+    checks.append(kept_with_retry("manual system cache point + middleware, 40 parallel calls", ["tools:40", "text"],
+                                  manual_point=True))
 
     # Context editing: stock clearing vs clearing in batches vs none, 13 calls with long tool results.
     tool_output["repeat"] = 40

scripts/live_litellm_checks.py

@@ -25,28 +25,10 @@
 MODEL = "bedrock/converse/us.anthropic.claude-sonnet-4-6"
 TOOLS = [{"type": "function", "function": {"name": "lookup_order", "description": "Look up an order.",
           "parameters": {"type": "object", "properties": {"id": {"type": "string"}}}}}]
+# The setup on the page (tests/test_site.py checks they match).
 SYSTEM_AND_LAST = [{"location": "message", "role": "system"}, {"location": "message", "index": -1}]
 
 
-# Same helper as on the page (tests/test_site.py checks they match).
-def cache_points(messages):
-    """Bedrock cache points for this call (Bedrock allows 4)."""
-    # The system prompt and the end of the conversation.
-    points = [{"location": "message", "role": "system"},
-              {"location": "message", "index": -1}]
-    users = [i for i, m in enumerate(messages) if m["role"] == "user"]
-    rounds = [i for i, m in enumerate(messages)
-              if m["role"] == "assistant" and m.get("tool_calls")]
-    # The newest user message.
-    if users:
-        points.append({"location": "message", "index": users[-1]})
-    # The first tool result of the newest round of tool calls.
-    if rounds and (not users or rounds[-1] > users[-1]) \
-            and rounds[-1] + 1 < len(messages):
-        points.append({"location": "message", "index": rounds[-1] + 1})
-    return points
-
-
 def lifetime_check() -> int:
     """One call asking for a 1-hour cache; prints the lifetime Bedrock reports it used."""
     model, extra = sys.argv[2], json.loads(sys.argv[3])
@@ -121,27 +103,27 @@ def tool_round(r, n):
         checks.append((label, call(f"{label}, turn 10", chat + [{"role": "user", "content": "Question 9?"}], points)))
     checks_ok = [checks[0][1]["uncached"] > 5000, checks[1][1]["uncached"] < 20]
 
-    # Finding 2: 4 vs 12 parallel tool calls with points on the system prompt and the last message.
+    # Finding 2: 4 vs 12 parallel tool calls with points on the system prompt and the last message. Since late
+    # Oct 2026 Bedrock counts a group of tool calls (and their results) as one lookback position, so both keep it.
     start = fresh()
-    call("agent start", start, SYSTEM_AND_LAST, TOOLS)
+    first = call("agent start", start, SYSTEM_AND_LAST, TOOLS)
     twelve = call("12 parallel calls", start + tool_round(1, 12), SYSTEM_AND_LAST, TOOLS)
     four = call("4 parallel calls", start + tool_round(1, 4), SYSTEM_AND_LAST, TOOLS)
-    checks_ok += [twelve["read"] < four["read"] / 2, four["write"] < 1000]
+    checks_ok += [twelve["read"] >= first["prompt"] - 5, four["read"] >= first["prompt"] - 5, four["write"] < 1000]
 
-    # The helper through a 7-step session, then a 21-call round.
+    # The page's setup through an agent session with wide rounds: every step reads the whole previous request.
     steps = [fresh()]
     for r, n in ((1, 12), (2, 1), (3, 20)):
         steps.append(steps[-1] + tool_round(r, n))
     steps.append(steps[-1] + [{"role": "assistant", "content": "All checked."}, {"role": "user", "content": "Refund order 3-4."}])
     steps.append(steps[-1] + tool_round(4, 3))
+    steps.append(steps[-1] + tool_round(5, 40))
     previous = None
     for i, messages in enumerate(steps):
-        row = call(f"helper step {i}", messages, cache_points(messages), TOOLS)
+        row = call(f"session step {i}", messages, SYSTEM_AND_LAST, TOOLS)
         if previous is not None:
             checks_ok.append(row["read"] >= previous["prompt"] - 5)  # read almost the whole previous request
         previous = row
-    wide = call("helper, 21 parallel calls", steps[-1] + tool_round(5, 21), cache_points(steps[-1] + tool_round(5, 21)), TOOLS)
-    checks_ok.append(wide["read"] < previous["prompt"] - 5)
 
     # Finding 4: the 1-hour lifetime. The price list is read at import, so each case runs in its own process.
     def lifetime(case, model, extra=None, built_in_prices=False):

scripts/live_lookback_boundary.py

@@ -1,4 +1,8 @@
-"""Measure where Bedrock's ~20-block cache lookback actually ends.
+"""Measure where Bedrock's ~20-block cache lookback actually ends (the first measurement, early Oct 2026).
+
+Historical: when this ran, Bedrock counted every block, so 22+ blocks of parallel tool calls missed.
+By Oct 28 2026 a run of tool calls (or tool results) counts as one position and these shapes all
+hit. scripts/live_lookback_positions.py is the current check.
 
 Request A: tools + a short uncached system prompt + one long user turn ending in a cache point.
 Request B: the same history with that cache point removed, then an assistant turn with N parallel