Phase 6: the accuracy loop, with sample B and the diagnosis split

[rohan-hotdata]

This PR builds the accuracy loop of phase 6. It adds a second fixed sample, a configuration for each arm, and a replay from the cache. It also adds a diagnosis that sorts each miss of the memory arm by cause. It records the baseline of both samples. It does not try to raise the score. Closes #16.

How

• bench/e2e.py: --sample a|b picks the sample. Sample B (bench/sample-b.json, seed 20261013) has 54 questions and no overlap with sample A.
• bench/e2e.py: Config holds the ranking of each arm, k, the budget, and the extractor variant. The options are --retrieval-ranking, --memory-ranking, --recall-k, --recall-budget, and --extractor. The result file records the configuration.
• bench/prompts.py: VARIANTS maps a name to a prompt and a model. baseline is the phase 5 prompt. Phase 7 adds its variants here.
• bench/models.py: --replay uses an Offline model that raises CacheMiss on a reply that the cache does not hold. A replay calls no model.
• bench/diagnose.py: reads the configuration from any result file and replays capture from the cache. It sorts each miss into not extracted, ranked below k, cut by as_of or the budget, or recalled and missed by the reader. A second GPT-4o call splits the recalled group into "all present" and "some present". The output is a table for each question type.
• bench/e2e.py: --workers N deals the questions to N processes, and each process has its own local database. This was not in the plan. It is recorded as a decision in plan.md.
• bench/models.py: the OpenAI client times out after 120 s. A diagnosis run stalled for 44 minutes on one call without it.
• bench/diagnose.py: each diagnosis gets its own database name. A killed run left facts in a shared database, BM25 ranks over the whole table, and the next replay missed the cache.
• bench/e2e.py: the run reads the git commit at its start, not at its end.

Why

Phase 7 to phase 11 change the extractor, the ranking, and capture. Each change needs a score on two samples, a cause for each miss, and a comparison that the runner itself does not disturb.

Results

Sample	Full context	Retrieval	Memory	Not extracted, of the memory misses
A	39	45	19	19 of 35
B	43	48	27	16 of 27

• The two samples differ by 14.8 points on the memory arm with the same code.
• In 2 of 25 recalled misses, the block held every needed fact.
• Phase spend: $21.80 ($21.42 for the sample B run, $0.37 for its diagnosis, $0.01 for the split on A).

Verification

• make verify: 204 passed, 99 skipped. There are new offline tests for the sample draw, Config, the diagnosis split, the Offline replay, and the worker split.
• Gate: the replay of sample A gave 19 of 54, with each answer and mark the same as phase 5, $0 spent, and 0 model calls. The diagnosis gave the same cause for each question (19 / 2 / 14).
• --replay --workers 4 --arms memory on sample A gave the same rows in 519 s, against 2,264 s with one process.
• I did not measure --workers on a paid run.

Notes

• bench/ is outside the package, so CHANGELOG.md has no entry. src/hotmemory does not change.
• The issue boxes on Phase 6: the accuracy loop #16 and the roadmap status after the merge are for the owner.