Phase 6: the accuracy loop, with sample B and the diagnosis split

README.md

@@ -138,18 +138,19 @@ as an episode in `HotdataStore` and searched with the question. The 419 scored q
 had these results with the vector ranking. For 98.3% of them, an evidence session was in
 the top 5. For 99.3% of them, every evidence session was in the top 10.
 
-The end-to-end score ran on a sample of 54 questions. The reader and the judge were
-`gpt-4o-2024-08-06`.
-
-| What the reader got | Correct |
-|---|---|
-| Every session of the question, about 114,000 tokens | 39 of 54 (72.2%) |
-| The 10 sessions that the search ranked first | 45 of 54 (83.3%) |
-| The block of `recall`, after `capture` with a `gpt-4o-mini` extractor | 19 of 54 (35.2%) |
-
-The memory contract, with this extractor, scores below the full history. In 19 of its 35
-misses, the extractor did not store the answer. With 54 questions, each score can be about
-10 points off. In the cloud, a search took 361 ms at p50 with the vector ranking, and
+The end-to-end score ran on two samples of 54 questions, sample A on 2026-10-10 and
+sample B on 2026-10-14. The reader and the judge were `gpt-4o-2024-08-06`.
+
+| What the reader got | Sample A | Sample B |
+|---|---|---|
+| Every session of the question, about 114,000 tokens | 39 of 54 (72.2%) | 43 of 54 (79.6%) |
+| The 10 sessions that the search ranked first | 45 of 54 (83.3%) | 48 of 54 (88.9%) |
+| The block of `recall`, after `capture` with a `gpt-4o-mini` extractor | 19 of 54 (35.2%) | 27 of 54 (50.0%) |
+
+The memory contract, with this extractor, scores below the full history. The extractor
+did not store the answer in 19 of the 35 misses on sample A, and in 16 of the 27 misses on
+sample B. With 54 questions, each score can be about 10 points off, and the memory arm
+differs by 14.8 points between the two samples. In the cloud, a search took 361 ms at p50 with the vector ranking, and
 998 ms with the fused ranking.
 
 ## Documents

bench/README.md

@@ -48,27 +48,54 @@ sessions of each question. Git ignores `bench/results/`, and
 
 ## The end-to-end score
 
-This command runs three arms on the sample of 54 questions in `bench/sample.json`, 9 of
-each question type:
+This command runs three arms on a sample of 54 questions, 9 of each question type:
 
 ```sh
 uv run --group bench --env-file .env python -m bench.e2e
 ```
 
-Without `--run`, it prints the cost estimate of each arm and calls no model. With `--run`,
-it stops if the estimate is over 50 dollars, and else it calls the models. Set
-`OPENAI_MODEL_KEY` in `.env` first.
+Without `--run` or `--replay`, it prints the cost estimate of each arm and calls no model.
+With `--run`, it stops if the estimate is over 50 dollars, and else it calls the models.
+Set `OPENAI_MODEL_KEY` in `.env` first. With `--replay`, it takes every reply from the
+cache, and it stops at the first reply that the cache does not hold. The estimate does not
+subtract the cached replies, so the stop at 50 dollars applies to a replay too.
 
 | Arm | What the reader gets |
 |---|---|
 | `full` | Every session of the question. |
-| `retrieval` | The 10 sessions that the fused search ranks first. |
+| `retrieval` | The 10 sessions that the search ranks first. |
 | `memory` | The block of `recall`, after `capture` ran on each session in date order. |
 
 The reader and the judge are `gpt-4o-2024-08-06` at temperature 0, with the prompts of
-the reference code. The extractor of the memory arm is `gpt-4o-mini-2024-07-18`. Each
-reply is kept in `bench/cache/`, so a second run with the same prompts calls no model.
-`--arms` runs some of the arms, and `--limit` runs the first N questions of the sample.
+the reference code. The baseline extractor of the memory arm is `gpt-4o-mini-2024-07-18`.
+Each reply is kept in `bench/cache/`, so a second run with the same prompts calls no model.
+
+There are two samples. Sample A is `bench/sample.json`, drawn with the seed `20261010`.
+Each phase iterates on sample A. Sample B is `bench/sample-b.json`, drawn with the seed
+`20261013` from the questions that are not in sample A. A phase runs sample B once, at its
+end.
+
+The options are these:
+
+| Option | Values | Default |
+|---|---|---|
+| `--sample` | `a` or `b` | `a` |
+| `--arms` | a comma-separated list of `full`, `retrieval`, and `memory` | every arm |
+| `--retrieval-ranking` | `fused` or `vector`, the ranking of the retrieval arm | `fused` |
+| `--memory-ranking` | `fused` or `vector`, the ranking of `capture` and `recall` | `fused` |
+| `--recall-k` | the `k` of the memory recall | `30` |
+| `--recall-budget` | the `budget` of the memory recall, in characters | `8000` |
+| `--extractor` | a variant name in `VARIANTS` of `bench/prompts.py` | `baseline` |
+| `--replay` | call no model, and stop at the first reply that the cache does not hold | off |
+| `--workers` | the number of processes that run the questions | `1` |
+| `--tag` | a suffix for the name of the result file | none |
+| `--limit` | the first N questions of the sample | every question |
+
+Each worker gets the questions in turn, and runs them over a new local database of its
+own. Each table then holds one question at a time, as in a run with one worker. BM25 ranks
+over the whole table before the scope filter, so the rows of a second question in a table
+can change the ranking. An extractor variant names a prompt and a model. The result file is
+`bench/results/e2e-sample-<a or b>[-<tag>].json`, and it records each option.
 
 ## The checks of the results
 
@@ -79,11 +106,26 @@ check does:
 uv run --group bench python -m bench.compare bench/results/<first>.json bench/results/<second>.json
 ```
 
-This command finds the cause of each miss of the memory arm in `e2e-sample.json`. It runs
-`capture` again from the cached replies, so the extractor is not called again. GPT-4o marks
-the stored facts that hold each answer. If the replay of `capture` spends more than 1
-dollar, the cache missed, and the command stops. `--result` names another result file:
+This command finds the cause of each miss of the memory arm in a result file. The default
+file is `e2e-sample-a.json`, and `--result` names another one:
 
 ```sh
-uv run --group bench --env-file .env python -m bench.diagnose
+uv run --group bench --env-file .env python -m bench.diagnose --result bench/results/e2e-sample-b.json
 ```
+
+The command runs `capture` again with the ranking, the `k`, the `budget`, and the extractor
+variant that the result file records. The replay calls no model, and it stops at the first
+reply that the cache does not hold. GPT-4o marks the stored facts that hold each answer.
+Each miss then gets one of four causes:
+
+| Cause | Meaning |
+|---|---|
+| not extracted | No stored fact holds the answer. |
+| extracted, ranked below k | A fact holds the answer, but the search ranks it below `k`. |
+| in the top k, cut by as_of or the budget | A fact is in the top `k`, but `recall` leaves it out. |
+| recalled, reader missed | A fact that holds the answer is in the block. |
+
+For the last cause, GPT-4o tells whether the marked facts in the block hold every value
+that the answer needs ("all present") or only some of them ("some present"). The command
+prints the groups for each question type, and writes them next to the result file, as
+`<result>-diagnosis.json`.