From 7ddbb833db18c72aba6c18aac19503d080f44d58 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 14 Oct 2026 15:27:52 +0530 Subject: [PATCH 1/2] Close phase 6 in the roadmap and open phase 7 Phase 6 closed with PR #18. The roadmap records what it changed for later phases. The plan drops phase 6, adds the sample B baseline to its start point, and gives phase 7 the commands that phase 6 built. Phase 7 is open as issue #19. --- docs/internal/plan.md | 92 ++++++++++------------------------------ docs/internal/roadmap.md | 16 ++++++- 2 files changed, 37 insertions(+), 71 deletions(-) diff --git a/docs/internal/plan.md b/docs/internal/plan.md index bf3cedb..326af20 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -1,10 +1,12 @@ # Plan: the second workstream, memory accuracy Status: agreed, 2026-10-13. The owner agreed to each decision and to the target on -2026-10-13. Phase 6 is open, as issue #16. This file holds the open workstream, phases 6 to 13. Each phase becomes one GitHub +2026-10-13. Phase 6 closed on 2026-10-14 with PR #18. Phase 7 is open, as issue +#19. This file holds the open workstream, phases 7 to 13. Each phase becomes one GitHub issue and one pull request, worked in task order. When a phase closes, `roadmap.md` gets its paragraph, and this file keeps only the phases still open. The plan of phase 5 is in -git history, at commit `4a6fb36`. +git history, at commit `4a6fb36`. The plan of phase 6, with the decisions taken during +its work, is at commit `8a7318d`. ## Goal @@ -34,7 +36,8 @@ writes, and does not try to reduce it. The roadmap lists the latency work under ## Where we start -All numbers are from `docs/benchmarks.md`, observed on 2026-10-10. +All numbers are from `docs/benchmarks.md`. The rows of sample A were observed on +2026-10-10, and the rows of sample B on 2026-10-14. | Item | Value | |---|---| @@ -45,7 +48,11 @@ All numbers are from `docs/benchmarks.md`, observed on 2026-10-10. | Sample of 54, memory arm | 19 correct, 1,462 reader tokens | | Memory misses, no stored fact held the answer | 19 of 35 | | Memory misses, stored but ranked below 30 | 2 of 35, both ranked higher by the vector ranking | -| Memory misses, a needed fact was in the block | 14 of 35, an upper bound | +| Memory misses, a needed fact was in the block | 14 of 35: 1 with every needed fact, 13 with some | +| Sample B, full context, retrieval, and memory arms | 43, 48, and 27 correct of 54 | +| Sample B memory misses, no stored fact held the answer | 16 of 27 | +| Sample B memory misses, stored but ranked below 30 | 0 of 27 | +| Sample B memory misses, a needed fact was in the block | 11 of 27: 1 with every needed fact, 10 with some | Four facts about the memory arm, read from `bench/e2e.py` and `src/hotmemory/contract.py`, shape the phases below: @@ -91,74 +98,17 @@ The owner agreed to each item on 2026-10-13. `docs/contracts.md`, and `docs/guarantees.md` in the same commit, and `MemoryStore` gets the same behavior as `HotdataStore`. -## Phase 6. The accuracy loop - -Goal: one command runs a variant of the memory arm on a sample, and one command explains -its misses. Every later phase uses these two commands. - -Tasks: - -1. Mark phase 6 open in `roadmap.md`. -2. `bench/e2e.py` takes `--sample a|b`. Sample B is drawn with a new fixed seed, has no - question of sample A, and its ids are committed as `bench/sample-b.json`. -3. The memory arm takes its configuration from the command line and writes it into the - result file: the ranking of the store, `k`, `budget`, and a name for the extractor - variant. The extractor variant names a prompt and a model in `bench/prompts.py`. -4. The retrieval arm takes the ranking from the command line too. Today it is fixed to - fused. -5. `bench/diagnose.py` runs on any result file of the memory arm. It prints the three - groups of misses for each question type. It splits the group "a needed fact was in the - block" in two. The marker tells whether the block holds every needed fact ("all - present") or only some of them ("some present"). -6. Rerun the memory arm on sample A from the caches. It must give 19 of 54 at no model - cost. If it does not, stop. -7. Run all three arms on sample B. This is the baseline of B. -8. Record both baselines in `docs/benchmarks.md`, with their configuration. - -Acceptance criteria: - -- `make verify` passes, with offline tests for the new options and for the sample draw. -- The rerun of sample A from the caches reproduces 19 of 54 and the 19, 2, and 14 split. -- `docs/benchmarks.md` reports the baseline of sample B for each arm and each type. - -Stop and ask if: - -- The cached rerun of sample A does not reproduce the published numbers. -- The estimate for sample B is over 50 dollars. - -Decisions taken during the work. The plan left these details open. The work took the -defaults below, and the owner can change any of them in review. - -- Sample B uses the seed `20261013` and the draw rule of sample A, over the questions that - are not in sample A. It holds 4 abstention questions, and sample A holds 2 (observed). -- One `Config` in `bench/e2e.py` holds the options of tasks 3 and 4, so the two tasks are - one commit. The memory arm uses one ranking for the search inside `capture` and for - `recall`. The options are `--retrieval-ranking`, `--memory-ranking`, `--recall-k`, - `--recall-budget`, `--extractor`, and `--tag`. -- A result file is named `e2e-sample-[-].json`. The phase 5 file - `e2e-sample.json` stays as the record of phase 5. -- `--replay` runs the arms with a model stand-in that raises at the first reply that the - cache does not hold. The diagnosis replays `capture` the same way, in place of the stop - at 1 dollar of spend. -- The diagnosis keeps the marker prompt of phase 5 unchanged, so the cache reproduces the - three groups. A second GPT-4o call splits the group "recalled, reader missed". It gets - the marked facts that are in the block. It tells whether they hold every value that the - answer needs ("all present") or not ("some present"). -- The diagnosis file is named after the result file, as `-diagnosis.json`. -- `--workers N` runs the questions in N processes, and each process has its own local - database. The plan did not ask for it. A run of the three arms took 3.6 hours in - phase 5 with one process, and each later phase runs the samples again. The runner reads - the commit at the start of a run, so a commit during the run does not change the commit - that the result file records. -- The phase spent $21.80 on models, from the result files: $21.42 for the run of sample - B, $0.37 for its diagnosis, and $0.01 for the split of the diagnosis of sample A. The - replays of sample A spent nothing. The first diagnosis run of sample A stalled on one - call and was stopped, and the cost of that call is not observed. - ## Phase 7. The extractor Goal: the extractor stores the facts that the questions ask about. The target is the -group of 19 misses where no stored fact held the answer. +group of misses where no stored fact held the answer: 19 on sample A and 16 on sample B. + +Phase 6 built the commands that this phase uses. `--extractor ` runs a variant +from `VARIANTS` in `bench/prompts.py`. `bench/diagnose.py --result ` gives the +groups of misses of any result file. `--replay` and `--workers N` are described in +`bench/README.md`. A new prompt misses the reply cache, so each variant pays for its +capture, reader, and judge calls. The full-context and retrieval arms do not change, so +run each variant with `--arms memory`. The library does not change in this phase. All work is in `bench/`. @@ -173,6 +123,10 @@ Tasks: - A reply that failed to parse. - A fact that the extractor left out because a near fact was in "Facts already in memory". + + Read from `bench/models.py`, not observed: a reply that reaches the limit raises + before the cache stores it, and a reply that fails to parse stops the run. So the + first two faults are not expected in the cache, and this task confirms it. 3. Write a new prompt variant with these rules: - If the user can ask later about what the assistant said, keep it. - Keep each item of a list as its own fact. diff --git a/docs/internal/roadmap.md b/docs/internal/roadmap.md index 8afa0a7..61211f5 100644 --- a/docs/internal/roadmap.md +++ b/docs/internal/roadmap.md @@ -29,8 +29,8 @@ calls a model. Latency is not a goal, and its work waits under Deferred. | Phase | Delivers | Status | |---|---|---| -| 6. The accuracy loop | A second sample of 54 questions, and the memory arm configured from the command line. The diagnosis split for each question type, and the baselines of both samples. | open, issue #16 | -| 7. The extractor | A benchmark extractor that keeps what the assistant said, with sources, measured against the 19 misses where no stored fact held the answer. No library change. | proposed | +| 6. The accuracy loop | A second sample of 54 questions, and the memory arm configured from the command line. The diagnosis split for each question type, and the baselines of both samples. | closed 2026-10-14, PR #18 | +| 7. The extractor | A benchmark extractor that keeps what the assistant said, with sources, measured against the 19 misses where no stored fact held the answer. No library change. | open, issue #19 | | 8. Episodes behind facts | `capture` stores its text as an episode that its facts cite. `recall` can add the cited or ranked episodes to its block, within one budget. | proposed | | 9. The ranking | An ablation of the fused ranking, and a default ranking chosen by the benchmark. | proposed | | 10. Supersede in capture | A fact can name the current record that it replaces, and `remember` closes that record. The extractor decides, and the library applies the rule. | proposed | @@ -110,3 +110,15 @@ A closed phase keeps its row above with the date it closed and the pull request. With every question in one table, the fused ranking scored higher at the top 5 and slightly lower at the top 10. A BM25 ranking inside the scope can wait. In the cloud, a fused search took 998 ms at p50 and a vector search 361 ms. +- Phase 6 closed on 2026-10-14 with PR #18, which closed issue #16. A replay of sample A + from the caches reproduced 19 of 54 and each cause of each miss, at no model cost. On + sample B, the reader scored 43 with every session, 48 with the top 10 sessions, and 27 + with the block of `recall`. It changed the plan for later phases in four ways. The + memory arm differs by 14.8 points between the two samples with the same configuration, + so a change counts only if both samples move the same way. The extractor is the main + loss on both samples: no stored fact held the answer in 19 of 35 misses on sample A and + 16 of 27 on sample B. When the block held a needed fact, it held every needed fact in + only 2 of 25 misses. A fact ranked below 30 caused 2 misses on sample A and none on + sample B, so the ranking has little to gain on these samples. The runner gained + `--replay` and `--workers`, and a replay of the memory arm of sample A took 519 s with + 4 workers, against 2,264 s with one process. From ab9f9a932f962f052c52cb6eebc30bbca4c75c5d Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 14 Oct 2026 15:33:03 +0530 Subject: [PATCH 2/2] Correct the phase 7 note: a reply that fails to parse is in the cache --- docs/internal/plan.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/internal/plan.md b/docs/internal/plan.md index 326af20..be7ed60 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -124,9 +124,10 @@ Tasks: - A fact that the extractor left out because a near fact was in "Facts already in memory". - Read from `bench/models.py`, not observed: a reply that reaches the limit raises - before the cache stores it, and a reply that fails to parse stops the run. So the - first two faults are not expected in the cache, and this task confirms it. + Read from `bench/models.py` and `bench/e2e.py`, not observed: a reply that reaches the + limit raises before the cache stores it. A reply that fails to parse is stored in the + cache, but it stops the run. So no completed result file holds either fault, and this + task confirms it. 3. Write a new prompt variant with these rules: - If the user can ask later about what the assistant said, keep it. - Keep each item of a list as its own fact.