Phase 5: a public benchmark #13

This issue is the task brief for phase 5. docs/internal/plan.md holds the same plan, and the pull request keeps the two matched. The plan is on branch phase-5-public-benchmark, after the commit that closes phase 4 in the roadmap (#12), and it rides in the phase pull request.

Goal

When this phase closes, hotmemory has its first public numbers on LongMemEval-S, a benchmark of 500 questions over long chat histories. The numbers come in two parts, in this order.

1. Retrieval recall at k. This score needs no model except the embedder. It compares the sessions that hotmemory ranks for a question with the sessions that hold the answer. It measures the store and the retrieval query alone.
2. End-to-end accuracy on a sample of the questions. An extractor writes facts, recall gives a block to a reader model, and a judge model grades the answer. The same sample also runs without memory, so the number has a baseline next to it.

docs/benchmarks.md holds the numbers. Each number has the configuration that produced it. The survey (section 2.8) names this run as the proof that hotmemory is a general memory and not only the feature of one consumer.

Decisions

The owner agreed to all of these on 2026-10-10, and asked to keep the phase simple and stay with this plan.

• The benchmark (agreed). LongMemEval-S, from the cleaned data file of 2025-09, which is under MIT. LoCoMo, MemoryBench, and DolphinBench wait, for the reasons in the research notes of 2026-10-08. The roadmap lists them under Deferred.
• The runner (agreed). The runner lives in bench/ at the root of the repository, outside the package. Its dependencies are in a bench dependency group, so the library gains no runtime dependency. The data downloads to a folder that git ignores. Ruff and mypy check bench/. Its offline tests use a small fixture in the shape of the data, and make verify runs them, so it stays under five seconds. No benchmark run is in CI.
• The retrieval score (agreed). One episode for each session, with the user turns of that session as its content, as in the session-level retrieval of the reference code. The episode key is the session id, with - for each character outside [A-Za-z0-9_-]. The episode keeps the session id in sources. A search with Filter(kind="episode") and k of 50 gives a ranked list of sessions. The runner scores recall_any@k, recall_all@k, and ndcg_any@k, with the formulas of the reference code. k is 1, 3, 5, 10, 30, and 50. It scores the questions that the reference code scores: not the abstention questions, and not the questions with no evidence in a user turn. The research notes count 419 such questions from the data. Task 3 counts them again with the reference rules.
• One question at a time (agreed). Each question gets its own namespace. The runner loads its sessions, searches, and then deletes its rows before the next question. Thus the tables hold one question at a time, as the reference code builds one index for each question. This matters for the fused ranking. BM25 takes its top rows from the whole table, and the scope filter applies after that. With 500 questions in one table, BM25 finds almost no row in the scope of one question. A second run keeps every question in one table, and the report shows the difference as a known limit of the fused ranking.
• The rankings and the drivers (agreed). The main numbers come from HotdataStore on the local stack, for both ranking="fused" and ranking="vector". MemoryStore runs the first 20 questions with the same embedder. Its vector ranking must give the same sessions in the same order as the vector ranking of HotdataStore. The oracle test already asks this.
• The embedder (agreed). OpenAIEmbedder with text-embedding-3-small, through a cache on disk that git ignores. The cache key is the model name and the text, so a second run calls no model, and the runs can be repeated.
• Where it runs (agreed). The ingest and the scores run on the local stack. A separate run of 20 questions on a cloud database measures the latency of search at p50, p95, and p99. It reports the first call to a cold engine as its own number. The report names the setup of every number, local or cloud.
• The end-to-end sample (agreed). 9 questions of each of the 6 question types, 54 in all, chosen with a fixed seed. The list of question ids is committed, so every arm and every later run uses the same questions. A full run of 500 questions waits for the owner's yes, because its cost is about ten times higher.
• The end-to-end arms (agreed). Three arms run on the sample, with the same reader and judge.
  1. Full context: the reader gets every session of the question. This is the baseline that the paper reports, about 60 percent with GPT-4o.
  2. Retrieval: the reader gets the top sessions that the retrieval score ranks. This arm needs no extractor.
  3. Memory: an extractor runs capture on each session, in date order, and the reader gets the block of recall.
• The models (agreed). The reader is GPT-4o and the judge is gpt-4o-2024-08-06 at temperature 0, with the judge prompts of the reference code for each question type. Then the numbers compare with the published GPT-4o numbers. The benchmark sets the judge: its scoring script takes GPT-4o, GPT-4o mini, or a Llama model, and its README runs it with GPT-4o. The benchmark leaves the reader to the caller, and its generation script runs readers from GPT-4o to small open models. So GPT-4o as the reader is our choice, made so that the numbers compare with the published ones. The extractor is Claude Haiku 4.5. The runner reads OPENAI_API_KEY and ANTHROPIC_API_KEY from the environment.
• The time rules (agreed). The runner sorts the sessions of a question by date before it loads them, because 211 haystacks are out of order. The session date becomes observed_at in UTC. recall takes as_of at 23:59:59 on the date of the question, because 76 questions have evidence later on the same day. These counts come from the research notes and are not observed yet. Task 2 counts them again.
• The cost limit (agreed). The runner prints an estimate before a paid run, and stops if the estimate is over 50 dollars. The sample with the three arms is about 25 to 35 dollars by the research estimates. That estimate is not observed. By the same estimate, the full-context arm is about half of it, because each of its 54 reads holds about 100,000 tokens.

Tasks

Worked in order on one branch, in one pull request.

• 1. Close phase 4 in roadmap.md and replace plan.md with the phase 5 plan.
• 2. bench/: the dependency group, the data download and its checks, the sort by date, the time rules, the session ids in keys and sources, and the embedder cache. Offline tests on a small fixture.
• 3. The retrieval metrics, ported from the reference code, with offline tests that compare them with values worked by hand.
• 4. The retrieval runner on the local stack: one question at a time, both rankings, all scored questions. Then the run with every question in one table.
• 5. The driver check: the first 20 questions on MemoryStore and on HotdataStore, with the vector ranking.
• 6. The cloud latency run of 20 questions.
• 7. The end-to-end runner: the extractor, the reader, the judge, the three arms, and the cost limit. Then the run on the sample of 54.
• 8. docs/benchmarks.md with every number and its configuration, a link from the README, and the docs audit.

Acceptance criteria

• AC1. make verify passes offline in under five seconds, with the offline tests of bench/.
• AC2. One command runs the retrieval score on the local stack, and docs/benchmarks.md reports recall_any@k, recall_all@k, and ndcg_any@k for every scored question, for both rankings.
• AC3. On the first 20 questions, the vector ranking of MemoryStore and of HotdataStore give the same sessions in the same order.
• AC4. docs/benchmarks.md reports the accuracy of the three arms on the same 54 questions, for each question type and in total, with the reader, the judge, and the extractor named.
• AC5. Every number in docs/benchmarks.md was observed in this phase, and its line names the configuration: the data file hash, the models, the embedder, k, the budget, the driver, and local or cloud.
• AC6. No public text names a private repository, a customer, or a deployment detail. A grep for the known names proves it over the files, the commit messages, the pull request, and the issue.

Verification

• Per commit: make verify.
• Per commit that touches a driver: make local-up, then make integration.
• Each run writes its configuration and its results to one JSON file. The numbers in docs/benchmarks.md come from those files.
• Before the pull request: the docs audit of the pr-workflow skill, and the grep for AC6.

Questions left open

• A search over history, so that recall(as_of=T) can return a fact superseded after T. Some temporal questions ask for a value before it changed. The report shows the score of each question type, so the cost of this gap becomes visible.
• After the engine fixes the filtered vector index search: let the vector rankings use the index again, and measure the gain.
• BM25 in a narrow scope. If the shared-table run shows a large loss, a later phase can rank BM25 inside the scope.
• An exact count for each namespace in the block of profile.
• The ideas from the study of another memory layer on 2026-10-09 wait until this roadmap reaches its basic memory layer.

Stop and ask if

• A benchmark needs a field that the record does not have, or a new Store method.
• The ingest of the local run takes more than two hours.
• The cost estimate of a paid run is over 50 dollars.
• The retrieval score of the two drivers differs on the first 20 questions.
• The reference code and the research notes disagree on the scored questions or on a formula.