Repository navigation
docs(memory): record the recall benchmark on two real models - #300
Merged
Merged
Conversation
B3's note said quality was unclaimed because nothing had been measured. Both runs are now on the page, against a local Ollama 0.40.11, each model run twice and byte-identical both times: - paraphrase (a query sharing no token with its fact): the lexical ranker finds none of the six in the top ten; nomic-embed-text (768 dims) puts four of six in the top three, bge-m3 three of six; - cross-lingual (Chinese query, English fact): lexical none of four; nomic one of four; bge-m3 all four, three of them first; - the word-sharing group stays at hit@1 1.00 for both models, so nothing was traded for the gain. The numbers separate the two claims the page was written to test: the paraphrase gap is closed by either model, and the cross-lingual gap is closed only by the multilingual one — same store, same corpus, same code. The harness now logs the vector dimension beside the model, because a vector's shape is part of what was measured, and the roadmap entry points at the result instead of saying the measurement does not exist. Refs #297, #168
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
docs/memory-bench.md: the measured table replaces the empty cell. Two models, 2026-10-26, localOllama 0.40.11, each run twice and byte-identical both times; the raw
MEMBENCHlines are quotedunder the table so the numbers can be re-derived from the page itself.
docs/roadmap.md: B3's Delivered note no longer says the measurement does not exist; it states theresult (0/6 → 4/6 and 3/6 at hit@3 on paraphrases; 0/4 → 1/4 and 4/4 cross-lingual).
agent/memory_bench_test.go: the gated run now logs the vector dimension, since a vector's shape ispart of what was measured.
The numbers
Both models keep hit@1 = 1.00 on the word-sharing group, so the semantic path did not trade the easy
case away.
Why
This is the measurement the previous PR deliberately left open. It also separates two claims that were
being made together: paraphrase recall is a property of having embeddings at all (either model closes
it), while cross-lingual recall is a property of which model — the English-focused one finds one of
four, the multilingual one finds four of four.
Verification
go test ./... -count=1→ 16 packagesok;gofmtandgo vetclean on the changed file.MEMBENCHlines from the second rundiff-identical to the firstfor both models.
make memory-benchis the same command the page documents; the doc's numbers were produced by it,not by hand.
git log --format='%h %G? %s'→G.Honest scope
they are not part of the repository.
says to read the ordering, not the decimals.
/api/embedwas exercised end to end; the OpenAI-compatible path is still covered byhttptestonly.Follow-ups
Refs #297, #168