Skip to content

docs(memory): record the recall benchmark on two real models - #300

Merged
yusiwen merged 1 commit into
masterfrom
docs/memory-bench-results
Oct 26, 2026
Merged

yusiwen merged 1 commit into
masterfrom
docs/memory-bench-results

Conversation

@yusiwen

@yusiwen yusiwen commented Oct 26, 2026

Copy link
Copy Markdown
Owner

What changed

  • docs/memory-bench.md: the measured table replaces the empty cell. Two models, 2026-10-26, local
    Ollama 0.40.11, each run twice and byte-identical both times; the raw MEMBENCH lines are quoted
    under the table so the numbers can be re-derived from the page itself.
  • docs/roadmap.md: B3's Delivered note no longer says the measurement does not exist; it states the
    result (0/6 → 4/6 and 3/6 at hit@3 on paraphrases; 0/4 → 1/4 and 4/4 cross-lingual).
  • agent/memory_bench_test.go: the gated run now logs the vector dimension, since a vector's shape is
    part of what was measured.

The numbers

Ranker Model Group n hit@1 hit@3 hit@5 MRR
lexical — paraphrase 6 0.00 0.00 0.00 0.00
semantic nomic-embed-text (768) paraphrase 6 0.33 0.67 0.83 0.49
semantic bge-m3 (1024) paraphrase 6 0.50 0.50 0.83 0.61
lexical — crosslingual 4 0.00 0.00 0.00 0.00
semantic nomic-embed-text (768) crosslingual 4 0.00 0.25 0.25 0.08
semantic bge-m3 (1024) crosslingual 4 0.75 1.00 1.00 0.83

Both models keep hit@1 = 1.00 on the word-sharing group, so the semantic path did not trade the easy
case away.

Why

This is the measurement the previous PR deliberately left open. It also separates two claims that were
being made together: paraphrase recall is a property of having embeddings at all (either model closes
it), while cross-lingual recall is a property of which model — the English-focused one finds one of
four, the multilingual one finds four of four.

Verification

  • go test ./... -count=1 → 16 packages ok; gofmt and go vet clean on the changed file.
  • Both benchmark runs repeated: the MEMBENCH lines from the second run diff-identical to the first
    for both models.
  • make memory-bench is the same command the page documents; the doc's numbers were produced by it,
    not by hand.
  • git log --format='%h %G? %s' → G.

Honest scope

  • The two models were installed locally for this run (274 MB + 1.2 GB in the machine's Ollama store);
    they are not part of the repository.
  • Six paraphrase and four cross-lingual queries: one query moves a column by 17 and 25 points. The page
    says to read the ordering, not the decimals.
  • Only Ollama's /api/embed was exercised end to end; the OpenAI-compatible path is still covered by
    httptest only.

Follow-ups

Refs #297, #168

B3's note said quality was unclaimed because nothing had been measured. Both
runs are now on the page, against a local Ollama 0.40.11, each model run twice
and byte-identical both times:

- paraphrase (a query sharing no token with its fact): the lexical ranker finds
  none of the six in the top ten; nomic-embed-text (768 dims) puts four of six in
  the top three, bge-m3 three of six;
- cross-lingual (Chinese query, English fact): lexical none of four; nomic one of
  four; bge-m3 all four, three of them first;
- the word-sharing group stays at hit@1 1.00 for both models, so nothing was
  traded for the gain.

The numbers separate the two claims the page was written to test: the paraphrase
gap is closed by either model, and the cross-lingual gap is closed only by the
multilingual one — same store, same corpus, same code.

The harness now logs the vector dimension beside the model, because a vector's
shape is part of what was measured, and the roadmap entry points at the result
instead of saying the measurement does not exist.

Refs #297, #168
@yusiwen yusiwen added the documentation Improvements or additions to documentation label Oct 26, 2026
@yusiwen
yusiwen merged commit d2e3e36 into master Oct 26, 2026
9 checks passed
@yusiwen
yusiwen deleted the docs/memory-bench-results branch October 26, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant