test(memory): a recall benchmark with a lexical baseline

CODEBASE.md

@@ -1,6 +1,6 @@
 # TinyCode — CODEBASE Map
 
-> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1224 test functions + 10 fuzz targets, race-detector clean.
+> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1228 test functions + 10 fuzz targets, race-detector clean.
 
 ## Quick Reference
 
@@ -849,7 +849,7 @@ User Input (textarea / CLI arg)
 
 ## Testing
 
-- **1224 test functions + 10 fuzz targets** in the main module — what `go test ./... -count=1` runs, since `tuiprobe/` is a nested module; each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks
+- **1228 test functions + 10 fuzz targets** in the main module — what `go test ./... -count=1` runs, since `tuiprobe/` is a nested module; each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks
 - `make fuzz` (`FUZZTIME=30s`) explores every fuzz target; `go test` already runs their seed corpora, so CI exercises them on every push
 - `make test-browser` (`BROWSER_TEST=1`, `-run TestBrowserSmoke`) renders a JavaScript page through both browser paths against a loopback server and asserts the request carried the proxy's `Via` header, which proves the filtering proxy was used; `TestBrowserSmokeRefusesABlockedSubresource` then serves a page whose own authority is exempted from the loopback rule while the policy stays enforced, and asserts a subresource pointing at a *second, live* loopback service is refused (the script's `onerror` marker in the DOM) and that the service was never reached — a refusal is proven, not only that the proxy was used (issue #8). Both skip without a browser, so `make test` never launches one
 - `make test-tui-visual` (`TUI_SHOT=1`) is the only place the TUI is looked at rather than asserted on: it renders the committed frame scenarios to PNGs through headless Chromium, starts `bin/tinycode` on a real 80x24 PTY (asserting the stream carried SGR styling and that the binary quits on the documented double Ctrl+C), runs it once more on a size-less PTY, and replays the live stream into a screen buffer that is screenshotted as `tinycode-pty-welcome-80x24.png`. Both tests skip without the variable, so `make test` needs neither a browser, nor a binary, nor a terminal device; PNGs land in `TUI_SHOT_DIR` (default `/tmp`)
@@ -895,7 +895,7 @@ Counts in this document are produced by these commands at the commit they descri
 
 ```bash
 find . -name '*_test.go' -not -path './tuiprobe/*' -print0 \
-  | xargs -0 grep -h '^func Test' | wc -l              # 1224 test functions (main module)
+  | xargs -0 grep -h '^func Test' | wc -l              # 1228 test functions (main module)
 grep -rn '^func Fuzz' --include=*_test.go . | wc -l    # 10 fuzz targets
 ls tui/testdata/golden/frames/*.txt | wc -l            # 29 plain-text frames
 ls tui/testdata/golden/ansi/*.ansi | wc -l             # 1 raw ANSI frame

Makefile

@@ -46,6 +46,22 @@ test-race:
 test-repeat:
 	go test ./... -count=3
 
+# The memory recall benchmark (docs/memory-bench.md) measured against a real
+# embedding endpoint — Ollama's /api/embed by default, or any OpenAI-compatible
+# /embeddings with MEMORY_BENCH_PROVIDER=openai. Without a URL and a model the
+# test skips with the reason, like every other gate, so an ordinary `make test`
+# never needs a model.
+#
+#	make memory-bench MEMORY_BENCH_URL=http://127.0.0.1:11434 MEMORY_BENCH_MODEL=nomic-embed-text
+#
+# -v on purpose: the per-group numbers and the per-miss lines are the output.
+memory-bench:
+	TINYCODE_MEMORY_BENCH_URL=$(MEMORY_BENCH_URL) \
+	TINYCODE_MEMORY_BENCH_MODEL=$(MEMORY_BENCH_MODEL) \
+	TINYCODE_MEMORY_BENCH_PROVIDER=$(MEMORY_BENCH_PROVIDER) \
+	TINYCODE_MEMORY_BENCH_API_KEY=$(MEMORY_BENCH_API_KEY) \
+	go test -count=1 -v -run TestMemoryBenchRealEndpoint ./agent/
+
 # Language server used by the gated LSP integration tests. Keep this in step
 # with the flake's pkgs.gopls (0.23.0).
 GOPLS_VERSION ?= v0.23.0

README.md

@@ -36,7 +36,7 @@ Custom **CellGrid** frame-buffer renders markdown directly in the terminal — n
 - **A measurement, before a framework**: `TINYCODE_EVAL=1 go run ./eval/runner` runs six small tasks through three agent configurations, grades each attempt by the task's own `go test`, and reports what the trajectories say — solved tasks, noise, tokens per solved task. Gated and never in CI, because it spends money. See [docs/eval.md](docs/eval.md).
 - **Every run leaves a trajectory**: one JSONL file per run — steps, per-call durations and TTFT, tokens, cost, and each tool call with a digest of its arguments, its result size and how it ended — written to `~/.tinycode/trajectories` by default, read back with `--trajectories` and `--trajectory <id>`. Metadata only unless you ask for text, bounded at 200 runs / 64 MiB, and not writable by a tool. See [docs/trajectories.md](docs/trajectories.md).
 - **A goal that continues itself**: `/goal <objective>` stores one objective in the session — phase, round budget, blocker — and when the session goes idle the agent starts another round toward it, up to 256 rounds, with the objective, the round number and the evidence rule in the prompt. A round may complete or block the goal but never rewrite it; Ctrl+C pauses instead of retrying; a resumed or forked session comes back disarmed until you say `/goal resume` (or just "continue"). The build agent is told to create one for work that will not fit a single turn; `/goal` is the explicit path. See [docs/goals.md](docs/goals.md).
-- **Memory that outlives the session**: recall is on by default and reads a local store — one file per project under `~/.tinycode/memory`, deliberately outside the project root and refused to the tool layer, because a fact the agent can rewrite is not one. Nothing is stored unless you ask: `/remember <label>: <fact>` is the only writer, `/memory` lists what is there with the id, and `/memory forget <id>` deletes it. A network backend (Hindsight) is reachable only when your own config file names the endpoint; a project-local config cannot turn it on. Recall matches keywords until you point `memory.embedding` at an embedding endpoint (Ollama, or any OpenAI-compatible `/embeddings`), which switches it to vector similarity with a lexical floor and a weighted rerank; the endpoint is yours to name, and the vectors live beside the facts, keyed by model. See [docs/memory-decision.md](docs/memory-decision.md).
+- **Memory that outlives the session**: recall is on by default and reads a local store — one file per project under `~/.tinycode/memory`, deliberately outside the project root and refused to the tool layer, because a fact the agent can rewrite is not one. Nothing is stored unless you ask: `/remember <label>: <fact>` is the only writer, `/memory` lists what is there with the id, and `/memory forget <id>` deletes it. A network backend (Hindsight) is reachable only when your own config file names the endpoint; a project-local config cannot turn it on. Recall matches keywords until you point `memory.embedding` at an embedding endpoint (Ollama, or any OpenAI-compatible `/embeddings`), which switches it to vector similarity with a lexical floor and a weighted rerank; the endpoint is yours to name, and the vectors live beside the facts, keyed by model. The recall benchmark and its lexical baseline live in [docs/memory-bench.md](docs/memory-bench.md). See [docs/memory-decision.md](docs/memory-decision.md).
 - **AI-generated session titles**: title hidden agent auto-names conversations via LLM. Falls back to first user message. (2deee5e)
 - Input history: Up/Down arrows to recall previous inputs, Esc to exit browse mode
 - Status bar shows `hist 3/5` during history browsing
@@ -107,7 +107,7 @@ Custom **CellGrid** frame-buffer renders markdown directly in the terminal — n
 - **GitHub Actions**: Two workflows — main.yml (build + lint + test on push/PR) and release.yml (cross-compile + GitHub Releases on tags v*)
 - **main.yml jobs**: `ci` (build, gofmt gate, vet, tests, `-race`, repeated run), `lsp` (real gopls), `browser` (real Chromium), `tui-visual` (frame PNGs + the built binary on a PTY), `cross` (linux/amd64, linux/arm64, darwin/arm64 build + vet) and `staticcheck`
 - **Makefile improvements**: test target preserves exit code with pass/fail message; releases target cross-compiles all platforms + .tar.gz archives
-- **1224 test functions + 10 fuzz targets** in the main module — the one `go test ./... -count=1` runs, since `tuiprobe/` is a nested module — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement
+- **1228 test functions + 10 fuzz targets** in the main module — the one `go test ./... -count=1` runs, since `tuiprobe/` is a nested module — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement
 - **Annotations**: the jobs carry one `ubuntu-latest` migration notice each, plus the `setup-chrome@v1` Node 20 warning on `browser` and `tui-visual`; the gate is no *new* annotations, not zero
 
 ### Running the checks locally

agent/memory_bench_test.go

@@ -0,0 +1,311 @@
+package agent
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"os"
+	"sort"
+	"strings"
+	"testing"
+
+	"github.com/yusiwen/tinycode/types"
+)
+
+// benchFact and benchQuery are the corpus file's shape (testdata/memory-bench.json).
+type benchFact struct {
+	Key   string `json:"key"`
+	Value string `json:"value"`
+}
+
+type benchQuery struct {
+	Q    string `json:"q"`
+	Want string `json:"want"`
+	Kind string `json:"kind"` // lexical | paraphrase | crosslingual
+}
+
+type benchCorpus struct {
+	Facts   []benchFact  `json:"facts"`
+	Queries []benchQuery `json:"queries"`
+}
+
+// The corpus's documented shape. docs/memory-bench.md quotes these three
+// numbers, so they are asserted here rather than left to drift with an edit.
+const (
+	benchFactCount        = 24
+	benchQueryCount       = 20
+	benchLexicalQueries   = 10
+	benchParaphraseQuery  = 6
+	benchCrosslingualQry  = 4
+	benchRecallDepth      = 10
+	benchLexicalFloorRank = 3
+)
+
+// loadBenchCorpus reads the fixed corpus the benchmark runs on.
+func loadBenchCorpus(t *testing.T) benchCorpus {
+	t.Helper()
+	raw, err := os.ReadFile("testdata/memory-bench.json")
+	if err != nil {
+		t.Fatalf("read corpus: %v", err)
+	}
+	var corpus benchCorpus
+	if err := json.Unmarshal(raw, &corpus); err != nil {
+		t.Fatalf("parse corpus: %v", err)
+	}
+	if len(corpus.Facts) == 0 || len(corpus.Queries) == 0 {
+		t.Fatal("the corpus is empty; a benchmark that runs on nothing passes vacuously")
+	}
+	return corpus
+}
+
+// TestMemoryBenchCorpusIsWellFormed guards the benchmark's own hygiene. Two
+// properties make its columns mean what the doc says they mean: every query
+// names a fact that exists, and a paraphrase or cross-lingual query shares no
+// token with the fact it wants — if it did, the lexical ranker could answer it
+// and the column would stop measuring what it claims to.
+func TestMemoryBenchCorpusIsWellFormed(t *testing.T) {
+	corpus := loadBenchCorpus(t)
+
+	if len(corpus.Facts) != benchFactCount {
+		t.Errorf("corpus has %d facts, docs/memory-bench.md says %d", len(corpus.Facts), benchFactCount)
+	}
+	if len(corpus.Queries) != benchQueryCount {
+		t.Errorf("corpus has %d queries, docs/memory-bench.md says %d", len(corpus.Queries), benchQueryCount)
+	}
+	byKind := map[string]int{}
+	facts := map[string]benchFact{}
+	for _, fact := range corpus.Facts {
+		if fact.Key == "" || fact.Value == "" {
+			t.Errorf("a fact is missing its key or value: %+v", fact)
+		}
+		if _, dup := facts[fact.Key]; dup {
+			t.Errorf("two facts share the key %q, so a rank cannot say which was found", fact.Key)
+		}
+		facts[fact.Key] = fact
+	}
+	for _, query := range corpus.Queries {
+		byKind[query.Kind]++
+		target, ok := facts[query.Want]
+		if !ok {
+			t.Errorf("query %q wants %q, which is not in the corpus", query.Q, query.Want)
+			continue
+		}
+		shared := sharedTerms(query.Q, target.Key+" "+target.Value)
+		switch query.Kind {
+		case "lexical":
+			if len(shared) == 0 {
+				t.Errorf("lexical query %q shares no token with %q, so it is mislabelled", query.Q, query.Want)
+			}
+		case "paraphrase", "crosslingual":
+			if len(shared) > 0 {
+				t.Errorf("%s query %q shares %v with %q; the lexical ranker could answer it", query.Kind, query.Q, shared, query.Want)
+			}
+		default:
+			t.Errorf("query %q has unknown kind %q", query.Q, query.Kind)
+		}
+	}
+	if byKind["lexical"] != benchLexicalQueries {
+		t.Errorf("corpus has %d lexical queries, docs/memory-bench.md says %d", byKind["lexical"], benchLexicalQueries)
+	}
+	if byKind["paraphrase"] != benchParaphraseQuery {
+		t.Errorf("corpus has %d paraphrase queries, docs/memory-bench.md says %d", byKind["paraphrase"], benchParaphraseQuery)
+	}
+	if byKind["crosslingual"] != benchCrosslingualQry {
+		t.Errorf("corpus has %d cross-lingual queries, docs/memory-bench.md says %d", byKind["crosslingual"], benchCrosslingualQry)
+	}
+}
+
+// TestMemoryBenchLexicalFloor is the CI half of the benchmark: the local
+// ranker must find a fact for a query that names one of its words, in the top
+// three. It is deterministic — same corpus, same store — so it can be a gate
+// rather than a report.
+func TestMemoryBenchLexicalFloor(t *testing.T) {
+	corpus := loadBenchCorpus(t)
+	store := newTestStore(t, t.TempDir(), "bench")
+	for _, fact := range corpus.Facts {
+		if err := store.Remember(fact.Key, fact.Value); err != nil {
+			t.Fatalf("remember %q: %v", fact.Key, err)
+		}
+	}
+
+	ranks := benchRanks(t, store, corpus)
+	for i, query := range corpus.Queries {
+		if query.Kind != "lexical" {
+			continue
+		}
+		if rank := ranks[i]; rank == 0 || rank > benchLexicalFloorRank {
+			t.Errorf("lexical query %q ranked its fact at %d (want <= %d)", query.Q, rank, benchLexicalFloorRank)
+		}
+	}
+	reportBench(t, "lexical", corpus, ranks)
+}
+
+// TestMemoryBenchMetrics pins the arithmetic the report is made of, on rankings
+// with a known answer, so a wrong hit@k cannot look like a result.
+func TestMemoryBenchMetrics(t *testing.T) {
+	m := benchMetricsFor([]int{1, 2, 3, 7, 0})
+	if m.N != 5 {
+		t.Fatalf("N = %d", m.N)
+	}
+	if m.Hit1 != 1 || m.Hit3 != 3 || m.Hit5 != 3 {
+		t.Fatalf("hits = %d/%d/%d, want 1/3/3 (rank 7 is outside the top five)", m.Hit1, m.Hit3, m.Hit5)
+	}
+	wantMRR := (1.0 + 1.0/2 + 1.0/3 + 1.0/7 + 0) / 5
+	if diff := m.MRR - wantMRR; diff > 1e-9 || diff < -1e-9 {
+		t.Fatalf("MRR = %.6f, want %.6f", m.MRR, wantMRR)
+	}
+	if empty := benchMetricsFor(nil); empty.N != 0 || empty.MRR != 0 {
+		t.Fatalf("an empty ranking produced %+v", empty)
+	}
+}
+
+// TestMemoryBenchRealEndpoint measures against a real model. It is gated: a
+// benchmark that silently runs against a fake reports the fake's opinions, and
+// a benchmark that fails without an endpoint is a gate nobody can keep.
+//
+//	TINYCODE_MEMORY_BENCH_URL=http://127.0.0.1:11434 \
+//	TINYCODE_MEMORY_BENCH_MODEL=nomic-embed-text \
+//	go test -count=1 -v -run TestMemoryBenchRealEndpoint ./agent/
+func TestMemoryBenchRealEndpoint(t *testing.T) {
+	base := os.Getenv("TINYCODE_MEMORY_BENCH_URL")
+	model := os.Getenv("TINYCODE_MEMORY_BENCH_MODEL")
+	if base == "" || model == "" {
+		t.Skip("set TINYCODE_MEMORY_BENCH_URL and TINYCODE_MEMORY_BENCH_MODEL to measure against a real embedding endpoint")
+	}
+	var embedder Embedder
+	if strings.EqualFold(os.Getenv("TINYCODE_MEMORY_BENCH_PROVIDER"), "openai") {
+		embedder = NewOpenAIEmbedder(base, model, os.Getenv("TINYCODE_MEMORY_BENCH_API_KEY"))
+	} else {
+		embedder = NewOllamaEmbedder(base, model)
+	}
+
+	corpus := loadBenchCorpus(t)
+	inner := newTestStore(t, t.TempDir(), "bench")
+	semantic, err := NewSemanticMemory(inner, embedder, SemanticOptions{})
+	if err != nil {
+		t.Fatalf("open semantic store: %v", err)
+	}
+	for _, fact := range corpus.Facts {
+		if err := semantic.Remember(fact.Key, fact.Value); err != nil {
+			t.Fatalf("remember %q: %v", fact.Key, err)
+		}
+	}
+	semantic.mu.Lock()
+	embedded := len(semantic.vectors)
+	semantic.mu.Unlock()
+	if embedded == 0 {
+		t.Fatalf("the endpoint at %s returned no vectors for %q; recall would silently fall back to lexical", base, model)
+	}
+	if embedded != len(corpus.Facts) {
+		t.Fatalf("%d of %d facts were embedded; a partial corpus misreports every column", embedded, len(corpus.Facts))
+	}
+
+	t.Logf("model=%s endpoint=%s facts=%d queries=%d", model, base, len(corpus.Facts), len(corpus.Queries))
+	reportBench(t, "lexical", corpus, benchRanks(t, inner, corpus))
+	reportBench(t, "semantic", corpus, benchRanks(t, semantic, corpus))
+}
+
+// benchRanks returns, per query, the 1-based rank of the wanted fact in a
+// recall of benchRecallDepth, or 0 when it is not there.
+func benchRanks(t *testing.T, store types.MemoryStore, corpus benchCorpus) []int {
+	t.Helper()
+	ranks := make([]int, len(corpus.Queries))
+	for i, query := range corpus.Queries {
+		hits, err := store.Recall(context.Background(), query.Q, benchRecallDepth)
+		if err != nil {
+			t.Fatalf("recall %q: %v", query.Q, err)
+		}
+		for position, hit := range hits {
+			if hit.Key == query.Want {
+				ranks[i] = position + 1
+				break
+			}
+		}
+	}
+	return ranks
+}
+
+// benchMetrics is one ranker's score over one group of queries.
+type benchMetrics struct {
+	N, Hit1, Hit3, Hit5 int
+	MRR                 float64
+}
+
+func benchMetricsFor(ranks []int) benchMetrics {
+	m := benchMetrics{N: len(ranks)}
+	var reciprocal float64
+	for _, rank := range ranks {
+		switch {
+		case rank == 1:
+			m.Hit1++
+			m.Hit3++
+			m.Hit5++
+		case rank == 2:
+			m.Hit3++
+			m.Hit5++
+		case rank == 3:
+			m.Hit3++
+			m.Hit5++
+		case rank > 3 && rank <= 5:
+			m.Hit5++
+		}
+		if rank > 0 {
+			reciprocal += 1 / float64(rank)
+		}
+	}
+	if m.N > 0 {
+		m.MRR = reciprocal / float64(m.N)
+	}
+	return m
+}
+
+// reportBench prints one line per (ranker, kind) in both a readable table and a
+// `MEMBENCH ...` form that is easy to quote in docs/memory-bench.md.
+func reportBench(t *testing.T, ranker string, corpus benchCorpus, ranks []int) {
+	t.Helper()
+	kinds := []string{"lexical", "paraphrase", "crosslingual"}
+	for _, kind := range kinds {
+		var group []int
+		for i, query := range corpus.Queries {
+			if query.Kind == kind {
+				group = append(group, ranks[i])
+			}
+		}
+		m := benchMetricsFor(group)
+		if m.N == 0 {
+			continue
+		}
+		t.Logf("MEMBENCH ranker=%-8s kind=%-12s n=%-2d hit1=%.2f hit3=%.2f hit5=%.2f mrr=%.2f",
+			ranker, kind, m.N, float64(m.Hit1)/float64(m.N), float64(m.Hit3)/float64(m.N),
+			float64(m.Hit5)/float64(m.N), m.MRR)
+		for i, query := range corpus.Queries {
+			if query.Kind == kind && ranks[i] == 0 {
+				t.Logf("  miss: %s -> %s", query.Q, query.Want)
+			}
+		}
+	}
+}
+
+// sharedTerms is the hygiene check's ruler: the tokens two texts have in common,
+// sorted so a failure message reads the same every run.
+func sharedTerms(a, b string) []string {
+	inB := map[string]bool{}
+	for _, term := range memoryTerms(b) {
+		inB[term] = true
+	}
+	var shared []string
+	seen := map[string]bool{}
+	for _, term := range memoryTerms(a) {
+		if inB[term] && !seen[term] {
+			seen[term] = true
+			shared = append(shared, term)
+		}
+	}
+	sort.Strings(shared)
+	return shared
+}
+
+// String keeps the metrics readable in a failure message.
+func (m benchMetrics) String() string {
+	return fmt.Sprintf("n=%d hit1=%d hit3=%d hit5=%d mrr=%.2f", m.N, m.Hit1, m.Hit3, m.Hit5, m.MRR)
+}