feat(eval): measure whether a repo map helps, with hidden tests (roadmap B4)

CODEBASE.md

@@ -1,6 +1,6 @@
 # TinyCode — CODEBASE Map
 
-> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1249 test functions + 10 fuzz targets, race-detector clean.
+> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1270 test functions + 10 fuzz targets, race-detector clean.
 
 ## Quick Reference
 
@@ -869,7 +869,7 @@ User Input (textarea / CLI arg)
 
 ## Testing
 
-- **1249 test functions + 10 fuzz targets** in the main module — what `go test ./... -count=1` runs, since `tuiprobe/` is a nested module; each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks
+- **1270 test functions + 10 fuzz targets** across the repository — what `go test ./... -count=1` runs for the main module, plus the evaluation corpora under `eval/` (nested modules the command still walks); each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks
 - `make fuzz` (`FUZZTIME=30s`) explores every fuzz target; `go test` already runs their seed corpora, so CI exercises them on every push
 - `make test-browser` (`BROWSER_TEST=1`, `-run TestBrowserSmoke`) renders a JavaScript page through both browser paths against a loopback server and asserts the request carried the proxy's `Via` header, which proves the filtering proxy was used; `TestBrowserSmokeRefusesABlockedSubresource` then serves a page whose own authority is exempted from the loopback rule while the policy stays enforced, and asserts a subresource pointing at a *second, live* loopback service is refused (the script's `onerror` marker in the DOM) and that the service was never reached — a refusal is proven, not only that the proxy was used (issue #8). Both skip without a browser, so `make test` never launches one
 - `make test-tui-visual` (`TUI_SHOT=1`) is the only place the TUI is looked at rather than asserted on: it renders the committed frame scenarios to PNGs through headless Chromium, starts `bin/tinycode` on a real 80x24 PTY (asserting the stream carried SGR styling and that the binary quits on the documented double Ctrl+C), runs it once more on a size-less PTY, and replays the live stream into a screen buffer that is screenshotted as `tinycode-pty-welcome-80x24.png`. Both tests skip without the variable, so `make test` needs neither a browser, nor a binary, nor a terminal device; PNGs land in `TUI_SHOT_DIR` (default `/tmp`)
@@ -915,7 +915,7 @@ Counts in this document are produced by these commands at the commit they descri
 
 ```bash
 find . -name '*_test.go' -not -path './tuiprobe/*' -print0 \
-  | xargs -0 grep -h '^func Test' | wc -l              # 1249 test functions (main module)
+  | xargs -0 grep -h '^func Test' | wc -l              # 1270 test functions (main module)
 grep -rn '^func Fuzz' --include=*_test.go . | wc -l    # 10 fuzz targets
 ls tui/testdata/golden/frames/*.txt | wc -l            # 29 plain-text frames
 ls tui/testdata/golden/ansi/*.ansi | wc -l             # 1 raw ANSI frame

README.md

@@ -107,7 +107,7 @@ Custom **CellGrid** frame-buffer renders markdown directly in the terminal — n
 - **GitHub Actions**: Two workflows — main.yml (build + lint + test on push/PR) and release.yml (cross-compile + GitHub Releases on tags v*)
 - **main.yml jobs**: `ci` (build, gofmt gate, vet, tests, `-race`, repeated run), `lsp` (real gopls), `browser` (real Chromium), `tui-visual` (frame PNGs + the built binary on a PTY), `cross` (linux/amd64, linux/arm64, darwin/arm64 build + vet) and `staticcheck`
 - **Makefile improvements**: test target preserves exit code with pass/fail message; releases target cross-compiles all platforms + .tar.gz archives
-- **1249 test functions + 10 fuzz targets** in the main module — the one `go test ./... -count=1` runs, since `tuiprobe/` is a nested module — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement
+- **1270 test functions + 10 fuzz targets** across the repository — the main module plus the evaluation corpora, with `tuiprobe/` a nested module the command below excludes — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement
 - **Annotations**: the jobs carry one `ubuntu-latest` migration notice each, plus the `setup-chrome@v1` Node 20 warning on `browser` and `tui-visual`; the gate is no *new* annotations, not zero
 
 ### Running the checks locally

docs/eval.md

@@ -8,7 +8,9 @@ unbuilt, on purpose: building it before knowing the signal exists would have bee
 ## What it measures
 
 Eighteen tasks, each a directory under [`eval/tasks/`](../eval/tasks) with a `go.mod`, a failing test, a
-`PROMPT` and a declared `TIER`. An attempt copies the task to a scratch directory, runs
+`PROMPT` and a declared `TIER`. (A separate corpus under `eval/spike-repo-index/` exists for the
+repository-index spike: six tasks whose tests are **hidden** until grading, and a `map` configuration the
+default runs never select. [repo-index-spike.md](repo-index-spike.md) is its result.) An attempt copies the task to a scratch directory, runs
 `tinycode "<prompt>"` there, and is graded by the task's own `go test ./...` exit code — no judge model, no
 rubric, no text comparison.
 

docs/repo-index-spike.md

@@ -0,0 +1,136 @@
+# Repository index: the spike's answer
+
+Roadmap B4 asked whether the agent needs a repository index or repo map, and said the question had to
+be **measured before it was built**. This is the measurement, from issue
+[#308](https://github.com/yusiwen/TinyCode/issues/308), run on 2026-10-28.
+
+## The verdict
+
+**No-go for an injected repo map**, under the condition the issue registered before the run: a map
+configuration had to solve at least two more of the six tasks than the baseline without losing one, **or**
+hold the solved rate and cut tokens per solved task by ≥ 20%. It did neither — it solved the same six
+and raised tokens per solved task by 35%.
+
+Two facts qualify that verdict, and both are in the numbers below:
+
+- **The corpus had no headroom left.** The baseline solved 6/6, so "two more solved" was not reachable on
+  this task set no matter how good the map was. The solved-rate branch of the condition could not fire.
+- **The cost figure is dominated by one run.** The map arm's *median* run was cheaper and shorter (87k
+  tokens and 10 steps, against 116k and 15); its *total* is worse because one `t3-cache` attempt spent
+  1.27M tokens over 35 steps. At two repetitions per cell, the mean is not a robust statistic here.
+
+So the answer is not "a repo map is useless". It is: **on the instrument this repository has, an injected
+map showed no benefit that survived the corpus's ceiling and the run-to-run spread**, and building an
+index would be spending the largest estimate in the roadmap on a case nobody has measured. The condition
+for revisiting is at the end.
+
+## What was measured
+
+**The tasks.** Six tasks adapted from the existing tier-2/3 suite, each a small Go module whose defect
+spans more than one file, and each **padded with ten unrelated packages** (`internal/metrics`, `configs`,
+`logging`, `textutil`, `clock`, `queue`, `retry`, `idgen`, `validate`, `version`) so that finding the
+defect means surveying a tree rather than reading two files. The corpus lives in
+`eval/spike-repo-index/tasks/`.
+
+| Task | Tier | Defect, as the prompt gives it |
+| --- | --- | --- |
+| `t2-inventory` | 2 | a refused reservation leaves state behind, across the stock and the order |
+| `t2-ledger` | 2 | a refused transfer is not atomic, across the ledger and the account |
+| `t2-report` | 2 | wrong output shape and ordering, across the model and the renderer |
+| `t3-boundary` | 3 | a half-open window is counted closed, in a helper outside the type |
+| `t3-cache` | 3 | expiry at the boundary and refresh on write, across the cache and its clock |
+| `t3-extension` | 3 | a conversion is missing, and the existing rounding must not move |
+
+**Why the tests are hidden.** The harness normally copies a task's tests into the work copy and grades
+with them. That makes the failing test *localise the defect*: `go test` prints the package and line, so a
+repo map's discovery half is measured away. For this spike the work copy carries **no test at all**; the
+hidden suite is installed after the run (`-hidden`), so the prompt is the only specification and finding
+the code is work. `TestSpikePromptsDoNotNameFiles` fails the build if a prompt names a file.
+
+**The two arms.**
+
+| Configuration | What the agent gets |
+| --- | --- |
+| `build` | the prompt, and the tree to explore |
+| `map` | the same, plus a bounded symbol map of the work copy prepended to the prompt |
+
+The map is generated by the runner (`repoMapDigest`): every non-test Go file, its package, its imports,
+and its exported functions, methods and types with signatures — sorted by path, capped at 2000 tokens.
+It is a survey, not a capability: the `map` arm gives the model no new tool.
+
+**The run.** 6 tasks × 2 configurations × 2 repetitions = 24 attempts, graded by each task's hidden suite,
+metrics read from the run's own trajectory. Model `deepseek-flash`, one machine, 2026-10-28.
+
+## Results
+
+| Task | `build` solved | `build` tokens (2 reps) | `build` steps | `map` solved | `map` tokens (2 reps) | `map` steps |
+| --- | --- | --- | --- | --- | --- | --- |
+| `t2-inventory` | 2/2 | 302,245 · 168,462 | 23 · 15 | 2/2 | 98,416 · 52,257 | 11 · 7 |
+| `t2-ledger` | 2/2 | 99,683 · 76,899 | 10 · 8 | 2/2 | 80,013 · 143,175 | 9 · 10 |
+| `t2-report` | 2/2 | 36,918 · 38,710 | 7 · 7 | 2/2 | 29,495 · 28,425 | 5 · 5 |
+| `t3-boundary` | 2/2 | 85,402 · 134,111 | 10 · 15 | 2/2 | 243,681 · 83,058 | 15 · 9 |
+| `t3-cache` | 2/2 | 120,440 · 488,093 | 13 · 22 | 2/2 | **1,273,531** · 223,148 | **35** · 16 |
+| `t3-extension` | 2/2 | 110,858 · 130,208 | 15 · 17 | 2/2 | 70,581 · 90,898 | 10 · 10 |
+
+| | `build` | `map` |
+| --- | ---: | ---: |
+| Solved | 6/6 (12/12 runs) | 6/6 (12/12 runs) |
+| Median tokens per run | 115,649 | 86,978 |
+| Mean tokens per run | 149,336 | 201,390 |
+| Total tokens | 1,792,029 | 2,416,678 |
+| Median steps per run | 15 | 10 |
+| Tokens per solved task | 298,671 | 402,779 |
+
+The median row is the one that says anything nice about the map: cheaper and shorter runs, which is what
+"a survey replaces exploration" would look like. The total is the one the pre-registered condition names,
+and it says the opposite, because of the single 35-step attempt. **Both are reported because either
+alone is misleading.**
+
+## What the spike does not answer
+
+- **Whether a *callable* index helps.** The tool-surface arm (query for definitions and references across
+  the tree — the LSP `workspace/symbol` and call-hierarchy route) was not built. The issue's method named
+  three configurations; this ran two, and the narrowing is recorded here rather than discovered later.
+  A tool that answers one question at a time is a different treatment from 2000 tokens of map in every
+  prompt, and its cost profile is the opposite: paid per use instead of per run.
+- **Whether an index helps on a real repository.** These modules are twenty to thirty files. The larger
+  the tree, the more a map should matter — and the harder it is to build a task whose hidden suite is
+  fair.
+- **Whether the map's shape was right.** Symbols and imports, capped at 2000 tokens, no ranking. A map
+  that ranked files by relevance to the prompt (which is what the memory store's vector machinery could
+  do) was not tried.
+
+## When to revisit
+
+Reopen B4 when **all** of these hold:
+
+1. the corpus can defeat the baseline — at least two of its tasks must fail `build` today, or the
+   solved-rate branch of this experiment cannot fire;
+2. repetitions are at least three per cell, so the mean is not one attempt's anecdote;
+3. the treatment is the **tool surface**, not the injected map: the map arm is measured and lost.
+
+If the tool arm also fails those tasks, the entry stays closed with this document as its answer, and the
+"largest bet in the roadmap" is recorded as not worth the bet.
+
+## Reproducing
+
+```bash
+make build
+cp bin/tinycode /tmp/tinycode-under-test     # see the note below
+TINYCODE_EVAL=1 go run ./eval/runner \
+  -tasks eval/spike-repo-index/tasks \
+  -hidden eval/spike-repo-index/hidden \
+  -configurations build,map -reps 2 \
+  -out /tmp/spike-out -work /tmp/spike-work \
+  -binary /tmp/tinycode-under-test
+```
+
+`go test ./eval/runner` proves the corpus offline: every spike task is red with its hidden suite
+installed and green after its reference solution, the work copies carry the padding, and the prompts name
+no file. None of that needs a provider.
+
+**Why the binary is copied out of the tree.** On this machine, executing `bin/tinycode` in place fails
+every provider call (`dial tcp …: connect: bad file descriptor`, or an i/o timeout) while a byte-identical
+copy at `/tmp/tinycode-under-test` succeeds. The cause is outside this repository (the agent harness's
+sandboxing of that path); the copy is the workaround, and the September release binary in `bin/` behaves
+the same way in place.

docs/roadmap.md

@@ -18,8 +18,8 @@ after it (see *Provenance and method*).
 | :--- | ---: | :--- |
 | Delivered | 16 | the entry's **Picked up** line names the issue that landed it |
 | Refuted | 1 | E5 — the capability exists, recorded so it is not re-filed |
-| In progress | 3 | F1 — three increments landed, the next one is not opened; B5 — S1 landed, S2 and S3 open; B4 — the spike is open, no implementation |
-| Not started | 20 | verified gaps, no issue opened |
+| In progress | 2 | F1 — three increments landed, the next one is not opened; B5 — S1 landed, S2 and S3 open |
+| Not started | 21 | verified gaps, no issue opened |
 | **Total** | **40** | `grep -Ec '^\| [A-H][0-9] \| .* \| (Delivered —\|Refuted\|In progress\|Not started)' docs/roadmap.md` |
 
 **Open work**: F1, whose next increment (a task set with headroom) is not opened, and
@@ -42,7 +42,7 @@ B4 (repository index), F1, then the larger D and E items.
 | B1 | The memory capability exists but is never called | Delivered — #293 |
 | B2 | No local memory backend | Delivered — #293 |
 | B3 | No embeddings, vector search or rerank | Delivered — #297 |
-| B4 | No repository index or repo map | In progress — #308 |
+| B4 | No repository index or repo map | Not started — spiked 2026-10-28, no-go (#308) |
 | B5 | Context management has a single strategy | In progress — #306 |
 | B6 | Truncation is fixed, not budget-aware | Delivered — #304 |
 | C1 | No checkpoint and no undo | Delivered — #234 |
@@ -369,10 +369,15 @@ current tree. D7 is the exception: it was recorded later, so its references are
 
 ### B4 — No repository index or repo map
 
-- **Picked up**: [#308](https://github.com/yusiwen/TinyCode/issues/308) — **S0, the spike**, and only
-  the spike: six cross-file tasks in `eval/tasks/`, three configurations (`build`, a repo map injected
-  into the prompt, a symbol query surface) and a pre-registered abandon condition. The implementation is
-  not started and becomes its own issue only if the numbers say go; the entry stays here until then.
+- **Spiked 2026-10-28** ([#308](https://github.com/yusiwen/TinyCode/issues/308)): **no-go for an
+  injected repo map**, under the condition the issue registered before the run. Six cross-file tasks with
+  hidden suites, ten packages of unrelated padding each, and two arms (`build`, `build` + a 2000-token
+  symbol map in the prompt): both solved 6/6, and the map arm raised tokens per solved task by 35%
+  (298,671 → 402,779). The verdict and its two qualifications — the corpus had no headroom left, and the
+  cost figure rests on one 35-step attempt while the median run got cheaper — are in
+  [repo-index-spike.md](repo-index-spike.md), which also states the three conditions under which this
+  entry reopens (a corpus that defeats the baseline, at least three reps per cell, and the callable
+  tool surface instead of the injected map, which was never built).
 - **Gap**: no tree-sitter, no symbol index, no call graph, no repo map. The agent cannot answer
   "who calls this symbol" or "what does changing this affect".
 - **Evidence**: searches for `tree-sitter`, `symbol.?index`, `call.?graph`, `codebase.?index`,