diff --git a/CODEBASE.md b/CODEBASE.md index f9b057a..3b64f7f 100644 --- a/CODEBASE.md +++ b/CODEBASE.md @@ -1,6 +1,6 @@ # TinyCode — CODEBASE Map -> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1277 test functions + 10 fuzz targets, race-detector clean. +> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1285 test functions + 10 fuzz targets, race-detector clean. ## Quick Reference @@ -258,6 +258,8 @@ provider is implemented — reaching IMDS from an agent process is a separate se A summarizer failure (including `context.Canceled`) returns `nil, err` rather than a no-op history, so a caller can tell a cancelled summary from a healthy "nothing to compress". The agent loop ignores the returned history in that case, exactly as it did when the error was swallowed, but `CompressHistory` surfaces it to the TUI. - **`(*Agent) compactRunMessages(ctx, messages, step)`** (`compaction.go`) — the *run's own* traffic is compacted at a step boundary when it crosses the same threshold; the cut snaps forward to an assistant message so a `tool_calls` message never loses its results. The head (system prompt, history, this run's prompt) and the last eight messages are kept verbatim; the rest becomes `[COMPACTED RUN — REFERENCE ONLY]`. A failure logs and leaves the messages alone. - **`(*Agent) summarizeText(...)`** — the only place compression talks to a provider, and it calls `recordCall`, so the summary's tokens and cost reach `UsageTotal`, the budgets and the status bar (they were invisible before). +- **`compressionPlan` / `(*Agent) planCompression(messages)`** — the partition (head, middle, pinned, tail) plus the token estimate and the threshold, computed **without a model call**. `compressHistory` executes this plan rather than re-deriving it, so `/compress preview` cannot disagree with the compression it previews. +- **`(*Agent) CompressionPreview()`** — the report behind `/compress preview`: whether automatic compression would fire, each kept region with its first lines and token count, and the middle's size. No provider call; refused while a run is active, because reading a slice the run appends to is still a data race. - **`(*Agent) HandleContextError(err error) bool`** — detects `context_length_exceeded`, lowers `discoveredCtxLen` (5 regex patterns, valid range 1024–10M, only lowers never raises) ### `context.go` — Turn-level Context Control @@ -871,7 +873,7 @@ User Input (textarea / CLI arg) ## Testing -- **1277 test functions + 10 fuzz targets** across the repository — what `go test ./... -count=1` runs for the main module, plus the evaluation corpora under `eval/` (nested modules the command still walks); each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks +- **1285 test functions + 10 fuzz targets** across the repository — what `go test ./... -count=1` runs for the main module, plus the evaluation corpora under `eval/` (nested modules the command still walks); each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks - `make fuzz` (`FUZZTIME=30s`) explores every fuzz target; `go test` already runs their seed corpora, so CI exercises them on every push - `make test-browser` (`BROWSER_TEST=1`, `-run TestBrowserSmoke`) renders a JavaScript page through both browser paths against a loopback server and asserts the request carried the proxy's `Via` header, which proves the filtering proxy was used; `TestBrowserSmokeRefusesABlockedSubresource` then serves a page whose own authority is exempted from the loopback rule while the policy stays enforced, and asserts a subresource pointing at a *second, live* loopback service is refused (the script's `onerror` marker in the DOM) and that the service was never reached — a refusal is proven, not only that the proxy was used (issue #8). Both skip without a browser, so `make test` never launches one - `make test-tui-visual` (`TUI_SHOT=1`) is the only place the TUI is looked at rather than asserted on: it renders the committed frame scenarios to PNGs through headless Chromium, starts `bin/tinycode` on a real 80x24 PTY (asserting the stream carried SGR styling and that the binary quits on the documented double Ctrl+C), runs it once more on a size-less PTY, and replays the live stream into a screen buffer that is screenshotted as `tinycode-pty-welcome-80x24.png`. Both tests skip without the variable, so `make test` needs neither a browser, nor a binary, nor a terminal device; PNGs land in `TUI_SHOT_DIR` (default `/tmp`) @@ -917,7 +919,7 @@ Counts in this document are produced by these commands at the commit they descri ```bash find . -name '*_test.go' -not -path './tuiprobe/*' -print0 \ - | xargs -0 grep -h '^func Test' | wc -l # 1277 test functions (main module) + | xargs -0 grep -h '^func Test' | wc -l # 1285 test functions (main module) grep -rn '^func Fuzz' --include=*_test.go . | wc -l # 10 fuzz targets ls tui/testdata/golden/frames/*.txt | wc -l # 29 plain-text frames ls tui/testdata/golden/ansi/*.ansi | wc -l # 1 raw ANSI frame diff --git a/README.md b/README.md index fdb62c0..f18b9cc 100644 --- a/README.md +++ b/README.md @@ -107,7 +107,7 @@ Custom **CellGrid** frame-buffer renders markdown directly in the terminal — n - **GitHub Actions**: Two workflows — main.yml (build + lint + test on push/PR) and release.yml (cross-compile + GitHub Releases on tags v*) - **main.yml jobs**: `ci` (build, gofmt gate, vet, tests, `-race`, repeated run), `lsp` (real gopls), `browser` (real Chromium), `tui-visual` (frame PNGs + the built binary on a PTY), `cross` (linux/amd64, linux/arm64, darwin/arm64 build + vet) and `staticcheck` - **Makefile improvements**: test target preserves exit code with pass/fail message; releases target cross-compiles all platforms + .tar.gz archives -- **1277 test functions + 10 fuzz targets** across the repository — the main module plus the evaluation corpora, with `tuiprobe/` a nested module the command below excludes — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement +- **1285 test functions + 10 fuzz targets** across the repository — the main module plus the evaluation corpora, with `tuiprobe/` a nested module the command below excludes — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement - **Annotations**: the jobs carry one `ubuntu-latest` migration notice each, plus the `setup-chrome@v1` Node 20 warning on `browser` and `tui-visual`; the gate is no *new* annotations, not zero ### Running the checks locally diff --git a/agent/compression.go b/agent/compression.go index f856fe0..3e6c1f4 100644 --- a/agent/compression.go +++ b/agent/compression.go @@ -132,18 +132,37 @@ func groupEnd(messages []types.Message, idx int) int { return len(messages) } -// compressHistory compresses a.History when it exceeds the threshold. It -// returns the history unchanged (and nil error) when there is nothing to -// compress, and a nil history plus the summarizer's error when the provider -// call fails or the context is cancelled. -func (a *Agent) compressHistory(ctx context.Context, messages []types.Message) ([]types.Message, error) { +// compressionPlan is the partition compressHistory makes, computed without a +// model call. The preview and the compression both read it, so they cannot +// disagree about what is kept (roadmap B5, S3). +type compressionPlan struct { + // Head and Tail are kept verbatim: the first two user groups and the last two. + Head []types.Message + Tail []types.Message + // Middle is the unpinned middle — what a summarizer would be given. + Middle []types.Message + // Pinned are middle groups the user asked to keep verbatim. + Pinned []types.Message + // Tokens is the history's estimated size and Threshold the bound that decides + // whether automatic compression fires. + Tokens int + Threshold int +} + +// OverThreshold reports whether automatic compression would fire. +func (p compressionPlan) OverThreshold() bool { + return p.Threshold > 0 && p.Tokens >= p.Threshold +} + +// planCompression works out what a compression of these messages would keep and +// what it would summarize. The second result is the reason there is no plan, so +// a caller can say why instead of "nothing to do". +func (a *Agent) planCompression(messages []types.Message) (compressionPlan, string) { if a.CompressionThreshold <= 0 || a.ContextLength <= 0 { - return messages, nil - } - estimatedTokens := EstimateMessagesTokens(messages) - if estimatedTokens < a.CompressionThreshold { - return messages, nil + return compressionPlan{}, "compression is disabled" } + plan := compressionPlan{Tokens: EstimateMessagesTokens(messages), Threshold: a.CompressionThreshold} + var userMsgIndices []int for i, m := range messages { if m.Role == types.RoleUser { @@ -151,7 +170,7 @@ func (a *Agent) compressHistory(ctx context.Context, messages []types.Message) ( } } if len(userMsgIndices) < 4 { - return messages, nil + return compressionPlan{}, "the history has fewer than four user turns" } protectFirst := 2 if protectFirst > len(userMsgIndices) { @@ -168,39 +187,46 @@ func (a *Agent) compressHistory(ctx context.Context, messages []types.Message) ( tailBudget := 2 tailStart := userMsgIndices[len(userMsgIndices)-tailBudget] if tailStart <= headEnd { - return messages, nil + return compressionPlan{}, "the head and the tail overlap: there is no middle to compress" } - head := messages[:headEnd] middle := messages[headEnd:tailStart] - tail := messages[tailStart:] if len(middle) < 2 { - return messages, nil + return compressionPlan{}, "the middle is too small to be worth a summary" } + plan.Head = messages[:headEnd] + plan.Tail = messages[tailStart:] // A pinned group is context the user asked to keep verbatim: it is never // fed to the summarizer, and it is put back after the summary and before the // tail. The order is therefore head, summary, pinned, recent tail — a // summary is already a lossy stand-in, and keeping the pinned text at all // matters more than where in the stand-in it sits (roadmap B5). - var ( - middleText strings.Builder - pinned []types.Message - ) for i := 0; i < len(middle); { end := groupEnd(middle, i) if groupPinned(middle[i:end]) { - pinned = append(pinned, middle[i:end]...) + plan.Pinned = append(plan.Pinned, middle[i:end]...) i = end continue } - middleText.WriteString(describeMessages(middle[i:end])) + plan.Middle = append(plan.Middle, middle[i:end]...) i = end } - if strings.TrimSpace(middleText.String()) == "" { - // Everything worth compressing is pinned. Compressing it would mean - // paying for a summary of nothing, so leave the history alone. + if len(plan.Middle) == 0 { + return compressionPlan{}, "every turn in the middle is pinned, so a summary would be of nothing" + } + return plan, "" +} + +// compressHistory compresses a.History when it exceeds the threshold. It +// returns the history unchanged (and nil error) when there is nothing to +// compress, and a nil history plus the summarizer's error when the provider +// call fails or the context is cancelled. +func (a *Agent) compressHistory(ctx context.Context, messages []types.Message) ([]types.Message, error) { + plan, _ := a.planCompression(messages) + if plan.Threshold == 0 || !plan.OverThreshold() { return messages, nil } + head, middleText, pinned, tail := plan.Head, describeMessages(plan.Middle), plan.Pinned, plan.Tail const instruction = `Summarize the following conversation history. Focus on: - What tasks were completed @@ -213,7 +239,7 @@ Conversation to summarize:` // A summarizer failure is reported rather than swallowed: the caller keeps // its history either way, but only this way can it tell a cancelled run // (context.Canceled) from a healthy no-op. - summary, err := a.summarizeText(ctx, instruction, middleText.String(), 1024, 0) + summary, err := a.summarizeText(ctx, instruction, middleText, 1024, 0) if err != nil { return nil, err } diff --git a/agent/compression_preview.go b/agent/compression_preview.go new file mode 100644 index 0000000..2a0e6c0 --- /dev/null +++ b/agent/compression_preview.go @@ -0,0 +1,73 @@ +package agent + +import ( + "fmt" + "strings" + "unicode/utf8" + + "github.com/yusiwen/tinycode/types" +) + +// previewLines bounds how many messages one section lists. The preview is meant +// to be read in a terminal, not to reproduce the history. +const previewLines = 12 + +// CompressionPreview describes what a compression of the current history would +// keep and what it would summarize, without calling a model (roadmap B5, S3). +// +// It reads the same partition compressHistory uses — planCompression — so the +// preview cannot disagree with the thing it previews about which turns are kept. +func (a *Agent) CompressionPreview() string { + plan, reason := a.planCompression(a.History) + if plan.Threshold == 0 { + return "Compression preview: nothing to show — " + reason + "." + } + var b strings.Builder + state := "would not fire (under the threshold)" + if plan.OverThreshold() { + state = "would fire" + } + fmt.Fprintf(&b, "Compression preview (no model call): %d messages, ~%d tokens, threshold %d — automatic compression %s.\n", + len(a.History), plan.Tokens, plan.Threshold, state) + if reason != "" { + fmt.Fprintf(&b, "Nothing would be compressed: %s.\n", reason) + return b.String() + } + writePreviewSection(&b, "kept: head", plan.Head) + fmt.Fprintf(&b, "summarized: middle — %d messages, ~%d tokens\n", + len(plan.Middle), EstimateMessagesTokens(plan.Middle)) + writePreviewSection(&b, "kept: pinned", plan.Pinned) + writePreviewSection(&b, "kept: tail", plan.Tail) + return b.String() +} + +// writePreviewSection lists one kept region: its size and the first line of each +// message, capped so the preview stays a glance. +func writePreviewSection(b *strings.Builder, label string, messages []types.Message) { + if len(messages) == 0 { + return + } + fmt.Fprintf(b, "%s — %d messages, ~%d tokens\n", label, len(messages), EstimateMessagesTokens(messages)) + for i, msg := range messages { + if i == previewLines { + fmt.Fprintf(b, " … (%d more)\n", len(messages)-i) + break + } + fmt.Fprintf(b, " %s: %s\n", msg.Role, firstLine(msg.Content, 60)) + } +} + +// firstLine returns the first non-empty line of text, clipped to max runes. +func firstLine(text string, max int) string { + trimmed := strings.TrimSpace(text) + for _, line := range strings.Split(trimmed, "\n") { + if line = strings.TrimSpace(line); line != "" { + if utf8.RuneCountInString(line) > max { + runes := []rune(line) + return string(runes[:max]) + "…" + } + return line + } + } + return "(no text)" +} diff --git a/agent/compression_preview_test.go b/agent/compression_preview_test.go new file mode 100644 index 0000000..965c23a --- /dev/null +++ b/agent/compression_preview_test.go @@ -0,0 +1,157 @@ +package agent + +import ( + "context" + "strings" + "testing" +) + +// previewHistory is seven turns with distinctive text, the fourth pinned: with +// this shape the head is turns 1-3, the middle is turns 4-5 (one pinned, one +// not), and the tail is turns 6-7. That mix is what a preview has to report. +func previewHistory(t *testing.T, pinned bool) *Agent { + t.Helper() + ag := New(&recordingProvider{content: "SUMMARY"}) + ag.ContextLength = 1000 + ag.CompressionThreshold = 1 + ag.History = turnHistory(7) + if pinned { + ag.History[6].Pinned = true + ag.History[7].Pinned = true + } + return ag +} + +// TestCompressionPreviewCallsNoModel is the acceptance that matters most: a +// preview that asks a provider is not a preview, it is a compression. +func TestCompressionPreviewCallsNoModel(t *testing.T) { + provider := &recordingProvider{content: "SUMMARY"} + ag := New(provider) + ag.ContextLength = 1000 + ag.CompressionThreshold = 1 + ag.History = turnHistory(6) + + if out := ag.CompressionPreview(); !strings.Contains(out, "Compression preview") { + t.Fatalf("preview = %q", out) + } + if len(provider.requests) != 0 { + t.Fatalf("the preview called the provider %d times", len(provider.requests)) + } +} + +// TestCompressionPreviewNamesGroups pins the report's shape: each kept region is +// named with its messages' first lines, and the middle is reported by size. +func TestCompressionPreviewNamesGroups(t *testing.T) { + ag := previewHistory(t, true) + + out := ag.CompressionPreview() + for _, want := range []string{ + "kept: head", "kept: pinned", "kept: tail", + "summarized: middle", + "user: question 0", // head + "assistant: answer 2", // head + "user: question 3", // pinned (turn 4) + "assistant: answer 3", + "user: question 5", // tail + } { + if !strings.Contains(out, want) { + t.Errorf("preview is missing %q:\n%s", want, out) + } + } + if !strings.Contains(out, "middle — 2 messages") { + t.Errorf("the middle should report the two messages it would summarize:\n%s", out) + } +} + +// TestCompressionPreviewSaysWhyNothingHappens: a history that cannot be +// compressed says why, instead of printing empty sections. +func TestCompressionPreviewSaysWhyNothingHappens(t *testing.T) { + ag := New(&recordingProvider{content: "SUMMARY"}) + ag.ContextLength = 1000 + ag.CompressionThreshold = 1 + ag.History = turnHistory(3) + + out := ag.CompressionPreview() + if !strings.Contains(out, "fewer than four user turns") { + t.Fatalf("preview = %q", out) + } + + ag.CompressionThreshold = 0 + if out := ag.CompressionPreview(); !strings.Contains(out, "compression is disabled") { + t.Fatalf("disabled preview = %q", out) + } +} + +// TestCompressionPreviewMatchesTheCompression is the agreement the acceptance +// asks for: the plan the preview reads is the plan the compression executes. +func TestCompressionPreviewMatchesTheCompression(t *testing.T) { + ag := previewHistory(t, true) + plan, reason := ag.planCompression(ag.History) + if reason != "" { + t.Fatalf("no plan: %s", reason) + } + + compressed, err := ag.compressHistory(context.Background(), ag.History) + if err != nil { + t.Fatalf("compress: %v", err) + } + if compressed == nil { + t.Fatal("nothing was compressed") + } + // The output is head, summary, pinned, tail — exactly the plan's partition. + if len(compressed) != len(plan.Head)+1+len(plan.Pinned)+len(plan.Tail) { + t.Fatalf("compressed has %d messages, the plan says %d", len(compressed), len(plan.Head)+1+len(plan.Pinned)+len(plan.Tail)) + } + for i, msg := range plan.Head { + if compressed[i].Content != msg.Content { + t.Fatalf("head message %d differs", i) + } + } + summary := compressed[len(plan.Head)] + if !strings.Contains(summary.Content, "SUMMARY") { + t.Fatalf("the summary is not where the plan says: %+v", summary) + } + offset := len(plan.Head) + 1 + for i, msg := range plan.Pinned { + if compressed[offset+i].Content != msg.Content { + t.Fatalf("pinned message %d differs", i) + } + } + offset += len(plan.Pinned) + for i, msg := range plan.Tail { + if compressed[offset+i].Content != msg.Content { + t.Fatalf("tail message %d differs", i) + } + } +} + +// TestCompressionPreviewUnderAndOverTheThreshold pins the one thing the report +// says about automatic compression: whether it would fire. +func TestCompressionPreviewUnderAndOverTheThreshold(t *testing.T) { + ag := previewHistory(t, false) + if out := ag.CompressionPreview(); !strings.Contains(out, "would fire") { + t.Fatalf("over the threshold, preview says:\n%s", out) + } + ag.CompressionThreshold = 1000000 + if out := ag.CompressionPreview(); !strings.Contains(out, "would not fire") { + t.Fatalf("under the threshold, preview says:\n%s", out) + } + // The partition is still reported under the threshold: the question "what + // would be kept" has an answer even when nothing would run. + if out := ag.CompressionPreview(); !strings.Contains(out, "kept: head") { + t.Fatalf("under the threshold, the partition is missing:\n%s", out) + } +} + +// TestFirstLine pins the clipping the sections rely on. +func TestFirstLine(t *testing.T) { + if got := firstLine("\n\n hello world\nsecond", 60); got != "hello world" { + t.Fatalf("firstLine = %q", got) + } + if got := firstLine("", 60); got != "(no text)" { + t.Fatalf("empty firstLine = %q", got) + } + if got := firstLine(strings.Repeat("x", 70), 10); got != strings.Repeat("x", 10)+"…" { + t.Fatalf("clipped firstLine = %q", got) + } +} diff --git a/docs/roadmap.md b/docs/roadmap.md index d359777..c58a295 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -393,9 +393,9 @@ current tree. D7 is the exception: it was recorded later, so its references are - **Picked up**: [#306](https://github.com/yusiwen/TinyCode/issues/306) — S1 (turn-level `/pin`, `/unpin` and `/drop`, and a compressor that keeps pinned groups verbatim), S2 (compaction of the run's own messages — the part D7 depends on — the summarizer's spend now counted, and pins carried through - the session file), S3 (a compression preview that calls no model), S4 (a path-targeted unload, split out - of S2 because the command channel it assumed does not exist). **S1 and S2 landed 2026-10-28**; S3 and - S4 are open. + the session file), S3 (a compression preview that calls no model — it reads the same partition + `compressHistory` executes), S4 (a path-targeted unload, split out of S2 because the command channel it + assumed does not exist). **S1–S3 landed** (2026-10-28, 2026-10-31); S4 is open. - **Gap**: compression is automatic and all-or-nothing. No `/drop`, no `/pin`, no unloading by message or by file, and no preview of what a compression would discard. - **Evidence**: one strategy, `compressHistory` (`agent/compression.go:59`) — head, LLM-summarized diff --git a/tui/context_command_test.go b/tui/context_command_test.go index de67803..159f1c9 100644 --- a/tui/context_command_test.go +++ b/tui/context_command_test.go @@ -184,3 +184,48 @@ func TestPinSurvivesTheSessionFile(t *testing.T) { t.Fatalf("historyFromSession lost the pin: %+v", restored) } } + +// TestCompressPreviewCommand prints the partition without compressing: the +// history must be untouched and the output must name the kept regions. +func TestCompressPreviewCommand(t *testing.T) { + m := newTestTUI() + m.agent.ContextLength = 1000 + m.agent.CompressionThreshold = 1 + seedHistory(m, 6) + before := len(m.agent.History) + + m.handleCommand("/compress preview") + + reply := lastMessage(m) + for _, want := range []string{"Compression preview", "kept: head", "summarized: middle", "kept: tail"} { + if !strings.Contains(reply, want) { + t.Fatalf("reply is missing %q:\n%s", want, reply) + } + } + if got := len(m.agent.History); got != before { + t.Fatalf("the preview changed the history: %d -> %d", before, got) + } +} + +// TestCompressPreviewRefusedDuringARun pins the guard: reading the history while +// the run goroutine appends to it is a data race even though nothing is written. +func TestCompressPreviewRefusedDuringARun(t *testing.T) { + m := newTestTUI() + m.agent.ContextLength = 1000 + m.agent.CompressionThreshold = 1 + seedHistory(m, 6) + m.runMu.Lock() + m.runActive = true + m.runMu.Unlock() + t.Cleanup(func() { + m.runMu.Lock() + m.runActive = false + m.runMu.Unlock() + }) + + m.handleCommand("/compress preview") + + if !strings.Contains(m.statusMsg, "a run is in progress") { + t.Fatalf("status = %q", m.statusMsg) + } +} diff --git a/tui/update.go b/tui/update.go index 5ea0b49..b715c9a 100644 --- a/tui/update.go +++ b/tui/update.go @@ -1166,6 +1166,17 @@ func (m *TuiModel) handleCommand(cmd string) (tea.Model, tea.Cmd) { m.ShowStatus("No agent available") return m, nil } + // `/compress preview` reads the history without replacing it. It is still + // refused while a run is active: the run goroutine appends to that slice, + // and reading it concurrently is a data race even when nothing is written. + if strings.TrimSpace(strings.TrimPrefix(cmd, "/compress")) == "preview" { + if m.runIsActive() { + m.ShowStatus("Cannot preview: a run is in progress") + return m, nil + } + m.appendSystem(m.agent.CompressionPreview()) + return m, nil + } // CompressHistory replaces a.History while the agent goroutine may be // appending to it, so it must never run concurrently with an active // run. Refusing while runIsActive() is sufficient: the run lifecycle