feat(agent): truncate tool output to a token budget, with a tail and a summary

CODEBASE.md

@@ -1,6 +1,6 @@
 # TinyCode — CODEBASE Map
 
-> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1230 test functions + 10 fuzz targets, race-detector clean.
+> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1237 test functions + 10 fuzz targets, race-detector clean.
 
 ## Quick Reference
 
@@ -308,10 +308,18 @@ What runs today, in order:
 
 ### `truncate.go` — Output Truncation
 
-- **`TruncationResult`** struct: `Content string` (preview + hint), `FullPath string`
-- **`TruncateOutput(output string) TruncationResult`** — limits come from `TruncMaxLines`/`TruncMaxBytes`/`truncDir`, overridable at startup with `SetTruncationConfig(maxLines, maxBytes, outputDir)`; the saved full output is written 0600 because tool output can contain secrets
-- Constants: `TruncMaxLines = 2000`, `TruncMaxBytes = 200KB`, `truncDir = "/tmp/tinycode/truncated"`
-- Saves full output to file, returns head preview with `[TRUNCATED]` hint
+- **`TruncationResult`** struct: `Content string` (preview + hint), `FullPath string`, `OmittedLines int`
+- **`TruncateOutput(output string) TruncationResult`** — the preview keeps the **head**, the **tail** and a
+  count of what fell between them: `... 18,431 of 20,000 lines omitted (4 mention an error, 9 a warning) ...`,
+  where the counts are a deterministic scan (`truncErrorSignal` / `truncWarningSignal`), never a model call.
+  A head-only slice lost the panic at the end, and a hint that said only "some bytes were dropped" could
+  not tell a reader whether the dropped part mattered.
+- Bounds: `TruncMaxLines` (2000), `TruncMaxBytes` (200KB), `TruncMaxTokens` (4000 — the bound that matters
+  once the text is dense; `bytesPerToken` is `EstimateTokens`' inverse and a test pins the pair) and
+  `TruncTailLines` (20). The token budget covers the whole preview: the tail is taken first, the head gets
+  the rest. Overridable at startup with
+  `SetTruncationConfig(maxLines, maxBytes, maxTokens, tailLines, outputDir)`; the saved full output is
+  written 0600 because tool output can contain secrets.
 
 ---
 
@@ -850,7 +858,7 @@ User Input (textarea / CLI arg)
 
 ## Testing
 
-- **1230 test functions + 10 fuzz targets** in the main module — what `go test ./... -count=1` runs, since `tuiprobe/` is a nested module; each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks
+- **1237 test functions + 10 fuzz targets** in the main module — what `go test ./... -count=1` runs, since `tuiprobe/` is a nested module; each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks
 - `make fuzz` (`FUZZTIME=30s`) explores every fuzz target; `go test` already runs their seed corpora, so CI exercises them on every push
 - `make test-browser` (`BROWSER_TEST=1`, `-run TestBrowserSmoke`) renders a JavaScript page through both browser paths against a loopback server and asserts the request carried the proxy's `Via` header, which proves the filtering proxy was used; `TestBrowserSmokeRefusesABlockedSubresource` then serves a page whose own authority is exempted from the loopback rule while the policy stays enforced, and asserts a subresource pointing at a *second, live* loopback service is refused (the script's `onerror` marker in the DOM) and that the service was never reached — a refusal is proven, not only that the proxy was used (issue #8). Both skip without a browser, so `make test` never launches one
 - `make test-tui-visual` (`TUI_SHOT=1`) is the only place the TUI is looked at rather than asserted on: it renders the committed frame scenarios to PNGs through headless Chromium, starts `bin/tinycode` on a real 80x24 PTY (asserting the stream carried SGR styling and that the binary quits on the documented double Ctrl+C), runs it once more on a size-less PTY, and replays the live stream into a screen buffer that is screenshotted as `tinycode-pty-welcome-80x24.png`. Both tests skip without the variable, so `make test` needs neither a browser, nor a binary, nor a terminal device; PNGs land in `TUI_SHOT_DIR` (default `/tmp`)
@@ -896,7 +904,7 @@ Counts in this document are produced by these commands at the commit they descri
 
 ```bash
 find . -name '*_test.go' -not -path './tuiprobe/*' -print0 \
-  | xargs -0 grep -h '^func Test' | wc -l              # 1230 test functions (main module)
+  | xargs -0 grep -h '^func Test' | wc -l              # 1237 test functions (main module)
 grep -rn '^func Fuzz' --include=*_test.go . | wc -l    # 10 fuzz targets
 ls tui/testdata/golden/frames/*.txt | wc -l            # 29 plain-text frames
 ls tui/testdata/golden/ansi/*.ansi | wc -l             # 1 raw ANSI frame

README.md

@@ -107,7 +107,7 @@ Custom **CellGrid** frame-buffer renders markdown directly in the terminal — n
 - **GitHub Actions**: Two workflows — main.yml (build + lint + test on push/PR) and release.yml (cross-compile + GitHub Releases on tags v*)
 - **main.yml jobs**: `ci` (build, gofmt gate, vet, tests, `-race`, repeated run), `lsp` (real gopls), `browser` (real Chromium), `tui-visual` (frame PNGs + the built binary on a PTY), `cross` (linux/amd64, linux/arm64, darwin/arm64 build + vet) and `staticcheck`
 - **Makefile improvements**: test target preserves exit code with pass/fail message; releases target cross-compiles all platforms + .tar.gz archives
-- **1230 test functions + 10 fuzz targets** in the main module — the one `go test ./... -count=1` runs, since `tuiprobe/` is a nested module — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement
+- **1237 test functions + 10 fuzz targets** in the main module — the one `go test ./... -count=1` runs, since `tuiprobe/` is a nested module — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement
 - **Annotations**: the jobs carry one `ubuntu-latest` migration notice each, plus the `setup-chrome@v1` Node 20 warning on `browser` and `tui-visual`; the gate is no *new* annotations, not zero
 
 ### Running the checks locally

agent/truncate.go

@@ -5,8 +5,10 @@ import (
 	"math/rand"
 	"os"
 	"path/filepath"
+	"regexp"
 	"strings"
 	"time"
+	"unicode/utf8"
 
 	"github.com/yusiwen/tinycode/tlog"
 )
@@ -21,20 +23,51 @@ var (
 	// have their own internal truncation (e.g. bash, code_review).
 	TruncMaxBytes = 200 * 1024 // 200KB
 
+	// TruncMaxTokens bounds the preview by what the model actually pays for.
+	// It is the bound that matters once text is dense: 200 KiB of it is ~50k
+	// tokens, a whole context window spent on one tool result. EstimateTokens
+	// is the estimator; bytesPerToken below is its inverse and a test keeps the
+	// two in step.
+	TruncMaxTokens = 4000
+
+	// TruncTailLines is how many of the output's last lines the preview keeps
+	// beside the head. A failure is usually in the last lines, and a head-only
+	// preview drops exactly the evidence a reader needs.
+	TruncTailLines = 20
+
 	// truncDir is where full truncated output files are saved.
 	truncDir = "/tmp/tinycode/truncated"
 )
 
+// bytesPerToken is the estimator's ratio: EstimateTokens reports len(text)/4.
+// The head loop counts bytes because that is O(1) per line; a test pins this
+// inverse to the estimator so the two cannot drift.
+const bytesPerToken = 4
+
+// The two signals the omitted-lines summary counts. They are matched, not
+// summarised: the count has to be cheap, deterministic and explainable, and a
+// model call would make an already-large tool result cost another turn.
+var (
+	truncErrorSignal   = regexp.MustCompile(`(?i)\b(error|failed|failure|panic|fatal|denied)\b`)
+	truncWarningSignal = regexp.MustCompile(`(?i)\bwarn(ing)?\b`)
+)
+
 // SetTruncationConfig overrides the output-truncation limits from the
 // configuration file. Zero or negative values keep the current setting. Call it
 // during startup, before any tool runs.
-func SetTruncationConfig(maxLines, maxBytes int, outputDir string) {
+func SetTruncationConfig(maxLines, maxBytes, maxTokens, tailLines int, outputDir string) {
 	if maxLines > 0 {
 		TruncMaxLines = maxLines
 	}
 	if maxBytes > 0 {
 		TruncMaxBytes = maxBytes
 	}
+	if maxTokens > 0 {
+		TruncMaxTokens = maxTokens
+	}
+	if tailLines > 0 {
+		TruncTailLines = tailLines
+	}
 	if outputDir != "" {
 		truncDir = outputDir
 	}
@@ -46,14 +79,26 @@ type TruncationResult struct {
 	Content string
 	// FullPath is the path to the saved full output, empty if not truncated.
 	FullPath string
+	// OmittedLines is how many lines did not fit the preview. Zero when the
+	// output fit.
+	OmittedLines int
 }
 
-// TruncateOutput truncates tool output that exceeds limits.
-// If truncation occurs, the full output is saved to a unique file and the
-// returned Content contains a preview plus instructions for reading the rest.
+// TruncateOutput truncates tool output that exceeds its bounds.
+//
+// The preview keeps the head, the tail, and a count of what fell between them:
+// a head-only slice loses the panic at the end, and a hint that says only "some
+// bytes were dropped" cannot tell a reader whether the dropped part mattered.
+// The full output is still written to a file, so nothing is lost.
 func TruncateOutput(output string) TruncationResult {
 	lines := strings.Split(output, "\n")
-	if len(lines) <= TruncMaxLines && len(output) <= TruncMaxBytes {
+	// A trailing newline leaves an empty final element; it is an artefact of the
+	// split, not a line, and counting it would make every "N lines" number one
+	// too high.
+	if n := len(lines); n > 0 && lines[n-1] == "" && strings.HasSuffix(output, "\n") {
+		lines = lines[:n-1]
+	}
+	if len(lines) <= TruncMaxLines && len(output) <= TruncMaxBytes && EstimateTokens(output) <= TruncMaxTokens {
 		return TruncationResult{Content: output}
 	}
 
@@ -66,36 +111,140 @@ func TruncateOutput(output string) TruncationResult {
 		return TruncationResult{Content: output}
 	}
 
-	tlog.Debug("truncate", "truncated", "file", filePath, "total_lines", len(lines), "total_bytes", len(output))
+	tlog.Debug("truncate", "truncated", "file", filePath, "total_lines", len(lines),
+		"total_bytes", len(output), "total_tokens", EstimateTokens(output))
+
+	// The tail comes first so the budget covers the whole preview: what the end
+	// needs is not negotiable, and the head gets what is left of it.
+	tailText, tailCount := truncTail(lines, TruncTailLines)
+	headBudget := TruncMaxTokens - EstimateTokens(tailText)
+	if headBudget < 0 {
+		headBudget = 0
+	}
+	headLimit := TruncMaxLines
+	if rest := len(lines) - tailCount; rest < headLimit {
+		headLimit = rest
+	}
+	headText, headCount := truncHead(lines, headLimit, headBudget)
+
+	omitted := lines[headCount : len(lines)-tailCount]
+	errs, warns := truncSignals(omitted)
 
-	// Build head-only preview respecting both limits
 	var preview strings.Builder
-	bytesAccum := 0
-	lineLimit := TruncMaxLines
-	if len(lines) < lineLimit {
-		lineLimit = len(lines)
-	}
-	for i := 0; i < lineLimit; i++ {
-		lineSize := len(lines[i]) + 1 // +1 for the newline
-		if bytesAccum+lineSize > TruncMaxBytes {
+	preview.WriteString(headText)
+	if len(omitted) > 0 {
+		fmt.Fprintf(&preview, "\n... %d of %d lines omitted (%d mention an error, %d a warning) ...\n\n",
+			len(omitted), len(lines), errs, warns)
+	}
+	preview.WriteString(tailText)
+
+	fmt.Fprintf(&preview,
+		"\n[TRUNCATED] Full output saved to: %s\n"+
+			"Use read_file with offset/limit to view specific sections.\n", filePath)
+	return TruncationResult{
+		Content:      preview.String(),
+		FullPath:     filePath,
+		OmittedLines: len(omitted),
+	}
+}
+
+// truncHead keeps the first lines that fit the byte and token budgets. The first
+// line is always kept, clipped if it alone is too large: an output whose first
+// line is enormous must still yield a preview that starts with it.
+func truncHead(lines []string, limit, tokenBudget int) (string, int) {
+	if limit > len(lines) {
+		limit = len(lines)
+	}
+	if limit <= 0 || tokenBudget <= 0 {
+		return "", 0
+	}
+	byteBudget := tokenBudget * bytesPerToken
+	if byteBudget > TruncMaxBytes {
+		byteBudget = TruncMaxBytes
+	}
+	var b strings.Builder
+	used := 0
+	count := 0
+	for i := 0; i < limit; i++ {
+		line := lines[i]
+		size := len(line) + 1
+		if used > 0 && used+size > byteBudget {
 			break
 		}
-		preview.WriteString(lines[i])
-		preview.WriteString("\n")
-		bytesAccum += lineSize
-	}
-
-	totalBytes := len(output)
-	removedBytes := totalBytes - bytesAccum
-
-	hint := fmt.Sprintf(
-		"\n... (%d bytes truncated) ...\n\n"+
-			"[TRUNCATED] Full output saved to: %s\n"+
-			"Use read_file with offset/limit to view specific sections.\n",
-		removedBytes, filePath,
-	)
-	preview.WriteString(hint)
-	return TruncationResult{Content: preview.String(), FullPath: filePath}
+		if used == 0 {
+			line = clipBytes(line, byteBudget)
+			size = len(line) + 1
+		}
+		b.WriteString(line)
+		b.WriteByte('\n')
+		used += size
+		count++
+	}
+	return b.String(), count
+}
+
+// truncTail keeps the last lines, bounded by the byte cap so one enormous last
+// line cannot smuggle the whole output back in.
+func truncTail(lines []string, want int) (string, int) {
+	if want <= 0 || len(lines) == 0 {
+		return "", 0
+	}
+	if want > len(lines) {
+		want = len(lines)
+	}
+	var kept []string
+	used := 0
+	for i := len(lines) - 1; i >= 0 && len(kept) < want; i-- {
+		size := len(lines[i]) + 1
+		if len(kept) > 0 && used+size > TruncMaxBytes {
+			break
+		}
+		line := lines[i]
+		if len(kept) == 0 {
+			line = clipBytes(line, TruncMaxBytes)
+			size = len(line) + 1
+		}
+		kept = append(kept, line)
+		used += size
+	}
+	for i, j := 0, len(kept)-1; i < j; i, j = i+1, j-1 {
+		kept[i], kept[j] = kept[j], kept[i]
+	}
+	if len(kept) == 0 {
+		return "", 0
+	}
+	return strings.Join(kept, "\n") + "\n", len(kept)
+}
+
+// truncSignals counts the omitted lines that mention a failure or a warning. An
+// error line is not also counted as a warning.
+func truncSignals(lines []string) (errs, warns int) {
+	for _, line := range lines {
+		if truncErrorSignal.MatchString(line) {
+			errs++
+			continue
+		}
+		if truncWarningSignal.MatchString(line) {
+			warns++
+		}
+	}
+	return errs, warns
+}
+
+// clipBytes cuts a string to at most limit bytes on a rune boundary, so a
+// multi-byte character is never split in half.
+func clipBytes(text string, limit int) string {
+	if limit <= 0 {
+		return ""
+	}
+	if len(text) <= limit {
+		return text
+	}
+	cut := limit
+	for cut > 0 && !utf8.RuneStart(text[cut]) {
+		cut--
+	}
+	return text[:cut]
 }
 
 // uniqueToolFile generates a unique file path for saving truncated tool output.