diff --git a/CODEBASE.md b/CODEBASE.md index f7babd7..08c61d2 100644 --- a/CODEBASE.md +++ b/CODEBASE.md @@ -1,6 +1,6 @@ # TinyCode — CODEBASE Map -> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1230 test functions + 10 fuzz targets, race-detector clean. +> AI coding agent in pure Go. Single binary, Bubble Tea TUI, ReAct agent loop, 25 built-in tools + MCP, LSP diagnostics, session persistence. 1237 test functions + 10 fuzz targets, race-detector clean. ## Quick Reference @@ -308,10 +308,18 @@ What runs today, in order: ### `truncate.go` — Output Truncation -- **`TruncationResult`** struct: `Content string` (preview + hint), `FullPath string` -- **`TruncateOutput(output string) TruncationResult`** — limits come from `TruncMaxLines`/`TruncMaxBytes`/`truncDir`, overridable at startup with `SetTruncationConfig(maxLines, maxBytes, outputDir)`; the saved full output is written 0600 because tool output can contain secrets -- Constants: `TruncMaxLines = 2000`, `TruncMaxBytes = 200KB`, `truncDir = "/tmp/tinycode/truncated"` -- Saves full output to file, returns head preview with `[TRUNCATED]` hint +- **`TruncationResult`** struct: `Content string` (preview + hint), `FullPath string`, `OmittedLines int` +- **`TruncateOutput(output string) TruncationResult`** — the preview keeps the **head**, the **tail** and a + count of what fell between them: `... 18,431 of 20,000 lines omitted (4 mention an error, 9 a warning) ...`, + where the counts are a deterministic scan (`truncErrorSignal` / `truncWarningSignal`), never a model call. + A head-only slice lost the panic at the end, and a hint that said only "some bytes were dropped" could + not tell a reader whether the dropped part mattered. +- Bounds: `TruncMaxLines` (2000), `TruncMaxBytes` (200KB), `TruncMaxTokens` (4000 — the bound that matters + once the text is dense; `bytesPerToken` is `EstimateTokens`' inverse and a test pins the pair) and + `TruncTailLines` (20). The token budget covers the whole preview: the tail is taken first, the head gets + the rest. Overridable at startup with + `SetTruncationConfig(maxLines, maxBytes, maxTokens, tailLines, outputDir)`; the saved full output is + written 0600 because tool output can contain secrets. --- @@ -850,7 +858,7 @@ User Input (textarea / CLI arg) ## Testing -- **1230 test functions + 10 fuzz targets** in the main module — what `go test ./... -count=1` runs, since `tuiprobe/` is a nested module; each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks +- **1237 test functions + 10 fuzz targets** in the main module — what `go test ./... -count=1` runs, since `tuiprobe/` is a nested module; each count below is listed with the command that produces it, and no count is carried in a copy that nothing checks - `make fuzz` (`FUZZTIME=30s`) explores every fuzz target; `go test` already runs their seed corpora, so CI exercises them on every push - `make test-browser` (`BROWSER_TEST=1`, `-run TestBrowserSmoke`) renders a JavaScript page through both browser paths against a loopback server and asserts the request carried the proxy's `Via` header, which proves the filtering proxy was used; `TestBrowserSmokeRefusesABlockedSubresource` then serves a page whose own authority is exempted from the loopback rule while the policy stays enforced, and asserts a subresource pointing at a *second, live* loopback service is refused (the script's `onerror` marker in the DOM) and that the service was never reached — a refusal is proven, not only that the proxy was used (issue #8). Both skip without a browser, so `make test` never launches one - `make test-tui-visual` (`TUI_SHOT=1`) is the only place the TUI is looked at rather than asserted on: it renders the committed frame scenarios to PNGs through headless Chromium, starts `bin/tinycode` on a real 80x24 PTY (asserting the stream carried SGR styling and that the binary quits on the documented double Ctrl+C), runs it once more on a size-less PTY, and replays the live stream into a screen buffer that is screenshotted as `tinycode-pty-welcome-80x24.png`. Both tests skip without the variable, so `make test` needs neither a browser, nor a binary, nor a terminal device; PNGs land in `TUI_SHOT_DIR` (default `/tmp`) @@ -896,7 +904,7 @@ Counts in this document are produced by these commands at the commit they descri ```bash find . -name '*_test.go' -not -path './tuiprobe/*' -print0 \ - | xargs -0 grep -h '^func Test' | wc -l # 1230 test functions (main module) + | xargs -0 grep -h '^func Test' | wc -l # 1237 test functions (main module) grep -rn '^func Fuzz' --include=*_test.go . | wc -l # 10 fuzz targets ls tui/testdata/golden/frames/*.txt | wc -l # 29 plain-text frames ls tui/testdata/golden/ansi/*.ansi | wc -l # 1 raw ANSI frame diff --git a/README.md b/README.md index e8f1c9e..ac36290 100644 --- a/README.md +++ b/README.md @@ -107,7 +107,7 @@ Custom **CellGrid** frame-buffer renders markdown directly in the terminal — n - **GitHub Actions**: Two workflows — main.yml (build + lint + test on push/PR) and release.yml (cross-compile + GitHub Releases on tags v*) - **main.yml jobs**: `ci` (build, gofmt gate, vet, tests, `-race`, repeated run), `lsp` (real gopls), `browser` (real Chromium), `tui-visual` (frame PNGs + the built binary on a PTY), `cross` (linux/amd64, linux/arm64, darwin/arm64 build + vet) and `staticcheck` - **Makefile improvements**: test target preserves exit code with pass/fail message; releases target cross-compiles all platforms + .tar.gz archives -- **1230 test functions + 10 fuzz targets** in the main module — the one `go test ./... -count=1` runs, since `tuiprobe/` is a nested module — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement +- **1237 test functions + 10 fuzz targets** in the main module — the one `go test ./... -count=1` runs, since `tuiprobe/` is a nested module — counted with `find . -name '*_test.go' -not -path './tuiprobe/*' -print0 | xargs -0 grep -h '^func Test' | wc -l` and `grep -rn '^func Fuzz' --include=*_test.go . | wc -l`; the command is part of the record, and `CODEBASE.md` → Testing lists every count with its measurement - **Annotations**: the jobs carry one `ubuntu-latest` migration notice each, plus the `setup-chrome@v1` Node 20 warning on `browser` and `tui-visual`; the gate is no *new* annotations, not zero ### Running the checks locally diff --git a/agent/truncate.go b/agent/truncate.go index 124d774..69cf513 100644 --- a/agent/truncate.go +++ b/agent/truncate.go @@ -5,8 +5,10 @@ import ( "math/rand" "os" "path/filepath" + "regexp" "strings" "time" + "unicode/utf8" "github.com/yusiwen/tinycode/tlog" ) @@ -21,20 +23,51 @@ var ( // have their own internal truncation (e.g. bash, code_review). TruncMaxBytes = 200 * 1024 // 200KB + // TruncMaxTokens bounds the preview by what the model actually pays for. + // It is the bound that matters once text is dense: 200 KiB of it is ~50k + // tokens, a whole context window spent on one tool result. EstimateTokens + // is the estimator; bytesPerToken below is its inverse and a test keeps the + // two in step. + TruncMaxTokens = 4000 + + // TruncTailLines is how many of the output's last lines the preview keeps + // beside the head. A failure is usually in the last lines, and a head-only + // preview drops exactly the evidence a reader needs. + TruncTailLines = 20 + // truncDir is where full truncated output files are saved. truncDir = "/tmp/tinycode/truncated" ) +// bytesPerToken is the estimator's ratio: EstimateTokens reports len(text)/4. +// The head loop counts bytes because that is O(1) per line; a test pins this +// inverse to the estimator so the two cannot drift. +const bytesPerToken = 4 + +// The two signals the omitted-lines summary counts. They are matched, not +// summarised: the count has to be cheap, deterministic and explainable, and a +// model call would make an already-large tool result cost another turn. +var ( + truncErrorSignal = regexp.MustCompile(`(?i)\b(error|failed|failure|panic|fatal|denied)\b`) + truncWarningSignal = regexp.MustCompile(`(?i)\bwarn(ing)?\b`) +) + // SetTruncationConfig overrides the output-truncation limits from the // configuration file. Zero or negative values keep the current setting. Call it // during startup, before any tool runs. -func SetTruncationConfig(maxLines, maxBytes int, outputDir string) { +func SetTruncationConfig(maxLines, maxBytes, maxTokens, tailLines int, outputDir string) { if maxLines > 0 { TruncMaxLines = maxLines } if maxBytes > 0 { TruncMaxBytes = maxBytes } + if maxTokens > 0 { + TruncMaxTokens = maxTokens + } + if tailLines > 0 { + TruncTailLines = tailLines + } if outputDir != "" { truncDir = outputDir } @@ -46,14 +79,26 @@ type TruncationResult struct { Content string // FullPath is the path to the saved full output, empty if not truncated. FullPath string + // OmittedLines is how many lines did not fit the preview. Zero when the + // output fit. + OmittedLines int } -// TruncateOutput truncates tool output that exceeds limits. -// If truncation occurs, the full output is saved to a unique file and the -// returned Content contains a preview plus instructions for reading the rest. +// TruncateOutput truncates tool output that exceeds its bounds. +// +// The preview keeps the head, the tail, and a count of what fell between them: +// a head-only slice loses the panic at the end, and a hint that says only "some +// bytes were dropped" cannot tell a reader whether the dropped part mattered. +// The full output is still written to a file, so nothing is lost. func TruncateOutput(output string) TruncationResult { lines := strings.Split(output, "\n") - if len(lines) <= TruncMaxLines && len(output) <= TruncMaxBytes { + // A trailing newline leaves an empty final element; it is an artefact of the + // split, not a line, and counting it would make every "N lines" number one + // too high. + if n := len(lines); n > 0 && lines[n-1] == "" && strings.HasSuffix(output, "\n") { + lines = lines[:n-1] + } + if len(lines) <= TruncMaxLines && len(output) <= TruncMaxBytes && EstimateTokens(output) <= TruncMaxTokens { return TruncationResult{Content: output} } @@ -66,36 +111,140 @@ func TruncateOutput(output string) TruncationResult { return TruncationResult{Content: output} } - tlog.Debug("truncate", "truncated", "file", filePath, "total_lines", len(lines), "total_bytes", len(output)) + tlog.Debug("truncate", "truncated", "file", filePath, "total_lines", len(lines), + "total_bytes", len(output), "total_tokens", EstimateTokens(output)) + + // The tail comes first so the budget covers the whole preview: what the end + // needs is not negotiable, and the head gets what is left of it. + tailText, tailCount := truncTail(lines, TruncTailLines) + headBudget := TruncMaxTokens - EstimateTokens(tailText) + if headBudget < 0 { + headBudget = 0 + } + headLimit := TruncMaxLines + if rest := len(lines) - tailCount; rest < headLimit { + headLimit = rest + } + headText, headCount := truncHead(lines, headLimit, headBudget) + + omitted := lines[headCount : len(lines)-tailCount] + errs, warns := truncSignals(omitted) - // Build head-only preview respecting both limits var preview strings.Builder - bytesAccum := 0 - lineLimit := TruncMaxLines - if len(lines) < lineLimit { - lineLimit = len(lines) - } - for i := 0; i < lineLimit; i++ { - lineSize := len(lines[i]) + 1 // +1 for the newline - if bytesAccum+lineSize > TruncMaxBytes { + preview.WriteString(headText) + if len(omitted) > 0 { + fmt.Fprintf(&preview, "\n... %d of %d lines omitted (%d mention an error, %d a warning) ...\n\n", + len(omitted), len(lines), errs, warns) + } + preview.WriteString(tailText) + + fmt.Fprintf(&preview, + "\n[TRUNCATED] Full output saved to: %s\n"+ + "Use read_file with offset/limit to view specific sections.\n", filePath) + return TruncationResult{ + Content: preview.String(), + FullPath: filePath, + OmittedLines: len(omitted), + } +} + +// truncHead keeps the first lines that fit the byte and token budgets. The first +// line is always kept, clipped if it alone is too large: an output whose first +// line is enormous must still yield a preview that starts with it. +func truncHead(lines []string, limit, tokenBudget int) (string, int) { + if limit > len(lines) { + limit = len(lines) + } + if limit <= 0 || tokenBudget <= 0 { + return "", 0 + } + byteBudget := tokenBudget * bytesPerToken + if byteBudget > TruncMaxBytes { + byteBudget = TruncMaxBytes + } + var b strings.Builder + used := 0 + count := 0 + for i := 0; i < limit; i++ { + line := lines[i] + size := len(line) + 1 + if used > 0 && used+size > byteBudget { break } - preview.WriteString(lines[i]) - preview.WriteString("\n") - bytesAccum += lineSize - } - - totalBytes := len(output) - removedBytes := totalBytes - bytesAccum - - hint := fmt.Sprintf( - "\n... (%d bytes truncated) ...\n\n"+ - "[TRUNCATED] Full output saved to: %s\n"+ - "Use read_file with offset/limit to view specific sections.\n", - removedBytes, filePath, - ) - preview.WriteString(hint) - return TruncationResult{Content: preview.String(), FullPath: filePath} + if used == 0 { + line = clipBytes(line, byteBudget) + size = len(line) + 1 + } + b.WriteString(line) + b.WriteByte('\n') + used += size + count++ + } + return b.String(), count +} + +// truncTail keeps the last lines, bounded by the byte cap so one enormous last +// line cannot smuggle the whole output back in. +func truncTail(lines []string, want int) (string, int) { + if want <= 0 || len(lines) == 0 { + return "", 0 + } + if want > len(lines) { + want = len(lines) + } + var kept []string + used := 0 + for i := len(lines) - 1; i >= 0 && len(kept) < want; i-- { + size := len(lines[i]) + 1 + if len(kept) > 0 && used+size > TruncMaxBytes { + break + } + line := lines[i] + if len(kept) == 0 { + line = clipBytes(line, TruncMaxBytes) + size = len(line) + 1 + } + kept = append(kept, line) + used += size + } + for i, j := 0, len(kept)-1; i < j; i, j = i+1, j-1 { + kept[i], kept[j] = kept[j], kept[i] + } + if len(kept) == 0 { + return "", 0 + } + return strings.Join(kept, "\n") + "\n", len(kept) +} + +// truncSignals counts the omitted lines that mention a failure or a warning. An +// error line is not also counted as a warning. +func truncSignals(lines []string) (errs, warns int) { + for _, line := range lines { + if truncErrorSignal.MatchString(line) { + errs++ + continue + } + if truncWarningSignal.MatchString(line) { + warns++ + } + } + return errs, warns +} + +// clipBytes cuts a string to at most limit bytes on a rune boundary, so a +// multi-byte character is never split in half. +func clipBytes(text string, limit int) string { + if limit <= 0 { + return "" + } + if len(text) <= limit { + return text + } + cut := limit + for cut > 0 && !utf8.RuneStart(text[cut]) { + cut-- + } + return text[:cut] } // uniqueToolFile generates a unique file path for saving truncated tool output. diff --git a/agent/truncate_test.go b/agent/truncate_test.go index 8840284..40f8d39 100644 --- a/agent/truncate_test.go +++ b/agent/truncate_test.go @@ -1,38 +1,54 @@ package agent import ( + "fmt" "os" "strings" "testing" + + "github.com/yusiwen/tinycode/config" ) +// withTruncation sets the package limits for one test and restores them, with a +// temporary output directory so a test never writes into another's. +func withTruncation(t *testing.T, maxLines, maxBytes, maxTokens, tailLines int) { + t.Helper() + lines, bytes, tokens, tail, dir := TruncMaxLines, TruncMaxBytes, TruncMaxTokens, TruncTailLines, truncDir + TruncMaxLines, TruncMaxBytes, TruncMaxTokens, TruncTailLines = maxLines, maxBytes, maxTokens, tailLines + truncDir = t.TempDir() + t.Cleanup(func() { + TruncMaxLines, TruncMaxBytes, TruncMaxTokens, TruncTailLines, truncDir = lines, bytes, tokens, tail, dir + }) +} + // TestSetTruncationConfig covers the config wiring for truncation limits. func TestSetTruncationConfig(t *testing.T) { - savedLines, savedBytes, savedDir := TruncMaxLines, TruncMaxBytes, truncDir + savedLines, savedBytes, savedTokens, savedTail, savedDir := + TruncMaxLines, TruncMaxBytes, TruncMaxTokens, TruncTailLines, truncDir defer func() { - TruncMaxLines, TruncMaxBytes, truncDir = savedLines, savedBytes, savedDir + TruncMaxLines, TruncMaxBytes, TruncMaxTokens, TruncTailLines, truncDir = + savedLines, savedBytes, savedTokens, savedTail, savedDir }() - SetTruncationConfig(123, 4567, "/tmp/custom-trunc") - if TruncMaxLines != 123 || TruncMaxBytes != 4567 || truncDir != "/tmp/custom-trunc" { - t.Fatalf("config not applied: lines=%d bytes=%d dir=%q", TruncMaxLines, TruncMaxBytes, truncDir) + SetTruncationConfig(123, 4567, 890, 12, "/tmp/custom-trunc") + if TruncMaxLines != 123 || TruncMaxBytes != 4567 || TruncMaxTokens != 890 || TruncTailLines != 12 || truncDir != "/tmp/custom-trunc" { + t.Fatalf("config not applied: lines=%d bytes=%d tokens=%d tail=%d dir=%q", + TruncMaxLines, TruncMaxBytes, TruncMaxTokens, TruncTailLines, truncDir) } // Zero/empty values must keep the current setting. - SetTruncationConfig(0, 0, "") - if TruncMaxLines != 123 || TruncMaxBytes != 4567 || truncDir != "/tmp/custom-trunc" { - t.Fatalf("zero values changed the settings: lines=%d bytes=%d dir=%q", TruncMaxLines, TruncMaxBytes, truncDir) + SetTruncationConfig(0, 0, 0, 0, "") + if TruncMaxLines != 123 || TruncMaxBytes != 4567 || TruncMaxTokens != 890 || TruncTailLines != 12 || truncDir != "/tmp/custom-trunc" { + t.Fatalf("zero values changed the settings: lines=%d bytes=%d tokens=%d tail=%d dir=%q", + TruncMaxLines, TruncMaxBytes, TruncMaxTokens, TruncTailLines, truncDir) } } // TestTruncateOutputSavedFileIsPrivate checks that the full-output file is // written 0600: tool output frequently contains secrets. func TestTruncateOutputSavedFileIsPrivate(t *testing.T) { - savedDir := truncDir - truncDir = t.TempDir() - defer func() { truncDir = savedDir }() + withTruncation(t, 2000, 200*1024, 4000, 20) - // One line past the line limit forces a dump. var sb strings.Builder for i := 0; i <= TruncMaxLines; i++ { sb.WriteString("line\n") @@ -41,7 +57,6 @@ func TestTruncateOutputSavedFileIsPrivate(t *testing.T) { if res.FullPath == "" { t.Fatal("expected the full output to be saved") } - info, err := os.Stat(res.FullPath) if err != nil { t.Fatalf("stat saved output: %v", err) @@ -49,8 +64,134 @@ func TestTruncateOutputSavedFileIsPrivate(t *testing.T) { if perm := info.Mode().Perm(); perm != 0600 { t.Errorf("saved output mode = %o, want 600", perm) } - // The preview must still tell the model where the rest is. if !strings.Contains(res.Content, "TRUNCATED") { t.Errorf("preview is missing the truncation hint: %q", res.Content) } } + +// TestTruncateLeavesAnOutputInsideEveryBoundAlone is the control: a preview that +// is under the line, byte and token bounds is returned byte for byte and no file +// is written. +func TestTruncateLeavesAnOutputInsideEveryBoundAlone(t *testing.T) { + withTruncation(t, 2000, 200*1024, 4000, 20) + const output = "one\ntwo\nthree\n" + res := TruncateOutput(output) + if res.Content != output || res.FullPath != "" || res.OmittedLines != 0 { + t.Fatalf("a small output was touched: %+v", res) + } +} + +// TestTruncateRespectsTheTokenBudget pins the bound that matters: the preview is +// measured in what the model pays for, not in bytes. +func TestTruncateRespectsTheTokenBudget(t *testing.T) { + withTruncation(t, 10000, 200*1024, 1000, 20) + + var sb strings.Builder + for i := 0; i < 5000; i++ { + fmt.Fprintf(&sb, "line %04d with a little text\n", i) + } + res := TruncateOutput(sb.String()) + if res.FullPath == "" { + t.Fatal("a 5000-line output was not truncated") + } + if res.OmittedLines < 4000 { + t.Fatalf("omitted %d lines, want most of 5000", res.OmittedLines) + } + // The budget covers head and tail; the message and the hint add a little. + if tokens := EstimateTokens(res.Content); tokens > TruncMaxTokens+150 { + t.Fatalf("preview is %d tokens, past the %d-token budget", tokens, TruncMaxTokens) + } +} + +// TestTruncateKeepsTheTail pins the half a head-only preview lost: the last line, +// where a failure usually is. +func TestTruncateKeepsTheTail(t *testing.T) { + withTruncation(t, 10, 200*1024, 4000, 5) + + var sb strings.Builder + for i := 0; i < 100; i++ { + fmt.Fprintf(&sb, "line %d\n", i) + } + sb.WriteString("FAIL: the thing that broke\n") + res := TruncateOutput(sb.String()) + + if !strings.Contains(res.Content, "FAIL: the thing that broke") { + t.Fatalf("the tail is missing from the preview:\n%s", res.Content) + } + if !strings.Contains(res.Content, "line 0") { + t.Fatalf("the head is missing from the preview:\n%s", res.Content) + } +} + +// TestTruncateSummarisesWhatItDropped pins the structured part of the preview: +// how many lines went, and how many of them mention a failure or a warning. +func TestTruncateSummarisesWhatItDropped(t *testing.T) { + withTruncation(t, 10, 200*1024, 4000, 5) + + var sb strings.Builder + for i := 0; i < 200; i++ { + switch i { + case 50, 51, 52: + sb.WriteString("error: something failed\n") + case 80, 81: + sb.WriteString("warning: deprecated call\n") + default: + fmt.Fprintf(&sb, "line %d\n", i) + } + } + res := TruncateOutput(sb.String()) + + for _, want := range []string{"3 mention an error", "2 a warning", "of 200 lines omitted"} { + if !strings.Contains(res.Content, want) { + t.Fatalf("the summary is missing %q:\n%s", want, res.Content) + } + } +} + +// TestTruncateClipsAnEnormousFirstLine pins the degenerate case: one line larger +// than every bound must still produce a preview that starts with it, not an +// empty one. +func TestTruncateClipsAnEnormousFirstLine(t *testing.T) { + withTruncation(t, 10, 1000, 4000, 5) + output := "HEAD-" + strings.Repeat("x", 5000) + "\nsecond\n" + res := TruncateOutput(output) + if res.FullPath == "" { + t.Fatal("an oversized single line was not truncated") + } + if !strings.HasPrefix(res.Content, "HEAD-") { + t.Fatalf("the preview does not start with the line: %q", res.Content[:40]) + } + if len(res.Content) > 4000 { + t.Fatalf("the preview is %d bytes, past the byte cap plus the hint", len(res.Content)) + } +} + +// TestBytesPerTokenMatchesTheEstimator keeps the loop's byte counting and the +// token estimate in step: bytesPerToken is the estimator's inverse, and a change +// to one without the other would silently move every budget. +func TestBytesPerTokenMatchesTheEstimator(t *testing.T) { + for _, n := range []int{0, 1, 4, 40} { + text := strings.Repeat("x", n*bytesPerToken) + if got := EstimateTokens(text); got != n { + t.Fatalf("EstimateTokens(%d bytes) = %d, want %d", n*bytesPerToken, got, n) + } + } +} + +// TestTruncationDefaultsMatchConfig pins the two layers' numbers: the package +// falls back to its own when it is handed zero, and the config defaults are what +// a fresh install starts with. +func TestTruncationDefaultsMatchConfig(t *testing.T) { + cfg := config.DefaultConfig() + if cfg.Truncation == nil { + t.Fatal("the shipped defaults have no truncation section") + } + if TruncMaxLines != cfg.Truncation.MaxLines || TruncMaxBytes != cfg.Truncation.MaxBytes { + t.Fatalf("line/byte defaults = %d/%d, config = %d/%d", + TruncMaxLines, TruncMaxBytes, cfg.Truncation.MaxLines, cfg.Truncation.MaxBytes) + } + if TruncMaxTokens != cfg.Truncation.MaxTokens || TruncTailLines != cfg.Truncation.TailLines { + t.Fatalf("token/tail defaults = %d/%d, config = %d/%d", + TruncMaxTokens, TruncTailLines, cfg.Truncation.MaxTokens, cfg.Truncation.TailLines) + } +} diff --git a/config.example.json b/config.example.json index 791d5b6..30812ee 100644 --- a/config.example.json +++ b/config.example.json @@ -89,6 +89,8 @@ "truncation": { "max_lines": 2000, "max_bytes": 204800, + "max_tokens": 4000, + "tail_lines": 20, "output_dir": "/tmp/tinycode/truncated" }, @@ -244,7 +246,7 @@ "checkpoint_dir": "Where the bytes a file tool replaced are kept, so /undo and --undo can restore them. Deliberately outside the project root: the fence that confines a tool write then confines the recovery store. It is not a tool path — no permission grants a model access to it. A bash command's writes are recorded too, from a snapshot of the project tree taken before it, bounded per file (2 MiB) and per snapshot (32 MiB / 4000 files); paths outside the project root are not captured. See docs/checkpoints.md.", "trajectory": "Where a run's record goes (dir), how much of it keeps text (capture: metadata|full), and the store's bounds (max_runs, max_bytes). enabled: false turns recording off. capture: full is honoured only from your own config file, never from a project-local one.", "memory": "Long-term memory: enabled, backend (local|hindsight), dir, base_url, bank, max_entries, max_bytes, recall_limit, embedding {enabled, provider, base_url, model, api_key_env, min_score, semantic_weight, lexical_weight, backfill}. Local by default and written only by /remember; a network backend, a base_url or an embedding endpoint is an egress request honoured only from your own config file.", - "truncation": "Tool-output truncation limits: max_lines, max_bytes, output_dir (used only when the output exceeds a limit).", + "truncation": "Tool-output truncation limits: max_lines, max_bytes, max_tokens (the bound that matters, since the preview is what the model pays for), tail_lines (how many of the last lines are kept beside the head, so a failure survives), output_dir (used only when the output exceeds a limit).", "agents": "allowed_tools replaces the agent's ruleset with a whitelist; denied_tools is applied on top of the built-in rules." } } diff --git a/config/config.go b/config/config.go index 30613b2..687f2ee 100644 --- a/config/config.go +++ b/config/config.go @@ -81,8 +81,15 @@ type PromptCacheConfig struct { // TruncationConfig holds tool output truncation settings. type TruncationConfig struct { - MaxLines int `json:"max_lines,omitempty"` - MaxBytes int `json:"max_bytes,omitempty"` + MaxLines int `json:"max_lines,omitempty"` + MaxBytes int `json:"max_bytes,omitempty"` + // MaxTokens bounds the preview by what the model pays for. It is the bound + // that matters once the text is dense; zero means the default. + MaxTokens int `json:"max_tokens,omitempty"` + // TailLines is how many of the output's last lines the preview keeps beside + // the head, so a failure at the end survives truncation. Zero means the + // default. + TailLines int `json:"tail_lines,omitempty"` OutputDir string `json:"output_dir,omitempty"` } @@ -698,6 +705,8 @@ func DefaultConfig() Config { Truncation: &TruncationConfig{ MaxLines: 2000, MaxBytes: 200 * 1024, + MaxTokens: 4000, + TailLines: 20, OutputDir: "/tmp/tinycode/truncated", }, Agents: map[string]AgentOverride{ diff --git a/docs/roadmap.md b/docs/roadmap.md index 2a33aba..50b4dc5 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -16,10 +16,10 @@ after it (see *Provenance and method*). | State | Entries | Note | | :--- | ---: | :--- | -| Delivered | 15 | the entry's **Picked up** line names the issue that landed it | +| Delivered | 16 | the entry's **Picked up** line names the issue that landed it | | Refuted | 1 | E5 — the capability exists, recorded so it is not re-filed | | In progress | 1 | F1 — three increments landed, the next one is not opened | -| Not started | 23 | verified gaps, no issue opened | +| Not started | 22 | verified gaps, no issue opened | | **Total** | **40** | `grep -Ec '^\| [A-H][0-9] \| .* \| (Delivered —\|Refuted\|In progress\|Not started)' docs/roadmap.md` | **Open work**: F1, whose next increment (a task set with headroom) is not opened, and @@ -44,7 +44,7 @@ B4 (repository index), F1, then the larger D and E items. | B3 | No embeddings, vector search or rerank | Delivered — #297 | | B4 | No repository index or repo map | Not started | | B5 | Context management has a single strategy | Not started | -| B6 | Truncation is fixed, not budget-aware | Not started | +| B6 | Truncation is fixed, not budget-aware | Delivered — #304 | | C1 | No checkpoint and no undo | Delivered — #234 | | C2 | No automatic change preview and commit loop, and no remote operations | Not started | | C3 | No isolation for parallel execution | Not started | @@ -395,6 +395,14 @@ current tree. D7 is the exception: it was recorded later, so its references are ### B6 — Truncation is fixed, not budget-aware +- **Picked up**: [#304](https://github.com/yusiwen/TinyCode/issues/304) — one slice: a token budget + beside the line and byte caps, the output's last lines kept beside the head, and a deterministic count + of what fell between them. +- **Delivered (2026-10-28)**: the preview is bounded by `truncation.max_tokens` (default 4000) as well as + by lines and bytes, keeps `tail_lines` (default 20) of the end, and replaces the old "N bytes + truncated" with "N of M lines omitted (E mention an error, W a warning)" — a `regexp` scan over the + dropped lines, not a model call. A head-only preview used to drop the panic at the end; it no longer + does. - **Gap**: truncation is a line/byte cap, not a token budget, and there is no structured summary in place of raw text. - **Evidence**: `config.example.json:53-57` sets `max_lines` 2000, `max_bytes` 204800, diff --git a/main.go b/main.go index 1cdd0c1..78f7ae0 100644 --- a/main.go +++ b/main.go @@ -215,7 +215,8 @@ func newRootCmd() *cobra.Command { } } if cfg.Truncation != nil { - agent.SetTruncationConfig(cfg.Truncation.MaxLines, cfg.Truncation.MaxBytes, expandPath(cfg.Truncation.OutputDir)) + agent.SetTruncationConfig(cfg.Truncation.MaxLines, cfg.Truncation.MaxBytes, + cfg.Truncation.MaxTokens, cfg.Truncation.TailLines, expandPath(cfg.Truncation.OutputDir)) } if cfg.SearXNGURL != "" { tool.SetSearXNG(cfg.SearXNGURL)