Repository navigation
feat: agent mode — let the search depend on what it already found - #26
Merged
Merged
Conversation
indexer only stored chunks so there was no way to read a whole file back without hitting github again. now the file text rides along in the same cache blob. also fetchRepoTree takes a ref (branch/tag/sha) so evals can index the exact commit a question was written against. ?ref= on the repo page wires it up. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
five tools the agent loop will call next. all of them read straight off
the vector store so a call is reproducible for a given index - no network,
no hidden state. each returns the same { result, truncated, bytes } envelope
with its own cap (200 lines a read, 40 grep hits, 4kb-ish snippets) so one
bad grep can't eat the context window.
read_file prints line numbers on purpose, that's what makes the model cite
lines that actually exist. find_definition goes chunk names first then falls
back to the dep graph, find_references is grep minus the definition site.
skipped the web worker for grep for now - the core is a pure fn over
{ path: text } like bm25-core, so moving it later is a wrapper, and a line
scan over a few hundred files isn't what's blocking the ui next to an llm call.
window.gitask on an indexed repo if you want to poke at them in devtools.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
runAgent is an async generator that yields thinking / tool_call / tool_result / answer / error, so the trace ui in phase 3 and the eval harness in phase 4 both just consume events instead of reaching inside the loop. the loop assumes the model misbehaves and bounds it rather than trusting it: - 12 turn cap, then one final call with no tools so it has to write prose - duplicate calls (same tool, same args, order-insensitive) get "you already ran this at step N" instead of re-running - costs nothing, breaks the loop - 60kb tool-output budget, tools get taken away when it's spent - ToolError goes back to the model as text so a wrong path is recoverable, anything else kills the run - no citations in the answer -> one rewrite ask, then ship it uncited rather than looping forever provider adapter so the loop never sees gemini. two impls: scripted (replays canned turns, this is what makes all 12 loop tests run with no key and no network) and gemini 2.5 flash function calling straight from the browser with the user's key, same as the groq path. also a new "agent" metric event with steps + tokens, and window.gitask.agent() to drive a whole run from devtools. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adds the mode switch in the chat header and a trace on agent answers - one row per tool call with args, elapsed, bytes and a truncated/repeat flag, expandable to the raw result. auto is inert for now, behaves as rag until the router lands. gemini provider now takes a KeyScope instead of a key string, since the byok vault only hands out the key inside withKey(). Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
30 from SWE-QA (flask + requests, pinned to their commits), 15 authored on gitask itself. Gold line spans transcribed from SWE-QA's own answers and checked against the pinned source - verify_eval_questions.py walks all 94 of them. chunk relevance is derived from goldLocations against whatever index is loaded instead of being annotated, so re-chunking doesn't invalidate the labels. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
retrieveForQuery / assembleRagContext / buildRagSystemPrompt now live in lib/ragAnswer.ts. the page keeps streaming, message state and the diagram flag; everything that decides what the model actually sees is shared. the eval arm needs to run the real baseline, not a lookalike - if the two drift every rag-vs-agent number in the deck is junk. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
both arms call gemini directly so they share a model id, a temperature and real usageMetadata instead of an estimate. rag runs the app's own retrieval and prompt via lib/ragAnswer, agent runs the app's own loop. judge never sees the retrieved context or which arm wrote the answer, and citations are scored mechanically against gold spans so that number isn't the judge's opinion. refusals get graded like any other answer - the user still got one. summary averages per question before averaging across questions, so a question doesn't count more just because it ran more times. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
picks a repo from the eval set, indexes it at the commit the gold line numbers were written against, and runs every question through both arms N times. rows stream into localStorage as they land so an hour-long run survives a reload. shows the gold files the index is missing up front - at the wrong ref, or dropped by isIndexable, those questions are unanswerable and the number has to be on the slide rather than discovered in q&a. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pulled the four embedding-only configs out of ablation.test.ts into lib/ablationConfigs.ts so the browser can run the same code against a real index. numbers on the codesearchnet corpus are unchanged (full pipeline still 62.9 / 0.7664 / 0.7102). relevance comes from deriveRelevance, so the ablation and the rag/agent table are about the same 45 questions instead of two unrelated corpora. also fixes the CodeRAG Multi-Path test that used to time out at 60s - it was rebuilding a vector store and a bm25 index for every query sample. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
drives the real page with playwright: index at the pinned ref, run the ablation, optionally run both arms with a key seeded into localStorage the way LLM Settings would. proves the no-key half on psf/requests@46e939b - 292 chunks, all 12 gold paths present, ablation over all 15 questions. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
three things the first real run against gemini turned up: - /eval never called initLLM, so getLLMStatus() was idle and both query expansion helpers silently no-op'd. the rag arm was running single query - a weaker baseline than the app ships, which would have made the agent look better than it is. - a 429 inside the judge took the whole run down. judge failures are now recorded as unreadable and dropped from correctness instead. - the gemini sdk takes no abort signal, so one rate-limited call hung an entire run for 20 minutes. there's a per-run timeout now; it can't cancel the call but it stops the eval waiting on it. plus shared retry/backoff for 429 and 503 in both the agent provider and the eval's one-shot calls, and a pause between runs. smoke run on psf/requests@46e939b, both arms, end to end in a browser: rag 1.00 / 25% cite prec / 4776 tok, agent 0.50 / 11% / 3229 tok / 2 steps. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
retrieval is deterministic so there's no run-to-run variance, but 15 questions per repo is a small sample and the spread between questions is the real uncertainty. the rag/agent table already carried stderr; this one didn't, which made it look more conclusive than it is. with the bars in, every pairwise config difference on all three repos sits inside ~1 standard error. n=15 can't separate them - the codesearchnet ablation (249 queries) is where that claim belongs. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
the agent arm was throwing away every tool_call and tool_result and keeping only aggregates, so five of the plan's eight failure buckets - looped on a tool, truncation hid the answer, grep too broad, wrong file chosen - were unrecoverable from an export. found this before the expensive run rather than after it. failures.ts buckets every run the judge didn't call correct, computed from the record instead of read off by hand so a rerun gets the same taxonomy for free. a failure usually matches several buckets; each run gets one primary by priority (error > judge > mechanism > outcome) and keeps the rest so co-occurrence stays answerable. no fixes yet. the plan says let the data pick the top two buckets and there is no data - one agent run so far. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…okens
gemini-2.5-flash still shows up in ListModels but 404s on generateContent
("no longer available to new users"), so anyone who made a key recently got
a dead model on the chat path too, not just the eval.
Pulled the model id into one constant since the eval's whole claim is that
RAG and agent differ only in retrieval - that shouldn't rest on three string
literals happening to match.
Also: Gemini 3 bills reasoning tokens at the output rate but reports them in
their own field, so reading candidatesTokenCount alone undercounts. Measured
44 thinking tokens against 24 visible ones on a trivial call. That would have
undercounted the agent arm worst, since it calls the model 6-12 times per
question where RAG calls once - flattering exactly the arm whose cost is the
argument against it. Pinned thinkingLevel to "low" so the counts reproduce;
"minimal" is rejected by 3.8 Flash and Flash can't disable thinking at all.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tures The smoke run caught this: every agent run came back 400, 0 steps, 0 tokens. Two separate breaking changes between 2.5 and 3.x, neither of which the compiler can see, because the pinned SDK predates Gemini 3 entirely. First, the `function` role is gone - "Role 'function' is not supported" - so tool results ride back on a user turn now. They're still functionResponse parts, so the model still reads them as evidence and not as instructions. Second, every functionCall part comes back with an opaque thoughtSignature, and replaying that call without it is a hard 400. The provider keeps them in a map for the length of one run and puts them back on the way out. While in there: Gemini 3 hands out its own call ids, so stop synthesizing name-index ones. Those collided anyway - the index restarted at 0 every reply, so semantic_search-0 at step 1 and step 3 were the same key. Worth the noise because of how this fails. It compiles, it ships, the first tool result kills the run, and the eval would have scored 135 agent runs at 0.00 and looked like a finding. Hence the regression tests on toContents. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The export reports tokens per run, and on gemini 3 those move with the thinking level because reasoning bills as output. Anyone reading the JSON later - including me, after the run - shouldn't have to infer the setting from the model id. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drives every repo in one page session, because records, ablation rows and repo summaries live in React state and only the records half survives a reload. Downloads the json after each repo too - a six hour run will die partway through eventually, and a checkpoint makes that a lost repo instead of a lost night. The status line matched loosely at first and locked onto the "run ablation" button label, which never changes. A log that stops moving looks exactly like a hung run, so match the page's three real progress formats instead. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reads an export and prints the arms table split by question type, agent stop reasons, and the bill. Takes --before so phase 5 can diff two exports; the before side is restricted to the questions the re-run covers, otherwise the delta measures the change in sample rather than the fix. Arithmetic only. Scoring, citation checks and the failure taxonomy stay in lib/eval and the /eval page - a second implementation here would be a second source of truth, and the two would drift. Cost is the one thing nothing else computes, since it needs the judge tokens and the per-model rates together. Checked against a synthetic export first, which caught toLocaleString picking up the system locale and printing 1,46,745. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The agent button was enabled for everyone, but agent mode calls Gemini directly with the user's own key - so someone on a local model or without a key could pick it, ask a question, and only then get told. Now the button is disabled with the reason in its tooltip. One function decides, for both the button and the run path. They were always going to drift as two separate checks, and the failure mode of drifting is exactly the bug being fixed here. Settings announce changes on a single event instead of each save handler firing its own, so adding a key enables the button without a reload and there's no handler left to forget. Selecting agent and then locking the vault drops you back to rag rather than leaving you on a dead mode. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It imported pooledMap from indexer.ts, which indexer.ts has never exported - the batching there is inline. Replaced with a local helper that mirrors that loop, with a note that the two have to move together. That only got it as far as the real problem: the embedder asks onnxruntime for the wasm device, which exists in a browser and not in node, so the file cannot pass under vitest and never could. Skipped with the reason written down, rather than deleted - the measurements are still worth taking, they just need a browser runner. Suite is green now instead of green-except-one-everybody-scrolls-past. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…es it The eval set is 18 swe-qa + 27 authored, not the 30/15 the plan claimed - and every swe-qa question is multihop while every lookup question is one I wrote. So "agent wins on multihop" and "agent wins on borrowed questions" predict the same table, and the type split on its own cannot tell them apart. The 9 authored multihop questions are the only cell where they come apart. If the multihop gap holds there too it is about question type; if it only shows up in the swe-qa half, the claim is much weaker and should be said that way. Prints both gaps side by side so the comparison is unavoidable. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All 9 authored multihop questions are gitask questions, so the cell meant to break the source/type confound is also "repo = gitask". It separates provenance from question type and cannot separate either from the repo. The repo column puts that on screen instead of leaving it for someone to work out from the question ids. Also merges several exports, keyed so a repo re-run replaces its rows rather than doubling them, and refuses outright when the exports used different models - a merged table across two models would be meaningless and nothing downstream would notice. A long run that gets interrupted is finished in a second session, and the per-repo checkpoints are separate files. The fixture caught a real one: with no --before, beforeIdx is -1, so beforeIdx + 1 is 0 and the filter silently dropped the first input file. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The config was described as "no BM25 and no RRF" but it also skips graph expansion - every other config calls expandCandidatesWithGraph and this one doesn't. So the gap between it and the full pipeline can't be pinned on any one of the three, and the old label invited exactly that reading. The description string ends up in AblationRow and from there in the export and on a slide, which is the part that made this worth fixing rather than leaving as a comment. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
180s was censoring the comparison, not bounding a hang. On the flask run it killed 12 of 27 agent multihop runs against 6 of 27 for rag - the agent makes six to twelve calls per question where rag makes one, so it gets caught more often, and it gets caught on the slowest questions, which are the multihop ones the whole eval exists to compare. Both readings of that data are wrong. Scoring the timeouts as wrong answers charges the agent for a provider slowdown and gives agent/multihop 0.56. Dropping them keeps only the runs that finished fast and gives 1.00. The honest answer is that those 18 runs have no measurement in them. Raised to 600s against a slowest observed real run of ~7 minutes. The timeout should bound a hang and never end a working run. The report now excludes runs that produced no answer and prints what was lost per arm and type, because exclusion isn't neutral either - lopsided losses mean the surviving samples aren't comparable, and that belongs on the slide rather than buried under a clean-looking mean. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-correct-answer figure still filtered only unreadable judge replies while the tables also dropped runs with no answer. It doesn't change the number - a run that never answered spent no tokens - but two sections disagreeing about what counts is how a slide ends up with a figure nobody can reproduce from the table above it. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Slide 4 needs a concrete question RAG gets wrong and the agent gets right. Picking one by eye invites "did you run it more than once?", so this only lists questions where every run of one arm beats every run of the other - a coin flip can't qualify. Prints both directions. The questions RAG wins are the honest half of the story and they're also the evidence the router slide runs on. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drives the real chat page, not a mock: indexes the repo, switches to agent mode, asks a question that needs more than one file, and saves a webm plus stills of the trace mid-investigation and the finished cited answer. Waits on the textarea being enabled rather than progress text - the text changes wording between phases and the disabled flag doesn't. Default question deliberately spans files. A question answerable from one place produces a two-step run, which shows nothing worth a slide. Don't run it next to the eval; in-browser embedding wants the same cores. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both arms answer the same questions, so comparing two independent means throws the pairing away and reports a wider interval than the data supports. Differencing per question cancels the "some questions are just harder" variance, which dominates at n=15. It also forces an honest restriction: when one arm loses a question outright - a timeout, a blocked candidate - the unpaired means are computed over different question sets and aren't comparable. On flask that's exactly what happened, with rag losing a question to a recitation filter. Flags a gap under two standard errors as inside noise, so the table says so rather than leaving it to whoever reads the slide. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
At this n the interesting gaps sit right on the two-SE line, and a binary label hides exactly those. flask alone put the multihop gap at 0.8 SE; adding requests moved it to 2.0. Both would have printed the same word. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Slide 3 needs this table and the report wasn't printing it at all - the rows were sitting unused in the export. Pooled means are weighted by question count and the pooled errors add in quadrature, since the per-repo means are independent. Per-repo rows stay because the arms data already showed one repo disagreeing sharply with another, and an average that hides that is worth less than the disagreement itself. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 5, and the data picked the target rather than the plan. The agent is at 96% correct with six failures across 135 runs, so there was no accuracy bucket worth fixing. What it does badly is cite: on multihop questions only 43% of its citations land on a gold span, against 90% on lookups. The breakdown says why. A third of multihop citations point at a file that isn't in the gold answer at all, and the cited spans are already tight - median 4 lines - so this isn't over-wide ranges, it's citing files that were only ever visited while narrowing down. The prompt caused it. "Cite only line numbers you have actually seen in a tool result" guards against invention and says nothing about relevance, so having read something was sufficient reason to cite it. On a lookup there's little to walk past; on a multihop the agent opens seven or eight files and cited them all. Also adds a question-type filter to /eval so the re-run covers only the questions the fix can affect. Re-running the lookups would just dilute the before/after. The select goes after the runs one because the headless runners address these by index. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3.8-flash went into "currently experiencing high demand" and every one of the 27 runs in the phase 5 re-run came back a 503 with all four retries exhausted. The page kept going and the runner kept going, and it would have produced two more repos of error rows before anyone looked. Now a repo whose runs are more than half errors stops the run and says which error, instead of checkpointing garbage and moving on. Retries go from four tries to six. The backoff doubles each time, so the window a single call can survive stretches from about 28 seconds to about two minutes. That rides out a spike; nothing rides out an outage, which is what the guard above is for. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both arms answered every question, so the export already says what each scored and what each cost. A routing policy is then just a choice of which column to read per question - no new api calls, and the always-rag and always-agent endpoints fall out of the same arithmetic. The result reframes the slide. Every policy lands near 0.96, so accuracy is not where routers differ on this set; cost is. A perfect router reaches the agent's accuracy while sending only 20% of questions to it, because rag already answers most of them correctly. The rule recovers all of the accuracy and only 6% of the 48% cost saving that's available - it is a good accuracy router and a bad cost router, which is the honest finding. Also reports how often the rule picks the winner on the questions where the arms actually disagreed, because a router can route badly and still score well wherever the arms agree, and that flatters it. Rule left deliberately untuned. At 45 questions, fitting it harder would be fitting it to this eval rather than to the problem. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
printDiff restricted the before side to the questions the re-run covered, but not the other way round. Each run loses its own questions to provider errors and unreadable judges, and those losses aren't random - they land on the slow, hard questions. The phase 5 before-run showed exactly this: requests lost 9 of 27 runs, one question losing all three, and the surviving sample scored noticeably better than the same slice on the earlier full run. Comparing that against an after-run that kept those questions would have measured the censoring and called it a fix. Both sides are now restricted to the questions both actually answered, and the report names any question the before-run lost that the re-run managed. Only that direction is flagged - the before side covering more questions is just the re-run being targeted, which is the point. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The harness flags both as judgeUnreadable, but they mean different things - one is the judge being unreliable, the other is the provider being down. The report said "unreadable judge replies" for both, which overstates the first. Across the main run and the phase 5 before-run, every single one has been a failed call. The judge has not produced an unparseable reply once in 270+ runs, which is worth being able to say. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scripts/record-agent-demo.js writes a ~30MB webm into demo/. It was untracked but not ignored, which is one careless git add away from being in the history permanently. Regenerating it is a single command. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The claim I could not support was that the baseline's import-graph expansion is what already answers most multi-file questions. Two things were in the way. Vector-Only drops bm25, rrf and graph expansion together, so nothing in the table isolates the hop. Added a config that removes only that step. And every ablation number was an aggregate over all 45 questions, which averages away exactly the signal being looked for: a component that only matters when the answer is spread across files. Rows now carry a question type, and the report prints what each component is worth on one-file questions against multi-file ones. A component that carries the multi-file case is positive in that difference; one that doesn't, isn't. No api calls in any of this - it's deterministic retrieval over a local index, which is why it can run while the provider is still down. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ablation needs no provider - it's deterministic retrieval over a local index - so it can run while the arms cannot. Two things stopped that: the runner always went on to the eval, and the download button keyed off records alone, so an ablation-only run had nothing to export. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…er uses An edit added three usages of ablationOnly without the declaration. node --check passes an undefined identifier, since that is a runtime error, so it only surfaced after three minutes of indexing. Also stopped requiring an api key for a run that makes no api calls. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
auto still answers exactly like rag, so the button was a lie. keeping the type and the plumbing for when the router lands, just not offering it. anyone stuck on auto in localStorage falls back to rag. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # scripts/verify-gemini.js # src/app/api/gemini/route.ts # src/lib/llm.ts
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an agent answer mode beside
rag. It searches, reads what came back, then searches again for names the question never contained — the thing one-shot retrieval structurally cannot do.Why
A user said the answers had "too much context overload." Turning that into a number: 18% of the lines a retrieval answer cited were where the answer actually was. Four out of five citations pointed somewhere else, so answers felt padded because they were.
The cause is structural. If a question needs the validator that
prepare_headerscalls, you cannot search for the validator — the question never names it. You only learn the name by reading the first file.What it does
Same index, same search, same model as the baseline, plus
read_file,find_definitionandfind_references. Capped at 12 steps and 60KB of context, repeat calls blocked, citations required.Runs on the user's own Gemini key, straight from the browser — never the server key. The button disables itself with an explanation when there isn't one, and drops back to
ragif the key goes away mid-session. Default staysrag.Measured before shipping, not after
45 questions across 3 repos, 3 runs each, 270 records. Same model, index and prompt on both sides; a judge blind to which arm wrote the answer; citations scored by span overlap with no model involved.
Two findings that cut against me, included deliberately: graph expansion is a no-op (removing it changes nothing on any repo, either question type, to four significant figures), and a prompt fix I tried moved citation accuracy 2.7 points when swapping the model alone moved it 2.0 — so I can't claim it.
Also here
/evalroute and offline harness (scripts/full-eval.js,eval-report.js,router-analysis.js)role: "user"for function responses,thought_signatureechoed on tool calls,thinkingLevelenum — with regression tests, since this broke the agent arm 100% once alreadyautomode is wired but not shown until the router lands; a button that behaves identically toragis a lieVerification
npm run typecheckclean, 194 tests pass (1 skipped — embedder needs awasmdevice Node lacks),npm run buildcleankarpathy/autoresearch, asked a cross-file question, got 5 steps in 18.6s with line-level citations across 4 filesNote for review
~9k insertions, but ~6k never reach a user — tests, fixtures, the eval harness and offline scripts. The feature is ~1.1k lines in
src/lib/agent/plus one component. The only file with heavy churn issrc/app/[owner]/[repo]/page.tsx; everything else is additive and dead when the mode israg.