Skip to content

feat: agent mode — let the search depend on what it already found - #26

Merged
FloareDor merged 41 commits into
mainfrom
agent-mode
Oct 26, 2026
Merged

FloareDor merged 41 commits into
mainfrom
agent-mode

Conversation

@FloareDor

@FloareDor FloareDor commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Adds an agent answer mode beside rag. It searches, reads what came back, then searches again for names the question never contained — the thing one-shot retrieval structurally cannot do.

Why

A user said the answers had "too much context overload." Turning that into a number: 18% of the lines a retrieval answer cited were where the answer actually was. Four out of five citations pointed somewhere else, so answers felt padded because they were.

The cause is structural. If a question needs the validator that prepare_headers calls, you cannot search for the validator — the question never names it. You only learn the name by reading the first file.

What it does

Same index, same search, same model as the baseline, plus read_file, find_definition and find_references. Capped at 12 steps and 60KB of context, repeat calls blocked, citations required.

Runs on the user's own Gemini key, straight from the browser — never the server key. The button disables itself with an explanation when there isn't one, and drops back to rag if the key goes away mid-session. Default stays rag.

Measured before shipping, not after

45 questions across 3 repos, 3 runs each, 270 records. Same model, index and prompt on both sides; a judge blind to which arm wrote the answer; citations scored by span overlap with no model involved.

Answer lives in n rag agent gap
One file 18 0.92 0.96 +0.05
Two or more 26 0.83 0.96 +0.13 ± 0.05
  • Citation precision 18% → 65%
  • Cost $0.0038 → $0.0222 per question (5.8×), latency 22.5s → 31.3s
  • Only 8 of 45 questions genuinely needed it — a perfect router would send 20% to the agent and halve the bill at identical accuracy

Two findings that cut against me, included deliberately: graph expansion is a no-op (removing it changes nothing on any repo, either question type, to four significant figures), and a prompt fix I tried moved citation accuracy 2.7 points when swapping the model alone moved it 2.0 — so I can't claim it.

Also here

  • /eval route and offline harness (scripts/full-eval.js, eval-report.js, router-analysis.js)
  • Gemini 3 wire-format fixes: role: "user" for function responses, thought_signature echoed on tool calls, thinkingLevel enum — with regression tests, since this broke the agent arm 100% once already
  • auto mode is wired but not shown until the router lands; a button that behaves identically to rag is a lie

Verification

  • npm run typecheck clean, 194 tests pass (1 skipped — embedder needs a wasm device Node lacks), npm run build clean
  • Ran end to end in a browser against the live API: indexed karpathy/autoresearch, asked a cross-file question, got 5 steps in 18.6s with line-level citations across 4 files

Note for review

~9k insertions, but ~6k never reach a user — tests, fixtures, the eval harness and offline scripts. The feature is ~1.1k lines in src/lib/agent/ plus one component. The only file with heavy churn is src/app/[owner]/[repo]/page.tsx; everything else is additive and dead when the mode is rag.

FloareDor and others added 30 commits October 17, 2026 21:10
indexer only stored chunks so there was no way to read a whole file back
without hitting github again. now the file text rides along in the same
cache blob. also fetchRepoTree takes a ref (branch/tag/sha) so evals can
index the exact commit a question was written against. ?ref= on the repo
page wires it up.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
five tools the agent loop will call next. all of them read straight off
the vector store so a call is reproducible for a given index - no network,
no hidden state. each returns the same { result, truncated, bytes } envelope
with its own cap (200 lines a read, 40 grep hits, 4kb-ish snippets) so one
bad grep can't eat the context window.

read_file prints line numbers on purpose, that's what makes the model cite
lines that actually exist. find_definition goes chunk names first then falls
back to the dep graph, find_references is grep minus the definition site.

skipped the web worker for grep for now - the core is a pure fn over
{ path: text } like bm25-core, so moving it later is a wrapper, and a line
scan over a few hundred files isn't what's blocking the ui next to an llm call.

window.gitask on an indexed repo if you want to poke at them in devtools.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
runAgent is an async generator that yields thinking / tool_call / tool_result /
answer / error, so the trace ui in phase 3 and the eval harness in phase 4 both
just consume events instead of reaching inside the loop.

the loop assumes the model misbehaves and bounds it rather than trusting it:
- 12 turn cap, then one final call with no tools so it has to write prose
- duplicate calls (same tool, same args, order-insensitive) get "you already
  ran this at step N" instead of re-running - costs nothing, breaks the loop
- 60kb tool-output budget, tools get taken away when it's spent
- ToolError goes back to the model as text so a wrong path is recoverable,
  anything else kills the run
- no citations in the answer -> one rewrite ask, then ship it uncited rather
  than looping forever

provider adapter so the loop never sees gemini. two impls: scripted (replays
canned turns, this is what makes all 12 loop tests run with no key and no
network) and gemini 2.5 flash function calling straight from the browser with
the user's key, same as the groq path.

also a new "agent" metric event with steps + tokens, and window.gitask.agent()
to drive a whole run from devtools.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adds the mode switch in the chat header and a trace on agent answers -
one row per tool call with args, elapsed, bytes and a truncated/repeat
flag, expandable to the raw result. auto is inert for now, behaves as
rag until the router lands.

gemini provider now takes a KeyScope instead of a key string, since the
byok vault only hands out the key inside withKey().

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
30 from SWE-QA (flask + requests, pinned to their commits), 15 authored
on gitask itself. Gold line spans transcribed from SWE-QA's own answers
and checked against the pinned source - verify_eval_questions.py walks
all 94 of them.

chunk relevance is derived from goldLocations against whatever index is
loaded instead of being annotated, so re-chunking doesn't invalidate the
labels.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
retrieveForQuery / assembleRagContext / buildRagSystemPrompt now live in
lib/ragAnswer.ts. the page keeps streaming, message state and the diagram
flag; everything that decides what the model actually sees is shared.

the eval arm needs to run the real baseline, not a lookalike - if the two
drift every rag-vs-agent number in the deck is junk.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
both arms call gemini directly so they share a model id, a temperature
and real usageMetadata instead of an estimate. rag runs the app's own
retrieval and prompt via lib/ragAnswer, agent runs the app's own loop.

judge never sees the retrieved context or which arm wrote the answer,
and citations are scored mechanically against gold spans so that number
isn't the judge's opinion. refusals get graded like any other answer -
the user still got one.

summary averages per question before averaging across questions, so a
question doesn't count more just because it ran more times.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
picks a repo from the eval set, indexes it at the commit the gold line
numbers were written against, and runs every question through both arms
N times. rows stream into localStorage as they land so an hour-long run
survives a reload.

shows the gold files the index is missing up front - at the wrong ref,
or dropped by isIndexable, those questions are unanswerable and the
number has to be on the slide rather than discovered in q&a.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pulled the four embedding-only configs out of ablation.test.ts into
lib/ablationConfigs.ts so the browser can run the same code against a
real index. numbers on the codesearchnet corpus are unchanged (full
pipeline still 62.9 / 0.7664 / 0.7102).

relevance comes from deriveRelevance, so the ablation and the rag/agent
table are about the same 45 questions instead of two unrelated corpora.

also fixes the CodeRAG Multi-Path test that used to time out at 60s - it
was rebuilding a vector store and a bm25 index for every query sample.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
drives the real page with playwright: index at the pinned ref, run the
ablation, optionally run both arms with a key seeded into localStorage
the way LLM Settings would.

proves the no-key half on psf/requests@46e939b - 292 chunks, all 12 gold
paths present, ablation over all 15 questions.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
three things the first real run against gemini turned up:

- /eval never called initLLM, so getLLMStatus() was idle and both query
  expansion helpers silently no-op'd. the rag arm was running single
  query - a weaker baseline than the app ships, which would have made
  the agent look better than it is.
- a 429 inside the judge took the whole run down. judge failures are now
  recorded as unreadable and dropped from correctness instead.
- the gemini sdk takes no abort signal, so one rate-limited call hung an
  entire run for 20 minutes. there's a per-run timeout now; it can't
  cancel the call but it stops the eval waiting on it.

plus shared retry/backoff for 429 and 503 in both the agent provider and
the eval's one-shot calls, and a pause between runs.

smoke run on psf/requests@46e939b, both arms, end to end in a browser:
rag 1.00 / 25% cite prec / 4776 tok, agent 0.50 / 11% / 3229 tok / 2 steps.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
retrieval is deterministic so there's no run-to-run variance, but 15
questions per repo is a small sample and the spread between questions is
the real uncertainty. the rag/agent table already carried stderr; this
one didn't, which made it look more conclusive than it is.

with the bars in, every pairwise config difference on all three repos
sits inside ~1 standard error. n=15 can't separate them - the
codesearchnet ablation (249 queries) is where that claim belongs.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
the agent arm was throwing away every tool_call and tool_result and
keeping only aggregates, so five of the plan's eight failure buckets -
looped on a tool, truncation hid the answer, grep too broad, wrong file
chosen - were unrecoverable from an export. found this before the
expensive run rather than after it.

failures.ts buckets every run the judge didn't call correct, computed
from the record instead of read off by hand so a rerun gets the same
taxonomy for free. a failure usually matches several buckets; each run
gets one primary by priority (error > judge > mechanism > outcome) and
keeps the rest so co-occurrence stays answerable.

no fixes yet. the plan says let the data pick the top two buckets and
there is no data - one agent run so far.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…okens

gemini-2.5-flash still shows up in ListModels but 404s on generateContent
("no longer available to new users"), so anyone who made a key recently got
a dead model on the chat path too, not just the eval.

Pulled the model id into one constant since the eval's whole claim is that
RAG and agent differ only in retrieval - that shouldn't rest on three string
literals happening to match.

Also: Gemini 3 bills reasoning tokens at the output rate but reports them in
their own field, so reading candidatesTokenCount alone undercounts. Measured
44 thinking tokens against 24 visible ones on a trivial call. That would have
undercounted the agent arm worst, since it calls the model 6-12 times per
question where RAG calls once - flattering exactly the arm whose cost is the
argument against it. Pinned thinkingLevel to "low" so the counts reproduce;
"minimal" is rejected by 3.8 Flash and Flash can't disable thinking at all.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tures

The smoke run caught this: every agent run came back 400, 0 steps, 0 tokens.
Two separate breaking changes between 2.5 and 3.x, neither of which the
compiler can see, because the pinned SDK predates Gemini 3 entirely.

First, the `function` role is gone - "Role 'function' is not supported" - so
tool results ride back on a user turn now. They're still functionResponse
parts, so the model still reads them as evidence and not as instructions.

Second, every functionCall part comes back with an opaque thoughtSignature,
and replaying that call without it is a hard 400. The provider keeps them in
a map for the length of one run and puts them back on the way out.

While in there: Gemini 3 hands out its own call ids, so stop synthesizing
name-index ones. Those collided anyway - the index restarted at 0 every
reply, so semantic_search-0 at step 1 and step 3 were the same key.

Worth the noise because of how this fails. It compiles, it ships, the first
tool result kills the run, and the eval would have scored 135 agent runs at
0.00 and looked like a finding. Hence the regression tests on toContents.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The export reports tokens per run, and on gemini 3 those move with the
thinking level because reasoning bills as output. Anyone reading the JSON
later - including me, after the run - shouldn't have to infer the setting
from the model id.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drives every repo in one page session, because records, ablation rows and
repo summaries live in React state and only the records half survives a
reload. Downloads the json after each repo too - a six hour run will die
partway through eventually, and a checkpoint makes that a lost repo instead
of a lost night.

The status line matched loosely at first and locked onto the "run ablation"
button label, which never changes. A log that stops moving looks exactly
like a hung run, so match the page's three real progress formats instead.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reads an export and prints the arms table split by question type, agent
stop reasons, and the bill. Takes --before so phase 5 can diff two exports;
the before side is restricted to the questions the re-run covers, otherwise
the delta measures the change in sample rather than the fix.

Arithmetic only. Scoring, citation checks and the failure taxonomy stay in
lib/eval and the /eval page - a second implementation here would be a second
source of truth, and the two would drift. Cost is the one thing nothing else
computes, since it needs the judge tokens and the per-model rates together.

Checked against a synthetic export first, which caught toLocaleString
picking up the system locale and printing 1,46,745.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The agent button was enabled for everyone, but agent mode calls Gemini
directly with the user's own key - so someone on a local model or without a
key could pick it, ask a question, and only then get told. Now the button is
disabled with the reason in its tooltip.

One function decides, for both the button and the run path. They were always
going to drift as two separate checks, and the failure mode of drifting is
exactly the bug being fixed here.

Settings announce changes on a single event instead of each save handler
firing its own, so adding a key enables the button without a reload and
there's no handler left to forget. Selecting agent and then locking the
vault drops you back to rag rather than leaving you on a dead mode.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It imported pooledMap from indexer.ts, which indexer.ts has never exported -
the batching there is inline. Replaced with a local helper that mirrors that
loop, with a note that the two have to move together.

That only got it as far as the real problem: the embedder asks onnxruntime
for the wasm device, which exists in a browser and not in node, so the file
cannot pass under vitest and never could. Skipped with the reason written
down, rather than deleted - the measurements are still worth taking, they
just need a browser runner.

Suite is green now instead of green-except-one-everybody-scrolls-past.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…es it

The eval set is 18 swe-qa + 27 authored, not the 30/15 the plan claimed -
and every swe-qa question is multihop while every lookup question is one I
wrote. So "agent wins on multihop" and "agent wins on borrowed questions"
predict the same table, and the type split on its own cannot tell them
apart.

The 9 authored multihop questions are the only cell where they come apart.
If the multihop gap holds there too it is about question type; if it only
shows up in the swe-qa half, the claim is much weaker and should be said
that way. Prints both gaps side by side so the comparison is unavoidable.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All 9 authored multihop questions are gitask questions, so the cell meant to
break the source/type confound is also "repo = gitask". It separates
provenance from question type and cannot separate either from the repo. The
repo column puts that on screen instead of leaving it for someone to work
out from the question ids.

Also merges several exports, keyed so a repo re-run replaces its rows rather
than doubling them, and refuses outright when the exports used different
models - a merged table across two models would be meaningless and nothing
downstream would notice. A long run that gets interrupted is finished in a
second session, and the per-repo checkpoints are separate files.

The fixture caught a real one: with no --before, beforeIdx is -1, so
beforeIdx + 1 is 0 and the filter silently dropped the first input file.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The config was described as "no BM25 and no RRF" but it also skips graph
expansion - every other config calls expandCandidatesWithGraph and this one
doesn't. So the gap between it and the full pipeline can't be pinned on any
one of the three, and the old label invited exactly that reading.

The description string ends up in AblationRow and from there in the export
and on a slide, which is the part that made this worth fixing rather than
leaving as a comment.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
180s was censoring the comparison, not bounding a hang. On the flask run it
killed 12 of 27 agent multihop runs against 6 of 27 for rag - the agent makes
six to twelve calls per question where rag makes one, so it gets caught more
often, and it gets caught on the slowest questions, which are the multihop
ones the whole eval exists to compare.

Both readings of that data are wrong. Scoring the timeouts as wrong answers
charges the agent for a provider slowdown and gives agent/multihop 0.56.
Dropping them keeps only the runs that finished fast and gives 1.00. The
honest answer is that those 18 runs have no measurement in them.

Raised to 600s against a slowest observed real run of ~7 minutes. The
timeout should bound a hang and never end a working run.

The report now excludes runs that produced no answer and prints what was
lost per arm and type, because exclusion isn't neutral either - lopsided
losses mean the surviving samples aren't comparable, and that belongs on
the slide rather than buried under a clean-looking mean.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-correct-answer figure still filtered only unreadable judge replies
while the tables also dropped runs with no answer. It doesn't change the
number - a run that never answered spent no tokens - but two sections
disagreeing about what counts is how a slide ends up with a figure nobody
can reproduce from the table above it.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Slide 4 needs a concrete question RAG gets wrong and the agent gets right.
Picking one by eye invites "did you run it more than once?", so this only
lists questions where every run of one arm beats every run of the other -
a coin flip can't qualify.

Prints both directions. The questions RAG wins are the honest half of the
story and they're also the evidence the router slide runs on.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drives the real chat page, not a mock: indexes the repo, switches to agent
mode, asks a question that needs more than one file, and saves a webm plus
stills of the trace mid-investigation and the finished cited answer.

Waits on the textarea being enabled rather than progress text - the text
changes wording between phases and the disabled flag doesn't.

Default question deliberately spans files. A question answerable from one
place produces a two-step run, which shows nothing worth a slide.

Don't run it next to the eval; in-browser embedding wants the same cores.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both arms answer the same questions, so comparing two independent means
throws the pairing away and reports a wider interval than the data supports.
Differencing per question cancels the "some questions are just harder"
variance, which dominates at n=15.

It also forces an honest restriction: when one arm loses a question outright
- a timeout, a blocked candidate - the unpaired means are computed over
different question sets and aren't comparable. On flask that's exactly what
happened, with rag losing a question to a recitation filter.

Flags a gap under two standard errors as inside noise, so the table says so
rather than leaving it to whoever reads the slide.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
At this n the interesting gaps sit right on the two-SE line, and a binary
label hides exactly those. flask alone put the multihop gap at 0.8 SE;
adding requests moved it to 2.0. Both would have printed the same word.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Slide 3 needs this table and the report wasn't printing it at all - the
rows were sitting unused in the export. Pooled means are weighted by
question count and the pooled errors add in quadrature, since the per-repo
means are independent.

Per-repo rows stay because the arms data already showed one repo
disagreeing sharply with another, and an average that hides that is worth
less than the disagreement itself.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FloareDor and others added 11 commits October 19, 2026 10:40
Phase 5, and the data picked the target rather than the plan. The agent is
at 96% correct with six failures across 135 runs, so there was no accuracy
bucket worth fixing. What it does badly is cite: on multihop questions only
43% of its citations land on a gold span, against 90% on lookups.

The breakdown says why. A third of multihop citations point at a file that
isn't in the gold answer at all, and the cited spans are already tight -
median 4 lines - so this isn't over-wide ranges, it's citing files that were
only ever visited while narrowing down.

The prompt caused it. "Cite only line numbers you have actually seen in a
tool result" guards against invention and says nothing about relevance, so
having read something was sufficient reason to cite it. On a lookup there's
little to walk past; on a multihop the agent opens seven or eight files and
cited them all.

Also adds a question-type filter to /eval so the re-run covers only the
questions the fix can affect. Re-running the lookups would just dilute the
before/after. The select goes after the runs one because the headless
runners address these by index.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3.8-flash went into "currently experiencing high demand" and every one of
the 27 runs in the phase 5 re-run came back a 503 with all four retries
exhausted. The page kept going and the runner kept going, and it would
have produced two more repos of error rows before anyone looked.

Now a repo whose runs are more than half errors stops the run and says
which error, instead of checkpointing garbage and moving on.

Retries go from four tries to six. The backoff doubles each time, so the
window a single call can survive stretches from about 28 seconds to about
two minutes. That rides out a spike; nothing rides out an outage, which is
what the guard above is for.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both arms answered every question, so the export already says what each
scored and what each cost. A routing policy is then just a choice of which
column to read per question - no new api calls, and the always-rag and
always-agent endpoints fall out of the same arithmetic.

The result reframes the slide. Every policy lands near 0.96, so accuracy is
not where routers differ on this set; cost is. A perfect router reaches the
agent's accuracy while sending only 20% of questions to it, because rag
already answers most of them correctly. The rule recovers all of the
accuracy and only 6% of the 48% cost saving that's available - it is a good
accuracy router and a bad cost router, which is the honest finding.

Also reports how often the rule picks the winner on the questions where the
arms actually disagreed, because a router can route badly and still score
well wherever the arms agree, and that flatters it.

Rule left deliberately untuned. At 45 questions, fitting it harder would be
fitting it to this eval rather than to the problem.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
printDiff restricted the before side to the questions the re-run covered,
but not the other way round. Each run loses its own questions to provider
errors and unreadable judges, and those losses aren't random - they land on
the slow, hard questions.

The phase 5 before-run showed exactly this: requests lost 9 of 27 runs, one
question losing all three, and the surviving sample scored noticeably better
than the same slice on the earlier full run. Comparing that against an
after-run that kept those questions would have measured the censoring and
called it a fix.

Both sides are now restricted to the questions both actually answered, and
the report names any question the before-run lost that the re-run managed.
Only that direction is flagged - the before side covering more questions is
just the re-run being targeted, which is the point.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The harness flags both as judgeUnreadable, but they mean different things -
one is the judge being unreliable, the other is the provider being down. The
report said "unreadable judge replies" for both, which overstates the first.

Across the main run and the phase 5 before-run, every single one has been a
failed call. The judge has not produced an unparseable reply once in 270+
runs, which is worth being able to say.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scripts/record-agent-demo.js writes a ~30MB webm into demo/. It was
untracked but not ignored, which is one careless git add away from being in
the history permanently. Regenerating it is a single command.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The claim I could not support was that the baseline's import-graph
expansion is what already answers most multi-file questions. Two things
were in the way.

Vector-Only drops bm25, rrf and graph expansion together, so nothing in
the table isolates the hop. Added a config that removes only that step.

And every ablation number was an aggregate over all 45 questions, which
averages away exactly the signal being looked for: a component that only
matters when the answer is spread across files. Rows now carry a question
type, and the report prints what each component is worth on one-file
questions against multi-file ones. A component that carries the multi-file
case is positive in that difference; one that doesn't, isn't.

No api calls in any of this - it's deterministic retrieval over a local
index, which is why it can run while the provider is still down.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ablation needs no provider - it's deterministic retrieval over a local
index - so it can run while the arms cannot. Two things stopped that: the
runner always went on to the eval, and the download button keyed off
records alone, so an ablation-only run had nothing to export.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…er uses

An edit added three usages of ablationOnly without the declaration. node
--check passes an undefined identifier, since that is a runtime error, so
it only surfaced after three minutes of indexing.

Also stopped requiring an api key for a run that makes no api calls.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
auto still answers exactly like rag, so the button was a lie. keeping
the type and the plumbing for when the router lands, just not offering
it. anyone stuck on auto in localStorage falls back to rag.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts:
#	scripts/verify-gemini.js
#	src/app/api/gemini/route.ts
#	src/lib/llm.ts
@vercel

vercel Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
gitask Building Building Preview Oct 26, 2026 4:49am UTC

@FloareDor
FloareDor merged commit fbda6e2 into main Oct 26, 2026
3 of 4 checks passed
@FloareDor
FloareDor deleted the agent-mode branch October 26, 2026 04:55

This branch was successfully deployed

1 active deployment
Preview — 84a8cfbb Deployed Oct 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant