feat(evals): run the SPACE text benchmarks through the agent-eval path - #3414
Open
mc856 wants to merge 13 commits into
Open
feat(evals): run the SPACE text benchmarks through the agent-eval path#3414mc856 wants to merge 13 commits into
mc856 wants to merge 13 commits into
Conversation
Contract (models.py): three new discriminated-union members -- source "none" (nothing to materialize), task "verbatim_prompt" (prompt handed to the agent untouched), validator "external_evaluator" (scoring delegated to an external suite, recorded as benchmark + revision). CompactEvalResult.recording/progress relax to optional: cases without a frozen environment have neither. Execution (single_case.py): kind-aware dispatch at oracle preflight, materialization, server environment, prompt assembly, system prompt, scoring, and both result constructors. EmptyEnvironment keeps python_exec on an empty kernel (code_policy_core.py). CLI rendering handles environment-less results (cli/eval.py). Existing tests unchanged and green; mypy union-attr clean.
… consumer BenchmarkAdapter protocol (name/revision + iter_items/to_case/score) is the seam an external benchmark plugs into; the answer key never enters an EvalCase -- BenchmarkItem has no answer field by construction. Group-aware sampling takes whole answer-order groups (4 permutations per stimulus) so subsets stay balanced. The SPACE text-task registry records upstream row counts; a generated grid suite is the second adapter implementation and the zero-data default test tier.
Source and data are fetched at run time to the user cache -- a pinned checkout of apple/ml-space-benchmark (564e4393) and the official release tarball (SHA256-verified); nothing SPACE-derived enters the repository. DimosQAAgent registers in SPACE's own registry and keeps the upstream question, parsing, and scoring code untouched: the only replaced step is the model call, which now runs execute_single_case. Every question writes a per-case record, and the run fails loudly if those records disagree with the official results.json. `dimos eval space --task <name>` is the documented entry point; the `space` extra declares what `import space` needs beyond dimos itself.
Re-running with the same --output used to spend the full set of agent sessions and only then fail on the duplicate results.json. The guard sits before source, data, and any network or agent work.
Preflight refuses a run that could never score before anything is fetched (bounds, API key, extra modules, Pi build), and a run where no question was ever answered no longer reports 0% as if the model had scored it; the summary counts infra failures next to the accuracy they distort. The downloader follows the repo HTTP rule (requests, with timeouts) and the cached release is verified against its provenance record on every run, which is now written before the release it describes. The adapter score path runs on every question as a consistency gate over the official predictions. Records that cannot be written no longer kill the worker pool. The kernel for environment-less cases no longer inherits the run- and data-directory variables, and the synthetic suite default seed no longer lets an ordinal predict its answer slot. Docstrings and the docs page claim exactly what the code does.
mc856
requested review from
Dreamsorcerer,
leshy,
mustafab0,
paul-nechifor and
spomichter
as code owners
August 9, 2026 10:09
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## cc/feat/frozen-qa-eval #3414 +/- ##
==========================================================
+ Coverage 76.26% 76.62% +0.35%
==========================================================
Files 1211 1231 +20
Lines 116918 119052 +2134
Branches 10512 10635 +123
==========================================================
+ Hits 89164 91220 +2056
- Misses 24670 24757 +87
+ Partials 3084 3075 -9
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
`dimos_revision` asked git for a HEAD from the directory this package sits in, and git answers from the nearest repository at or above that, so an installed copy under an unrelated checkout would have signed the manifest with that checkout's commit. It now returns nothing unless the toplevel git names is this source tree, and a git that hangs no longer takes the run down with it. The docs row says what the field still cannot see: uncommitted changes here, unlike the SPACE checkout, which a run refuses outright when its tree is dirty. The run claims its --output directory beside the preflight instead of beside the first write, so a path nothing can create is refused before the 3.6 GB download rather than after it. Three tests claimed more than they checked. The stub upstream parser skipped the integer coercion the real one does, so a parametrized case asserted an outcome no run produces; the ledger's atomicity test passed for a plain write; and nothing pinned the four-consecutive-rows-per-stimulus layout that whole-group sampling rests on, now asserted against the release for every registered task. The evaluation page follows the command it documents, which grew a third rendering for externally scored cases, and the benchmark page joins the docs nav. Sampling docs say how to ask for a whole task, and that anything smaller reports one draw.
`--seed` was required so two runs of one command asked about the same stimuli, which is what a comparison needs. But it made the ordinary case -- score this task -- carry a flag whose value nobody had a reason to choose, and it pushed everyone toward whatever number they typed last, so a task was always measured on one subset of itself. The seed is now drawn when it is absent, printed beside the score as `(drawn)`, and written to the manifest as it already was, so the run is reproducible from its own record. Pinning it is still what turns two runs into a comparison, and the sampling docs say that where they say what --groups does to coverage. The draw happens in the CLI rather than in run_space_task, which still takes an integer and stays deterministic under test.
git's toplevel answer was split on any whitespace, so a checkout at a path with a space in it failed the two-line check and lost its dimos revision -- on exactly the kind of machine the guard was written for. The answer now splits by lines. mkdir proves nothing about a directory that already exists, so the pre-download claim let an unwritable --output through to the far side of the 3.6 GB fetch; claiming now writes and removes a probe file. The revision guard's hanging-git path, half of its motivation, gets the test it was missing, and one overlong docs line is rewrapped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contribution path
cc/feat/frozen-qa-eval(feat: add frozen recording agent evaluation #3378); I will rebase ontomainonce feat: add frozen recording agent evaluation #3378 lands.
Problem
The #3378 agent-eval framework scores agents only against internally authored suites with local
oracles. #3399 asks for an external benchmark — Apple's SPACE (spatial cognition) — wired through
the same execution path, so that an unmodified official scorer grades what a DimOS agent actually
does. The integration must not redistribute any SPACE content (ND-licensed data), and the seams it
adds should be reusable for the next external benchmark rather than SPACE-specific.
Solution
One seam. SPACE's official code keeps everything it already owns — question loading, the
evaluation loop, answer parsing, scoring — and upstream
space.evaluate_qas.mainwrites theonly
results.jsonthat counts. DimOS enters at exactly one point:DimosQAAgentregisters inSPACE's own agent registry and replaces the model call — nothing else — with an
execute_single_caserun (the #3378 path,python_execon an empty kernel). Localcross-checks can only refuse a run, never alter a score.
Agent-eval contract — three new generic case kinds. No SPACE naming anywhere, so a future
external benchmark can reuse them unchanged:
none— nothing to materialize (no recording to replay);verbatim_prompt— the benchmark's prompt passes through untouched (any injectedinstruction would fight SPACE's own JSON format directive);
external_evaluator— no local oracle; scoring is delegated and recorded asbenchmark+revision.One change is not purely additive and is called out here deliberately:
CompactEvalResult.recording/progressrelax from required toOptional— environment-lesscases have neither. Old result files still parse; new files write
nullthere. Whether thisshould bump
schema_versionis left as a maintainer decision (see below).Scope. SPACE breaks down into 10 text QA tasks, 14 image QA tasks, and 4 interactive
task families; the image tasks need an image input channel this eval path does not have yet,
and the interactive ones need a stepwise multi-turn environment loop rather than one-shot QA.
So this PR integrates exactly the 10 text tasks — the other two slices ship as design only
(scope rationale in the #3399 comment; roadmap write-up available on request).
space_qapackage — the SPACE-specific side. A smallBenchmarkAdapterprotocol(
iter_items/to_case/score) turns benchmark rows into eval cases.BenchmarkItemhasno answer field, so the answer key cannot enter an
EvalCaseby construction. Subset drawingfollows the sampling design below — whole answer-order groups on a recorded seed. A generated
grid suite is the second adapter implementation and the zero-data test tier.
Data & provenance — fetched at run time, never committed. Nothing SPACE-derived enters the
repository. On first run, the tool clones
apple/ml-space-benchmarkat a pinned revision anddownloads the official 3.6 GB data release into the user cache, checking the tarball's SHA256;
a checkout with local edits, or off the pin, is refused. Every run writes a manifest naming
exactly what it consumed: task, seed (given or drawn), group count, the upstream row ordinals,
a SHA256 per question, digests for the archive and for the question file the run actually read,
and the dimos revision — recorded only when the package really runs from this source tree, so
an installed copy under some unrelated checkout cannot get signed with that checkout's HEAD.
Anyone holding the official release can re-identify from the manifest exactly which items a run
graded; the outputs sit beside it — per-question records and upstream's own
results.json.What SPACE ships, and what a run samples. Each of the ten tasks is a pool of stimuli,
and SPACE counterbalances every stimulus into four answer-order variants — four consecutive
rows in the released question file, the correct answer rotated into a different slot in each.
The pools are uneven: from 30 stimuli (
MapSketchingBEVText) to 150(
DirectionEstimationBEVText), 755 stimuli / 3,020 questions across the ten (per-tasksizes in the results table below). Scoring the whole release is tens of hours of agent time at
the observed cadence, so a run samples:
--groups Nsets how many stimuli it draws (4Nquestions),
--seedsets which ones — omitted, a seed is drawn, printed beside the score, andwritten to the manifest, so every run is reproducible from its record; passed explicitly, it
turns two runs into a comparison. Upstream itself has no sampling —
evaluate_qasscoresevery row it is handed — and
--groups <pool size>reproduces exactly that.Two properties of the release, both re-checkable by anyone holding it, decided how to sample:
follows the same ①→④ order — 600 of 600, 540 of 540, 120 of 120 rows — so a row-wise subset
has its answer in one fixed slot, and its score cannot be told apart from a slot preference.
Whole groups keep a stimulus's four variants together and make the answer-position
distribution flat by construction; they also fix the unit of evidence — 32 rows are 8
independent stimuli, and inference below is per stimulus.
Direction's 150stimuli are 30 scenes × 5 consecutive groups), so the first 8 groups of that file cover 2
scenes; a seeded random 8 covers ~7. The seed is a budget knob, not part of the benchmark.
Results — an integration-validation profile, not a controlled benchmark. All ten tasks
were scored at the same subset size —
--groups 8 --seed 20260808: 8 stimuli × 4 variants =32 questions per task, 320 questions, 80 of the release's 755 stimuli (10.6%) — model
gpt-5.6-lunathrough this path. Accuracy, exactly as SPACE's unmodified scorer wrote it toresults.json(infrastructure failures count as misses): 90.94% (291/320) — macro andmicro alike, since every task has the same n. The paper's GPT-4o macro-mean on the same ten tasks is
59.8 — context for scale, not a comparison this profile can support: the paper's baselines
are bare no-tool chat models while this path carries
python_exec, and these runs accumulatedover the course of development, against more than one locally configured OpenAI-compatible
endpoint (the full accounting is in the companion report).
SAtt_textMPFB_textMapSketchingBEVTextCBTT_textSAdd_textMRT_textDistanceEstimationBEVTextPTT_textJLO_textDirectionEstimationBEVTextCompleted: 315 of 320 — two transient HTTP 503s and three sessions that ended
terminated. Accuracy among completed replies: 92.38% (291/315). Malformed completedreplies: 0 — the upstream parser read an integer out of every reply that arrived.
Direction carries two of the five losses: 23 of its 30 completed answers were correct (76.67%).
Three observations worth carrying out of the per-item data (all of it in the companion report,
available on request):
JLO_text's six misses are one stimulus answered wrong on all four of its answer orders plustwo single-variant misses — a stable per-stimulus failure, not slot noise.
python_execcall, while the three BEV tasks account for 119 of the profile's 144.the same 32 Direction questions, same channel, behind a local patch not in this PR — 71.88
with the tool, 71.88 without, 37% faster without. Read it as no benefit detected, not no
benefit (8 stimulus groups, sign test p = 1.0; the two runs differ in more than the tool —
caveats in the companion report).
Coverage is the boundary on every claim above. Eight stimuli per task is a quarter of
MapSketching's pool and a nineteenth ofDirection's; nothing here is a full-benchmarkresult.
--groupsis the whole knob — the subset a task is given is everything that task ran,and full coverage is a spending decision, not a code change.
Known limitations (each disclosed on purpose; the first one wants your read):
python_execruns unsandboxed on the host that also holds the answer key on disk; droppingthe two pointer env vars is defence in depth, not isolation. Every completed transcript under
the scored-run tree was audited — 419 of them, carrying 199
python_execcalls: zerofilesystem, environment, subprocess, or network access, no imports beyond
numpy,math,itertools,collections. That certifies these runs, not the runner. The structural fix isa per-case tool surface (also what a paper-comparable no-tool run and vision need); until
then the benchmark leans on after-the-fact audit — whether that interim is acceptable is
your call.
combinations are not rejected at construction. A capability preflight is the right general
mechanism and is left as a follow-up rather than grown ad hoc here.
Infra fails N of Mnext to the accuracy it distorts, and an all-failures run is refused. Abounded per-question retry (original failure kept in the record) would re-ask exactly the
five questions this profile lost; left out because it changes what a score means.
the dimos tree (the SPACE checkout, by contrast, is refused outright when dirty), and the
run-root
cases.jsonlledger is written only after the three consistency gates — a refusedrun keeps its per-question records; only the aggregated file is missing.
One decision requested
schema_version: this PR changesCompactEvalResult.recording/progressfrom required tooptional (environment-less cases have neither, so new result files may hold
nullthere —a value the 1.0 shape never produced). Kept at 1.0 for now — readers that treat the two
fields as optional stay compatible, and this PR discloses the change here rather than hiding
it. Say the word and I will bump it to 1.1 in this PR.
Roadmap
Both remaining SPACE slices keep the one-seam invariant.
Vision is next, and is the prerequisite for the VQA benchmark that follows SPACE. One more
task kind (
multimodal_prompt, parts instead of a string); the Pi CLI already accepts imagefiles as positional
@filearguments, so the change concentrates in the transport layer. Oncethat channel exists, SPACE's own
*_visionvariants become a zero-integration regression loadfor it — same adapter, same scorer, different presentation.
Interactive (route retracing, shortcut discovery, MCT, CSWM) needs one decision before any
code: expose the benchmark environment as a tool inside one persistent session, or drive one
agent call per step from the upstream script. The trade-offs are in the companion report — I
would rather argue them against your pick than pre-empt it.
Smaller follow-ups — a configurable tool surface, the bounded infra retry, capability
preflight, a wording pass on the shared
python_execdescription — will each get their ownissue.
How to Test
Offline (no SPACE data, no network, no API key):
The feature itself (first run clones the pinned source and downloads the 3.6 GB release into the
user cache):
One command, four sizes:
Timing, measured on the runs behind the results: at
--workers 2a 32-question subset rangedfrom about a minute of wall clock (
SAtt_text) to about half an hour(
DirectionEstimationBEVText— the long BEV prompts dominate), and the full ten-task profilecame to roughly an hour and a half. The smoke and the offline tier are seconds-scale.
Scores print from upstream's own
results.json; the run directory holds the manifest,per-question records, and provenance. Docs:
docs/capabilities/agents/space-benchmark.md.AI assistance
Claude Code (Fable 5 and Opus 5) wrote most of the implementation under my direction, and the
branch went through several adversarial review rounds with Claude Code and Codex before
submission. I can explain every line in this diff.
Checklist