This is Symgliph's first public, paired study of context efficiency on real software-maintenance tasks. It compares full repository context, lexical BM25, embedding retrieval, and Symgliph's compiled context on the same 30 tasks and the same localization model.
The result supports the efficiency hypothesis, but it does not establish quality parity. Symgliph used 97.58% fewer prompt tokens on the mean paired task and 98.54% less localization-model spend in aggregate, while exact fix-path localization recall declined by 11.56 percentage points versus full context.
A separate v0.2 follow-up adds the symgliph_adaptive arm defined by SACP-0.2.
It records the complete budget ladder and stop reason per task under the same
pinned suite, model, and localization profile. Its result remains separate so
this v0.1 record stays reproducible; see the
adaptive study.
| Strategy | Mean token savings | Evidence recall (95% CI) | Fix-path recall | Retrieval cost | Localization cost |
|---|---|---|---|---|---|
| Full context | 0.00% | 100.00% (fixed) | 53.32% | $0.0000 | $4.5755 |
| Lexical BM25 | 95.82% | 58.55% (45.49–70.83%) | 46.87% | $0.0000 | $0.0974 |
| Embedding cosine | 95.78% | 56.49% (43.31–69.58%) | 43.34% | $0.2266 | $0.0971 |
| Symgliph compiled | 97.58% | 51.85% (38.88–65.12%) | 41.76% | $0.0000 | $0.0668 |
Costs are the provider-reported charges recorded during this run, in US dollars. Embedding retrieval cost includes cold corpus indexing and query embedding. Local BM25 and Symgliph graph construction used no paid API. A warm embedding index would have a materially lower retrieval cost, so cold and warm economics must not be conflated.
The compiled arm's $0.066826 localization spend was 68.47 times smaller than full context's $4.575497, a 98.54% reduction. It also achieved the best mean reciprocal rank (52.49%) and evidence precision (30.73%) of the three retrieval arms, but the lowest evidence recall and localization recall. Lexical BM25 is the strongest quality/cost baseline in this first study.
The suite is deterministically sampled from the Rust portion of
Multi-SWE-bench at commit
56ff018c04a38e27ada1e9d0a6d5839a51f88f0d.
| Repository | Tasks |
|---|---|
BurntSushi/ripgrep |
6 |
sharkdp/fd |
11 |
tokio-rs/bytes |
4 |
tokio-rs/tracing |
9 |
Ten selections were drawn from each declared sampling stratum: documentation, multi-Rust-file, and single-Rust-file changes. Strata may overlap; the resulting suite contains 10 documentation-linked, 16 multi-file, and 14 single-file tasks. Selection is without replacement in BLAKE3 task-ID order. Every task pins the repository's pre-fix commit and uses the upstream patch's changed file paths as objective labels.
The suite stores issue text, provenance URLs, revisions, and gold paths. It does not redistribute source repositories. The dataset card's license declaration does not replace the licenses and rights of each upstream repository; users who reproduce the run fetch those pinned sources directly.
All retrieval arms use a local o200k_base count, an 8,000-token budget, at
most 16 chunks or nodes, and the same issue-grounded query.
- Full context renders every supported non-binary repository file.
- Lexical BM25 ranks 512-token chunks with fixed
k1=1.2,b=0.75. - Embedding cosine ranks the same chunks using
openai/text-embedding-3-smallthrough OpenRouter. - Symgliph compiled uses BM25-ranked paths as routing hints, then expands verified semantic nodes and dependencies under the token budget.
openai/gpt-4.1-mini localizes files for every rendered context with
temperature zero, JSON-only output, a 1,024-token output cap, and reasoning
disabled. The API runner caches only responses whose model, profile, output
cap, and exact context BLAKE3 digest match.
- Evidence recall is the fraction of exact gold fix paths present in the context's ranked selected paths.
- Evidence precision is the fraction of selected paths that are gold.
- MRR is reciprocal rank of the first exact gold path.
- Localization recall/precision score the model's predicted paths, kept separate from retrieval evidence.
- Token savings is paired per task against that task's full context, then averaged.
- The 95% intervals use 10,000 deterministic paired task bootstrap resamples.
passed: true means that all strategy runs are structurally complete,
digest-valid, and repeatable under the study schema. It does not mean Symgliph
beat the baselines or met a quality noninferiority margin.
This study does not measure patch generation, tests passing, answer factuality, policy enforcement, warm-cache embedding economics, repeated model variance, or enterprise-private corpora. Thirty tasks across four Rust repositories are enough to falsify simplistic claims, not enough for a universal conclusion.
The suite, four raw run files, and generated report are checked in. This gate validates all three schemas, recomputes the report byte-for-byte, asserts the published result, and proves that a missing paired observation is rejected:
scripts/eceb-study-check.shThe raw artifacts live in
benchmarks/results/eceb-multiswe-rust-v0.1.
To recompute directly:
cargo run --features openrouter --bin symgliph-eceb-study -- evaluate \
benchmarks/eceb-multiswe-rust-v0.1.json \
benchmarks/results/eceb-multiswe-rust-v0.1/{full_context,lexical_bm25,embedding_cosine,symgliph_compiled}.jsonThis fetches pinned repository commits and spends OpenRouter credit:
cargo run --release --features openrouter --bin symgliph-eceb-study -- run \
benchmarks/eceb-multiswe-rust-v0.1.json \
--env-file /path/to/openrouter.env \
--localizer-model openai/gpt-4.1-mini \
--localizer-output-tokens 1024Use --task-id OWNER__repo-NUMBER to reproduce one task. API results can vary
over time even when prompts and provider-facing parameters are fixed; the
checked-in records preserve the exact observed model identity, usage, cost,
latency, predictions, context digest, and retrieval output.