Proposal: per-repository sealed storage with a pluggable backend and prebuilt vector indexes #48

Staging draft. This issue lives on the fork while the study and implementation continue. It will be updated as results arrive, then filed on zzet/gortex once the design is chosen and the implementation plan is ready.

Is your feature request related to a problem?

Gortex keeps every tracked repository in one SQLite store that changes in place. Reported effects:

• WAL checkpoint path appears to stall on large stores, forcing full re-index far below documented throughput zzet/gortex#843: on a 16.95 GiB store, WAL checkpoints stall and a full re-index runs at about 1 file/s, against the documented 143 to 300 files/s.
• v0.64.0: sustained write amplification (~549 GB/day) — non-converging dirty/sha reconciliation with worktree-tracking state (store 2.16× legitimate size) zzet/gortex#767: one daemon wrote 549.76 GB in 13.55 hours for a repository of about 1,700 files. The store was 2.16 times its legitimate size.
• Improve incremental indexing and resumable contract analysis zzet/gortex#806, merged on 2026-10-23, lists as a known limit that dirty-generation retirement falls behind at high edit rates (store grew from 8.1 to 19.6 GB).
• In After #473: optional PostgreSQL storage for shared Gortex deployments? zzet/gortex#512, SQLite is described as having "quickly became a bottleneck" with parallel agents, and an alternative storage layer as possible "after completing Improve incremental indexing and resumable contract analysis zzet/gortex#806".

Describe the solution you'd like

Build on the capability interfaces from zzet#806:

1. One folder per repository, sealed versions. Each indexed version is written once and never changed in place. A small catalog records which version each repository serves. Publishing is one atomic pointer update, and the previous version stays as a fallback.
2. A storage-neutral contract with one shared conformance suite. SQLite stays the embedded default. Postgres or another backend implements the same contract and passes the same tests.
3. Vector search with two methods, chosen automatically. Each repository keeps its full-precision vectors and an 8-bit copy. Below a latency budget, an 8-bit exact scan with full-precision re-scoring gives exact results. Above it, a USearch index, built in the background and opened straight from the file, answers instead. A built-in eval tool measures both on the user's machine and repositories and suggests the switch point. Every default can be overridden in config. Exact scan is also the fallback when no index is present.
4. Optional reuse of a CI-built index for a commit the user has already cloned. The source stays local, so tools that read source keep working. Enabled by a config opt-in (design pending).
5. Replace the current store, not run beside it. The change lands as a short stack of PRs: the storage foundation, the indexer build path, daemon and MCP serving, vectors, then the default switch. When the new storage becomes the default, the old store is deprecated but stays selectable for at least one release. A background migration rebuilds each repository from source while the old store keeps serving, reuses stored vectors only when their recipe matches, checks the result against the old store, and then switches in one step. The proposal is to start it from gortex upgrade, which already runs post-upgrade steps, with a gortex storage migrate command for manual control. Rolling back means selecting the old store again.

How it fits together

Today, every repository shares one store that changes in place:

flowchart LR
  subgraph today["Today: one shared store"]
    A0["repo A"] --> DB[("store.sqlite<br/>every repository<br/>changed in place")]
    B0["repo B"] --> DB
    C0["repo C"] --> DB
  end

Proposed, one folder per repository with sealed versions and a small catalog:

flowchart LR
  CAT[("catalog<br/>current version<br/>per repository")]
  subgraph A["repo A folder"]
    A1["version 1, sealed"]
    A2["version 2, sealed"]
  end
  subgraph B["repo B folder"]
    B1["version 1, sealed"]
  end
  CAT -- "serves" --> A2
  CAT -. "fallback" .-> A1
  CAT -- "serves" --> B1

Publishing a new version is one atomic pointer switch:

sequenceDiagram
  participant I as Indexer
  participant F as repo A folder
  participant C as catalog
  participant Q as Queries
  I->>F: write version 3 and seal it
  I->>C: switch repo A from version 2 to 3 (one atomic update)
  Q->>C: which version does repo A serve?
  C-->>Q: version 3 (version 2 kept as fallback)

Every backend sits behind the same contract and passes the same tests:

flowchart TD
  T["MCP tools and CLI"] --> CI["capability interfaces from zzet/gortex#806"]
  CI --> S["SQLite segments<br/>embedded default"]
  CI --> P["Postgres or another backend"]
  CT["one shared conformance suite"] -. "tests" .-> S
  CT -. "tests" .-> P

Reusing a CI-built index (opt-in):

sequenceDiagram
  participant CI
  participant A as Release artifacts
  participant U as User machine
  CI->>CI: index commit X, embed symbols, build USearch file
  CI->>A: upload index and vectors, signed
  U->>U: clone commit X (source stays local)
  U->>A: download the index for commit X (opt-in)
  U->>U: check signature, commit, model and USearch version
  alt all match
    U->>U: open the index, then apply local edits
  else any mismatch
    U->>U: index locally instead
  end

Use case

The daemon and MCP tools for users who track many or large repositories, and teams that want CI to prepare indexes.

Evidence so far

The study is in progress. It uses public repositories only, and Gortex builds of current main plus our open PRs.

Vector search, 10,000 vectors of 384 dimensions, 1,000 queries, recall@10 against exact results, worst of 5 builds:

How to read the table:

• Recall@10: of the 10 truly nearest vectors, the share the index returned. 1.00 means all 10. Higher is better.
• Exact scan compares the query with every vector, so it is correct by definition. It is the baseline the other rows are measured against.
• Query p50: the median time for one query, with one reader. Lower is better.
• Build: the time to build and save the index for these 10,000 vectors. Lower is better.
• File size: the index on disk, including the stored vectors.
• 384 dimensions: the size of the MiniLM embeddings that Gortex uses for semantic search.
• M: links per node in an HNSW graph. More links usually raise recall, but cost build time and disk.
• Worst of 5 builds: a graph index can come out differently on each build, so each was built 5 times and the lowest recall is shown.
• Search width (efSearch): how many candidates a graph search examines. Wider is more accurate and slower.
• Clustered vectors: the synthetic set has 32 tight groups. A graph index can get stuck inside one group and miss neighbors in another, so this set is a hard case.

What it means for Gortex users: at 10,000 vectors, USearch matches the exact results and answers about 60 times faster in these timings, with a 10% larger file. At 100,000 vectors it needs a wider search to stay above 0.99. coder/hnsw is accurate at M=32, but much slower to build.

• coder/hnsw v0.6.1, its latest release, built disconnected graphs on clustered vectors. Fixes merged later (fix: heap Max() and PopLast() return wrong element coder/hnsw#20, fix: search termination and result set bounded by k instead of efSearch coder/hnsw#22) are not released yet.
• At 100,000 vectors, exact scan stays at recall 1.00 with a 50 ms p50. USearch at M=16 reaches 0.90 at its default search width. Raising only the search width to 512, on the same saved indexes, gives 0.997 to 0.999 on all 5 builds. coder/hnsw main at M=32 reaches 0.99 but takes about 13 minutes per build.
• The synthetic vectors form 32 tight clusters, which is hard for any graph index.
• Real code embeddings (voyage-code-4, 1,024 dimensions, about 10,000 symbols in each of 3 public repositories, 1,000 queries built from doc comments): USearch at M=32 and search width 1,024 reached worst-build recall of 0.997 (Go), 0.982 (Rust) and 0.996 (TypeScript). Fetching 50 to 200 candidates and re-scoring them exactly did not help, because the missed results were never among the candidates. An 8-bit exact scan with full-precision re-scoring of the top 128 returned every true top-10 result for every query, at about 6 ms per 10,000 vectors on one core. On about 100,000 real vectors from one Go repository, the 8-bit scan with re-scoring again matched the exact results on all 1,000 queries, and USearch at M=32 and search width 1,024 reached a worst-build recall of 0.994.
• Embedding model quality on the same doc-comment queries (top-10 recall, first with the symbol text Gortex builds today, then with the symbol's code body added): MiniLM, the default for the local provider, 54.9% and 59.9%. Gortex's overall default is static GloVe word vectors, which the study did not measure. CodeRankEmbed (137M parameters, runs locally on CPU) 78.8% and 86.0%. OpenAI text-embedding-3-large 80.5% and 89.2%. On a smaller 5,000-symbol TypeScript sample, nomic-embed-code (7B, 4-bit) reached 83.2% and 91.6%; its 8-bit and 16-bit versions scored within 0.3 points of it. Chosen direction: CodeRankEmbed as the local default model, OpenAI text-embedding-3-large as a hosted opt-in (it sends code to a third party), nomic-embed-code 7B as an opt-in for users with a GPU, and the symbol's code body added to the embedded text. MiniLM stays selectable.
• Embedding in process: CodeRankEmbed and nomic-embed-text-v1.5 export to ONNX with Hugging Face Optimum, and their vectors match PyTorch (cosine above 0.9999999). Loaded from Go through ONNX Runtime, with the weights in an external file and prepacking off, a session starts at about 50 MB resident. Dynamic 8-bit quantization cost CodeRankEmbed 0.3 to 1.5 points of top-10 recall on the TypeScript sample.
• Storage formats, correctness first, on one Kubernetes repository (client-go: 81,179 nodes, 448,952 edges, 694 queries). Every answer is checked against an exact reference, with readers running during publication and 24 kill-during-publish trials per candidate. Passed: SQLite segments, Pebble with Bleve, bbolt with Bleve, Badger with Bleve, all five catalog forms (SQLite, ref files, bbolt, Pebble, manifest with lock file), exact vector scan and USearch. Failed: libSQL, whose built-in vector index fell below the 0.8 recall floor. Next: the same screen on 10 repositories (423,567 nodes, 2.3 million edges), then timings for the finalists.
• Current store, for comparison (provisional; it will be measured once more on one pinned build of main plus our open PRs). Stopped SQLite store size and time to track all members:

• SQLite identity reindex scans the whole generation when planner statistics describe a much smaller store zzet/gortex#987: a long-lived store that collected planner statistics on a small corpus later picks a slow plan for the identity-refresh reindex (125 CPU seconds instead of about 33). fix(sqlite): keep identity-list joins on the edge key under stale statistics zzet/gortex#990, merged on 2026-10-25, fixes it. A sealed segment can collect its statistics once, when it is sealed, so they always match its data.
• Timings come from a shared laptop and are noisy. One session on a dedicated cloud machine will time the finalists, and the draft PR's CI will repeat the comparison.

Plan

• A stack of draft PRs, each building and passing CI on its own, with CI jobs that compare the current store and this proposal on public repositories: index time, store size, bytes written, query latency and recall.
• A CI job that builds a prebuilt index in one job and reuses it in another, to show the time saved.

Open questions

• Which capability interfaces from Improve incremental indexing and resumable contract analysis zzet/gortex#806 a sealed backend must implement first.
• The opt-in and trust model for downloading a prebuilt index, including signature checks before opening a file.
• USearch packaging: a static build links on macOS arm64. The native library adds about 385 KB, and the binary needs no separate USearch library. Linux static linking and the Windows MinGW build are not tested yet; the draft PR's CI will test them.

Related upstream items

• Make every read tool's latency independent of how many other repos the store holds zzet/gortex#1003: "Make every read tool's latency independent of how many other repos the store holds". One folder per repository addresses this directly.
• Grow the SQLite reader pool, split interactive from background reads, then raise the dispatch cap zzet/gortex#1045: a SQLite reader pool for the current store.
• Add a re-track measurement harness and publish large-store re-track baselines zzet/gortex#1055: a re-track measurement harness and large-store baselines. The study's current-store numbers use the same kind of measurements.