Skip to content

Add mrplex verify — read-only integrity scrub - #20

Merged
usergenic merged 6 commits into
mainfrom
verify-command
Sep 8, 2026
Merged

usergenic merged 6 commits into
mainfrom
verify-command

Conversation

@usergenic

Copy link
Copy Markdown
Owner

Summary

Adds mrplex verify, a read-only integrity scrub over the store — mrplex's git fsck. In an append-only store the version chain is the source of truth and every other table (FTS, links, chunks, embedding backlog) is a derived index that can silently drift; verify re-derives what should be derivable, diffs it against what's stored, and reports inconsistencies as structured findings without ever writing. Implements docs/verify-plan.md in full (WS1–WS5).

kernel.verify(ctx, spec) returns a VerifyReport — findings are data, never exceptions, so a clean store returns an empty list and a corrupt one still returns a report. Six check families:

  • chain — prev/next symmetry, one-current-per-doc, one-live-per-path, cycles, repo-id drift, orphan documents
  • hash — recompute contentHash vs. the stored content_hash
  • frontmatterfrontmatter_rawfrontmatter round-trip, $-intrinsic leaks
  • fts — SQLite fts_docs rowid ↔ version bijection (skipped-with-note on Postgres, whose fts_tsv is a generated column that can't drift)
  • chunks — embedding provenance (nonexistent-version orphans, backlog orphans, mixed-dim)
  • links — re-extractEdges vs. stored rows + resolution correctness

Findings never repair; each carries a suggested_fix pointing at the relevant backfill.

Surfaces

  • CLI mrplex verify--check (repeatable), --severity, --max-findings, --json, --ci (exit 1 on any finding at/above threshold). Repo optional: omit -r for a whole-store scan, required for the whole-store fts/chunks-orphan checks.
  • MCP verify tool with full outputSchema (25 tools now).
  • REST GET /verify (whole-store) and GET /repos/{repo}/verify (scoped); GET-only, no ETag.
  • Threaded through KernelClient (local + remote) and the auth shell (read-only, scope-narrowing).

Notable design refinements found during implementation

  • FTS is engine-asymmetric — SQLite membership is a whole-history rowid↔version bijection read via the fts_docs_docsize shadow table; Postgres has no separate FTS structure. The fts family is SQLite-only via an optional VerifyFtsScans capability.
  • "Orphan" means a nonexistent version, not a superseded one — the embed worker deliberately keeps chunks on superseded versions.
  • chunks / embedding_backlog / fts_docs are global tables — their orphan checks are whole-store, skipped-with-note under --repo.

Cost

verify is O(total versions) — it walks history, unlike everything else on the read path. Keyset-paginated throughout; --check/--repo narrow the work; findings capped with exact counts.

Test plan

  • npm run typecheck clean
  • Full suite green: 1144 passed, 2 Postgres suites skipped (no MRPLEX_TEST_POSTGRES_URL)
  • Corruption-injection tests prove each finding fires (test/verify-checks.test.ts, 21 cases)
  • Adapter parity + corruption scans (test/verify-scans.test.ts)
  • CLI / MCP / REST surface tests (test/cli-verify.test.ts, http-mcp, http-rest)
  • Postgres parity for the new adapter methods is written but unverified locally — needs a CI run with MRPLEX_TEST_POSTGRES_URL set

🤖 Generated with Claude Code

usergenic and others added 6 commits September 4, 2026 17:41
Read-only integrity scrub over the store: re-derives FTS/links/hashes
and checks the version chain, reporting inconsistencies as structured
findings. Six check families (chain, hash, frontmatter, fts, chunks,
links), never writes, CLI/MCP/REST surfaces. Design decisions settled
in §8.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add VerifyReport/VerifyFinding/VerifySpec/VerifySeverity wire types and a
kernel.verify(ctx, spec) op returning a well-formed empty report. The
verify module owns pre-flight (repo_not_found), scope narrowing, the
finding accumulator (exact counts past the max_findings cap, min_severity
filtering, skipped-check notes), and check selection by family/code. The
six check families plug into runChecks in WS3.

Threads embedderConfigured from the query-embed hook to gate the future
chunks.unembedded check, and proxies verify through the auth shell as a
read-only op.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add versions_all / documents_all / chunks_all_version_ids /
backlog_all_version_ids to the Storage interface (keyset by id, the one
place mrplex walks all versions incl. superseded), on both SQLite and
Postgres adapters.

The fts family is SQLite-only: correct the plan's §2.4 (FTS is a
version-wide rowid bijection, not a live-set cover; Postgres fts_tsv is a
generated column that can't drift). Add an optional VerifyFtsScans
capability the SQLite adapter implements and Postgres omits; the kernel
skips fts-with-note when absent. Membership reads from the fts_docs_docsize
shadow table, since an external-content FTS5 table can't surface orphaned
rowids via a plain scan.

Add VerifyReport.checks_skipped to the wire shape. Parity + corruption
tests in test/verify-scans.test.ts.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements chain, hash, frontmatter, fts, chunks, and links checks as pure
functions over the WS2 scans, wired into runVerify. Per-repo families
(chain/hash/frontmatter/links + chunks.unembedded) run in the repo loop;
whole-store families (fts, chunks orphan/mixed-dim) run once and are
skipped-with-note under a --repo filter.

Notable corrections found during implementation:
- chunks.orphan means a NONEXISTENT version, not a superseded one — the
  embed worker deliberately leaves chunks on superseded versions.
- chunks/backlog/fts are global tables, not repo-partitioned.
- fixed an infinite loop: the chain scan's keyset cursor advanced after a
  `continue` for orphan docs, stalling the page. Cursor now advances first.

Adds versions_by_document + chunk orphan/dim scans to both adapters.
365 lines of corruption-injection tests prove each finding fires; full
suite green (1133 passed).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Exposes kernel.verify through all three doors:
- CLI `mrplex verify` (local or remote via the client seam): --check
  (repeatable), --severity, --max-findings, --json, --ci (exit 1 on any
  finding at/above threshold). Repo optional — omit -r for a whole-store
  scan, required for the whole-store fts/chunks orphan checks.
- MCP `verify` tool with a full outputSchema (25 tools now).
- REST GET /verify (whole-store) and GET /repos/{repo}/verify (scoped);
  GET-only, no ETag (a scan is point-in-time, not cacheable).

Adds verify to KernelClient (local + remote-mcp) and a shared
renderVerifyReport used by both the MCP text channel and the CLI's
non-JSON output. Surface tests across cli/mcp/rest; full suite green
(1144 passed).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
README gains a `mrplex verify` blurb in "Versions, always" (the chain is
the guarantee; verify confirms it) and a Verify row in the concepts table.
Flips the design.md §11 verify bullet to Shipped with the as-built shape,
noting where implementation refined the sketch (nonexistent-version
orphans, whole-store fts/chunks checks, embedder-gated unembedded).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@usergenic
usergenic marked this pull request as ready for review September 8, 2026 20:26
@usergenic
usergenic merged commit a9be36b into main Sep 8, 2026
1 of 2 checks passed
@usergenic
usergenic deleted the verify-command branch September 8, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant