Add mrplex verify — read-only integrity scrub - #20
Merged
Merged
Conversation
Read-only integrity scrub over the store: re-derives FTS/links/hashes and checks the version chain, reporting inconsistencies as structured findings. Six check families (chain, hash, frontmatter, fts, chunks, links), never writes, CLI/MCP/REST surfaces. Design decisions settled in §8. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add VerifyReport/VerifyFinding/VerifySpec/VerifySeverity wire types and a kernel.verify(ctx, spec) op returning a well-formed empty report. The verify module owns pre-flight (repo_not_found), scope narrowing, the finding accumulator (exact counts past the max_findings cap, min_severity filtering, skipped-check notes), and check selection by family/code. The six check families plug into runChecks in WS3. Threads embedderConfigured from the query-embed hook to gate the future chunks.unembedded check, and proxies verify through the auth shell as a read-only op. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add versions_all / documents_all / chunks_all_version_ids / backlog_all_version_ids to the Storage interface (keyset by id, the one place mrplex walks all versions incl. superseded), on both SQLite and Postgres adapters. The fts family is SQLite-only: correct the plan's §2.4 (FTS is a version-wide rowid bijection, not a live-set cover; Postgres fts_tsv is a generated column that can't drift). Add an optional VerifyFtsScans capability the SQLite adapter implements and Postgres omits; the kernel skips fts-with-note when absent. Membership reads from the fts_docs_docsize shadow table, since an external-content FTS5 table can't surface orphaned rowids via a plain scan. Add VerifyReport.checks_skipped to the wire shape. Parity + corruption tests in test/verify-scans.test.ts. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements chain, hash, frontmatter, fts, chunks, and links checks as pure functions over the WS2 scans, wired into runVerify. Per-repo families (chain/hash/frontmatter/links + chunks.unembedded) run in the repo loop; whole-store families (fts, chunks orphan/mixed-dim) run once and are skipped-with-note under a --repo filter. Notable corrections found during implementation: - chunks.orphan means a NONEXISTENT version, not a superseded one — the embed worker deliberately leaves chunks on superseded versions. - chunks/backlog/fts are global tables, not repo-partitioned. - fixed an infinite loop: the chain scan's keyset cursor advanced after a `continue` for orphan docs, stalling the page. Cursor now advances first. Adds versions_by_document + chunk orphan/dim scans to both adapters. 365 lines of corruption-injection tests prove each finding fires; full suite green (1133 passed). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Exposes kernel.verify through all three doors:
- CLI `mrplex verify` (local or remote via the client seam): --check
(repeatable), --severity, --max-findings, --json, --ci (exit 1 on any
finding at/above threshold). Repo optional — omit -r for a whole-store
scan, required for the whole-store fts/chunks orphan checks.
- MCP `verify` tool with a full outputSchema (25 tools now).
- REST GET /verify (whole-store) and GET /repos/{repo}/verify (scoped);
GET-only, no ETag (a scan is point-in-time, not cacheable).
Adds verify to KernelClient (local + remote-mcp) and a shared
renderVerifyReport used by both the MCP text channel and the CLI's
non-JSON output. Surface tests across cli/mcp/rest; full suite green
(1144 passed).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
README gains a `mrplex verify` blurb in "Versions, always" (the chain is the guarantee; verify confirms it) and a Verify row in the concepts table. Flips the design.md §11 verify bullet to Shipped with the as-built shape, noting where implementation refined the sketch (nonexistent-version orphans, whole-store fts/chunks checks, embedder-gated unembedded). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
mrplex verify, a read-only integrity scrub over the store — mrplex'sgit fsck. In an append-only store the version chain is the source of truth and every other table (FTS, links, chunks, embedding backlog) is a derived index that can silently drift;verifyre-derives what should be derivable, diffs it against what's stored, and reports inconsistencies as structured findings without ever writing. Implementsdocs/verify-plan.mdin full (WS1–WS5).kernel.verify(ctx, spec)returns aVerifyReport— findings are data, never exceptions, so a clean store returns an empty list and a corrupt one still returns a report. Six check families:contentHashvs. the storedcontent_hashfrontmatter_raw↔frontmatterround-trip,$-intrinsic leaksfts_docsrowid ↔ version bijection (skipped-with-note on Postgres, whosefts_tsvis a generated column that can't drift)extractEdgesvs. stored rows + resolution correctnessFindings never repair; each carries a
suggested_fixpointing at the relevant backfill.Surfaces
mrplex verify—--check(repeatable),--severity,--max-findings,--json,--ci(exit 1 on any finding at/above threshold). Repo optional: omit-rfor a whole-store scan, required for the whole-store fts/chunks-orphan checks.verifytool with fulloutputSchema(25 tools now).GET /verify(whole-store) andGET /repos/{repo}/verify(scoped); GET-only, no ETag.KernelClient(local + remote) and the auth shell (read-only, scope-narrowing).Notable design refinements found during implementation
fts_docs_docsizeshadow table; Postgres has no separate FTS structure. Theftsfamily is SQLite-only via an optionalVerifyFtsScanscapability.chunks/embedding_backlog/fts_docsare global tables — their orphan checks are whole-store, skipped-with-note under--repo.Cost
verifyis O(total versions) — it walks history, unlike everything else on the read path. Keyset-paginated throughout;--check/--reponarrow the work; findings capped with exact counts.Test plan
npm run typecheckcleanMRPLEX_TEST_POSTGRES_URL)test/verify-checks.test.ts, 21 cases)test/verify-scans.test.ts)test/cli-verify.test.ts,http-mcp,http-rest)MRPLEX_TEST_POSTGRES_URLset🤖 Generated with Claude Code