From 8639be028e3d3d11a92fcaa8d88796be59ed16db Mon Sep 17 00:00:00 2001 From: Brendan Baldwin Date: Fri, 4 Sep 2026 17:41:57 -0600 Subject: [PATCH 1/8] Add mrplex verify implementation plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Read-only integrity scrub over the store: re-derives FTS/links/hashes and checks the version chain, reporting inconsistencies as structured findings. Six check families (chain, hash, frontmatter, fts, chunks, links), never writes, CLI/MCP/REST surfaces. Design decisions settled in §8. Co-Authored-By: Claude Opus 4.7 --- docs/verify-plan.md | 225 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 docs/verify-plan.md diff --git a/docs/verify-plan.md b/docs/verify-plan.md new file mode 100644 index 0000000..a5a242b --- /dev/null +++ b/docs/verify-plan.md @@ -0,0 +1,225 @@ +# Verify Implementation Plan — `mrplex verify`, an integrity scrub over the store + +Target: the **`mrplex verify`** bullet in [design.md §11](archive/design.md) ("Future work"): + +> **`mrplex verify`.** Integrity scrub over the version chain: walk each document oldest-to-newest, recompute body/frontmatter hashes, confirm `frontmatter_raw` ↔ `frontmatter` round-trips byte-exact (§3.2), check `prev_id`/`next_id` symmetry, verify FTS/chunk/link derived tables against their source versions, report orphans. No writes. CLI + kernel op + optional CI mode that exits non-zero on any inconsistency. Cheap insurance for an append-only store where the chain *is* the guarantee. + +In an append-only store the version chain **is** the source of truth, and every other table (`fts_docs`, `chunks`, `embedding_backlog`, `links`) is a derived index that can silently drift out of agreement with it — a bad migration, a hand-edited DB, a bug in in-tx maintenance, a partial crash. Nothing today detects that drift. `verify` is mrplex's `git fsck`: a read-only scan that re-derives what should be derivable and diffs it against what's stored, plus a set of structural-invariant checks that the partial indexes (§3.2) are *supposed* to make impossible but which a corrupted database can still violate. + +The logic mostly exists already, scattered: `test/invariants.test.ts` asserts the index invariants at the storage layer; `contentHash` (`src/markdown/content-hash.ts`) and `extractEdges` (`src/links/extract.ts`) are the pure re-derivation functions; `split`/`join` (`src/markdown/frontmatter.ts`) drive the round-trip. `verify` is chiefly **wiring these into a runtime kernel op + CLI command**, held to SQLite/Postgres parity by the shared kernel suite (§7.2). + +Branch `verify` is cut from `main`. + +## 1. Scope + +**In:** + +- **`kernel.verify(ctx, spec)`** — a new read-only kernel op returning a structured `VerifyReport` (findings + counts), never throwing on inconsistency (a finding is data, not an error). Kernel-level so all three surfaces reach it; CLI is the primary consumer. +- **Six check families** (§2), each independently toggleable, each emitting `Finding` rows with a stable `check` code, severity, the offending `version_id`/`document_id`/`path`, and a `detail` payload. +- **A read-only adapter surface** (§4) — new `Storage` methods that stream rows for scanning (whole version chains, derived-table membership) without materializing the corpus. Keyset-paginated, id-ordered, on both adapters. +- **`mrplex verify` CLI** (§5) — human table + `--json` structured output; `--ci` exits non-zero on any finding at or above a threshold severity; `--repo`, `--check`, `--severity` filters. +- **Scope-respecting.** Like every read op, `verify` narrows to the caller's read claims (§8.2): a scoped caller verifies only the slice it can read. `--unsafe` / full-trust verifies everything. (§2.7 covers the one wrinkle: derived-table orphans that reference *unreadable* versions.) +- **SQLite + Postgres parity.** Same findings on the same corrupted fixture across both engines; enforced by the shared kernel suite. + +**Out (deliberately):** + +- **Repair.** `verify` never writes. Fixing what it finds is a separate concern: derived-index drift is repaired by the existing backfills (`mrplex links backfill`, `mrplex embed backfill`, `mrplex hash backfill`); structural chain corruption is not auto-repairable and warrants a human (or a future `mrplex repair` that re-ingests into a fresh DB). The plan defines a **`suggested_fix` hint** per finding (§3) so the report *points at* the remedy without performing it. +- **Cross-repo / cross-database checks.** Links are repo-local (§11.2); `verify` is too. Comparing two databases (e.g. a Postgres follower vs. its primary) is a replication concern, not this. +- **Semantic/vector *quality* checks.** `verify` confirms a chunk row's *existence and provenance* (belongs to a live version, model recorded), not that the embedding vector is "good." Vector correctness isn't checkable without re-running the hook, and the hook is non-deterministic across models. +- **Filesystem/sync-state checks.** The sync daemon's on-disk `$version`/`$content_hash` provenance and cursor files are a client concern (§4 sync), not store integrity. A future `mrplex sync verify` could diff a vault against the store; out of scope here. +- **Performance guarantees on pathological chains.** `verify` is O(total versions) — it walks history, unlike everything else in the read path which is O(live set). This is inherent (see §6) and documented, not optimized away in v1. + +## 2. The six check families + +Each family has a stable `check` code (the string clients discriminate on) and a default severity. Severity is one of `error` (a real inconsistency — the store is lying) or `warn` (suspicious but possibly benign — e.g. a legacy row a backfill would fix). A `--ci` run fails on `error` by default; `--severity warn` lowers the bar. + +### 2.1 `chain` — version-chain structural integrity + +Per document, walk `version_history` oldest→newest and confirm the doubly-linked `prev_id`/`next_id` chain is well-formed (design §3.2: *"writing a new version Y with `prev_id = X` also sets `X.next_id = Y` in the same transaction"*). This is the family that catches what the partial indexes can't (a directly-corrupted DB, an FK left dangling by a bad migration). + +| `check` code | Severity | Condition | +|---|---|---| +| `chain.prev_next_asymmetry` | error | `X.next_id = Y` but `Y.prev_id ≠ X` (or vice-versa) — the inverse-link invariant broke. | +| `chain.multiple_current` | error | A document has >1 version with `next_id IS NULL`. The partial unique index (§3.2) should forbid this; a finding means the index is missing/corrupt. | +| `chain.no_current` | error | A document has ≥1 version but none with `next_id IS NULL` — a headless chain. | +| `chain.broken_prev` | error | A non-root version's `prev_id` points at a nonexistent version, or one in a different document. | +| `chain.cycle` | error | The `prev_id` walk revisits a version — a loop, not a chain. | +| `chain.repo_mismatch` | error | A version's `repo_id` disagrees with its document's `repo_id` (the denormalized column §3.2 drifted). | +| `chain.orphan_document` | warn | A `documents` row with zero versions. Benign-ish (invisible to every query) but shouldn't exist. | +| `chain.multiple_live_at_path` | error | Two live versions share `(repo_id, path_norm)` — the second partial unique index (§3.2) broke. | + +### 2.2 `hash` — content-fingerprint fidelity + +Recompute `contentHash(frontmatter_raw, body)` (`src/markdown/content-hash.ts`) for every version and compare to the stored `content_hash` column. This is the fingerprint sync relies on (`$content_hash` clean-state detection) — a mismatch means an outside writer or a bug corrupted it. + +| `check` code | Severity | Condition | +|---|---|---| +| `hash.mismatch` | error | Stored `content_hash ≠` recomputed. `detail: { stored, computed }`. | +| `hash.missing` | warn | `content_hash IS NULL` on a row written before migration 0002 (§2.6). `suggested_fix: "mrplex hash backfill"`. | + +Only the two families that check *stored derived scalars* (`hash`, and `links` resolution) need to re-run pure functions; both are cheap CPU. + +### 2.3 `frontmatter` — raw ↔ parsed round-trip + +Design §3.2 stores frontmatter twice by design: `frontmatter_raw` (byte-verbatim YAML) and `frontmatter` (parsed JSON, the query index). `verify` re-parses `frontmatter_raw` and confirms it still yields the stored `frontmatter` JSON — catching a YAML-parser upgrade that changed semantics, or a write that let the two diverge. + +| `check` code | Severity | Condition | +|---|---|---| +| `frontmatter.parse_error` | error | Stored `frontmatter_raw` no longer parses as YAML at all. | +| `frontmatter.divergence` | error | Re-parsed raw ≠ stored `frontmatter` JSON (deep-equal). `detail: { keys_differing }`. | +| `frontmatter.system_leak` | error | A `$`-prefixed key is present in stored `frontmatter_raw`/`frontmatter` — `$*` intrinsics must be stripped at write time (`canonicalizeFrontmatter`) and never persisted (sync/history §2.4). A leak corrupts `$content_hash` and re-injection. | + +### 2.4 `fts` — full-text index membership + +The FTS index (`fts_docs`, external-content mode, trigger-maintained — `migrations/0003_fts_docs.sql`) must cover **exactly the current versions' bodies**: one FTS row per live version, none for superseded/deleted-out-of-namespace versions, none orphaned. + +| `check` code | Severity | Condition | +|---|---|---| +| `fts.missing` | error | A live version has no FTS row. `suggested_fix: "rebuild FTS (reindex)"`. | +| `fts.orphan` | error | An FTS row references a version that isn't live (or doesn't exist). | + +`fts.missing` / `fts.orphan` need only id-set membership (which live versions have/lack an FTS row), so they hold at full SQLite/Postgres parity. A body-content freshness check (`fts.stale_body` — "the indexed text matches the live body") is **deliberately not in v1**: SQLite's FTS5 runs in external-content mode and doesn't store the body redundantly (the `versions` table is the content source), so there's no cheap way to compare stored FTS text on SQLite, and a parity-breaking Postgres-only check isn't worth it here. Deferred; additive if a cheap path appears. + +### 2.5 `chunks` — embedding provenance + +Chunks + the embedding backlog (§5.3). `verify` checks *structural* consistency, not vector quality (§1 Out): + +| `check` code | Severity | Condition | +|---|---|---| +| `chunks.orphan` | error | A `chunks` row references a non-live or nonexistent version. | +| `chunks.backlog_orphan` | error | An `embedding_backlog` row references a nonexistent version. | +| `chunks.unembedded` | warn | A live version has neither chunk rows nor a backlog entry — it fell out of the embedding pipeline. `suggested_fix: "mrplex embed backfill"`. **Only runs when an embedder is configured** (see below); otherwise skipped entirely, not reported clean. | +| `chunks.mixed_dim` | error | Chunk rows for one version carry vectors of differing dimensionality (§5.3 "refuse mixed-dim writes" — a finding means that guard was bypassed). | + +**Embedder-gated coverage.** `chunks.unembedded` is meaningful only when embedding is actually intended for this store. A corpus that never configured an embedder has *every* live version "unembedded" — that's noise, not a finding. The check therefore runs **only when an embedder is configured**, resolved through the standard precedence (flag → `MRPLEX_EMBEDDER` env → config `embedder` → none; the §"Configuration" resolution the whole CLI already uses). No embedder configured ⇒ `chunks.unembedded` is *skipped* (the report notes it was skipped for lack of an embedder — not silently omitted, not reported clean). The orphan/provenance checks (`chunks.orphan`, `chunks.backlog_orphan`, `chunks.mixed_dim`) are unconditional: stray chunk rows referencing dead versions are a real inconsistency whether or not an embedder is *currently* wired up (they'd be residue from a past one). + +### 2.6 `links` — link index vs. re-extraction + +Re-run `extractEdges({ body, frontmatter, config })` (`src/links/extract.ts`) for each live version under the repo's **effective link-config** (`effectiveLinkConfig`, §11.2 cascade) and diff the resolved edge set against the stored `links` rows. Then check resolution correctness against the live path set — the class of bug that would make `graph` / `$backlinks()` quietly wrong. + +| `check` code | Severity | Condition | +|---|---|---| +| `links.set_mismatch` | error | Re-extracted `(ord, field, target_raw)` set ≠ stored rows for the source. `detail: { missing, extra }`. In-tx maintenance drifted from extraction. | +| `links.misresolved_dangling` | error | An edge is `target_id IS NULL` but a live document *does* exist at its folded `target_norm` — a dangler that should have bound (missed `links_resolve_dangling`). | +| `links.misresolved_bound` | error | An edge's `target_id` points at a document whose current path's fold ≠ the edge's `target_norm`, **or** at a nonexistent/non-live document. Identity binding went stale in a way renames alone can't explain. | +| `links.self_link` | warn | `source_id == target_id` — excluded by construction (§11.2 `source_id <> document_id`); a finding means one slipped in. | +| `links.deleted_source_has_outbound` | error | A document currently in the system namespace (`:deleted/…`) still has outbound `links` rows — `docs.delete` should `links_clear` them (§11.2). | + +### 2.7 Scope interaction (all families) + +`verify` respects read scope (§8.2), which creates one honest subtlety: a **derived-table orphan** (`fts.orphan`, `chunks.orphan`, `links.misresolved_bound`) may reference a version the caller *can't read*. Rule: a scoped caller sees an orphan finding only when it can read the referenced version; otherwise the row is silently dropped (same posture as `links.stale`, kernel.ts:753 — *"surface a stale link only when the caller can read both endpoints"*). The finding is still discoverable by a full-trust (`--unsafe`) verify, which is the intended operator context for integrity scrubs anyway. Document this so a scoped `verify` reporting "clean" is understood as "clean within your scope," not "clean globally." + +## 3. Report shape (wire types) + +New in `src/kernel/wire.ts`: + +```ts +export type VerifySeverity = "error" | "warn"; + +export type VerifyFinding = { + check: string; // stable code, e.g. "chain.prev_next_asymmetry" + severity: VerifySeverity; + repo: string; // slug + document_id?: string; // opaque, when the finding is doc-scoped + version_id?: string; // opaque, when version-scoped + path?: string; // the offending version's path, when known + detail: Record; // check-specific payload (stored vs computed, etc.) + suggested_fix?: string; // human hint, e.g. "mrplex hash backfill" — never auto-run +}; + +export type VerifyReport = { + findings: VerifyFinding[]; + counts: { + versions_scanned: number; + documents_scanned: number; + by_check: Record; // findings per check code + by_severity: Record; + }; + truncated: boolean; // true if max_findings capped the list (counts stay exact) +}; +``` + +Design notes: +- **Findings are data, never exceptions.** `verify` returns a report even when the store is on fire; the only throws are the usual pre-flight ones (`repo_not_found` for a bad `--repo`, `forbidden` for scope). This mirrors `links.repair` returning `{ repaired, skipped }` rather than throwing on a skip. +- **`counts` stay exact even when `findings` is truncated.** A corpus with a million broken rows shouldn't OOM the report; `max_findings` (default e.g. 10_000) caps the emitted list but the scan still tallies `counts` and sets `truncated: true`. The operator learns the true scale and re-runs with `--check X` to enumerate one family. +- **Opaque ids only.** `document_id`/`version_id` cross the wire as encoded strings (`encodeVersionId`), consistent with §3.3 — internal integer ids never leak. + +`VerifySpec` (input): + +```ts +export type VerifySpec = { + repo?: string; // omitted = every repo the caller can see + checks?: string[]; // family prefixes ("chain", "links") or full codes; omitted = all + min_severity?: VerifySeverity; // filter findings below this (counts still full); default "warn" + max_findings?: number; // cap emitted findings; default 10_000 +}; +``` + +## 4. Adapter surface + +`verify` reads a lot but must not materialize the corpus. New `Storage` methods (both adapters, parity-tested), all keyset-paginated by id and read-only: + +- **`versions_all(opts: { repo_id?; after_id; limit })`** → full `VersionRow[]` in id order. The backbone scan for `hash`, `frontmatter`, and `links` re-derivation (they need `frontmatter_raw`, `frontmatter`, `body`). This is the one place mrplex walks *all* versions including superseded ones; keyset by id so batches don't re-scan. +- **`documents_all(opts: { repo_id?; after_id; limit })`** → `{ id, repo_id }[]` for the `chain` family's per-document walk and `chain.orphan_document`. Existing `version_history(document_id)` walks each chain. +- **`fts_all_refs(opts)`** → `{ version_id, has_row: bool, text_hash? }` sufficient to compute `fts.missing`/`fts.orphan`/`fts.stale_body` by joining against the live set. (SQLite external-content FTS makes stored-text retrieval awkward — if `stale_body` can't be done cheaply, the method returns `text_hash: null` and the check is skipped with a one-line report note, not a silent omission.) +- **`chunks_all_version_ids(opts)`** and **`backlog_all_version_ids(opts)`** → the id sets for `chunks.*` orphan/coverage checks; intersect with live-version ids in the kernel. +- **Reuse existing** `links_by_repo(repo_id)` (already returns every link row ordered by `(source_id, ord)` — tests use it) and `versions_live_by_repo(repo_id)` for the `links` family; `versions_current_by_documents` to resolve `target_id` → current path for `links.misresolved_bound`. + +No new indexes required — every scan is either an existing partial index (live set) or a full id-ordered table walk (history), which is acceptable for an operator-invoked scrub (§6). + +The kernel op composes these behind `kernel.verify`, applies scope (§2.7), runs the pure re-derivation functions (`contentHash`, `extractEdges`, YAML parse via `frontmatter.ts`), and assembles the `VerifyReport`. The heavy comparison logic lives in a new `src/kernel/verify/` module (mirroring `src/kernel/query/`), pure and unit-testable against hand-built corrupted fixtures. + +## 5. Surfaces + +### CLI — `mrplex verify` + +``` +mrplex verify [--repo ] [--check ]... [--severity error|warn] + [--max-findings ] [--json] [--ci] +``` + +- Default: human table grouped by `check`, a summary line (`scanned N versions across M docs; K findings (E error, W warn)`), and per-finding `path` + `detail`. +- `--json`: the full `VerifyReport` (the MCP/REST structured shape), for piping. +- `--ci`: exit non-zero when any finding at or above the threshold severity exists. Reuses the exit-code families (`src/cli/exit-codes.ts`): a clean run exits 0; findings exit **1** (validation family — "the data failed validation"). A pre-flight `repo_not_found` still exits 4, `forbidden` exits 3, unchanged. This keeps `mrplex verify --ci` a drop-in CI gate. +- `--check` repeatable; accepts a family prefix (`chain`) or a full code (`links.set_mismatch`). + +Registered under the top-level program alongside `hash`/`links`/`embed` (main.ts) — it's a maintenance command, not a `docs` subcommand. + +### MCP — `verify` tool + +A read tool mirroring the kernel op (the §6.2 one-to-one pattern), `outputSchema` = `VerifyReport`. Description leads with *"Read-only integrity scrub — re-derives FTS/links/hashes and checks the version chain, reporting inconsistencies as structured findings; never writes."* An agent maintaining a corpus (the worknotes AGENTS.md discipline) can call this after a batch of writes to confirm it didn't corrupt the graph. Scope arg + `X-Mrplex-Scope` header as every other read tool. + +### REST — `GET /repos/{repo}/verify` + +Read-only, so a plain `GET` with query params (`?check=chain&severity=error`). Returns the JSON envelope. A whole-database verify (no repo) is `GET /verify`. ETag semantics: none — a verify result is a point-in-time scan, not a cacheable resource (document this; don't emit a misleading ETag). + +## 6. Cost and the honest limit + +`verify` is **O(total versions)**, not O(live set) — it's the one read path that walks history. On a corpus with heavy edit history (an Obsidian vault synced through autosave storms — the exact case the §11 rollup bullet worries about) that's a real cost. Mitigations, in order of preference: + +1. **Scoping by `--check` and `--repo`.** The `chain`/`hash`/`frontmatter` families need the full walk; `fts`/`chunks`/`links` only touch the live set + derived tables and are far cheaper. Default to running everything, but let an operator target the cheap families for a frequent smoke check and reserve the full history walk for periodic/CI runs. +2. **Keyset pagination throughout** (§4) so memory stays bounded regardless of corpus size — the scan streams, the report caps findings, counts stay exact. +3. **No write locks.** `verify` runs in ordinary read transactions; a concurrent write during a scan simply means the scan reflects a slightly-earlier snapshot per document, which is fine — findings are advisory, not transactional guarantees. + +This is inherent to "the chain is the guarantee": verifying the guarantee means reading the chain. v1 documents the cost rather than hiding it; a future incremental mode (verify only versions with id > last-verified watermark) is additive and noted as follow-up. + +## 7. Workstreams + +- **WS1 — wire types + kernel skeleton.** `VerifyReport`/`VerifyFinding`/`VerifySpec` in `wire.ts`; `kernel.verify` returning an empty report; `src/kernel/verify/` module scaffold. Scope + pre-flight (`repo_not_found`, `forbidden`) wired. +- **WS2 — adapter reads + parity.** `versions_all`, `documents_all`, `fts_all_refs`, `chunks_all_version_ids`, `backlog_all_version_ids` on both adapters; kernel-suite parity tests on a shared corrupted fixture. +- **WS3 — the six check families.** Each family as a pure function over the scanned rows + re-derivation helpers; exhaustive unit tests with hand-built corrupt inputs (a chain with a broken `next_id`, a hash-mismatched row, a divergent frontmatter, an orphaned FTS row, a mixed-dim chunk set, a misresolved dangling link). This is the bulk of the work and where correctness is proven. +- **WS4 — surfaces.** CLI (`--json`/`--ci`/filters, exit codes), MCP tool + `outputSchema`, REST route. CLI tests in `test/cli-verify.test.ts` asserting exit codes and JSON shape against a seeded-then-corrupted DB. +- **WS5 — docs.** README "How it works" gets a one-liner; a `mrplex verify` section near the sync/history material; flip the §11 bullet to **shipped** with an inline note (the links-plan.md precedent). + +## 8. Resolved decisions + +The design questions from the first draft are settled: + +- **`fts.stale_body` — dropped from v1.** SQLite's external-content FTS5 doesn't store the body redundantly, so a body-freshness check has no cheap SQLite path and a Postgres-only check isn't worth the parity break. Ship `fts.missing`/`fts.orphan` (id-set membership, full parity) only; a freshness check is an additive follow-up if a cheap path appears (§2.4). +- **`chunks.unembedded` — gated on embedder configuration.** The check runs only when an embedder is configured (flag → `MRPLEX_EMBEDDER` → config `embedder`); with no embedder it's *skipped and noted as skipped*, never reported clean and never firing on every version. The `chunks` orphan/mixed-dim checks stay unconditional — stray rows are residue worth flagging regardless (§2.5). +- **Incremental-verify watermark — deferred, not designed in.** `verify` runs during maintenance or when an issue is suspected, and in exactly those moments a **full** analysis is what's wanted, not an incremental slice trusting a prior clean watermark. v1 always does the complete O(total-versions) walk (§6); an incremental mode can be added later without disturbing this shape, but isn't a v1 concern. +- **`frontmatter.divergence` — strict, always an `error`.** In a correct store `frontmatter_raw` and `frontmatter` are written together in one transaction and can't drift, so this check should essentially never fire; when it does, it means the query index is lying about a document's content. It stays strict with no "parser tolerance" fudge. The one scenario that could mass-trigger it — a deliberate YAML-parser upgrade that reparses old bytes differently — is a **migration event**: any such upgrade ships with a bulk re-parse-and-rewrite backfill, so a divergence finding always means real trouble, never an expected upgrade artifact (§2.3). + +## 9. Open questions + +None outstanding — the §8 decisions are settled. Implementation can proceed from WS1. From 4d520dbe5afd85222c309286dace6bbf84997a3b Mon Sep 17 00:00:00 2001 From: Brendan Baldwin Date: Fri, 4 Sep 2026 17:57:11 -0600 Subject: [PATCH 2/8] verify WS1: wire types + kernel skeleton Add VerifyReport/VerifyFinding/VerifySpec/VerifySeverity wire types and a kernel.verify(ctx, spec) op returning a well-formed empty report. The verify module owns pre-flight (repo_not_found), scope narrowing, the finding accumulator (exact counts past the max_findings cap, min_severity filtering, skipped-check notes), and check selection by family/code. The six check families plug into runChecks in WS3. Threads embedderConfigured from the query-embed hook to gate the future chunks.unembedded check, and proxies verify through the auth shell as a read-only op. Co-Authored-By: Claude Opus 4.7 --- src/kernel/kernel.ts | 17 +++ src/kernel/verify/verify.test.ts | 122 +++++++++++++++++++++ src/kernel/verify/verify.ts | 178 +++++++++++++++++++++++++++++++ src/kernel/wire.ts | 66 ++++++++++++ src/shell/guard.ts | 6 ++ 5 files changed, 389 insertions(+) create mode 100644 src/kernel/verify/verify.test.ts create mode 100644 src/kernel/verify/verify.ts diff --git a/src/kernel/kernel.ts b/src/kernel/kernel.ts index e4664c4..2538435 100644 --- a/src/kernel/kernel.ts +++ b/src/kernel/kernel.ts @@ -66,6 +66,7 @@ import { validateRepoOverride, } from "./path-config.js"; import { type QuerySpec, DEFAULT_QUERY_LIMIT, runQuery } from "./query/query.js"; +import { type VerifyDeps, runVerify } from "./verify/verify.js"; import { isPathGlobPattern, normalizeExactDocumentPath, @@ -83,6 +84,8 @@ import type { PathWarning, QueryHit, Repo, + VerifyReport, + VerifySpec, Version, } from "./wire.js"; @@ -179,6 +182,8 @@ export type Kernel = { query(ctx: CallContext, spec: QuerySpec): Promise; /** Neighborhood expansion over the links index (docs/graph-plan.md). */ graph(ctx: CallContext, spec: GraphSpec): Promise; + /** Read-only integrity scrub over the store (docs/verify-plan.md). */ + verify(ctx: CallContext, spec: VerifySpec): Promise; /** Change-log read surface keyed by version-log position (sync/history §3). */ history: { /** The global change feed — the longest gap-free run after the cursor. */ @@ -834,6 +839,18 @@ export function createKernel(config: KernelConfig | Storage): Kernel { return runGraph(claimsFor(ctx), spec, deps); }, + async verify(ctx: CallContext, spec: VerifySpec): Promise { + // A configured query-embed hook is the resolved "embedder configured" + // signal — it comes from the same flag→env→config resolution the surfaces + // apply. Gates the chunks.unembedded check (verify-plan §2.5). + const deps: VerifyDeps = { + storage, + serverPathConfig, + embedderConfigured: queryEmbed !== undefined, + }; + return runVerify(claimsFor(ctx), spec, deps); + }, + history: { async since(ctx, input) { // Resolve the optional repo filter to an id (and gate its existence diff --git a/src/kernel/verify/verify.test.ts b/src/kernel/verify/verify.test.ts new file mode 100644 index 0000000..aec9246 --- /dev/null +++ b/src/kernel/verify/verify.test.ts @@ -0,0 +1,122 @@ +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { beforeEach, describe, expect, it } from "vitest"; +import { sqliteAdapter } from "../../storage-sqlite/adapter.js"; +import type { Storage } from "../../storage/types.js"; +import { type Kernel, createKernel } from "../kernel.js"; +import { VerifyAccumulator, checkSelected } from "./verify.js"; + +describe("checkSelected", () => { + it("selects everything when the list is omitted or empty", () => { + expect(checkSelected("chain.cycle", undefined)).toBe(true); + expect(checkSelected("chain.cycle", [])).toBe(true); + }); + + it("matches a full code", () => { + expect(checkSelected("chain.cycle", ["chain.cycle"])).toBe(true); + expect(checkSelected("chain.cycle", ["hash.mismatch"])).toBe(false); + }); + + it("matches a family prefix", () => { + expect(checkSelected("chain.cycle", ["chain"])).toBe(true); + expect(checkSelected("chain.no_current", ["chain"])).toBe(true); + expect(checkSelected("links.set_mismatch", ["chain"])).toBe(false); + }); +}); + +describe("VerifyAccumulator", () => { + it("tallies counts by check and severity", () => { + const acc = new VerifyAccumulator("warn", 100); + acc.countVersions(5); + acc.countDocuments(2); + acc.add({ check: "chain.cycle", severity: "error", repo: "r", detail: {} }); + acc.add({ check: "chain.cycle", severity: "error", repo: "r", detail: {} }); + acc.add({ check: "hash.missing", severity: "warn", repo: "r", detail: {} }); + + const report = acc.report(); + expect(report.counts).toEqual({ + versions_scanned: 5, + documents_scanned: 2, + by_check: { "chain.cycle": 2, "hash.missing": 1 }, + by_severity: { warn: 1, error: 2 }, + }); + expect(report.findings).toHaveLength(3); + expect(report.truncated).toBe(false); + }); + + it("drops findings below min_severity but never counts them", () => { + const acc = new VerifyAccumulator("error", 100); + acc.add({ check: "hash.missing", severity: "warn", repo: "r", detail: {} }); + acc.add({ check: "chain.cycle", severity: "error", repo: "r", detail: {} }); + + const report = acc.report(); + expect(report.findings).toHaveLength(1); + expect(report.findings[0]?.check).toBe("chain.cycle"); + expect(report.counts.by_severity).toEqual({ warn: 0, error: 1 }); + }); + + it("caps the emitted list at max_findings but keeps counts exact", () => { + const acc = new VerifyAccumulator("warn", 2); + for (let i = 0; i < 5; i++) { + acc.add({ check: "chain.cycle", severity: "error", repo: "r", detail: { i } }); + } + const report = acc.report(); + expect(report.findings).toHaveLength(2); + expect(report.truncated).toBe(true); + expect(report.counts.by_check).toEqual({ "chain.cycle": 5 }); + expect(report.counts.by_severity.error).toBe(5); + }); + + it("records skipped checks", () => { + const acc = new VerifyAccumulator("warn", 100); + acc.skip("chunks.unembedded", "no embedder configured"); + expect(acc.report().checks_skipped).toEqual([ + { check: "chunks.unembedded", reason: "no embedder configured" }, + ]); + }); +}); + +describe("kernel.verify skeleton (WS1)", () => { + let storage: Storage; + let kernel: Kernel; + + beforeEach(async () => { + const path = join(tmpdir(), `mrplex-verify-${Date.now()}-${Math.random()}.db`); + storage = await sqliteAdapter.open({ database: `sqlite:${path}` }); + kernel = createKernel(storage); + await storage.repos_create({ slug: "notes", created_at: new Date().toISOString() }); + await storage.repos_create({ slug: "secret", created_at: new Date().toISOString() }); + }); + + it("returns a well-formed empty report over all repos", async () => { + const report = await kernel.verify({}, {}); + expect(report.findings).toEqual([]); + expect(report.truncated).toBe(false); + expect(report.checks_skipped).toEqual([]); + expect(report.counts.versions_scanned).toBe(0); + expect(report.counts.documents_scanned).toBe(0); + }); + + it("throws repo_not_found for an unknown --repo", async () => { + await expect(kernel.verify({}, { repo: "nope" })).rejects.toMatchObject({ + code: "repo_not_found", + }); + }); + + it("hides an out-of-scope repo as not-found", async () => { + await expect( + kernel.verify({ scope: [{ repo: "notes" }] }, { repo: "secret" }), + ).rejects.toMatchObject({ code: "repo_not_found" }); + }); + + it("scopes a repo-less run to visible repos without throwing", async () => { + const report = await kernel.verify({ scope: [{ repo: "notes" }] }, {}); + expect(report.findings).toEqual([]); + }); + + it("verifies a system-namespaced repo when named explicitly", async () => { + await storage.repos_create({ slug: ":deleted-old", created_at: new Date().toISOString() }); + const report = await kernel.verify({}, { repo: ":deleted-old" }); + expect(report.findings).toEqual([]); + }); +}); diff --git a/src/kernel/verify/verify.ts b/src/kernel/verify/verify.ts new file mode 100644 index 0000000..8252cc2 --- /dev/null +++ b/src/kernel/verify/verify.ts @@ -0,0 +1,178 @@ +/** + * kernel.verify — a read-only integrity scrub over the store (docs/verify-plan.md). + * + * In an append-only store the version chain IS the source of truth, and every + * other table (fts_docs, chunks, embedding_backlog, links) is a derived index + * that can silently drift. `verify` re-derives what should be derivable and + * diffs it against what's stored, plus a set of structural-invariant checks the + * partial indexes (§3.2) are supposed to make impossible but which a corrupted + * database can still violate. mrplex's `git fsck`. + * + * Findings are DATA, never exceptions (verify-plan §3): the report comes back + * even when the store is on fire. The only throws are the usual pre-flight ones + * (`repo_not_found` for a bad `--repo`, `forbidden` never — scope narrows + * silently, like `query`). Six check families run over a full O(total-versions) + * history walk; see the check modules under this folder. + * + * WS1 (this file, initial): pre-flight + scope + empty report scaffold. The + * check families (WS3) fill in `runChecks` per repo. + */ + +import type { RepoRow, Storage } from "../../storage/types.js"; +import { type ClaimMatcher, claimsGrantRepo } from "../auth/scope.js"; +import { repoNotFound } from "../errors.js"; +import type { PathConfig } from "../path-config.js"; +import { effectivePathConfig, parseRepoOverride } from "../path-config.js"; +import type { VerifyFinding, VerifyReport, VerifySeverity, VerifySpec } from "../wire.js"; + +/** Default cap on emitted findings; `counts` stay exact past it (verify-plan §3). */ +export const DEFAULT_MAX_FINDINGS = 10_000; + +export type VerifyDeps = { + storage: Storage; + serverPathConfig: PathConfig; + /** + * Whether an embedder is configured for this store (flag → MRPLEX_EMBEDDER → + * config). Gates the `chunks.unembedded` check: with no embedder it's skipped + * and noted, never fired on every version (verify-plan §2.5). + */ + embedderConfigured: boolean; +}; + +const SEVERITY_RANK: Record = { warn: 0, error: 1 }; + +/** + * True when `check` (e.g. `chain.prev_next_asymmetry`) is selected by the + * caller's `checks` list — matched by full code or by family prefix + * (`chain` selects every `chain.*`). Empty/omitted list = all checks. + */ +export function checkSelected(check: string, checks: readonly string[] | undefined): boolean { + if (checks === undefined || checks.length === 0) return true; + const family = check.split(".")[0]; + return checks.some((c) => c === check || c === family); +} + +/** + * Accumulates findings + exact counts across the scan. Findings past + * `maxFindings` are dropped from the emitted list but still tallied, and + * `truncated` is set (verify-plan §3). + */ +export class VerifyAccumulator { + private readonly findings: VerifyFinding[] = []; + private versionsScanned = 0; + private documentsScanned = 0; + private readonly byCheck: Record = {}; + private readonly bySeverity: Record = { warn: 0, error: 0 }; + private readonly skipped: { check: string; reason: string }[] = []; + private truncated = false; + + constructor( + private readonly minSeverity: VerifySeverity, + private readonly maxFindings: number, + ) {} + + countVersions(n: number): void { + this.versionsScanned += n; + } + + countDocuments(n: number): void { + this.documentsScanned += n; + } + + skip(check: string, reason: string): void { + this.skipped.push({ check, reason }); + } + + add(finding: VerifyFinding): void { + if (SEVERITY_RANK[finding.severity] < SEVERITY_RANK[this.minSeverity]) return; + this.byCheck[finding.check] = (this.byCheck[finding.check] ?? 0) + 1; + this.bySeverity[finding.severity] += 1; + if (this.findings.length < this.maxFindings) { + this.findings.push(finding); + } else { + this.truncated = true; + } + } + + report(): VerifyReport { + return { + findings: this.findings, + counts: { + versions_scanned: this.versionsScanned, + documents_scanned: this.documentsScanned, + by_check: this.byCheck, + by_severity: this.bySeverity, + }, + checks_skipped: this.skipped, + truncated: this.truncated, + }; + } +} + +export async function runVerify( + claims: ClaimMatcher[] | null, + spec: VerifySpec, + deps: VerifyDeps, +): Promise { + const repos = await resolveRepos(claims, spec.repo, deps); + + const acc = new VerifyAccumulator( + spec.min_severity ?? "warn", + spec.max_findings ?? DEFAULT_MAX_FINDINGS, + ); + + for (const repo of repos) { + await runChecks(acc, repo, claims, spec, deps); + } + + return acc.report(); +} + +/** + * The repos this call verifies. A named `repo` resolves to exactly one (and + * gates existence through scope, same shape as `resolveRepo` in kernel.ts — an + * out-of-scope repo looks not-found). Omitted = every repo the caller can see, + * with system-namespaced (deleted) repos excluded, matching `repos.list`. + */ +async function resolveRepos( + claims: ClaimMatcher[] | null, + repoSlug: string | undefined, + deps: VerifyDeps, +): Promise { + const { storage, serverPathConfig } = deps; + // A named repo resolves even when system-namespaced (deleted) — an operator + // may deliberately verify a `:deleted-…` repo's integrity — so this branch + // does not apply the sigil filter; only the repo-less "all repos" case does. + if (repoSlug !== undefined) { + const row = await storage.repos_by_slug(repoSlug); + if (!row) throw repoNotFound(repoSlug); + if (claims && !claimsGrantRepo(claims, row.slug)) throw repoNotFound(repoSlug); + return [row]; + } + const isSystem = (slug: string): boolean => + serverPathConfig.system_sigils.some((sigil) => slug.startsWith(sigil)); + const rows = await storage.repos_list(); + return rows.filter( + (r) => !isSystem(r.slug) && (claims === null || claimsGrantRepo(claims, r.slug)), + ); +} + +/** + * Run the selected check families against one repo, appending findings to + * `acc`. WS1 scaffold — the six families (WS3) plug in here. `effectiveConfig` + * is resolved once per repo so sigil-aware checks share it. + */ +async function runChecks( + acc: VerifyAccumulator, + repo: RepoRow, + _claims: ClaimMatcher[] | null, + _spec: VerifySpec, + deps: VerifyDeps, +): Promise { + const _effectiveConfig = effectivePathConfig( + deps.serverPathConfig, + parseRepoOverride(repo.path_config), + ); + // WS3 wires the check families in here; the accumulator + config are the + // seam they hang off. Intentionally empty in the WS1 skeleton. +} diff --git a/src/kernel/wire.ts b/src/kernel/wire.ts index 696d70a..8eefdc1 100644 --- a/src/kernel/wire.ts +++ b/src/kernel/wire.ts @@ -169,3 +169,69 @@ export type DocGetManyResult = { items: Version[]; errors: DocGetManyError[]; }; + +// ----------------------------------------------------------------------------- +// Verify — read-only integrity scrub (docs/verify-plan.md). Re-derives the +// FTS / links / hash indexes and checks the version chain, reporting +// inconsistencies as structured findings rather than throwing. +// ----------------------------------------------------------------------------- + +/** + * A finding's severity. `error` = a real inconsistency (the store is lying); + * `warn` = suspicious but possibly benign (e.g. a legacy row a backfill fixes). + * `--ci` fails on `error` by default. (verify-plan §2.) + */ +export type VerifySeverity = "error" | "warn"; + +/** + * One inconsistency found by `verify`. `check` is a stable code (e.g. + * `chain.prev_next_asymmetry`) clients discriminate on; `detail` carries the + * check-specific payload (stored vs. computed, missing/extra edges, …). + * `document_id` / `version_id` are opaque encoded strings (§3.3) — internal + * integer ids never cross the wire. (verify-plan §3.) + */ +export type VerifyFinding = { + check: string; + severity: VerifySeverity; + repo: string; // slug + document_id?: string; + version_id?: string; + path?: string; + detail: Record; + /** Human hint at the remedy (e.g. "mrplex hash backfill"); never auto-run. */ + suggested_fix?: string; +}; + +/** + * Input to `kernel.verify` (verify-plan §3). `repo` omitted = every repo the + * caller can see. `checks` are family prefixes ("chain", "links") or full + * codes; omitted = all. `min_severity` filters emitted findings below the bar + * (counts stay full); `max_findings` caps the emitted list (counts stay exact, + * `truncated` is set). + */ +export type VerifySpec = { + repo?: string; + checks?: string[]; + min_severity?: VerifySeverity; + max_findings?: number; +}; + +/** + * Result of `kernel.verify` (verify-plan §3). Findings are data, never + * exceptions — the report comes back even when the store is corrupt. `counts` + * stay exact even when `findings` is capped by `max_findings` (then + * `truncated` is true). `checks_skipped` names families that didn't run and + * why (e.g. `chunks.unembedded` with no embedder configured) so a clean report + * isn't mistaken for full coverage. + */ +export type VerifyReport = { + findings: VerifyFinding[]; + counts: { + versions_scanned: number; + documents_scanned: number; + by_check: Record; + by_severity: Record; + }; + checks_skipped: { check: string; reason: string }[]; + truncated: boolean; +}; diff --git a/src/shell/guard.ts b/src/shell/guard.ts index a9fa579..e261775 100644 --- a/src/shell/guard.ts +++ b/src/shell/guard.ts @@ -238,6 +238,12 @@ export function guardKernel(kernel: Kernel, entitlement: Entitlement, audit?: Au graph: (_ctx, spec) => forward("graph", { repo: spec.repo }, () => kernel.graph(readCtx(), spec)), + // Read-only integrity scrub; scoped to read visibility like query/graph. + // The engine drops derived-table orphans that reference unreadable versions + // (verify-plan §2.7), so a scoped caller verifies only its slice. + verify: (_ctx, spec) => + forward("verify", { repo: spec.repo }, () => kernel.verify(readCtx(), spec)), + history: { // Read-only change feed; the engine applies read scope to each ref's // endpoints (a ref is delivered only if the caller can see either end). From 20664035409c2dda6059ac1711e26b9efca739c3 Mon Sep 17 00:00:00 2001 From: Brendan Baldwin Date: Sat, 5 Sep 2026 00:05:31 -0600 Subject: [PATCH 3/8] verify WS2: read-only adapter scans + parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add versions_all / documents_all / chunks_all_version_ids / backlog_all_version_ids to the Storage interface (keyset by id, the one place mrplex walks all versions incl. superseded), on both SQLite and Postgres adapters. The fts family is SQLite-only: correct the plan's §2.4 (FTS is a version-wide rowid bijection, not a live-set cover; Postgres fts_tsv is a generated column that can't drift). Add an optional VerifyFtsScans capability the SQLite adapter implements and Postgres omits; the kernel skips fts-with-note when absent. Membership reads from the fts_docs_docsize shadow table, since an external-content FTS5 table can't surface orphaned rowids via a plain scan. Add VerifyReport.checks_skipped to the wire shape. Parity + corruption tests in test/verify-scans.test.ts. Co-Authored-By: Claude Opus 4.7 --- docs/verify-plan.md | 24 +++- src/storage-postgres/adapter.ts | 73 ++++++++++ src/storage-sqlite/adapter.ts | 100 ++++++++++++++ src/storage/types.ts | 66 +++++++++ test/verify-scans.test.ts | 234 ++++++++++++++++++++++++++++++++ 5 files changed, 490 insertions(+), 7 deletions(-) create mode 100644 test/verify-scans.test.ts diff --git a/docs/verify-plan.md b/docs/verify-plan.md index a5a242b..849dbbd 100644 --- a/docs/verify-plan.md +++ b/docs/verify-plan.md @@ -69,16 +69,23 @@ Design §3.2 stores frontmatter twice by design: `frontmatter_raw` (byte-verbati | `frontmatter.divergence` | error | Re-parsed raw ≠ stored `frontmatter` JSON (deep-equal). `detail: { keys_differing }`. | | `frontmatter.system_leak` | error | A `$`-prefixed key is present in stored `frontmatter_raw`/`frontmatter` — `$*` intrinsics must be stripped at write time (`canonicalizeFrontmatter`) and never persisted (sync/history §2.4). A leak corrupts `$content_hash` and re-injection. | -### 2.4 `fts` — full-text index membership +### 2.4 `fts` — full-text index membership (SQLite-only) -The FTS index (`fts_docs`, external-content mode, trigger-maintained — `migrations/0003_fts_docs.sql`) must cover **exactly the current versions' bodies**: one FTS row per live version, none for superseded/deleted-out-of-namespace versions, none orphaned. +**This family is SQLite-specific by construction, and the invariant is a bijection with *all* versions, not the live set.** The original plan draft mis-stated both points; the schema is the authority. How FTS actually works in each engine: -| `check` code | Severity | Condition | +- **SQLite** — `fts_docs` is an FTS5 external-content table maintained by an `AFTER INSERT` trigger on `versions` (`0001_init.sql:89-102`). The store is append-only — versions are never `DELETE`d (a "delete" moves the doc under `:deleted/`) — so the trigger fires once per version insert and `fts_docs` holds **one row per version row, live and superseded alike**. `versions_search` filters `next_id IS NULL` at query time; the index itself is whole-history. The integrity invariant is therefore a **rowid↔version bijection over all versions**: every `versions.id` has an `fts_docs` rowid and vice-versa. A break means a trigger didn't fire (missing) or a stray rowid survived (orphan). +- **Postgres** — there is no separate FTS structure. `fts_tsv` is a `GENERATED ALWAYS AS (to_tsvector('english', body)) STORED` column on `versions` (`0001_init.sql:52`); the database regenerates it on every write. It cannot be missing, orphaned, or stale relative to its own row. So there is nothing to verify. + +Consequently the `fts` family **runs on SQLite and is skipped-with-note on Postgres** (`checks_skipped: { check: "fts", reason: "postgres: fts_tsv is a generated column, structurally consistent by construction" }`). This is the one genuinely engine-specific family; it's justified because it's the only check that catches a broken or absent FTS trigger, and the skip note keeps a clean Postgres report from being mistaken for "fts verified." + +| `check` code | Severity | Condition (SQLite) | |---|---|---| -| `fts.missing` | error | A live version has no FTS row. `suggested_fix: "rebuild FTS (reindex)"`. | -| `fts.orphan` | error | An FTS row references a version that isn't live (or doesn't exist). | +| `fts.missing` | error | A `versions` row has no matching `fts_docs` rowid — the insert trigger didn't fire. `suggested_fix: "rebuild the fts_docs index"`. | +| `fts.orphan` | error | An `fts_docs` rowid has no matching `versions` row — a stray index entry. | -`fts.missing` / `fts.orphan` need only id-set membership (which live versions have/lack an FTS row), so they hold at full SQLite/Postgres parity. A body-content freshness check (`fts.stale_body` — "the indexed text matches the live body") is **deliberately not in v1**: SQLite's FTS5 runs in external-content mode and doesn't store the body redundantly (the `versions` table is the content source), so there's no cheap way to compare stored FTS text on SQLite, and a parity-breaking Postgres-only check isn't worth it here. Deferred; additive if a cheap path appears. +Both are pure id-set membership, so no body text is read — which is also why the dropped `fts.stale_body` (below) was the awkward one. **Implementation note:** the membership set is read from the FTS5 shadow table `fts_docs_docsize` (one row per indexed rowid), not by scanning `fts_docs` directly. An external-content FTS5 table resolves each row's columns by joining back to `versions`, so `select rowid from fts_docs` for an *orphaned* rowid returns nothing (its content row is gone) and can't surface the very orphans we hunt; the shadow table is the authoritative index-membership set. + +A body-content freshness check (`fts.stale_body` — "the indexed text matches the live body") is **deliberately not in v1**: SQLite's FTS5 external-content mode doesn't store the body redundantly (the `versions` table is the content source), so there's no cheap way to compare stored FTS text, and Postgres has no separate text to compare against at all. Deferred; additive only if a cheap path appears. ### 2.5 `chunks` — embedding provenance @@ -135,10 +142,13 @@ export type VerifyReport = { by_check: Record; // findings per check code by_severity: Record; }; + checks_skipped: { check: string; reason: string }[]; // e.g. fts on postgres, chunks.unembedded w/o embedder truncated: boolean; // true if max_findings capped the list (counts stay exact) }; ``` +`checks_skipped` names families/checks that did not run and why — so a clean report is never mistaken for full coverage. Two sources feed it: the SQLite-only `fts` family skipped on Postgres (§2.4), and `chunks.unembedded` skipped when no embedder is configured (§2.5). + Design notes: - **Findings are data, never exceptions.** `verify` returns a report even when the store is on fire; the only throws are the usual pre-flight ones (`repo_not_found` for a bad `--repo`, `forbidden` for scope). This mirrors `links.repair` returning `{ repaired, skipped }` rather than throwing on a skip. - **`counts` stay exact even when `findings` is truncated.** A corpus with a million broken rows shouldn't OOM the report; `max_findings` (default e.g. 10_000) caps the emitted list but the scan still tallies `counts` and sets `truncated: true`. The operator learns the true scale and re-runs with `--check X` to enumerate one family. @@ -161,7 +171,7 @@ export type VerifySpec = { - **`versions_all(opts: { repo_id?; after_id; limit })`** → full `VersionRow[]` in id order. The backbone scan for `hash`, `frontmatter`, and `links` re-derivation (they need `frontmatter_raw`, `frontmatter`, `body`). This is the one place mrplex walks *all* versions including superseded ones; keyset by id so batches don't re-scan. - **`documents_all(opts: { repo_id?; after_id; limit })`** → `{ id, repo_id }[]` for the `chain` family's per-document walk and `chain.orphan_document`. Existing `version_history(document_id)` walks each chain. -- **`fts_all_refs(opts)`** → `{ version_id, has_row: bool, text_hash? }` sufficient to compute `fts.missing`/`fts.orphan`/`fts.stale_body` by joining against the live set. (SQLite external-content FTS makes stored-text retrieval awkward — if `stale_body` can't be done cheaply, the method returns `text_hash: null` and the check is skipped with a one-line report note, not a silent omission.) +- **`fts_missing_rowids(opts: { after_id; limit })`** and **`fts_orphan_rowids(opts: { after_id; limit })`** (SQLite adapter only) → the two sides of the `versions.id` ↔ `fts_docs` rowid bijection diff (§2.4), keyset-paginated. `missing` = version ids with no `fts_docs` rowid; `orphan` = `fts_docs` rowids with no version. Not on the `Storage` interface — declared on an optional `VerifyFtsScans` capability the SQLite adapter implements and Postgres does not, so the kernel skips the `fts` family (with a note) when the capability is absent. This keeps the whole-history FTS check off the shared interface where Postgres has nothing to implement. - **`chunks_all_version_ids(opts)`** and **`backlog_all_version_ids(opts)`** → the id sets for `chunks.*` orphan/coverage checks; intersect with live-version ids in the kernel. - **Reuse existing** `links_by_repo(repo_id)` (already returns every link row ordered by `(source_id, ord)` — tests use it) and `versions_live_by_repo(repo_id)` for the `links` family; `versions_current_by_documents` to resolve `target_id` → current path for `links.misresolved_bound`. diff --git a/src/storage-postgres/adapter.ts b/src/storage-postgres/adapter.ts index c657047..ad5c3e4 100644 --- a/src/storage-postgres/adapter.ts +++ b/src/storage-postgres/adapter.ts @@ -772,6 +772,79 @@ class PostgresStorage implements Storage { }); } + // Verify scans (docs/verify-plan.md §4). Read-only, keyset by id. No fts + // scans here: Postgres's fts_tsv is a generated column that cannot drift, so + // the kernel skips the `fts` family (§2.4) — this adapter deliberately omits + // the VerifyFtsScans capability. + + async versions_all(opts: { + repo_id?: number; + after_id: number; + limit: number; + }): Promise { + return this.withClient(async (c) => { + const repoClause = opts.repo_id === undefined ? "" : " and repo_id = $3"; + const params = + opts.repo_id === undefined + ? [opts.after_id, opts.limit] + : [opts.after_id, opts.limit, opts.repo_id]; + const res = await c.query( + `select id, document_id, repo_id, prev_id, next_id, path, + frontmatter_raw, frontmatter, body, author, created_at, content_hash + from versions + where id > $1${repoClause} + order by id asc limit $2`, + params, + ); + return res.rows as VersionRow[]; + }); + } + + async documents_all(opts: { + repo_id?: number; + after_id: number; + limit: number; + }): Promise { + return this.withClient(async (c) => { + const repoClause = opts.repo_id === undefined ? "" : " and repo_id = $3"; + const params = + opts.repo_id === undefined + ? [opts.after_id, opts.limit] + : [opts.after_id, opts.limit, opts.repo_id]; + const res = await c.query( + `select id, repo_id from documents + where id > $1${repoClause} + order by id asc limit $2`, + params, + ); + return res.rows as DocumentRow[]; + }); + } + + async chunks_all_version_ids(opts: { after_id: number; limit: number }): Promise { + return this.withClient(async (c) => { + const res = await c.query<{ version_id: number }>( + `select distinct version_id from chunks + where version_id > $1 + order by version_id asc limit $2`, + [opts.after_id, opts.limit], + ); + return res.rows.map((r) => Number(r.version_id)); + }); + } + + async backlog_all_version_ids(opts: { after_id: number; limit: number }): Promise { + return this.withClient(async (c) => { + const res = await c.query<{ version_id: number }>( + `select version_id from embedding_backlog + where version_id > $1 + order by version_id asc limit $2`, + [opts.after_id, opts.limit], + ); + return res.rows.map((r) => Number(r.version_id)); + }); + } + async chunks_upsert( version_id: number, model: string, diff --git a/src/storage-sqlite/adapter.ts b/src/storage-sqlite/adapter.ts index c2c89f5..101f9fa 100644 --- a/src/storage-sqlite/adapter.ts +++ b/src/storage-sqlite/adapter.ts @@ -719,6 +719,106 @@ class SqliteStorage implements Storage { }); } + // Verify scans (docs/verify-plan.md §4). Read-only, keyset by id. + + async versions_all(opts: { + repo_id?: number; + after_id: number; + limit: number; + }): Promise { + const repoClause = opts.repo_id === undefined ? "" : " and repo_id = ?"; + const params = + opts.repo_id === undefined + ? [opts.after_id, opts.limit] + : [opts.after_id, opts.repo_id, opts.limit]; + const rows = this.db + .prepare( + `select id, document_id, repo_id, prev_id, next_id, path, + frontmatter_raw, frontmatter, body, author, created_at, content_hash + from versions + where id > ?${repoClause} + order by id asc limit ?`, + ) + .all(...params) as VersionRawRow[]; + return rows.map(hydrateVersion); + } + + async documents_all(opts: { + repo_id?: number; + after_id: number; + limit: number; + }): Promise { + const repoClause = opts.repo_id === undefined ? "" : " and repo_id = ?"; + const params = + opts.repo_id === undefined + ? [opts.after_id, opts.limit] + : [opts.after_id, opts.repo_id, opts.limit]; + return this.db + .prepare( + `select id, repo_id from documents + where id > ?${repoClause} + order by id asc limit ?`, + ) + .all(...params) as DocumentRow[]; + } + + async chunks_all_version_ids(opts: { after_id: number; limit: number }): Promise { + const rows = this.db + .prepare( + `select distinct version_id from chunks + where version_id > ? + order by version_id asc limit ?`, + ) + .all(opts.after_id, opts.limit) as { version_id: number }[]; + return rows.map((r) => r.version_id); + } + + async backlog_all_version_ids(opts: { after_id: number; limit: number }): Promise { + const rows = this.db + .prepare( + `select version_id from embedding_backlog + where version_id > ? + order by version_id asc limit ?`, + ) + .all(opts.after_id, opts.limit) as { version_id: number }[]; + return rows.map((r) => r.version_id); + } + + // fts verify scans (VerifyFtsScans capability, verify-plan §2.4). SQLite-only: + // the version↔fts_docs rowid bijection can drift here (trigger-maintained + // external-content table), unlike Postgres's generated fts_tsv column. + // + // Membership is read from the FTS5 shadow table `fts_docs_docsize` (one row + // per indexed rowid), NOT by scanning `fts_docs` itself: an external-content + // FTS5 table resolves each row's columns by joining back to `versions`, so a + // `select rowid from fts_docs` for an orphaned rowid returns nothing (the + // content row is gone) — it can't surface the very orphans we're hunting. + // The shadow table is the authoritative index-membership set. + + async fts_missing_rowids(opts: { after_id: number; limit: number }): Promise { + const rows = this.db + .prepare( + `select v.id as id from versions v + left join fts_docs_docsize d on d.id = v.id + where v.id > ? and d.id is null + order by v.id asc limit ?`, + ) + .all(opts.after_id, opts.limit) as { id: number }[]; + return rows.map((r) => r.id); + } + + async fts_orphan_rowids(opts: { after_id: number; limit: number }): Promise { + const rows = this.db + .prepare( + `select d.id as id from fts_docs_docsize d + left join versions v on v.id = d.id + where d.id > ? and v.id is null + order by d.id asc limit ?`, + ) + .all(opts.after_id, opts.limit) as { id: number }[]; + return rows.map((r) => r.id); + } + async chunks_upsert( version_id: number, model: string, diff --git a/src/storage/types.ts b/src/storage/types.ts index ee662a4..0c6156f 100644 --- a/src/storage/types.ts +++ b/src/storage/types.ts @@ -327,6 +327,44 @@ export type Storage = { updates: readonly { id: number; content_hash: string }[], ): Promise; + // Verify scans (docs/verify-plan.md §4). Read-only, keyset-paginated by id. + // These are the ONE place mrplex walks all versions (including superseded + // ones), not just the live set — an integrity scrub reads the whole chain. + + /** + * One keyset page of ALL versions (live and superseded) in id order, id > + * `after_id`, capped at `limit`, optionally scoped to `repo_id`. The backbone + * scan for the `chain` / `hash` / `frontmatter` / `links` check families, + * which re-derive from `frontmatter_raw` / `frontmatter` / `body`. Keyset by + * id so batches don't re-scan (verify-plan §4, §6). + */ + versions_all(opts: { repo_id?: number; after_id: number; limit: number }): Promise; + + /** + * One keyset page of `documents` rows in id order, id > `after_id`, capped at + * `limit`, optionally scoped to `repo_id`. Feeds the `chain` family's + * per-document walk (via `version_history`) and `chain.orphan_document` + * (documents with zero versions). Keyset by id (verify-plan §4). + */ + documents_all(opts: { + repo_id?: number; + after_id: number; + limit: number; + }): Promise; + + /** + * Distinct version ids present in the `chunks` table, keyset-paginated by + * version id (id > `after_id`, capped at `limit`). The kernel intersects + * these against live/all version ids for `chunks.orphan` (verify-plan §2.5). + */ + chunks_all_version_ids(opts: { after_id: number; limit: number }): Promise; + + /** + * Version ids present in the `embedding_backlog` table, keyset-paginated by + * version id. Feeds `chunks.backlog_orphan` (verify-plan §2.5). + */ + backlog_all_version_ids(opts: { after_id: number; limit: number }): Promise; + /** * All currently-live versions in a repo (i.e. rows where next_id IS NULL). * Used by `repos.set_path_config` to produce the advisory PathWarning[] @@ -504,6 +542,34 @@ export type Storage = { backlog_status(now: string): Promise; }; +/** + * Optional adapter capability for the SQLite-only `fts` verify family + * (verify-plan §2.4). SQLite maintains a separate `fts_docs` external-content + * table via triggers, so its rowid set can drift from `versions.id` (a trigger + * that didn't fire, a stray row). Postgres has no separate structure — + * `fts_tsv` is a generated column that cannot drift — so the Postgres adapter + * does NOT implement this, and the kernel skips the `fts` family with a note. + * + * Both methods diff the `versions.id` ↔ `fts_docs` rowid bijection over ALL + * versions (not the live set), keyset-paginated by id. + */ +export type VerifyFtsScans = { + /** Version ids with no matching `fts_docs` rowid (trigger didn't fire). */ + fts_missing_rowids(opts: { after_id: number; limit: number }): Promise; + /** `fts_docs` rowids with no matching `versions` row (stray index entry). */ + fts_orphan_rowids(opts: { after_id: number; limit: number }): Promise; +}; + +/** Runtime probe: does this storage implement the SQLite-only fts verify scans? */ +export function hasVerifyFtsScans(storage: unknown): storage is VerifyFtsScans { + return ( + typeof storage === "object" && + storage !== null && + typeof (storage as VerifyFtsScans).fts_missing_rowids === "function" && + typeof (storage as VerifyFtsScans).fts_orphan_rowids === "function" + ); +} + export type OpenConfig = { /** Database url — sqlite:./path.db or postgres://… */ database: string; diff --git a/test/verify-scans.test.ts b/test/verify-scans.test.ts new file mode 100644 index 0000000..c7b0322 --- /dev/null +++ b/test/verify-scans.test.ts @@ -0,0 +1,234 @@ +/** + * Verify scan surface (docs/verify-plan.md §4) — direct adapter tests, parity + * across SQLite and (when MRPLEX_TEST_POSTGRES_URL is set) Postgres. These pin + * the read-only storage contract the verify check families (WS3) build on: + * versions_all / documents_all / chunks_all_version_ids / + * backlog_all_version_ids, plus the SQLite-only VerifyFtsScans capability. + */ + +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import Database from "better-sqlite3"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { sqliteAdapter } from "../src/storage-sqlite/adapter.js"; +import { type Storage, hasVerifyFtsScans } from "../src/storage/types.js"; +import { PG_URL, openTestPostgres } from "./pg-harness.js"; + +type Factory = { + name: string; + open: () => Promise<{ storage: Storage; cleanup?: () => Promise }>; +}; + +// Each SQLite factory tracks its file path so a corruption test can open a +// second raw handle and mutate fts_docs behind the adapter's back (WAL lets a +// second connection see committed rows). Keyed per-open so parallel tests don't +// collide. +const sqlitePaths = new Map(); + +const factories: Factory[] = [ + { + name: "sqlite", + open: async () => { + const path = join(tmpdir(), `mrplex-verify-scans-${Date.now()}-${Math.random()}.db`); + const storage = await sqliteAdapter.open({ database: `sqlite:${path}` }); + sqlitePaths.set(storage, path); + return { storage }; + }, + }, +]; + +if (PG_URL) { + factories.push({ + name: "postgres", + open: async () => { + const { storage, cleanup } = await openTestPostgres(); + return { storage, cleanup }; + }, + }); +} + +const T = (sec: number): string => new Date(Date.UTC(2026, 7, 14, 0, 0, sec)).toISOString(); + +/** Insert a fresh document + one version; return {docId, versionId}. */ +async function seedDoc( + storage: Storage, + repoId: number, + path: string, + body: string, +): Promise<{ docId: number; versionId: number }> { + const doc = await storage.documents_create(repoId); + const v = await storage.version_insert({ + document_id: doc.id, + repo_id: repoId, + prev_id: null, + path, + frontmatter_raw: "", + frontmatter: {}, + body, + author: "alice", + created_at: T(1), + }); + return { docId: doc.id, versionId: v.id }; +} + +for (const factory of factories) { + describe(`verify scans [${factory.name}]`, () => { + let storage: Storage; + let cleanup: (() => Promise) | undefined; + let repoId: number; + let otherRepoId: number; + + beforeEach(async () => { + const opened = await factory.open(); + storage = opened.storage; + cleanup = opened.cleanup; + repoId = (await storage.repos_create({ slug: "notes", created_at: T(0) })).id; + otherRepoId = (await storage.repos_create({ slug: "other", created_at: T(0) })).id; + }); + + afterEach(async () => { + if (cleanup) await cleanup(); + else await storage.close(); + }); + + describe("versions_all", () => { + it("returns all versions (live and superseded) in id order", async () => { + const doc = await storage.documents_create(repoId); + const v1 = await storage.version_insert({ + document_id: doc.id, + repo_id: repoId, + prev_id: null, + path: "a.md", + frontmatter_raw: "", + frontmatter: {}, + body: "one\n", + author: "alice", + created_at: T(1), + }); + const v2 = await storage.version_insert({ + document_id: doc.id, + repo_id: repoId, + prev_id: v1.id, + path: "a.md", + frontmatter_raw: "", + frontmatter: {}, + body: "two\n", + author: "alice", + created_at: T(2), + }); + const rows = await storage.versions_all({ after_id: 0, limit: 100 }); + // Both the superseded v1 and the live v2 come back. + expect(rows.map((r) => r.id)).toEqual([v1.id, v2.id]); + }); + + it("keyset-paginates by id", async () => { + const a = await seedDoc(storage, repoId, "a.md", "a"); + const b = await seedDoc(storage, repoId, "b.md", "b"); + const c = await seedDoc(storage, repoId, "c.md", "c"); + const page1 = await storage.versions_all({ after_id: 0, limit: 2 }); + expect(page1.map((r) => r.id)).toEqual([a.versionId, b.versionId]); + const page2 = await storage.versions_all({ after_id: page1[1]!.id, limit: 2 }); + expect(page2.map((r) => r.id)).toEqual([c.versionId]); + }); + + it("scopes to repo_id when given", async () => { + await seedDoc(storage, repoId, "a.md", "a"); + const other = await seedDoc(storage, otherRepoId, "x.md", "x"); + const rows = await storage.versions_all({ repo_id: otherRepoId, after_id: 0, limit: 100 }); + expect(rows.map((r) => r.id)).toEqual([other.versionId]); + }); + }); + + describe("documents_all", () => { + it("returns document rows in id order, optionally scoped", async () => { + const a = await seedDoc(storage, repoId, "a.md", "a"); + const b = await seedDoc(storage, repoId, "b.md", "b"); + const other = await seedDoc(storage, otherRepoId, "x.md", "x"); + + const all = await storage.documents_all({ after_id: 0, limit: 100 }); + expect(all.map((d) => d.id)).toEqual([a.docId, b.docId, other.docId]); + + const scoped = await storage.documents_all({ + repo_id: repoId, + after_id: 0, + limit: 100, + }); + expect(scoped.map((d) => d.id)).toEqual([a.docId, b.docId]); + }); + + it("includes a document with zero versions (orphan)", async () => { + const orphan = await storage.documents_create(repoId); + const rows = await storage.documents_all({ after_id: 0, limit: 100 }); + expect(rows.map((d) => d.id)).toContain(orphan.id); + }); + }); + + describe("chunks_all_version_ids / backlog_all_version_ids", () => { + it("returns distinct version ids present in each table, keyset by id", async () => { + const a = await seedDoc(storage, repoId, "a.md", "a"); + const b = await seedDoc(storage, repoId, "b.md", "b"); + + await storage.chunks_upsert(a.versionId, "m", [ + { ix: 0, text: "a0", text_hash: "h0", model: "m", embedding: [1, 0] }, + { ix: 1, text: "a1", text_hash: "h1", model: "m", embedding: [0, 1] }, + ]); + await storage.backlog_enqueue(b.versionId); + + const chunkIds = await storage.chunks_all_version_ids({ after_id: 0, limit: 100 }); + expect(chunkIds).toEqual([a.versionId]); // distinct — two chunks collapse to one id + + const backlogIds = await storage.backlog_all_version_ids({ after_id: 0, limit: 100 }); + expect(backlogIds).toEqual([b.versionId]); + }); + + it("returns empty arrays when the tables are empty", async () => { + expect(await storage.chunks_all_version_ids({ after_id: 0, limit: 100 })).toEqual([]); + expect(await storage.backlog_all_version_ids({ after_id: 0, limit: 100 })).toEqual([]); + }); + }); + + describe("VerifyFtsScans capability", () => { + it("is present on SQLite and absent on Postgres", () => { + expect(hasVerifyFtsScans(storage)).toBe(factory.name === "sqlite"); + }); + + it("reports no missing/orphan rowids on a healthy SQLite store", async () => { + if (!hasVerifyFtsScans(storage)) return; + await seedDoc(storage, repoId, "a.md", "a"); + await seedDoc(storage, repoId, "b.md", "b"); + expect(await storage.fts_missing_rowids({ after_id: 0, limit: 100 })).toEqual([]); + expect(await storage.fts_orphan_rowids({ after_id: 0, limit: 100 })).toEqual([]); + }); + + it("detects a missing rowid (a version with no fts_docs row)", async () => { + if (!hasVerifyFtsScans(storage)) return; + const a = await seedDoc(storage, repoId, "a.md", "a"); + // Simulate a trigger that didn't fire: delete this version's fts row via + // a second raw handle. external-content FTS5 delete uses the special + // 'delete' command with the OLD body. + const path = sqlitePaths.get(storage) as string; + const raw = new Database(path); + raw + .prepare("insert into fts_docs(fts_docs, rowid, body) values('delete', ?, ?)") + .run(a.versionId, "a"); + raw.close(); + expect(await storage.fts_missing_rowids({ after_id: 0, limit: 100 })).toEqual([ + a.versionId, + ]); + expect(await storage.fts_orphan_rowids({ after_id: 0, limit: 100 })).toEqual([]); + }); + + it("detects an orphan rowid (an fts_docs row with no version)", async () => { + if (!hasVerifyFtsScans(storage)) return; + await seedDoc(storage, repoId, "a.md", "a"); + const path = sqlitePaths.get(storage) as string; + const raw = new Database(path); + // Insert an fts row for a rowid that no version claims. + raw.prepare("insert into fts_docs(rowid, body) values (?, ?)").run(9999, "ghost"); + raw.close(); + expect(await storage.fts_orphan_rowids({ after_id: 0, limit: 100 })).toEqual([9999]); + expect(await storage.fts_missing_rowids({ after_id: 0, limit: 100 })).toEqual([]); + }); + }); + }); +} From 8d40ab944cc397feb36cb89f7c05148b9d389ec4 Mon Sep 17 00:00:00 2001 From: Brendan Baldwin Date: Sat, 5 Sep 2026 02:12:34 -0600 Subject: [PATCH 4/8] verify WS3: the six check families MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements chain, hash, frontmatter, fts, chunks, and links checks as pure functions over the WS2 scans, wired into runVerify. Per-repo families (chain/hash/frontmatter/links + chunks.unembedded) run in the repo loop; whole-store families (fts, chunks orphan/mixed-dim) run once and are skipped-with-note under a --repo filter. Notable corrections found during implementation: - chunks.orphan means a NONEXISTENT version, not a superseded one — the embed worker deliberately leaves chunks on superseded versions. - chunks/backlog/fts are global tables, not repo-partitioned. - fixed an infinite loop: the chain scan's keyset cursor advanced after a `continue` for orphan docs, stalling the page. Cursor now advances first. Adds versions_by_document + chunk orphan/dim scans to both adapters. 365 lines of corruption-injection tests prove each finding fires; full suite green (1133 passed). Co-Authored-By: Claude Opus 4.7 --- docs/verify-plan.md | 6 +- src/kernel/kernel.ts | 1 + src/kernel/verify/chain.ts | 225 +++++++++++++++++++ src/kernel/verify/checks.ts | 58 +++++ src/kernel/verify/chunks.ts | 139 ++++++++++++ src/kernel/verify/content.ts | 190 ++++++++++++++++ src/kernel/verify/fts.ts | 61 ++++++ src/kernel/verify/links.ts | 216 ++++++++++++++++++ src/kernel/verify/verify.test.ts | 6 +- src/kernel/verify/verify.ts | 112 ++++++++-- src/storage-postgres/adapter.ts | 68 ++++++ src/storage-sqlite/adapter.ts | 68 ++++++ src/storage/types.ts | 52 ++++- test/verify-checks.test.ts | 365 +++++++++++++++++++++++++++++++ 14 files changed, 1537 insertions(+), 30 deletions(-) create mode 100644 src/kernel/verify/chain.ts create mode 100644 src/kernel/verify/checks.ts create mode 100644 src/kernel/verify/chunks.ts create mode 100644 src/kernel/verify/content.ts create mode 100644 src/kernel/verify/fts.ts create mode 100644 src/kernel/verify/links.ts create mode 100644 test/verify-checks.test.ts diff --git a/docs/verify-plan.md b/docs/verify-plan.md index 849dbbd..ffa67d5 100644 --- a/docs/verify-plan.md +++ b/docs/verify-plan.md @@ -93,11 +93,15 @@ Chunks + the embedding backlog (§5.3). `verify` checks *structural* consistency | `check` code | Severity | Condition | |---|---|---| -| `chunks.orphan` | error | A `chunks` row references a non-live or nonexistent version. | +| `chunks.orphan` | error | A `chunks` row references a **nonexistent** version (an FK violation — not merely a superseded one). | | `chunks.backlog_orphan` | error | An `embedding_backlog` row references a nonexistent version. | | `chunks.unembedded` | warn | A live version has neither chunk rows nor a backlog entry — it fell out of the embedding pipeline. `suggested_fix: "mrplex embed backfill"`. **Only runs when an embedder is configured** (see below); otherwise skipped entirely, not reported clean. | | `chunks.mixed_dim` | error | Chunk rows for one version carry vectors of differing dimensionality (§5.3 "refuse mixed-dim writes" — a finding means that guard was bypassed). | +**"Orphan" means nonexistent, not non-live.** The embedding worker deliberately leaves chunks on *superseded* versions (it reuses their vectors on the next re-embed via `chunks_by_version(prev_id)` and never deletes them — `worker.ts`, `backfill.ts`). A superseded version keeping its chunks is normal, so `chunks.orphan` fires only when the referenced version row does not exist at all — reachable only if a foreign key was disabled. Cheap to check, and the only genuine inconsistency here. + +**Global vs. per-repo.** `chunks`, `embedding_backlog`, and `fts_docs` are **not repo-partitioned**. The orphan checks (`chunks.orphan`, `chunks.backlog_orphan`) reference versions that could belong to any repo — or to none, when the version is gone — so they, `chunks.mixed_dim`, and the whole `fts` family are **whole-store checks that run once per verify call and only in an all-repos run**. Under a `--repo` filter they are skipped-with-note (`checks_skipped: { check, reason: "whole-store check; omit --repo to run" }`), since a repo-scoped run can neither attribute nor bound them correctly. The exception is `chunks.unembedded`, which is about a *repo's* live versions lacking embeddings and therefore runs in the per-repo loop. + **Embedder-gated coverage.** `chunks.unembedded` is meaningful only when embedding is actually intended for this store. A corpus that never configured an embedder has *every* live version "unembedded" — that's noise, not a finding. The check therefore runs **only when an embedder is configured**, resolved through the standard precedence (flag → `MRPLEX_EMBEDDER` env → config `embedder` → none; the §"Configuration" resolution the whole CLI already uses). No embedder configured ⇒ `chunks.unembedded` is *skipped* (the report notes it was skipped for lack of an embedder — not silently omitted, not reported clean). The orphan/provenance checks (`chunks.orphan`, `chunks.backlog_orphan`, `chunks.mixed_dim`) are unconditional: stray chunk rows referencing dead versions are a real inconsistency whether or not an embedder is *currently* wired up (they'd be residue from a past one). ### 2.6 `links` — link index vs. re-extraction diff --git a/src/kernel/kernel.ts b/src/kernel/kernel.ts index 2538435..e556a40 100644 --- a/src/kernel/kernel.ts +++ b/src/kernel/kernel.ts @@ -846,6 +846,7 @@ export function createKernel(config: KernelConfig | Storage): Kernel { const deps: VerifyDeps = { storage, serverPathConfig, + serverLinkConfig, embedderConfigured: queryEmbed !== undefined, }; return runVerify(claimsFor(ctx), spec, deps); diff --git a/src/kernel/verify/chain.ts b/src/kernel/verify/chain.ts new file mode 100644 index 0000000..9dc863a --- /dev/null +++ b/src/kernel/verify/chain.ts @@ -0,0 +1,225 @@ +/** + * `chain` verify family (docs/verify-plan.md §2.1) — version-chain structural + * integrity. Catches what the partial unique indexes (§3.2) are supposed to + * make impossible but a directly-corrupted DB can still violate. + * + * Walks every document chain-independently (`versions_by_document`, not the + * recursive-from-current walk, which a broken chain defeats) and checks the + * prev/next inverse-link invariant, current-version cardinality, prev target + * validity, cycles, and repo-id agreement. Also flags orphan documents (zero + * versions) and the one-live-per-path invariant across the repo's live set. + */ + +import { normalizeKey } from "../casefold.js"; +import { type CheckContext, SCAN_BATCH, canRead, did, finding, vid } from "./checks.js"; + +export async function checkChain(ctx: CheckContext): Promise { + const asym = ctx.selected("chain.prev_next_asymmetry"); + const multiCurrent = ctx.selected("chain.multiple_current"); + const noCurrent = ctx.selected("chain.no_current"); + const brokenPrev = ctx.selected("chain.broken_prev"); + const cycle = ctx.selected("chain.cycle"); + const repoMismatch = ctx.selected("chain.repo_mismatch"); + const orphanDoc = ctx.selected("chain.orphan_document"); + const multiLive = ctx.selected("chain.multiple_live_at_path"); + + // Per-document chain checks — walk documents keyset-paginated by id. + let afterDoc = 0; + for (;;) { + const docs = await ctx.storage.documents_all({ + repo_id: ctx.repo.id, + after_id: afterDoc, + limit: SCAN_BATCH, + }); + if (docs.length === 0) break; + ctx.acc.countDocuments(docs.length); + + for (const doc of docs) { + // Advance the keyset cursor FIRST — before any `continue` below — so an + // orphan doc (zero versions) that lands last in a batch can't stall the + // scan into refetching the same page forever. + afterDoc = doc.id; + + const versions = await ctx.storage.versions_by_document(doc.id); + ctx.acc.countVersions(versions.length); + + if (versions.length === 0) { + if (orphanDoc) { + ctx.acc.add( + finding(ctx, { + check: "chain.orphan_document", + severity: "warn", + document_id: did(doc.id), + detail: {}, + }), + ); + } + continue; + } + + const byId = new Map(versions.map((v) => [v.id, v])); + const currents = versions.filter((v) => v.next_id === null); + + if (multiCurrent && currents.length > 1) { + ctx.acc.add( + finding(ctx, { + check: "chain.multiple_current", + severity: "error", + document_id: did(doc.id), + detail: { current_version_ids: currents.map((v) => vid(v.id)) }, + }), + ); + } + if (noCurrent && currents.length === 0) { + ctx.acc.add( + finding(ctx, { + check: "chain.no_current", + severity: "error", + document_id: did(doc.id), + detail: { version_count: versions.length }, + }), + ); + } + + for (const v of versions) { + const path = canRead(ctx, v.path) ? v.path : undefined; + + // repo_id agreement with the denormalized column. + if (repoMismatch && v.repo_id !== doc.repo_id) { + ctx.acc.add( + finding(ctx, { + check: "chain.repo_mismatch", + severity: "error", + document_id: did(doc.id), + version_id: vid(v.id), + path, + detail: { version_repo_id: v.repo_id, document_repo_id: doc.repo_id }, + }), + ); + } + + // prev target validity + prev↔next symmetry. + if (v.prev_id !== null) { + const prev = byId.get(v.prev_id); + if (brokenPrev && prev === undefined) { + ctx.acc.add( + finding(ctx, { + check: "chain.broken_prev", + severity: "error", + document_id: did(doc.id), + version_id: vid(v.id), + path, + detail: { prev_version_id: vid(v.prev_id) }, + }), + ); + } else if (asym && prev !== undefined && prev.next_id !== v.id) { + ctx.acc.add( + finding(ctx, { + check: "chain.prev_next_asymmetry", + severity: "error", + document_id: did(doc.id), + version_id: vid(v.id), + path, + detail: { + prev_version_id: vid(v.prev_id), + prev_next_id: prev.next_id === null ? null : vid(prev.next_id), + }, + }), + ); + } + } + + // next target validity + next↔prev symmetry (the other direction). + if (v.next_id !== null) { + const next = byId.get(v.next_id); + if (brokenPrev && next === undefined) { + ctx.acc.add( + finding(ctx, { + check: "chain.broken_prev", + severity: "error", + document_id: did(doc.id), + version_id: vid(v.id), + path, + detail: { next_version_id: vid(v.next_id), reason: "next points nowhere" }, + }), + ); + } else if (asym && next !== undefined && next.prev_id !== v.id) { + ctx.acc.add( + finding(ctx, { + check: "chain.prev_next_asymmetry", + severity: "error", + document_id: did(doc.id), + version_id: vid(v.id), + path, + detail: { + next_version_id: vid(v.next_id), + next_prev_id: next.prev_id === null ? null : vid(next.prev_id), + }, + }), + ); + } + } + } + + // Cycle detection: follow prev_id from each current back to a root, + // bounded by version count. A revisit (or overrun) is a loop. + if (cycle && currents.length > 0) { + for (const start of currents) { + const seen = new Set(); + let cur = start; + let looped = false; + while (cur.prev_id !== null) { + if (seen.has(cur.id)) { + looped = true; + break; + } + seen.add(cur.id); + const prev = byId.get(cur.prev_id); + if (prev === undefined) break; // broken_prev already covers this + if (seen.has(prev.id)) { + looped = true; + break; + } + cur = prev; + } + if (looped) { + ctx.acc.add( + finding(ctx, { + check: "chain.cycle", + severity: "error", + document_id: did(doc.id), + detail: { from_version_id: vid(start.id) }, + }), + ); + } + } + } + } + } + + // Repo-wide: one live document per normalized path. The partial unique index + // should forbid duplicates; a finding means it's missing or corrupt. + if (multiLive) { + const live = await ctx.storage.versions_live_by_repo(ctx.repo.id); + const byNorm = new Map(); + for (const v of live) { + const key = normalizeKey(v.path); + const ids = byNorm.get(key); + if (ids) ids.push(v.id); + else byNorm.set(key, [v.id]); + } + for (const [, ids] of byNorm) { + if (ids.length > 1) { + const first = live.find((v) => v.id === ids[0]); + ctx.acc.add( + finding(ctx, { + check: "chain.multiple_live_at_path", + severity: "error", + path: first && canRead(ctx, first.path) ? first.path : undefined, + detail: { version_ids: ids.map(vid) }, + }), + ); + } + } + } +} diff --git a/src/kernel/verify/checks.ts b/src/kernel/verify/checks.ts new file mode 100644 index 0000000..be0b5a6 --- /dev/null +++ b/src/kernel/verify/checks.ts @@ -0,0 +1,58 @@ +/** + * Shared context + helpers for the verify check families (docs/verify-plan.md + * §2). Each family is a pure-ish async function `(ctx) => void` that reads + * through the storage scans (WS2) and pushes findings onto the accumulator. + * + * Scope (§2.7): a family emits a finding only when the caller can read the + * referenced path. For derived-table orphans that reference an UNREADABLE + * version, the finding is dropped (a scoped caller verifies only its slice; + * a full-trust `--unsafe` run sees everything). `canRead` centralizes that. + */ + +import type { LinkConfig } from "../../links/link-config.js"; +import type { RepoRow, Storage } from "../../storage/types.js"; +import type { ClaimMatcher } from "../auth/scope.js"; +import { claimsGrantRead } from "../auth/scope.js"; +import type { PathConfig } from "../path-config.js"; +import { encodeVersionId } from "../version-id.js"; +import type { VerifyFinding } from "../wire.js"; +import type { VerifyAccumulator } from "./verify.js"; + +/** Page size for the keyset-paginated verify scans. */ +export const SCAN_BATCH = 500; + +export type CheckContext = { + storage: Storage; + repo: RepoRow; + /** Effective path config for the repo (system/hidden sigils, etc.). */ + pathConfig: PathConfig; + /** Effective link-extraction config for the repo (the `links` family). */ + linkConfig: LinkConfig; + claims: ClaimMatcher[] | null; + acc: VerifyAccumulator; + /** Whether `check` (family or full code) is selected this run. */ + selected: (check: string) => boolean; +}; + +/** Encode an internal version id for the wire (opaque string, §3.3). */ +export function vid(id: number): string { + return encodeVersionId(id); +} + +/** Encode an internal document id for the wire (reuses the version-id codec). */ +export function did(id: number): string { + return encodeVersionId(id); +} + +/** + * True when the caller may read `path` in this repo. Absent scope = full + * visibility. Used to drop findings that would leak an unreadable path (§2.7). + */ +export function canRead(ctx: CheckContext, path: string): boolean { + return ctx.claims === null || claimsGrantRead(ctx.claims, ctx.repo.slug, path); +} + +/** Build a finding pre-filled with the repo slug. */ +export function finding(ctx: CheckContext, f: Omit): VerifyFinding { + return { repo: ctx.repo.slug, ...f }; +} diff --git a/src/kernel/verify/chunks.ts b/src/kernel/verify/chunks.ts new file mode 100644 index 0000000..8b2c0cd --- /dev/null +++ b/src/kernel/verify/chunks.ts @@ -0,0 +1,139 @@ +/** + * `chunks` verify family (docs/verify-plan.md §2.5) — embedding provenance. + * + * Structural consistency only, never vector quality (§1 Out). Two shapes: + * + * • Whole-store (chunks/backlog aren't repo-partitioned): `chunks.orphan`, + * `chunks.backlog_orphan`, `chunks.mixed_dim`. Run once per verify call in + * an all-repos run; skipped-with-note under a `--repo` filter (a scoped run + * can't attribute a version that no longer exists to a repo). + * • Per-repo: `chunks.unembedded` — a repo's live version with neither chunk + * rows nor a backlog entry. Gated on an embedder being configured (§2.5): + * with no embedder every live version is "unembedded", which is noise. + * + * "Orphan" means the referenced version does NOT exist — not merely that it's + * superseded. The embed worker leaves chunks on superseded versions on purpose + * (`worker.ts`), so a non-live-but-existing version keeping chunks is normal. + */ + +import { type CheckContext, SCAN_BATCH, finding, vid } from "./checks.js"; + +/** Whole-store chunk checks — call once per verify, not per repo. */ +export async function checkChunksStore(ctx: CheckContext): Promise { + const orphan = ctx.selected("chunks.orphan"); + const backlogOrphan = ctx.selected("chunks.backlog_orphan"); + const mixedDim = ctx.selected("chunks.mixed_dim"); + + if (orphan) { + let after = 0; + for (;;) { + const ids = await ctx.storage.chunks_orphan_version_ids({ + after_id: after, + limit: SCAN_BATCH, + }); + if (ids.length === 0) break; + for (const id of ids) { + ctx.acc.add( + finding(ctx, { + check: "chunks.orphan", + severity: "error", + version_id: vid(id), + detail: { reason: "chunk references a nonexistent version" }, + }), + ); + } + after = ids[ids.length - 1] as number; + } + } + + if (backlogOrphan) { + let after = 0; + for (;;) { + const ids = await ctx.storage.backlog_orphan_version_ids({ + after_id: after, + limit: SCAN_BATCH, + }); + if (ids.length === 0) break; + for (const id of ids) { + ctx.acc.add( + finding(ctx, { + check: "chunks.backlog_orphan", + severity: "error", + version_id: vid(id), + detail: { reason: "backlog entry references a nonexistent version" }, + }), + ); + } + after = ids[ids.length - 1] as number; + } + } + + if (mixedDim) { + let after = 0; + for (;;) { + const rows = await ctx.storage.chunks_dims_by_version({ after_id: after, limit: SCAN_BATCH }); + if (rows.length === 0) break; + for (const row of rows) { + if (row.dims.length > 1) { + ctx.acc.add( + finding(ctx, { + check: "chunks.mixed_dim", + severity: "error", + version_id: vid(row.version_id), + detail: { dims: row.dims.slice().sort((a, b) => a - b) }, + }), + ); + } + } + after = rows[rows.length - 1]?.version_id as number; + } + } +} + +/** + * Per-repo `chunks.unembedded`. Only meaningful when an embedder is configured + * — the caller (kernel) decides that and skips-with-note otherwise. A live + * version with neither chunks nor a pending backlog entry has fallen out of + * the pipeline. + */ +export async function checkChunksUnembedded(ctx: CheckContext): Promise { + if (!ctx.selected("chunks.unembedded")) return; + + // Membership sets are whole-store; build them once as Sets of version ids. + const chunked = await collectAll((after) => + ctx.storage.chunks_all_version_ids({ after_id: after, limit: SCAN_BATCH }), + ); + const backlogged = await collectAll((after) => + ctx.storage.backlog_all_version_ids({ after_id: after, limit: SCAN_BATCH }), + ); + + const live = await ctx.storage.versions_live_by_repo(ctx.repo.id); + for (const v of live) { + if (!chunked.has(v.id) && !backlogged.has(v.id)) { + ctx.acc.add( + finding(ctx, { + check: "chunks.unembedded", + severity: "warn", + document_id: vid(v.document_id), + version_id: vid(v.id), + path: v.path, + detail: {}, + suggested_fix: "mrplex embed backfill", + }), + ); + } + } +} + +/** Drain a keyset-paginated id scan into a Set. */ +async function collectAll(page: (afterId: number) => Promise): Promise> { + const out = new Set(); + let after = 0; + for (;;) { + const ids = await page(after); + if (ids.length === 0) break; + for (const id of ids) out.add(id); + after = ids[ids.length - 1] as number; + } + return out; +} diff --git a/src/kernel/verify/content.ts b/src/kernel/verify/content.ts new file mode 100644 index 0000000..478ba0a --- /dev/null +++ b/src/kernel/verify/content.ts @@ -0,0 +1,190 @@ +/** + * `hash` + `frontmatter` verify families (docs/verify-plan.md §2.2, §2.3). + * + * Both re-derive from a version's stored bytes, so they share one + * `versions_all` walk (a version scan is the expensive part; running both + * families over it is free). Per version: + * + * • hash.mismatch / hash.missing — recompute contentHash(frontmatter_raw, + * body) and compare to the stored content_hash column. + * • frontmatter.parse_error / .divergence — re-parse frontmatter_raw as YAML + * and deep-equal it against the stored frontmatter JSON. + * • frontmatter.system_leak — a `$`-prefixed key must never be persisted + * (canonicalizeFrontmatter strips them at write time); a leak corrupts + * $content_hash and re-injection. + * + * `frontmatter.divergence` is strict and always an error: in a correct store + * raw and JSON are written together in one tx and cannot drift, so a finding + * means the query index is lying (verify-plan §2.3, §8). + */ + +import { contentHash } from "../../markdown/content-hash.js"; +import { FrontmatterInvalidError, parse as parseFrontmatter } from "../../markdown/frontmatter.js"; +import type { FrontmatterJson } from "../../storage/types.js"; +import { INTRINSIC_SIGIL } from "../constants.js"; +import { type CheckContext, SCAN_BATCH, canRead, did, finding, vid } from "./checks.js"; + +export async function checkContent(ctx: CheckContext): Promise { + const hashMismatch = ctx.selected("hash.mismatch"); + const hashMissing = ctx.selected("hash.missing"); + const parseError = ctx.selected("frontmatter.parse_error"); + const divergence = ctx.selected("frontmatter.divergence"); + const systemLeak = ctx.selected("frontmatter.system_leak"); + + const anyHash = hashMismatch || hashMissing; + const anyFm = parseError || divergence || systemLeak; + if (!anyHash && !anyFm) return; + + let afterId = 0; + for (;;) { + const versions = await ctx.storage.versions_all({ + repo_id: ctx.repo.id, + after_id: afterId, + limit: SCAN_BATCH, + }); + if (versions.length === 0) break; + // Note: the chain family already tallies versions_scanned via + // versions_by_document; this family doesn't double-count. + + for (const v of versions) { + afterId = v.id; // advance the keyset cursor first (defensive: no continue today) + const path = canRead(ctx, v.path) ? v.path : undefined; + + if (anyHash) { + if (v.content_hash === null) { + if (hashMissing) { + ctx.acc.add( + finding(ctx, { + check: "hash.missing", + severity: "warn", + document_id: did(v.document_id), + version_id: vid(v.id), + path, + detail: {}, + suggested_fix: "mrplex hash backfill", + }), + ); + } + } else if (hashMismatch) { + const computed = contentHash(v.frontmatter_raw, v.body); + if (computed !== v.content_hash) { + ctx.acc.add( + finding(ctx, { + check: "hash.mismatch", + severity: "error", + document_id: did(v.document_id), + version_id: vid(v.id), + path, + detail: { stored: v.content_hash, computed }, + }), + ); + } + } + } + + if (anyFm) { + checkFrontmatter(ctx, v, path, { parseError, divergence, systemLeak }); + } + } + } +} + +function checkFrontmatter( + ctx: CheckContext, + v: { id: number; document_id: number; frontmatter_raw: string; frontmatter: FrontmatterJson }, + path: string | undefined, + which: { parseError: boolean; divergence: boolean; systemLeak: boolean }, +): void { + // system_leak: a `$`-prefixed top-level key in the stored raw or JSON. + if (which.systemLeak) { + const rawLeak = hasSystemLine(v.frontmatter_raw); + const jsonLeak = Object.keys(v.frontmatter).some((k) => k.startsWith(INTRINSIC_SIGIL)); + if (rawLeak || jsonLeak) { + ctx.acc.add( + finding(ctx, { + check: "frontmatter.system_leak", + severity: "error", + document_id: did(v.document_id), + version_id: vid(v.id), + path, + detail: { in_raw: rawLeak, in_json: jsonLeak }, + }), + ); + } + } + + if (!which.parseError && !which.divergence) return; + + let reparsed: FrontmatterJson; + try { + reparsed = parseFrontmatter(v.frontmatter_raw); + } catch (err) { + if (which.parseError) { + ctx.acc.add( + finding(ctx, { + check: "frontmatter.parse_error", + severity: "error", + document_id: did(v.document_id), + version_id: vid(v.id), + path, + detail: { + reason: err instanceof FrontmatterInvalidError ? err.message : String(err), + }, + }), + ); + } + return; // can't diverge-check what won't parse + } + + if (which.divergence && !deepEqual(reparsed, v.frontmatter)) { + ctx.acc.add( + finding(ctx, { + check: "frontmatter.divergence", + severity: "error", + document_id: did(v.document_id), + version_id: vid(v.id), + path, + detail: { keys_differing: differingKeys(reparsed, v.frontmatter) }, + }), + ); + } +} + +/** True if any top-level line begins with the intrinsic sigil (`$key:`). */ +function hasSystemLine(raw: string): boolean { + if (!raw.includes(INTRINSIC_SIGIL)) return false; + for (const line of raw.split("\n")) { + if (line.startsWith(INTRINSIC_SIGIL)) return true; + } + return false; +} + +/** Top-level keys whose values differ (or are present in only one side). */ +function differingKeys(a: FrontmatterJson, b: FrontmatterJson): string[] { + const keys = new Set([...Object.keys(a), ...Object.keys(b)]); + const out: string[] = []; + for (const k of keys) { + if (!deepEqual(a[k], b[k])) out.push(k); + } + return out.sort(); +} + +/** Structural deep-equality for JSON values (stored frontmatter is JSON). */ +function deepEqual(a: unknown, b: unknown): boolean { + if (a === b) return true; + if (typeof a !== typeof b) return false; + if (a === null || b === null) return a === b; + if (Array.isArray(a) || Array.isArray(b)) { + if (!Array.isArray(a) || !Array.isArray(b) || a.length !== b.length) return false; + return a.every((x, i) => deepEqual(x, b[i])); + } + if (typeof a === "object" && typeof b === "object") { + const ao = a as Record; + const bo = b as Record; + const ak = Object.keys(ao); + const bk = Object.keys(bo); + if (ak.length !== bk.length) return false; + return ak.every((k) => Object.hasOwn(bo, k) && deepEqual(ao[k], bo[k])); + } + return false; +} diff --git a/src/kernel/verify/fts.ts b/src/kernel/verify/fts.ts new file mode 100644 index 0000000..0254a58 --- /dev/null +++ b/src/kernel/verify/fts.ts @@ -0,0 +1,61 @@ +/** + * `fts` verify family (docs/verify-plan.md §2.4) — SQLite-only. + * + * SQLite maintains a separate `fts_docs` external-content FTS5 table via + * triggers, so its rowid membership can drift from `versions.id` (a trigger + * that didn't fire → missing; a stray shadow-table row → orphan). The + * invariant is a bijection over ALL versions, not the live set. Postgres has + * no separate structure (`fts_tsv` is a generated column that can't drift), so + * this family is skipped-with-note there — the kernel gates it on the + * `VerifyFtsScans` capability before calling. + * + * Both scans are repo-agnostic (rowids are global), so this runs once per + * verify call, not once per repo — the kernel calls it against the first + * repo's context only. + */ + +import type { VerifyFtsScans } from "../../storage/types.js"; +import { type CheckContext, SCAN_BATCH, finding, vid } from "./checks.js"; + +export async function checkFts(ctx: CheckContext, scans: VerifyFtsScans): Promise { + const missing = ctx.selected("fts.missing"); + const orphan = ctx.selected("fts.orphan"); + + if (missing) { + let afterId = 0; + for (;;) { + const ids = await scans.fts_missing_rowids({ after_id: afterId, limit: SCAN_BATCH }); + if (ids.length === 0) break; + for (const id of ids) { + ctx.acc.add( + finding(ctx, { + check: "fts.missing", + severity: "error", + version_id: vid(id), + detail: {}, + suggested_fix: "rebuild the fts_docs index", + }), + ); + } + afterId = ids[ids.length - 1] as number; + } + } + + if (orphan) { + let afterId = 0; + for (;;) { + const ids = await scans.fts_orphan_rowids({ after_id: afterId, limit: SCAN_BATCH }); + if (ids.length === 0) break; + for (const id of ids) { + ctx.acc.add( + finding(ctx, { + check: "fts.orphan", + severity: "error", + detail: { fts_rowid: id }, + }), + ); + } + afterId = ids[ids.length - 1] as number; + } + } +} diff --git a/src/kernel/verify/links.ts b/src/kernel/verify/links.ts new file mode 100644 index 0000000..39f4f16 --- /dev/null +++ b/src/kernel/verify/links.ts @@ -0,0 +1,216 @@ +/** + * `links` verify family (docs/verify-plan.md §2.6) — link index vs. re-extraction. + * + * Re-runs the pure extraction+normalization pipeline (`extractEdges` → + * `normalizeEdges`) for each live version under the repo's effective link + * config, resolves candidates against an in-memory snapshot of the live path + * set (the same first-candidate-wins, self-link-drop, dense-ord logic as the + * write path's `reindexOutboundLinks`), and diffs the result against the stored + * `links` rows. Then checks resolution correctness against the live set. + * + * Findings: + * • links.set_mismatch — re-extracted (ord,field,target_raw) set ≠ stored + * • links.misresolved_dangling — target_id null but a live doc exists at target_norm + * • links.misresolved_bound — target_id points at a missing/non-live doc, or one + * whose current path's fold ≠ target_norm + * • links.self_link — source_id == target_id (excluded by construction) + * • links.deleted_source_has_outbound — a doc in the system namespace still has edges + */ + +import { extractEdges } from "../../links/extract.js"; +import { normalizeEdges } from "../../links/resolve.js"; +import type { LinkRow, VersionRow } from "../../storage/types.js"; +import { normalizeKey } from "../casefold.js"; +import { pathIsInSystemNamespace } from "../deletion.js"; +import { type CheckContext, canRead, did, finding } from "./checks.js"; + +/** The identity-independent shape of an edge, for set comparison. */ +type EdgeKey = { ord: number; field: string; target_raw: string }; + +export async function checkLinks(ctx: CheckContext): Promise { + const setMismatch = ctx.selected("links.set_mismatch"); + const misDangling = ctx.selected("links.misresolved_dangling"); + const misBound = ctx.selected("links.misresolved_bound"); + const selfLink = ctx.selected("links.self_link"); + const deletedOutbound = ctx.selected("links.deleted_source_has_outbound"); + if (!setMismatch && !misDangling && !misBound && !selfLink && !deletedOutbound) return; + + const live = await ctx.storage.versions_live_by_repo(ctx.repo.id); + // Live path snapshot: folded path → document id. This is the resolution + // universe the write path binds against (via version_current's fold). + const liveByNorm = new Map(); + for (const v of live) { + liveByNorm.set(normalizeKey(v.path), v.document_id); + } + + const storedRows = await ctx.storage.links_by_repo(ctx.repo.id); + const storedBySource = new Map(); + for (const row of storedRows) { + const list = storedBySource.get(row.source_id); + if (list) list.push(row); + else storedBySource.set(row.source_id, [row]); + } + + // --- set_mismatch: re-extract each live doc and diff against stored rows. + if (setMismatch) { + for (const v of live) { + const expected = resolveExpected(ctx, v, liveByNorm); + const stored = (storedBySource.get(v.document_id) ?? []) + .slice() + .sort((a, b) => a.ord - b.ord) + .map((r) => ({ ord: r.ord, field: r.field, target_raw: r.target_raw })); + const { missing, extra } = diffEdgeSets(expected, stored); + if (missing.length > 0 || extra.length > 0) { + ctx.acc.add( + finding(ctx, { + check: "links.set_mismatch", + severity: "error", + document_id: did(v.document_id), + path: canRead(ctx, v.path) ? v.path : undefined, + detail: { missing, extra }, + }), + ); + } + } + } + + // --- per-row resolution correctness + self-link. + const currentPathByDoc = new Map(); + for (const v of live) currentPathByDoc.set(v.document_id, v.path); + + for (const row of storedRows) { + if (row.target_id === null) { + if (misDangling && liveByNorm.has(row.target_norm)) { + ctx.acc.add( + finding(ctx, { + check: "links.misresolved_dangling", + severity: "error", + document_id: did(row.source_id), + detail: { + target_norm: row.target_norm, + should_bind_to: did(liveByNorm.get(row.target_norm) as number), + }, + }), + ); + } + continue; + } + + if (selfLink && row.target_id === row.source_id) { + ctx.acc.add( + finding(ctx, { + check: "links.self_link", + severity: "warn", + document_id: did(row.source_id), + detail: { ord: row.ord }, + }), + ); + } + + if (misBound) { + const targetPath = currentPathByDoc.get(row.target_id); + // A bound edge should point at a live document whose current folded path + // equals the edge's target_norm. Missing (not live) or a fold mismatch + // is a stale binding renames alone can't explain. + if (targetPath === undefined) { + ctx.acc.add( + finding(ctx, { + check: "links.misresolved_bound", + severity: "error", + document_id: did(row.source_id), + detail: { target_id: did(row.target_id), reason: "target not live in this repo" }, + }), + ); + } else if (normalizeKey(targetPath) !== row.target_norm) { + ctx.acc.add( + finding(ctx, { + check: "links.misresolved_bound", + severity: "error", + document_id: did(row.source_id), + detail: { + target_id: did(row.target_id), + target_norm: row.target_norm, + actual_norm: normalizeKey(targetPath), + reason: "bound target's folded path differs from target_norm", + }, + }), + ); + } + } + } + + // --- deleted_source_has_outbound: a doc under a system sigil keeps edges. + if (deletedOutbound) { + // Sources with stored edges whose current path is in the system namespace. + // We need the current path of each source doc, incl. deleted ones — the + // live snapshot only has user-territory docs, so scan the source ids that + // have edges but are absent from the live set. + const liveDocIds = new Set(live.map((v) => v.document_id)); + for (const [sourceId] of storedBySource) { + if (liveDocIds.has(sourceId)) continue; // live doc — edges are expected + // Not in the live set: either deleted (system namespace) or gone. Either + // way it shouldn't have outbound edges. Resolve its current path to + // report it precisely. + const versions = await ctx.storage.versions_by_document(sourceId); + const current = versions.find((v) => v.next_id === null); + const inSystem = + current !== undefined && + pathIsInSystemNamespace(current.path, ctx.pathConfig.system_sigils); + ctx.acc.add( + finding(ctx, { + check: "links.deleted_source_has_outbound", + severity: "error", + document_id: did(sourceId), + detail: { + edge_count: (storedBySource.get(sourceId) as LinkRow[]).length, + reason: inSystem ? "source is deleted (system namespace)" : "source not live", + }, + }), + ); + } + } +} + +/** + * Re-derive the expected stored edge keys for one live version — the pure twin + * of `reindexOutboundLinks`: extract → normalize → resolve against the live + * snapshot, drop external (no-candidate) and self edges, re-pack ord densely. + */ +function resolveExpected( + ctx: CheckContext, + v: VersionRow, + liveByNorm: Map, +): EdgeKey[] { + const raw = extractEdges({ body: v.body, frontmatter: v.frontmatter, config: ctx.linkConfig }); + const normalized = normalizeEdges(raw, v.path, ctx.linkConfig); + const out: EdgeKey[] = []; + let ord = 0; + for (const edge of normalized) { + if (edge.candidates.length === 0) continue; // external / unresolvable + // Resolve: first candidate that maps to a live doc; else dangling. + let targetId: number | null = null; + for (const candidate of edge.candidates) { + const docId = liveByNorm.get(normalizeKey(candidate)); + if (docId !== undefined) { + targetId = docId; + break; + } + } + if (targetId === v.document_id) continue; // self-link dropped + out.push({ ord: ord++, field: edge.field, target_raw: edge.target_raw }); + } + return out; +} + +/** Diff two ord-ordered edge-key lists positionally by (ord,field,target_raw). */ +function diffEdgeSets( + expected: EdgeKey[], + stored: EdgeKey[], +): { missing: EdgeKey[]; extra: EdgeKey[] } { + const key = (e: EdgeKey) => `${e.ord}${e.field}${e.target_raw}`; + const expSet = new Set(expected.map(key)); + const storedSet = new Set(stored.map(key)); + const missing = expected.filter((e) => !storedSet.has(key(e))); + const extra = stored.filter((e) => !expSet.has(key(e))); + return { missing, extra }; +} diff --git a/src/kernel/verify/verify.test.ts b/src/kernel/verify/verify.test.ts index aec9246..8f8c1b0 100644 --- a/src/kernel/verify/verify.test.ts +++ b/src/kernel/verify/verify.test.ts @@ -92,9 +92,13 @@ describe("kernel.verify skeleton (WS1)", () => { const report = await kernel.verify({}, {}); expect(report.findings).toEqual([]); expect(report.truncated).toBe(false); - expect(report.checks_skipped).toEqual([]); expect(report.counts.versions_scanned).toBe(0); expect(report.counts.documents_scanned).toBe(0); + // No embedder configured in this harness, so chunks.unembedded is + // skipped-and-noted (once, deduped across repos). + expect(report.checks_skipped).toEqual([ + { check: "chunks.unembedded", reason: "no embedder configured" }, + ]); }); it("throws repo_not_found for an unknown --repo", async () => { diff --git a/src/kernel/verify/verify.ts b/src/kernel/verify/verify.ts index 8252cc2..3ce445e 100644 --- a/src/kernel/verify/verify.ts +++ b/src/kernel/verify/verify.ts @@ -18,12 +18,24 @@ * check families (WS3) fill in `runChecks` per repo. */ -import type { RepoRow, Storage } from "../../storage/types.js"; +import { + HARDCODED_DEFAULTS as LINK_DEFAULTS, + type LinkConfig, + effectiveLinkConfig, + parseRepoOverride as parseLinkOverride, +} from "../../links/link-config.js"; +import { type RepoRow, type Storage, hasVerifyFtsScans } from "../../storage/types.js"; import { type ClaimMatcher, claimsGrantRepo } from "../auth/scope.js"; import { repoNotFound } from "../errors.js"; import type { PathConfig } from "../path-config.js"; import { effectivePathConfig, parseRepoOverride } from "../path-config.js"; import type { VerifyFinding, VerifyReport, VerifySeverity, VerifySpec } from "../wire.js"; +import { checkChain } from "./chain.js"; +import type { CheckContext } from "./checks.js"; +import { checkChunksStore, checkChunksUnembedded } from "./chunks.js"; +import { checkContent } from "./content.js"; +import { checkFts } from "./fts.js"; +import { checkLinks } from "./links.js"; /** Default cap on emitted findings; `counts` stay exact past it (verify-plan §3). */ export const DEFAULT_MAX_FINDINGS = 10_000; @@ -31,6 +43,8 @@ export const DEFAULT_MAX_FINDINGS = 10_000; export type VerifyDeps = { storage: Storage; serverPathConfig: PathConfig; + /** Server-level link-extraction config; per-repo overrides layer on top. */ + serverLinkConfig?: LinkConfig; /** * Whether an embedder is configured for this store (flag → MRPLEX_EMBEDDER → * config). Gates the `chunks.unembedded` check: with no embedder it's skipped @@ -80,6 +94,9 @@ export class VerifyAccumulator { } skip(check: string, reason: string): void { + // Dedupe: a per-repo skip (e.g. chunks.unembedded with no embedder) is + // noted once for the whole run, not once per repo. + if (this.skipped.some((s) => s.check === check)) return; this.skipped.push({ check, reason }); } @@ -115,19 +132,90 @@ export async function runVerify( deps: VerifyDeps, ): Promise { const repos = await resolveRepos(claims, spec.repo, deps); + const wholeStore = spec.repo === undefined; // whole-store checks need all repos const acc = new VerifyAccumulator( spec.min_severity ?? "warn", spec.max_findings ?? DEFAULT_MAX_FINDINGS, ); + const selected = (check: string): boolean => checkSelected(check, spec.checks); + const serverLinkConfig = deps.serverLinkConfig ?? LINK_DEFAULTS; + // Per-repo families: chain, hash/frontmatter, links, chunks.unembedded. for (const repo of repos) { - await runChecks(acc, repo, claims, spec, deps); + const ctx: CheckContext = { + storage: deps.storage, + repo, + pathConfig: effectivePathConfig(deps.serverPathConfig, parseRepoOverride(repo.path_config)), + linkConfig: effectiveLinkConfig(serverLinkConfig, parseLinkOverride(repo.link_config)), + claims, + acc, + selected, + }; + await checkChain(ctx); + await checkContent(ctx); + await checkLinks(ctx); + if (deps.embedderConfigured) { + await checkChunksUnembedded(ctx); + } else if (selected("chunks.unembedded")) { + acc.skip("chunks.unembedded", "no embedder configured"); + } + } + + // Whole-store families (fts, chunks orphan/mixed-dim): not repo-partitioned, + // so they run once over the first repo's context — and only in an all-repos + // run, since a --repo filter can neither attribute nor bound them (§2.5). + if (repos.length > 0) { + const anchor = repos[0] as RepoRow; + const storeCtx: CheckContext = { + storage: deps.storage, + repo: anchor, + pathConfig: effectivePathConfig(deps.serverPathConfig, parseRepoOverride(anchor.path_config)), + linkConfig: effectiveLinkConfig(serverLinkConfig, parseLinkOverride(anchor.link_config)), + claims, + acc, + selected, + }; + await runWholeStoreChecks(storeCtx, wholeStore, deps, acc, selected); } return acc.report(); } +/** + * fts + chunks orphan/mixed-dim families. Skipped-with-note under a `--repo` + * filter (can't attribute a gone version to a repo), and the fts family is + * further gated on the SQLite-only `VerifyFtsScans` capability (§2.4). + */ +async function runWholeStoreChecks( + ctx: CheckContext, + wholeStore: boolean, + deps: VerifyDeps, + acc: VerifyAccumulator, + selected: (check: string) => boolean, +): Promise { + const ftsSelected = selected("fts.missing") || selected("fts.orphan"); + const chunkStoreSelected = + selected("chunks.orphan") || selected("chunks.backlog_orphan") || selected("chunks.mixed_dim"); + + if (!wholeStore) { + if (ftsSelected) acc.skip("fts", "whole-store check; omit --repo to run"); + if (chunkStoreSelected) acc.skip("chunks", "whole-store check; omit --repo to run"); + return; + } + + if (ftsSelected) { + if (hasVerifyFtsScans(deps.storage)) { + await checkFts(ctx, deps.storage); + } else { + acc.skip("fts", "postgres: fts_tsv is a generated column, structurally consistent"); + } + } + if (chunkStoreSelected) { + await checkChunksStore(ctx); + } +} + /** * The repos this call verifies. A named `repo` resolves to exactly one (and * gates existence through scope, same shape as `resolveRepo` in kernel.ts — an @@ -156,23 +244,3 @@ async function resolveRepos( (r) => !isSystem(r.slug) && (claims === null || claimsGrantRepo(claims, r.slug)), ); } - -/** - * Run the selected check families against one repo, appending findings to - * `acc`. WS1 scaffold — the six families (WS3) plug in here. `effectiveConfig` - * is resolved once per repo so sigil-aware checks share it. - */ -async function runChecks( - acc: VerifyAccumulator, - repo: RepoRow, - _claims: ClaimMatcher[] | null, - _spec: VerifySpec, - deps: VerifyDeps, -): Promise { - const _effectiveConfig = effectivePathConfig( - deps.serverPathConfig, - parseRepoOverride(repo.path_config), - ); - // WS3 wires the check families in here; the accumulator + config are the - // seam they hang off. Intentionally empty in the WS1 skeleton. -} diff --git a/src/storage-postgres/adapter.ts b/src/storage-postgres/adapter.ts index ad5c3e4..01dafdd 100644 --- a/src/storage-postgres/adapter.ts +++ b/src/storage-postgres/adapter.ts @@ -821,6 +821,20 @@ class PostgresStorage implements Storage { }); } + async versions_by_document(document_id: number): Promise { + return this.withClient(async (c) => { + const res = await c.query( + `select id, document_id, repo_id, prev_id, next_id, path, + frontmatter_raw, frontmatter, body, author, created_at, content_hash + from versions + where document_id = $1 + order by id asc`, + [document_id], + ); + return res.rows as VersionRow[]; + }); + } + async chunks_all_version_ids(opts: { after_id: number; limit: number }): Promise { return this.withClient(async (c) => { const res = await c.query<{ version_id: number }>( @@ -845,6 +859,60 @@ class PostgresStorage implements Storage { }); } + async chunks_orphan_version_ids(opts: { after_id: number; limit: number }): Promise { + return this.withClient(async (c) => { + const res = await c.query<{ version_id: number }>( + `select distinct c.version_id as version_id from chunks c + left join versions v on v.id = c.version_id + where c.version_id > $1 and v.id is null + order by c.version_id asc limit $2`, + [opts.after_id, opts.limit], + ); + return res.rows.map((r) => Number(r.version_id)); + }); + } + + async backlog_orphan_version_ids(opts: { after_id: number; limit: number }): Promise { + return this.withClient(async (c) => { + const res = await c.query<{ version_id: number }>( + `select b.version_id as version_id from embedding_backlog b + left join versions v on v.id = b.version_id + where b.version_id > $1 and v.id is null + order by b.version_id asc limit $2`, + [opts.after_id, opts.limit], + ); + return res.rows.map((r) => Number(r.version_id)); + }); + } + + async chunks_dims_by_version(opts: { + after_id: number; + limit: number; + }): Promise<{ version_id: number; dims: number[] }[]> { + return this.withClient(async (c) => { + const ids = await c.query<{ version_id: number }>( + `select distinct version_id from chunks + where version_id > $1 + order by version_id asc limit $2`, + [opts.after_id, opts.limit], + ); + if (ids.rows.length === 0) return []; + const versionIds = ids.rows.map((r) => Number(r.version_id)); + const res = await c.query<{ version_id: number; dim: number }>( + `select distinct version_id, vector_dims(embedding) as dim from chunks + where version_id = ANY($1::bigint[]) and embedding is not null`, + [versionIds], + ); + const byVersion = new Map(); + for (const id of versionIds) byVersion.set(id, []); + for (const r of res.rows) byVersion.get(Number(r.version_id))?.push(Number(r.dim)); + return versionIds.map((version_id) => ({ + version_id, + dims: byVersion.get(version_id) ?? [], + })); + }); + } + async chunks_upsert( version_id: number, model: string, diff --git a/src/storage-sqlite/adapter.ts b/src/storage-sqlite/adapter.ts index 101f9fa..3458f32 100644 --- a/src/storage-sqlite/adapter.ts +++ b/src/storage-sqlite/adapter.ts @@ -762,6 +762,19 @@ class SqliteStorage implements Storage { .all(...params) as DocumentRow[]; } + async versions_by_document(document_id: number): Promise { + const rows = this.db + .prepare( + `select id, document_id, repo_id, prev_id, next_id, path, + frontmatter_raw, frontmatter, body, author, created_at, content_hash + from versions + where document_id = ? + order by id asc`, + ) + .all(document_id) as VersionRawRow[]; + return rows.map(hydrateVersion); + } + async chunks_all_version_ids(opts: { after_id: number; limit: number }): Promise { const rows = this.db .prepare( @@ -784,6 +797,61 @@ class SqliteStorage implements Storage { return rows.map((r) => r.version_id); } + async chunks_orphan_version_ids(opts: { after_id: number; limit: number }): Promise { + const rows = this.db + .prepare( + `select distinct c.version_id as version_id from chunks c + left join versions v on v.id = c.version_id + where c.version_id > ? and v.id is null + order by c.version_id asc limit ?`, + ) + .all(opts.after_id, opts.limit) as { version_id: number }[]; + return rows.map((r) => r.version_id); + } + + async backlog_orphan_version_ids(opts: { after_id: number; limit: number }): Promise { + const rows = this.db + .prepare( + `select b.version_id as version_id from embedding_backlog b + left join versions v on v.id = b.version_id + where b.version_id > ? and v.id is null + order by b.version_id asc limit ?`, + ) + .all(opts.after_id, opts.limit) as { version_id: number }[]; + return rows.map((r) => r.version_id); + } + + async chunks_dims_by_version(opts: { + after_id: number; + limit: number; + }): Promise<{ version_id: number; dims: number[] }[]> { + // One page of DISTINCT versions (id > after_id), then their distinct + // non-null embedding sizes. length(blob) is the byte count in SQLite — + // proportional to dimension, and only distinctness matters here. + const versionIds = this.db + .prepare( + `select distinct version_id from chunks + where version_id > ? + order by version_id asc limit ?`, + ) + .all(opts.after_id, opts.limit) as { version_id: number }[]; + if (versionIds.length === 0) return []; + const ph = versionIds.map(() => "?").join(","); + const rows = this.db + .prepare( + `select distinct version_id, length(embedding) as dim from chunks + where version_id in (${ph}) and embedding is not null`, + ) + .all(...versionIds.map((r) => r.version_id)) as { version_id: number; dim: number }[]; + const byVersion = new Map(); + for (const { version_id } of versionIds) byVersion.set(version_id, []); + for (const r of rows) byVersion.get(r.version_id)?.push(r.dim); + return versionIds.map(({ version_id }) => ({ + version_id, + dims: byVersion.get(version_id) ?? [], + })); + } + // fts verify scans (VerifyFtsScans capability, verify-plan §2.4). SQLite-only: // the version↔fts_docs rowid bijection can drift here (trigger-maintained // external-content table), unlike Postgres's generated fts_tsv column. diff --git a/src/storage/types.ts b/src/storage/types.ts index 0c6156f..ab38a27 100644 --- a/src/storage/types.ts +++ b/src/storage/types.ts @@ -343,8 +343,8 @@ export type Storage = { /** * One keyset page of `documents` rows in id order, id > `after_id`, capped at * `limit`, optionally scoped to `repo_id`. Feeds the `chain` family's - * per-document walk (via `version_history`) and `chain.orphan_document` - * (documents with zero versions). Keyset by id (verify-plan §4). + * per-document walk and `chain.orphan_document` (documents with zero + * versions). Keyset by id (verify-plan §4). */ documents_all(opts: { repo_id?: number; @@ -353,18 +353,58 @@ export type Storage = { }): Promise; /** - * Distinct version ids present in the `chunks` table, keyset-paginated by - * version id (id > `after_id`, capped at `limit`). The kernel intersects - * these against live/all version ids for `chunks.orphan` (verify-plan §2.5). + * ALL versions of one document, ascending by id, chain-INDEPENDENT (a plain + * `where document_id = ?`, not a walk from the current version). The `chain` + * verify family needs this: a corrupt chain may have zero or two `next_id IS + * NULL` rows, which `version_history`'s recursive-from-current walk can't + * traverse. Empty when the document has no versions (verify-plan §2.1). + */ + versions_by_document(document_id: number): Promise; + + /** + * Distinct version ids present in the `chunks` table (whether or not the + * version still exists / is live), keyset-paginated by version id. The + * `chunks.unembedded` check intersects live-version ids against this set to + * find live versions lacking chunks (verify-plan §2.5). */ chunks_all_version_ids(opts: { after_id: number; limit: number }): Promise; /** * Version ids present in the `embedding_backlog` table, keyset-paginated by - * version id. Feeds `chunks.backlog_orphan` (verify-plan §2.5). + * version id. With `chunks_all_version_ids`, tells `chunks.unembedded` + * which live versions have neither chunks nor a pending backlog entry. */ backlog_all_version_ids(opts: { after_id: number; limit: number }): Promise; + /** + * Distinct `chunks.version_id`s that have NO matching `versions` row — true + * orphans (an FK violation), keyset-paginated by version id. NOT + * merely-superseded versions: the embed worker legitimately leaves chunks on + * superseded versions (`worker.ts`), so "orphan" means the version is gone + * entirely (verify-plan §2.5). Whole-store (chunks aren't repo-partitioned). + */ + chunks_orphan_version_ids(opts: { after_id: number; limit: number }): Promise; + + /** + * `embedding_backlog.version_id`s with no matching `versions` row, keyset by + * version id. Feeds `chunks.backlog_orphan` (verify-plan §2.5). + */ + backlog_orphan_version_ids(opts: { after_id: number; limit: number }): Promise; + + /** + * Per-version distinct embedding dimensions across the `chunks` table, for + * version ids > `after_id`, capped at `limit` DISTINCT versions. "Dimension" + * is engine-defined but internally consistent (SQLite: blob byte length; + * Postgres: `vector_dims`) — only DISTINCTNESS matters: a version with more + * than one value has mixed-dimension vectors → `chunks.mixed_dim` (§2.5, the + * §5.3 "refuse mixed-dim writes" guard was bypassed). Null embeddings are + * ignored. Whole-store. + */ + chunks_dims_by_version(opts: { + after_id: number; + limit: number; + }): Promise<{ version_id: number; dims: number[] }[]>; + /** * All currently-live versions in a repo (i.e. rows where next_id IS NULL). * Used by `repos.set_path_config` to produce the advisory PathWarning[] diff --git a/test/verify-checks.test.ts b/test/verify-checks.test.ts new file mode 100644 index 0000000..d9e6d24 --- /dev/null +++ b/test/verify-checks.test.ts @@ -0,0 +1,365 @@ +/** + * Verify check families (docs/verify-plan.md §2) — the correctness proof. + * + * Each test seeds a healthy corpus through the kernel, asserts a clean report, + * then injects one specific corruption (usually via a second raw SQLite handle, + * since the store's own write path won't produce these states) and asserts the + * exact finding. SQLite-only: corruption injection is engine-specific, and the + * check logic is pure/shared, so SQLite coverage proves the family. + */ + +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import Database from "better-sqlite3"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import type { CallContext } from "../src/kernel/context.js"; +import { type Kernel, createKernel } from "../src/kernel/kernel.js"; +import { decodeVersionId } from "../src/kernel/version-id.js"; +import type { VerifyFinding } from "../src/kernel/wire.js"; +import { sqliteAdapter } from "../src/storage-sqlite/adapter.js"; +import type { Storage } from "../src/storage/types.js"; + +const ROOT: CallContext = {}; + +let dbPath: string; +let storage: Storage; +let kernel: Kernel; + +beforeEach(async () => { + dbPath = join(tmpdir(), `mrplex-verify-checks-${Date.now()}-${Math.random()}.db`); + storage = await sqliteAdapter.open({ database: `sqlite:${dbPath}` }); + kernel = createKernel(storage); + await kernel.repos.create(ROOT, "notes"); +}); + +afterEach(async () => { + await storage.close(); +}); + +/** + * Run `fn` against a second raw handle to the DB file, always closing it — an + * un-closed better-sqlite3 handle keeps the vitest worker alive (looks like a + * hang). Corruption injections must also respect the partial unique indexes + * (e.g. can't null a next_id if that would create two live rows at one path), + * or the UPDATE throws and leaks the handle. + */ +function withRaw(fn: (db: Database.Database) => T): T { + const db = new Database(dbPath); + try { + return fn(db); + } finally { + db.close(); + } +} + +/** + * Inject corruption that the schema's own constraints normally forbid — the + * whole point of some checks is to catch states a healthy write path can't + * produce. Disables FK enforcement and drops the partial unique indexes for the + * mutation, so e.g. a dangling chunk ref or a two-live-versions document can be + * created. The indexes aren't restored (the DB is thrown away after the test). + */ +function withRawUnsafe(fn: (db: Database.Database) => T): T { + return withRaw((db) => { + db.pragma("foreign_keys = OFF"); + db.exec("drop index if exists versions_document_current_uidx"); + db.exec("drop index if exists versions_repo_path_current_uidx"); + db.exec("drop index if exists versions_repo_pathnorm_current_uidx"); + return fn(db); + }); +} + +/** Findings for a given check code. */ +function findingsFor(report: { findings: VerifyFinding[] }, check: string): VerifyFinding[] { + return report.findings.filter((f) => f.check === check); +} + +describe("verify: clean corpus", () => { + it("reports no findings on a healthy repo", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "title: A\n", + body: "see [B](b.md)\n", + }); + await kernel.docs.create(ROOT, "notes", "b.md", { + frontmatter_raw: "title: B\n", + body: "hello\n", + }); + const report = await kernel.verify(ROOT, {}); + expect(report.findings).toEqual([]); + expect(report.counts.documents_scanned).toBe(2); + }); +}); + +describe("verify: chain family", () => { + it("chain.orphan_document — a document with no versions", async () => { + // documents_create with no version_insert leaves an orphan. + await storage.documents_create(1); + const report = await kernel.verify(ROOT, { checks: ["chain.orphan_document"] }); + expect(findingsFor(report, "chain.orphan_document")).toHaveLength(1); + }); + + it("chain.prev_next_asymmetry — v1.next_id doesn't point back at v2", async () => { + // One document, two versions (edit) + a decoy version in another doc. v2's + // prev_id still points at v1, but repoint v1.next_id at the decoy: within + // the document, v1 (prev of v2) no longer claims v2 → asymmetry. Using a + // real decoy id satisfies the next_id FK; v1 stays non-current (its next_id + // is non-null) so a.md's single live slot is unaffected. + const v1 = await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "", + body: "one\n", + }); + await kernel.docs.put(ROOT, "notes", v1.version_id, "a.md", { body: "two\n" }); + const decoy = await kernel.docs.create(ROOT, "notes", "d.md", { + frontmatter_raw: "", + body: "d\n", + }); + const v1Id = decodeVersionId(v1.version_id); + const decoyId = decodeVersionId(decoy.version_id); + withRaw((r) => r.prepare("update versions set next_id = ? where id = ?").run(decoyId, v1Id)); + const report = await kernel.verify(ROOT, { checks: ["chain.prev_next_asymmetry"] }); + expect(findingsFor(report, "chain.prev_next_asymmetry").length).toBeGreaterThanOrEqual(1); + }); + + it("chain.multiple_current — two live versions in one document", async () => { + // Move a.md → b.md so the document has two versions at DIFFERENT paths. + // Nulling the superseded version's next_id then makes both live without + // colliding on (repo, path_norm). + const v1 = await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "", + body: "one\n", + }); + await kernel.docs.put(ROOT, "notes", v1.version_id, "b.md", { body: "one\n" }); + const v1Id = decodeVersionId(v1.version_id); + // Two live rows in one document violates versions_document_current_uidx, so + // the injection must drop that index (withRawUnsafe). + withRawUnsafe((r) => r.prepare("update versions set next_id = null where id = ?").run(v1Id)); + const report = await kernel.verify(ROOT, { checks: ["chain.multiple_current"] }); + expect(findingsFor(report, "chain.multiple_current")).toHaveLength(1); + }); + + it("chain.no_current — no live version in a document", async () => { + const a = await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "", + body: "one\n", + }); + const b = await kernel.docs.create(ROOT, "notes", "b.md", { + frontmatter_raw: "", + body: "two\n", + }); + const aId = decodeVersionId(a.version_id); + const bId = decodeVersionId(b.version_id); + // Point a's only version at b's version (a real id → satisfies the FK), so + // a's document has no next_id-null row → headless. Frees a.md's live slot, + // so the (repo, path_norm) unique index isn't violated either. + withRaw((r) => r.prepare("update versions set next_id = ? where id = ?").run(bId, aId)); + const report = await kernel.verify(ROOT, { checks: ["chain.no_current"] }); + expect(findingsFor(report, "chain.no_current")).toHaveLength(1); + }); + + it("chain.repo_mismatch — version repo_id disagrees with its document", async () => { + await kernel.repos.create(ROOT, "other"); + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "one\n" }); + withRaw((r) => { + const otherId = ( + r.prepare("select id from repos where slug='other'").get() as { + id: number; + } + ).id; + r.prepare("update versions set repo_id = ? where path = 'a.md'").run(otherId); + }); + const report = await kernel.verify(ROOT, { checks: ["chain.repo_mismatch"] }); + expect(findingsFor(report, "chain.repo_mismatch").length).toBeGreaterThanOrEqual(1); + }); +}); + +describe("verify: hash family", () => { + it("hash.mismatch — stored content_hash disagrees with recomputed", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "real body\n" }); + withRaw((r) => + r.prepare("update versions set content_hash = 'deadbeef' where path = 'a.md'").run(), + ); + const report = await kernel.verify(ROOT, { checks: ["hash.mismatch"] }); + const hits = findingsFor(report, "hash.mismatch"); + expect(hits).toHaveLength(1); + expect(hits[0]?.detail.stored).toBe("deadbeef"); + }); + + it("hash.missing — a pre-backfill null content_hash (warn)", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "body\n" }); + withRaw((r) => r.prepare("update versions set content_hash = null where path = 'a.md'").run()); + const report = await kernel.verify(ROOT, { checks: ["hash.missing"] }); + const hits = findingsFor(report, "hash.missing"); + expect(hits).toHaveLength(1); + expect(hits[0]?.severity).toBe("warn"); + expect(hits[0]?.suggested_fix).toBe("mrplex hash backfill"); + }); +}); + +describe("verify: frontmatter family", () => { + it("frontmatter.divergence — stored JSON disagrees with re-parsed raw", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "status: draft\n", + body: "b\n", + }); + // Tamper the parsed JSON so it no longer matches the raw YAML. + withRaw((r) => + r + .prepare("update versions set frontmatter = ? where path = 'a.md'") + .run(JSON.stringify({ status: "published" })), + ); + const report = await kernel.verify(ROOT, { checks: ["frontmatter.divergence"] }); + const hits = findingsFor(report, "frontmatter.divergence"); + expect(hits).toHaveLength(1); + expect(hits[0]?.detail.keys_differing).toEqual(["status"]); + }); + + it("frontmatter.parse_error — stored raw no longer parses as YAML", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "title: A\n", + body: "b\n", + }); + // Unbalanced bracket → YAML parse failure. + withRaw((r) => + r + .prepare("update versions set frontmatter_raw = ? where path = 'a.md'") + .run("title: [oops\n"), + ); + const report = await kernel.verify(ROOT, { checks: ["frontmatter.parse_error"] }); + expect(findingsFor(report, "frontmatter.parse_error")).toHaveLength(1); + }); + + it("frontmatter.system_leak — a $-prefixed key persisted in storage", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "title: A\n", body: "b\n" }); + withRaw((r) => + r + .prepare("update versions set frontmatter_raw = ? where path = 'a.md'") + .run("title: A\n$version: v99\n"), + ); + const report = await kernel.verify(ROOT, { checks: ["frontmatter.system_leak"] }); + const hits = findingsFor(report, "frontmatter.system_leak"); + expect(hits).toHaveLength(1); + expect(hits[0]?.detail.in_raw).toBe(true); + }); +}); + +describe("verify: fts family", () => { + it("fts.missing — a version with no fts_docs row", async () => { + const v = await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "a\n" }); + const vId = decodeVersionId(v.version_id); + withRaw((r) => + r.prepare("insert into fts_docs(fts_docs, rowid, body) values('delete', ?, ?)").run(vId, "a"), + ); + const report = await kernel.verify(ROOT, { checks: ["fts.missing"] }); + expect(findingsFor(report, "fts.missing")).toHaveLength(1); + }); + + it("fts.orphan — an fts_docs row with no version", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "a\n" }); + withRaw((r) => r.prepare("insert into fts_docs(rowid, body) values (?, ?)").run(9999, "ghost")); + const report = await kernel.verify(ROOT, { checks: ["fts.orphan"] }); + expect(findingsFor(report, "fts.orphan")).toHaveLength(1); + }); + + it("fts family is skipped-with-note under a --repo filter", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "a\n" }); + const report = await kernel.verify(ROOT, { repo: "notes", checks: ["fts.missing"] }); + expect(report.checks_skipped).toContainEqual({ + check: "fts", + reason: "whole-store check; omit --repo to run", + }); + }); +}); + +describe("verify: chunks family", () => { + it("chunks.orphan — a chunk row referencing a nonexistent version", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "a\n" }); + // A chunk referencing a nonexistent version is an FK violation by design — + // inject with FK off (withRawUnsafe) to simulate the corruption. + withRawUnsafe((r) => + r + .prepare( + "insert into chunks(version_id, ix, text, text_hash, model, embedding) values (?,?,?,?,?,?)", + ) + .run(9999, 0, "t", "h", "m", Buffer.from(new Float32Array([1, 0]).buffer)), + ); + const report = await kernel.verify(ROOT, { checks: ["chunks.orphan"] }); + expect(findingsFor(report, "chunks.orphan")).toHaveLength(1); + }); + + it("chunks.backlog_orphan — a backlog row referencing a nonexistent version", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "a\n" }); + withRawUnsafe((r) => + r + .prepare( + "insert into embedding_backlog(version_id, attempts, last_error, next_retry_at) values (?,?,?,?)", + ) + .run(9999, 0, null, null), + ); + const report = await kernel.verify(ROOT, { checks: ["chunks.backlog_orphan"] }); + expect(findingsFor(report, "chunks.backlog_orphan")).toHaveLength(1); + }); + + it("chunks.mixed_dim — one version with vectors of differing dimension", async () => { + const v = await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "a\n" }); + const vId = decodeVersionId(v.version_id); + withRaw((r) => { + const ins = r.prepare( + "insert into chunks(version_id, ix, text, text_hash, model, embedding) values (?,?,?,?,?,?)", + ); + ins.run(vId, 0, "t0", "h0", "m", Buffer.from(new Float32Array([1, 0]).buffer)); // 8 bytes + ins.run(vId, 1, "t1", "h1", "m", Buffer.from(new Float32Array([1, 0, 0]).buffer)); // 12 bytes + }); + const report = await kernel.verify(ROOT, { checks: ["chunks.mixed_dim"] }); + expect(findingsFor(report, "chunks.mixed_dim")).toHaveLength(1); + }); + + it("chunks.unembedded is skipped-with-note when no embedder is configured", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { frontmatter_raw: "", body: "a\n" }); + const report = await kernel.verify(ROOT, { checks: ["chunks.unembedded"] }); + expect(report.checks_skipped).toContainEqual({ + check: "chunks.unembedded", + reason: "no embedder configured", + }); + expect(findingsFor(report, "chunks.unembedded")).toHaveLength(0); + }); +}); + +describe("verify: links family", () => { + it("links.set_mismatch — stored edges disagree with re-extraction", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "", + body: "see [B](b.md)\n", + }); + await kernel.docs.create(ROOT, "notes", "b.md", { frontmatter_raw: "", body: "hi\n" }); + // Corrupt the stored edge target so re-extraction won't match. + withRaw((r) => + r.prepare("update links set target_raw = 'wrong.md', target_norm = 'wrong.md'").run(), + ); + const report = await kernel.verify(ROOT, { checks: ["links.set_mismatch"] }); + expect(findingsFor(report, "links.set_mismatch").length).toBeGreaterThanOrEqual(1); + }); + + it("links.misresolved_dangling — a dangling edge that should have bound", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "", + body: "see [B](b.md)\n", + }); + await kernel.docs.create(ROOT, "notes", "b.md", { frontmatter_raw: "", body: "hi\n" }); + // Force the a→b edge to dangle even though b.md is live. + withRaw((r) => r.prepare("update links set target_id = null").run()); + const report = await kernel.verify(ROOT, { checks: ["links.misresolved_dangling"] }); + expect(findingsFor(report, "links.misresolved_dangling").length).toBeGreaterThanOrEqual(1); + }); + + it("is clean on a correct link graph", async () => { + await kernel.docs.create(ROOT, "notes", "a.md", { + frontmatter_raw: "", + body: "see [B](b.md)\n", + }); + await kernel.docs.create(ROOT, "notes", "b.md", { frontmatter_raw: "", body: "hi\n" }); + const report = await kernel.verify(ROOT, { checks: ["links"] }); + expect(findingsFor(report, "links.set_mismatch")).toHaveLength(0); + expect(findingsFor(report, "links.misresolved_dangling")).toHaveLength(0); + expect(findingsFor(report, "links.misresolved_bound")).toHaveLength(0); + }); +}); From 7da747c1264fc4b15373af05a210cdcb198fb925 Mon Sep 17 00:00:00 2001 From: Brendan Baldwin Date: Sat, 5 Sep 2026 08:04:24 -0600 Subject: [PATCH 5/8] verify WS4: CLI / MCP / REST surfaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exposes kernel.verify through all three doors: - CLI `mrplex verify` (local or remote via the client seam): --check (repeatable), --severity, --max-findings, --json, --ci (exit 1 on any finding at/above threshold). Repo optional — omit -r for a whole-store scan, required for the whole-store fts/chunks orphan checks. - MCP `verify` tool with a full outputSchema (25 tools now). - REST GET /verify (whole-store) and GET /repos/{repo}/verify (scoped); GET-only, no ETag (a scan is point-in-time, not cacheable). Adds verify to KernelClient (local + remote-mcp) and a shared renderVerifyReport used by both the MCP text channel and the CLI's non-JSON output. Surface tests across cli/mcp/rest; full suite green (1144 passed). Co-Authored-By: Claude Opus 4.7 --- src/cli/main.ts | 73 ++++++++++++++++++- src/client/kernel-client.ts | 3 + src/client/local.ts | 1 + src/client/remote-mcp.ts | 4 ++ src/mcp/render.ts | 44 +++++++++++- src/mcp/tools.ts | 139 ++++++++++++++++++++++++++++++++++-- src/rest/routes.ts | 43 ++++++++++- test/cli-verify.test.ts | 126 ++++++++++++++++++++++++++++++++ test/http-mcp.test.ts | 24 ++++++- test/http-rest.test.ts | 42 +++++++++++ 10 files changed, 488 insertions(+), 11 deletions(-) create mode 100644 test/cli-verify.test.ts diff --git a/src/cli/main.ts b/src/cli/main.ts index 3abbc83..2da709c 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -39,7 +39,7 @@ import { createKernel } from "../kernel/kernel.js"; import type { GraphSpec } from "../kernel/wire.js"; import { extractSystemProperties, split as splitFrontmatter } from "../markdown/frontmatter.js"; import { backfillContentHashes } from "../markdown/hash-backfill.js"; -import { renderDocGetManyText, renderGraphSummary } from "../mcp/render.js"; +import { renderDocGetManyText, renderGraphSummary, renderVerifyReport } from "../mcp/render.js"; import { startMcpStdio } from "../mcp/server.js"; import { startServer } from "../server/serve.js"; import { fileAuditSink } from "../shell/audit.js"; @@ -1927,6 +1927,77 @@ function buildProgram(): Command { })(); }); + // -------- verify -------- + // Read-only integrity scrub (docs/verify-plan.md). Top-level maintenance + // command like `hash`; works local or remote via the client seam. The repo + // is OPTIONAL here (unlike most commands): with a global -r it scans one + // repo, without it scans the whole store — required for the whole-store + // fts / chunks orphan checks. + program + .command("verify") + .description("read-only integrity scrub over the store (docs/verify-plan.md)") + .option( + "-c, --check ", + "check family (chain) or full code (links.set_mismatch); repeatable", + (val: string, prev: string[]) => [...prev, val], + [] as string[], + ) + .option("--severity ", "minimum finding severity to report: error | warn (default warn)") + .option("--max-findings ", "cap the emitted findings list (counts stay exact)") + .option("--ci", "exit non-zero if any finding at/above the severity threshold exists", false) + .action(function (this: Command) { + const localOpts = this.opts<{ + check: string[]; + severity?: string; + maxFindings?: string; + ci: boolean; + }>(); + const globals = this.optsWithGlobals(); + // Repo is optional: an explicit -r scopes to one repo; a bare glob is + // rejected (verify addresses one repo or the whole store, never a glob). + const repo = globals.repo; + if (repo !== undefined && isRepoPattern(repo)) { + const err = new Error( + `repo "${repo}" is a pattern — verify scans one repo (-r ) or the whole store (omit -r)`, + ); + (err as unknown as { code: string }).code = "cli_usage"; + reportError(err); + } + const severity = + localOpts.severity === "error" || localOpts.severity === "warn" + ? localOpts.severity + : undefined; + if (localOpts.severity !== undefined && severity === undefined) { + const err = new Error(`--severity must be "error" or "warn", got "${localOpts.severity}"`); + (err as unknown as { code: string }).code = "cli_usage"; + reportError(err); + } + const maxFindings = + localOpts.maxFindings !== undefined ? Number.parseInt(localOpts.maxFindings, 10) : undefined; + if (maxFindings !== undefined && (!Number.isSafeInteger(maxFindings) || maxFindings < 1)) { + const err = new Error("--max-findings must be a positive integer"); + (err as unknown as { code: string }).code = "cli_usage"; + reportError(err); + } + withClient(this, async (client, opts) => { + const report = await client.verify({ + ...(repo !== undefined && { repo }), + ...(localOpts.check.length > 0 && { checks: localOpts.check }), + ...(severity !== undefined && { min_severity: severity }), + ...(maxFindings !== undefined && { max_findings: maxFindings }), + }); + emit(report, opts, renderVerifyReport(report)); + // --ci: fail the process when the report isn't clean at the threshold. + // Findings counted are already filtered to min_severity, so any counted + // finding is at/above the bar. Exit 1 (validation family — the data + // failed validation), leaving 3/4 for the pre-flight forbidden/not-found. + if (localOpts.ci) { + const total = report.counts.by_severity.error + report.counts.by_severity.warn; + if (total > 0) process.exitCode = 1; + } + }).catch(reportError); + }); + return program; } diff --git a/src/client/kernel-client.ts b/src/client/kernel-client.ts index b796593..7013f66 100644 --- a/src/client/kernel-client.ts +++ b/src/client/kernel-client.ts @@ -30,6 +30,8 @@ import type { PathWarning, QueryHit, Repo, + VerifyReport, + VerifySpec, Version, } from "../kernel/wire.js"; import type { LinkConfigOverride } from "../links/link-config.js"; @@ -87,6 +89,7 @@ export type KernelClient = { }; query(spec: QuerySpec): Promise; graph(spec: GraphSpec): Promise; + verify(spec: VerifySpec): Promise; history: { since(input: { after_version: string; diff --git a/src/client/local.ts b/src/client/local.ts index 5f5405c..137f219 100644 --- a/src/client/local.ts +++ b/src/client/local.ts @@ -100,6 +100,7 @@ function buildClient( }, query: (spec) => kernel.query(ctx, spec), graph: (spec) => kernel.graph(ctx, spec), + verify: (spec) => kernel.verify(ctx, spec), history: { since: (input) => kernel.history.since(ctx, input), index: (input) => kernel.history.index(ctx, input), diff --git a/src/client/remote-mcp.ts b/src/client/remote-mcp.ts index a02b1fc..23c247b 100644 --- a/src/client/remote-mcp.ts +++ b/src/client/remote-mcp.ts @@ -34,6 +34,8 @@ import type { PathWarning, QueryHit, Repo, + VerifyReport, + VerifySpec, Version, } from "../kernel/wire.js"; import type { LinkConfigOverride } from "../links/link-config.js"; @@ -208,6 +210,8 @@ function buildRemoteClient(client: Client): KernelClient { }, graph: (spec: GraphSpec) => call("graph", spec as unknown as Record), + verify: (spec: VerifySpec) => + call("verify", spec as unknown as Record), history: { since: (input) => call("history_since", input as unknown as Record), diff --git a/src/mcp/render.ts b/src/mcp/render.ts index 1956f4f..70ea8fc 100644 --- a/src/mcp/render.ts +++ b/src/mcp/render.ts @@ -8,7 +8,7 @@ * humans; the MCP text rendering just has to be *readable*, not pretty. */ -import type { GraphResult, QueryHit, Repo, Version } from "../kernel/wire.js"; +import type { GraphResult, QueryHit, Repo, VerifyReport, Version } from "../kernel/wire.js"; export function renderJson(x: unknown): string { return JSON.stringify(x, null, 2); @@ -122,3 +122,45 @@ export function renderGraphSummary(result: GraphResult): string { return lines.join("\n"); } + +/** + * Text half for `verify` (docs/verify-plan.md §5). A per-finding block grouped + * by check code, a summary line, and any skipped-check notes. Shared by the MCP + * tool and the CLI's default (non-JSON) output. + */ +export function renderVerifyReport(report: VerifyReport): string { + const lines: string[] = []; + const { versions_scanned, documents_scanned, by_severity } = report.counts; + + if (report.findings.length === 0 && !report.truncated) { + lines.push("clean — no findings"); + } else { + // Group findings by check code, preserving first-seen order. + const byCheck = new Map(); + for (const f of report.findings) { + const list = byCheck.get(f.check); + if (list) list.push(f); + else byCheck.set(f.check, [f]); + } + for (const [check, findings] of byCheck) { + const sev = findings[0]?.severity ?? "error"; + lines.push(`${check} [${sev}] — ${findings.length}`); + for (const f of findings) { + const loc = f.path ?? f.version_id ?? f.document_id ?? ""; + const fix = f.suggested_fix ? ` (fix: ${f.suggested_fix})` : ""; + lines.push(` ${f.repo}${loc ? `/${loc}` : ""}${fix}`); + } + } + } + + const vLabel = `${versions_scanned} version${versions_scanned === 1 ? "" : "s"}`; + const dLabel = `${documents_scanned} document${documents_scanned === 1 ? "" : "s"}`; + const trunc = report.truncated ? " (findings truncated; counts exact)" : ""; + lines.push( + `scanned ${vLabel} across ${dLabel}; ${by_severity.error} error, ${by_severity.warn} warn${trunc}`, + ); + for (const s of report.checks_skipped) { + lines.push(`skipped ${s.check}: ${s.reason}`); + } + return lines.join("\n"); +} diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 41b08aa..7e53847 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -19,7 +19,7 @@ import { KernelError } from "../kernel/errors.js"; import type { Kernel } from "../kernel/kernel.js"; import type { PathConfigOverride } from "../kernel/path-config.js"; import type { QuerySpec } from "../kernel/query/query.js"; -import type { GraphSpec, Version } from "../kernel/wire.js"; +import type { GraphSpec, VerifyReport, VerifySpec, Version } from "../kernel/wire.js"; import type { LinkConfigOverride } from "../links/link-config.js"; import { appendSystemProperty, extractSystemProperties } from "../markdown/frontmatter.js"; import { QUERY_SYNTAX_DOC } from "./query-syntax.js"; @@ -29,6 +29,7 @@ import { renderJson, renderQueryHitList, renderRepoList, + renderVerifyReport, renderVersion, renderVersionList, } from "./render.js"; @@ -477,6 +478,67 @@ const GRAPH_LINK_SCHEMA: JsonSchemaProp = { required: ["source", "target", "field"], }; +const VERIFY_FINDING_SCHEMA: JsonSchemaProp = { + type: "object", + description: "One inconsistency found by verify (docs/verify-plan.md §3).", + properties: { + check: { type: "string", description: "Stable check code, e.g. `chain.prev_next_asymmetry`." }, + severity: { + type: "string", + enum: ["error", "warn"], + description: "`error` = a real inconsistency; `warn` = suspicious/legacy.", + }, + repo: { type: "string", description: "Repo slug." }, + document_id: { type: "string", description: "Opaque document id, when doc-scoped." }, + version_id: { type: "string", description: "Opaque version id, when version-scoped." }, + path: { type: "string", description: "Offending version's path, when known + readable." }, + detail: { type: "object", additionalProperties: true, description: "Check-specific payload." }, + suggested_fix: { type: "string", description: "Human hint at the remedy; never auto-run." }, + }, + required: ["check", "severity", "repo", "detail"], +}; + +const VERIFY_RESULT_SCHEMA: JsonSchema = { + type: "object", + description: "Structured integrity report (docs/verify-plan.md §3).", + properties: { + findings: { + type: "array", + items: VERIFY_FINDING_SCHEMA, + description: "Inconsistencies found.", + }, + counts: { + type: "object", + properties: { + versions_scanned: { type: "integer" }, + documents_scanned: { type: "integer" }, + by_check: { type: "object", additionalProperties: { type: "integer" } }, + by_severity: { + type: "object", + properties: { error: { type: "integer" }, warn: { type: "integer" } }, + required: ["error", "warn"], + }, + }, + required: ["versions_scanned", "documents_scanned", "by_check", "by_severity"], + }, + checks_skipped: { + type: "array", + description: + "Families/checks that did not run and why (e.g. fts on postgres, chunks.unembedded with no embedder).", + items: { + type: "object", + properties: { check: { type: "string" }, reason: { type: "string" } }, + required: ["check", "reason"], + }, + }, + truncated: { + type: "boolean", + description: "True if max_findings capped the list (counts stay exact).", + }, + }, + required: ["findings", "counts", "checks_skipped", "truncated"], +}; + const GRAPH_RESULT_SCHEMA: JsonSchema = { type: "object", description: "A graph neighborhood: documents and the links between them (docs/graph-plan.md).", @@ -715,7 +777,11 @@ export const TOOL_REGISTRY: ToolEntry[] = [ outputSchema: DOC_GET_MANY_RESULT_SCHEMA, handler: async (kernel, ctx, args) => { const raw = args.raw === true; - const result = await kernel.docs.get_many(ctx, argStr(args, "repo"), argStrArray(args, "paths")); + const result = await kernel.docs.get_many( + ctx, + argStr(args, "repo"), + argStrArray(args, "paths"), + ); const items = result.items.map((v) => withInjectedSystemProps(v, raw)); const structured = { items, errors: result.errors }; return { structured, text: renderDocGetManyText(items, result.errors) }; @@ -820,8 +886,7 @@ export const TOOL_REGISTRY: ToolEntry[] = [ }, { name: "docs_diff", - description: - `Unified diff between two versions of the document at (repo, path). ${EXACT_PATH_DOC} Both versions must belong to that document — otherwise version_not_in_document.`, + description: `Unified diff between two versions of the document at (repo, path). ${EXACT_PATH_DOC} Both versions must belong to that document — otherwise version_not_in_document.`, inputSchema: { type: "object", properties: { @@ -846,8 +911,7 @@ export const TOOL_REGISTRY: ToolEntry[] = [ }, { name: "docs_create", - description: - `Create a new document at (repo, path). ${EXACT_PATH_DOC} Fails with create_conflict if the path is occupied. Provide exactly one of \`frontmatter\` (JSON map) or \`frontmatter_raw\` (verbatim YAML).`, + description: `Create a new document at (repo, path). ${EXACT_PATH_DOC} Fails with create_conflict if the path is occupied. Provide exactly one of \`frontmatter\` (JSON map) or \`frontmatter_raw\` (verbatim YAML).`, inputSchema: { type: "object", properties: { @@ -1239,6 +1303,69 @@ export const TOOL_REGISTRY: ToolEntry[] = [ } }, }, + { + name: "verify", + description: + "Read-only integrity scrub — re-derives the FTS / links / hash indexes and checks the " + + "version chain, reporting inconsistencies as structured `findings`; never writes. Run it " + + "during maintenance or when you suspect corruption (e.g. after a batch of writes). Six check " + + "families — `chain` (version-chain structure), `hash` (content-hash fidelity), `frontmatter` " + + "(raw↔parsed round-trip), `fts` (SQLite index membership), `chunks` (embedding provenance), " + + "`links` (link index vs. re-extraction). `findings` are data, not errors — a clean store " + + "returns an empty list. Omit `repo` to scan the whole store (required for the whole-store " + + "`fts` / `chunks` orphan checks); pass `repo` to scan one. `checks` selects families " + + "(`chain`) or full codes (`links.set_mismatch`). `checks_skipped` names what didn't run and " + + "why. O(total versions) — it walks history, so it's heavier than ordinary reads.", + inputSchema: { + type: "object", + properties: { + repo: { + type: "string", + description: + "Repo slug to scan; omit for the whole store. Whole-store checks (fts, chunks orphan) " + + "run only in an all-repos scan.", + }, + checks: { + type: "array", + items: { type: "string" }, + description: + "Family prefixes (`chain`, `links`) or full codes (`hash.mismatch`); omit for all.", + }, + min_severity: { + type: "string", + enum: ["error", "warn"], + description: + "Drop findings below this severity from the list (counts stay full). Default warn.", + }, + max_findings: { + type: "integer", + minimum: 1, + description: + "Cap the emitted findings list (counts stay exact; sets `truncated`). Default 10000.", + }, + scope: { + type: "array", + description: "Read-visibility claims (ScopeClaim[]); the X-Mrplex-Scope header wins.", + items: { type: "object", additionalProperties: true }, + }, + }, + }, + outputSchema: VERIFY_RESULT_SCHEMA, + handler: async (kernel, ctx, args) => { + const order = argStrOpt(args, "min_severity"); + const spec: VerifySpec = { + repo: argStrOpt(args, "repo"), + checks: Array.isArray(args.checks) ? (args.checks as string[]) : undefined, + min_severity: order === "error" || order === "warn" ? order : undefined, + max_findings: argIntOpt(args, "max_findings"), + }; + const report = await kernel.verify(queryCtx(ctx, args), spec); + return { + structured: report as unknown as Record, + text: renderVerifyReport(report), + }; + }, + }, { name: "history_since", description: diff --git a/src/rest/routes.ts b/src/rest/routes.ts index de19547..e6dd58b 100644 --- a/src/rest/routes.ts +++ b/src/rest/routes.ts @@ -19,7 +19,7 @@ import { KernelError } from "../kernel/errors.js"; import type { Kernel } from "../kernel/kernel.js"; import type { PathConfigOverride } from "../kernel/path-config.js"; import type { QuerySpec } from "../kernel/query/query.js"; -import type { GraphSpec, QueryHit, Version } from "../kernel/wire.js"; +import type { GraphSpec, QueryHit, VerifySpec, Version } from "../kernel/wire.js"; import { appendSystemProperty, extractSystemProperties } from "../markdown/frontmatter.js"; import { type ContextForRequest, @@ -256,6 +256,14 @@ async function dispatch( return dispatchQuery(req, res, kernel, query, method, contextForRequest); } + // /verify — whole-store integrity scrub (GET). Repo-less: runs every family + // including the whole-store fts/chunks orphan checks (verify-plan §2.5). + if (segments[0] === "verify" && segments.length === 1) { + if (method !== "GET") return methodNotAllowed(res, method, ["GET"]); + const ctx = await contextForRequest(req); + return dispatchVerify(res, kernel, ctx, undefined, query); + } + // /repos and everything under it (config, docs, versions, history). if (segments[0] === "repos") { return dispatchRepos(req, res, kernel, storage, segments, query, method, contextForRequest); @@ -426,6 +434,13 @@ async function dispatchRepos( return dispatchGraph(req, res, kernel, ctx, repoSlug, query, method); } + // /repos/{repo}/verify — scoped integrity scrub (GET). Per-repo: the + // whole-store fts/chunks orphan checks are skipped-with-note (verify-plan §2.5). + if (segments[2] === "verify" && segments.length === 3) { + if (method !== "GET") return methodNotAllowed(res, method, ["GET"]); + return dispatchVerify(res, kernel, ctx, repoSlug, query); + } + notFound(res); } @@ -488,6 +503,32 @@ function graphSpecFromQueryString(query: URLSearchParams, repoSlug: string): Gra return spec; } +/** + * Verify read surface (docs/verify-plan.md §5). GET only — a scan is a + * point-in-time result, not a cacheable resource, so no ETag. `repoSlug` + * undefined = whole-store. Query params: `check` (repeatable/comma-joined), + * `severity` (min), `max_findings`. + */ +async function dispatchVerify( + res: ServerResponse, + kernel: Kernel, + ctx: CallContext, + repoSlug: string | undefined, + query: URLSearchParams, +): Promise { + const checks = collectListParam(query, "check"); + const severity = query.get("severity"); + const maxFindings = readOptionalIntQueryParam(query, "max_findings"); + const spec: VerifySpec = { + ...(repoSlug !== undefined && { repo: repoSlug }), + ...(checks.length > 0 && { checks }), + ...((severity === "error" || severity === "warn") && { min_severity: severity }), + ...(maxFindings !== undefined && { max_findings: maxFindings }), + }; + const report = await kernel.verify(ctx, spec); + writeJson(res, 200, report); +} + /** Flatten repeated and/or comma-joined query params into a string[]. */ function collectListParam(query: URLSearchParams, key: string): string[] { const flat: string[] = []; diff --git a/test/cli-verify.test.ts b/test/cli-verify.test.ts new file mode 100644 index 0000000..d4e0e41 --- /dev/null +++ b/test/cli-verify.test.ts @@ -0,0 +1,126 @@ +/** + * CLI `mrplex verify` end-to-end (docs/verify-plan.md §5). Spawns the CLI in + * local mode against a SQLite db, asserts the human/JSON output, the --ci exit + * code, and repo-scoped skip notes. Corruption is injected via a raw + * better-sqlite3 handle (the store's own write path won't produce these states). + */ + +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import Database from "better-sqlite3"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import type { VerifyReport } from "../src/kernel/wire.js"; + +const REPO_ROOT = join(fileURLToPath(new URL(".", import.meta.url)), ".."); +const CLI = join(REPO_ROOT, "src", "cli", "main.ts"); + +let workDir: string; +let dbFile: string; +let dbUrl: string; + +function run(args: string[], input?: string): { stdout: string; stderr: string; status: number } { + const res = spawnSync("node", ["--import", "tsx", CLI, "--database", dbUrl, ...args], { + cwd: REPO_ROOT, + encoding: "utf8", + ...(input !== undefined && { input }), + env: { + ...(process.env as Record), + XDG_CONFIG_HOME: workDir, + MRPLEX_EMBEDDER: "", + }, + }); + return { stdout: res.stdout, stderr: res.stderr, status: res.status ?? 1 }; +} + +function createDoc(path: string, body: string): void { + const out = run(["-r", "notes", "docs", "create", path, "--from-file", "-"], body); + if (out.status !== 0) throw new Error(`create ${path} failed: ${out.stderr}`); +} + +/** Mutate the db via a raw handle (corruption the write path won't produce). */ +function corrupt(sql: string): void { + const db = new Database(dbFile); + try { + db.exec(sql); + } finally { + db.close(); + } +} + +beforeEach(() => { + workDir = mkdtempSync(join(tmpdir(), "mrplex-cli-verify-")); + mkdirSync(workDir, { recursive: true }); + dbFile = join(workDir, "verify.db"); + dbUrl = `sqlite:${dbFile}`; + run(["repos", "create", "notes"]); +}); + +afterEach(() => { + rmSync(workDir, { recursive: true, force: true }); +}); + +describe("cli verify", () => { + it("reports clean on a healthy store and exits 0 under --ci", () => { + createDoc("a.md", "see [B](b.md)"); + createDoc("b.md", "hi"); + const out = run(["verify", "--ci"]); + expect(out.status).toBe(0); + expect(out.stdout).toContain("clean — no findings"); + expect(out.stdout).toContain("scanned 2 versions across 2 documents"); + }); + + it("emits a structured report with --json", () => { + createDoc("a.md", "body"); + const out = run(["--json", "verify"]); + expect(out.status).toBe(0); + const report = JSON.parse(out.stdout) as VerifyReport; + expect(report.findings).toEqual([]); + expect(report.counts.documents_scanned).toBe(1); + // No embedder in this env → chunks.unembedded skipped-and-noted. + expect(report.checks_skipped).toContainEqual({ + check: "chunks.unembedded", + reason: "no embedder configured", + }); + }); + + it("finds a hash mismatch and exits 1 under --ci", () => { + createDoc("a.md", "real body"); + corrupt("update versions set content_hash = 'deadbeef' where path = 'a.md'"); + const out = run(["verify", "--check", "hash", "--ci"]); + expect(out.status).toBe(1); + expect(out.stdout).toContain("hash.mismatch"); + expect(out.stdout).toContain("1 error"); + }); + + it("--json surfaces the finding detail", () => { + createDoc("a.md", "real body"); + corrupt("update versions set content_hash = 'deadbeef' where path = 'a.md'"); + const out = run(["--json", "verify", "--check", "hash.mismatch"]); + const report = JSON.parse(out.stdout) as VerifyReport; + expect(report.findings).toHaveLength(1); + expect(report.findings[0]?.check).toBe("hash.mismatch"); + expect(report.findings[0]?.detail.stored).toBe("deadbeef"); + }); + + it("skips the fts family with a note under a --repo filter", () => { + createDoc("a.md", "body"); + const out = run(["-r", "notes", "verify", "--check", "fts"]); + expect(out.status).toBe(0); + expect(out.stdout).toContain("skipped fts"); + }); + + it("rejects a glob repo", () => { + const out = run(["-r", "note*", "verify"]); + expect(out.status).not.toBe(0); + expect(out.stderr).toContain("pattern"); + }); + + it("rejects a bad --severity value", () => { + const out = run(["verify", "--severity", "loud"]); + expect(out.status).not.toBe(0); + expect(out.stderr).toContain("severity"); + }); +}); diff --git a/test/http-mcp.test.ts b/test/http-mcp.test.ts index 0de59d5..302bac1 100644 --- a/test/http-mcp.test.ts +++ b/test/http-mcp.test.ts @@ -37,9 +37,9 @@ afterEach(async () => { }); describe("MCP lifecycle + tools/list", () => { - it("lists 24 tools (no user/token tools after noauth; links_* + set_link_config + query_syntax + graph + history_since/index/list + docs_get_many)", async () => { + it("lists 25 tools (no user/token tools after noauth; links_* + set_link_config + query_syntax + graph + verify + history_since/index/list + docs_get_many)", async () => { const r = await client.listTools(); - expect(r.tools.length).toBe(24); + expect(r.tools.length).toBe(25); // Sample the important names. const names = new Set(r.tools.map((t) => t.name)); for (const name of [ @@ -54,6 +54,7 @@ describe("MCP lifecycle + tools/list", () => { "query", "query_syntax", "graph", + "verify", "history_since", "links_backfill", "links_stale", @@ -146,6 +147,25 @@ describe("MCP tools/call round-trip", () => { expect((fetched.structuredContent as { version_id: string }).version_id).toBe("v1"); }); + it("verify returns a structured report; clean store has no findings", async () => { + await client.callTool({ name: "repos_create", arguments: { repo: "notes" } }); + await client.callTool({ + name: "docs_create", + arguments: { repo: "notes", path: "a.md", body: "a\n", frontmatter: {} }, + }); + const r = await client.callTool({ name: "verify", arguments: {} }); + expect(r.isError).toBeFalsy(); + const report = r.structuredContent as { + findings: unknown[]; + counts: { documents_scanned: number }; + checks_skipped: { check: string }[]; + truncated: boolean; + }; + expect(report.findings).toEqual([]); + expect(report.counts.documents_scanned).toBe(1); + expect(report.truncated).toBe(false); + }); + it("docs_get_many returns items and per-path errors; isError is false on partial miss", async () => { await client.callTool({ name: "repos_create", arguments: { repo: "notes" } }); await client.callTool({ diff --git a/test/http-rest.test.ts b/test/http-rest.test.ts index 325c5e2..d407701 100644 --- a/test/http-rest.test.ts +++ b/test/http-rest.test.ts @@ -665,3 +665,45 @@ describe("REST error mapping", () => { expect([201, 400]).toContain(r.status); }); }); + +describe("REST verify", () => { + beforeEach(async () => { + await fetch(`${base}/repos`, { + method: "POST", + headers: { ...authHeaders(), "Content-Type": "application/json" }, + body: JSON.stringify({ slug: "notes" }), + }); + await fetch(`${base}/repos/notes/docs/a.md`, { + method: "PUT", + headers: { ...authHeaders(), "Content-Type": "text/markdown", "If-None-Match": "*" }, + body: "---\nstatus: draft\n---\nfoo\n", + }); + }); + + it("GET /verify returns a clean whole-store report", async () => { + const r = await fetch(`${base}/verify`, { headers: authHeaders() }); + expect(r.status).toBe(200); + const report = await readJson<{ + findings: unknown[]; + counts: { documents_scanned: number }; + truncated: boolean; + }>(r); + expect(report.findings).toEqual([]); + expect(report.counts.documents_scanned).toBe(1); + }); + + it("GET /repos/{repo}/verify scopes to one repo and skips whole-store fts", async () => { + const r = await fetch(`${base}/repos/notes/verify?check=fts`, { headers: authHeaders() }); + expect(r.status).toBe(200); + const report = await readJson<{ checks_skipped: { check: string; reason: string }[] }>(r); + expect(report.checks_skipped).toContainEqual({ + check: "fts", + reason: "whole-store check; omit --repo to run", + }); + }); + + it("verify is GET-only", async () => { + const r = await fetch(`${base}/verify`, { method: "POST", headers: authHeaders() }); + expect(r.status).toBe(405); + }); +}); From a1f627d8fd7b84f9d2750fe7999386de45456eb9 Mon Sep 17 00:00:00 2001 From: Brendan Baldwin Date: Sat, 5 Sep 2026 08:11:15 -0600 Subject: [PATCH 6/8] verify WS5: docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README gains a `mrplex verify` blurb in "Versions, always" (the chain is the guarantee; verify confirms it) and a Verify row in the concepts table. Flips the design.md §11 verify bullet to Shipped with the as-built shape, noting where implementation refined the sketch (nonexistent-version orphans, whole-store fts/chunks checks, embedder-gated unembedded). Co-Authored-By: Claude Opus 4.7 --- README.md | 10 ++++++++++ docs/archive/design.md | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e2d5a6e..43c36b2 100644 --- a/README.md +++ b/README.md @@ -158,6 +158,15 @@ Use `--render mermaid` instead to get a diagram you can paste into any Markdown Every write appends a new version rather than overwriting — so history is never lost. That's also what makes `--prev` work: it's *optimistic concurrency*. You tell a write which version you started from; if someone else has written in the meantime, yours is rejected and you're handed the current version to reconcile against, instead of silently clobbering their change. Deleting a document moves it aside to `:deleted/…` rather than erasing it, so a delete can be undone with an ordinary write. When you need the record, `docs history` lists a document's versions and `docs diff --from v1 --to v2` shows what changed. +Because the chain *is* the guarantee, `mrplex verify` is the way to confirm it holds. It's a read-only scrub — mrplex's `git fsck` — that re-derives the search, link, and hash indexes and checks the version chain, reporting any inconsistency as a structured finding without ever writing. Reach for it during maintenance, after a bulk import, or whenever you suspect something drifted: + +```sh +$ mrplex verify # scan the whole store +clean — no findings +scanned 128 versions across 96 documents; 0 error, 0 warn +$ mrplex verify --check links --ci # one family; exit non-zero on any finding (CI gate) +``` + ## Connect an agent The CLI, the MCP server, and the REST API are three doors into the same store. For a database only you touch, the quickest way to give an agent access is MCP over local stdio — for example, in a client's MCP configuration: @@ -205,6 +214,7 @@ mrplex is built from a few concepts: | **Query** | A CEL filter over frontmatter and `$path` / `$body` / `$updated_at`, combined with full-text and semantic search | | **Link graph** | Links from Markdown syntax, wikilinks, and frontmatter paths, tracked by identity so renames don't break them | | **Graph walk** | A breadth-first tour of that link graph — *how* documents connect, not just which ones match | +| **Verify** | A read-only integrity scrub that re-derives every index and checks the version chain, reporting drift as findings | | **Surfaces** | The CLI, an MCP server, and a REST API, all over the same store | Underneath, mrplex is two layers. The **kernel** is the store itself — documents, versions, queries, the graph — and it is full-trust: it has no notion of users, so whoever holds the database file holds everything. Around it is an optional **access-and-identity shell** that adds keys, OIDC, per-path permissions, and an audit log when you need them. The store runs on SQLite by default, or Postgres with pgvector when you want it; the same test suite runs against both, so behavior matches. diff --git a/docs/archive/design.md b/docs/archive/design.md index b948852..6c77f8e 100644 --- a/docs/archive/design.md +++ b/docs/archive/design.md @@ -982,7 +982,7 @@ Remaining `[OPEN]` markers throughout the doc are narrower questions (query cach - **Per-repo frontmatter schema** `[OPEN]`. Declared shape stored per repo (types, required fields, enums, string patterns), validated at write time. Turns the "YAML soup" reality into typed records without giving up prose, and is load-bearing for three downstream features: MCP tools shaped like the domain (`notes.create_task(title, due, tags)` derived from the schema, not just generic `docs.put`), LSP completion (below), and aggregations (above) that can trust field types. Open: schema language (JSON Schema for reach, or a small mrplex-native dialect that maps directly to CEL types?); enforcement mode (strict / warn / advisory, per repo *and* per field?); evolution when a required field is added to a repo with existing docs (reject writes, allow with defaults, or trigger a bulk-update pass via the future bulk-update op?); scope (schema lives in repo config, or as a system-namespace doc so it versions like anything else?). - **Computed frontmatter as `$`-intrinsics.** `$word_count`, `$reading_time`, `$outgoing_links` (once §11.2 lands), `$last_body_edit` (most recent version whose body hash actually differed — distinguishes real edits from frontmatter-only touches). All derivable from data mrplex already stores; queryable via the same CEL surface as `$path`/`$updated_at`. Read-only — writes rejected with `computed_field`. Kills the "stash derived value in frontmatter and forget to update it" antipattern that otherwise grows in every corpus. - **Change feed — webhooks and SSE.** Every write (create/update/move/delete) emits `{ repo, path, document_id, version_id, prev_version_id, author, kind }`. Two subscribers: outbound webhooks (per-repo config, HMAC-signed, at-least-once with a small retry queue) and a `GET /repos/{repo}/events` SSE stream filtered by the caller's read scope claim (§8.2) — same scope filter as `query`, so no subscriber ever sees an event it couldn't have read. MCP notifications already exist; this is the plain-HTTP twin so a static-site builder, Meilisearch indexer, or an agent doesn't have to poll. Ordering guarantee is per-document, not global (matches the write model). Resume via `?from_version_id=…`. -- **`mrplex verify`.** Integrity scrub over the version chain: walk each document oldest-to-newest, recompute body/frontmatter hashes, confirm `frontmatter_raw` ↔ `frontmatter` round-trips byte-exact (§3.2), check `prev_id`/`next_id` symmetry, verify FTS/chunk/link derived tables against their source versions, report orphans. No writes. CLI + kernel op + optional CI mode that exits non-zero on any inconsistency. Cheap insurance for an append-only store where the chain *is* the guarantee. +- **`mrplex verify`. Shipped** (`docs/verify-plan.md`). Read-only integrity scrub: `kernel.verify` re-derives the FTS / links / hash indexes and checks the version chain, reporting inconsistencies as structured findings (never throwing on a finding). Six families — `chain` (prev/next symmetry, one-current, one-live-per-path, cycles, repo-id drift), `hash` (recompute `contentHash` vs. stored), `frontmatter` (raw↔parsed round-trip, `$`-leak), `fts` (SQLite `fts_docs` rowid↔version bijection — skipped-with-note on Postgres, whose `fts_tsv` is a generated column), `chunks` (embedding provenance: nonexistent-version orphans, mixed-dim), `links` (re-`extractEdges` vs. stored rows + resolution correctness). No writes — findings carry a `suggested_fix` pointing at the relevant backfill. Surfaces: CLI `mrplex verify` (`--check`/`--severity`/`--max-findings`/`--json`/`--ci`), MCP `verify` tool, REST `GET /verify` and `GET /repos/{repo}/verify`. O(total versions) — it walks history, unlike everything else on the read path. Where implementation refined the sketch: "orphan" means a *nonexistent* version (the embed worker deliberately keeps chunks on superseded versions), the `fts` and `chunks`-orphan checks are whole-store (not repo-partitioned, skipped-with-note under `--repo`), and `chunks.unembedded` runs only when an embedder is configured. - **Retention: rollup of autosave storms.** Per-repo policy that collapses contiguous same-author versions within N seconds into a single displayed step in `docs.history` — underlying versions retained, a `rollup_of` link identifies the group. Complements the embedding damper (§5.3): history stays readable when a WebDAV/Obsidian client sprays 40 saves per minute during an edit session. This is a view-time policy over an untouched underlying chain, not a delete — `docs.get --version` still resolves every intermediate step and `docs.diff` still spans them. - **`mrplex-lsp`.** LSP over stdio for Markdown+YAML: frontmatter completion and diagnostics from the per-repo schema (above), go-to-definition and hover on links once §11.2 lands, code-actions surfacing `mrplex links repair`. Editors get the mrplex surface (VS Code, Neovim, Helix, Zed) through their existing LSP clients, without a per-editor plugin. Runs against `--database` or `--server` the same way the CLI does. - **`mrplex export`.** Materialize a repo's live current versions as a filesystem tree — `path/` structure, `frontmatter_raw` + body written verbatim. Flags for `:deleted/` inclusion, historical version fanout (`--as-of` once PIT reads land), and index files. Hands the corpus to Hugo/Zola/11ty/rsync in one command; the byte-exact round-trip is what makes it trustworthy where a naive dump wouldn't be. From f9b2750132b3895f9174468eb1628ae9776b2964 Mon Sep 17 00:00:00 2001 From: Brendan Baldwin Date: Tue, 8 Sep 2026 15:55:34 -0600 Subject: [PATCH 7/8] ignore .clause --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index f9f5813..c5d4db9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ .claude/settings.local.json +.claude/* .DS_Store .mrplex node_modules/ From 76216425f8529d94118e93dad42c1e8cf1dd21af Mon Sep 17 00:00:00 2001 From: Brendan Baldwin Date: Tue, 8 Sep 2026 19:49:24 -0600 Subject: [PATCH 8/8] docs_put: create-if-absent when no prev supplied Make docs_put an upsert across the MCP and CLI surfaces: with no prev_version_id (arg or embedded $version), create the document at the path instead of erroring. An occupied path still yields create_conflict rather than a blind overwrite, preserving optimistic concurrency. REST already handled create-if-absent via If-None-Match: *. Co-Authored-By: Claude Opus 4.7 --- src/cli/main.ts | 20 +++++++++++------ src/mcp/tools.ts | 18 ++++++++++++--- test/cli-m1.test.ts | 25 +++++++++++++++++++++ test/http-mcp.test.ts | 51 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 104 insertions(+), 10 deletions(-) diff --git a/src/cli/main.ts b/src/cli/main.ts index 2da709c..a2a860b 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1306,10 +1306,12 @@ function buildProgram(): Command { docs .command("put ") - .description("update or move a document — path may differ from prev's path") + .description( + "upsert or move a document — with --prev: update/move (path may differ from prev's); with no prev: create at (create_conflict if occupied, so re-read and pass --prev to update)", + ) .option( "--prev ", - "current version id (from get / history) — optional if the input's frontmatter carries `$version: `", + "current version id (from get / history) — optional if the input's frontmatter carries `$version: `; omit entirely to create a new document", ) .option("--from-file ", "read the markdown from a file or '-' for stdin") .action(function (this: Command, path: string) { @@ -1334,12 +1336,16 @@ function buildProgram(): Command { } } const prev = localOpts.prev ?? embeddedVersion; + // No prev → create-if-absent. Delegating to create means an occupied + // path raises create_conflict rather than clobbering — the caller must + // re-read and pass --prev (or `$version`) to update the existing doc. if (prev === undefined) { - const err = new Error( - "no prev version — pass --prev, or provide `$version: ` in the input frontmatter", - ); - (err as unknown as { code: string }).code = "cli_usage"; - throw err; + const created = await client.docs.create(repo, path, { + frontmatter_raw: input.frontmatter_raw ?? "", + body: input.body ?? "", + }); + emitVersionWrite(created, opts); + return; } const result = await client.docs.put(repo, prev, path, input); emitVersionWrite(result, opts); diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 7e53847..b9e0e0c 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -945,7 +945,7 @@ export const TOOL_REGISTRY: ToolEntry[] = [ { name: "docs_put", description: - "Update or move a document (optimistic concurrency). `path` may differ from prev's path (= move). Exactly one of `frontmatter` | `frontmatter_raw` if changing frontmatter; both may be omitted to keep prev's. `prev_version_id` may be omitted if `frontmatter_raw` embeds `$version: ` from a prior `docs_get`. Conflicts: stale_prev (someone else wrote first — re-read and retry), path_taken (move onto an occupied path).", + "Upsert or move a document (optimistic concurrency). With a prev: update/move the existing document — `path` may differ from prev's path (= move). Exactly one of `frontmatter` | `frontmatter_raw` if changing frontmatter; both may be omitted to keep prev's. `prev_version_id` may be omitted if `frontmatter_raw` embeds `$version: ` from a prior `docs_get`. With NO prev: create a new document at `path` — but if a document already exists there you get create_conflict (carrying the current version id), so re-read and pass its `prev_version_id` to update it (this guards against blind overwrites). Conflicts: stale_prev (someone else wrote first — re-read and retry), path_taken (move onto an occupied path), create_conflict (no-prev create onto an occupied path).", inputSchema: { type: "object", properties: { @@ -991,10 +991,22 @@ export const TOOL_REGISTRY: ToolEntry[] = [ } } const prev = argStrOpt(args, "prev_version_id") ?? embeddedVersion; + // No prev → create-if-absent. Delegating to `create` means an occupied + // path raises create_conflict (carrying the current version id) rather + // than silently overwriting — the caller must re-read and pass the prev. if (prev === undefined) { - throw new Error( - "prev_version_id is required (either as an argument or as `$version` in frontmatter_raw)", + const createInput: { frontmatter?: never; frontmatter_raw?: string; body: string } = { + body: input.body ?? "", + }; + if (input.frontmatter !== undefined) createInput.frontmatter = input.frontmatter as never; + else createInput.frontmatter_raw = input.frontmatter_raw ?? ""; + const created = await kernel.docs.create( + writeCtx(ctx, args), + argStr(args, "repo"), + argStr(args, "path"), + createInput, ); + return { structured: created, text: renderVersion(created) }; } const v = await kernel.docs.put( diff --git a/test/cli-m1.test.ts b/test/cli-m1.test.ts index 9ac1b23..5e48e2a 100644 --- a/test/cli-m1.test.ts +++ b/test/cli-m1.test.ts @@ -103,6 +103,31 @@ describe("cli — end-to-end write flow", () => { expect(history[0]?.version_id).toBe(v5.version_id); }); + it("put with no --prev creates a new document at an empty path", () => { + expect(run(["repos", "create", "notes"]).status).toBe(0); + const put = run(["--json", "docs", "put", "fresh.md", "--from-file", "-"], { + stdin: "---\ntitle: Fresh\n---\nborn via put\n", + }); + expect(put.status).toBe(0); + const v = JSON.parse(put.stdout) as { version_id: string; body: string; path: string }; + expect(v.version_id).toBe("v1"); + expect(v.body).toBe("born via put\n"); + expect(v.path).toBe("fresh.md"); + }); + + it("put with no --prev on an occupied path → create_conflict (exit 2)", () => { + expect(run(["repos", "create", "notes"]).status).toBe(0); + expect( + run(["docs", "create", "a.md", "--from-file", "-"], { stdin: "---\n---\noriginal\n" }).status, + ).toBe(0); + const put = run(["docs", "put", "a.md", "--from-file", "-"], { stdin: "---\n---\nclobber\n" }); + expect(put.status).toBe(2); + expect(put.stderr).toContain("create_conflict"); + // Original must survive. + const got = run(["--json", "docs", "get", "a.md"]); + expect((JSON.parse(got.stdout) as { body: string }).body).toBe("original\n"); + }); + it("stale_prev exits 2 with current attached", () => { expect(run(["repos", "create", "notes"]).status).toBe(0); const created = run(["--json", "docs", "create", "x.md", "--from-file", "-"], { diff --git a/test/http-mcp.test.ts b/test/http-mcp.test.ts index 302bac1..dc4d32b 100644 --- a/test/http-mcp.test.ts +++ b/test/http-mcp.test.ts @@ -268,6 +268,57 @@ describe("MCP $version round-trip", () => { const rRaw = (rawResp.structuredContent as { frontmatter_raw: string }).frontmatter_raw; expect(rRaw).not.toContain("$version"); }); + + it("docs_put with no prev creates a new document at an empty path", async () => { + await client.callTool({ name: "repos_create", arguments: { repo: "notes" } }); + const put = await client.callTool({ + name: "docs_put", + arguments: { + repo: "notes", + path: "fresh.md", + body: "brand new", + frontmatter: { status: "draft" }, + }, + }); + expect(put.isError).toBeFalsy(); + const v = put.structuredContent as { version_id: string; prev_version_id: string | null }; + expect(v.version_id).toBe("v1"); + expect(v.prev_version_id).toBeNull(); + const got = await client.callTool({ + name: "docs_get", + arguments: { repo: "notes", path: "fresh.md" }, + }); + expect((got.structuredContent as { body: string }).body).toBe("brand new"); + }); + + it("docs_put with no prev on an occupied path → create_conflict (no blind overwrite)", async () => { + await client.callTool({ name: "repos_create", arguments: { repo: "notes" } }); + await client.callTool({ + name: "docs_create", + arguments: { + repo: "notes", + path: "a.md", + body: "original", + frontmatter: { status: "draft" }, + }, + }); + const put = await client.callTool({ + name: "docs_put", + arguments: { repo: "notes", path: "a.md", body: "clobber" }, + }); + expect(put.isError).toBe(true); + const content = (put.content as { type: string; text: string }[])[0]; + expect(content).toBeDefined(); + const parsed = JSON.parse((content as { text: string }).text); + expect(parsed.code).toBe("create_conflict"); + expect(parsed.data.current_version_id).toBe("v1"); + // The original body must be untouched. + const got = await client.callTool({ + name: "docs_get", + arguments: { repo: "notes", path: "a.md" }, + }); + expect((got.structuredContent as { body: string }).body).toBe("original"); + }); }); describe("MCP in-band errors", () => {