fix: harden memory writes and CLI reliability

.github/workflows/ci.yml

@@ -11,7 +11,10 @@ permissions:
 
 jobs:
   test:
-    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
 
     steps:
       - name: Checkout
@@ -26,20 +29,14 @@ jobs:
       - name: Install
         run: npm ci
 
-      - name: Source tests
-        run: npm run test:src:readonly
-
-      - name: Markdown links
-        run: npm run check:links
+      - name: Source, compiled, links and installed-package tests
+        run: npm test
 
       - name: Memory lint
-        run: npm run lint:memory
+        run: node scripts/dejavu-lint-memory.mjs && node scripts/dejavu-lint-memory.mjs --memory-root starter-kit/memory && node scripts/dejavu-lint-memory.mjs --memory-root examples/protocol-project/memory && node scripts/dejavu-lint-memory.mjs --memory-root docs/examples/settings-project/memory
+
+      - name: Project readiness
+        run: node scripts/deja-vu.mjs doctor --json
 
       - name: Feedback report
         run: npm run report:feedback
-
-      - name: Build
-        run: npm run build
-
-      - name: Pack verification
-        run: npm run test:pack

CHANGELOG.md

@@ -2,6 +2,17 @@
 
 All notable changes to this project will be documented in this file.
 
+## Unreleased
+
+- Use the same validated rules block for readiness and scope checks; reject ambiguous scopes and omit raw JSONL parse messages from diagnostics.
+
+- Prepare engine writes before mutation; compensate failed writes, expose recovery errors, preserve update metadata, reject duplicate IDs, and serialize mutations.
+- Preflight initialization, back up overwritten/appended files, stage replacements, and report rollback failures as JSON.
+- Share schema and route diagnostics across CLI tools; reject invalid input, inactive routes, missing successors, cycles, and scope mismatches.
+- Recognize actual recall rules instead of filename mentions; validate all three setup files in project scans.
+- Add fault-injection regression tests, executable settings-demo fixtures, installed-package smoke checks, and full-suite CI coverage.
+- Clarify lexical recall limits, cue-first manual fallback, and non-transactional recovery boundaries.
+
 ## [0.7.0] - 2026-08-09
 
 - Rebuilt the README around a one-command start, concrete before/after proof, agent compatibility, safety boundaries, and a shareable product visual.

README.md

@@ -143,7 +143,9 @@ npx @focaxisdev/deja-vu doctor --json
 npx @focaxisdev/deja-vu explain
 ```
 
-`init` creates missing files only. It never replaces existing files unless `--force` is explicitly passed. When `AGENTS.md` already exists without Deja Vu rules, the command reports that setup is incomplete; `--merge-agents` appends a marked block without deleting existing rules.
+`init` creates missing files only. It never replaces existing files unless `--force` is explicitly passed. When `AGENTS.md` already exists without Deja Vu rules, the command reports that setup is incomplete; `--merge-agents` appends a marked block to unrelated rules and leaves conflicting or partial blocks for manual review.
+
+Initialization backs up files before overwriting or appending and reports the backup directory. Check `ready` and run `doctor` before use; see [reliability and recovery](docs/reliability.md) for restoration instructions and transaction limits.
 
 Focused tools are also available:
 

docs/demo-walkthrough.md

@@ -1,6 +1,11 @@
 # Demo Walkthrough
 
-This walkthrough shows the behavior Deja Vu is designed to create.
+This walkthrough uses fictional, sanitized data. The checked-in
+[summary](examples/settings-project/memory/summary.md),
+[cue index](examples/settings-project/memory/impressions.jsonl),
+[decision](examples/settings-project/memory/decisions/settings-sync.md), and
+[follow-up](examples/settings-project/memory/open-loops/settings-migration.md)
+make the example reproducible.
 
 ## Before Deja Vu
 
@@ -44,17 +49,34 @@ User: Continue the settings refactor.
 Agent: I will follow AGENTS.md and scan memory/impressions.jsonl before planning.
 ```
 
-The cue scan finds a strong match for `settings` and `sync`.
+That exact request does **not** produce a strong scripted match: it returns
+`level: "none"` and a top score of `0.2437`. The lexical scanner does not infer
+the word "sync" from "refactor". No detail should be loaded on that result.
 
-The agent reads only:
+When the task supplies the explicit cues `settings sync`, the scanner returns
+`strong`, score `0.975`, with the settings decision first. The second route is
+the migration follow-up (score `0.475`).
+
+Run both from the repository root:
+
+```bash
+node scripts/dejavu-scan-memory.mjs --memory-root docs/examples/settings-project/memory "Continue the settings refactor."
+node scripts/dejavu-scan-memory.mjs --memory-root docs/examples/settings-project/memory "settings sync"
+```
+
+These exact outcomes are regression-tested. This demonstrates cue-based
+retrieval, not measured improvements in agent task success.
+
+For the explicit-cue task, the agent can read the decision and its relevant
+migration link, within the one-to-three-record budget:
 
 - `memory/decisions/settings-sync.md`
 - `memory/open-loops/settings-migration.md`
 
 Then it plans with the right constraints:
 
 - do not move synced preferences to localStorage
-- preserve the existing IME workaround
+- preserve the IME composition guard documented in the settings decision
 - continue the legacy setting-name migration
 
 ## Post-Task Writeback

docs/engine/semantic-engine.md

@@ -48,6 +48,31 @@ await engine.updateMemory(id, input);
 await engine.deleteMemory(id);
 ```
 
+## Write failure contract
+
+Adds and updates prepare summaries, chunks, and all embeddings before changing
+storage. Updates retain importance, tags, and creation time unless explicitly
+changed where supported. Adding an existing ID is rejected; use `updateMemory`.
+Mutations are serialized within one engine instance.
+
+If a storage or vector write fails, the engine attempts compensating cleanup
+and restoration. `MemoryWriteError.memoryId` identifies the affected memory,
+including auto-generated IDs on failed adds. `recoveryErrors` is empty when compensation
+succeeds. Otherwise, repair the adapter using `recoverySnapshot`; it may contain
+private raw content and must not be logged or included in telemetry.
+
+This is **not a durable transaction**. Process termination, persistent adapter
+outages, concurrent readers, multiple engine instances, and direct adapter
+writes are not isolated by this queue. Deletion is serialized but does not
+provide compensating rollback. Production hosts must supply transactional
+persistence/coordination where those guarantees are required.
+
+Storage adapters may implement optional `getMemorySnapshot(id)` to return a
+complete detached snapshot, including raw creation/update timestamps. The
+legacy getter fallback reconstructs those timestamps from chunk or access
+metadata and cannot guarantee their exact preservation. The built-in in-memory
+adapter implements complete snapshots; it is still not persistent storage.
+
 ## Optional npm install
 
 ```bash

docs/examples/settings-project/AGENTS.md

@@ -0,0 +1,53 @@
+# Deja Vu Project Memory Rules
+
+## Memory Identity
+
+- Protocol: Deja Vu Protocol v0.4
+- Scope: `project:demo`
+- Memory root: `memory/`
+
+## Recall Before Substantial Work
+
+Before substantial planning, coding, refactoring, debugging, or architecture discussion:
+
+1. Inspect `memory/impressions.jsonl` for familiar cues.
+2. If there is no familiarity, do not load memory by default.
+3. If familiarity is weak, read `memory/summary.md`.
+4. If familiarity is strong, read only the 1-3 linked detailed records needed for the task.
+5. Do not load the whole memory tree unless the user explicitly asks.
+
+Default budget:
+
+- impression scan: always allowed
+- summary: at most one file
+- detailed records: one to three files
+- full memory tree: forbidden unless explicitly requested
+
+## Durable Writeback Only
+
+After meaningful work, write back only memory that should change future agent behavior:
+
+- accepted decisions
+- architecture intent
+- stable preferences
+- unresolved follow-ups
+- milestone summaries
+- recall feedback that should improve future cue quality
+
+Route durable writeback:
+
+- project-level truth -> `memory/summary.md` plus `memory/impressions.jsonl`
+- accepted decision -> `memory/decisions/` plus `memory/impressions.jsonl`
+- unresolved follow-up -> `memory/open-loops/` plus `memory/impressions.jsonl`
+- recall quality signal -> `memory/recall-feedback.jsonl`
+
+## Never Store
+
+- secrets, tokens, credentials, or private keys
+- full chat transcripts
+- customer or user PII
+- low-signal chatter
+- disposable exploration noise
+- raw debug logs or build output
+
+Memory should feel like recognition, not replay.

docs/examples/settings-project/memory/decisions/settings-sync.md

@@ -0,0 +1,25 @@
+---
+id: decision-settings-sync
+title: Settings sync stays server-backed
+status: active
+scope: project:demo
+updated: 2026-05-16
+---
+
+# Settings sync
+
+## Decision
+
+Keep synced preferences on the server; do not move them to localStorage.
+Preserve the input-method editor (IME) composition guard: do not submit
+unfinished text while composition is active.
+
+## Rationale
+
+The fictional application shares settings across devices. Premature submission
+during IME composition can persist incomplete text.
+
+## Consequences
+
+The refactor must retain server-backed persistence and composition handling.
+Track old setting names in the [migration follow-up](../open-loops/settings-migration.md).

docs/examples/settings-project/memory/impressions.jsonl

@@ -0,0 +1,2 @@
+{"schema_version":1,"id":"decision-settings-sync","scope":"project:demo","title":"Settings sync stays server-backed","keywords":["settings","sync","localstorage","preferences"],"record_path":"memory/decisions/settings-sync.md","updated":"2026-05-16","weight":0.9,"status":"active"}
+{"schema_version":1,"id":"open-loop-settings-migration","scope":"project:demo","title":"Migrate old setting names","keywords":["settings","migration","legacy","names"],"record_path":"memory/open-loops/settings-migration.md","updated":"2026-05-16","weight":0.8,"status":"active"}

docs/examples/settings-project/memory/open-loops/settings-migration.md

@@ -0,0 +1,26 @@
+---
+id: open-loop-settings-migration
+title: Migrate old setting names
+status: active
+scope: project:demo
+updated: 2026-05-16
+---
+
+# Settings migration
+
+## Owner
+
+Demo maintainer.
+
+## Opened
+
+2026-05-16.
+
+## Next trigger
+
+Before removing legacy setting names, implement and test their migration.
+
+## Why it matters
+
+Existing preferences must survive the refactor without losing server sync.
+The [settings decision](../decisions/settings-sync.md) remains authoritative.

docs/examples/settings-project/memory/summary.md

@@ -0,0 +1,16 @@
+---
+id: settings-summary
+title: Settings demo
+status: active
+scope: project:demo
+updated: 2026-05-16
+---
+
+# Settings demo
+
+This is fictional, sanitized test data, not a claim about a real application.
+Preferences remain server-backed. The settings refactor must preserve the IME
+composition guard. Legacy setting-name migration is still open.
+
+See the [decision](decisions/settings-sync.md) and
+[migration follow-up](open-loops/settings-migration.md).

docs/protocol.md

@@ -101,7 +101,7 @@ Before substantial planning, coding, or answering:
 3. If the scan is `none`, do not load project memory by default.
 4. If the scan is `weak`, read `memory/summary.md`.
 5. If the scan is `strong`, open only the detailed records needed for the current task.
-6. If no script is available, fall back to `memory/summary.md` and only then to `memory/index.md` when present.
+6. If no script is available, inspect `memory/impressions.jsonl` directly for familiar cues and apply the same none/weak/strong loading rules. Do not unconditionally load the summary or index.
 7. Track the recall budget used for the task: impression scan, summary count, detailed records, and why anything was loaded.
 
 Default recall budget:

docs/reliability.md

@@ -0,0 +1,49 @@
+# Reliability and recovery
+
+## Initialization
+
+`deja-vu init` preflights all target files before writing. It stages replacements
+beside their destinations and restores completed writes if a later operation
+throws. Existing files overwritten by `--force` or appended by `--merge-agents`
+are backed up under `.deja-vu-backups/init-*/`; JSON and text output report the
+directory. `manifest.json` maps original paths to byte-for-byte backups.
+
+Backups may contain private rules or memory. Their directory contains a
+`*` ignore rule, but Git ignore is not encryption or an access-control boundary.
+Keep backups local, inspect them before sharing, and manage retention yourself.
+
+`--dry-run` creates neither files nor backups; readiness is a projection, not
+a full validation of the proposed final tree. A successful command can return
+`ok: true, ready: false` when existing rules or memory need manual review.
+Run `deja-vu doctor --json` after initialization.
+
+Known shipped rules and version-1 generated blocks are recognized structurally.
+Mentions of memory filenames alone are not proof of readiness. Partial blocks,
+unknown versions, differing scopes, and conflicting existing rules require
+manual review. Recognition is not natural-language contradiction detection.
+
+Readiness and scope checks use the same validated block. Scope mentions outside
+a marked block do not override its identity; duplicate scopes inside it require
+manual review. JSONL parse failures report a fixed code, path and line number,
+without echoing the malformed record in the diagnostic.
+
+## Recovering interrupted initialization
+
+Caught write failures return `ok: false`, the backup directory when available,
+and any `recovery_errors` in JSON mode. Successful compensation restores original
+bytes. The operation does not promise an all-files transaction across a process
+crash or concurrent external edits.
+
+After a crash or recovery error, stop writers, inspect the backup manifest,
+compare each current file against its backup, and restore only the intended
+targets. Do not blindly overwrite newer user edits. Run doctor before resuming.
+
+## Validation boundaries
+
+Doctor and lint reject broken routes, cross-project targets, active cues pointing
+to inactive records, missing successors, and supersession cycles. Legacy Markdown
+without frontmatter still produces a warning: add metadata to enable complete
+lifecycle validation. These checks do not prove the truth of a memory's content.
+
+See [scripted recall](scripted-recall.md) for scan exit codes and cue-only mode,
+and the [engine contract](engine/semantic-engine.md) for adapter failure recovery.

docs/scripted-recall.md

@@ -39,11 +39,21 @@ The linter also warns about low-quality cue routes that make future recall more
 
 ## Inputs
 
-The script reads:
+In project mode (default or `--memory-root`), the script requires all three
+setup files to exist as files:
 
 - `memory/impressions.jsonl`
+- `memory/summary.md`
+- `AGENTS.md`
 
-Use `--memory-root` or `--file` when the memory directory is not under the current working directory:
+For weak/strong results it also checks returned route targets and their lifecycle
+metadata, including successor links. These local validation reads do not inject
+record bodies into the agent context: the reported loading budget measures
+context loading, not filesystem I/O. Run `doctor` for a full health check.
+
+Use `--memory-root` for another project's memory directory. Use `--file`
+only for a standalone cue-index scan: it validates JSONL schema but does not
+validate setup files or target documents; the host owns those checks.
 
 ```bash
 node scripts/dejavu-scan-memory.mjs --memory-root ./examples/protocol-project/memory "protocol memory"
@@ -84,7 +94,12 @@ The script prints JSON:
 }
 ```
 
-If `memory/impressions.jsonl` is missing, the script returns `level: "not_initialized"` with a bootstrap hint instead of pretending the task had no familiarity match.
+If any required setup file is missing, project mode returns
+`ok: false, level: "not_initialized"` and exit code 1 with a bootstrap hint.
+Malformed JSONL, invalid schema, and unusable matched routes return
+`ok: false, level: "error"` and exit code 1. Healthy scans (including `none`)
+return `ok: true` and exit code 0. Never interpret an error as no familiarity.
+The linter, doctor, and feedback report also return nonzero on validation errors.
 
 ## Host Workflow
 
@@ -97,6 +112,8 @@ If `memory/impressions.jsonl` is missing, the script returns `level: "not_initia
 7. Record recall feedback only when the result should tune future cue quality.
 8. Continue to apply normal writeback and compaction rules after work completes.
 
+Stop recall on `error`; repair the indicated record or route first.
+
 ## Bootstrap Rule
 
 When installing Deja Vu into a project, copy or create the three required files:

package.json

@@ -32,9 +32,9 @@
     "clean": "node -e \"require('node:fs').rmSync('dist',{recursive:true,force:true})\"",
     "prepack": "npm run clean && npm run build",
     "prepublishOnly": "npm test",
-    "test": "npm run test:src:readonly && npm run check:links && npm run build && node --test dist/tests/semantic-recall-engine.test.js dist/tests/memory-cli.test.js && npm run test:pack",
+    "test": "npm run test:src:readonly && npm run check:links && npm run build && node --test dist/tests/semantic-recall-engine.test.js dist/tests/memory-cli.test.js dist/tests/reliability.test.js && npm run test:pack",
     "test:src": "npm run test:src:readonly",
-    "test:src:readonly": "node --import \"data:text/javascript,import { register } from 'node:module'; import { pathToFileURL } from 'node:url'; register('ts-node/esm', pathToFileURL('./'));\" --test tests/semantic-recall-engine.test.ts tests/memory-cli.test.ts",
+    "test:src:readonly": "node --import \"data:text/javascript,import { register } from 'node:module'; import { pathToFileURL } from 'node:url'; register('ts-node/esm', pathToFileURL('./'));\" --test tests/semantic-recall-engine.test.ts tests/memory-cli.test.ts tests/reliability.test.ts",
     "test:pack": "node --import \"data:text/javascript,import { register } from 'node:module'; import { pathToFileURL } from 'node:url'; register('ts-node/esm', pathToFileURL('./'));\" --test tests/package-pack.test.ts",
     "check:links": "node scripts/check-markdown-links.mjs",
     "scan:memory": "node scripts/dejavu-scan-memory.mjs",