fix(projections): route erase obligations to erasure reconciler, not orphan path

[bgausden]

Problem

A successful dashboard DELETE surfaced as DEGRADED with an unfixable failure. Deleting fact ff8a4c44 created three kind=erase obligations under erasure id 210a524c (one per owner: bm25/temporal/vector). The 30s background _reconcile_pending_projections feeds every pending operation id to the ingestion-only _context_for_operation, which looks up ingestion_operations — a table erasure ids never appear in. Context resolves to None, so _terminalize_orphan_operation FAILED all three obligations with {"phase":"orphan","error":"canonical record missing"} and bumped attempts. After 10 passes the op was exhausted. retry can never succeed: there is no ingestion row to reconcile. Error text is misleading twice over — the missing row is the ingestion record, and the fact itself was correctly deleted and tombstoned.

Solution

Fan the redrive out by obligation kind instead of treating every operation as an ingestion:

• _pending_obligation_kinds reads the DISTINCT non-terminal kinds per op (loud on query error — never a silent skip).
• erase-kind groups go to a new _reconcile_erase_operation: requires canonical absent (profile-scoped) + tombstone present + every pending owner proving no residue via read-only ErasureService.prove_erased (new public wrapper over _prove_owner). Full proof → ERASED with no attempt bump, plus a manifest write so completed erasures leave the missing-manifest feed. Anything unproven stays pending, untouched.
• _terminalize_orphan_operation now only touches kind=apply and says ingestion record missing.

Alternatives rejected: bumping attempts on unproven erasures (ages them into silent abandonment); marking ERASED on tombstone alone (tombstone is written before owner purge in remove(), so presence proves nothing about residue); filtering operations_missing_manifest SQL (shared surface, wider blast radius — handled instead by skipping empty-kind ops and writing manifests on close).

Call tree:

_reconcile_pending_projections (30s redrive)
├── _pending_obligation_kinds ── empty → skip (terminal churn guard)
├── erase kinds → _reconcile_erase_operation → per (profile, subject) group
│   └── _reconcile_erase_group: canonical? tombstone? prove all → ERASED + manifest
└── apply kinds → _context_for_operation → reconcile | orphan-terminalize (apply only)

Changes

src/superlocalmemory/server/unified_daemon.py — the fix

Change	Why
_pending_obligation_kinds()	Kind fan-out; warning on query failure
_reconcile_erase_operation() + _reconcile_erase_group()	Read-only re-proof → ERASED, no bump, manifest on close
Redrive loop partitions by kind, skips empty	Erase never enters ingestion path; completed ops stop consuming slots
_terminalize_orphan_operation skips non-apply, renames error	Removes the false FAILED; names the actually-missing table

src/superlocalmemory/core/transactions/erasure.py — additive only

Change	Why
ErasureService.prove_erased() public wrapper	Redrive needs read-only proof without touching _prove_owner internals

tests/core/test_projection_spine_integration.py — regression tests

• test_redrive_does_not_orphan_erase_obligations — erase-only op + tombstone, no ingestion row → all erased, attempts 0, manifest written
• test_redrive_leaves_erase_pending_when_residue_remains — bm25 row present → stays pending, attempts 0
• test_redrive_leaves_erase_pending_without_tombstone — no tombstone → stays pending, attempts 0 across 2 passes

What Does Not Change

• Apply-path reconciliation, manifest derivation, and the orphan path for genuine ingestion orphans are untouched.
• No audit emission for redrive-completed erasures (delete-time finalize already emits the receipt path); no receipt rows written by the redrive.
• Unprovable erase ops stay pending without bumping — visible but never DEGRADED; follow-up may add observability/aging.
• No new dependencies; test helpers use stdlib + existing fixtures.

Breaking Changes

None.

Test Plan

Regression test (fails before fix, passes after):

• tests/core/test_projection_spine_integration.py::test_redrive_does_not_orphan_erase_obligations
  • Before: AssertionError: bm25: {'owner': 'bm25', 'state': 'failed', 'attempts': 1} — assert 'failed' == 'erased'
  • After: PASS (all 3 erased, attempts 0, manifest present)

Existing suites (no regression):

• pytest tests/core/test_projection_spine_integration.py — 9 passed
• pytest tests/core/test_erasure_service.py tests/core/test_erasure_saga.py tests/core/test_erasure_integration.py — 25 passed

Design review: three rubber-duck rounds with opencode/space-bunny-free (hole → fix → re-verify); round-2 findings applied (kind-query logging, collect-then-mark in one txn, per-subject groups, manifest-on-close, negative tests).

Manual: slm restart clean (integrity ok); slm ops status → HEALTHY after remediating pre-existing poisoned rows on the local 4.1.17 daemon (fix itself ships here, not yet installed locally).

Reviewer Notes

• Residual risk: reconcile writes the manifest in the same txn as the erase close; a commit failure mid-way can leave manifest and obligation rows divergent until the next pass.
• Door: two-way — pure background-path change, git revert restores old redrive; no schema migration.
• Blast radius: redrive + orphan-terminalize call sites only; erasure owners and ingestion paths untouched.

Fixes the 210a524c/ff8a4c44 orphan-erase incident. See also #148 (adjacent ops resolve CLI robustness, out of scope here).

[varun369]

Merged into 4.1.21, now on PyPI and npm, with your authorship kept. Thank you, this was a real bug.

On top of your fix, unconfirmed deletions now back off instead of retrying forever, they are re-checked in every profile, entity erasures heal, and an unreadable vector store never counts as erased. slm ops and the Ops Health page say what each entry is and what to do.