Skip to content
Merged
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ Before adding code or content, search for an existing definition and extend it.
- **Uninstall preservation**: `--purge` snapshots canonical `identity.md` / `DISTILLED.md` with the databases and materializes them into Claude's MEMORY directory when that does not overwrite a foreign file. Installed root hooks and recursive `hooks/lib/` helpers must also appear in the matching `uninstall.sh` inventories; the uninstall test audits both.
- **MCP registration**: User scope in `~/.claude/settings.json` (or `~/.claude.json` if managed by `claude mcp add`) under `mcpServers`. The `env.RECALL_DB_PATH` block is populated by the installer.
- **Hook registration**: Claude events live in `~/.claude/settings.json`; Codex events live in `hosts/plugins/recall/hooks/hooks.json`; Grok capture uses the managed `~/.grok/hooks/RecallLifecycle.json` file.
- **omp capture ownership**: the root `package.json#omp.extensions` loads `hosts/omp/recall.ts` through omp's native plugin manager. Only the awaited main-session `session_stop` event captures; native task/subagent sessions are excluded by omp. The installer still owns skills only. The extension sends ordered branch text to package-local `dist/index.js capture` and does not write SQLite. No duplicate database writer or Pi fallback.
- **omp capture ownership**: the root `package.json#omp.extensions` loads `hosts/omp/recall.ts` through omp's native plugin manager. Only the awaited main-session `session_stop` event captures; native task/subagent sessions are excluded by omp. The installer still owns skills only. The extension sends ordered branch text to package-local `dist/index.js capture` and does not write SQLite. No duplicate database writer or Pi fallback. After a successful capture it detaches `recall loa omp-end` only when that session's Curated snapshot is missing or behind and no child is already running β€” not on every turn; fabric failure keeps the basic summary. See [omp integration](docs/OMP_INTEGRATION.md).
- **Hooks are self-contained**: `RecallExtract.ts`, `RecallStart.ts`, etc. are standalone scripts symlinked into `~/.claude/hooks/` from `~/.agents/Recall/shared/hooks/`. They don't import from `src/`. DB-path ownership follows the [source contracts](src/AGENTS.md#local-contracts).
- **Input-scaled SQL bind lists**: any `IN (...)` or multi-row `VALUES` whose placeholder count grows with input MUST chunk through `src/lib/chunk.ts`'s `chunked()` (conservative 500, under SQLite's bind-variable limit) β€” never bind the whole list at once. Because hooks can't import `src/` (see above), a hook with an input-scaled list inlines a local equivalent. Partly enforced by `tests/lib/chunk-audit.test.ts`, which fails when a new input-scaled placeholder list appears in a file that doesn't already route some query through `chunked()` (and isn't allowlisted fixed-size). The guard is **file-granular**, not per-statement: a file that already calls `chunked()` anywhere is exempt wholesale, so a new un-chunked sibling `IN (...)` added to such a file (e.g. `memory.ts`, `dump.ts`, `aging.ts`, `dedup.ts` β€” the bulk-SQL files most likely to grow one) is NOT caught. Chunk those by hand and rely on review, not the guard, there.
- **Shell-to-JS safety**: `install.sh` passes variables to `bun -e` via environment variables, never shell interpolation in JS strings.
Expand Down
4 changes: 2 additions & 2 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@ Recall's core product posture: it captures and curates memory in the background
## LoA

**Automatic-capture LoA**:
An extracted Library of Alexandria entry created by terminal session capture without an explicit user LoA command.
A Library of Alexandria entry produced by terminal session capture in the automatic-capture tier, separate from Curated LoA.
_Avoid_: hook LoA, Stop-hook LoA, Fabric extract (the column name is not the writer)

**Curated LoA**:
A Library of Alexandria entry created by an explicit user or agent LoA command.
A Library of Alexandria entry created by a user or agent LoA command, a session dump, or host background curation.
_Avoid_: manual LoA, Fabric LoA

## Extractor
Expand Down
4 changes: 4 additions & 0 deletions docs/OMP_INTEGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ Do not install the npm package named `recall-memory`; that name is an unrelated

This is turn-completion capture, not crash recovery. An interrupted turn before `session_stop` is not guaranteed captured. Shutdown and pre-compaction hooks are not installed. The active branch is read again on the next stop after a failure.

After a successful capture, the extension opens the database read-only to check curation freshness. It detaches the package-local `recall loa omp-end <sessionId>` worker when the session has published lifecycle messages and its Curated entry is missing or its source pins differ from those messages, provided no worker holds the session lock. Revisiting an earlier snapshot also triggers curation. The worker releases its lock and checks again before finishing so a capture that deferred to it is still covered.

The worker uses the configured [Curated Extractor](architecture.md#extractor-config), Fabric by default, and creates one separate entry tagged `curated,omp`. Later successful extracts replace that entry's body, source pins, message count, and snapshot cursor while retaining its ID. Only the session's ordered published lifecycle messages are extracted; explicit dumps on the same session stay separate. A changed extract invalidates the entry's old embedding; this worker does not generate a replacement embedding. Extraction failure leaves the basic summary and any previous Curated entry unchanged; a later successful capture can retry curation. Basic capture continues to work without Fabric.

Capture bounds its CLI child to 30 seconds and its serialized JSON payload to 25 MiB, including metadata, escaping, and the truncation marker. Excluded content does not count toward this limit. When the scrubbed text would exceed the limit, capture allocates 60% of the remaining JSON-escaped byte budget to the beginning and 40% to the end of the text, separated by `[Recall omp capture truncated]`; a warning reports the UTF-8 bytes dropped from the scrubbed text. Malformed branches are rejected before starting the child; malformed branches, cancellations, and child failures warn without stopping omp.

## Explicit dump: `recall_dump` tool
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,9 @@ MCP (`recall-mcp`) and `agent-skills/` remain the cross-host agent surfaces.
| host_ingest_generations | Lifecycle generation identity and publication status | No |
| host_ingest_generation_messages | Scrubbed generation records activated by one checkpoint pointer | Yes |
| host_ingest_embedding_invalidations | Pending semantic-index cleanup for an activated lifecycle generation | No |
| loa_message_sources | Retention-aware exact message lineage for automatic terminal summaries | No |
| loa_message_sources | Retention-aware exact message lineage for terminal summaries and pinned LoA entries | No |
| messages | Conversation turns (user + assistant content); includes `importance` (1-10) and a nullable internal lifecycle-publication token | Yes |
| loa_entries | Library of Alexandria entries (Automatic-capture LoA and Curated LoA); extract body in `fabric_extract` (column name, not the writer); includes `importance` (1-10, floor 5) and an immutable snapshot cursor independent of retention-nullable display ranges | Yes |
| loa_entries | Library of Alexandria entries (Automatic-capture LoA and Curated LoA); extract body in `fabric_extract` (column name, not the writer); includes `importance` (1-10, floor 5) and a snapshot cursor independent of retention-nullable display ranges ([omp update semantics](OMP_INTEGRATION.md#capture-contract)) | Yes |
| decisions | Architectural decisions with reasoning; includes `status` (active/superseded/reverted), `confidence` (high/medium/low), and `importance` (1-10) columns | Yes |
| learnings | Problems solved and patterns discovered; includes `confidence` (high/medium/low) and `importance` (1-10) columns | Yes |
| breadcrumbs | Contextual notes, references, and TODOs (with importance 1-10) | Yes |
Expand Down
2 changes: 2 additions & 0 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,8 @@ recall loa quote 1 # View raw source messages for ent

LoA is the primary knowledge capture mechanism. Raw transcripts are high-noise; Curated LoA entries contain `fabric` Extractor output, message lineage, continuation chains, and project context. Chaining (`-c <id>`) links a new entry to a prior one, forming a thread of related sessions. Automatic-capture LoA is written by session extract, not this command. Extractor config: [architecture](architecture.md#extractor-config).

The omp extension invokes `recall loa omp-end <sessionId>` for [background curation](OMP_INTEGRATION.md#capture-contract).

### Decisions

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/mcp-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ memory_stats()

## loa_show

Show a full Library of Alexandria entry. The body is the `fabric_extract` column (name, not the writer). Entries may be Automatic-capture LoA or Curated LoA; only Curated LoA from `recall loa` or `memory_dump` with `skip_fabric: false` is Fabric `extract_wisdom` output.
Show a full Library of Alexandria entry. The body is the `fabric_extract` column (name, not the writer). For entry types, see [LoA terminology](../CONTEXT.md#loa).

**Parameters**

Expand Down
3 changes: 2 additions & 1 deletion hooks/lib/db-path.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
// 2. MEM_DB_PATH (legacy fallback)
// 3. ~/.agents/Recall/recall.db (default)

import { homedir } from 'node:os';
import { join } from 'path';

/** Preserve the suffix: normalizing symlink/.. paths can select a different database. */
Expand All @@ -18,7 +19,7 @@ export function expandDbPathTilde(raw: string, home: string): string {
}

export function resolveDbPath(): string {
const home = process.env.HOME || process.env.USERPROFILE || '';
const home = process.env.HOME || process.env.USERPROFILE || homedir();
const raw = process.env.RECALL_DB_PATH || process.env.MEM_DB_PATH;
return raw ? expandDbPathTilde(raw, home) : join(home, '.agents', 'Recall', 'recall.db');
}
59 changes: 59 additions & 0 deletions hosts/omp/loa-protocol.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
import type { Database } from 'bun:sqlite';
import { closeSync, mkdirSync, openSync, unlinkSync } from 'node:fs';
import { homedir } from 'node:os';
import { dirname, join } from 'node:path';

export const OMP_CURATED_DESCRIPTION = 'Curated fabric extraction from omp session end.';

function ompLoaLockPath(sessionId: string): string {
return join(process.env.RECALL_HOME || join(homedir(), '.agents', 'Recall'), `omp-loa-${sessionId}.pid`);
}

export function exclusiveLoaLock(sessionId: string): boolean {
const path = ompLoaLockPath(sessionId);
mkdirSync(dirname(path), { recursive: true });
try {
closeSync(openSync(path, 'wx'));
return true;
} catch (error) {
if (error && typeof error === 'object' && 'code' in error && error.code === 'EEXIST') return false;
throw error;
}
}

export function releaseLoaLock(sessionId: string): void {
try {
unlinkSync(ompLoaLockPath(sessionId));
} catch {}
}

export function loadOmpLifecycleMessages(db: Database, sessionId: string): unknown[] {
return db.prepare(`
SELECT m.id, m.session_id, m.timestamp, m.role, m.content, m.project, m.importance, m.provenance
FROM active_host_ingest_messages h
JOIN published_messages m ON m.id = h.message_id
WHERE h.source = 'omp' AND h.session_id = ? AND h.source_position IS NOT NULL
ORDER BY h.source_position, h.message_id
`).all(sessionId);
}

export function ompSnapshotsMatch(left: readonly unknown[], right: readonly unknown[]): boolean {
return JSON.stringify(left) === JSON.stringify(right);
}

export function ompCuratedSnapshotBehind(db: Database, sessionId: string): boolean {
return db.transaction(() => {
const messages = loadOmpLifecycleMessages(db, sessionId);
if (messages.length === 0) return false;
const row = db.prepare(`
SELECT id FROM loa_entries WHERE session_id = ? AND description = ?
ORDER BY id DESC LIMIT 1
`).get(sessionId, OMP_CURATED_DESCRIPTION) as { id: number } | undefined;
if (!row) return true;
const pins = db.prepare(`
SELECT message_id AS id, session_id, timestamp, role, content, project, importance, provenance
FROM loa_message_sources WHERE loa_id = ? ORDER BY ordinal
`).all(row.id);
return !ompSnapshotsMatch(messages, pins);
})();
}
61 changes: 59 additions & 2 deletions hosts/omp/recall.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,13 @@
import { spawn } from 'node:child_process';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { Database } from 'bun:sqlite';
import type { z } from 'zod';
import { resolveDbPath as getDbPath } from '../../hooks/lib/db-path';
import { scrub } from '../../hooks/lib/write-safety';
import { exclusiveLoaLock, ompCuratedSnapshotBehind, releaseLoaLock } from './loa-protocol';

export { exclusiveLoaLock } from './loa-protocol';

const PACKAGE_ROOT = fileURLToPath(new URL('../..', import.meta.url));
const CLI_PATH = join(PACKAGE_ROOT, 'dist', 'index.js');
Expand Down Expand Up @@ -334,11 +339,47 @@ export function ompBranchMessages(entries: unknown): OmpMessage[] {
return messages;
}

export function curatedSnapshotBehind(sessionId: string): boolean {
let db: Database;
try {
db = new Database(getDbPath(), { readonly: true });
} catch {
return false;
}
try {
return ompCuratedSnapshotBehind(db, sessionId);
} catch {
return false;
} finally {
db.close();
}
}

export function spawnLoaOmpEnd(
sessionId: string,
launch: typeof spawn = spawn,
onError?: () => void,
): void {
const child = launch('bun', [CLI_PATH, 'loa', 'omp-end', sessionId], {
detached: true,
stdio: 'ignore',
});
child.once('error', () => {
releaseLoaLock(sessionId);
onError?.();
});
child.unref();
}

export async function captureOmpSessionStop(
event: OmpSessionStopEvent,
ctx: OmpExtensionContext,
pi?: OmpExtensionAPI,
runChild: RunBoundedChild = runBoundedChild,
curatedAhead: (sessionId: string) => boolean = curatedSnapshotBehind,
acquireLock: (sessionId: string) => boolean = exclusiveLoaLock,
spawnDetached: (sessionId: string) => void = spawnLoaOmpEnd,
launch: typeof spawn = spawn,
): Promise<void> {
try {
if (event.signal?.aborted) {
Expand Down Expand Up @@ -368,8 +409,24 @@ export async function captureOmpSessionStop(
return;
}
const { code, aborted } = await runCli(runChild, ['capture'], payload, event.signal, CHILD_TIMEOUT_MS);
if (code === 0) return;
warn(ctx, pi, aborted ? 'Recall omp capture cancelled' : 'Recall omp capture failed');
if (code !== 0) {
warn(ctx, pi, aborted ? 'Recall omp capture cancelled' : 'Recall omp capture failed');
return;
}
if (aborted) return;
const detach = spawnDetached === spawnLoaOmpEnd
? (id: string) => spawnLoaOmpEnd(id, launch, () => warn(ctx, pi, 'Recall omp curated extract failed'))
: spawnDetached;
let locked = false;
try {
if (!curatedAhead(sessionId)) return;
if (!acquireLock(sessionId)) return;
locked = true;
detach(sessionId);
} catch {
if (locked) releaseLoaLock(sessionId);
warn(ctx, pi, 'Recall omp curated extract failed');
}
} catch {
warn(ctx, pi, 'Recall omp capture failed');
}
Expand Down
123 changes: 122 additions & 1 deletion src/commands/loa.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
// recall loa command - Library of Alexandria capture

import { createLoaEntryFromMessages, getMessagesSinceLastLoa, getLastLoaEntry, getLoaEntry, getLoaMessages } from '../lib/memory.js';
import { createLoaEntry, createLoaEntryFromMessages, getMessagesSinceLastLoa, getLastLoaEntry, getLoaEntry, getLoaMessages, invalidateRecordEmbedding, replaceLoaMessagePins } from '../lib/memory.js';
import { detectProject } from '../lib/project.js';
import { embed, embeddingToBlob, checkEmbeddingService } from '../lib/embeddings.js';
import { getDb } from '../db/connection.js';
import { formatMessagesForExtraction, runFabricExtract } from '../lib/extraction.js';
import { spawn } from 'node:child_process';
import type { Message } from '../types/index.js';
import { exclusiveLoaLock, loadOmpLifecycleMessages, OMP_CURATED_DESCRIPTION, ompCuratedSnapshotBehind, ompSnapshotsMatch, releaseLoaLock } from '../../hosts/omp/loa-protocol.js';

interface LoaOptions {
continues?: number;
Expand Down Expand Up @@ -196,3 +199,121 @@ export function runLoaList(limit: number = 10): void {
console.log('');
}
}

const OMP_LOA_ATTEMPTS = 8;

function requeueOmpLoa(sessionId: string): void {
const child = spawn('bun', [process.argv[1], 'loa', 'omp-end', sessionId], {
detached: true,
stdio: 'ignore',
});
child.once('error', () => releaseLoaLock(sessionId));
child.unref();
}

function writeOmpCurated(
sessionId: string,
messages: Message[],
text: string,
): boolean {
const db = getDb();
db.exec('BEGIN IMMEDIATE');
try {
if (!ompSnapshotsMatch(loadOmpLifecycleMessages(db, sessionId), messages)) {
db.exec('ROLLBACK');
return false;
}
const session = db.prepare('SELECT project FROM sessions WHERE session_id = ?')
.get(sessionId) as { project: string | null } | undefined;
const existing = db.prepare(`
SELECT id, fabric_extract FROM loa_entries WHERE session_id = ? AND description = ?
ORDER BY id DESC LIMIT 1
`).get(sessionId, OMP_CURATED_DESCRIPTION) as { id: number; fabric_extract: string } | undefined;
const highWater = messages.reduce((max, message) => Math.max(max, message.id!), 0);
let loaId: number;
if (!existing) {
loaId = createLoaEntry({
title: `Omp session ${sessionId}`,
description: OMP_CURATED_DESCRIPTION,
fabric_extract: text,
snapshot_max_message_id: highWater,
session_id: sessionId,
project: session?.project || undefined,
tags: 'curated,omp',
message_count: messages.length,
provenance: 'extracted',
});
} else {
loaId = existing.id;
db.prepare(`
UPDATE loa_entries
SET fabric_extract = ?, message_count = ?, snapshot_max_message_id = ?
WHERE id = ?
`).run(text, messages.length, highWater, existing.id);
if (existing.fabric_extract !== text) invalidateRecordEmbedding(db, 'loa_entries', existing.id);
}
replaceLoaMessagePins(loaId, messages, db);
db.exec('COMMIT');
return true;
} catch (error) {
try { db.exec('ROLLBACK'); } catch { /* already closed */ }
throw error;
}
}

export async function runLoaOmpEnd(
sessionId: string,
deps: {
extract?: typeof runFabricExtract;
afterLoad?: () => void;
beforeWrite?: () => void;
afterWrite?: () => void;
} = {},
): Promise<number> {
const extract = deps.extract ?? runFabricExtract;
let handoff = false;
let hooked = false;
let completed = false;
try {
for (let attempt = 0; attempt < OMP_LOA_ATTEMPTS; attempt++) {
const messages = loadOmpLifecycleMessages(getDb(), sessionId) as Message[];
if (messages.length === 0) {
completed = true;
return 0;
}
if (attempt === 0) deps.afterLoad?.();
let text: string;
try {
text = extract(formatMessagesForExtraction(messages));
} catch {
return 0;
}
if (!ompSnapshotsMatch(loadOmpLifecycleMessages(getDb(), sessionId), messages)) continue;
if (!hooked) {
hooked = true;
deps.beforeWrite?.();
if (!ompSnapshotsMatch(loadOmpLifecycleMessages(getDb(), sessionId), messages)) continue;
}
if (writeOmpCurated(sessionId, messages, text)) {
completed = true;
deps.afterWrite?.();
return 0;
}
}
requeueOmpLoa(sessionId);
handoff = true;
return 0;
} finally {
if (!handoff) {
releaseLoaLock(sessionId);
if (completed && ompCuratedSnapshotBehind(getDb(), sessionId) && exclusiveLoaLock(sessionId)) {
try {
requeueOmpLoa(sessionId);
} catch (error) {
releaseLoaLock(sessionId);
throw error;
}
}
}
}
}
Loading
Loading