From 09ddeefc78beca428bbb433b223008110fb2380f Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 09:50:48 -0400 Subject: [PATCH 01/15] docs: plan controlled Cortex retrieval for Bot memory --- docs/ROADMAP.md | 8 ++ docs/design/bot-memory-integration.md | 180 ++++++++++++++++++++++++++ 2 files changed, 188 insertions(+) create mode 100644 docs/design/bot-memory-integration.md diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 9b38184..8b1340b 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -8,6 +8,14 @@ zero telemetry, cryptographically hardened, defense-in-depth. This is the prioritized forward plan. The per-iteration security audits (`SECURITY_AUDIT_ITER_*.md`) are the historical record; this is where we're going next. +### Bot memory integration proposal (2026-10-22) + +[Implementation plan](design/bot-memory-integration.md): reuse Cortex as an optional, +controlled retrieval index behind the Bot MemoryManager. PostgreSQL keeps memory and +permission authority; existing personal Cortex behavior remains compatible. W0-W4 cover +the first retrieval delivery; inference/tidy is a separate W5 enhancement. Design only: +no implementation, release, or deployment, and no change to the shipped status below. + --- ## Current state (verified 2026-06-10) diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md new file mode 100644 index 0000000..9546d83 --- /dev/null +++ b/docs/design/bot-memory-integration.md @@ -0,0 +1,180 @@ +# Cortex integration with Bot memory: implementation plan + +Date: 2026-10-22. Status: design only; no runtime implementation, migration, release, or deployment. + +## 1. Goal and fixed boundaries + +Reuse Cortex for retrieval of confirmed Bot memories before building another retrieval engine. Start with a disposable index; add inference and consolidation only after the controlled retrieval contract passes acceptance. + +The upstream Bot contract is [kol-agent PR #24](https://github.com/HEchooo/kol-agent/pull/24), final head `eb52ceafdc854be832183169ee428280f7278cce`, especially its [memory, context, and acceptance design](https://github.com/HEchooo/kol-agent/blob/eb52ceafdc854be832183169ee428280f7278cce/docs/bot-workbench-memory-session-design.md). This plan refines its PostgreSQL-first retrieval direction: PostgreSQL remains authoritative, Cortex is an optional derived retrieval backend. Adoption must be recorded in the Bot design/roadmap by its owner before enabling the integration. It does not require embedding-based retrieval to ship the Bot. + +- Bot MemoryManager/PostgreSQL owns identity, owner/bot/scope permissions, explicit versus candidate status, source validity, revisions, source ordering, expiry, correction, withdrawal, and public acknowledgements. +- Cortex owns derived storage, indexing, bounded ranking, and later candidate suggestions. It cannot acknowledge a successful business memory write or grant access. +- ContextProjector remains the sole model-context composer. Raw Cortex `get_context()` output is not injected alongside it. +- Workflow retains runs, checkpoints, approvals, actions, budget barriers, and recovery. Cortex receives no wallet, publishing, or execution authority. +- A local revision is an object concurrency token, not a cross-repository version/hash admission rule. No shared ORM or writes to another service's database. +- Existing personal Cortex, MCP, HTTP, sync, gateway, and cloud behavior remains compatible. The new integration uses a separate controlled store and explicit opt-in; personal archives and Muse exports are never imported implicitly. + +## 2. Evidence and reuse at the planning baseline + +Cortex baseline: `00f0e38da36bdafc8b40cf235aaac9e53bdf8548` (origin/main when planning began). Recheck functions against current main before implementing. Source links below are repository-relative; names, not historical line numbers, identify implementation seams. + +| Existing code | Reuse | Gap for Bot integration | +| --- | --- | --- | +| [core ingest lifecycle](../../cortex-core/src/lib.rs), `ingest_with_options`, `prepare_ingest`, `finalize_ingest` | Existing storage/index machinery and input checks | Normal ingest performs inference, deduplication, derived preference writes, and periodic consolidation; disabling the interval alone is insufficient | +| [MemObject and MemSource](../../cortex-core/src/types.rs) | Stable internal UUID, namespace, metadata, source, and privacy | Add a validated external projection envelope; generic metadata alone does not enforce revision or scope invariants | +| [retrieval](../../cortex-core/src/retrieval.rs) | Existing ranking, lexical/vector retrieval, and QueryBudget | Controlled requests require authorized scope filtering throughout retrieval, structured results, and explicit degraded/truncated reporting | +| [storage](../../cortex-core/src/storage/sqlite.rs) and [storage trait](../../cortex-core/src/storage/traits.rs) | SQLite transactions, CRUD, FTS triggers, memory cache | Atomic projection payload/state changes, stale-write fencing, and controlled-store lifecycle | +| [core deletion](../../cortex-core/src/lib.rs), `delete_memory` | SQL removal, index removal, cache generation, deletion event | External identity/tombstone handling and crash/race acceptance; a normal delete alone does not reject a later stale projection | +| [Python binding](../../cortex-python/src/lib.rs) | Existing PyO3 package and native engine | Current API lacks namespace arguments and delete exposure; retrieve returns only ID/score/text, without verification metadata | +| [HTTP guard](../../cortex-http/src/main.rs), `guard_origin` | Preserve existing local-only use | The local HTTP API has no authentication and can read Private memories; it is not a tenant service | + +The Python README example currently uses `cortex_python.PyCortex`, while the module/class declarations are `cortex_ai_memory.Cortex`. Verify the installed wheel and repair examples as part of the binding package work; do not infer installability from a source import. + +An open wheel-build PR (#14 at planning time) owns macOS deployment-target fixes. Coordinate with that work instead of duplicating or overwriting its changes. This plan makes no claims about current published wheel availability. + +The original local checkout also contains security commits outside this main baseline, including `c1d8658` (owner-only database files) and `5f522d6` (plaintext deletion). Verify their current PR/merge ownership before implementing related protections. They are not treated as shipped main capabilities here, and this plan neither copies nor overrides them. + +## 3. First delivery and responsibilities + +P0 indexes only confirmed, active, authorized memory supplied by MemoryManager. No raw transcript import, automatic fact/preference activation, people-identity merging, memory tidy application, cross-device sync, background inference, public API, or new UI is included. + +Use the existing Python binding as the first integration surface. Extend it with a controlled-store object, leaving existing `Cortex` calls unchanged. Use Rust core and existing storage/retrieval helpers underneath; do not replicate ranking or expose direct SQLite mutation to Python. HTTP and MCP expansion are not P0 prerequisites. If supported wheel/platform packaging cannot satisfy the host, stop enablement and document that concrete gap; do not silently switch transport or expose cortex-http. + +| Owner | Deliverables | +| --- | --- | +| Cortex | Controlled core API, persisted projection fences, scope-safe structured search, deletion/rebuild lifecycle, Python binding, distribution tests and documentation | +| chatbot-agent | PostgreSQL memory/outbox authority; adapter; replica cursors/reconciliation; permission/version revalidation; ContextProjector integration; fallback and end-to-end tests | +| Studio | Install the reviewed chatbot-agent/vendor package using the existing release process; retain current identity and permissions; no direct Cortex browser calls | +| Bot design owner | Record the optional retrieval backend and precise acceptance mapping in the existing design/roadmap | + +All API names below are proposed. No listed method or acceptance result is implemented by this document. + +## 4. Controlled projection contract (P0) + +### 4.1 Identity and data + +One projection key is `(owner_type, owner_id, bot_id, external_memory_id)`. Scope is immutable for that key; changing scope withdraws the old memory and creates a new authoritative identity. Two different external IDs remain separate even when their text is identical. Revisions are positive monotonic integers supplied by MemoryManager for this one key. + +Upsert accepts a typed envelope containing that key, `revision`, active state, `scope` (type and canonical entity ID where applicable), `kind`, bounded text, source_refs, and optional expiry. Store the envelope in existing metadata/source fields where possible. Internal UUID allocation/mapping is stable and durable; client display names, email labels, or text similarity do not determine identity. Data always remains Private and sync-disabled in this store; no privacy flag supplied by a client can enable sharing. + +Delete accepts the same key and withdrawal revision, without the old text. The managed scope is supplied by the trusted adapter, never chosen by a browser or inferred by a model. Core validates type/size, required identity, scope consistency, and supported fields; execution-relevant unknown fields are rejected. Contract tests must establish concrete input limits before implementation is considered ready; reuse existing limits where suitable rather than inventing unlimited payloads. + +### 4.2 Durable idempotency and ordering + +Persist one local projection-state record per key: maximum adopted revision, live/tombstone state, and internal memory ID. It is a derived replay fence, not another business authority. Store it in the same SQLite transaction as the corresponding memory replacement/removal, reusing transactional storage helpers. Do not call two separately committing CRUD methods and assume atomicity. + +- Higher revision replaces the live projection or writes a tombstone. +- Equal revision and equal normalized intent is a replay; return the original adopted outcome. Equal revision with different intent returns a conflict. +- Lower revision returns stale without mutating payload, index, or state. A tombstone also fences late upserts. Only a higher, explicitly reactivated authoritative revision may recreate the key. +- A duplicate delete is successful and reports already absent. A delete for a never-indexed key still persists its revision fence. +- Normalize supported semantic fields deterministically; do not compare arbitrary JSON serialization or logging/display metadata. Retain no withdrawn text merely to compare a tombstone replay. +- The adapter sends current authority, not a queued stale body: before replaying an outbox event, reload current memory/withdrawal state and authorize its scope. Delivery identities and cursors persist in PostgreSQL using the Bot's existing delivery patterns. + +Core serializes a controlled mutation and its index/cache publication within one instance. A search must not observe a newly adopted revision paired with old indexed text. After SQL commit, failure to publish index/cache state makes the instance unavailable until repaired/reopened; it cannot return success and continue serving stale state. After a crash, opening reconstructs indexes from committed controlled rows before search becomes ready. No network or model work occurs under a SQLite write transaction. + +### 4.3 Store and process isolation + +Use a dedicated database per owner/bot and per serving-process replica. Derive paths from server-maintained identity mappings under an owner-only data directory, not caller path strings. Require owner-only file handling, including WAL/SHM, and reject symlink/path escapes; coordinate the unmerged protections noted in section 2 rather than assuming this baseline provides them. Never open a personal Cortex archive as a controlled store. + +Persist a controlled-store marker. New legacy open/export/sync paths must refuse a marked controlled store, and controlled open refuses a personal store; do not rely only on caller discipline. The controlled binding exposes no general core/storage handle, legacy mutation, export, sync, or plugin access. Older binaries cannot honor a new marker, so deployment paths/access policy and rollback prohibit opening the controlled files with them. + +Each replica has its own persistent replay cursor and instance ID; do not use a shared work-queue acknowledgement as proof that every replica has applied an event. P0 supports independent local replicas, not multiple processes sharing one index database. Enforce one live controlled handle per database with an in-process registry and an OS-released-on-exit lock; a second process fails with an explicit busy result. This does not replace SQLite locks or imply a distributed lease. + +The Bot adapter manages replay for each replica. A new/lost replica builds from current PostgreSQL state; an expired replay cursor forces rebuild. A paused or lagging replica may serve only results that survive current authoritative revalidation, otherwise the Bot uses its PostgreSQL fallback. Record bounded lag and recovery state without memory text. + +Physical isolation is only the first boundary. Within the owner/bot store, each request carries a nonempty trusted set of currently allowed scopes; an empty set returns empty, never an unfiltered query. Missing scope is an error. All retrieval branches and caches, including FTS, vectors, graph expansion, keyword/semantic helpers, and cache keys, must preserve the same scope predicate. Filter before ranking/top-k and do not expose excluded IDs, snippets, counts, or metadata. If an existing branch cannot enforce the predicate, disable that branch in controlled mode until it can. + +### 4.4 Search and final authority checks + +`search(query, allowed_scopes, limit, budget)` returns structured hits with external identity, revision, scope, source_refs, score, and indexed text, plus explicit `degraded/truncated` information. These are internal candidates, not browser-safe responses. Enforce bounded query/result sizes and retain QueryBudget; namespace is a secondary partition, not authorization proof. + +The adapter batch-loads current PostgreSQL memories and checks owner/bot/scope, active status, expiry, source access/withdrawal, and revision before exposing anything to UI, logs, or ContextProjector. Rejected candidates cannot enter excerpts, telemetry, summaries, or explanations. Emit current PostgreSQL text and source metadata after validation, not stale indexed text. Candidate shortage/timeout is not proof that no memory exists: fall back to the existing bounded PostgreSQL retrieval and report optional-context degradation. Required execution facts continue to come from the current work or their authoritative service. + +At model-call/action boundaries, the existing Bot rules still invalidate withdrawn memory manifests, summaries, and checkpoints. Cortex does not claim to revoke content already sent to a model. Revalidate at the same boundary as the existing permission checks; no new distributed transaction with a model supplier is claimed. + +### 4.5 Forget, rebuild, and maintenance + +MemoryManager commits withdrawal and its public acknowledgement independently of index availability. All reads honor that withdrawal immediately through authoritative validation. Cortex cleanup is tracked separately: reuse core deletion, FTS triggers, index removal, cache generation and events, with the projection tombstone committed atomically. Keep retrying the same identity through the existing finite retry/operator-recovery mechanism; failed cleanup remains visible. + +Rebuild constructs a separate controlled database from a consistent, authorized PostgreSQL snapshot plus a durable committed replay watermark, then applies every later change in order. Until the adapter revalidates state, the rebuilt store is not ready. Withdrawals, expirations, or permission changes during rebuild must not leak through the swap. Swap/retire handles under a local gate; pin each active query to one instance generation and drain old handles before deletion. A crash before/after the swap must reopen either a validated ready generation or a recoverable not-ready state, never a half-built database. + +Do not discard tombstones while their revision range can still be replayed. A rebuild may omit old tombstones only when the old replica is retired, its generation/cursor cannot publish again, a new snapshot/replay boundary is established, and every later delivery is resolved from current authority. Backup restoration follows the same reconciliation before readiness. Local projection files, WAL/SHM, staging files and retired generations stay owner-only and are included in cleanup/retention inventory. + +Logical withdrawal and physical erasure are distinct: no immediate wiping of existing backups or already-sent model requests is promised. State exactly what cleanup finished and what remains. No automatic device sync or export is enabled for these projections. + +## 5. Python delivery and operational fallback (P0) + +Expose controlled open/upsert/delete/search/status/close through one documented binding; choose final names in the first contract package. Errors distinguish invalid scope/input, stale revision, equal-revision conflict, busy/not-ready store, and bounded retrieval degradation. Search results must not require parsing a generated prompt or tuple text to recover identity. + +P0 runtime uses a supported wheel installed in a clean environment, not an adjacent Rust checkout. Add a built-wheel smoke/integration test covering import/class name, controlled APIs, restart, and scope/delete behavior. Rust workspace CI excludes cortex-python and therefore cannot substitute for this test. Verify the host's actual Python, OS, and architecture; add the binding job to the required integration checks rather than treating a skipped release workflow as acceptance. Coordinate macOS fixes with PR #14. + +The current Python crate disables core's encrypted-db feature. Do not describe this wheel's SQLite data as encrypted at rest. Controlled deployment must use the host's existing approved private-data storage and retention controls; if those require application-level encryption, provide and test an encrypted wheel configuration before enablement. Never weaken storage policy just to make the wheel install. + +Native work must not block the host event loop. Define binding GIL release/thread-safety and call it through the host's bounded worker mechanism, with concurrency, queue, and timeout limits. Cancellation cannot leave SQL committed with a permanently unpublished index. No detached unbounded indexing threads. Reuse core/query budgets, expose metadata-only health/lag/error metrics, and do not add telemetry/network dependencies to the core. Embedding initialization must be explicit; tests are offline with supplied synthetic vectors or lexical mode, and production model availability is checked before enabling optional semantic retrieval. + +Cortex failure does not reject an otherwise valid Bot memory write or bypass permission checks. If Cortex is missing, not ready, or unusable, use PostgreSQL retrieval with a clear capability/degradation state. The integration feature flag defaults off. Disablement stops new controlled work and drains/closes handles; the original Bot and personal Cortex paths remain usable. Do not roll back by opening a controlled database with an older binary that does not understand its fences. + +## 6. Later candidate extraction and tidy (P1, separate enablement) + +Only after P0 acceptance, reuse `infer` and consolidation analysis to propose structured candidates with provenance and confidence. Current live `run_consolidation` is not a read-only preview and cannot be wired to the Bot preview button. Add a no-write analysis path sharing existing analysis helpers, or run analysis on an explicitly disposable isolated copy; test that previews do not mutate live rows or derived state. + +MemoryManager remains responsible for accepting/rejecting a candidate, preserving explicit user corrections and causal source ordering. Consolidation cannot activate speculation, overwrite a user-confirmed value, or merge scopes. Any accepted change returns through the P0 projection path. Tidy apply/undo and expected revisions reuse the upstream Bot contract rather than introducing a second report or approval protocol. + +Before enabling derived candidates, persist sufficient parent memory/input identities and revisions to invalidate all derived facts, preferences, summaries and proposals when a source is corrected, forgotten or loses access. Failure to establish that lineage keeps P1 disabled. People graph, cross-Bot sharing, and device/cloud sync remain separate proposals, not an automatic consequence of enabling P1. + +## 7. Work packages and exits + +Implement on isolated branches against current main, preserving others' work. Each implementation PR carries its own final diff review/CI; review of this plan does not approve later code. + +| Package | Owner and code seams | Deliverable and exit | +| --- | --- | --- | +| W0: contract fixtures | Cortex + chatbot-agent owners; types and adapter interface | Freeze typed envelope, input limits, errors, allowed-scope predicate, replica/rebuild sequence and supported wheel target; document Bot adoption; agree synthetic fixtures | +| W1: controlled core | Cortex `types.rs`, `lib.rs`, storage trait/SQLite, index/retrieval | Implement projection-state transaction, controlled open/mutation/read lifecycle and fail-closed guards; C01-C10 and C12 pass | +| W2: binding/distribution | Cortex Python binding, its README, existing build/release workflows | Expose one supported surface; correct import example; installed-wheel C11/C13 pass; retain existing public API behavior | +| W3: Bot adapter | chatbot-agent memory/deliveries/ContextProjector and fixtures | Implement per-replica delivery/reconciliation, authority gate, PostgreSQL fallback and feature flag; C14-C17 pass with real test PostgreSQL | +| W4: host verification | chatbot-agent + Studio existing vendor/release flow | Install reviewed package in isolated host; C18 plus upstream memory/context regression pass; authorized rollout/rollback evidence | +| W5: candidate enhancement | Cortex + MemoryManager, separately reviewed | P1 no-write candidate preview and lineage; C19/C20 pass before any activation | + +W1 and W2 follow W0; W3 can use contract fixtures while W1/W2 develop, but integration waits for both. W4 waits for W1-W3. W5 is optional after W4 and cannot block the P0 retrieval release. + +## 8. Acceptance matrix (required, not executed) + +Use synthetic memory identities/text, real temporary SQLite and real test PostgreSQL where indicated. No production archive import, private model photos, paid generation, or publishing is necessary. Failure injection uses actual process termination and real files/transactions, not mocks in place of storage acceptance. + +| ID | Scenario | Required result | +| --- | --- | --- | +| C01 | Controlled single/batch projection; same text under different IDs/scopes | No automatic facts/preferences/people/sync, dedup merge, decay or consolidation; independent IDs and permissions preserved | +| C02 | Replay same envelope; equal revision with different text/scope/source intent; lower revision | Replay is stable; conflicting intent rejected; lower revision has no effect | +| C03 | Commit rev 2 while rev 1 is delayed; restart between delivery and acknowledgement | Only rev 2 visible; original identities reused; persisted state and payload agree | +| C04 | Delete unseen key; repeat delete; delayed upsert; explicit higher reactivation | Durable tombstone, idempotent deletion, stale body rejected; only authoritative higher reactivation returns | +| C05 | Kill writer before SQL commit, after commit/before index publication, and after publication/before response | Reopen either original or fully committed state; no ready instance exposes mixed revisions/text; retries preserve outcome | +| C06 | Concurrent search/update/delete, cache hit, injected publication failure | Results bind one valid state; instance fails closed on publication failure; SQL/index/cache agree after repair | +| C07 | Same text/IDs in two owners and two Bots; path manipulation; two processes opening one replica DB | No cross-store data; safe owner-only paths; shared-handle attempts rejected and restart releases lock | +| C08 | Entity scope A/B within one Bot across every retrieval branch and warm cache | Allowed predicate holds before top-k; no excluded snippets/IDs/counts; empty scopes return empty, missing scope errors | +| C09 | Inspect structured search results; budget exhaustion and malformed oversized inputs | Identity/version/source/scope intact; bounded results; degraded search is explicit; validation is deterministic | +| C10 | Delete then inspect SQL, FTS, vector index/cache; restart | No queryable withdrawn content; fence remains without old plaintext; cleanup failures reported rather than claimed complete | +| C11 | Install built wheel in clean supported host environment, then upsert/search/delete/reopen | Correct module/class and structured types; no checkout dependency; no missing controlled APIs | +| C12 | Existing personal core/MCP/HTTP/sync/gateway regressions | Legacy calls and opt-in behavior preserved; controlled data cannot be exported or synced through legacy entrypoints | +| C13 | Native call under host concurrency, timeout/cancellation, embedding unavailable, encrypted-storage policy | Event loop stays responsive; bounded work; no publication gap; offline mode usable; required at-rest controls verified | +| C14 | Real test PG commit plus lost event/ack, restart, two replicas at different cursors, retention expiry | Each replica reconciles independently; stale replicas only return authority-approved results or fallback; expired cursor rebuilds | +| C15 | Correct/forget/revoke/expire after indexing or during search, including rejected source access | Current PG gate removes invalid hits and uses authoritative text; model-call boundary checks hold; no rejected content reaches logs/UI/context | +| C16 | Rebuild from empty with concurrent changes, permission revocation, stale delivery, kill/swap, restore backup | New snapshot/replay generation valid; no resurrection/leak; half-built or obsolete generation never becomes ready | +| C17 | Core unavailable, flag off, slow search, too few valid candidates, invalid authority lookup | PG fallback with truthful status; no unvalidated hit and no lost canonical write; invalid permissions fail closed | +| C18 | Host installed package, feature toggle, worker restart, rollback, new-work recall and forget | Verify actual package/config/process and end-to-end behavior; upstream M01-M06/N01-N03/L08 and existing work/approval regressions pass | +| C19 | Candidate inference and tidy preview with explicit correction/conflicting scope | No live mutation; suggestion retains provenance; no overwrite, activation or cross-scope merge | +| C20 | Forget/correct/revoke parent while preview or derived processing is delayed | Descendants invalidated; replay/undo cannot restore withdrawn evidence; P0 behavior unchanged | + +An acceptance subagent must be context-isolated from implementation: give it this plan and public schemas/docs, not implementation diffs or unit tests. Implementers write unit tests separately. Adversarial review is a third independent context, per repository rules. + +## 9. Review, CI, rollout and evidence + +For this documentation change, validate local links, fences, acceptance IDs, and `git diff --check`. No business acceptance is claimed. Keep the exact reviewed document content and head/base identifiers with raw model outputs in a traceable review record. Each model first reviews independently; cross-check only after both initial findings are saved. Fix every valid actionable finding at every severity, then repeat both reviews over the entire final diff; disputed findings require the originating reviewer to confirm closure. + +Before any implementation merge, require two different underlying reviewer models independently clean on the final complete head/base and all required CI on that same integration version. Recheck head/base, conflicts, findings and checks immediately before merging with exact-head constraint. Follow the authorized admin merge policy only after these gates; do not bypass failed/missing/skipped required checks. A main failure or base movement requires investigation and fresh evidence, not inherited clean conclusions. + +Required implementation verification includes `cargo test --workspace --exclude cortex-python --exclude cortex-wasm`, the current required Clippy/privacy jobs, installed-wheel tests, and the integration matrix above for that package. Relevant existing snapshots/golden/compatibility assertions must be checked; do not rerecord baselines to mask regressions. UI remains outside this plan; any later UI change needs desktop/mobile and affected flows. + +Rollout uses a clean main containing the latest release and the existing package/vendor process: test environment first, then explicitly authorized scope. Record Cortex/chatbot-agent/Studio SHAs, wheel identity, flags, storage/replica configuration, CI runs, each reviewer model/round, fault-injection results and unresolved limits. Verify actual installed package, processes, health, replay lag, cleanup failures, logs and recall/correction/forget behavior after release. Separate implementation, isolated tests, deployment and real authorized end-to-end acceptance; anything not deployed must be labeled not deployed. + +Completion of this plan means a reviewed design artifact only. Completion of P0 requires W0-W4 and C01-C18 evidence; P1 remains disabled until W5/C19-C20 also pass. This document grants no production operation or data-sharing authorization. From d9867092596b9633abd097b410186c5ddad9a96a Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 09:56:41 -0400 Subject: [PATCH 02/15] docs: define Bot integration boundaries and durable replay recovery --- README.md | 15 +++++++++++++++ docs/design/bot-memory-integration.md | 6 ++++-- docs/guides/integrations.md | 6 ++++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6eb8f80..aad3b23 100644 --- a/README.md +++ b/README.md @@ -137,6 +137,21 @@ Cortex implements a 4-tier memory model inspired by human cognition: **Working** holds the current session scratch pad. **Episodic** stores raw experiences with timestamps and source metadata. The **Consolidation Engine** periodically promotes recurring patterns into **Semantic** facts and decays stale episodes. **Procedural** captures learned workflows and routines. +### Bot integration architecture (proposed) + +For a hosted Bot with existing PostgreSQL memory and account permissions, Cortex is an +optional derived retrieval backend behind its MemoryManager. PostgreSQL remains the +authority for confirmed values, scope, provenance, revisions and withdrawal; +ContextProjector remains the single model-context composer, and Workflow keeps execution +and approvals. Search hits must pass current authoritative validation before entering +model context. A controlled projection store must not import personal archives or enable +sync, automatic inference or consolidation by default. + +The [Bot memory integration plan](docs/design/bot-memory-integration.md) defines the +proposed controlled-store contract, Cortex versus host responsibilities, implementation +order, failure recovery and acceptance. It is a design, not an available SDK API or a +deployed Bot integration; existing personal-agent integration paths below are unchanged. + ## Key Components ### People Graph diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index 9546d83..a23192f 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -73,6 +73,8 @@ Persist one local projection-state record per key: maximum adopted revision, liv Core serializes a controlled mutation and its index/cache publication within one instance. A search must not observe a newly adopted revision paired with old indexed text. After SQL commit, failure to publish index/cache state makes the instance unavailable until repaired/reopened; it cannot return success and continue serving stale state. After a crash, opening reconstructs indexes from committed controlled rows before search becomes ready. No network or model work occurs under a SQLite write transaction. +Projection durability must precede acknowledgement of its replay position in PostgreSQL. Persist the replica generation and local applied replay position in the same SQLite transaction as payload/fence changes; make those changes durable with a tested controlled-store durability mode or the existing durable-flush mechanism before advancing the PostgreSQL cursor. `synchronous=NORMAL` commit by itself is not this guarantee. After restart/restore, compare local generation/applied position with the PostgreSQL cursor before readiness; if local data may have rolled back or the identities disagree, rewind to a proven retained position or rebuild. Never trust the remote cursor alone. Crash after durable local commit but before remote acknowledgement safely replays the same identities. C05/C14 must simulate local acknowledged-transaction loss with the PostgreSQL cursor surviving, not just kill a process. + ### 4.3 Store and process isolation Use a dedicated database per owner/bot and per serving-process replica. Derive paths from server-maintained identity mappings under an owner-only data directory, not caller path strings. Require owner-only file handling, including WAL/SHM, and reject symlink/path escapes; coordinate the unmerged protections noted in section 2 rather than assuming this baseline provides them. Never open a personal Cortex archive as a controlled store. @@ -148,7 +150,7 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr | C02 | Replay same envelope; equal revision with different text/scope/source intent; lower revision | Replay is stable; conflicting intent rejected; lower revision has no effect | | C03 | Commit rev 2 while rev 1 is delayed; restart between delivery and acknowledgement | Only rev 2 visible; original identities reused; persisted state and payload agree | | C04 | Delete unseen key; repeat delete; delayed upsert; explicit higher reactivation | Durable tombstone, idempotent deletion, stale body rejected; only authoritative higher reactivation returns | -| C05 | Kill writer before SQL commit, after commit/before index publication, and after publication/before response | Reopen either original or fully committed state; no ready instance exposes mixed revisions/text; retries preserve outcome | +| C05 | Kill writer before SQL commit, after commit/before index publication, and after publication/before response; simulate loss of an acknowledged local transaction | Reopen either original or fully committed state; readiness requires generation/replay-position reconciliation; no mixed revisions/text or silently skipped lost projection | | C06 | Concurrent search/update/delete, cache hit, injected publication failure | Results bind one valid state; instance fails closed on publication failure; SQL/index/cache agree after repair | | C07 | Same text/IDs in two owners and two Bots; path manipulation; two processes opening one replica DB | No cross-store data; safe owner-only paths; shared-handle attempts rejected and restart releases lock | | C08 | Entity scope A/B within one Bot across every retrieval branch and warm cache | Allowed predicate holds before top-k; no excluded snippets/IDs/counts; empty scopes return empty, missing scope errors | @@ -157,7 +159,7 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr | C11 | Install built wheel in clean supported host environment, then upsert/search/delete/reopen | Correct module/class and structured types; no checkout dependency; no missing controlled APIs | | C12 | Existing personal core/MCP/HTTP/sync/gateway regressions | Legacy calls and opt-in behavior preserved; controlled data cannot be exported or synced through legacy entrypoints | | C13 | Native call under host concurrency, timeout/cancellation, embedding unavailable, encrypted-storage policy | Event loop stays responsive; bounded work; no publication gap; offline mode usable; required at-rest controls verified | -| C14 | Real test PG commit plus lost event/ack, restart, two replicas at different cursors, retention expiry | Each replica reconciles independently; stale replicas only return authority-approved results or fallback; expired cursor rebuilds | +| C14 | Real test PG commit plus lost event/ack, restart, two replicas at different cursors, retention expiry; PG cursor ahead of restored local data | Local durable position precedes remote ack; each replica reconciles independently, rewinds/rebuilds on mismatch; stale replicas only return authority-approved results or fallback | | C15 | Correct/forget/revoke/expire after indexing or during search, including rejected source access | Current PG gate removes invalid hits and uses authoritative text; model-call boundary checks hold; no rejected content reaches logs/UI/context | | C16 | Rebuild from empty with concurrent changes, permission revocation, stale delivery, kill/swap, restore backup | New snapshot/replay generation valid; no resurrection/leak; half-built or obsolete generation never becomes ready | | C17 | Core unavailable, flag off, slow search, too few valid candidates, invalid authority lookup | PG fallback with truthful status; no unvalidated hit and no lost canonical write; invalid permissions fail closed | diff --git a/docs/guides/integrations.md b/docs/guides/integrations.md index 219ff03..435465f 100644 --- a/docs/guides/integrations.md +++ b/docs/guides/integrations.md @@ -2,6 +2,12 @@ Cortex provides persistent, local-first memory for AI agents. This guide shows how to integrate Cortex with popular multi-agent frameworks. +For hosted Bots that already own memory and permissions in PostgreSQL, follow the +[controlled Bot integration architecture and plan](../design/bot-memory-integration.md). +That proposed path uses Cortex only as a derived retrieval backend, with mandatory +authoritative scope/version/withdrawal checks. Its controlled APIs are not implemented +yet; the personal-agent examples below do not provide that hosted isolation contract. + **What Cortex adds to any framework:** - **Persistent memory** -- 4-tier (Working / Episodic / Semantic / Procedural) that survives across sessions From d4194c50f8e9d6c5ffaf2528f6c640fbae56da4b Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 09:59:31 -0400 Subject: [PATCH 03/15] docs: retain scope through withdrawal and repair integration links --- docs/design/bot-memory-integration.md | 6 +++--- docs/guides/integrations.md | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index a23192f..4079ae5 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -58,11 +58,11 @@ One projection key is `(owner_type, owner_id, bot_id, external_memory_id)`. Scop Upsert accepts a typed envelope containing that key, `revision`, active state, `scope` (type and canonical entity ID where applicable), `kind`, bounded text, source_refs, and optional expiry. Store the envelope in existing metadata/source fields where possible. Internal UUID allocation/mapping is stable and durable; client display names, email labels, or text similarity do not determine identity. Data always remains Private and sync-disabled in this store; no privacy flag supplied by a client can enable sharing. -Delete accepts the same key and withdrawal revision, without the old text. The managed scope is supplied by the trusted adapter, never chosen by a browser or inferred by a model. Core validates type/size, required identity, scope consistency, and supported fields; execution-relevant unknown fields are rejected. Contract tests must establish concrete input limits before implementation is considered ready; reuse existing limits where suitable rather than inventing unlimited payloads. +Delete accepts the same key, canonical scope and withdrawal revision, without the old text. An unseen-key deletion must also provide that scope from authoritative state; missing scope is rejected, not replaced by a wildcard. The managed scope is supplied by the trusted adapter, never chosen by a browser or inferred by a model. Core validates type/size, required identity, scope consistency, and supported fields; execution-relevant unknown fields are rejected. Contract tests must establish concrete input limits before implementation is considered ready; reuse existing limits where suitable rather than inventing unlimited payloads. ### 4.2 Durable idempotency and ordering -Persist one local projection-state record per key: maximum adopted revision, live/tombstone state, and internal memory ID. It is a derived replay fence, not another business authority. Store it in the same SQLite transaction as the corresponding memory replacement/removal, reusing transactional storage helpers. Do not call two separately committing CRUD methods and assume atomicity. +Persist one local projection-state record per key: immutable canonical scope, maximum adopted revision, live/tombstone state, and internal memory ID (allocated when first materialized). Retain scope in tombstones across deletion, restart and reactivation; compare every later mutation against it and reject changed scope even at a higher revision. An unseen delete establishes the scope from its authoritative envelope. It is a derived replay fence, not another business authority. Store it in the same SQLite transaction as the corresponding memory replacement/removal, reusing transactional storage helpers. Do not call two separately committing CRUD methods and assume atomicity. - Higher revision replaces the live projection or writes a tombstone. - Equal revision and equal normalized intent is a replay; return the original adopted outcome. Equal revision with different intent returns a conflict. @@ -149,7 +149,7 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr | C01 | Controlled single/batch projection; same text under different IDs/scopes | No automatic facts/preferences/people/sync, dedup merge, decay or consolidation; independent IDs and permissions preserved | | C02 | Replay same envelope; equal revision with different text/scope/source intent; lower revision | Replay is stable; conflicting intent rejected; lower revision has no effect | | C03 | Commit rev 2 while rev 1 is delayed; restart between delivery and acknowledgement | Only rev 2 visible; original identities reused; persisted state and payload agree | -| C04 | Delete unseen key; repeat delete; delayed upsert; explicit higher reactivation | Durable tombstone, idempotent deletion, stale body rejected; only authoritative higher reactivation returns | +| C04 | Delete unseen key with/without scope; repeat delete; delayed upsert; restart then higher reactivation in same/different scope | Missing scope rejected; durable scoped tombstone; idempotent deletion and stale-write rejection; only authoritative higher reactivation in original scope succeeds, changed scope rejected | | C05 | Kill writer before SQL commit, after commit/before index publication, and after publication/before response; simulate loss of an acknowledged local transaction | Reopen either original or fully committed state; readiness requires generation/replay-position reconciliation; no mixed revisions/text or silently skipped lost projection | | C06 | Concurrent search/update/delete, cache hit, injected publication failure | Results bind one valid state; instance fails closed on publication failure; SQL/index/cache agree after repair | | C07 | Same text/IDs in two owners and two Bots; path manipulation; two processes opening one replica DB | No cross-store data; safe owner-only paths; shared-handle attempts rejected and restart releases lock | diff --git a/docs/guides/integrations.md b/docs/guides/integrations.md index 435465f..f2faa8c 100644 --- a/docs/guides/integrations.md +++ b/docs/guides/integrations.md @@ -298,7 +298,7 @@ async with MultiServerMCPClient({ Your LangGraph agent gets instant access to `memory_search`, `memory_ingest`, `fact_add`, `belief_observe`, `person_resolve`, and 24 more tools -- all running locally. -See the [LangGraph section in README.md](../README.md#integration-with-langgraph) for full details. +See the [LangGraph section in README.md](../../README.md#integration-with-langgraph) for full details. --- @@ -328,7 +328,7 @@ mcp_servers: All DeerFlow agents (Telegram, Slack, Feishu) get instant access to 29 memory tools -- cross-session memory, fact storage, people graph, and belief tracking across all channels. -See the [DeerFlow section in README.md](../README.md#integration-with-deerflow-bytedance) for full details. +See the [DeerFlow section in README.md](../../README.md#integration-with-deerflow-bytedance) for full details. --- @@ -366,7 +366,7 @@ openclaw plugin add @cortex-ai-memory/cortex-memory - `autoRecall` -- injects relevant memories before each turn (your agent "remembers") - 7 tools: `memory_search`, `memory_store`, `fact_add`, `belief_observe`, `person_resolve`, and more -See the [OpenClaw section in README.md](../README.md#openclaw-plugin) and [`openclaw-plugin/README.md`](../openclaw-plugin/README.md) for full configuration options. +See the [OpenClaw section in README.md](../../README.md#openclaw-plugin) and [`openclaw-plugin/README.md`](../../openclaw-plugin/README.md) for full configuration options. --- @@ -406,4 +406,4 @@ curl -X POST http://localhost:3315/v1/facts \ -d '{"subject": "Alice", "predicate": "works_at", "object": "Stripe", "confidence": 0.95, "channel": "api"}' ``` -See the [HTTP API section in README.md](../README.md#http-api) for the full endpoint reference. +See the [HTTP API section in README.md](../../README.md#http-api) for the full endpoint reference. From eb5bbe05a723c63ce3fa8c259e52e5a540f1604e Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 10:02:49 -0400 Subject: [PATCH 04/15] docs: preserve authoritative exact-key defaults before optional recall --- docs/design/bot-memory-integration.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index 4079ae5..9a487db 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -91,6 +91,8 @@ Physical isolation is only the first boundary. Within the owner/bot store, each `search(query, allowed_scopes, limit, budget)` returns structured hits with external identity, revision, scope, source_refs, score, and indexed text, plus explicit `degraded/truncated` information. These are internal candidates, not browser-safe responses. Enforce bounded query/result sizes and retain QueryBudget; namespace is a secondary partition, not authorization proof. +MemoryManager first resolves required defaults by exact key from current PostgreSQL state with the existing scope/precedence rules, independently of Cortex readiness, hit count, or index lag. Current explicit instructions and work constraints still override defaults. Optional Cortex retrieval supplements this mandatory lookup; it cannot replace it. In particular, an execution action depending on a just-committed memory action must see that default even if no replica has indexed it yet and Cortex returns a full page of unrelated valid hits. + The adapter batch-loads current PostgreSQL memories and checks owner/bot/scope, active status, expiry, source access/withdrawal, and revision before exposing anything to UI, logs, or ContextProjector. Rejected candidates cannot enter excerpts, telemetry, summaries, or explanations. Emit current PostgreSQL text and source metadata after validation, not stale indexed text. Candidate shortage/timeout is not proof that no memory exists: fall back to the existing bounded PostgreSQL retrieval and report optional-context degradation. Required execution facts continue to come from the current work or their authoritative service. At model-call/action boundaries, the existing Bot rules still invalidate withdrawn memory manifests, summaries, and checkpoints. Cortex does not claim to revoke content already sent to a model. Revalidate at the same boundary as the existing permission checks; no new distributed transaction with a model supplier is claimed. @@ -162,7 +164,7 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr | C14 | Real test PG commit plus lost event/ack, restart, two replicas at different cursors, retention expiry; PG cursor ahead of restored local data | Local durable position precedes remote ack; each replica reconciles independently, rewinds/rebuilds on mismatch; stale replicas only return authority-approved results or fallback | | C15 | Correct/forget/revoke/expire after indexing or during search, including rejected source access | Current PG gate removes invalid hits and uses authoritative text; model-call boundary checks hold; no rejected content reaches logs/UI/context | | C16 | Rebuild from empty with concurrent changes, permission revocation, stale delivery, kill/swap, restore backup | New snapshot/replay generation valid; no resurrection/leak; half-built or obsolete generation never becomes ready | -| C17 | Core unavailable, flag off, slow search, too few valid candidates, invalid authority lookup | PG fallback with truthful status; no unvalidated hit and no lost canonical write; invalid permissions fail closed | +| C17 | Core unavailable, flag off, slow/short search, invalid authority lookup; just-confirmed required default missing from delayed replica that returns a full page of other valid hits | Mandatory exact-key PG defaults always precede optional search; dependent action uses the committed default; fallback is truthful, writes retained, invalid permissions fail closed | | C18 | Host installed package, feature toggle, worker restart, rollback, new-work recall and forget | Verify actual package/config/process and end-to-end behavior; upstream M01-M06/N01-N03/L08 and existing work/approval regressions pass | | C19 | Candidate inference and tidy preview with explicit correction/conflicting scope | No live mutation; suggestion retains provenance; no overwrite, activation or cross-scope merge | | C20 | Forget/correct/revoke parent while preview or derived processing is delayed | Descendants invalidated; replay/undo cannot restore withdrawn evidence; P0 behavior unchanged | From 4f5e8ef15cf8e888b75b83fa7d17f06178aed2ed Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 10:05:04 -0400 Subject: [PATCH 05/15] docs: enforce both memory validity bounds at authority boundaries --- docs/design/bot-memory-integration.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index 9a487db..ecb1f70 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -56,7 +56,7 @@ All API names below are proposed. No listed method or acceptance result is imple One projection key is `(owner_type, owner_id, bot_id, external_memory_id)`. Scope is immutable for that key; changing scope withdraws the old memory and creates a new authoritative identity. Two different external IDs remain separate even when their text is identical. Revisions are positive monotonic integers supplied by MemoryManager for this one key. -Upsert accepts a typed envelope containing that key, `revision`, active state, `scope` (type and canonical entity ID where applicable), `kind`, bounded text, source_refs, and optional expiry. Store the envelope in existing metadata/source fields where possible. Internal UUID allocation/mapping is stable and durable; client display names, email labels, or text similarity do not determine identity. Data always remains Private and sync-disabled in this store; no privacy flag supplied by a client can enable sharing. +Upsert accepts a typed envelope containing that key, `revision`, active state, `scope` (type and canonical entity ID where applicable), `kind`, bounded text, source_refs, and optional validity bounds (`valid_from`, `valid_until`). Store the envelope in existing metadata/source fields where possible. Internal UUID allocation/mapping is stable and durable; client display names, email labels, or text similarity do not determine identity. Data always remains Private and sync-disabled in this store; no privacy flag supplied by a client can enable sharing. Delete accepts the same key, canonical scope and withdrawal revision, without the old text. An unseen-key deletion must also provide that scope from authoritative state; missing scope is rejected, not replaced by a wildcard. The managed scope is supplied by the trusted adapter, never chosen by a browser or inferred by a model. Core validates type/size, required identity, scope consistency, and supported fields; execution-relevant unknown fields are rejected. Contract tests must establish concrete input limits before implementation is considered ready; reuse existing limits where suitable rather than inventing unlimited payloads. @@ -91,11 +91,11 @@ Physical isolation is only the first boundary. Within the owner/bot store, each `search(query, allowed_scopes, limit, budget)` returns structured hits with external identity, revision, scope, source_refs, score, and indexed text, plus explicit `degraded/truncated` information. These are internal candidates, not browser-safe responses. Enforce bounded query/result sizes and retain QueryBudget; namespace is a secondary partition, not authorization proof. -MemoryManager first resolves required defaults by exact key from current PostgreSQL state with the existing scope/precedence rules, independently of Cortex readiness, hit count, or index lag. Current explicit instructions and work constraints still override defaults. Optional Cortex retrieval supplements this mandatory lookup; it cannot replace it. In particular, an execution action depending on a just-committed memory action must see that default even if no replica has indexed it yet and Cortex returns a full page of unrelated valid hits. +MemoryManager first resolves required defaults by exact key from current PostgreSQL state with the existing scope/precedence and validity rules, independently of Cortex readiness, hit count, or index lag. Current explicit instructions and work constraints still override defaults. Optional Cortex retrieval supplements this mandatory lookup; it cannot replace it. In particular, an execution action depending on a just-committed memory action must see that default even if no replica has indexed it yet and Cortex returns a full page of unrelated valid hits. -The adapter batch-loads current PostgreSQL memories and checks owner/bot/scope, active status, expiry, source access/withdrawal, and revision before exposing anything to UI, logs, or ContextProjector. Rejected candidates cannot enter excerpts, telemetry, summaries, or explanations. Emit current PostgreSQL text and source metadata after validation, not stale indexed text. Candidate shortage/timeout is not proof that no memory exists: fall back to the existing bounded PostgreSQL retrieval and report optional-context degradation. Required execution facts continue to come from the current work or their authoritative service. +The adapter batch-loads current PostgreSQL memories and checks owner/bot/scope, active status, both validity bounds, source access/withdrawal, and revision before exposing anything to UI, logs, or ContextProjector. For both required exact-key defaults and optional hits, the authoritative server time must satisfy `valid_from <= now < valid_until`; a missing bound is unbounded on that side. Use timezone-aware timestamps, not browser/model clocks. Active status alone does not make a future-effective memory usable. Rejected candidates cannot enter excerpts, telemetry, summaries, or explanations. Emit current PostgreSQL text and source metadata after validation, not stale indexed text. Candidate shortage/timeout is not proof that no memory exists: fall back to the existing bounded PostgreSQL retrieval and report optional-context degradation. Required execution facts continue to come from the current work or their authoritative service. -At model-call/action boundaries, the existing Bot rules still invalidate withdrawn memory manifests, summaries, and checkpoints. Cortex does not claim to revoke content already sent to a model. Revalidate at the same boundary as the existing permission checks; no new distributed transaction with a model supplier is claimed. +At model-call/action boundaries, the existing Bot rules still invalidate withdrawn memory manifests, summaries, and checkpoints. Recheck both validity bounds for defaults and optional memories at these boundaries, including a memory that expires while waiting. Cortex does not claim to revoke content already sent to a model. Revalidate at the same boundary as the existing permission checks; no new distributed transaction with a model supplier is claimed. ### 4.5 Forget, rebuild, and maintenance @@ -162,7 +162,7 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr | C12 | Existing personal core/MCP/HTTP/sync/gateway regressions | Legacy calls and opt-in behavior preserved; controlled data cannot be exported or synced through legacy entrypoints | | C13 | Native call under host concurrency, timeout/cancellation, embedding unavailable, encrypted-storage policy | Event loop stays responsive; bounded work; no publication gap; offline mode usable; required at-rest controls verified | | C14 | Real test PG commit plus lost event/ack, restart, two replicas at different cursors, retention expiry; PG cursor ahead of restored local data | Local durable position precedes remote ack; each replica reconciles independently, rewinds/rebuilds on mismatch; stale replicas only return authority-approved results or fallback | -| C15 | Correct/forget/revoke/expire after indexing or during search, including rejected source access | Current PG gate removes invalid hits and uses authoritative text; model-call boundary checks hold; no rejected content reaches logs/UI/context | +| C15 | Correct/forget/revoke/expire after indexing or during search; future-effective active defaults/hits before/at activation and at expiry; rejected source access | Current PG gate enforces both validity bounds for exact-key defaults and optional hits, including model-call/action boundaries; authoritative text only, no rejected content reaches logs/UI/context | | C16 | Rebuild from empty with concurrent changes, permission revocation, stale delivery, kill/swap, restore backup | New snapshot/replay generation valid; no resurrection/leak; half-built or obsolete generation never becomes ready | | C17 | Core unavailable, flag off, slow/short search, invalid authority lookup; just-confirmed required default missing from delayed replica that returns a full page of other valid hits | Mandatory exact-key PG defaults always precede optional search; dependent action uses the committed default; fallback is truthful, writes retained, invalid permissions fail closed | | C18 | Host installed package, feature toggle, worker restart, rollback, new-work recall and forget | Verify actual package/config/process and end-to-end behavior; upstream M01-M06/N01-N03/L08 and existing work/approval regressions pass | From e82eded2cb0127abccf686dcec64c17fa006045b Mon Sep 17 00:00:00 2001 From: gambletan Date: Sun, 11 Oct 2026 09:14:00 -0400 Subject: [PATCH 06/15] fix: make the local memory database owner-only (category: privacy) The local store is plaintext SQLite holding every memory, Private included, but it was created under the process umask: memory.db, -wal and -shm were 0644 and ~/.cortex 0755. Any other local account that can traverse the home directory (macOS staff group with a 0750 home, Linux 0755 homes, custom DB paths) could read every memory. - SqliteStorage creates a new DB file as 0600 before SQLite opens it; SQLite gives -wal/-shm the main file's mode. - Existing DB/-wal/-shm/-journal files with group/other bits are tightened on open (warn, not fail, if chmod is refused). - cortex-mcp-server and cortex-http create the DB directory as 0700. Fixes self-evolution iteration 47. Co-Authored-By: Claude Opus 5.5 --- cortex-core/src/storage/sqlite.rs | 32 ++++++++++ cortex-core/tests/test_db_file_permissions.rs | 59 +++++++++++++++++++ cortex-http/src/main.rs | 6 +- cortex-mcp-server/src/main.rs | 7 ++- 4 files changed, 102 insertions(+), 2 deletions(-) create mode 100644 cortex-core/tests/test_db_file_permissions.rs diff --git a/cortex-core/src/storage/sqlite.rs b/cortex-core/src/storage/sqlite.rs index 063d0cc..3e42769 100644 --- a/cortex-core/src/storage/sqlite.rs +++ b/cortex-core/src/storage/sqlite.rs @@ -127,7 +127,39 @@ impl SqliteStorage { Ok(()) } + /// The database holds every memory (Private included), usually unencrypted, so it + /// must be owner-only. Creates a new file as 0600 — SQLite gives `-wal`/`-shm` the + /// main file's mode — and tightens files left group/world-readable by older releases. + #[cfg(unix)] + fn restrict_db_file_mode(path: &str) -> Result<(), CortexError> { + use std::os::unix::fs::{OpenOptionsExt, PermissionsExt}; + if path.is_empty() || path == ":memory:" || path.starts_with("file:") { + return Ok(()); + } + std::fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(false) + .mode(0o600) + .open(path) + .map_err(|e| CortexError::Storage(e.to_string()))?; + for suffix in ["", "-wal", "-shm", "-journal"] { + let p = format!("{path}{suffix}"); + let Ok(meta) = std::fs::metadata(&p) else { continue }; + let mode = meta.permissions().mode(); + if mode & 0o077 != 0 { + // Not fatal (e.g. a file owned by someone else), but never silent. + if let Err(e) = std::fs::set_permissions(&p, std::fs::Permissions::from_mode(mode & 0o700)) { + tracing::warn!("could not restrict permissions on {p}: {e}"); + } + } + } + Ok(()) + } + fn open_keyed(path: &str, key: Option, readers: usize) -> Result { + #[cfg(unix)] + Self::restrict_db_file_mode(path)?; let conn = Connection::open(path).map_err(|e| CortexError::Storage(e.to_string()))?; Self::apply_key(&conn, key.as_ref())?; diff --git a/cortex-core/tests/test_db_file_permissions.rs b/cortex-core/tests/test_db_file_permissions.rs new file mode 100644 index 0000000..a67aa09 --- /dev/null +++ b/cortex-core/tests/test_db_file_permissions.rs @@ -0,0 +1,59 @@ +//! The local database is plaintext SQLite holding every memory, Private included. +//! It must be readable by its owner only — never by other local accounts. +#![cfg(unix)] + +use std::os::unix::fs::PermissionsExt; +use std::path::Path; + +use cortex_core::Cortex; + +fn mode(p: &Path) -> u32 { + std::fs::metadata(p).unwrap().permissions().mode() & 0o777 +} + +fn assert_owner_only(p: &Path) { + let m = mode(p); + assert_eq!(m & 0o077, 0, "{} is {:o}; group/other must have no access", p.display(), m); +} + +#[test] +fn new_database_and_wal_files_are_owner_only() { + let dir = tempfile::tempdir().unwrap(); + let db = dir.path().join("memory.db"); + let cortex = Cortex::open(db.to_str().unwrap()).unwrap(); + cortex.ingest("my private diagnosis", "test", None, None, None).unwrap(); + + assert_owner_only(&db); + for suffix in ["-wal", "-shm"] { + let side = dir.path().join(format!("memory.db{suffix}")); + if side.exists() { + assert_owner_only(&side); + } + } +} + +#[test] +fn existing_world_readable_database_is_tightened_on_open() { + let dir = tempfile::tempdir().unwrap(); + let db = dir.path().join("memory.db"); + { + let cortex = Cortex::open(db.to_str().unwrap()).unwrap(); + cortex.ingest("written by an older version", "test", None, None, None).unwrap(); + } + // Simulate a database created by an older release under a 022 umask. + for name in ["memory.db", "memory.db-wal", "memory.db-shm"] { + let p = dir.path().join(name); + if p.exists() { + std::fs::set_permissions(&p, std::fs::Permissions::from_mode(0o644)).unwrap(); + } + } + + let _cortex = Cortex::open(db.to_str().unwrap()).unwrap(); + + for name in ["memory.db", "memory.db-wal", "memory.db-shm"] { + let p = dir.path().join(name); + if p.exists() { + assert_owner_only(&p); + } + } +} diff --git a/cortex-http/src/main.rs b/cortex-http/src/main.rs index 82f222f..a1b45f9 100644 --- a/cortex-http/src/main.rs +++ b/cortex-http/src/main.rs @@ -59,7 +59,11 @@ async fn main() { }); if let Some(parent) = std::path::Path::new(&db_path).parent() { - let _ = std::fs::create_dir_all(parent); + let mut builder = std::fs::DirBuilder::new(); + builder.recursive(true); + #[cfg(unix)] + std::os::unix::fs::DirBuilderExt::mode(&mut builder, 0o700); + let _ = builder.create(parent); } info!(db = %db_path, "opening cortex database"); diff --git a/cortex-mcp-server/src/main.rs b/cortex-mcp-server/src/main.rs index fa6f403..d1257dc 100644 --- a/cortex-mcp-server/src/main.rs +++ b/cortex-mcp-server/src/main.rs @@ -265,7 +265,12 @@ fn resolve_db_path(cli_path: Option<&str>) -> String { fn ensure_db_dir(db_path: &str) { if let Some(parent) = std::path::Path::new(db_path).parent() { - let _ = std::fs::create_dir_all(parent); + // Owner-only: the directory holds the (usually unencrypted) memory database. + let mut builder = std::fs::DirBuilder::new(); + builder.recursive(true); + #[cfg(unix)] + std::os::unix::fs::DirBuilderExt::mode(&mut builder, 0o700); + let _ = builder.create(parent); } } From 2a3f6d2e9611097488030a5927e6341c473b64a5 Mon Sep 17 00:00:00 2001 From: gambletan Date: Sat, 17 Oct 2026 09:16:04 -0400 Subject: [PATCH 07/15] fix: make memory deletion erase the plaintext from disk (category: privacy) delete_memory is documented as permanent, but on the plaintext database the deleted text stayed recoverable: SQLite's default fast secure_delete left the row in freed pages, FTS5 kept the lowercased terms in index segments, and the WAL kept the original frames until a later checkpoint. Any backup/copy of memory.db recovered it. - write connection sets PRAGMA secure_delete=ON - FTS5 'secure-delete' enabled where supported (SQLite 3.42+); otherwise the FTS index is optimized after a delete so deleted terms are dropped - after every delete path (single, batch, sync-merge) the WAL is checkpointed and truncated, best effort - test greps the raw .db and -wal bytes for the deleted marker; each of the three steps is individually required for it to pass Fixes self-evolution iteration 50. Co-Authored-By: Claude Opus 5.5 --- cortex-core/src/storage/sqlite.rs | 47 +++++++++++++++++++++- cortex-core/tests/test_secure_delete.rs | 52 +++++++++++++++++++++++++ 2 files changed, 98 insertions(+), 1 deletion(-) create mode 100644 cortex-core/tests/test_secure_delete.rs diff --git a/cortex-core/src/storage/sqlite.rs b/cortex-core/src/storage/sqlite.rs index 3e42769..ce3e03c 100644 --- a/cortex-core/src/storage/sqlite.rs +++ b/cortex-core/src/storage/sqlite.rs @@ -171,7 +171,8 @@ impl SqliteStorage { PRAGMA temp_store=MEMORY; PRAGMA foreign_keys=ON; PRAGMA wal_autocheckpoint=1000; - PRAGMA busy_timeout=5000;", + PRAGMA busy_timeout=5000; + PRAGMA secure_delete=ON;", ) .map_err(|e| CortexError::Storage(e.to_string()))?; @@ -201,9 +202,48 @@ impl SqliteStorage { )), }; storage.init()?; + // Deletes are permanent: FTS5 otherwise keeps deleted terms in index segments + // until a merge (secure_delete only scrubs freed b-tree pages). Needs SQLite + // 3.42+; on older builds scrub_after_delete falls back to an index rebuild. + let _ = storage + .write_conn + .lock() + .execute_batch("INSERT INTO memories_fts(memories_fts, rank) VALUES('secure-delete', 1);"); Ok(storage) } + /// Make a committed delete permanent on disk: purge the deleted terms from the FTS + /// index, then fold the WAL back into the (zero-scrubbed) database and truncate it so + /// the plaintext does not linger in old WAL frames. Best effort: a reader holding a + /// snapshot can block the checkpoint; the next delete or autocheckpoint retries. + fn scrub_after_delete(conn: &Connection) { + let fts_secure_delete = conn + .query_row( + "SELECT v FROM memories_fts_config WHERE k = 'secure-delete'", + [], + |r| r.get::<_, i64>(0), + ) + .is_ok_and(|v| v != 0); + if !fts_secure_delete { + if let Err(e) = + conn.execute_batch("INSERT INTO memories_fts(memories_fts) VALUES('optimize');") + { + tracing::warn!(error = %e, "FTS scrub after delete failed"); + } + } + for attempt in 0..3 { + match conn.query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |r| r.get::<_, i64>(0)) { + Ok(0) => return, + Ok(_) => std::thread::sleep(std::time::Duration::from_millis(10 * (attempt + 1))), + Err(e) => { + tracing::warn!(error = %e, "WAL scrub after delete failed"); + return; + } + } + } + tracing::warn!("WAL scrub after delete skipped: database busy"); + } + pub fn open_in_memory() -> Result { let conn = Connection::open_in_memory().map_err(|e| CortexError::Storage(e.to_string()))?; @@ -801,8 +841,10 @@ impl StorageBackend for SqliteStorage { .map_err(|e| CortexError::Storage(e.to_string()))?; stmt.execute(params![id_str]) .map_err(|e| CortexError::Storage(e.to_string()))?; + drop(stmt); // FTS5 synced via trigger (memories_fts_ad) + Self::scrub_after_delete(&conn); self.cache_evict(&id); Ok(()) } @@ -1683,6 +1725,9 @@ impl StorageBackend for SqliteStorage { conn.execute_batch("COMMIT;") .map_err(|e| CortexError::Storage(e.to_string()))?; + if count > 0 { + Self::scrub_after_delete(&conn); + } // Evict deleted entries from cache { let mut cache = self.mem_cache.lock(); diff --git a/cortex-core/tests/test_secure_delete.rs b/cortex-core/tests/test_secure_delete.rs new file mode 100644 index 0000000..35f8a8b --- /dev/null +++ b/cortex-core/tests/test_secure_delete.rs @@ -0,0 +1,52 @@ +//! `delete_memory` is documented as permanent. On a plaintext database the deleted text +//! must not survive in the database file's free pages, the FTS index, or the WAL — +//! otherwise any later copy of the file (backup, forensics) still recovers it. + +use std::path::Path; + +use cortex_core::Cortex; + +const MARKER: &str = "ZQXDELMARKER"; + +fn file_contains(p: &Path, needle: &[u8]) -> bool { + match std::fs::read(p) { + Ok(bytes) => bytes.windows(needle.len()).any(|w| w == needle), + Err(_) => false, + } +} + +fn assert_marker_gone(dir: &Path) { + for name in ["memory.db", "memory.db-wal"] { + let p = dir.join(name); + // The FTS index stores lowercased tokens. + let lower = MARKER.to_lowercase(); + assert!( + !file_contains(&p, MARKER.as_bytes()) && !file_contains(&p, lower.as_bytes()), + "deleted text still recoverable from {}", + p.display() + ); + } +} + +#[test] +fn deleted_memory_text_is_not_recoverable_from_disk() { + let dir = tempfile::tempdir().unwrap(); + let db = dir.path().join("memory.db"); + let cortex = Cortex::open(db.to_str().unwrap()).unwrap(); + cortex.ingest("the weather is mild today", "test", None, None, None).unwrap(); + let secret = cortex + .ingest(&format!("{MARKER} my medical diagnosis is confidential"), "test", None, None, None) + .unwrap(); + cortex.ingest("lunch was a sandwich", "test", None, None, None).unwrap(); + + let on_disk = |n: &str| file_contains(&dir.path().join(n), MARKER.as_bytes()); + assert!(on_disk("memory.db") || on_disk("memory.db-wal"), "precondition: text is on disk"); + + cortex.delete_memory(secret.id).unwrap(); + + // Checked while the database is still open (the WAL is live) … + assert_marker_gone(dir.path()); + drop(cortex); + // … and after close. + assert_marker_gone(dir.path()); +} From a61719824ba1d8c2d16fed3ce3b4b4253464d1e8 Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 11:46:01 -0400 Subject: [PATCH 08/15] feat: add controlled Bot memory projections and Python API --- .github/workflows/ci.yml | 28 + Cargo.lock | 6 +- README.md | 11 +- cortex-core/Cargo.toml | 5 +- cortex-core/src/controlled.rs | 785 +++++++++++++++++++ cortex-core/src/lib.rs | 1 + cortex-core/src/retrieval.rs | 11 +- cortex-core/src/storage/sqlite.rs | 125 ++- cortex-core/tests/test_sync.rs | 9 +- cortex-python/Cargo.toml | 5 +- cortex-python/README.md | 17 +- cortex-python/pyproject.toml | 4 +- cortex-python/src/lib.rs | 217 ++++- cortex-python/tests/test_controlled_store.py | 507 ++++++++++++ docs/ROADMAP.md | 8 +- docs/controlled-memory-api.md | 80 ++ docs/design/bot-memory-integration.md | 18 +- docs/guides/integrations.md | 35 +- 18 files changed, 1776 insertions(+), 96 deletions(-) create mode 100644 cortex-core/src/controlled.rs create mode 100644 cortex-python/tests/test_controlled_store.py create mode 100644 docs/controlled-memory-api.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 699820a..4d47338 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -42,6 +42,34 @@ jobs: - name: Assert cortex-core default build has no network/telemetry deps run: bash scripts/check-no-network-egress.sh + controlled-wheel: + name: Controlled wheel (${{ matrix.os }}) + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, macos-latest] + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - uses: Swatinem/rust-cache@v2 + - name: Build native wheel + run: | + python -m pip install 'maturin>=1.0,<2.0' + maturin build --release -m cortex-python/Cargo.toml --out dist --interpreter python + - name: Install wheel and run isolated acceptance + run: | + python -m pip install --force-reinstall dist/*.whl + python -I "$GITHUB_WORKSPACE/cortex-python/tests/test_controlled_store.py" + - uses: actions/upload-artifact@v4 + with: + name: controlled-wheel-${{ matrix.os }} + path: dist/*.whl + retention-days: 14 + bench: name: Benchmark runs-on: ubuntu-latest diff --git a/Cargo.lock b/Cargo.lock index 92f53d4..f6fb5ab 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -613,6 +613,7 @@ dependencies = [ "half", "hmac", "instant-distance", + "libc", "lru", "notify", "parking_lot", @@ -678,10 +679,13 @@ dependencies = [ [[package]] name = "cortex-python" -version = "2.4.0" +version = "2.5.0" dependencies = [ "cortex-core", + "parking_lot", "pyo3", + "serde", + "serde_json", "uuid", ] diff --git a/README.md b/README.md index aad3b23..740fc06 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ Cortex implements a 4-tier memory model inspired by human cognition: **Working** holds the current session scratch pad. **Episodic** stores raw experiences with timestamps and source metadata. The **Consolidation Engine** periodically promotes recurring patterns into **Semantic** facts and decays stale episodes. **Procedural** captures learned workflows and routines. -### Bot integration architecture (proposed) +### Bot integration architecture For a hosted Bot with existing PostgreSQL memory and account permissions, Cortex is an optional derived retrieval backend behind its MemoryManager. PostgreSQL remains the @@ -147,10 +147,11 @@ and approvals. Search hits must pass current authoritative validation before ent model context. A controlled projection store must not import personal archives or enable sync, automatic inference or consolidation by default. -The [Bot memory integration plan](docs/design/bot-memory-integration.md) defines the -proposed controlled-store contract, Cortex versus host responsibilities, implementation -order, failure recovery and acceptance. It is a design, not an available SDK API or a -deployed Bot integration; existing personal-agent integration paths below are unchanged. +The [controlled API contract](docs/controlled-memory-api.md) documents the implemented +`ControlledStore` Python interface, snapshot/replay lifecycle and error codes. The +[implementation plan](docs/design/bot-memory-integration.md) separates Cortex-side +acceptance from host-owned PostgreSQL and Bot integration. Package publication and Bot +deployment are separate steps; existing personal-agent interfaces remain available. ## Key Components diff --git a/cortex-core/Cargo.toml b/cortex-core/Cargo.toml index 6c44636..2f1445d 100644 --- a/cortex-core/Cargo.toml +++ b/cortex-core/Cargo.toml @@ -14,7 +14,7 @@ uuid = { version = "1", features = ["v4", "serde"] } chrono = { version = "0.4", features = ["serde"] } serde = { version = "1", features = ["derive"] } serde_json = "1" -rusqlite = { version = "0.31", features = ["bundled"] } +rusqlite = { version = "0.31", features = ["bundled", "hooks"] } thiserror = "1" tracing = "0.1" parking_lot = "0.12" @@ -47,3 +47,6 @@ harness = false [[bench]] name = "comparison_bench" harness = false + +[target.'cfg(unix)'.dependencies] +libc = "0.2" diff --git a/cortex-core/src/controlled.rs b/cortex-core/src/controlled.rs new file mode 100644 index 0000000..4d490cb --- /dev/null +++ b/cortex-core/src/controlled.rs @@ -0,0 +1,785 @@ +//! A disposable, scope-constrained projection of host-authoritative memory. +//! Never invokes ingest, plugins, inference, sync, or consolidation. +use crate::retrieval::{QueryBudget, RetrievalEngine, RetrievalQuery}; +use crate::storage::{memory_index::MemoryIndex, sqlite::SqliteStorage}; +use crate::types::{MemContent, MemObject, MemObjectBuilder, MemSource, MemoryTier}; +use chrono::{DateTime, Utc}; +use parking_lot::Mutex; +use rusqlite::{params, params_from_iter, OptionalExtension}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use std::collections::{HashMap, HashSet}; +use std::fs::File; +use std::path::{Component, Path, PathBuf}; +use std::sync::{Arc, OnceLock}; +use std::time::{Duration, Instant}; +use uuid::Uuid; + +#[derive(Debug, thiserror::Error)] +pub enum ControlledError { + #[error("invalid controlled input: {0}")] + InvalidInput(&'static str), + #[error("controlled state conflict: {0}")] + Conflict(&'static str), + #[error("replica is already open")] + Busy, + #[error("replica needs authoritative reconciliation")] + NotReady, + #[error("controlled store is closed")] + Closed, + #[error("controlled storage failure")] + Storage, +} +impl ControlledError { + pub fn code(&self) -> &'static str { + match self { + Self::InvalidInput(_) => "invalid_input", + Self::Conflict(_) => "conflict", + Self::Busy => "busy", + Self::NotReady => "not_ready", + Self::Closed => "closed", + Self::Storage => "storage", + } + } +} +impl From for ControlledError { + fn from(_: rusqlite::Error) -> Self { + Self::Storage + } +} +impl From for ControlledError { + fn from(_: crate::CortexError) -> Self { + Self::Storage + } +} +type Fence = (String, i64, Option, bool, Option); +type Result = std::result::Result; + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct StoreIdentity { + pub owner_type: String, + pub owner_id: String, + pub bot_id: String, + pub replica_id: String, +} +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq, Hash)] +#[serde(deny_unknown_fields)] +pub struct Scope { + pub kind: String, + pub id: String, +} +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)] +#[serde(deny_unknown_fields)] +pub struct SourceRef { + pub kind: String, + pub provider: String, + pub id: String, +} +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Projection { + pub memory_id: String, + pub revision: i64, + pub scope: Scope, + pub kind: String, + pub text: String, + pub source_refs: Vec, + #[serde(default)] + pub valid_from: Option>, + #[serde(default)] + pub valid_until: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub embedding: Option>, +} +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Deletion { + pub memory_id: String, + pub revision: i64, + pub scope: Scope, +} +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(tag = "op", rename_all = "snake_case", deny_unknown_fields)] +pub enum Mutation { + Upsert { projection: Projection }, + Delete { deletion: Deletion }, +} +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct Outcome { + pub memory_id: String, + pub revision: i64, + pub state: String, + pub outcome: String, +} +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct Receipt { + pub generation: String, + pub position: i64, + pub durable: bool, + pub outcomes: Vec, +} +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct Status { + pub generation: String, + pub position: i64, + pub sealed: bool, + pub ready: bool, +} +#[derive(Clone, Debug, Serialize)] +pub struct Hit { + pub owner_type: String, + pub owner_id: String, + pub bot_id: String, + #[serde(flatten)] + pub projection: Projection, + pub score: f32, +} +#[derive(Clone, Debug, Serialize)] +pub struct SearchResult { + pub hits: Vec, + pub degraded: bool, + pub truncated: bool, +} + +/// Host-supplied bounds; no implicit embedding model is loaded. +#[derive(Clone, Debug)] +pub struct SearchOptions { + pub limit: usize, + pub embedding: Option>, + pub max_candidates: usize, + pub max_duration_ms: u64, +} +impl Default for SearchOptions { + fn default() -> Self { + Self { + limit: 10, + embedding: None, + max_candidates: 1000, + max_duration_ms: 250, + } + } +} + +struct State { + storage: SqliteStorage, + ready: bool, + closed: bool, +} +static OPEN: OnceLock>> = OnceLock::new(); +struct Registry(PathBuf); +impl Drop for Registry { + fn drop(&mut self) { + OPEN.get().unwrap().lock().remove(&self.0); + } +} +/// Locks are dropped only after SQLite state and all calls have drained. +pub struct ControlledStore { + state: Mutex, + identity: StoreIdentity, + _file: File, + _registry: Registry, +} + +fn ident(s: &str) -> Result<()> { + if s.is_empty() || s.len() > 256 || s.chars().any(char::is_control) { + return Err(ControlledError::InvalidInput("identifier")); + } + Ok(()) +} +fn json(x: &T) -> Result { + serde_json::to_string(x).map_err(|_| ControlledError::InvalidInput("serialization")) +} +fn digest(x: &T) -> Result { + Ok(format!("{:x}", Sha256::digest(json(x)?.as_bytes()))) +} +fn decode(s: &str) -> Result { + serde_json::from_str(s).map_err(|_| ControlledError::Storage) +} +fn scope_check(scope: &Scope, owner: &StoreIdentity) -> Result<()> { + ident(&scope.id)?; + if !["owner", "entity", "project", "workspace", "user"].contains(&scope.kind.as_str()) + || (scope.kind == "owner" && scope.id != owner.owner_id) + { + return Err(ControlledError::InvalidInput("scope")); + } + Ok(()) +} +fn vector(v: &Option>) -> Result<()> { + if let Some(v) = v { + if v.is_empty() + || v.len() > 4096 + || v.iter().any(|x| !x.is_finite()) + || !v.iter().map(|x| x * x).sum::().is_finite() + { + return Err(ControlledError::InvalidInput("embedding")); + } + } + Ok(()) +} +fn normalize(batch: &mut [Mutation], owner: &StoreIdentity) -> Result<()> { + if batch.len() > 100 { + return Err(ControlledError::InvalidInput("batch limit")); + } + let mut seen = HashSet::new(); + for m in batch { + let (id, rev, scope) = match m { + Mutation::Upsert { projection: p } => { + if !["fact", "preference", "experience", "method"].contains(&p.kind.as_str()) + || p.text.len() > 48000 + || p.text.chars().count() > 12000 + { + return Err(ControlledError::InvalidInput("projection")); + } + if p.source_refs.is_empty() || p.source_refs.len() > 16 { + return Err(ControlledError::InvalidInput("sources")); + } + for r in &p.source_refs { + ident(&r.kind)?; + ident(&r.provider)?; + ident(&r.id)?; + } + p.source_refs.sort(); + p.source_refs.dedup(); + vector(&p.embedding)?; + if let (Some(a), Some(b)) = (p.valid_from, p.valid_until) { + if a >= b { + return Err(ControlledError::InvalidInput("validity")); + } + } + (&p.memory_id, p.revision, &p.scope) + } + Mutation::Delete { deletion: d } => (&d.memory_id, d.revision, &d.scope), + }; + ident(id)?; + if rev <= 0 { + return Err(ControlledError::InvalidInput("revision")); + } + scope_check(scope, owner)?; + if !seen.insert(id.clone()) { + return Err(ControlledError::InvalidInput("duplicate batch identity")); + } + } + Ok(()) +} + +#[cfg(unix)] +fn private_file(path: &Path) -> Result { + use std::os::unix::fs::{MetadataExt, OpenOptionsExt, PermissionsExt}; + let f = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .mode(0o600) + .custom_flags(libc::O_NOFOLLOW) + .open(path) + .map_err(|_| ControlledError::InvalidInput("unsafe store file"))?; + let m = f.metadata().map_err(|_| ControlledError::Storage)?; + if !m.is_file() || m.nlink() != 1 || m.uid() != unsafe { libc::geteuid() } { + return Err(ControlledError::InvalidInput("unsafe store file")); + } + f.set_permissions(std::fs::Permissions::from_mode(0o600)) + .map_err(|_| ControlledError::Storage)?; + Ok(f) +} +#[cfg(not(unix))] +fn private_file(_: &Path) -> Result { + Err(ControlledError::InvalidInput( + "controlled storage currently requires Unix permissions", + )) +} + +impl ControlledStore { + pub fn open(root: &Path, identity: StoreIdentity) -> Result { + for x in [ + &identity.owner_type, + &identity.owner_id, + &identity.bot_id, + &identity.replica_id, + ] { + ident(x)?; + } + if !root.is_absolute() || root.components().any(|c| matches!(c, Component::ParentDir)) { + return Err(ControlledError::InvalidInput( + "absolute private root required", + )); + } + let mut prefix = PathBuf::new(); + for c in root.components() { + prefix.push(c); + if let Ok(m) = std::fs::symlink_metadata(&prefix) { + if m.file_type().is_symlink() { + return Err(ControlledError::InvalidInput("symlink root")); + } + } + } + let mut builder = std::fs::DirBuilder::new(); + builder.recursive(true); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt; + builder.mode(0o700); + } + builder.create(root).map_err(|_| ControlledError::Storage)?; + #[cfg(unix)] + { + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + let m = std::fs::metadata(root).map_err(|_| ControlledError::Storage)?; + if m.uid() != unsafe { libc::geteuid() } || m.permissions().mode() & 0o077 != 0 { + return Err(ControlledError::InvalidInput("root must be owner-only")); + } + } + let path = root.join(format!("{}.db", digest(&identity)?)); + let registry = OPEN.get_or_init(|| Mutex::new(HashSet::new())); + if !registry.lock().insert(path.clone()) { + return Err(ControlledError::Busy); + } + let registry = Registry(path.clone()); + let file = private_file(&path.with_extension("lock"))?; + file.try_lock().map_err(|_| ControlledError::Busy)?; + for suffix in ["", "-wal", "-shm", "-journal"] { + let p = PathBuf::from(format!("{}{suffix}", path.display())); + if suffix.is_empty() || p.exists() || std::fs::symlink_metadata(&p).is_ok() { + drop(private_file(&p)?); + } + } + let storage = SqliteStorage::open_controlled( + path.to_str() + .ok_or(ControlledError::InvalidInput("path encoding"))?, + )?; + storage.with_controlled_conn(|conn| -> Result<()> { + conn.execute_batch("CREATE TABLE IF NOT EXISTS controlled_meta(id INTEGER PRIMARY KEY CHECK(id=1), identity TEXT NOT NULL, generation TEXT NOT NULL, position INTEGER NOT NULL DEFAULT 0, sealed INTEGER NOT NULL DEFAULT 0, snapshot_position INTEGER, last_batch TEXT, last_receipt TEXT, dimension INTEGER); + CREATE TABLE IF NOT EXISTS controlled_fences(memory_id TEXT PRIMARY KEY, scope TEXT NOT NULL, revision INTEGER NOT NULL, internal_id TEXT, live INTEGER NOT NULL, intent TEXT, valid_from INTEGER, valid_until INTEGER); + CREATE INDEX IF NOT EXISTS controlled_scope ON controlled_fences(scope,live,internal_id);")?; + conn.execute("INSERT OR IGNORE INTO controlled_meta(id,identity,generation) VALUES(1,?1,?2)",params![json(&identity)?,Uuid::new_v4().to_string()])?; + let stored:String=conn.query_row("SELECT identity FROM controlled_meta WHERE id=1",[],|r|r.get(0))?; + if decode::(&stored)?!=identity { return Err(ControlledError::Conflict("store identity")); } + let check:String=conn.query_row("PRAGMA quick_check",[],|r|r.get(0))?; + if check!="ok" { return Err(ControlledError::Storage); } + Ok(()) + })?; + Ok(Self { + state: Mutex::new(State { + storage, + ready: false, + closed: false, + }), + identity, + _file: file, + _registry: registry, + }) + } + fn status_locked(s: &State) -> Result { + if s.closed { + return Err(ControlledError::Closed); + } + s.storage.with_controlled_conn(|c| { + Ok(c.query_row( + "SELECT generation,position,sealed FROM controlled_meta WHERE id=1", + [], + |r| { + Ok(Status { + generation: r.get(0)?, + position: r.get(1)?, + sealed: r.get(2)?, + ready: s.ready, + }) + }, + )?) + }) + } + pub fn status(&self) -> Result { + Self::status_locked(&self.state.lock()) + } + pub fn close(&self) { + let mut s = self.state.lock(); + s.ready = false; + s.closed = true; + } + pub fn reconcile(&self, generation: &str, position: i64) -> Result { + let mut s = self.state.lock(); + s.ready = false; + let v = Self::status_locked(&s)?; + if !v.sealed { + return Err(ControlledError::NotReady); + } + if position < 0 || v.generation != generation || v.position != position { + return Err(ControlledError::Conflict( + "reconciliation position or generation", + )); + } + s.ready = true; + Self::status_locked(&s) + } + pub fn finish_snapshot(&self, position: i64) -> Result { + if position < 0 { + return Err(ControlledError::InvalidInput("position")); + } + let mut s = self.state.lock(); + let v = Self::status_locked(&s)?; + if v.sealed { + let original: i64 = s.storage.with_controlled_conn(|c| -> Result { + Ok(c.query_row( + "SELECT snapshot_position FROM controlled_meta WHERE id=1", + [], + |r| r.get(0), + )?) + })?; + if original == position { + return Ok(v); + } + return Err(ControlledError::Conflict("snapshot already sealed")); + } + let r = s.storage.with_controlled_conn(|c| -> Result<()> { + c.execute( + "UPDATE controlled_meta SET position=?1,snapshot_position=?1,sealed=1 WHERE id=1", + [position], + )?; + Ok(()) + }); + if r.is_err() { + s.ready = false; + } + r?; + Self::status_locked(&s) + } + pub fn seed(&self, projections: Vec) -> Result { + self.mutate( + projections + .into_iter() + .map(|projection| Mutation::Upsert { projection }) + .collect(), + None, + ) + } + pub fn apply_batch(&self, mutations: Vec, position: i64) -> Result { + if position < 0 { + return Err(ControlledError::InvalidInput("position")); + } + self.mutate(mutations, Some(position)) + } + pub fn upsert(&self, projection: Projection, position: i64) -> Result { + self.apply_batch(vec![Mutation::Upsert { projection }], position) + } + pub fn delete(&self, deletion: Deletion, position: i64) -> Result { + self.apply_batch(vec![Mutation::Delete { deletion }], position) + } + fn mutate(&self, mut batch: Vec, position: Option) -> Result { + normalize(&mut batch, &self.identity)?; + let batch_digest = digest(&batch)?; + let mut s = self.state.lock(); + let before = Self::status_locked(&s)?; + if position.is_none() && before.sealed { + return Err(ControlledError::Conflict("snapshot is sealed")); + } + if position.is_some() && !before.sealed { + return Err(ControlledError::NotReady); + } + let result=s.storage.with_controlled_conn(|conn| -> Result { + let tx=conn.unchecked_transaction()?; + if let Some(pos)=position { + if pos==before.position { + let (d,r):(Option,Option)=tx.query_row("SELECT last_batch,last_receipt FROM controlled_meta WHERE id=1",[],|r|Ok((r.get(0)?,r.get(1)?)))?; + if d.as_deref()==Some(&batch_digest) { return decode(r.as_deref().ok_or(ControlledError::Storage)?); } + return Err(ControlledError::Conflict("position replay differs")); + } + if before.position.checked_add(1)!=Some(pos) { return Err(ControlledError::Conflict("nonsequential position")); } + } + let mut dimension:Option=tx.query_row("SELECT dimension FROM controlled_meta WHERE id=1",[],|r|r.get(0))?; + let mut outcomes=Vec::new(); + for mutation in &batch { + let (id,revision,scope,p)=match mutation { Mutation::Upsert{projection:p}=>(&p.memory_id,p.revision,&p.scope,Some(p)),Mutation::Delete{deletion:d}=>(&d.memory_id,d.revision,&d.scope,None) }; + let scope_json=json(scope)?; + let old:Option=tx.query_row("SELECT scope,revision,internal_id,live,intent FROM controlled_fences WHERE memory_id=?1",[id],|r|Ok((r.get(0)?,r.get(1)?,r.get(2)?,r.get(3)?,r.get(4)?))).optional()?; + let intent=p.map(digest).transpose()?; + let live=p.is_some(); + let mut outcome="applied"; + if let Some((old_scope,old_rev,_,old_live,old_intent))=&old { + if old_scope!=&scope_json { return Err(ControlledError::Conflict("immutable scope")); } + if revision<*old_rev { outcome="stale"; } + else if revision==*old_rev { + if *old_live!=live || old_intent!=&intent { return Err(ControlledError::Conflict("revision replay differs")); } + outcome="replayed"; + } + } + if outcome=="applied" { + let internal=old.as_ref().and_then(|x|x.2.clone()).unwrap_or_else(||Uuid::new_v4().to_string()); + tx.execute("DELETE FROM memories WHERE id=?1",[&internal])?; + if let Some(p)=p { + if let Some(v)=&p.embedding { + if dimension.is_some_and(|d|d!=v.len()) { return Err(ControlledError::InvalidInput("embedding dimension")); }dimension=Some(v.len()); + } + let mut envelope=p.clone();envelope.embedding=None; + let mut mem=MemObjectBuilder::new(MemoryTier::Episodic,MemContent::Text(p.text.clone()),MemSource::new("bot_projection")) + .namespace(scope_json.clone()).meta("projection",serde_json::to_value(&envelope).map_err(|_|ControlledError::Storage)?).build(); + mem.id=Uuid::parse_str(&internal).map_err(|_|ControlledError::Storage)?; + mem.embedding=p.embedding.clone().map(Arc::new); + SqliteStorage::insert_on_connection(&tx,&mem)?; + } + tx.execute("INSERT INTO controlled_fences(memory_id,scope,revision,internal_id,live,intent,valid_from,valid_until) VALUES(?1,?2,?3,?4,?5,?6,?7,?8) ON CONFLICT(memory_id) DO UPDATE SET revision=excluded.revision,internal_id=excluded.internal_id,live=excluded.live,intent=excluded.intent,valid_from=excluded.valid_from,valid_until=excluded.valid_until",params![id,scope_json,revision,internal,live,intent,p.and_then(|x|x.valid_from.map(|t|t.timestamp_millis())),p.and_then(|x|x.valid_until.map(|t|t.timestamp_millis()))])?; + } + let (adopted_rev,adopted_live)=if outcome=="stale" { let x=old.as_ref().unwrap();(x.1,x.3) }else{(revision,live)}; + outcomes.push(Outcome{memory_id:id.clone(),revision:adopted_rev,state:if adopted_live{"live"}else{"withdrawn"}.into(),outcome:outcome.into()}); + } + let receipt=Receipt{generation:before.generation.clone(),position:position.unwrap_or(before.position),durable:true,outcomes}; + tx.execute("UPDATE controlled_meta SET dimension=?1 WHERE id=1",[dimension])?; + if let Some(pos)=position { tx.execute("UPDATE controlled_meta SET position=?1,last_batch=?2,last_receipt=?3 WHERE id=1",params![pos,batch_digest,json(&receipt)?])?; } + tx.commit()?;Ok(receipt) + }); + s.storage.invalidate_controlled_cache(); + if matches!(result, Err(ControlledError::Storage)) { + s.ready = false; + } + if result.is_ok() && batch.iter().any(|m| matches!(m, Mutation::Delete { .. })) { + s.storage.with_controlled_conn(|c| { + SqliteStorage::scrub_after_delete(c); + Ok::<_, ControlledError>(()) + })?; + } + result + } + pub fn search( + &self, + query: &str, + scopes: &[Scope], + options: SearchOptions, + ) -> Result { + let SearchOptions { + limit, + embedding, + max_candidates, + max_duration_ms, + } = options; + if query.trim().is_empty() + || query.chars().count() > 4000 + || !(1..=100).contains(&limit) + || scopes.len() > 64 + || !(1..=10000).contains(&max_candidates) + || !(1..=250).contains(&max_duration_ms) + { + return Err(ControlledError::InvalidInput("search bounds")); + } + vector(&embedding)?; + for scope in scopes { + scope_check(scope, &self.identity)?; + } + let s = self.state.lock(); + let status = Self::status_locked(&s)?; + if !status.ready { + return Err(ControlledError::NotReady); + } + let empty = || SearchResult { + hits: vec![], + degraded: false, + truncated: false, + }; + if scopes.is_empty() { + return Ok(empty()); + } + let started = Instant::now(); + let duration = Duration::from_millis(max_duration_ms); + let mut scope_json: Vec = scopes.iter().map(json).collect::>()?; + scope_json.sort(); + scope_json.dedup(); + let terms: Vec = query + .split(|c: char| !c.is_alphanumeric()) + .filter(|s| !s.is_empty()) + .take(65) + .map(|s| format!("\"{s}\"")) + .collect(); + if terms.len() > 64 { + return Err(ControlledError::InvalidInput("query term limit")); + } + let fts = if terms.is_empty() { + "\"\"".into() + } else { + terms.join(" AND ") + }; + let fetched=s.storage.with_controlled_conn(|conn| -> Result<_> { + let dim:Option=conn.query_row("SELECT dimension FROM controlled_meta WHERE id=1",[],|r|r.get(0))?; + if let (Some(v),Some(d))=(&embedding,dim) { if v.len()!=d { return Err(ControlledError::InvalidInput("embedding dimension")); }} + conn.progress_handler(1000,Some(move ||started.elapsed()>=duration)); + let run=(|| -> std::result::Result,rusqlite::Error> { + let placeholders=vec!["?";scope_json.len()].join(","); + let sql=format!("SELECT m.id,m.tier,m.content_json,m.embedding_blob,m.temporal_json,m.source_json,m.salience_json,m.privacy_json,m.tags_json,m.metadata_json,m.links_json,m.content_hash,m.namespace, m.id IN (SELECT id FROM memories_fts WHERE memories_fts MATCH ?) AS matched FROM memories m JOIN controlled_fences f ON f.internal_id=m.id WHERE f.live=1 AND f.scope IN ({placeholders}) AND (f.valid_from IS NULL OR f.valid_from<=?) AND (f.valid_until IS NULL OR f.valid_until>?) ORDER BY matched DESC,m.id LIMIT ?"); + let now=Utc::now().timestamp_millis(); + let mut args:Vec=vec![fts.clone().into()];args.extend(scope_json.iter().cloned().map(Into::into));args.extend([now.into(),now.into(),((max_candidates+1)as i64).into()]); + let mut stmt=conn.prepare(&sql)?;let rows=stmt.query_map(params_from_iter(args),|r|Ok((SqliteStorage::parse_mem_row(r)?,r.get(13)?)))?;rows.collect() + })(); + conn.progress_handler(0,None::bool>); + match run { Ok(v)=>Ok(Some(v)),Err(rusqlite::Error::SqliteFailure(e,_)) if e.code==rusqlite::ErrorCode::OperationInterrupted => Ok(None),Err(_)=>Err(ControlledError::Storage) } + })?; + let Some(mut fetched) = fetched else { + return Ok(SearchResult { + hits: vec![], + degraded: true, + truncated: true, + }); + }; + let truncated = fetched.len() > max_candidates; + fetched.truncate(max_candidates); + let index = MemoryIndex::new(); + let mut mems = HashMap::new(); + let mut lexical = HashMap::new(); + let mut scores = HashMap::new(); + for (mem, matched) in fetched { + if let Some(v) = &mem.embedding { + index.insert_arc(mem.id, v); + } + if matched { + lexical.insert(mem.id, 1.0); + } + scores.insert(mem.id, 0.5); + mems.insert(mem.id, Arc::new(mem)); + } + if let Some(v) = &embedding { + for (id, score) in index.search(v, max_candidates) { + scores.insert(id, score); + } + } + let mut q = RetrievalQuery::new(query, limit); + q.embedding = embedding; + q.budget = QueryBudget { + max_candidates, + max_duration: duration, + }; + let ranked = RetrievalEngine::new(&s.storage, &index) + .rank_candidates(&q, &scores, &mems, &lexical)?; + let mut hits = Vec::new(); + for r in ranked { + let projection: Projection = serde_json::from_value( + r.memory + .metadata + .get("projection") + .ok_or(ControlledError::Storage)? + .clone(), + ) + .map_err(|_| ControlledError::Storage)?; + hits.push(Hit { + owner_type: self.identity.owner_type.clone(), + owner_id: self.identity.owner_id.clone(), + bot_id: self.identity.bot_id.clone(), + projection, + score: r.score, + }); + } + Ok(SearchResult { + hits, + degraded: truncated || started.elapsed() >= duration, + truncated, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + fn identity() -> StoreIdentity { + StoreIdentity { + owner_type: "merchant".into(), + owner_id: "test-owner".into(), + bot_id: "test-bot".into(), + replica_id: "test-replica".into(), + } + } + fn projection(id: &str) -> Projection { + Projection { + memory_id: id.into(), + revision: 1, + scope: Scope { + kind: "owner".into(), + id: "test-owner".into(), + }, + kind: "preference".into(), + text: "Remember the exact confirmed preference".into(), + source_refs: vec![SourceRef { + kind: "input".into(), + provider: "bot".into(), + id: "input-1".into(), + }], + valid_from: None, + valid_until: None, + embedding: None, + } + } + #[test] + fn controlled_writes_are_durable_and_do_not_infer_or_merge() { + let root = tempfile::tempdir().unwrap(); + let store = ControlledStore::open( + &root.path().canonicalize().unwrap().join("private"), + identity(), + ) + .unwrap(); + store.seed(vec![projection("a"), projection("b")]).unwrap(); + let state = store.state.lock(); + state + .storage + .with_controlled_conn(|c| -> Result<()> { + assert_eq!( + c.query_row("PRAGMA synchronous", [], |r| r.get::<_, i64>(0))?, + 2 + ); + assert_eq!( + c.query_row("SELECT COUNT(*) FROM memories", [], |r| r.get::<_, i64>(0))?, + 2 + ); + assert_eq!( + c.query_row("SELECT COUNT(*) FROM people", [], |r| r.get::<_, i64>(0))?, + 0 + ); + assert_eq!( + c.query_row("SELECT COUNT(*) FROM beliefs", [], |r| r.get::<_, i64>(0))?, + 0 + ); + Ok(()) + }) + .unwrap(); + } + #[test] + fn sql_failure_rolls_back_payload_fence_cursor_and_disables_queries() { + let root = tempfile::tempdir().unwrap(); + let store = ControlledStore::open( + &root.path().canonicalize().unwrap().join("private"), + identity(), + ) + .unwrap(); + store.seed(vec![projection("a")]).unwrap(); + store.finish_snapshot(10).unwrap(); + let status = store.status().unwrap(); + store.reconcile(&status.generation, 10).unwrap(); + store.state.lock().storage.with_controlled_conn(|c| -> Result<()> {c.execute_batch("CREATE TRIGGER controlled_failure BEFORE INSERT ON memories BEGIN SELECT RAISE(ABORT,'injected failure'); END;")?;Ok(())}).unwrap(); + let error = store.upsert(projection("b"), 11).unwrap_err(); + assert_eq!(error.code(), "storage"); + assert!(!store.status().unwrap().ready); + assert_eq!(store.status().unwrap().position, 10); + store + .state + .lock() + .storage + .with_controlled_conn(|c| -> Result<()> { + assert_eq!( + c.query_row("SELECT COUNT(*) FROM controlled_fences", [], |r| r + .get::<_, i64>(0))?, + 1 + ); + Ok(()) + }) + .unwrap(); + } + #[test] + fn equal_revision_replay_with_changed_embedding_conflicts() { + let root = tempfile::tempdir().unwrap(); + let store = ControlledStore::open( + &root.path().canonicalize().unwrap().join("private"), + identity(), + ) + .unwrap(); + let mut p = projection("a"); + p.embedding = Some(vec![1.0, 0.0]); + store.seed(vec![p.clone()]).unwrap(); + store.seed(vec![p.clone()]).unwrap(); + p.embedding = Some(vec![0.0, 1.0]); + assert_eq!(store.seed(vec![p]).unwrap_err().code(), "conflict"); + } +} diff --git a/cortex-core/src/lib.rs b/cortex-core/src/lib.rs index e6b85ad..37bbb1a 100644 --- a/cortex-core/src/lib.rs +++ b/cortex-core/src/lib.rs @@ -2,6 +2,7 @@ pub mod belief; pub mod compression; pub mod consolidation; pub mod context; +pub mod controlled; pub mod embedder; pub mod episode; pub mod events; diff --git a/cortex-core/src/retrieval.rs b/cortex-core/src/retrieval.rs index daf741d..ef9e64d 100644 --- a/cortex-core/src/retrieval.rs +++ b/cortex-core/src/retrieval.rs @@ -283,6 +283,15 @@ impl<'a> RetrievalEngine<'a> { let memories = self.storage.get_memories_batch(&all_ids)?; let mem_map: HashMap> = memories.into_iter().map(|m| (m.id, Arc::new(m))).collect(); + self.rank_candidates(query, &candidate_scores, &mem_map, &fts_scores) + } + + /// Shared ranking for candidates already constrained by a trusted scope predicate. + pub(crate) fn rank_candidates( + &self, query: &RetrievalQuery, candidate_scores: &HashMap, + mem_map: &HashMap>, fts_scores: &HashMap, + ) -> Result, CortexError> { + let temporal_intent = detect_temporal_intent(&query.text); // Adapt weights for temporal queries let weights = match temporal_intent { TemporalIntent::None => self.weights.clone(), @@ -343,7 +352,7 @@ impl<'a> RetrievalEngine<'a> { // Score each candidate let mut results = Vec::new(); - for (&id, &sim_score) in &candidate_scores { + for (&id, &sim_score) in candidate_scores { if let Some(mem) = mem_map.get(&id) { // Namespace isolation: skip memories that don't match requested namespace if let Some(ref ns) = query.namespace { diff --git a/cortex-core/src/storage/sqlite.rs b/cortex-core/src/storage/sqlite.rs index ce3e03c..0ba32ea 100644 --- a/cortex-core/src/storage/sqlite.rs +++ b/cortex-core/src/storage/sqlite.rs @@ -158,11 +158,44 @@ impl SqliteStorage { } fn open_keyed(path: &str, key: Option, readers: usize) -> Result { + Self::open_keyed_mode(path, key, readers, false) + } + + pub(crate) fn open_controlled(path: &str) -> Result { + let storage = Self::open_keyed_mode(path, None, 1, true)?; + storage.with_write_conn(|conn| { + conn.execute_batch("PRAGMA synchronous=FULL;") + .map_err(|e| CortexError::Storage(e.to_string())) + })?; + Ok(storage) + } + + fn open_keyed_mode(path: &str, key: Option, readers: usize, controlled: bool) -> Result { #[cfg(unix)] Self::restrict_db_file_mode(path)?; let conn = Connection::open(path).map_err(|e| CortexError::Storage(e.to_string()))?; Self::apply_key(&conn, key.as_ref())?; + let marked: bool = conn.query_row( + "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name='cortex_controlled_marker')", [], |r| r.get(0) + ).map_err(|e| CortexError::Storage(e.to_string()))?; + if marked && !controlled { + return Err(CortexError::InvalidInput("controlled database requires ControlledStore".into())); + } + if controlled && !marked { + let nonempty: bool = conn.query_row( + "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name NOT LIKE 'sqlite_%')", [], |r| r.get(0) + ).map_err(|e| CortexError::Storage(e.to_string()))?; + if nonempty { return Err(CortexError::InvalidInput("cannot import a personal database".into())); } + conn.execute_batch("BEGIN IMMEDIATE; CREATE TABLE cortex_controlled_marker(version INTEGER NOT NULL); INSERT INTO cortex_controlled_marker VALUES(1); COMMIT;") + .map_err(|e| CortexError::Storage(e.to_string()))?; + } + if controlled { + let version: i64 = conn.query_row("SELECT version FROM cortex_controlled_marker", [], |r| r.get(0)) + .map_err(|e| CortexError::Storage(e.to_string()))?; + if version != 1 { return Err(CortexError::InvalidInput("unsupported controlled store format".into())); } + } + conn.execute_batch( "PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; @@ -216,7 +249,7 @@ impl SqliteStorage { /// index, then fold the WAL back into the (zero-scrubbed) database and truncate it so /// the plaintext does not linger in old WAL frames. Best effort: a reader holding a /// snapshot can block the checkpoint; the next delete or autocheckpoint retries. - fn scrub_after_delete(conn: &Connection) { + pub(crate) fn scrub_after_delete(conn: &Connection) { let fts_secure_delete = conn .query_row( "SELECT v FROM memories_fts_config WHERE k = 'secure-delete'", @@ -288,6 +321,56 @@ impl SqliteStorage { f(&conn) } + pub(crate) fn insert_on_connection(conn: &Connection, mem: &MemObject) -> Result<(), CortexError> { + let now = Utc::now().to_rfc3339(); + let embedding_blob = mem.embedding.as_ref().map(|e| f32_vec_to_bytes(e)); + let content_json = serde_json::to_string(&mem.content).unwrap(); + let id_str = mem.id.to_string(); + let (fact_subject, fact_object, pref_key) = extract_materialized_columns(&mem.content); + let mut stmt = conn + .prepare_cached( + "INSERT INTO memories (id, tier, content_json, embedding_blob, temporal_json, source_json, salience_json, privacy_json, tags_json, metadata_json, links_json, content_hash, namespace, salience_score, fact_subject, fact_object, pref_key, source_channel, source_identity_id, created_at, updated_at) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18, ?19, ?20, ?21)", + ) + .map_err(|e| CortexError::Storage(e.to_string()))?; + stmt.execute(params![ + id_str, + mem.tier.as_str(), + content_json, + embedding_blob, + serde_json::to_string(&mem.temporal).unwrap(), + serde_json::to_string(&mem.source).unwrap(), + serde_json::to_string(&mem.salience).unwrap(), + serialize_privacy(&mem.privacy), + serialize_tags(&mem.tags), + serialize_metadata(&mem.metadata), + serialize_links(&mem.links), + mem.content_hash, + mem.namespace, + mem.salience.effective_score, + fact_subject, + fact_object, + pref_key, + mem.source.channel, + mem.source.identity_id.map(|id| id.to_string()), + now, + now, + ]) + .map_err(|e| CortexError::Storage(e.to_string()))?; + + Ok(()) + } + + pub(crate) fn invalidate_controlled_cache(&self) { + self.mem_cache.lock().clear(); + self.entity_cache.lock().clear(); + } + + pub(crate) fn with_controlled_conn(&self, f: F) -> Result + where F: FnOnce(&Connection) -> Result { + f(&self.write_conn.lock()) + } + /// Invalidate a single entry from the memory cache. fn cache_evict(&self, id: &Uuid) { self.mem_cache.lock().pop(id); @@ -317,7 +400,7 @@ impl SqliteStorage { }) } - fn parse_mem_row(row: &rusqlite::Row) -> Result { + pub(crate) fn parse_mem_row(row: &rusqlite::Row) -> Result { let id_str: String = row.get(0)?; let tier_str: String = row.get(1)?; let content_json: String = row.get(2)?; @@ -726,43 +809,7 @@ impl StorageBackend for SqliteStorage { fn store_memory(&self, mem: &MemObject) -> Result<(), CortexError> { let conn = self.write_conn.lock(); - let now = Utc::now().to_rfc3339(); - let embedding_blob = mem.embedding.as_ref().map(|e| f32_vec_to_bytes(e)); - let content_json = serde_json::to_string(&mem.content).unwrap(); - let id_str = mem.id.to_string(); - let (fact_subject, fact_object, pref_key) = extract_materialized_columns(&mem.content); - let mut stmt = conn - .prepare_cached( - "INSERT INTO memories (id, tier, content_json, embedding_blob, temporal_json, source_json, salience_json, privacy_json, tags_json, metadata_json, links_json, content_hash, namespace, salience_score, fact_subject, fact_object, pref_key, source_channel, source_identity_id, created_at, updated_at) - VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18, ?19, ?20, ?21)", - ) - .map_err(|e| CortexError::Storage(e.to_string()))?; - stmt.execute(params![ - id_str, - mem.tier.as_str(), - content_json, - embedding_blob, - serde_json::to_string(&mem.temporal).unwrap(), - serde_json::to_string(&mem.source).unwrap(), - serde_json::to_string(&mem.salience).unwrap(), - serialize_privacy(&mem.privacy), - serialize_tags(&mem.tags), - serialize_metadata(&mem.metadata), - serialize_links(&mem.links), - mem.content_hash, - mem.namespace, - mem.salience.effective_score, - fact_subject, - fact_object, - pref_key, - mem.source.channel, - mem.source.identity_id.map(|id| id.to_string()), - now, - now, - ]) - .map_err(|e| CortexError::Storage(e.to_string()))?; - - // FTS5 synced via trigger (memories_fts_ai) + Self::insert_on_connection(&conn, mem)?; self.cache_put(mem); Ok(()) } diff --git a/cortex-core/tests/test_sync.rs b/cortex-core/tests/test_sync.rs index cbf1f4b..18a836a 100644 --- a/cortex-core/tests/test_sync.rs +++ b/cortex-core/tests/test_sync.rs @@ -1097,8 +1097,13 @@ fn test_background_sync_detects_changes() { ) .unwrap(); - // Wait for debounce + processing time - std::thread::sleep(Duration::from_millis(1500)); + // Native FSEvents may coalesce delivery for several seconds before our + // debounce even starts. Wait for the observable callback, retaining a bounded + // failure deadline rather than assuming a platform scheduling latency. + let deadline = std::time::Instant::now() + Duration::from_secs(10); + while call_count.load(Ordering::SeqCst) == 0 && std::time::Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(20)); + } let count = call_count.load(Ordering::SeqCst); assert!( diff --git a/cortex-python/Cargo.toml b/cortex-python/Cargo.toml index 65ceda1..a6f5ae9 100644 --- a/cortex-python/Cargo.toml +++ b/cortex-python/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cortex-python" -version = "2.4.0" +version = "2.5.0" edition = "2021" rust-version = "1.89" @@ -15,5 +15,8 @@ crate-type = ["cdylib"] # cortex-core proper and is unaffected. (Opt back in with a feature if a wheel ever needs # SQLCipher and the build env supports it.) cortex-core = { path = "../cortex-core", default-features = false } +serde = { version = "1", features = ["derive"] } +serde_json = "1" +parking_lot = "0.12" pyo3 = { version = "0.23", features = ["extension-module"] } uuid = { version = "1", features = ["v4"] } diff --git a/cortex-python/README.md b/cortex-python/README.md index 0792918..ace50bf 100644 --- a/cortex-python/README.md +++ b/cortex-python/README.md @@ -13,10 +13,10 @@ pip install cortex-ai-memory ## Quick Start ```python -from cortex_python import PyCortex +from cortex_ai_memory import Cortex # Open or create a memory database -cx = PyCortex("memory.db") +cx = Cortex("memory.db") # Ingest memories cx.ingest("Met Alice at the Q3 planning meeting", "slack", user_id="alice_123") @@ -76,3 +76,16 @@ results = cx.retrieve("where do I live?", 5, embedding=embed("where do I live?") ## License MIT + +## Hosted Bot projections + +`ControlledStore` provides an isolated, durable projection backend without ingest-time +inference or consolidation. See the [controlled API contract](../docs/controlled-memory-api.md) +for snapshot/replay reconciliation, immutable scopes, typed dictionaries, error codes, +query bounds and a complete example. The caller retains PostgreSQL authority and must +revalidate every hit before use. `Cortex` remains the personal-agent interface. + +The wheel uses plaintext SQLite. Controlled roots and all artifacts are owner-only; +this does not replace your host's approved storage encryption or retention policy. +Native controlled operations release the GIL, but the host still needs bounded worker +capacity and a PostgreSQL fallback. No public HTTP service or cloud sync is enabled. diff --git a/cortex-python/pyproject.toml b/cortex-python/pyproject.toml index 77db1cf..9129897 100644 --- a/cortex-python/pyproject.toml +++ b/cortex-python/pyproject.toml @@ -4,9 +4,9 @@ build-backend = "maturin" [project] name = "cortex-ai-memory" -version = "2.4.0" +version = "2.5.0" description = "Private. Free. Local. — Memory engine for personal AI agents." -requires-python = ">=3.9" +requires-python = ">=3.9,<3.14" license = "MIT" authors = [ { name = "Alvin Tang" } diff --git a/cortex-python/src/lib.rs b/cortex-python/src/lib.rs index db43ca8..5ee8503 100644 --- a/cortex-python/src/lib.rs +++ b/cortex-python/src/lib.rs @@ -64,7 +64,11 @@ impl PyCortex { .map(|r| { let summary = match &r.memory.content { cortex_core::types::MemContent::Text(t) => t.clone(), - cortex_core::types::MemContent::Fact { subject, predicate, object } => { + cortex_core::types::MemContent::Fact { + subject, + predicate, + object, + } => { format!("{} {} {}", subject, predicate, object) } cortex_core::types::MemContent::Preference { key, value, .. } => { @@ -96,12 +100,7 @@ impl PyCortex { } /// Add or resolve a person by channel identity. - fn add_person( - &self, - name: &str, - channel: &str, - channel_user_id: &str, - ) -> PyResult { + fn add_person(&self, name: &str, channel: &str, channel_user_id: &str) -> PyResult { let person = self .inner .add_person(name, channel, channel_user_id) @@ -123,16 +122,14 @@ impl PyCortex { /// Get beliefs above a confidence threshold. fn get_beliefs(&self, threshold: f32) -> PyResult> { let beliefs = self.inner.get_beliefs(threshold).map_err(to_py_err)?; - Ok(beliefs.into_iter().map(|b| (b.key, b.probability)).collect()) + Ok(beliefs + .into_iter() + .map(|b| (b.key, b.probability)) + .collect()) } /// Observe evidence for a belief key. - fn observe_belief( - &self, - key: &str, - supports: bool, - strength: f32, - ) -> PyResult { + fn observe_belief(&self, key: &str, supports: bool, strength: f32) -> PyResult { let belief = self .inner .observe_belief(key, supports, strength) @@ -157,12 +154,7 @@ impl PyCortex { } /// Add a user preference. - fn add_preference( - &self, - key: &str, - value: &str, - confidence: f32, - ) -> PyResult { + fn add_preference(&self, key: &str, value: &str, confidence: f32) -> PyResult { let mem = self .inner .add_preference(key, value, confidence) @@ -171,8 +163,193 @@ impl PyCortex { } } +pyo3::create_exception!(cortex_ai_memory, ControlledError, PyRuntimeError); + +fn controlled_error(py: Python<'_>, e: cortex_core::controlled::ControlledError) -> PyErr { + let error = ControlledError::new_err(e.to_string()); + let _ = error.value(py).setattr("code", e.code()); + error +} +fn invalid_controlled(py: Python<'_>) -> PyErr { + controlled_error( + py, + cortex_core::controlled::ControlledError::InvalidInput("request"), + ) +} +fn parse_controlled( + py: Python<'_>, + value: &Bound<'_, PyAny>, +) -> PyResult { + let body: String = (|| -> PyResult<_> { + py.import("json")? + .call_method1("dumps", (value,))? + .extract() + })() + .map_err(|_| invalid_controlled(py))?; + if body.len() > 8 * 1024 * 1024 { + return Err(invalid_controlled(py)); + } + serde_json::from_str(&body).map_err(|_| invalid_controlled(py)) +} +fn controlled_result( + py: Python<'_>, + value: Result, +) -> PyResult> { + let value = value.map_err(|e| controlled_error(py, e))?; + let body = serde_json::to_string(&value).map_err(|_| invalid_controlled(py))?; + Ok(py.import("json")?.call_method1("loads", (body,))?.unbind()) +} + +#[pyclass(name = "ControlledStore")] +struct PyControlledStore { + inner: parking_lot::Mutex>>, +} +impl PyControlledStore { + fn handle( + &self, + py: Python<'_>, + ) -> PyResult> { + self.inner + .lock() + .clone() + .ok_or_else(|| controlled_error(py, cortex_core::controlled::ControlledError::Closed)) + } +} +#[pymethods] +impl PyControlledStore { + #[new] + fn new( + py: Python<'_>, + root: String, + owner_type: String, + owner_id: String, + bot_id: String, + replica_id: String, + ) -> PyResult { + let identity = cortex_core::controlled::StoreIdentity { + owner_type, + owner_id, + bot_id, + replica_id, + }; + let store = py + .allow_threads(|| { + cortex_core::controlled::ControlledStore::open( + std::path::Path::new(&root), + identity, + ) + }) + .map_err(|e| controlled_error(py, e))?; + Ok(Self { + inner: parking_lot::Mutex::new(Some(std::sync::Arc::new(store))), + }) + } + fn status(&self, py: Python<'_>) -> PyResult> { + let s = self.handle(py)?; + controlled_result(py, py.allow_threads(|| s.status())) + } + fn seed(&self, py: Python<'_>, projections: &Bound<'_, PyAny>) -> PyResult> { + let s = self.handle(py)?; + let p = parse_controlled(py, projections)?; + controlled_result(py, py.allow_threads(|| s.seed(p))) + } + fn finish_snapshot(&self, py: Python<'_>, position: &Bound<'_, PyAny>) -> PyResult> { + let s = self.handle(py)?; + let p = parse_controlled(py, position)?; + controlled_result(py, py.allow_threads(|| s.finish_snapshot(p))) + } + fn reconcile( + &self, + py: Python<'_>, + generation: String, + position: &Bound<'_, PyAny>, + ) -> PyResult> { + let s = self.handle(py)?; + let p = parse_controlled(py, position)?; + controlled_result(py, py.allow_threads(|| s.reconcile(&generation, p))) + } + fn apply_batch( + &self, + py: Python<'_>, + mutations: &Bound<'_, PyAny>, + position: &Bound<'_, PyAny>, + ) -> PyResult> { + let s = self.handle(py)?; + let m = parse_controlled(py, mutations)?; + let p = parse_controlled(py, position)?; + controlled_result(py, py.allow_threads(|| s.apply_batch(m, p))) + } + fn upsert( + &self, + py: Python<'_>, + projection: &Bound<'_, PyAny>, + position: &Bound<'_, PyAny>, + ) -> PyResult> { + let s = self.handle(py)?; + let m = parse_controlled(py, projection)?; + let p = parse_controlled(py, position)?; + controlled_result(py, py.allow_threads(|| s.upsert(m, p))) + } + fn delete( + &self, + py: Python<'_>, + deletion: &Bound<'_, PyAny>, + position: &Bound<'_, PyAny>, + ) -> PyResult> { + let s = self.handle(py)?; + let m = parse_controlled(py, deletion)?; + let p = parse_controlled(py, position)?; + controlled_result(py, py.allow_threads(|| s.delete(m, p))) + } + #[pyo3(signature = (query, allowed_scopes, limit=10, embedding=None, max_candidates=1000, max_duration_ms=250))] + #[allow(clippy::too_many_arguments)] + fn search( + &self, + py: Python<'_>, + query: String, + allowed_scopes: &Bound<'_, PyAny>, + limit: i64, + embedding: Option>, + max_candidates: i64, + max_duration_ms: i64, + ) -> PyResult> { + let s = self.handle(py)?; + let scopes: Vec = parse_controlled(py, allowed_scopes)?; + if limit < 0 || max_candidates < 0 || max_duration_ms < 0 { + return Err(invalid_controlled(py)); + } + controlled_result( + py, + py.allow_threads(|| { + s.search( + &query, + &scopes, + cortex_core::controlled::SearchOptions { + limit: limit as usize, + embedding, + max_candidates: max_candidates as usize, + max_duration_ms: max_duration_ms as u64, + }, + ) + }), + ) + } + fn close(&self, py: Python<'_>) -> PyResult<()> { + let s = self.inner.lock().take().ok_or_else(|| { + controlled_error(py, cortex_core::controlled::ControlledError::Closed) + })?; + py.allow_threads(move || { + s.close(); + drop(s); + }); + Ok(()) + } +} + #[pymodule] fn cortex_ai_memory(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; + m.add_class::()?; + m.add("ControlledError", m.py().get_type::())?; Ok(()) } diff --git a/cortex-python/tests/test_controlled_store.py b/cortex-python/tests/test_controlled_store.py new file mode 100644 index 0000000..36ba0e4 --- /dev/null +++ b/cortex-python/tests/test_controlled_store.py @@ -0,0 +1,507 @@ +"""Independent black-box acceptance for the documented installed-wheel API. + +Run outside the checkout with an installed wheel, for example: + python -I /absolute/path/to/tests/test_controlled_store.py +No implementation imports, network, models, production data, or SQL mutations. +""" +import concurrent.futures +import copy +import datetime +import gc +import importlib.metadata +import json +import math +import os +from pathlib import Path +import queue +import shutil +import subprocess +import sys +import tempfile +import threading +import time +import unittest + +import cortex_ai_memory as memory +from cortex_ai_memory import ControlledError, ControlledStore + + +SCOPE = {"kind": "owner", "id": "merchant-acceptance"} +OTHER = {"kind": "entity", "id": "entity-other"} + + +def projection(key="memory-one", revision=1, scope=None, text="orchid report preference", **extra): + value = {"memory_id": key, "revision": revision, "scope": dict(scope or SCOPE), + "kind": "preference", "text": text, + "source_refs": [{"kind": "input", "provider": "acceptance", "id": "source-one"}]} + value.update(extra) + return value + + +def upsert(value): + return {"op": "upsert", "projection": value} + + +def deletion(key="memory-one", revision=2, scope=None): + return {"memory_id": key, "revision": revision, "scope": dict(scope or SCOPE)} + + +CHILD = r''' +import json, sys, time +from cortex_ai_memory import ControlledStore, ControlledError +root, mode = sys.argv[1:] +try: + s = ControlledStore(root, 'merchant', 'merchant-acceptance', 'bot-one', 'replica-one') +except ControlledError as e: + print(json.dumps({'error': e.code}), flush=True) + sys.exit(0) +if mode == 'lock': + print(json.dumps({'opened': True}), flush=True) + time.sleep(60) +else: + state = s.status() + s.reconcile(state['generation'], state['position']) + values = [{'op':'upsert', 'projection': {'memory_id':'crash-%d'%i, 'revision':2, + 'scope':{'kind':'owner','id':'merchant-acceptance'}, 'kind':'fact', + 'text':'orchid committed revision two ' + 'x'*11000, + 'source_refs':[{'kind':'input','provider':'acceptance','id':'crash-source'}]}} + for i in range(100)] + print(json.dumps({'starting': True}), flush=True) + receipt = s.apply_batch(values, 101) + print(json.dumps({'committed': receipt}), flush=True) + time.sleep(60) +''' + + +class ControlledStoreAcceptance(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory(prefix="controlled-acceptance-") + self.root = str(Path(self.temp.name).resolve() / "private") + self.handles = [] + self.children = [] + + def tearDown(self): + for child in self.children: + if child.poll() is None: + child.kill() + child.communicate(timeout=10) + for store in reversed(self.handles): + try: + store.close() + except ControlledError as error: + self.assertEqual(error.code, "closed") + self.temp.cleanup() + + def open(self, replica="replica-one", bot="bot-one", owner="merchant-acceptance", root=None): + value = ControlledStore(root or self.root, "merchant", owner, bot, replica) + self.handles.append(value) + return value + + def ready(self, values=(), position=100, **kwargs): + store = self.open(**kwargs) + store.seed(list(values)) + store.finish_snapshot(position) + state = store.status() + store.reconcile(state["generation"], position) + return store + + def error(self, code, function, *args, **kwargs): + with self.assertRaises(ControlledError) as caught: + function(*args, **kwargs) + self.assertEqual(caught.exception.code, code) + return caught.exception + + def hits(self, store, scopes=None, **kwargs): + result = store.search("orchid", [SCOPE] if scopes is None else scopes, **kwargs) + self.assertIsInstance(result["degraded"], bool) + self.assertIsInstance(result["truncated"], bool) + return result["hits"] + + def reconcile(self, store): + state = store.status() + store.reconcile(state["generation"], state["position"]) + return state + + def child(self, mode): + process = subprocess.Popen([sys.executable, "-I", "-c", CHILD, self.root, mode], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) + self.children.append(process) + return process + + def line(self, process): + lines = queue.Queue() + reader = threading.Thread(target=lambda: lines.put(process.stdout.readline()), daemon=True) + reader.start() + value = lines.get(timeout=20) + self.assertTrue(value, "child exited before its public-API checkpoint") + return json.loads(value) + + def test_installed_distribution_and_api(self): + module_file = Path(memory.__file__).resolve() + checkout = Path(__file__).resolve().parents[2] + self.assertNotIn(checkout, module_file.parents, "test requires the installed built wheel") + distributions = importlib.metadata.packages_distributions().get("cortex_ai_memory", []) + self.assertTrue(distributions, "native module must have installed wheel distribution metadata") + self.assertTrue(any(importlib.metadata.distribution(name).read_text("WHEEL") for name in distributions)) + self.assertTrue(issubclass(ControlledError, Exception)) + self.assertTrue(callable(memory.Cortex)) + + def test_snapshot_reconcile_restart_and_closed_lifecycle(self): + store = self.open() + initial = store.status() + self.assertFalse(initial["ready"]) + self.assertFalse(initial["sealed"]) + self.error("not_ready", store.search, "orchid", [SCOPE]) + self.error("not_ready", store.reconcile, initial["generation"], initial["position"]) + store.seed([projection()]) + store.seed([projection()]) + store.finish_snapshot(100) + store.finish_snapshot(100) + self.error("conflict", store.finish_snapshot, 101) + self.error("conflict", store.seed, [projection("late")]) + self.error("not_ready", store.search, "orchid", [SCOPE]) + state = self.reconcile(store) + self.assertTrue(store.status()["ready"]) + self.assertEqual(len(self.hits(store)), 1) + store.apply_batch([], 101) + store.finish_snapshot(100) + self.assertEqual(store.status()["position"], 101) + self.assertTrue(store.status()["sealed"]) + self.assertTrue(store.status()["ready"]) + self.error("conflict", store.finish_snapshot, 101) + self.assertEqual(store.status()["position"], 101) + self.error("conflict", store.reconcile, state["generation"] + "-wrong", 101) + self.assertFalse(store.status()["ready"]) + self.reconcile(store) + self.error("conflict", store.reconcile, state["generation"], 100) + self.assertFalse(store.status()["ready"]) + store.close() + for call in (store.status, lambda: store.search("orchid", [SCOPE]), + lambda: store.apply_batch([], 101), lambda: store.seed([])): + self.error("closed", call) + reopened = self.open() + self.assertFalse(reopened.status()["ready"]) + self.assertEqual(reopened.status()["generation"], state["generation"]) + self.assertEqual(reopened.status()["position"], 101) + self.reconcile(reopened) + self.assertEqual(self.hits(reopened)[0]["memory_id"], "memory-one") + + def test_revisions_normalized_replay_and_sequential_positions(self): + store = self.ready() + item = projection(revision=2, source_refs=[ + {"kind": "input", "provider": "acceptance", "id": "b"}, + {"kind": "input", "provider": "acceptance", "id": "a"}]) + receipt = store.upsert(item, 101) + self.assertTrue(receipt["durable"]) + self.assertEqual(receipt["position"], 101) + self.assertNotIn(item["text"], json.dumps(receipt)) + normalized = copy.deepcopy(item) + normalized["source_refs"].reverse() + self.assertEqual(store.apply_batch([upsert(normalized)], 101), receipt) + changed = dict(item, text="orchid conflicting text") + self.error("conflict", store.upsert, changed, 101) + self.error("conflict", store.upsert, changed, 102) + self.assertEqual(store.status()["position"], 101) + stale = store.upsert(projection(revision=1), 102) + self.assertEqual(stale["outcomes"][0]["outcome"], "stale") + replay = store.upsert(normalized, 103) + self.assertEqual(replay["outcomes"][0]["outcome"], "replayed") + self.error("conflict", store.apply_batch, [], 105) + self.error("conflict", store.apply_batch, [], 102) + empty = store.apply_batch([], 104) + self.assertEqual(empty["outcomes"], []) + self.assertEqual(store.apply_batch([], 104), empty) + self.error("conflict", store.upsert, projection("different"), 104) + hit = self.hits(store)[0] + self.assertEqual((hit["revision"], hit["text"]), (2, item["text"])) + + def test_tombstone_scope_and_unseen_delete_survive_restart(self): + store = self.ready([projection()]) + store.delete(deletion(), 101) + self.assertEqual(self.hits(store), []) + self.assertEqual(store.upsert(projection(), 102)["outcomes"][0]["outcome"], "stale") + self.error("conflict", store.upsert, projection(revision=3, scope=OTHER), 103) + self.error("invalid_input", store.delete, {"memory_id": "unseen", "revision": 5}, 103) + store.delete(deletion("unseen", 5, OTHER), 103) + store.close() + store = self.open() + self.reconcile(store) + self.error("conflict", store.upsert, projection("unseen", 6), 104) + self.assertEqual(store.upsert(projection("unseen", 4, OTHER), 104)["outcomes"][0]["outcome"], "stale") + duplicate = store.delete(deletion("unseen", 5, OTHER), 105) + self.assertEqual(duplicate["outcomes"][0]["outcome"], "replayed") + store.upsert(projection("unseen", 6, OTHER), 106) + store.upsert(projection(revision=3), 107) + self.assertEqual({h["memory_id"] for h in self.hits(store, [SCOPE, OTHER])}, {"memory-one", "unseen"}) + + def test_batch_atomicity_invalid_second_and_conflict_second(self): + store = self.ready([projection()]) + invalid = projection("invalid", revision=True) + self.error("invalid_input", store.apply_batch, + [upsert(projection("first")), upsert(invalid)], 101) + self.assertEqual(store.status()["position"], 100) + self.assertEqual({h["memory_id"] for h in self.hits(store)}, {"memory-one"}) + self.error("conflict", store.apply_batch, + [upsert(projection("first")), upsert(projection(text="orchid changed"))], 101) + self.assertEqual(store.status()["position"], 100) + store.upsert(projection("first"), 101) + self.assertEqual(len(self.hits(store)), 2) + + def test_scope_filter_precedes_topk_and_cache_and_vectors(self): + excluded = [projection("excluded-%02d" % i, scope=OTHER, + text="orchid " * 20, embedding=[1.0, 0.0]) for i in range(20)] + included = [projection("allowed-a", embedding=[0.9, 0.1]), + projection("allowed-b", scope={"kind": "project", "id": "project-a"}, embedding=[0.8, 0.2])] + store = self.ready(excluded + included) + allowed = [SCOPE, included[1]["scope"]] + for vector in (None, [1.0, 0.0], None): + hits = self.hits(store, allowed, limit=2, embedding=vector) + self.assertEqual({h["memory_id"] for h in hits}, {"allowed-a", "allowed-b"}) + for hit in hits: + self.assertEqual((hit["owner_type"], hit["owner_id"], hit["bot_id"]), + ("merchant", "merchant-acceptance", "bot-one")) + self.assertIn(hit["scope"], allowed) + self.assertEqual(hit["revision"], 1) + self.assertEqual(hit["source_refs"], included[0]["source_refs"]) + self.assertIsInstance(hit["score"], (int, float)) + self.assertEqual(self.hits(store, []), []) + with self.assertRaises((TypeError, ControlledError)): + store.search("orchid") + bounded = store.search("orchid", [SCOPE, OTHER], max_candidates=1) + self.assertTrue(bounded["degraded"] or bounded["truncated"]) + + def test_validity_roundtrip_and_interval_validation(self): + item = projection(valid_from="2000-01-01T00:00:00Z", valid_until="2090-01-01T00:00:00Z") + store = self.ready([item]) + # Native hits are candidates; current-time authorization belongs to PG. + hit = self.hits(store)[0] + self.assertEqual(hit["valid_from"], item["valid_from"]) + self.assertEqual(hit["valid_until"], item["valid_until"]) + for start, end in (("2031-01-01T00:00:00Z", "2030-01-01T00:00:00Z"), + ("2030-01-01T00:00:00Z", "2030-01-01T00:00:00Z"), + ("not-a-timestamp", "2031-01-01T00:00:00Z")): + with self.subTest(start=start, end=end): + self.error("invalid_input", store.upsert, + projection("bad-interval", valid_from=start, valid_until=end), 101) + store.upsert(projection("unbounded"), 101) + hit = next(h for h in self.hits(store) if h["memory_id"] == "unbounded") + self.assertIsNone(hit["valid_from"]) + self.assertIsNone(hit["valid_until"]) + + def test_input_bounds_unknown_fields_and_integer_precision(self): + store = self.ready() + bad_items = [projection(revision=r) for r in (True, 1.5, 0, -1, 2**63)] + bad_items += [projection(memory_id=""), projection(memory_id="x"*257), + projection(memory_id="line\nbreak"), projection(text="x"*12001), + projection(source_refs=[]), projection(extra_unknown=True), + projection(scope={"kind": "owner", "id": "another-owner"}), + projection(embedding=[math.nan]), projection(embedding=[math.inf]), + projection(embedding=[]), projection(embedding=[0.0]*4097)] + for item in bad_items: + with self.subTest(item_fields=list(item), revision=item["revision"]): + error = self.error("invalid_input", store.upsert, item, 101) + self.assertNotIn("orchid report preference", str(error)) + self.assertEqual(store.status()["position"], 100) + for position in (True, 1.5, -1, 2**63): + self.error("invalid_input", store.apply_batch, [], position) + for kwargs in ({"limit": 0}, {"limit": 101}, {"max_candidates": 0}, + {"max_candidates": 10001}, {"max_duration_ms": 0}, {"max_duration_ms": 251}): + self.error("invalid_input", store.search, "orchid", [SCOPE], **kwargs) + self.error("invalid_input", store.search, "", [SCOPE]) + self.error("invalid_input", store.search, "x"*4001, [SCOPE]) + self.error("invalid_input", store.search, "orchid", [OTHER]*65) + self.error("invalid_input", store.apply_batch, [upsert(projection(str(i))) for i in range(101)], 101) + large = 2**53 + 19 + store.upsert(projection(revision=large, embedding=[1.0, 0.0]), 101) + self.assertEqual(self.hits(store)[0]["revision"], large) + self.error("invalid_input", store.upsert, projection("wrong-dimension", embedding=[1.0]), 102) + other = self.ready(position=large, replica="large-position") + other.apply_batch([], large+1) + self.assertEqual(other.status()["position"], large+1) + + def test_owner_bot_replica_isolation_and_private_files(self): + first = self.ready([projection()]) + bot = self.ready([projection("other-bot", text="orchid bot independent")], bot="bot-two") + owner_scope = {"kind": "owner", "id": "merchant-two"} + owner = self.ready([projection("other-owner", scope=owner_scope)], owner="merchant-two") + replica = self.ready(replica="replica-two") + self.assertEqual([h["memory_id"] for h in self.hits(first)], ["memory-one"]) + self.assertEqual([h["memory_id"] for h in self.hits(bot)], ["other-bot"]) + self.assertEqual([h["memory_id"] for h in self.hits(owner, [owner_scope])], ["other-owner"]) + self.assertEqual(self.hits(replica), []) + self.assertNotEqual(first.status()["generation"], replica.status()["generation"]) + for path in Path(self.root).rglob("*"): + self.assertFalse(path.stat().st_mode & 0o077, "store artifact must be owner-only: " + str(path)) + link = Path(self.temp.name) / "symlink-root" + link.symlink_to(self.root, target_is_directory=True) + self.error("invalid_input", self.open, root=str(link)) + self.error("invalid_input", self.open, root=self.root + "/../escape") + + def test_personal_cortex_refuses_controlled_database(self): + store = self.ready([projection()]) + store.close() + # Inspect only fixture file headers to find the generated SQLite path; + # generated path layout and internal tables are deliberately unknown. + databases = [] + for path in Path(self.root).rglob("*"): + if path.is_file(): + with path.open("rb") as handle: + if handle.read(16) == b"SQLite format 3\x00": + databases.append(path) + self.assertEqual(len(databases), 1) + with self.assertRaises(RuntimeError): + personal = memory.Cortex(str(databases[0])) + del personal + gc.collect() + reopened = self.open() + self.reconcile(reopened) + self.assertEqual(len(self.hits(reopened)), 1) + + def test_native_validity_defense_in_depth(self): + now = datetime.datetime.now(datetime.timezone.utc) + stamp = lambda delta: (now + datetime.timedelta(days=delta)).isoformat() + store = self.ready([ + projection("current", valid_from=stamp(-1), valid_until=stamp(1)), + projection("future", valid_from=stamp(1), valid_until=stamp(2)), + projection("expired", valid_from=stamp(-2), valid_until=stamp(-1)), + ]) + self.assertEqual({h["memory_id"] for h in self.hits(store)}, {"current"}) + + def test_long_native_operation_releases_gil(self): + store = self.open() + values = [projection("gil-%d" % i, text="orchid " + "x"*11990, + embedding=[0.001]*4096) for i in range(100)] + stopped = threading.Event() + started = threading.Event() + ticks = [] + def heartbeat(): + started.set() + while not stopped.is_set(): + ticks.append(time.monotonic()) + time.sleep(0.0005) + worker = threading.Thread(target=heartbeat) + worker.start() + started.wait(timeout=5) + old_interval = sys.getswitchinterval() + try: + # Prevent normal Python bytecode switching from masquerading as a + # GIL release inside this one deliberately substantial native call. + sys.setswitchinterval(10.0) + begin = time.monotonic() + store.seed(values) + end = time.monotonic() + finally: + sys.setswitchinterval(old_interval) + stopped.set() + worker.join(timeout=5) + self.assertFalse(worker.is_alive()) + self.assertTrue(any(begin < tick < end for tick in ticks), + "heartbeat made no progress during the native seed call") + store.finish_snapshot(100) + self.reconcile(store) + self.assertEqual(len(self.hits(store, limit=100)), 100) + + def test_process_and_inprocess_lock_released_on_termination(self): + store = self.ready() + self.error("busy", self.open) + competing = self.child("lock") + self.assertEqual(self.line(competing), {"error": "busy"}) + competing.wait(timeout=10) + store.close() + holder = self.child("lock") + self.assertEqual(self.line(holder), {"opened": True}) + self.error("busy", self.open) + holder.kill() + holder.wait(timeout=10) + reopened = self.open() + self.assertFalse(reopened.status()["ready"]) + self.reconcile(reopened) + + def test_threaded_queries_mutations_and_close_drain(self): + store = self.ready([projection()]) + start = threading.Event() + results = [] + def searcher(): + start.wait() + for _ in range(50): + try: + for hit in self.hits(store): + self.assertIn((hit["revision"], hit["text"]), + {(1, "orchid report preference"), (2, "orchid revision two")}) + except ControlledError as error: + self.assertEqual(error.code, "closed") + return + results.append(True) + with concurrent.futures.ThreadPoolExecutor(max_workers=5) as pool: + futures = [pool.submit(searcher) for _ in range(4)] + start.set() + store.upsert(projection(revision=2, text="orchid revision two"), 101) + store.close() + for future in futures: + future.result(timeout=20) + reopened = self.open() + self.reconcile(reopened) + self.assertEqual(self.hits(reopened)[0]["revision"], 2) + + def test_child_crash_after_commit_and_during_write(self): + for phase in ("committed", "inflight"): + with self.subTest(phase=phase): + root = str((Path(self.temp.name) / phase).resolve()) + store = self.ready([projection("crash-%d" % i) for i in range(100)], root=root) + original_generation = store.status()["generation"] + store.close() + original_root = self.root + self.root = root + try: + writer = self.child("write") + self.assertEqual(self.line(writer), {"starting": True}) + if phase == "committed": + receipt = self.line(writer)["committed"] + self.assertTrue(receipt["durable"]) + writer.kill() + writer.wait(timeout=10) + reopened = self.open() + state = reopened.status() + self.assertFalse(state["ready"]) + self.assertEqual(state["generation"], original_generation) + self.assertIn(state["position"], (100, 101)) + if phase == "committed": + self.assertEqual(state["position"], 101) + self.error("not_ready", reopened.search, "orchid", [SCOPE]) + self.reconcile(reopened) + hits = self.hits(reopened, limit=100) + self.assertEqual(len(hits), 100) + expected_revision = 2 if state["position"] == 101 else 1 + self.assertEqual({h["revision"] for h in hits}, {expected_revision}) + for hit in hits: + self.assertEqual(hit["text"].startswith("orchid committed revision two"), expected_revision == 2) + reopened.close() + finally: + self.root = original_root + + def test_restored_local_transaction_loss_rejects_remote_cursor(self): + store = self.ready([projection()]) + state = store.status() + store.close() + backup = Path(self.temp.name) / "backup" + shutil.copytree(self.root, backup) + store = self.open() + self.reconcile(store) + store.upsert(projection(revision=2, text="orchid durable later text"), 101) + remote_acknowledged_position = store.status()["position"] + store.close() + shutil.rmtree(self.root) + shutil.copytree(backup, self.root) + restored = self.open() + self.assertEqual(restored.status()["position"], 100) + self.assertFalse(restored.status()["ready"]) + self.error("conflict", restored.reconcile, state["generation"], remote_acknowledged_position) + self.error("not_ready", restored.search, "orchid", [SCOPE]) + # A proven retained replay may repair the replica before reconciliation. + restored.upsert(projection(revision=2, text="orchid durable later text"), 101) + restored.reconcile(state["generation"], remote_acknowledged_position) + self.assertEqual(self.hits(restored)[0]["revision"], 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 8b1340b..65b64bf 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -8,13 +8,15 @@ zero telemetry, cryptographically hardened, defense-in-depth. This is the prioritized forward plan. The per-iteration security audits (`SECURITY_AUDIT_ITER_*.md`) are the historical record; this is where we're going next. -### Bot memory integration proposal (2026-10-22) +### Bot memory integration (2026-10-22) [Implementation plan](design/bot-memory-integration.md): reuse Cortex as an optional, controlled retrieval index behind the Bot MemoryManager. PostgreSQL keeps memory and permission authority; existing personal Cortex behavior remains compatible. W0-W4 cover -the first retrieval delivery; inference/tidy is a separate W5 enhancement. Design only: -no implementation, release, or deployment, and no change to the shipped status below. +the full retrieval delivery; Cortex W1/W2 are implemented on the integration branch, +while Bot/Studio W3/W4 remain caller-owned. See the [API contract](controlled-memory-api.md). +Inference/tidy is a separate disabled W5 enhancement. No package publication or Bot +deployment is claimed, and the historical shipped status below is unchanged. --- diff --git a/docs/controlled-memory-api.md b/docs/controlled-memory-api.md new file mode 100644 index 0000000..d50e47b --- /dev/null +++ b/docs/controlled-memory-api.md @@ -0,0 +1,80 @@ +# Controlled memory API + +Implementation contract for Cortex-side P0. Python distribution version: **2.5.0** (this branch; not yet published on PyPI). The installed `cortex_ai_memory` module exposes `ControlledStore` in addition to the unchanged personal `Cortex` class. The integration plan remains the scope/acceptance reference. Bot/Studio integration, PG authorization and candidate inference are outside this package. + +## Install the reviewed build + +The existing PyPI 2.4.0 release does not contain this API. Use the exact 2.5.0 wheel +from the reviewed PR's `controlled-wheel-*` CI artifact for your platform, or build +from the reviewed source commit. The CI covers Python 3.12 on Linux x86_64 and macOS +arm64. Wheel files are platform/interpreter specific; do not install an incompatible +artifact or assume other targets have been verified. + +```sh +python3.12 -m venv .venv +.venv/bin/python -m pip install 'maturin>=1.0,<2.0' +.venv/bin/maturin build --release -m cortex-python/Cargo.toml --out dist --interpreter .venv/bin/python +.venv/bin/python -m pip install --force-reinstall dist/cortex_ai_memory-2.5.0-*.whl +.venv/bin/python -c 'from cortex_ai_memory import ControlledStore, ControlledError' +``` + +Do not use `pip install cortex-ai-memory` as proof this API is available until 2.5.0 +has actually been published. Use a clean wheel output directory and retain the source +SHA and artifact checksum with the host's existing dependency manifest. + +## Store and lifecycle + +`ControlledStore(root, owner_type, owner_id, bot_id, replica_id)` opens a dedicated owner/bot/replica database under an absolute, private root. Paths inside that root are generated from the identity. The root must not be a symlink or contain `..`; resolve platform temporary-directory aliases before passing it. One process/handle may own a replica at a time. Different replicas have separate generation IDs and cursors. A personal database is never imported. Current legacy open/export paths refuse controlled databases. Older binaries must not be given these files. + +- `status()` returns `{generation, position, sealed, ready}`. Every reopen starts not ready. +- `seed(projections)` atomically loads a batch of current authoritative memories while the snapshot is unsealed. Retrying identical revisions is safe. No seed after snapshot completion. +- `finish_snapshot(position)` durably seals a new snapshot at the PG committed snapshot watermark; does not enable queries. Repeating the same watermark succeeds; changing it after sealing conflicts. +- `reconcile(generation, position)` enables queries only after both exactly match local durable status and snapshot is sealed. An unsealed snapshot raises `not_ready`; a generation/position mismatch raises `conflict` and queries remain disabled. The caller must validate the source snapshot, reconcile retained changes or rebuild a new replica before this call; equality alone does not authorize data. +- `apply_batch(mutations, position)` durably and atomically applies the next sequential replay position. Empty batches can advance a position (e.g. an irrelevant event). Equal latest position is accepted only for the identical normalized batch and returns its original receipt. Older/gapped positions conflict without modifying data. Upsert/delete convenience methods accept one projection/deletion and the same position. +- `close()` refuses subsequent calls and drains current native operations before releasing the store lock. A new store object can reopen/reconcile the replica. Use a new replica ID for a fresh rebuild, reconcile it, atomically route new calls to it, drain/close the previous object, then retire old files under retention policy. No unsafe in-place file swap API is exposed. + +Each projection has `memory_id` (opaque external ID), positive integer `revision`, `scope: {kind,id}`, `kind` (`fact`, `preference`, `experience`, `method`), `text`, `source_refs: [{kind,provider,id}]`, optional RFC3339 `valid_from`/`valid_until`, and optional numeric `embedding`. Missing validity bounds are unbounded. Validity is `[valid_from, valid_until)`. Scope kind is `owner`, `entity`, `project`, `workspace`, or `user`; owner scope ID must equal the store owner ID. Scope never changes for an existing memory ID, including after deletion. To change it, withdraw that identity and create another. Projection state is implicitly active; no inferred/candidate memory is accepted automatically. + +A deletion is `{memory_id, revision, scope}` and retains an immutable-scope tombstone without text. Even unseen deletions require canonical scope. Lower memory revisions return a stale outcome; equal revision with changed semantic content conflicts; identical revision/content replays. A higher same-scope revision can restore a memory only when the caller has confirmed reactivation in PG. Different external IDs with identical text never merge. + +Mutations are `{"op":"upsert","projection":...}` or `{"op":"delete","deletion":...}`. A receipt is `{generation, position, durable:true, outcomes:[{memory_id,revision,state,outcome}]}`; state is `live`/`withdrawn`, outcome is `applied`/`replayed`/`stale`. Receipts contain no memory text. The local payload, revision fence, receipt and replay position commit in one SQLite WAL transaction with synchronous FULL. Only then may the caller acknowledge PG delivery. After restore, compare local status against the remote cursor and reconcile/rebuild; never trust the remote cursor alone. + +## Retrieval + +`search(query, allowed_scopes, limit=10, embedding=None, max_candidates=1000, max_duration_ms=250)` returns `{hits, degraded, truncated}`. Hits include owner_type/owner_id/bot_id, memory_id, revision, scope, kind, text, source_refs, validity bounds and score. Native methods release the Python GIL. Call them through bounded host worker capacity, not unlimited async tasks. Cancellation cannot roll back an already committed mutation; retry the original position/batch. + +`allowed_scopes` is required, supplied by the trusted host, and an empty list returns no hits. Candidates are filtered in SQL for scopes and current validity bounds before ranking/top-k; disabled graph/unscoped expansion cannot contribute. The existing FTS, vector index and ranking machinery are reused. FTS contributes a match signal without global BM25 statistics; bounded authorized candidates feed a per-query vector index, so excluded records do not influence ranking. This initial backend prioritizes isolation over an unbounded persistent vector index. Work is bounded by candidate/time limits; exhaustion reports degradation, never proof that no memory exists. Semantic lookup uses explicitly supplied vectors; no model download/inference occurs. Cortex enforces one vector dimension per store; the host must also keep the embedding model consistent. Model changes require rebuilding with new vectors. + +Every hit is an **internal candidate**. Before any UI/log/context exposure, the Bot must reload current PG state, validate identity/scope/source access, active/withdrawn state, exact revision and both validity bounds using server time, and emit current PG text. Resolve required defaults by exact key from PG independently of Cortex hits/readiness. Recheck validity and withdrawal at model/action boundaries. Never inject Cortex-generated prompts alongside ContextProjector. If lookup is unavailable or incomplete use the existing bounded PG fallback; authority lookup failures fail closed. + +## Limits and errors + +Identifiers: 1–256 UTF-8 bytes, no control characters. Owner/scope kinds are constrained as above. Text: at most 12,000 characters / 48,000 bytes. Sources: 1–16; each field uses identifier limits. Batches: at most 100 entries. Query: 1–4,000 characters and at most 64 alphanumeric terms; limit 1–100; at most 64 allowed scopes; candidates 1–10,000; time budget 1–250 ms; embeddings 1–4,096 finite floats with finite squared norm, with a consistent store-wide dimension. Parsed JSON request size is capped at 8 MiB. Duplicate memory IDs inside one batch are rejected. Unknown fields, bool/fractional/negative revisions or positions, invalid intervals and malformed vectors fail validation; source order is normalized for replay identity. + +`ControlledError` has stable `.code`: `invalid_input`, `conflict`, `busy`, `not_ready`, `closed`, or `storage`. Stale memory revisions are receipt outcomes, not exceptions. Storage/commit ambiguity disables queries until reopen/reconciliation. No error text contains projection bodies. Python integer positions/revisions preserve precision up to SQLite's signed 64-bit maximum. + +## Example (after installation) + +```python +from cortex_ai_memory import ControlledStore + +store = ControlledStore('/srv/private/cortex', 'merchant', 'm1', 'b1', 'worker1') +scope = {'kind': 'owner', 'id': 'm1'} +projection = { + 'memory_id': 'lang', 'revision': 1, 'scope': scope, + 'kind': 'preference', 'text': 'Reports use English', + 'source_refs': [{'kind': 'input', 'provider': 'bot', 'id': 'input1'}], +} +store.seed([projection]) +store.finish_snapshot(100) # real PG snapshot watermark, not a guessed event number +state = store.status() +store.reconcile(state['generation'], 100) # only after authoritative reconciliation +hits = store.search('Reports', [scope]) +# Validate each candidate against current PG permissions/state before use. +store.delete({'memory_id': 'lang', 'revision': 2, 'scope': scope}, 101) +store.close() +``` + +The current wheel uses plaintext SQLite; private filesystem controls are mandatory. No automatic sync/export/inference/consolidation is enabled. Logical withdrawal does not promise immediate erasure from backups or already-sent model requests. See the [integration plan](design/bot-memory-integration.md) for host-owned replay, rebuild, rollout and real-PG acceptance. + +The existing personal API constructor is `Cortex(db_path)`; it raises `RuntimeError` for a controlled database. It has no explicit close method; release its references when no longer used. diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index ecb1f70..e6499f5 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -1,6 +1,6 @@ # Cortex integration with Bot memory: implementation plan -Date: 2026-10-22. Status: design only; no runtime implementation, migration, release, or deployment. +Date: 2026-10-22. Status: Cortex controlled core and Python binding implemented on this branch; host integration, package publication and deployment are not completed. ## 1. Goal and fixed boundaries @@ -27,13 +27,13 @@ Cortex baseline: `00f0e38da36bdafc8b40cf235aaac9e53bdf8548` (origin/main when pl | [storage](../../cortex-core/src/storage/sqlite.rs) and [storage trait](../../cortex-core/src/storage/traits.rs) | SQLite transactions, CRUD, FTS triggers, memory cache | Atomic projection payload/state changes, stale-write fencing, and controlled-store lifecycle | | [core deletion](../../cortex-core/src/lib.rs), `delete_memory` | SQL removal, index removal, cache generation, deletion event | External identity/tombstone handling and crash/race acceptance; a normal delete alone does not reject a later stale projection | | [Python binding](../../cortex-python/src/lib.rs) | Existing PyO3 package and native engine | Current API lacks namespace arguments and delete exposure; retrieve returns only ID/score/text, without verification metadata | -| [HTTP guard](../../cortex-http/src/main.rs), `guard_origin` | Preserve existing local-only use | The local HTTP API has no authentication and can read Private memories; it is not a tenant service | +| [HTTP guard](../../cortex-http/src/main.rs), local server entrypoint | Preserve existing local-only use | The local HTTP API has no authentication and can read Private memories; it is not a tenant service | The Python README example currently uses `cortex_python.PyCortex`, while the module/class declarations are `cortex_ai_memory.Cortex`. Verify the installed wheel and repair examples as part of the binding package work; do not infer installability from a source import. An open wheel-build PR (#14 at planning time) owns macOS deployment-target fixes. Coordinate with that work instead of duplicating or overwriting its changes. This plan makes no claims about current published wheel availability. -The original local checkout also contains security commits outside this main baseline, including `c1d8658` (owner-only database files) and `5f522d6` (plaintext deletion). Verify their current PR/merge ownership before implementing related protections. They are not treated as shipped main capabilities here, and this plan neither copies nor overrides them. +The original owner-only database and plaintext-deletion fixes (`c1d8658`, `5f522d6`) have been reused as original cherry-picked commits in this branch, with conflicts resolved to preserve current main. Their tests and full diff are included in implementation review; they were not shipped main capabilities at the planning baseline. ## 3. First delivery and responsibilities @@ -48,7 +48,7 @@ Use the existing Python binding as the first integration surface. Extend it with | Studio | Install the reviewed chatbot-agent/vendor package using the existing release process; retain current identity and permissions; no direct Cortex browser calls | | Bot design owner | Record the optional retrieval backend and precise acceptance mapping in the existing design/roadmap | -All API names below are proposed. No listed method or acceptance result is implemented by this document. +The implemented Cortex/Python surface is specified in the [controlled memory API](../controlled-memory-api.md). Host PostgreSQL delivery and authorization behavior below remains the integration contract; no host or real-account acceptance is implied. ## 4. Controlled projection contract (P0) @@ -109,7 +109,7 @@ Logical withdrawal and physical erasure are distinct: no immediate wiping of exi ## 5. Python delivery and operational fallback (P0) -Expose controlled open/upsert/delete/search/status/close through one documented binding; choose final names in the first contract package. Errors distinguish invalid scope/input, stale revision, equal-revision conflict, busy/not-ready store, and bounded retrieval degradation. Search results must not require parsing a generated prompt or tuple text to recover identity. +The Python `ControlledStore` binding exposes open/seed/finish_snapshot/reconcile/apply_batch/upsert/delete/search/status/close; exact dictionary schemas and limits are in the [API contract](../controlled-memory-api.md). Errors distinguish invalid scope/input, stale revision, equal-revision conflict, busy/not-ready store, and bounded retrieval degradation. Search results must not require parsing a generated prompt or tuple text to recover identity. P0 runtime uses a supported wheel installed in a clean environment, not an adjacent Rust checkout. Add a built-wheel smoke/integration test covering import/class name, controlled APIs, restart, and scope/delete behavior. Rust workspace CI excludes cortex-python and therefore cannot substitute for this test. Verify the host's actual Python, OS, and architecture; add the binding job to the required integration checks rather than treating a skipped release workflow as acceptance. Coordinate macOS fixes with PR #14. @@ -142,7 +142,7 @@ Implement on isolated branches against current main, preserving others' work. Ea W1 and W2 follow W0; W3 can use contract fixtures while W1/W2 develop, but integration waits for both. W4 waits for W1-W3. W5 is optional after W4 and cannot block the P0 retrieval release. -## 8. Acceptance matrix (required, not executed) +## 8. Acceptance matrix and ownership Use synthetic memory identities/text, real temporary SQLite and real test PostgreSQL where indicated. No production archive import, private model photos, paid generation, or publishing is necessary. Failure injection uses actual process termination and real files/transactions, not mocks in place of storage acceptance. @@ -171,9 +171,11 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr An acceptance subagent must be context-isolated from implementation: give it this plan and public schemas/docs, not implementation diffs or unit tests. Implementers write unit tests separately. Adversarial review is a third independent context, per repository rules. +Cortex-side automated evidence includes the independent installed-wheel acceptance suite (16 cases covering C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. + ## 9. Review, CI, rollout and evidence -For this documentation change, validate local links, fences, acceptance IDs, and `git diff --check`. No business acceptance is claimed. Keep the exact reviewed document content and head/base identifiers with raw model outputs in a traceable review record. Each model first reviews independently; cross-check only after both initial findings are saved. Fix every valid actionable finding at every severity, then repeat both reviews over the entire final diff; disputed findings require the originating reviewer to confirm closure. +For this implementation, validate local links, fences, acceptance IDs and `git diff --check`, plus the required source and installed-wheel checks. No Bot/Studio business acceptance is claimed. Keep the exact reviewed document content and head/base identifiers with raw model outputs in a traceable review record. Each model first reviews independently; cross-check only after both initial findings are saved. Fix every valid actionable finding at every severity, then repeat both reviews over the entire final diff; disputed findings require the originating reviewer to confirm closure. Before any implementation merge, require two different underlying reviewer models independently clean on the final complete head/base and all required CI on that same integration version. Recheck head/base, conflicts, findings and checks immediately before merging with exact-head constraint. Follow the authorized admin merge policy only after these gates; do not bypass failed/missing/skipped required checks. A main failure or base movement requires investigation and fresh evidence, not inherited clean conclusions. @@ -181,4 +183,4 @@ Required implementation verification includes `cargo test --workspace --exclude Rollout uses a clean main containing the latest release and the existing package/vendor process: test environment first, then explicitly authorized scope. Record Cortex/chatbot-agent/Studio SHAs, wheel identity, flags, storage/replica configuration, CI runs, each reviewer model/round, fault-injection results and unresolved limits. Verify actual installed package, processes, health, replay lag, cleanup failures, logs and recall/correction/forget behavior after release. Separate implementation, isolated tests, deployment and real authorized end-to-end acceptance; anything not deployed must be labeled not deployed. -Completion of this plan means a reviewed design artifact only. Completion of P0 requires W0-W4 and C01-C18 evidence; P1 remains disabled until W5/C19-C20 also pass. This document grants no production operation or data-sharing authorization. +Cortex-side completion covers W1-W2: controlled storage/search/lifecycle, binding and installed-wheel tests. Full Bot P0 still requires host-owned W3-W4 and real-PG C14-C18 evidence. C19-C20/P1 remain disabled. See the public API contract for caller responsibilities; this code does not grant production-operation or data-sharing authorization. diff --git a/docs/guides/integrations.md b/docs/guides/integrations.md index f2faa8c..460e364 100644 --- a/docs/guides/integrations.md +++ b/docs/guides/integrations.md @@ -4,9 +4,10 @@ Cortex provides persistent, local-first memory for AI agents. This guide shows h For hosted Bots that already own memory and permissions in PostgreSQL, follow the [controlled Bot integration architecture and plan](../design/bot-memory-integration.md). -That proposed path uses Cortex only as a derived retrieval backend, with mandatory -authoritative scope/version/withdrawal checks. Its controlled APIs are not implemented -yet; the personal-agent examples below do not provide that hosted isolation contract. +That path uses Cortex only as a derived retrieval backend, with mandatory authoritative +scope/version/withdrawal checks. The [ControlledStore API](../controlled-memory-api.md) +defines the implemented Cortex-side contract; the host adapter remains the caller's +responsibility. The personal-agent examples below do not provide hosted isolation. **What Cortex adds to any framework:** @@ -50,10 +51,13 @@ pip install crewai cortex-ai-memory ```python from crewai import Agent, Task, Crew -from cortex_ai_memory import PyCortex +from cortex_ai_memory import Cortex +from pathlib import Path # Initialize Cortex (creates or opens a local SQLite database) -cx = PyCortex("~/.cortex/crewai.db") +memory_dir = Path.home() / ".cortex" +memory_dir.mkdir(mode=0o700, parents=True, exist_ok=True) +cx = Cortex(str(memory_dir / "crewai.db")) def cortex_recall(query: str) -> str: @@ -123,9 +127,12 @@ cx.add_preference("report_style", "concise with bullet points", 0.9) ### Storing Facts and Beliefs Between Runs ```python -from cortex_ai_memory import PyCortex +from cortex_ai_memory import Cortex +from pathlib import Path -cx = PyCortex("~/.cortex/crewai.db") +memory_dir = Path.home() / ".cortex" +memory_dir.mkdir(mode=0o700, parents=True, exist_ok=True) +cx = Cortex(str(memory_dir / "crewai.db")) # Store structured knowledge that agents can query later cx.add_fact("Alice", "works_at", "Stripe", 0.95, "crewai") @@ -165,10 +172,13 @@ from autogen_agentchat.agents import AssistantAgent from autogen_agentchat.teams import RoundRobinGroupChat from autogen_agentchat.conditions import TextMentionTermination from autogen_ext.models.openai import OpenAIChatCompletionClient -from cortex_ai_memory import PyCortex +from cortex_ai_memory import Cortex +from pathlib import Path # Initialize Cortex -cx = PyCortex("~/.cortex/autogen.db") +memory_dir = Path.home() / ".cortex" +memory_dir.mkdir(mode=0o700, parents=True, exist_ok=True) +cx = Cortex(str(memory_dir / "autogen.db")) def memory_search(query: str) -> str: @@ -242,9 +252,12 @@ asyncio.run(main()) ### Session Lifecycle Pattern ```python -from cortex_ai_memory import PyCortex +from cortex_ai_memory import Cortex +from pathlib import Path -cx = PyCortex("~/.cortex/autogen.db") +memory_dir = Path.home() / ".cortex" +memory_dir.mkdir(mode=0o700, parents=True, exist_ok=True) +cx = Cortex(str(memory_dir / "autogen.db")) # --- Start of session: load context --- context = cx.get_context(2000) From c4b00c1691b5c04189f0cba20d809a8955bab24b Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 11:52:46 -0400 Subject: [PATCH 09/15] fix: bound controlled vector scoring and native lifecycle checks --- cortex-core/src/controlled.rs | 63 ++++++++++++++------ cortex-core/src/retrieval.rs | 4 +- cortex-python/src/lib.rs | 4 +- cortex-python/tests/test_controlled_store.py | 33 ++++++++++ docs/controlled-memory-api.md | 2 +- docs/design/bot-memory-integration.md | 2 +- 6 files changed, 86 insertions(+), 22 deletions(-) diff --git a/cortex-core/src/controlled.rs b/cortex-core/src/controlled.rs index 4d490cb..fc7b936 100644 --- a/cortex-core/src/controlled.rs +++ b/cortex-core/src/controlled.rs @@ -1,7 +1,10 @@ //! A disposable, scope-constrained projection of host-authoritative memory. //! Never invokes ingest, plugins, inference, sync, or consolidation. use crate::retrieval::{QueryBudget, RetrievalEngine, RetrievalQuery}; -use crate::storage::{memory_index::MemoryIndex, sqlite::SqliteStorage}; +use crate::storage::{ + memory_index::{cosine_similarity, MemoryIndex}, + sqlite::SqliteStorage, +}; use crate::types::{MemContent, MemObject, MemObjectBuilder, MemSource, MemoryTier}; use chrono::{DateTime, Utc}; use parking_lot::Mutex; @@ -48,8 +51,11 @@ impl From for ControlledError { } } impl From for ControlledError { - fn from(_: crate::CortexError) -> Self { - Self::Storage + fn from(error: crate::CortexError) -> Self { + match error { + crate::CortexError::InvalidInput(_) => Self::InvalidInput("database mode"), + _ => Self::Storage, + } } } type Fence = (String, i64, Option, bool, Option); @@ -398,6 +404,10 @@ impl ControlledStore { s.closed = true; } pub fn reconcile(&self, generation: &str, position: i64) -> Result { + ident(generation)?; + if position < 0 { + return Err(ControlledError::InvalidInput("position")); + } let mut s = self.state.lock(); s.ready = false; let v = Self::status_locked(&s)?; @@ -564,7 +574,16 @@ impl ControlledStore { for scope in scopes { scope_check(scope, &self.identity)?; } - let s = self.state.lock(); + let started = Instant::now(); + let duration = Duration::from_millis(max_duration_ms); + let deadline = started + duration; + let Some(s) = self.state.try_lock_until(deadline) else { + return Ok(SearchResult { + hits: vec![], + degraded: true, + truncated: true, + }); + }; let status = Self::status_locked(&s)?; if !status.ready { return Err(ControlledError::NotReady); @@ -577,8 +596,6 @@ impl ControlledStore { if scopes.is_empty() { return Ok(empty()); } - let started = Instant::now(); - let duration = Duration::from_millis(max_duration_ms); let mut scope_json: Vec = scopes.iter().map(json).collect::>()?; scope_json.sort(); scope_json.dedup(); @@ -617,37 +634,49 @@ impl ControlledStore { truncated: true, }); }; - let truncated = fetched.len() > max_candidates; + let mut truncated = fetched.len() > max_candidates; fetched.truncate(max_candidates); + // Share the existing scorer and cosine math, but never construct an HNSW + // graph for a request-local candidate set. Each vector is bounded to 4096 + // dimensions and the deadline is checked between candidates. let index = MemoryIndex::new(); let mut mems = HashMap::new(); let mut lexical = HashMap::new(); let mut scores = HashMap::new(); for (mem, matched) in fetched { - if let Some(v) = &mem.embedding { - index.insert_arc(mem.id, v); + if Instant::now() >= deadline { + truncated = true; + break; } + let score = match (&embedding, &mem.embedding) { + (Some(query), Some(vector)) => cosine_similarity(query, vector), + _ => 0.5, + }; if matched { lexical.insert(mem.id, 1.0); } - scores.insert(mem.id, 0.5); + scores.insert(mem.id, score); mems.insert(mem.id, Arc::new(mem)); } - if let Some(v) = &embedding { - for (id, score) in index.search(v, max_candidates) { - scores.insert(id, score); - } - } let mut q = RetrievalQuery::new(query, limit); q.embedding = embedding; q.budget = QueryBudget { max_candidates, max_duration: duration, }; - let ranked = RetrievalEngine::new(&s.storage, &index) - .rank_candidates(&q, &scores, &mems, &lexical)?; + let ranked = RetrievalEngine::new(&s.storage, &index).rank_candidates( + &q, + &scores, + &mems, + &lexical, + Some(deadline), + )?; let mut hits = Vec::new(); for r in ranked { + if Instant::now() >= deadline { + truncated = true; + break; + } let projection: Projection = serde_json::from_value( r.memory .metadata diff --git a/cortex-core/src/retrieval.rs b/cortex-core/src/retrieval.rs index ef9e64d..f52b32c 100644 --- a/cortex-core/src/retrieval.rs +++ b/cortex-core/src/retrieval.rs @@ -283,13 +283,14 @@ impl<'a> RetrievalEngine<'a> { let memories = self.storage.get_memories_batch(&all_ids)?; let mem_map: HashMap> = memories.into_iter().map(|m| (m.id, Arc::new(m))).collect(); - self.rank_candidates(query, &candidate_scores, &mem_map, &fts_scores) + self.rank_candidates(query, &candidate_scores, &mem_map, &fts_scores, None) } /// Shared ranking for candidates already constrained by a trusted scope predicate. pub(crate) fn rank_candidates( &self, query: &RetrievalQuery, candidate_scores: &HashMap, mem_map: &HashMap>, fts_scores: &HashMap, + deadline: Option, ) -> Result, CortexError> { let temporal_intent = detect_temporal_intent(&query.text); // Adapt weights for temporal queries @@ -353,6 +354,7 @@ impl<'a> RetrievalEngine<'a> { // Score each candidate let mut results = Vec::new(); for (&id, &sim_score) in candidate_scores { + if deadline.is_some_and(|end| std::time::Instant::now() >= end) { break; } if let Some(mem) = mem_map.get(&id) { // Namespace isolation: skip memories that don't match requested namespace if let Some(ref ns) = query.namespace { diff --git a/cortex-python/src/lib.rs b/cortex-python/src/lib.rs index 5ee8503..09f7828 100644 --- a/cortex-python/src/lib.rs +++ b/cortex-python/src/lib.rs @@ -50,7 +50,7 @@ impl PyCortex { embedding: Option>, ) -> PyResult> { let pid = person_id - .map(|s| uuid::Uuid::parse_str(s)) + .map(uuid::Uuid::parse_str) .transpose() .map_err(|e| PyRuntimeError::new_err(format!("Invalid UUID: {}", e)))?; @@ -90,7 +90,7 @@ impl PyCortex { person_id: Option<&str>, ) -> PyResult { let pid = person_id - .map(|s| uuid::Uuid::parse_str(s)) + .map(uuid::Uuid::parse_str) .transpose() .map_err(|e| PyRuntimeError::new_err(format!("Invalid UUID: {}", e)))?; diff --git a/cortex-python/tests/test_controlled_store.py b/cortex-python/tests/test_controlled_store.py index 36ba0e4..3acdd9b 100644 --- a/cortex-python/tests/test_controlled_store.py +++ b/cortex-python/tests/test_controlled_store.py @@ -367,6 +367,39 @@ def test_native_validity_defense_in_depth(self): ]) self.assertEqual({h["memory_id"] for h in self.hits(store)}, {"current"}) + def test_large_semantic_search_obeys_budget_and_preserves_responsiveness(self): + store = self.open() + vector = [0.01 * ((i % 7) + 1) for i in range(128)] + for offset in range(0, 1100, 100): + store.seed([projection("budget-%d" % i, embedding=vector) + for i in range(offset, offset + 100)]) + store.finish_snapshot(100) + self.reconcile(store) + begin = time.monotonic() + result = store.search("orchid", [SCOPE], limit=100, embedding=vector, + max_candidates=1100, max_duration_ms=250) + elapsed = time.monotonic() - begin + # Allow substantial CI scheduling/serialization overhead beyond the + # native budget; this catches runaway work, not a real-time SLA. + self.assertLess(elapsed, 2.0, "semantic search exceeded bounded-work allowance") + self.assertLessEqual(len(result["hits"]), 100) + self.assertIsInstance(result["degraded"], bool) + self.assertIsInstance(result["truncated"], bool) + if elapsed >= 0.250: + self.assertTrue(result["degraded"] or result["truncated"], + "budget exhaustion must be explicit") + self.assertTrue(all(hit["scope"] == SCOPE for hit in result["hits"])) + begin = time.monotonic() + receipt = store.upsert(projection("after-budget-search", embedding=vector), 101) + self.assertLess(time.monotonic() - begin, 2.0, + "search left later mutation unresponsive") + self.assertTrue(receipt["durable"]) + self.assertEqual(store.status()["position"], 101) + begin = time.monotonic() + store.close() + self.assertLess(time.monotonic() - begin, 2.0, + "search left close unresponsive") + def test_long_native_operation_releases_gil(self): store = self.open() values = [projection("gil-%d" % i, text="orchid " + "x"*11990, diff --git a/docs/controlled-memory-api.md b/docs/controlled-memory-api.md index d50e47b..25f6046 100644 --- a/docs/controlled-memory-api.md +++ b/docs/controlled-memory-api.md @@ -43,7 +43,7 @@ Mutations are `{"op":"upsert","projection":...}` or `{"op":"delete","deletion":. `search(query, allowed_scopes, limit=10, embedding=None, max_candidates=1000, max_duration_ms=250)` returns `{hits, degraded, truncated}`. Hits include owner_type/owner_id/bot_id, memory_id, revision, scope, kind, text, source_refs, validity bounds and score. Native methods release the Python GIL. Call them through bounded host worker capacity, not unlimited async tasks. Cancellation cannot roll back an already committed mutation; retry the original position/batch. -`allowed_scopes` is required, supplied by the trusted host, and an empty list returns no hits. Candidates are filtered in SQL for scopes and current validity bounds before ranking/top-k; disabled graph/unscoped expansion cannot contribute. The existing FTS, vector index and ranking machinery are reused. FTS contributes a match signal without global BM25 statistics; bounded authorized candidates feed a per-query vector index, so excluded records do not influence ranking. This initial backend prioritizes isolation over an unbounded persistent vector index. Work is bounded by candidate/time limits; exhaustion reports degradation, never proof that no memory exists. Semantic lookup uses explicitly supplied vectors; no model download/inference occurs. Cortex enforces one vector dimension per store; the host must also keep the embedding model consistent. Model changes require rebuilding with new vectors. +`allowed_scopes` is required, supplied by the trusted host, and an empty list returns no hits. Candidates are filtered in SQL for scopes and current validity bounds before ranking/top-k; disabled graph/unscoped expansion cannot contribute. The existing FTS, cosine-similarity and ranking machinery are reused. FTS contributes a match signal without global BM25 statistics; only bounded authorized candidates are scored, so excluded records do not influence ranking. No request-local HNSW graph is built. The soft deadline covers lock acquisition, SQL work, vector scoring, ranking and hit projection, with cooperative checks between bounded work units; it is not a hard real-time OS scheduling SLA. Work is bounded by candidate/time limits; exhaustion reports degradation, never proof that no memory exists. Semantic lookup uses explicitly supplied vectors; no model download/inference occurs. Cortex enforces one vector dimension per store; the host must also keep the embedding model consistent. Model changes require rebuilding with new vectors. Every hit is an **internal candidate**. Before any UI/log/context exposure, the Bot must reload current PG state, validate identity/scope/source access, active/withdrawn state, exact revision and both validity bounds using server time, and emit current PG text. Resolve required defaults by exact key from PG independently of Cortex hits/readiness. Recheck validity and withdrawal at model/action boundaries. Never inject Cortex-generated prompts alongside ContextProjector. If lookup is unavailable or incomplete use the existing bounded PG fallback; authority lookup failures fail closed. diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index e6499f5..620cb20 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -171,7 +171,7 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr An acceptance subagent must be context-isolated from implementation: give it this plan and public schemas/docs, not implementation diffs or unit tests. Implementers write unit tests separately. Adversarial review is a third independent context, per repository rules. -Cortex-side automated evidence includes the independent installed-wheel acceptance suite (16 cases covering C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. +Cortex-side automated evidence includes the independent installed-wheel acceptance suite (17 cases covering C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. ## 9. Review, CI, rollout and evidence From 534d13ba7999a100199196321bc893673302f82a Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 12:22:23 -0400 Subject: [PATCH 10/15] fix: close controlled memory privacy and lifecycle review findings --- .github/workflows/ci.yml | 10 + Cargo.toml | 5 + cortex-core/src/controlled.rs | 689 +++++++++++++----- cortex-core/src/retrieval.rs | 258 +++++-- cortex-core/src/storage/sqlite.rs | 448 +++++++----- cortex-core/tests/test_db_file_permissions.rs | 39 +- cortex-python/src/lib.rs | 220 +++++- cortex-python/tests/test_controlled_store.py | 206 ++++++ .../tests/test_foreign_owned_database.py | 52 ++ docs/controlled-memory-api.md | 15 +- docs/design/bot-memory-integration.md | 2 +- 11 files changed, 1491 insertions(+), 453 deletions(-) create mode 100644 cortex-python/tests/test_foreign_owned_database.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4d47338..ed34173 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -64,6 +64,16 @@ jobs: run: | python -m pip install --force-reinstall dist/*.whl python -I "$GITHUB_WORKSPACE/cortex-python/tests/test_controlled_store.py" + - name: Reject writable foreign-owned database (real filesystem) + if: runner.os == 'Linux' + run: | + fixture_dir=$(mktemp -d) + chmod 755 "$fixture_dir" + fixture_path="$fixture_dir/foreign.db" + touch "$fixture_path" + sudo chown 65534:65534 "$fixture_path" + sudo chmod 666 "$fixture_path" + CORTEX_FOREIGN_OWNED_DB="$fixture_path" python -I "$GITHUB_WORKSPACE/cortex-python/tests/test_foreign_owned_database.py" - uses: actions/upload-artifact@v4 with: name: controlled-wheel-${{ matrix.os }} diff --git a/Cargo.toml b/Cargo.toml index 8931172..53f6ebc 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -7,3 +7,8 @@ lto = true codegen-units = 1 strip = true panic = "abort" + +# Keep native wheel symbols: stripping this cdylib can produce a misaligned +# LINKEDIT string table with current macOS linker/install-name tooling. +[profile.release.package.cortex-python] +strip = false diff --git a/cortex-core/src/controlled.rs b/cortex-core/src/controlled.rs index fc7b936..79b0394 100644 --- a/cortex-core/src/controlled.rs +++ b/cortex-core/src/controlled.rs @@ -14,7 +14,7 @@ use sha2::{Digest, Sha256}; use std::collections::{HashMap, HashSet}; use std::fs::File; use std::path::{Component, Path, PathBuf}; -use std::sync::{Arc, OnceLock}; +use std::sync::Arc; use std::time::{Duration, Instant}; use uuid::Uuid; @@ -131,6 +131,7 @@ pub struct Status { pub position: i64, pub sealed: bool, pub ready: bool, + pub cleanup_pending: bool, } #[derive(Clone, Debug, Serialize)] pub struct Hit { @@ -168,23 +169,21 @@ impl Default for SearchOptions { } struct State { - storage: SqliteStorage, + storage: Option, ready: bool, closed: bool, + file: Option, } -static OPEN: OnceLock>> = OnceLock::new(); -struct Registry(PathBuf); -impl Drop for Registry { - fn drop(&mut self) { - OPEN.get().unwrap().lock().remove(&self.0); +impl State { + fn storage(&self) -> Result<&SqliteStorage> { + self.storage.as_ref().ok_or(ControlledError::Closed) } } /// Locks are dropped only after SQLite state and all calls have drained. pub struct ControlledStore { state: Mutex, identity: StoreIdentity, - _file: File, - _registry: Registry, + creator_pid: u32, } fn ident(s: &str) -> Result<()> { @@ -296,6 +295,22 @@ fn private_file(_: &Path) -> Result { )) } +fn effective(p: &Projection, now: DateTime) -> bool { + p.valid_from.is_none_or(|t| t <= now) && p.valid_until.is_none_or(|t| now < t) +} +impl Drop for ControlledStore { + fn drop(&mut self) { + if self.creator_pid != std::process::id() { + // Never invoke copied SQLite state or a copied mutex in a fork child. + // The OS reclaims this child's resources at process exit. + let s = self.state.get_mut(); + if let Some(storage) = s.storage.take() { + std::mem::forget(storage); + } + } + } +} + impl ControlledStore { pub fn open(root: &Path, identity: StoreIdentity) -> Result { for x in [ @@ -318,6 +333,18 @@ impl ControlledStore { if m.file_type().is_symlink() { return Err(ControlledError::InvalidInput("symlink root")); } + #[cfg(unix)] + { + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + let uid = unsafe { libc::geteuid() }; + let mode = m.permissions().mode(); + // An untrusted ancestor owner or writable non-sticky parent + // could replace the private root between validation and open. + if (m.uid() != 0 && m.uid() != uid) || (mode & 0o022 != 0 && mode & 0o1000 == 0) + { + return Err(ControlledError::InvalidInput("untrusted root ancestor")); + } + } } } let mut builder = std::fs::DirBuilder::new(); @@ -337,11 +364,6 @@ impl ControlledStore { } } let path = root.join(format!("{}.db", digest(&identity)?)); - let registry = OPEN.get_or_init(|| Mutex::new(HashSet::new())); - if !registry.lock().insert(path.clone()) { - return Err(ControlledError::Busy); - } - let registry = Registry(path.clone()); let file = private_file(&path.with_extension("lock"))?; file.try_lock().map_err(|_| ControlledError::Busy)?; for suffix in ["", "-wal", "-shm", "-journal"] { @@ -355,7 +377,7 @@ impl ControlledStore { .ok_or(ControlledError::InvalidInput("path encoding"))?, )?; storage.with_controlled_conn(|conn| -> Result<()> { - conn.execute_batch("CREATE TABLE IF NOT EXISTS controlled_meta(id INTEGER PRIMARY KEY CHECK(id=1), identity TEXT NOT NULL, generation TEXT NOT NULL, position INTEGER NOT NULL DEFAULT 0, sealed INTEGER NOT NULL DEFAULT 0, snapshot_position INTEGER, last_batch TEXT, last_receipt TEXT, dimension INTEGER); + conn.execute_batch("CREATE TABLE IF NOT EXISTS controlled_meta(id INTEGER PRIMARY KEY CHECK(id=1), identity TEXT NOT NULL, generation TEXT NOT NULL, position INTEGER NOT NULL DEFAULT 0, sealed INTEGER NOT NULL DEFAULT 0, snapshot_position INTEGER, last_batch TEXT, last_receipt TEXT, dimension INTEGER, cleanup_pending INTEGER NOT NULL DEFAULT 0); CREATE TABLE IF NOT EXISTS controlled_fences(memory_id TEXT PRIMARY KEY, scope TEXT NOT NULL, revision INTEGER NOT NULL, internal_id TEXT, live INTEGER NOT NULL, intent TEXT, valid_from INTEGER, valid_until INTEGER); CREATE INDEX IF NOT EXISTS controlled_scope ON controlled_fences(scope,live,internal_id);")?; conn.execute("INSERT OR IGNORE INTO controlled_meta(id,identity,generation) VALUES(1,?1,?2)",params![json(&identity)?,Uuid::new_v4().to_string()])?; @@ -367,22 +389,30 @@ impl ControlledStore { })?; Ok(Self { state: Mutex::new(State { - storage, + storage: Some(storage), ready: false, closed: false, + file: Some(file), }), identity, - _file: file, - _registry: registry, + creator_pid: std::process::id(), }) } + fn check_process(&self) -> Result<()> { + if self.creator_pid != std::process::id() { + return Err(ControlledError::Conflict( + "inherited handle; open a new replica after fork", + )); + } + Ok(()) + } fn status_locked(s: &State) -> Result { if s.closed { return Err(ControlledError::Closed); } - s.storage.with_controlled_conn(|c| { + s.storage()?.with_controlled_conn(|c| { Ok(c.query_row( - "SELECT generation,position,sealed FROM controlled_meta WHERE id=1", + "SELECT generation,position,sealed,cleanup_pending FROM controlled_meta WHERE id=1", [], |r| { Ok(Status { @@ -390,69 +420,120 @@ impl ControlledStore { position: r.get(1)?, sealed: r.get(2)?, ready: s.ready, + cleanup_pending: r.get(3)?, }) }, )?) }) } pub fn status(&self) -> Result { - Self::status_locked(&self.state.lock()) + self.check_process()?; + let mut s = self.state.lock(); + let result = Self::status_locked(&s); + Self::complete(&mut s, result) + } + fn complete(s: &mut State, result: Result) -> Result { + if matches!(result, Err(ControlledError::Storage)) { + s.ready = false; + } + result } - pub fn close(&self) { + fn cleanup_locked(s: &State) -> Result<()> { + s.storage()?.with_controlled_conn(|conn| -> Result<()> { + let pending: bool = conn.query_row( + "SELECT cleanup_pending FROM controlled_meta WHERE id=1", + [], + |r| r.get(0), + )?; + if pending && SqliteStorage::scrub_after_delete(conn) { + // All calls are serialized by State; a newer delete cannot race + // the checkpoint and have its pending flag cleared by this call. + conn.execute( + "UPDATE controlled_meta SET cleanup_pending=0 WHERE id=1", + [], + )?; + } + Ok(()) + }) + } + pub fn retry_cleanup(&self) -> Result { + self.check_process()?; + let mut s = self.state.lock(); + let result = (|| { + Self::cleanup_locked(&s)?; + Self::status_locked(&s) + })(); + Self::complete(&mut s, result) + } + pub fn close(&self) -> Result<()> { + self.check_process()?; let mut s = self.state.lock(); s.ready = false; s.closed = true; + // The operation mutex drains executing calls. Queued calls see Closed; + // no old Arc can retain SQLite or the OS lock after close returns. + drop(s.storage.take()); + drop(s.file.take()); + Ok(()) } pub fn reconcile(&self, generation: &str, position: i64) -> Result { + self.check_process()?; ident(generation)?; if position < 0 { return Err(ControlledError::InvalidInput("position")); } let mut s = self.state.lock(); - s.ready = false; - let v = Self::status_locked(&s)?; - if !v.sealed { - return Err(ControlledError::NotReady); - } - if position < 0 || v.generation != generation || v.position != position { - return Err(ControlledError::Conflict( - "reconciliation position or generation", - )); - } - s.ready = true; - Self::status_locked(&s) + let operation = (|| { + s.ready = false; + let v = Self::status_locked(&s)?; + if !v.sealed { + return Err(ControlledError::NotReady); + } + if position < 0 || v.generation != generation || v.position != position { + return Err(ControlledError::Conflict( + "reconciliation position or generation", + )); + } + s.ready = true; + Self::status_locked(&s) + })(); + Self::complete(&mut s, operation) } pub fn finish_snapshot(&self, position: i64) -> Result { + self.check_process()?; if position < 0 { return Err(ControlledError::InvalidInput("position")); } let mut s = self.state.lock(); - let v = Self::status_locked(&s)?; - if v.sealed { - let original: i64 = s.storage.with_controlled_conn(|c| -> Result { - Ok(c.query_row( - "SELECT snapshot_position FROM controlled_meta WHERE id=1", - [], - |r| r.get(0), - )?) - })?; - if original == position { - return Ok(v); + let operation = (|| { + let v = Self::status_locked(&s)?; + if v.sealed { + let original: i64 = s.storage()?.with_controlled_conn(|c| -> Result { + Ok(c.query_row( + "SELECT snapshot_position FROM controlled_meta WHERE id=1", + [], + |r| r.get(0), + )?) + })?; + if original == position { + return Ok(v); + } + return Err(ControlledError::Conflict("snapshot already sealed")); } - return Err(ControlledError::Conflict("snapshot already sealed")); - } - let r = s.storage.with_controlled_conn(|c| -> Result<()> { - c.execute( + let r = s.storage()?.with_controlled_conn(|c| -> Result<()> { + c.execute( "UPDATE controlled_meta SET position=?1,snapshot_position=?1,sealed=1 WHERE id=1", [position], )?; - Ok(()) - }); - if r.is_err() { - s.ready = false; - } - r?; - Self::status_locked(&s) + Ok(()) + }); + if r.is_err() { + s.ready = false; + } + r?; + Self::status_locked(&s) + })(); + Self::complete(&mut s, operation) } pub fn seed(&self, projections: Vec) -> Result { self.mutate( @@ -476,17 +557,19 @@ impl ControlledStore { self.apply_batch(vec![Mutation::Delete { deletion }], position) } fn mutate(&self, mut batch: Vec, position: Option) -> Result { + self.check_process()?; normalize(&mut batch, &self.identity)?; let batch_digest = digest(&batch)?; let mut s = self.state.lock(); - let before = Self::status_locked(&s)?; - if position.is_none() && before.sealed { - return Err(ControlledError::Conflict("snapshot is sealed")); - } - if position.is_some() && !before.sealed { - return Err(ControlledError::NotReady); - } - let result=s.storage.with_controlled_conn(|conn| -> Result { + let operation = (|| { + let before = Self::status_locked(&s)?; + if position.is_none() && before.sealed { + return Err(ControlledError::Conflict("snapshot is sealed")); + } + if position.is_some() && !before.sealed { + return Err(ControlledError::NotReady); + } + let result=s.storage()?.with_controlled_conn(|conn| -> Result { let tx=conn.unchecked_transaction()?; if let Some(pos)=position { if pos==before.position { @@ -529,25 +612,34 @@ impl ControlledStore { } tx.execute("INSERT INTO controlled_fences(memory_id,scope,revision,internal_id,live,intent,valid_from,valid_until) VALUES(?1,?2,?3,?4,?5,?6,?7,?8) ON CONFLICT(memory_id) DO UPDATE SET revision=excluded.revision,internal_id=excluded.internal_id,live=excluded.live,intent=excluded.intent,valid_from=excluded.valid_from,valid_until=excluded.valid_until",params![id,scope_json,revision,internal,live,intent,p.and_then(|x|x.valid_from.map(|t|t.timestamp_millis())),p.and_then(|x|x.valid_until.map(|t|t.timestamp_millis()))])?; } + #[cfg(test)] + crash_barrier("inside_transaction"); let (adopted_rev,adopted_live)=if outcome=="stale" { let x=old.as_ref().unwrap();(x.1,x.3) }else{(revision,live)}; outcomes.push(Outcome{memory_id:id.clone(),revision:adopted_rev,state:if adopted_live{"live"}else{"withdrawn"}.into(),outcome:outcome.into()}); } let receipt=Receipt{generation:before.generation.clone(),position:position.unwrap_or(before.position),durable:true,outcomes}; tx.execute("UPDATE controlled_meta SET dimension=?1 WHERE id=1",[dimension])?; if let Some(pos)=position { tx.execute("UPDATE controlled_meta SET position=?1,last_batch=?2,last_receipt=?3 WHERE id=1",params![pos,batch_digest,json(&receipt)?])?; } - tx.commit()?;Ok(receipt) + if !batch.is_empty() { tx.execute("UPDATE controlled_meta SET cleanup_pending=1 WHERE id=1", [])?; } + tx.commit()?; + #[cfg(test)] + crash_barrier("after_commit"); + Ok(receipt) }); - s.storage.invalidate_controlled_cache(); - if matches!(result, Err(ControlledError::Storage)) { - s.ready = false; - } - if result.is_ok() && batch.iter().any(|m| matches!(m, Mutation::Delete { .. })) { - s.storage.with_controlled_conn(|c| { - SqliteStorage::scrub_after_delete(c); - Ok::<_, ControlledError>(()) - })?; - } - result + s.storage()?.invalidate_controlled_cache(); + if matches!(result, Err(ControlledError::Storage)) { + s.ready = false; + } + if result.is_ok() { + // The logical receipt already committed. Cleanup failure must never + // turn it into an ambiguous failed mutation; expose pending status. + if Self::cleanup_locked(&s).is_err() { + s.ready = false; + } + } + result + })(); + Self::complete(&mut s, operation) } pub fn search( &self, @@ -555,6 +647,7 @@ impl ControlledStore { scopes: &[Scope], options: SearchOptions, ) -> Result { + self.check_process()?; let SearchOptions { limit, embedding, @@ -577,49 +670,50 @@ impl ControlledStore { let started = Instant::now(); let duration = Duration::from_millis(max_duration_ms); let deadline = started + duration; - let Some(s) = self.state.try_lock_until(deadline) else { + let Some(mut s) = self.state.try_lock_until(deadline) else { return Ok(SearchResult { hits: vec![], degraded: true, truncated: true, }); }; - let status = Self::status_locked(&s)?; - if !status.ready { - return Err(ControlledError::NotReady); - } - let empty = || SearchResult { - hits: vec![], - degraded: false, - truncated: false, - }; - if scopes.is_empty() { - return Ok(empty()); - } - let mut scope_json: Vec = scopes.iter().map(json).collect::>()?; - scope_json.sort(); - scope_json.dedup(); - let terms: Vec = query - .split(|c: char| !c.is_alphanumeric()) - .filter(|s| !s.is_empty()) - .take(65) - .map(|s| format!("\"{s}\"")) - .collect(); - if terms.len() > 64 { - return Err(ControlledError::InvalidInput("query term limit")); - } - let fts = if terms.is_empty() { - "\"\"".into() - } else { - terms.join(" AND ") - }; - let fetched=s.storage.with_controlled_conn(|conn| -> Result<_> { + let operation = (|| { + let status = Self::status_locked(&s)?; + if !status.ready { + return Err(ControlledError::NotReady); + } + let empty = || SearchResult { + hits: vec![], + degraded: false, + truncated: false, + }; + if scopes.is_empty() { + return Ok(empty()); + } + let mut scope_json: Vec = scopes.iter().map(json).collect::>()?; + scope_json.sort(); + scope_json.dedup(); + let terms: Vec = query + .split(|c: char| !c.is_alphanumeric()) + .filter(|s| !s.is_empty()) + .take(65) + .map(|s| format!("\"{s}\"")) + .collect(); + if terms.len() > 64 { + return Err(ControlledError::InvalidInput("query term limit")); + } + let fts = if terms.is_empty() { + "\"\"".into() + } else { + terms.join(" AND ") + }; + let fetched=s.storage()?.with_controlled_conn(|conn| -> Result<_> { let dim:Option=conn.query_row("SELECT dimension FROM controlled_meta WHERE id=1",[],|r|r.get(0))?; if let (Some(v),Some(d))=(&embedding,dim) { if v.len()!=d { return Err(ControlledError::InvalidInput("embedding dimension")); }} conn.progress_handler(1000,Some(move ||started.elapsed()>=duration)); let run=(|| -> std::result::Result,rusqlite::Error> { let placeholders=vec!["?";scope_json.len()].join(","); - let sql=format!("SELECT m.id,m.tier,m.content_json,m.embedding_blob,m.temporal_json,m.source_json,m.salience_json,m.privacy_json,m.tags_json,m.metadata_json,m.links_json,m.content_hash,m.namespace, m.id IN (SELECT id FROM memories_fts WHERE memories_fts MATCH ?) AS matched FROM memories m JOIN controlled_fences f ON f.internal_id=m.id WHERE f.live=1 AND f.scope IN ({placeholders}) AND (f.valid_from IS NULL OR f.valid_from<=?) AND (f.valid_until IS NULL OR f.valid_until>?) ORDER BY matched DESC,m.id LIMIT ?"); + let sql=format!("SELECT m.id,m.tier,m.content_json,m.embedding_blob,m.temporal_json,m.source_json,m.salience_json,m.privacy_json,m.tags_json,m.metadata_json,m.links_json,m.content_hash,m.namespace, EXISTS(SELECT 1 FROM memories_fts WHERE rowid=m.rowid AND memories_fts MATCH ?) AS matched FROM memories m JOIN controlled_fences f ON f.internal_id=m.id WHERE f.live=1 AND f.scope IN ({placeholders}) AND (f.valid_from IS NULL OR f.valid_from<=?) AND (f.valid_until IS NULL OR f.valid_until>=?) ORDER BY matched DESC,m.id LIMIT ?"); let now=Utc::now().timestamp_millis(); let mut args:Vec=vec![fts.clone().into()];args.extend(scope_json.iter().cloned().map(Into::into));args.extend([now.into(),now.into(),((max_candidates+1)as i64).into()]); let mut stmt=conn.prepare(&sql)?;let rows=stmt.query_map(params_from_iter(args),|r|Ok((SqliteStorage::parse_mem_row(r)?,r.get(13)?)))?;rows.collect() @@ -627,77 +721,103 @@ impl ControlledStore { conn.progress_handler(0,None::bool>); match run { Ok(v)=>Ok(Some(v)),Err(rusqlite::Error::SqliteFailure(e,_)) if e.code==rusqlite::ErrorCode::OperationInterrupted => Ok(None),Err(_)=>Err(ControlledError::Storage) } })?; - let Some(mut fetched) = fetched else { - return Ok(SearchResult { - hits: vec![], - degraded: true, - truncated: true, - }); - }; - let mut truncated = fetched.len() > max_candidates; - fetched.truncate(max_candidates); - // Share the existing scorer and cosine math, but never construct an HNSW - // graph for a request-local candidate set. Each vector is bounded to 4096 - // dimensions and the deadline is checked between candidates. - let index = MemoryIndex::new(); - let mut mems = HashMap::new(); - let mut lexical = HashMap::new(); - let mut scores = HashMap::new(); - for (mem, matched) in fetched { - if Instant::now() >= deadline { - truncated = true; - break; - } - let score = match (&embedding, &mem.embedding) { - (Some(query), Some(vector)) => cosine_similarity(query, vector), - _ => 0.5, + let Some(mut fetched) = fetched else { + return Ok(SearchResult { + hits: vec![], + degraded: true, + truncated: true, + }); }; - if matched { - lexical.insert(mem.id, 1.0); + let mut truncated = fetched.len() > max_candidates; + fetched.truncate(max_candidates); + // Share the existing scorer and cosine math, but never construct an HNSW + // graph for a request-local candidate set. Each vector is bounded to 4096 + // dimensions and the deadline is checked between candidates. + let index = MemoryIndex::new(); + let mut mems = HashMap::new(); + let mut lexical = HashMap::new(); + let mut scores = HashMap::new(); + for (mem, matched) in fetched { + if Instant::now() >= deadline { + truncated = true; + break; + } + let envelope: Projection = serde_json::from_value( + mem.metadata + .get("projection") + .ok_or(ControlledError::Storage)? + .clone(), + ) + .map_err(|_| ControlledError::Storage)?; + if !effective(&envelope, Utc::now()) { + continue; + } + let score = match (&embedding, &mem.embedding) { + (Some(query), Some(vector)) => cosine_similarity(query, vector), + _ => 0.5, + }; + if matched { + lexical.insert(mem.id, 1.0); + } + scores.insert(mem.id, score); + mems.insert(mem.id, Arc::new(mem)); } - scores.insert(mem.id, score); - mems.insert(mem.id, Arc::new(mem)); - } - let mut q = RetrievalQuery::new(query, limit); - q.embedding = embedding; - q.budget = QueryBudget { - max_candidates, - max_duration: duration, - }; - let ranked = RetrievalEngine::new(&s.storage, &index).rank_candidates( - &q, - &scores, - &mems, - &lexical, - Some(deadline), - )?; - let mut hits = Vec::new(); - for r in ranked { - if Instant::now() >= deadline { - truncated = true; - break; + let mut q = RetrievalQuery::new(query, limit); + q.embedding = embedding; + q.budget = QueryBudget { + max_candidates, + max_duration: duration, + }; + let ranked = RetrievalEngine::new(s.storage()?, &index).rank_candidates( + &q, + &scores, + &mems, + &lexical, + Some(deadline), + )?; + let mut hits = Vec::new(); + for r in ranked { + if Instant::now() >= deadline { + truncated = true; + break; + } + let projection: Projection = serde_json::from_value( + r.memory + .metadata + .get("projection") + .ok_or(ControlledError::Storage)? + .clone(), + ) + .map_err(|_| ControlledError::Storage)?; + if !effective(&projection, Utc::now()) { + continue; + } + hits.push(Hit { + owner_type: self.identity.owner_type.clone(), + owner_id: self.identity.owner_id.clone(), + bot_id: self.identity.bot_id.clone(), + projection, + score: r.score, + }); } - let projection: Projection = serde_json::from_value( - r.memory - .metadata - .get("projection") - .ok_or(ControlledError::Storage)? - .clone(), - ) - .map_err(|_| ControlledError::Storage)?; - hits.push(Hit { - owner_type: self.identity.owner_type.clone(), - owner_id: self.identity.owner_id.clone(), - bot_id: self.identity.bot_id.clone(), - projection, - score: r.score, - }); + Ok(SearchResult { + hits, + degraded: truncated || started.elapsed() >= duration, + truncated, + }) + })(); + Self::complete(&mut s, operation) + } +} + +#[cfg(test)] +fn crash_barrier(phase: &str) { + if std::env::var("CORTEX_TEST_CRASH_PHASE").as_deref() == Ok(phase) { + let marker = std::env::var("CORTEX_TEST_CRASH_MARKER").unwrap(); + std::fs::write(marker, phase).unwrap(); + loop { + std::thread::park(); } - Ok(SearchResult { - hits, - degraded: truncated || started.elapsed() >= duration, - truncated, - }) } } @@ -743,7 +863,8 @@ mod tests { store.seed(vec![projection("a"), projection("b")]).unwrap(); let state = store.state.lock(); state - .storage + .storage() + .unwrap() .with_controlled_conn(|c| -> Result<()> { assert_eq!( c.query_row("PRAGMA synchronous", [], |r| r.get::<_, i64>(0))?, @@ -777,7 +898,7 @@ mod tests { store.finish_snapshot(10).unwrap(); let status = store.status().unwrap(); store.reconcile(&status.generation, 10).unwrap(); - store.state.lock().storage.with_controlled_conn(|c| -> Result<()> {c.execute_batch("CREATE TRIGGER controlled_failure BEFORE INSERT ON memories BEGIN SELECT RAISE(ABORT,'injected failure'); END;")?;Ok(())}).unwrap(); + store.state.lock().storage().unwrap().with_controlled_conn(|c| -> Result<()> {c.execute_batch("CREATE TRIGGER controlled_failure BEFORE INSERT ON memories BEGIN SELECT RAISE(ABORT,'injected failure'); END;")?;Ok(())}).unwrap(); let error = store.upsert(projection("b"), 11).unwrap_err(); assert_eq!(error.code(), "storage"); assert!(!store.status().unwrap().ready); @@ -785,7 +906,8 @@ mod tests { store .state .lock() - .storage + .storage() + .unwrap() .with_controlled_conn(|c| -> Result<()> { assert_eq!( c.query_row("SELECT COUNT(*) FROM controlled_fences", [], |r| r @@ -797,6 +919,16 @@ mod tests { .unwrap(); } #[test] + fn native_close_releases_resources_before_old_handles_drop() { + let root = tempfile::tempdir().unwrap(); + let path = root.path().canonicalize().unwrap().join("private"); + let old = ControlledStore::open(&path, identity()).unwrap(); + old.close().unwrap(); + let reopened = ControlledStore::open(&path, identity()).unwrap(); + assert_eq!(old.status().unwrap_err().code(), "closed"); + assert!(!reopened.status().unwrap().ready); + } + #[test] fn equal_revision_replay_with_changed_embedding_conflicts() { let root = tempfile::tempdir().unwrap(); let store = ControlledStore::open( @@ -811,4 +943,205 @@ mod tests { p.embedding = Some(vec![0.0, 1.0]); assert_eq!(store.seed(vec![p]).unwrap_err().code(), "conflict"); } + #[test] + fn precise_validity_preserves_submillisecond_boundaries() { + let instant: DateTime = "2026-10-22T00:00:00.123456789Z".parse().unwrap(); + let mut p = projection("precision"); + p.valid_from = Some(instant); + p.valid_until = Some(instant + chrono::Duration::nanoseconds(100)); + assert!(!effective(&p, instant - chrono::Duration::nanoseconds(1))); + assert!(effective(&p, instant)); + assert!(effective(&p, instant + chrono::Duration::nanoseconds(99))); + assert!(!effective(&p, instant + chrono::Duration::nanoseconds(100))); + } + #[test] + fn read_storage_failure_disables_queries_until_reconciliation() { + let root = tempfile::tempdir().unwrap(); + let store = ControlledStore::open( + &root.path().canonicalize().unwrap().join("private"), + identity(), + ) + .unwrap(); + store.seed(vec![projection("a")]).unwrap(); + store.finish_snapshot(10).unwrap(); + let status = store.status().unwrap(); + store.reconcile(&status.generation, 10).unwrap(); + store + .state + .lock() + .storage() + .unwrap() + .with_controlled_conn(|c| -> Result<()> { + c.execute("UPDATE memories SET metadata_json='{}'", [])?; + Ok(()) + }) + .unwrap(); + let error = store + .search( + "Remember", + &[projection("a").scope], + SearchOptions::default(), + ) + .unwrap_err(); + assert_eq!(error.code(), "storage"); + assert!(!store.status().unwrap().ready); + } + #[test] + fn crash_child() { + let Ok(root) = std::env::var("CORTEX_TEST_CRASH_ROOT") else { + return; + }; + let store = ControlledStore::open(Path::new(&root), identity()).unwrap(); + let status = store.status().unwrap(); + store + .reconcile(&status.generation, status.position) + .unwrap(); + let mut p = projection("a"); + p.revision = 2; + p.text = "committed revision two".into(); + store.upsert(p, 11).unwrap(); + panic!("crash barrier was not reached"); + } + #[test] + fn sigkill_inside_transaction_and_after_commit_preserves_atomic_replay() { + use std::process::{Command, Stdio}; + struct ChildGuard(std::process::Child); + impl Drop for ChildGuard { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } + } + for (phase, expected_position) in [("inside_transaction", 10), ("after_commit", 11)] { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().canonicalize().unwrap().join("private"); + let marker = dir.path().join("barrier"); + let store = ControlledStore::open(&root, identity()).unwrap(); + store.seed(vec![projection("a")]).unwrap(); + store.finish_snapshot(10).unwrap(); + store.close().unwrap(); + let mut child = ChildGuard( + Command::new(std::env::current_exe().unwrap()) + .args(["controlled::tests::crash_child", "--exact", "--nocapture"]) + .env("CORTEX_TEST_CRASH_ROOT", &root) + .env("CORTEX_TEST_CRASH_PHASE", phase) + .env("CORTEX_TEST_CRASH_MARKER", &marker) + .stdout(Stdio::null()) + .spawn() + .unwrap(), + ); + let deadline = Instant::now() + Duration::from_secs(15); + while !marker.exists() { + assert!( + child.0.try_wait().unwrap().is_none(), + "child exited before barrier" + ); + assert!(Instant::now() < deadline, "child did not reach barrier"); + std::thread::sleep(Duration::from_millis(10)); + } + assert_eq!(std::fs::read_to_string(&marker).unwrap(), phase); + child.0.kill().unwrap(); + child.0.wait().unwrap(); + let reopened = ControlledStore::open(&root, identity()).unwrap(); + let status = reopened.status().unwrap(); + assert_eq!(status.position, expected_position); + assert!(!status.ready); + reopened.state.lock().storage().unwrap().with_controlled_conn(|c| -> Result<()> { + let (revision, body): (i64, String) = c.query_row("SELECT f.revision,m.content_json FROM controlled_fences f JOIN memories m ON m.id=f.internal_id WHERE f.memory_id='a'", [], |r| Ok((r.get(0)?,r.get(1)?)))?; + assert_eq!(revision, if expected_position == 10 { 1 } else { 2 }); + assert_eq!(body.contains("committed revision two"), expected_position == 11); + Ok(()) + }).unwrap(); + let mut p = projection("a"); + p.revision = 2; + p.text = "committed revision two".into(); + let receipt = reopened.upsert(p.clone(), 11).unwrap(); + assert!(receipt.durable); + assert_eq!(receipt.position, 11); + assert_eq!( + json(&receipt).unwrap(), + json(&reopened.upsert(p, 11).unwrap()).unwrap() + ); + } + } + #[test] + fn cleanup_failure_remains_visible_and_retryable() { + let dir = tempfile::tempdir().unwrap(); + let store = ControlledStore::open( + &dir.path().canonicalize().unwrap().join("private"), + identity(), + ) + .unwrap(); + store.seed(vec![projection("a")]).unwrap(); + store + .state + .lock() + .storage() + .unwrap() + .with_controlled_conn(|c| -> Result<()> { + c.execute_batch( + "UPDATE controlled_meta SET cleanup_pending=1; DROP TABLE memories_fts;", + )?; + Ok(()) + }) + .unwrap(); + assert!(store.retry_cleanup().unwrap().cleanup_pending); + assert!(store.status().unwrap().cleanup_pending); + store + .state + .lock() + .storage() + .unwrap() + .with_controlled_conn(|c| -> Result<()> { + c.execute_batch( + "CREATE VIRTUAL TABLE memories_fts USING fts5(id UNINDEXED, content);", + )?; + Ok(()) + }) + .unwrap(); + assert!(!store.retry_cleanup().unwrap().cleanup_pending); + } + #[cfg(unix)] + #[test] + fn untrusted_ancestor_swap_race_cannot_open_outside_root() { + use std::os::unix::fs::PermissionsExt; + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + let dir = tempfile::tempdir().unwrap(); + let base = dir.path().canonicalize().unwrap(); + let parent = base.join("unsafe"); + std::fs::create_dir(&parent).unwrap(); + std::fs::set_permissions(&parent, std::fs::Permissions::from_mode(0o777)).unwrap(); + let target = base.join("outside"); + std::fs::create_dir(&target).unwrap(); + let stop = Arc::new(AtomicBool::new(false)); + let swaps = Arc::new(AtomicUsize::new(0)); + let running = stop.clone(); + let count = swaps.clone(); + let attacker_parent = parent.clone(); + let attacker_target = target.clone(); + let attacker = std::thread::spawn(move || { + let candidate = attacker_parent.join("candidate"); + while !running.load(Ordering::SeqCst) { + std::fs::create_dir(&candidate).unwrap(); + let parked = attacker_parent.join("parked"); + std::fs::rename(&candidate, &parked).unwrap(); + std::os::unix::fs::symlink(&attacker_target, &candidate).unwrap(); + count.fetch_add(1, Ordering::SeqCst); + std::fs::remove_file(&candidate).unwrap(); + std::fs::remove_dir(&parked).unwrap(); + } + }); + while swaps.load(Ordering::SeqCst) == 0 { + std::thread::yield_now(); + } + let results: Vec<_> = (0..100) + .map(|_| ControlledStore::open(&parent.join("candidate").join("private"), identity())) + .collect(); + stop.store(true, Ordering::SeqCst); + attacker.join().unwrap(); + assert!(results + .iter() + .all(|r| matches!(r, Err(ControlledError::InvalidInput(_))))); + assert_eq!(std::fs::read_dir(&target).unwrap().count(), 0); + } } diff --git a/cortex-core/src/retrieval.rs b/cortex-core/src/retrieval.rs index f52b32c..3b1f1aa 100644 --- a/cortex-core/src/retrieval.rs +++ b/cortex-core/src/retrieval.rs @@ -281,15 +281,19 @@ impl<'a> RetrievalEngine<'a> { // Batch fetch all candidates in a single query let all_ids: Vec = candidate_scores.keys().copied().collect(); let memories = self.storage.get_memories_batch(&all_ids)?; - let mem_map: HashMap> = memories.into_iter().map(|m| (m.id, Arc::new(m))).collect(); + let mem_map: HashMap> = + memories.into_iter().map(|m| (m.id, Arc::new(m))).collect(); self.rank_candidates(query, &candidate_scores, &mem_map, &fts_scores, None) } /// Shared ranking for candidates already constrained by a trusted scope predicate. pub(crate) fn rank_candidates( - &self, query: &RetrievalQuery, candidate_scores: &HashMap, - mem_map: &HashMap>, fts_scores: &HashMap, + &self, + query: &RetrievalQuery, + candidate_scores: &HashMap, + mem_map: &HashMap>, + fts_scores: &HashMap, deadline: Option, ) -> Result, CortexError> { let temporal_intent = detect_temporal_intent(&query.text); @@ -354,7 +358,9 @@ impl<'a> RetrievalEngine<'a> { // Score each candidate let mut results = Vec::new(); for (&id, &sim_score) in candidate_scores { - if deadline.is_some_and(|end| std::time::Instant::now() >= end) { break; } + if deadline.is_some_and(|end| std::time::Instant::now() >= end) { + break; + } if let Some(mem) = mem_map.get(&id) { // Namespace isolation: skip memories that don't match requested namespace if let Some(ref ns) = query.namespace { @@ -369,7 +375,8 @@ impl<'a> RetrievalEngine<'a> { } let fts_score = fts_scores.get(&id).copied().unwrap_or(0.0); - let breakdown = self.compute_scores(mem, sim_score, query, &temporal_intent, fts_score); + let breakdown = + self.compute_scores(mem, sim_score, query, &temporal_intent, fts_score); let final_score = weights.similarity * breakdown.similarity + weights.temporal * breakdown.temporal + weights.salience * breakdown.salience @@ -435,7 +442,12 @@ impl<'a> RetrievalEngine<'a> { if expanded.len() >= MAX_EXPANSION_TERMS { break 'entities; } - if let MemContent::Fact { subject, predicate, object } = &mem.content { + if let MemContent::Fact { + subject, + predicate, + object, + } = &mem.content + { let entity_lower = entity.to_lowercase(); if subject.to_lowercase().contains(&entity_lower) { expanded.insert(object.clone()); @@ -476,7 +488,10 @@ impl<'a> RetrievalEngine<'a> { ) -> Result, CortexError> { let terms: Vec = entities.iter().cloned().collect(); let matches = self.storage.search_episodic_by_terms(&terms, limit)?; - Ok(matches.into_iter().map(|(id, _text)| (id, 0.30_f32)).collect()) + Ok(matches + .into_iter() + .map(|(id, _text)| (id, 0.30_f32)) + .collect()) } /// Multi-hop expansion: look at top initial candidates, extract entities @@ -495,7 +510,9 @@ impl<'a> RetrievalEngine<'a> { let mut entities: Vec = Vec::new(); for mem in &top_mems { match &mem.content { - MemContent::Fact { subject, object, .. } => { + MemContent::Fact { + subject, object, .. + } => { if subject != "User" && !entities.contains(subject) { entities.push(subject.clone()); } @@ -503,7 +520,9 @@ impl<'a> RetrievalEngine<'a> { entities.push(object.clone()); } } - MemContent::Relationship { person_a, person_b, .. } => { + MemContent::Relationship { + person_a, person_b, .. + } => { // person_a/person_b are UUIDs, skip for now let _ = (person_a, person_b); } @@ -537,10 +556,9 @@ impl<'a> RetrievalEngine<'a> { limit: usize, ) -> Result, CortexError> { match intent { - TemporalIntent::Recent => { - self.storage - .list_by_tier_ordered_by_ingestion(MemoryTier::Episodic, limit) - } + TemporalIntent::Recent => self + .storage + .list_by_tier_ordered_by_ingestion(MemoryTier::Episodic, limit), TemporalIntent::First | TemporalIntent::Earliest => { // Get all and reverse (oldest first) let mut mems = self @@ -606,10 +624,16 @@ impl<'a> RetrievalEngine<'a> { match &mem.source.identity_id { Some(id) if *id == person_id => 1.0, _ => match &mem.content { - MemContent::Relationship { person_a, person_b, .. } => { + MemContent::Relationship { + person_a, person_b, .. + } => { let matches_a = *person_a == person_id; let matches_b = *person_b == person_id; - if matches_a | matches_b { 1.0 } else { 0.0 } + if matches_a | matches_b { + 1.0 + } else { + 0.0 + } } _ => 0.0, }, @@ -667,7 +691,13 @@ pub fn detect_query_type(query: &str) -> QueryType { } // Preference patterns (check before fact patterns since "what do I like" contains "what") - let pref_en = ["what do i like", "my preference", "do i prefer", "my favorite", "my favourite"]; + let pref_en = [ + "what do i like", + "my preference", + "do i prefer", + "my favorite", + "my favourite", + ]; let pref_zh = ["我喜欢什么", "我的偏好", "我偏好", "我喜欢的"]; for pat in &pref_en { if lower.contains(pat) { @@ -681,7 +711,14 @@ pub fn detect_query_type(query: &str) -> QueryType { } // Person patterns - let person_en = ["who is", "who was", "who are", "about him", "about her", "about them"]; + let person_en = [ + "who is", + "who was", + "who are", + "about him", + "about her", + "about them", + ]; for pat in &person_en { if lower.contains(pat) { return QueryType::PersonQuery; @@ -692,10 +729,15 @@ pub fn detect_query_type(query: &str) -> QueryType { return QueryType::PersonQuery; } // "about [Name]" — look for "about" followed by a capitalized word - if let Some(pos) = lower.find("about ") { - let after = &query[pos + 6..]; - if after.chars().next().is_some_and(|c| c.is_uppercase()) { - return QueryType::PersonQuery; + for (pos, _) in query.char_indices() { + if query[pos..] + .get(..6) + .is_some_and(|part| part.eq_ignore_ascii_case("about ")) + { + let after = &query[pos + 6..]; + if after.chars().next().is_some_and(|c| c.is_uppercase()) { + return QueryType::PersonQuery; + } } } // Chinese person patterns @@ -714,7 +756,10 @@ pub fn detect_query_type(query: &str) -> QueryType { .take_while(|c| !c.is_whitespace() && !"的了吗呢吧？！，。".contains(*c)) .collect(); let char_count = name.chars().count(); - let non_name_words = ["这个", "那个", "什么", "哪个", "一些", "所有", "我们", "他们", "项目", "问题", "情况", "事情"]; + let non_name_words = [ + "这个", "那个", "什么", "哪个", "一些", "所有", "我们", "他们", "项目", "问题", "情况", + "事情", + ]; let is_non_name = non_name_words.iter().any(|w| name.contains(w)); if (2..=4).contains(&char_count) && !is_non_name { return QueryType::PersonQuery; @@ -776,9 +821,9 @@ fn extract_query_entities(text: &str) -> Vec { let name = name_parts.join(" "); // Skip common English words that happen to start sentences let skip = [ - "I", "The", "A", "An", "What", "Where", "When", "Who", "How", - "Tell", "Do", "Does", "Is", "Are", "Was", "Were", "Can", "Could", - "Will", "Would", "Should", "My", "Your", "His", "Her", "Their", + "I", "The", "A", "An", "What", "Where", "When", "Who", "How", "Tell", "Do", "Does", + "Is", "Are", "Was", "Were", "Can", "Could", "Will", "Would", "Should", "My", + "Your", "His", "Her", "Their", ]; if !skip.contains(&name.as_str()) { entities.push(name); @@ -810,8 +855,10 @@ fn extract_query_entities(text: &str) -> Vec { for word in lower.split_whitespace() { let clean: String = word.chars().filter(|c| c.is_alphanumeric()).collect(); if clean.len() >= 4 - && !["what", "where", "when", "about", "know", "tell", "does", "have"] - .contains(&clean.as_str()) + && ![ + "what", "where", "when", "about", "know", "tell", "does", "have", + ] + .contains(&clean.as_str()) && !entities.iter().any(|e| e.to_lowercase() == clean) { entities.push(clean); @@ -828,13 +875,31 @@ fn detect_temporal_intent(query: &str) -> TemporalIntent { // Recent intent let recent_en = [ - "last time", "recently", "latest", "most recent", "just now", - "yesterday", "today", "this week", "last week", "this month", - "last month", "the other day", + "last time", + "recently", + "latest", + "most recent", + "just now", + "yesterday", + "today", + "this week", + "last week", + "this month", + "last month", + "the other day", ]; let recent_zh = [ - "最近", "上次", "昨天", "今天", "刚才", "这周", "上周", - "这个月", "上个月", "前几天", "最新", + "最近", + "上次", + "昨天", + "今天", + "刚才", + "这周", + "上周", + "这个月", + "上个月", + "前几天", + "最新", ]; for pat in &recent_en { @@ -850,12 +915,17 @@ fn detect_temporal_intent(query: &str) -> TemporalIntent { // First/earliest intent let first_en = [ - "first time", "the first", "originally", "initially", "at first", - "when did", "earliest", "began", "started", - ]; - let first_zh = [ - "第一次", "最初", "一开始", "起初", "最早", "什么时候开始", + "first time", + "the first", + "originally", + "initially", + "at first", + "when did", + "earliest", + "began", + "started", ]; + let first_zh = ["第一次", "最初", "一开始", "起初", "最早", "什么时候开始"]; for pat in &first_en { if lower.contains(pat) { @@ -870,13 +940,28 @@ fn detect_temporal_intent(query: &str) -> TemporalIntent { // Sequence intent (asking about order of events) let seq_en = [ - "before", "after", "then", "followed by", "in order", - "timeline", "sequence", "chronolog", "when was", "what happened", - "how many times", "how often", + "before", + "after", + "then", + "followed by", + "in order", + "timeline", + "sequence", + "chronolog", + "when was", + "what happened", + "how many times", + "how often", ]; let seq_zh = [ - "之前", "之后", "然后", "顺序", "时间线", "多少次", - "什么时候", "先后", + "之前", + "之后", + "然后", + "顺序", + "时间线", + "多少次", + "什么时候", + "先后", ]; for pat in &seq_en { @@ -915,8 +1000,14 @@ mod tests { // A just-created, never-recalled memory: recency high, frequency zero → ~0.5. let fresh = TemporalInfo::now(); let f = compute_frecency(&fresh, now); - assert!((0.0..=1.0).contains(&f), "frecency must stay in [0,1], got {f}"); - assert!((0.45..=0.55).contains(&f), "fresh frecency should be ~0.5, got {f}"); + assert!( + (0.0..=1.0).contains(&f), + "frecency must stay in [0,1], got {f}" + ); + assert!( + (0.45..=0.55).contains(&f), + "fresh frecency should be ~0.5, got {f}" + ); } #[test] @@ -979,13 +1070,22 @@ mod tests { #[test] fn test_query_type_person_who() { assert_eq!(detect_query_type("Who is Alice?"), QueryType::PersonQuery); - assert_eq!(detect_query_type("who was that person?"), QueryType::PersonQuery); + assert_eq!( + detect_query_type("who was that person?"), + QueryType::PersonQuery + ); } #[test] fn test_query_type_person_about_name() { - assert_eq!(detect_query_type("Tell me about Alice"), QueryType::PersonQuery); - assert_eq!(detect_query_type("What do you know about Bob Smith?"), QueryType::PersonQuery); + assert_eq!( + detect_query_type("Tell me about Alice"), + QueryType::PersonQuery + ); + assert_eq!( + detect_query_type("What do you know about Bob Smith?"), + QueryType::PersonQuery + ); } #[test] @@ -993,14 +1093,20 @@ mod tests { assert_eq!(detect_query_type("谁是张三"), QueryType::PersonQuery); assert_eq!(detect_query_type("关于张三的信息"), QueryType::PersonQuery); // Non-name phrases after "关于" should NOT be PersonQuery - assert_ne!(detect_query_type("关于这个项目的进展"), QueryType::PersonQuery); + assert_ne!( + detect_query_type("关于这个项目的进展"), + QueryType::PersonQuery + ); assert_ne!(detect_query_type("关于问题的解答"), QueryType::PersonQuery); } #[test] fn test_query_type_fact() { assert_eq!(detect_query_type("What is Rust?"), QueryType::FactQuery); - assert_eq!(detect_query_type("Where does Alice work?"), QueryType::FactQuery); + assert_eq!( + detect_query_type("Where does Alice work?"), + QueryType::FactQuery + ); assert_eq!(detect_query_type("What does Bob do?"), QueryType::FactQuery); } @@ -1012,51 +1118,85 @@ mod tests { #[test] fn test_query_type_preference() { - assert_eq!(detect_query_type("What do I like to eat?"), QueryType::PreferenceQuery); - assert_eq!(detect_query_type("What is my preference for editors?"), QueryType::PreferenceQuery); + assert_eq!( + detect_query_type("What do I like to eat?"), + QueryType::PreferenceQuery + ); + assert_eq!( + detect_query_type("What is my preference for editors?"), + QueryType::PreferenceQuery + ); } #[test] fn test_query_type_preference_chinese() { - assert_eq!(detect_query_type("我喜欢什么颜色"), QueryType::PreferenceQuery); - assert_eq!(detect_query_type("我的偏好是什么"), QueryType::PreferenceQuery); + assert_eq!( + detect_query_type("我喜欢什么颜色"), + QueryType::PreferenceQuery + ); + assert_eq!( + detect_query_type("我的偏好是什么"), + QueryType::PreferenceQuery + ); } #[test] fn test_query_type_temporal_defers() { // Temporal queries should be classified as Temporal, not other types - assert_eq!(detect_query_type("What did I do last time?"), QueryType::Temporal); + assert_eq!( + detect_query_type("What did I do last time?"), + QueryType::Temporal + ); assert_eq!(detect_query_type("最近聊了什么"), QueryType::Temporal); } #[test] fn test_query_type_general() { assert_eq!(detect_query_type("hello there"), QueryType::General); - assert_eq!(detect_query_type("summarize everything"), QueryType::General); + assert_eq!( + detect_query_type("summarize everything"), + QueryType::General + ); } #[test] fn test_extract_query_entities_english() { let entities = extract_query_entities("What do I know about Alice?"); - assert!(entities.iter().any(|e| e == "Alice"), "Should extract Alice: {:?}", entities); + assert!( + entities.iter().any(|e| e == "Alice"), + "Should extract Alice: {:?}", + entities + ); } #[test] fn test_extract_query_entities_full_name() { let entities = extract_query_entities("Tell me about Alice Smith"); - assert!(entities.iter().any(|e| e == "Alice Smith"), "Should extract full name: {:?}", entities); + assert!( + entities.iter().any(|e| e == "Alice Smith"), + "Should extract full name: {:?}", + entities + ); } #[test] fn test_extract_query_entities_chinese() { let entities = extract_query_entities("关于张三的信息"); - assert!(entities.iter().any(|e| e == "张三"), "Should extract Chinese name: {:?}", entities); + assert!( + entities.iter().any(|e| e == "张三"), + "Should extract Chinese name: {:?}", + entities + ); } #[test] fn test_extract_query_entities_skip_common() { let entities = extract_query_entities("What is the weather?"); // "What" should be skipped - assert!(!entities.iter().any(|e| e == "What"), "Should skip common words: {:?}", entities); + assert!( + !entities.iter().any(|e| e == "What"), + "Should skip common words: {:?}", + entities + ); } } diff --git a/cortex-core/src/storage/sqlite.rs b/cortex-core/src/storage/sqlite.rs index 0ba32ea..60e2030 100644 --- a/cortex-core/src/storage/sqlite.rs +++ b/cortex-core/src/storage/sqlite.rs @@ -64,20 +64,26 @@ impl SqliteStorage { // (another connection holds the checkpoint lock): retry, never report success. for attempt in 0..20 { let (busy, log, done): (i64, i64, i64) = conn - .query_row("PRAGMA wal_checkpoint(FULL)", [], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?))) + .query_row("PRAGMA wal_checkpoint(FULL)", [], |r| { + Ok((r.get(0)?, r.get(1)?, r.get(2)?)) + }) .map_err(|e| CortexError::Storage(e.to_string()))?; if busy == 0 && done >= log { return Ok(()); } std::thread::sleep(std::time::Duration::from_millis(25 * (attempt + 1))); } - Err(CortexError::Storage("could not make the change durable (database busy); try again".into())) + Err(CortexError::Storage( + "could not make the change durable (database busy); try again".into(), + )) } /// The database file, or `None` for an in-memory database. pub fn db_path(&self) -> Option { let conn = self.write_conn.lock(); - conn.path().filter(|p| !p.is_empty()).map(std::path::PathBuf::from) + conn.path() + .filter(|p| !p.is_empty()) + .map(std::path::PathBuf::from) } /// Open a Cortex database with optional encryption passphrase. @@ -90,9 +96,15 @@ impl SqliteStorage { /// Open with a raw 256-bit SQLCipher key (no per-open PBKDF2), `readers` read /// connections, and fully durable commits (`synchronous=FULL`). For services that open many small databases (one per tenant). /// Fails if SQLCipher is not compiled in: a raw key must never silently mean plaintext. - pub fn open_with_raw_key(path: &str, key: &[u8; 32], readers: usize) -> Result { + pub fn open_with_raw_key( + path: &str, + key: &[u8; 32], + readers: usize, + ) -> Result { if !cfg!(feature = "encrypted-db") { - return Err(CortexError::Storage("encrypted-db feature is required for a raw key".into())); + return Err(CortexError::Storage( + "encrypted-db feature is required for a raw key".into(), + )); } let storage = Self::open_keyed(path, Some(DbKey::Raw(key)), readers.max(1))?; // Services acknowledge writes to remote clients (e.g. "unshared"): a commit must @@ -108,14 +120,16 @@ impl SqliteStorage { fn apply_key(conn: &Connection, key: Option<&DbKey>) -> Result<(), CortexError> { #[cfg(feature = "encrypted-db")] match key { - Some(DbKey::Passphrase(p)) => conn - .pragma_update(None, "key", p) - .map_err(|e| CortexError::Storage(format!("Failed to set encryption key: {}", e)))?, + Some(DbKey::Passphrase(p)) => conn.pragma_update(None, "key", p).map_err(|e| { + CortexError::Storage(format!("Failed to set encryption key: {}", e)) + })?, Some(DbKey::Raw(k)) => { let hex: String = k.iter().map(|b| format!("{b:02x}")).collect(); // SQLCipher raw-key syntax: PRAGMA key = "x'<64 hex>'" (hex only; no injection). conn.execute_batch(&format!("PRAGMA key = \"x'{hex}'\";")) - .map_err(|e| CortexError::Storage(format!("Failed to set encryption key: {}", e)))? + .map_err(|e| { + CortexError::Storage(format!("Failed to set encryption key: {}", e)) + })? } None => {} } @@ -132,27 +146,62 @@ impl SqliteStorage { /// main file's mode — and tightens files left group/world-readable by older releases. #[cfg(unix)] fn restrict_db_file_mode(path: &str) -> Result<(), CortexError> { - use std::os::unix::fs::{OpenOptionsExt, PermissionsExt}; - if path.is_empty() || path == ":memory:" || path.starts_with("file:") { + use std::os::unix::fs::{MetadataExt, OpenOptionsExt, PermissionsExt}; + if path == ":memory:" { return Ok(()); } - std::fs::OpenOptions::new() + // URI filenames have aliases and query parameters; accepting them here + // would let SQLite open a different file than the descriptor we secure. + if path.is_empty() || path.starts_with("file:") { + return Err(CortexError::InvalidInput( + "use a filesystem path or :memory:".into(), + )); + } + let main = std::fs::OpenOptions::new() .write(true) .create(true) .truncate(false) .mode(0o600) + .custom_flags(libc::O_NOFOLLOW) .open(path) .map_err(|e| CortexError::Storage(e.to_string()))?; for suffix in ["", "-wal", "-shm", "-journal"] { - let p = format!("{path}{suffix}"); - let Ok(meta) = std::fs::metadata(&p) else { continue }; - let mode = meta.permissions().mode(); - if mode & 0o077 != 0 { - // Not fatal (e.g. a file owned by someone else), but never silent. - if let Err(e) = std::fs::set_permissions(&p, std::fs::Permissions::from_mode(mode & 0o700)) { - tracing::warn!("could not restrict permissions on {p}: {e}"); + let file = if suffix.is_empty() { + main.try_clone() + } else { + match std::fs::OpenOptions::new() + .read(true) + .write(true) + .custom_flags(libc::O_NOFOLLOW) + .open(format!("{path}{suffix}")) + { + Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue, + other => other, } } + .map_err(|e| CortexError::Storage(e.to_string()))?; + let meta = file + .metadata() + .map_err(|e| CortexError::Storage(e.to_string()))?; + if !meta.is_file() || meta.uid() != unsafe { libc::geteuid() } || meta.nlink() != 1 { + return Err(CortexError::Storage( + "database must be an owner-owned regular file".into(), + )); + } + file.set_permissions(std::fs::Permissions::from_mode(0o600)) + .map_err(|e| CortexError::Storage(e.to_string()))?; + if file + .metadata() + .map_err(|e| CortexError::Storage(e.to_string()))? + .permissions() + .mode() + & 0o777 + != 0o600 + { + return Err(CortexError::Storage( + "database permissions are not private".into(), + )); + } } Ok(()) } @@ -164,36 +213,78 @@ impl SqliteStorage { pub(crate) fn open_controlled(path: &str) -> Result { let storage = Self::open_keyed_mode(path, None, 1, true)?; storage.with_write_conn(|conn| { - conn.execute_batch("PRAGMA synchronous=FULL;") + // Controlled queries probe exact authorized rowids. Keep FTS docids + // aligned with memories; legacy databases retain their old layout. + conn.execute_batch("PRAGMA synchronous=FULL; + BEGIN IMMEDIATE; + DROP TRIGGER IF EXISTS memories_fts_ai; + DROP TRIGGER IF EXISTS memories_fts_ad; + DROP TRIGGER IF EXISTS memories_fts_au; + CREATE TRIGGER memories_fts_ai AFTER INSERT ON memories BEGIN + INSERT INTO memories_fts(rowid,id,content) VALUES(NEW.rowid,NEW.id,NEW.content_json); END; + CREATE TRIGGER memories_fts_ad AFTER DELETE ON memories BEGIN + DELETE FROM memories_fts WHERE rowid=OLD.rowid; END; + CREATE TRIGGER memories_fts_au AFTER UPDATE OF content_json ON memories BEGIN + DELETE FROM memories_fts WHERE rowid=OLD.rowid; + INSERT INTO memories_fts(rowid,id,content) VALUES(NEW.rowid,NEW.id,NEW.content_json); END; + DELETE FROM memories_fts; + INSERT INTO memories_fts(rowid,id,content) SELECT rowid,id,content_json FROM memories; + COMMIT;") .map_err(|e| CortexError::Storage(e.to_string())) })?; Ok(storage) } - fn open_keyed_mode(path: &str, key: Option, readers: usize, controlled: bool) -> Result { + fn open_keyed_mode( + path: &str, + key: Option, + readers: usize, + controlled: bool, + ) -> Result { #[cfg(unix)] Self::restrict_db_file_mode(path)?; let conn = Connection::open(path).map_err(|e| CortexError::Storage(e.to_string()))?; Self::apply_key(&conn, key.as_ref())?; - let marked: bool = conn.query_row( - "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name='cortex_controlled_marker')", [], |r| r.get(0) - ).map_err(|e| CortexError::Storage(e.to_string()))?; + let marked: bool = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name='cortex_controlled_marker')", + [], + |r| r.get(0), + ) + .map_err(|e| CortexError::Storage(e.to_string()))?; if marked && !controlled { - return Err(CortexError::InvalidInput("controlled database requires ControlledStore".into())); + return Err(CortexError::InvalidInput( + "controlled database requires ControlledStore".into(), + )); } if controlled && !marked { - let nonempty: bool = conn.query_row( - "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name NOT LIKE 'sqlite_%')", [], |r| r.get(0) - ).map_err(|e| CortexError::Storage(e.to_string()))?; - if nonempty { return Err(CortexError::InvalidInput("cannot import a personal database".into())); } + let nonempty: bool = conn + .query_row( + "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name NOT LIKE 'sqlite_%')", + [], + |r| r.get(0), + ) + .map_err(|e| CortexError::Storage(e.to_string()))?; + if nonempty { + return Err(CortexError::InvalidInput( + "cannot import a personal database".into(), + )); + } conn.execute_batch("BEGIN IMMEDIATE; CREATE TABLE cortex_controlled_marker(version INTEGER NOT NULL); INSERT INTO cortex_controlled_marker VALUES(1); COMMIT;") .map_err(|e| CortexError::Storage(e.to_string()))?; } if controlled { - let version: i64 = conn.query_row("SELECT version FROM cortex_controlled_marker", [], |r| r.get(0)) + let version: i64 = conn + .query_row("SELECT version FROM cortex_controlled_marker", [], |r| { + r.get(0) + }) .map_err(|e| CortexError::Storage(e.to_string()))?; - if version != 1 { return Err(CortexError::InvalidInput("unsupported controlled store format".into())); } + if version != 1 { + return Err(CortexError::InvalidInput( + "unsupported controlled store format".into(), + )); + } } conn.execute_batch( @@ -238,10 +329,9 @@ impl SqliteStorage { // Deletes are permanent: FTS5 otherwise keeps deleted terms in index segments // until a merge (secure_delete only scrubs freed b-tree pages). Needs SQLite // 3.42+; on older builds scrub_after_delete falls back to an index rebuild. - let _ = storage - .write_conn - .lock() - .execute_batch("INSERT INTO memories_fts(memories_fts, rank) VALUES('secure-delete', 1);"); + let _ = storage.write_conn.lock().execute_batch( + "INSERT INTO memories_fts(memories_fts, rank) VALUES('secure-delete', 1);", + ); Ok(storage) } @@ -249,37 +339,40 @@ impl SqliteStorage { /// index, then fold the WAL back into the (zero-scrubbed) database and truncate it so /// the plaintext does not linger in old WAL frames. Best effort: a reader holding a /// snapshot can block the checkpoint; the next delete or autocheckpoint retries. - pub(crate) fn scrub_after_delete(conn: &Connection) { - let fts_secure_delete = conn - .query_row( - "SELECT v FROM memories_fts_config WHERE k = 'secure-delete'", - [], - |r| r.get::<_, i64>(0), - ) - .is_ok_and(|v| v != 0); - if !fts_secure_delete { - if let Err(e) = - conn.execute_batch("INSERT INTO memories_fts(memories_fts) VALUES('optimize');") - { - tracing::warn!(error = %e, "FTS scrub after delete failed"); - } - } - for attempt in 0..3 { - match conn.query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |r| r.get::<_, i64>(0)) { - Ok(0) => return, - Ok(_) => std::thread::sleep(std::time::Duration::from_millis(10 * (attempt + 1))), - Err(e) => { - tracing::warn!(error = %e, "WAL scrub after delete failed"); - return; - } + pub(crate) fn scrub_after_delete(conn: &Connection) -> bool { + // A snapshot reader can block truncation indefinitely. Cleanup is bounded; + // controlled callers persist and expose a retryable pending flag. + if conn.busy_timeout(std::time::Duration::ZERO).is_err() { + return false; + } + let started = std::time::Instant::now(); + conn.progress_handler(1000, Some(move || started.elapsed() >= std::time::Duration::from_millis(250))); + let result = (|| -> rusqlite::Result { + let secure = conn + .query_row( + "SELECT v FROM memories_fts_config WHERE k = 'secure-delete'", + [], + |r| r.get::<_, i64>(0), + ) + .is_ok_and(|v| v != 0); + if !secure { + conn.execute_batch("INSERT INTO memories_fts(memories_fts) VALUES('optimize');")?; } - } - tracing::warn!("WAL scrub after delete skipped: database busy"); + Ok(conn.query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |r| { + r.get::<_, i64>(0) + })? == 0) + })() + .unwrap_or(false); + conn.progress_handler(0, None:: bool>); + let restored = conn + .busy_timeout(std::time::Duration::from_millis(5000)) + .is_ok(); + if !result || !restored { tracing::warn!("SQLite physical cleanup remains pending"); } + result && restored } pub fn open_in_memory() -> Result { - let conn = - Connection::open_in_memory().map_err(|e| CortexError::Storage(e.to_string()))?; + let conn = Connection::open_in_memory().map_err(|e| CortexError::Storage(e.to_string()))?; conn.execute_batch("PRAGMA foreign_keys=ON;") .map_err(|e| CortexError::Storage(e.to_string()))?; // In-memory DBs can't share connections, so read_pool is empty @@ -321,7 +414,10 @@ impl SqliteStorage { f(&conn) } - pub(crate) fn insert_on_connection(conn: &Connection, mem: &MemObject) -> Result<(), CortexError> { + pub(crate) fn insert_on_connection( + conn: &Connection, + mem: &MemObject, + ) -> Result<(), CortexError> { let now = Utc::now().to_rfc3339(); let embedding_blob = mem.embedding.as_ref().map(|e| f32_vec_to_bytes(e)); let content_json = serde_json::to_string(&mem.content).unwrap(); @@ -334,29 +430,29 @@ impl SqliteStorage { ) .map_err(|e| CortexError::Storage(e.to_string()))?; stmt.execute(params![ - id_str, - mem.tier.as_str(), - content_json, - embedding_blob, - serde_json::to_string(&mem.temporal).unwrap(), - serde_json::to_string(&mem.source).unwrap(), - serde_json::to_string(&mem.salience).unwrap(), - serialize_privacy(&mem.privacy), - serialize_tags(&mem.tags), - serialize_metadata(&mem.metadata), - serialize_links(&mem.links), - mem.content_hash, - mem.namespace, - mem.salience.effective_score, - fact_subject, - fact_object, - pref_key, - mem.source.channel, - mem.source.identity_id.map(|id| id.to_string()), - now, - now, - ]) - .map_err(|e| CortexError::Storage(e.to_string()))?; + id_str, + mem.tier.as_str(), + content_json, + embedding_blob, + serde_json::to_string(&mem.temporal).unwrap(), + serde_json::to_string(&mem.source).unwrap(), + serde_json::to_string(&mem.salience).unwrap(), + serialize_privacy(&mem.privacy), + serialize_tags(&mem.tags), + serialize_metadata(&mem.metadata), + serialize_links(&mem.links), + mem.content_hash, + mem.namespace, + mem.salience.effective_score, + fact_subject, + fact_object, + pref_key, + mem.source.channel, + mem.source.identity_id.map(|id| id.to_string()), + now, + now, + ]) + .map_err(|e| CortexError::Storage(e.to_string()))?; Ok(()) } @@ -367,7 +463,9 @@ impl SqliteStorage { } pub(crate) fn with_controlled_conn(&self, f: F) -> Result - where F: FnOnce(&Connection) -> Result { + where + F: FnOnce(&Connection) -> Result, + { f(&self.write_conn.lock()) } @@ -450,11 +548,12 @@ impl SqliteStorage { } else { Self::json_col(8, &tags_json)? }; - let metadata: std::collections::HashMap = if metadata_json == "{}" { - std::collections::HashMap::new() - } else { - Self::json_col(9, &metadata_json)? - }; + let metadata: std::collections::HashMap = + if metadata_json == "{}" { + std::collections::HashMap::new() + } else { + Self::json_col(9, &metadata_json)? + }; let links: Vec = if links_json == "[]" { Vec::new() } else { @@ -529,14 +628,18 @@ fn serialize_links(links: &[MemLink]) -> String { /// Extract materialized column values from memory content. /// Returns (fact_subject, fact_object, pref_key) — all lowercased for indexed LIKE queries. -fn extract_materialized_columns(content: &MemContent) -> (Option, Option, Option) { +fn extract_materialized_columns( + content: &MemContent, +) -> (Option, Option, Option) { match content { - MemContent::Fact { subject, object, .. } => { - (Some(subject.to_lowercase()), Some(object.to_lowercase()), None) - } - MemContent::Preference { key, .. } => { - (None, None, Some(key.to_lowercase())) - } + MemContent::Fact { + subject, object, .. + } => ( + Some(subject.to_lowercase()), + Some(object.to_lowercase()), + None, + ), + MemContent::Preference { key, .. } => (None, None, Some(key.to_lowercase())), _ => (None, None, None), } } @@ -555,8 +658,12 @@ fn f32_vec_to_bytes(v: &[f32]) -> Vec { let mut min_val = f32::INFINITY; let mut max_val = f32::NEG_INFINITY; for &f in v { - if f < min_val { min_val = f; } - if f > max_val { max_val = f; } + if f < min_val { + min_val = f; + } + if f > max_val { + max_val = f; + } } let range = max_val - min_val; @@ -711,9 +818,7 @@ impl StorageBackend for SqliteStorage { .map_err(|e| CortexError::Storage(e.to_string()))?; // Migration: create FTS5 virtual table for full-text search - let has_fts: bool = conn - .prepare("SELECT * FROM memories_fts LIMIT 0") - .is_ok(); + let has_fts: bool = conn.prepare("SELECT * FROM memories_fts LIMIT 0").is_ok(); if !has_fts { conn.execute_batch( "CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5( @@ -838,7 +943,6 @@ impl StorageBackend for SqliteStorage { Ok(result) } - fn update_memory(&self, mem: &MemObject) -> Result<(), CortexError> { let conn = self.write_conn.lock(); let now = Utc::now().to_rfc3339(); @@ -852,28 +956,28 @@ impl StorageBackend for SqliteStorage { ) .map_err(|e| CortexError::Storage(e.to_string()))?; stmt.execute(params![ - id_str, - mem.tier.as_str(), - content_json, - embedding_blob, - serde_json::to_string(&mem.temporal).unwrap(), - serde_json::to_string(&mem.source).unwrap(), - serde_json::to_string(&mem.salience).unwrap(), - serialize_privacy(&mem.privacy), - serialize_tags(&mem.tags), - serialize_metadata(&mem.metadata), - serialize_links(&mem.links), - mem.content_hash, - mem.namespace, - mem.salience.effective_score, - fact_subject, - fact_object, - pref_key, - mem.source.channel, - mem.source.identity_id.map(|id| id.to_string()), - now, - ]) - .map_err(|e| CortexError::Storage(e.to_string()))?; + id_str, + mem.tier.as_str(), + content_json, + embedding_blob, + serde_json::to_string(&mem.temporal).unwrap(), + serde_json::to_string(&mem.source).unwrap(), + serde_json::to_string(&mem.salience).unwrap(), + serialize_privacy(&mem.privacy), + serialize_tags(&mem.tags), + serialize_metadata(&mem.metadata), + serialize_links(&mem.links), + mem.content_hash, + mem.namespace, + mem.salience.effective_score, + fact_subject, + fact_object, + pref_key, + mem.source.channel, + mem.source.identity_id.map(|id| id.to_string()), + now, + ]) + .map_err(|e| CortexError::Storage(e.to_string()))?; // FTS5 synced via trigger (memories_fts_au) self.cache_put(mem); @@ -896,11 +1000,7 @@ impl StorageBackend for SqliteStorage { Ok(()) } - fn list_by_tier( - &self, - tier: MemoryTier, - limit: usize, - ) -> Result, CortexError> { + fn list_by_tier(&self, tier: MemoryTier, limit: usize) -> Result, CortexError> { let conn = self.read_conn()?; let mut stmt = conn .prepare_cached( @@ -943,11 +1043,7 @@ impl StorageBackend for SqliteStorage { Ok(results) } - fn list_by_channel( - &self, - channel: &str, - limit: usize, - ) -> Result, CortexError> { + fn list_by_channel(&self, channel: &str, limit: usize) -> Result, CortexError> { let conn = self.read_conn()?; let mut stmt = conn .prepare_cached( @@ -1021,15 +1117,17 @@ impl StorageBackend for SqliteStorage { let mut stmt = conn .prepare_cached("INSERT OR REPLACE INTO links (source_id, target_id, relation, strength) VALUES (?1, ?2, ?3, ?4)") .map_err(|e| CortexError::Storage(e.to_string()))?; - stmt.execute(params![source_id.to_string(), target_id.to_string(), relation.as_str(), strength]) + stmt.execute(params![ + source_id.to_string(), + target_id.to_string(), + relation.as_str(), + strength + ]) .map_err(|e| CortexError::Storage(e.to_string()))?; Ok(()) } - fn get_links( - &self, - source_id: Uuid, - ) -> Result, CortexError> { + fn get_links(&self, source_id: Uuid) -> Result, CortexError> { let conn = self.read_conn()?; let mut stmt = conn .prepare_cached("SELECT target_id, relation, strength FROM links WHERE source_id = ?1") @@ -1046,10 +1144,9 @@ impl StorageBackend for SqliteStorage { for row in rows { let (target_str, rel_str, strength) = row.map_err(|e| CortexError::Storage(e.to_string()))?; - let target = Uuid::parse_str(&target_str) - .map_err(|e| CortexError::Storage(e.to_string()))?; - let relation = - LinkRelation::parse(&rel_str).unwrap_or(LinkRelation::RelatedTo); + let target = + Uuid::parse_str(&target_str).map_err(|e| CortexError::Storage(e.to_string()))?; + let relation = LinkRelation::parse(&rel_str).unwrap_or(LinkRelation::RelatedTo); results.push((target, relation, strength)); } Ok(results) @@ -1126,10 +1223,7 @@ impl StorageBackend for SqliteStorage { ) .map_err(|e| CortexError::Storage(e.to_string()))?; let person = stmt - .query_row( - params![channel, channel_user_id], - parse_person_row, - ) + .query_row(params![channel, channel_user_id], parse_person_row) .optional() .map_err(|e| CortexError::Storage(e.to_string()))?; @@ -1373,9 +1467,14 @@ impl StorageBackend for SqliteStorage { "SELECT id, tier, content_json, embedding_blob, temporal_json, source_json, salience_json, privacy_json, tags_json, metadata_json, links_json, content_hash, namespace FROM memories WHERE id IN ({})", placeholders.join(", ") ); - let mut stmt = conn.prepare(&sql).map_err(|e| CortexError::Storage(e.to_string()))?; + let mut stmt = conn + .prepare(&sql) + .map_err(|e| CortexError::Storage(e.to_string()))?; let id_strings: Vec = miss_ids.iter().map(|id| id.to_string()).collect(); - let params: Vec<&dyn rusqlite::types::ToSql> = id_strings.iter().map(|s| s as &dyn rusqlite::types::ToSql).collect(); + let params: Vec<&dyn rusqlite::types::ToSql> = id_strings + .iter() + .map(|s| s as &dyn rusqlite::types::ToSql) + .collect(); let rows = stmt .query_map(params.as_slice(), Self::parse_mem_row) .map_err(|e| CortexError::Storage(e.to_string()))?; @@ -1444,12 +1543,16 @@ impl StorageBackend for SqliteStorage { ORDER BY created_at DESC", conditions.join(" OR ") ); - let mut stmt = conn.prepare(&sql).map_err(|e| CortexError::Storage(e.to_string()))?; - let mut params_vec: Vec> = Vec::with_capacity(entities.len()); + let mut stmt = conn + .prepare(&sql) + .map_err(|e| CortexError::Storage(e.to_string()))?; + let mut params_vec: Vec> = + Vec::with_capacity(entities.len()); for entity in entities { params_vec.push(Box::new(format!("%{}%", entity.to_lowercase()))); } - let param_refs: Vec<&dyn rusqlite::types::ToSql> = params_vec.iter().map(|p| p.as_ref()).collect(); + let param_refs: Vec<&dyn rusqlite::types::ToSql> = + params_vec.iter().map(|p| p.as_ref()).collect(); let rows = stmt .query_map(param_refs.as_slice(), Self::parse_mem_row) .map_err(|e| CortexError::Storage(e.to_string()))?; @@ -1621,7 +1724,10 @@ impl StorageBackend for SqliteStorage { ) .map_err(|e| CortexError::Storage(e.to_string()))?; let rows = stmt - .query_map(params![tier.as_str(), ns, limit as i64], Self::parse_mem_row) + .query_map( + params![tier.as_str(), ns, limit as i64], + Self::parse_mem_row, + ) .map_err(|e| CortexError::Storage(e.to_string()))?; for row in rows { results.push(row.map_err(|e| CortexError::Storage(e.to_string()))?); @@ -1649,13 +1755,17 @@ impl StorageBackend for SqliteStorage { conditions.join(" OR "), limit ); - let mut stmt = conn.prepare(&sql).map_err(|e| CortexError::Storage(e.to_string()))?; - let mut params_vec: Vec> = Vec::with_capacity(terms.len() + 1); + let mut stmt = conn + .prepare(&sql) + .map_err(|e| CortexError::Storage(e.to_string()))?; + let mut params_vec: Vec> = + Vec::with_capacity(terms.len() + 1); params_vec.push(Box::new("episodic".to_string())); for term in terms { params_vec.push(Box::new(format!("%{}%", term.to_lowercase()))); } - let param_refs: Vec<&dyn rusqlite::types::ToSql> = params_vec.iter().map(|p| p.as_ref()).collect(); + let param_refs: Vec<&dyn rusqlite::types::ToSql> = + params_vec.iter().map(|p| p.as_ref()).collect(); let rows = stmt .query_map(param_refs.as_slice(), |row| { let id_str: String = row.get(0)?; @@ -1701,7 +1811,10 @@ impl StorageBackend for SqliteStorage { ) .map_err(|e| CortexError::Storage(e.to_string()))?; let rows = stmt - .query_map(params![tier.as_str(), limit as i64, offset as i64], Self::parse_mem_row) + .query_map( + params![tier.as_str(), limit as i64, offset as i64], + Self::parse_mem_row, + ) .map_err(|e| CortexError::Storage(e.to_string()))?; let mut results = Vec::new(); for row in rows { @@ -1710,10 +1823,7 @@ impl StorageBackend for SqliteStorage { Ok(results) } - fn batch_update_salience( - &self, - updates: &[(Uuid, String, f32)], - ) -> Result { + fn batch_update_salience(&self, updates: &[(Uuid, String, f32)]) -> Result { if updates.is_empty() { return Ok(0); } @@ -1763,7 +1873,8 @@ impl StorageBackend for SqliteStorage { .map_err(|e| CortexError::Storage(e.to_string()))?; for id in ids { let id_str = id.to_string(); - del_stmt.execute(params![id_str]) + del_stmt + .execute(params![id_str]) .map_err(|e| CortexError::Storage(e.to_string()))?; count += 1; } @@ -1832,7 +1943,7 @@ impl SqliteStorage { let mut min_rank = 0.0_f64; let mut max_rank = 0.0_f64; let mut raw: Vec<(Uuid, f64)> = Vec::new(); - let mut seen_ids = std::collections::HashSet::new(); // Prevent duplicates across retries + let mut seen_ids = std::collections::HashSet::new(); // Prevent duplicates across retries // Progressive over-fetching: start with 10x multiplier, scale up if needed // This prevents result count from leaking privacy distribution info @@ -2071,6 +2182,9 @@ mod row_parsing_tests { // Malformed JSON (e.g. a corrupted/tampered row) returns an Err, not a panic. let err = SqliteStorage::json_col::>(8, "not valid json"); - assert!(err.is_err(), "malformed JSON must map to an error, never panic"); + assert!( + err.is_err(), + "malformed JSON must map to an error, never panic" + ); } } diff --git a/cortex-core/tests/test_db_file_permissions.rs b/cortex-core/tests/test_db_file_permissions.rs index a67aa09..0e99344 100644 --- a/cortex-core/tests/test_db_file_permissions.rs +++ b/cortex-core/tests/test_db_file_permissions.rs @@ -13,7 +13,13 @@ fn mode(p: &Path) -> u32 { fn assert_owner_only(p: &Path) { let m = mode(p); - assert_eq!(m & 0o077, 0, "{} is {:o}; group/other must have no access", p.display(), m); + assert_eq!( + m & 0o077, + 0, + "{} is {:o}; group/other must have no access", + p.display(), + m + ); } #[test] @@ -21,7 +27,9 @@ fn new_database_and_wal_files_are_owner_only() { let dir = tempfile::tempdir().unwrap(); let db = dir.path().join("memory.db"); let cortex = Cortex::open(db.to_str().unwrap()).unwrap(); - cortex.ingest("my private diagnosis", "test", None, None, None).unwrap(); + cortex + .ingest("my private diagnosis", "test", None, None, None) + .unwrap(); assert_owner_only(&db); for suffix in ["-wal", "-shm"] { @@ -38,7 +46,9 @@ fn existing_world_readable_database_is_tightened_on_open() { let db = dir.path().join("memory.db"); { let cortex = Cortex::open(db.to_str().unwrap()).unwrap(); - cortex.ingest("written by an older version", "test", None, None, None).unwrap(); + cortex + .ingest("written by an older version", "test", None, None, None) + .unwrap(); } // Simulate a database created by an older release under a 022 umask. for name in ["memory.db", "memory.db-wal", "memory.db-shm"] { @@ -57,3 +67,26 @@ fn existing_world_readable_database_is_tightened_on_open() { } } } + +#[test] +fn sqlite_uri_cannot_bypass_private_file_guard() { + let dir = tempfile::tempdir().unwrap(); + let db = dir.path().join("uri.db"); + let uri = format!("file:{}?mode=rwc", db.display()); + assert!(Cortex::open(&uri).is_err()); + assert!(!db.exists()); + assert!(Cortex::open(":memory:").is_ok()); +} + +#[test] +fn symlink_and_hardlink_database_aliases_fail_closed() { + let dir = tempfile::tempdir().unwrap(); + let target = dir.path().join("original.db"); + drop(Cortex::open(target.to_str().unwrap()).unwrap()); + let link = dir.path().join("symlink.db"); + std::os::unix::fs::symlink(&target, &link).unwrap(); + assert!(Cortex::open(link.to_str().unwrap()).is_err()); + std::fs::remove_file(&link).unwrap(); + std::fs::hard_link(&target, &link).unwrap(); + assert!(Cortex::open(link.to_str().unwrap()).is_err()); +} diff --git a/cortex-python/src/lib.rs b/cortex-python/src/lib.rs index 09f7828..c17db04 100644 --- a/cortex-python/src/lib.rs +++ b/cortex-python/src/lib.rs @@ -176,43 +176,161 @@ fn invalid_controlled(py: Python<'_>) -> PyErr { cortex_core::controlled::ControlledError::InvalidInput("request"), ) } +fn bounded_value( + py: Python<'_>, + value: &Bound<'_, PyAny>, + depth: usize, + budget: &mut usize, + list_limit: usize, +) -> PyResult { + use pyo3::types::{PyBool, PyDict, PyFloat, PyInt, PyList, PyString}; + let invalid = || invalid_controlled(py); + if depth > 8 { + return Err(invalid()); + } + let debit = |budget: &mut usize, bytes: usize| -> PyResult<()> { + *budget = budget.checked_sub(bytes).ok_or_else(invalid)?; + Ok(()) + }; + debit(budget, 8)?; + if value.is_none() { + return Ok(serde_json::Value::Null); + } + if value.is_exact_instance_of::() { + return Ok(serde_json::Value::Bool( + value.extract().map_err(|_| invalid())?, + )); + } + if value.is_exact_instance_of::() { + return Ok(serde_json::Value::from( + value.extract::().map_err(|_| invalid())?, + )); + } + if value.is_exact_instance_of::() { + let number = value.extract::().map_err(|_| invalid())?; + return serde_json::Number::from_f64(number) + .map(serde_json::Value::Number) + .ok_or_else(invalid); + } + if value.is_exact_instance_of::() { + let string = value.downcast::().map_err(|_| invalid())?; + // Check Python's character length before asking for a UTF-8 buffer. + if string.len().map_err(|_| invalid())? > 12000 { + return Err(invalid()); + } + let text = string.to_str().map_err(|_| invalid())?; + if text.len() > 48000 { + return Err(invalid()); + } + let bytes = text + .bytes() + .map(|b| { + if b < 32 { + 6 + } else if b == b'"' || b == b'\\' { + 2 + } else { + 1 + } + }) + .sum::(); + debit(budget, bytes)?; + return Ok(serde_json::Value::String(text.to_owned())); + } + if value.is_exact_instance_of::() { + let list = value.downcast::().map_err(|_| invalid())?; + if list.len() > list_limit { + return Err(invalid()); + } + debit(budget, list.len())?; + let mut result = Vec::with_capacity(list.len()); + for item in list.iter() { + result.push(bounded_value(py, &item, depth + 1, budget, 100)?); + } + return Ok(serde_json::Value::Array(result)); + } + if value.is_exact_instance_of::() { + let dict = value.downcast::().map_err(|_| invalid())?; + if dict.len() > 16 { + return Err(invalid()); + } + let mut result = serde_json::Map::new(); + for (key, item) in dict.iter() { + if !key.is_exact_instance_of::() { + return Err(invalid()); + } + let key = key.downcast::().map_err(|_| invalid())?; + if key.len().map_err(|_| invalid())? > 64 { + return Err(invalid()); + } + let key = key.to_str().map_err(|_| invalid())?; + if key.len() > 64 { + return Err(invalid()); + } + debit(budget, key.len() * 6 + 3)?; + let cap = match key { + "embedding" => 4096, + "source_refs" => 16, + _ => 100, + }; + result.insert( + key.to_owned(), + bounded_value(py, &item, depth + 1, budget, cap)?, + ); + } + return Ok(serde_json::Value::Object(result)); + } + Err(invalid()) +} +fn parse_bounded( + py: Python<'_>, + value: &Bound<'_, PyAny>, + cap: usize, +) -> PyResult { + let value = bounded_value(py, value, 0, &mut (8 * 1024 * 1024), cap)?; + serde_json::from_value(value).map_err(|_| invalid_controlled(py)) +} fn parse_controlled( py: Python<'_>, value: &Bound<'_, PyAny>, ) -> PyResult { - let body: String = (|| -> PyResult<_> { - py.import("json")? - .call_method1("dumps", (value,))? - .extract() - })() - .map_err(|_| invalid_controlled(py))?; - if body.len() > 8 * 1024 * 1024 { - return Err(invalid_controlled(py)); - } - serde_json::from_str(&body).map_err(|_| invalid_controlled(py)) + parse_bounded(py, value, 100) } -fn controlled_result( +fn controlled_result( py: Python<'_>, value: Result, ) -> PyResult> { let value = value.map_err(|e| controlled_error(py, e))?; - let body = serde_json::to_string(&value).map_err(|_| invalid_controlled(py))?; + let body = py + .allow_threads(move || serde_json::to_string(&value)) + .map_err(|_| invalid_controlled(py))?; Ok(py.import("json")?.call_method1("loads", (body,))?.unbind()) } #[pyclass(name = "ControlledStore")] struct PyControlledStore { + creator_pid: u32, inner: parking_lot::Mutex>>, } impl PyControlledStore { + fn check_process(&self, py: Python<'_>) -> PyResult<()> { + if self.creator_pid != std::process::id() { + return Err(controlled_error( + py, + cortex_core::controlled::ControlledError::Conflict( + "inherited handle; open after fork", + ), + )); + } + Ok(()) + } fn handle( &self, py: Python<'_>, ) -> PyResult> { - self.inner - .lock() - .clone() - .ok_or_else(|| controlled_error(py, cortex_core::controlled::ControlledError::Closed)) + self.check_process(py)?; + let handle = { self.inner.lock().clone() }; + handle.ok_or_else(|| controlled_error(py, cortex_core::controlled::ControlledError::Closed)) } } #[pymethods] @@ -220,12 +338,17 @@ impl PyControlledStore { #[new] fn new( py: Python<'_>, - root: String, - owner_type: String, - owner_id: String, - bot_id: String, - replica_id: String, + root: &Bound<'_, PyAny>, + owner_type: &Bound<'_, PyAny>, + owner_id: &Bound<'_, PyAny>, + bot_id: &Bound<'_, PyAny>, + replica_id: &Bound<'_, PyAny>, ) -> PyResult { + let root: String = parse_controlled(py, root)?; + let owner_type: String = parse_controlled(py, owner_type)?; + let owner_id: String = parse_controlled(py, owner_id)?; + let bot_id: String = parse_controlled(py, bot_id)?; + let replica_id: String = parse_controlled(py, replica_id)?; let identity = cortex_core::controlled::StoreIdentity { owner_type, owner_id, @@ -241,6 +364,7 @@ impl PyControlledStore { }) .map_err(|e| controlled_error(py, e))?; Ok(Self { + creator_pid: std::process::id(), inner: parking_lot::Mutex::new(Some(std::sync::Arc::new(store))), }) } @@ -261,11 +385,12 @@ impl PyControlledStore { fn reconcile( &self, py: Python<'_>, - generation: String, + generation: &Bound<'_, PyAny>, position: &Bound<'_, PyAny>, ) -> PyResult> { let s = self.handle(py)?; let p = parse_controlled(py, position)?; + let generation: String = parse_controlled(py, generation)?; controlled_result(py, py.allow_threads(|| s.reconcile(&generation, p))) } fn apply_batch( @@ -301,22 +426,32 @@ impl PyControlledStore { let p = parse_controlled(py, position)?; controlled_result(py, py.allow_threads(|| s.delete(m, p))) } - #[pyo3(signature = (query, allowed_scopes, limit=10, embedding=None, max_candidates=1000, max_duration_ms=250))] - #[allow(clippy::too_many_arguments)] + #[pyo3(signature = (query, allowed_scopes, **options))] fn search( &self, py: Python<'_>, - query: String, + query: &Bound<'_, PyAny>, allowed_scopes: &Bound<'_, PyAny>, - limit: i64, - embedding: Option>, - max_candidates: i64, - max_duration_ms: i64, + options: Option<&Bound<'_, pyo3::types::PyDict>>, ) -> PyResult> { let s = self.handle(py)?; - let scopes: Vec = parse_controlled(py, allowed_scopes)?; - if limit < 0 || max_candidates < 0 || max_duration_ms < 0 { - return Err(invalid_controlled(py)); + let query: String = parse_controlled(py, query)?; + let scopes: Vec = parse_bounded(py, allowed_scopes, 64)?; + let mut limit: usize = 10; + let mut embedding: Option> = None; + let mut max_candidates: usize = 1000; + let mut max_duration_ms: u64 = 250; + if let Some(options) = options { + for (key, value) in options.iter() { + let key: String = parse_controlled(py, &key)?; + match key.as_str() { + "limit" => limit = parse_controlled(py, &value)?, + "embedding" => embedding = parse_bounded(py, &value, 4096)?, + "max_candidates" => max_candidates = parse_controlled(py, &value)?, + "max_duration_ms" => max_duration_ms = parse_controlled(py, &value)?, + _ => return Err(invalid_controlled(py)), + } + } } controlled_result( py, @@ -325,24 +460,31 @@ impl PyControlledStore { &query, &scopes, cortex_core::controlled::SearchOptions { - limit: limit as usize, + limit, embedding, - max_candidates: max_candidates as usize, - max_duration_ms: max_duration_ms as u64, + max_candidates, + max_duration_ms, }, ) }), ) } + fn retry_cleanup(&self, py: Python<'_>) -> PyResult> { + let s = self.handle(py)?; + controlled_result(py, py.allow_threads(|| s.retry_cleanup())) + } fn close(&self, py: Python<'_>) -> PyResult<()> { - let s = self.inner.lock().take().ok_or_else(|| { + self.check_process(py)?; + let handle = { self.inner.lock().take() }; + let s = handle.ok_or_else(|| { controlled_error(py, cortex_core::controlled::ControlledError::Closed) })?; py.allow_threads(move || { - s.close(); + let result = s.close(); drop(s); - }); - Ok(()) + result + }) + .map_err(|e| controlled_error(py, e)) } } diff --git a/cortex-python/tests/test_controlled_store.py b/cortex-python/tests/test_controlled_store.py index 3acdd9b..61eb5f3 100644 --- a/cortex-python/tests/test_controlled_store.py +++ b/cortex-python/tests/test_controlled_store.py @@ -512,6 +512,212 @@ def test_child_crash_after_commit_and_during_write(self): finally: self.root = original_root + def test_cleanup_pending_with_readonly_reader_then_retry_and_reopen(self): + store = self.ready([projection()]) + self.assertIsInstance(store.status()["cleanup_pending"], bool) + databases = [] + for path in Path(self.root).rglob("*.db"): + with path.open("rb") as handle: + if handle.read(16) == b"SQLite format 3\x00": + databases.append(path) + self.assertEqual(len(databases), 1) + blocker = r''' +import sqlite3, sys +connection = sqlite3.connect(sys.argv[1] + '?mode=ro', uri=True) +connection.execute('BEGIN') +connection.execute('SELECT count(*) FROM sqlite_master').fetchone() +print('ready', flush=True) +sys.stdin.readline() +connection.close() +''' + reader = subprocess.Popen([sys.executable, "-I", "-c", blocker, databases[0].as_uri()], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, text=True) + self.children.append(reader) + ready_lines = queue.Queue() + threading.Thread(target=lambda: ready_lines.put(reader.stdout.readline()), daemon=True).start() + self.assertEqual(ready_lines.get(timeout=10).strip(), "ready") + try: + receipt = store.upsert(projection(revision=2, text="orchid replacement"), 101) + self.assertTrue(receipt["durable"]) + self.assertTrue(store.status()["cleanup_pending"]) + begin = time.monotonic() + self.assertTrue(store.retry_cleanup()["cleanup_pending"]) + self.assertLess(time.monotonic() - begin, 5.0) + withdrawn = store.delete(deletion(revision=3), 102) + self.assertTrue(withdrawn["durable"]) + self.assertTrue(store.status()["cleanup_pending"]) + self.assertEqual(self.hits(store), []) + finally: + reader.communicate(input="release\n", timeout=10) + self.assertEqual(reader.returncode, 0) + begin = time.monotonic() + self.assertFalse(store.retry_cleanup()["cleanup_pending"]) + self.assertLess(time.monotonic() - begin, 5.0) + self.assertEqual(store.delete(deletion(revision=3), 102), withdrawn) + store.close() + reopened = self.open() + self.assertFalse(reopened.status()["cleanup_pending"]) + self.reconcile(reopened) + self.assertEqual(self.hits(reopened), []) + + def test_search_malformed_options_builtin_only_and_unicode(self): + store = self.ready([projection(text="İabout 猫 orchid")]) + result = store.search("İabout 猫", [SCOPE]) + self.assertIsInstance(result["hits"], list) + with self.assertRaises(TypeError): + store.search("orchid", [SCOPE], 10) + for name in ("limit", "max_candidates", "max_duration_ms"): + for value in (True, False, "10", None, 1.5, 2**100, -1, [], {}): + with self.subTest(option=name, value_type=type(value).__name__): + self.error("invalid_input", store.search, "orchid", [SCOPE], **{name: value}) + for vector in (True, "vector", [True], [1e308], [1.0, object()]): + self.error("invalid_input", store.search, "orchid", [SCOPE], embedding=vector) + self.error("invalid_input", store.search, " ".join("term%d" % i for i in range(65)), [SCOPE]) + class Custom: + pass + class DictSubclass(dict): + pass + class ListSubclass(list): + pass + class StringSubclass(str): + pass + class IntSubclass(int): + pass + for value in (Custom(), DictSubclass(projection()), + projection(text=StringSubclass("orchid")), projection(revision=IntSubclass(2))): + self.error("invalid_input", store.upsert, value, 101) + self.error("invalid_input", store.seed, ListSubclass([projection("new")])) + cyclic = [] + cyclic.append(cyclic) + self.error("invalid_input", store.seed, cyclic) + deep = [] + for _ in range(12): + deep = [deep] + self.error("invalid_input", store.seed, deep) + self.error("invalid_input", store.upsert, projection(text="x"*(8*1024*1024+1)), 101) + duplicate = projection("duplicate") + self.error("invalid_input", store.apply_batch, [upsert(duplicate), upsert(duplicate)], 101) + self.assertEqual(store.status()["position"], 100) + + def test_exact_submillisecond_interval_and_clock_filter(self): + start = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(seconds=0.15) + until = start + datetime.timedelta(microseconds=200) + item = projection("precise", valid_from=start.isoformat(), valid_until=until.isoformat()) + store = self.ready([item]) + deadline = time.monotonic() + 2.0 + while time.monotonic() < deadline: + before = datetime.datetime.now(datetime.timezone.utc) + hits = self.hits(store) + after = datetime.datetime.now(datetime.timezone.utc) + for hit in hits: + self.assertEqual(datetime.datetime.fromisoformat(hit["valid_from"].replace("Z", "+00:00")), start) + self.assertEqual(datetime.datetime.fromisoformat(hit["valid_until"].replace("Z", "+00:00")), until) + self.assertLess(before, until, "expired submillisecond candidate returned") + self.assertGreaterEqual(after, start, "future submillisecond candidate returned") + if before > until + datetime.timedelta(milliseconds=5): + self.assertEqual(hits, []) + break + else: + self.fail("precise interval test failed to reach its expiry checkpoint") + # Distinct bounds inside one millisecond must remain a valid interval. + tiny = projection("nanosecond", valid_from="2000-01-01T00:00:00.000000001Z", + valid_until="2000-01-01T00:00:00.000000002Z") + store.upsert(tiny, 101) + self.assertEqual(self.hits(store), []) + + def test_unsafe_nonsticky_ancestor_rejected(self): + ancestor = Path(self.temp.name).resolve() / "unsafe-parent" + ancestor.mkdir(mode=0o700) + try: + for mode in (0o770, 0o707, 0o777): + ancestor.chmod(mode) + self.error("invalid_input", self.open, root=str(ancestor / "private")) + finally: + ancestor.chmod(0o700) + + def test_fork_inherited_handle_fails_before_copied_locks(self): + script = r''' +import json, os, signal, sys +from cortex_ai_memory import ControlledStore, ControlledError +s = ControlledStore(sys.argv[1], 'merchant', 'merchant-acceptance', 'bot-one', 'replica-one') +s.finish_snapshot(100) +s.reconcile(s.status()['generation'], 100) +r, w = os.pipe() +pid = os.fork() +if pid == 0: + signal.alarm(5) + os.close(r) + codes = [] + try: + for call in (s.status, lambda:s.search('orchid', []), s.retry_cleanup, s.close): + try: + call() + codes.append('unexpected-success') + except ControlledError as e: + codes.append(e.code) + os.write(w, json.dumps(codes).encode()) + finally: + os._exit(0) +os.close(w) +data = os.read(r, 65536) +os.waitpid(pid, 0) +print(data.decode(), flush=True) +s.close() +''' + completed = subprocess.run([sys.executable, "-I", "-c", script, self.root], + capture_output=True, text=True, timeout=15) + self.assertEqual(completed.returncode, 0, completed.stderr) + self.assertEqual(json.loads(completed.stdout), ["conflict"]*4) + reopened = self.open() + self.reconcile(reopened) + + def test_immediate_reopen_while_old_thread_calls_are_queued(self): + store = self.ready([projection()]) + barrier = threading.Barrier(9) + def caller(): + barrier.wait(timeout=10) + for _ in range(100): + try: + store.search("orchid", [SCOPE]) + except ControlledError as error: + self.assertEqual(error.code, "closed") + return + with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool: + futures = [pool.submit(caller) for _ in range(8)] + barrier.wait(timeout=10) + store.close() + # Reopen before waiting for old Python callers to finish: none may + # reacquire ownership or touch the newly opened generation. + reopened = self.open() + self.reconcile(reopened) + reopened.upsert(projection(revision=2, text="orchid reopened"), 101) + for future in futures: + future.result(timeout=15) + self.assertEqual(self.hits(reopened)[0]["text"], "orchid reopened") + + def test_unauthorized_term_load_cannot_change_authorized_ranking(self): + allowed = [projection("allowed-exact", text="orchid orchid report", embedding=[1.0, 0.0]), + projection("allowed-other", text="orchid other report", embedding=[0.0, 1.0])] + store = self.open() + store.seed(allowed) + store.finish_snapshot(100) + self.reconcile(store) + baseline = store.search("orchid report", [SCOPE], limit=2, max_candidates=2, embedding=[1.0, 0.0]) + signature = [(hit["memory_id"], hit["score"]) for hit in baseline["hits"]] + self.assertEqual({key for key, _ in signature}, {"allowed-exact", "allowed-other"}) + position = 100 + for offset in range(0, 1500, 100): + position += 1 + store.apply_batch([upsert(projection("excluded-load-%d" % i, scope=OTHER, + text="orchid "*30 + "report")) + for i in range(offset, offset+100)], position) + for _ in range(3): + result = store.search("orchid report", [SCOPE], limit=2, max_candidates=2, embedding=[1.0, 0.0]) + self.assertEqual([(hit["memory_id"], hit["score"]) for hit in result["hits"]], signature) + self.assertEqual(result["degraded"], baseline["degraded"]) + self.assertEqual(result["truncated"], baseline["truncated"]) + def test_restored_local_transaction_loss_rejects_remote_cursor(self): store = self.ready([projection()]) state = store.status() diff --git a/cortex-python/tests/test_foreign_owned_database.py b/cortex-python/tests/test_foreign_owned_database.py new file mode 100644 index 0000000..c35a8c7 --- /dev/null +++ b/cortex-python/tests/test_foreign_owned_database.py @@ -0,0 +1,52 @@ +"""Installed-wheel acceptance requiring a real foreign-owned Linux fixture. + +CI must create a regular file, chown it to another uid (for example nobody), +chmod it 0666, and set CORTEX_FOREIGN_OWNED_DB to its absolute path. Run this +script with the installed-wheel interpreter and -I. Missing prerequisites fail; +this test neither fabricates foreign ownership nor skips the security boundary. +""" +import gc +import hashlib +import os +from pathlib import Path +import stat +import unittest + +import cortex_ai_memory as memory + + +def fingerprint(path): + digest = hashlib.sha256() + with path.open("rb") as handle: + for block in iter(lambda: handle.read(65536), b""): + digest.update(block) + metadata = path.lstat() + return (metadata.st_dev, metadata.st_ino, metadata.st_uid, metadata.st_gid, + stat.S_IMODE(metadata.st_mode), metadata.st_size, digest.hexdigest()) + + +class ForeignOwnedDatabaseAcceptance(unittest.TestCase): + def test_personal_open_refuses_foreign_owned_world_writable_file_unchanged(self): + configured = os.environ.get("CORTEX_FOREIGN_OWNED_DB") + self.assertTrue(configured, "CI must supply CORTEX_FOREIGN_OWNED_DB") + path = Path(configured) + self.assertTrue(path.is_absolute(), "fixture path must be absolute") + metadata = path.lstat() + self.assertTrue(stat.S_ISREG(metadata.st_mode), "fixture must be a regular file") + self.assertNotEqual(metadata.st_uid, os.geteuid(), "fixture must have a genuinely foreign uid") + self.assertEqual(stat.S_IMODE(metadata.st_mode), 0o666, + "fixture must start world-writable, without special mode bits") + checkout = Path(__file__).resolve().parents[2] + self.assertNotIn(checkout, Path(memory.__file__).resolve().parents, + "run against the installed wheel, not adjacent source") + before = fingerprint(path) + with self.assertRaises(RuntimeError): + instance = memory.Cortex(str(path.resolve())) + del instance + gc.collect() + self.assertEqual(fingerprint(path), before, + "refused open must preserve foreign file identity, owner, mode and content") + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/controlled-memory-api.md b/docs/controlled-memory-api.md index 25f6046..abb3979 100644 --- a/docs/controlled-memory-api.md +++ b/docs/controlled-memory-api.md @@ -7,7 +7,7 @@ Implementation contract for Cortex-side P0. Python distribution version: **2.5.0 The existing PyPI 2.4.0 release does not contain this API. Use the exact 2.5.0 wheel from the reviewed PR's `controlled-wheel-*` CI artifact for your platform, or build from the reviewed source commit. The CI covers Python 3.12 on Linux x86_64 and macOS -arm64. Wheel files are platform/interpreter specific; do not install an incompatible +arm64. The Python release profile retains native symbols to avoid the reproduced macOS stripped-LINKEDIT load failure. Wheel files are platform/interpreter specific; do not install an incompatible artifact or assume other targets have been verified. ```sh @@ -24,13 +24,14 @@ SHA and artifact checksum with the host's existing dependency manifest. ## Store and lifecycle -`ControlledStore(root, owner_type, owner_id, bot_id, replica_id)` opens a dedicated owner/bot/replica database under an absolute, private root. Paths inside that root are generated from the identity. The root must not be a symlink or contain `..`; resolve platform temporary-directory aliases before passing it. One process/handle may own a replica at a time. Different replicas have separate generation IDs and cursors. A personal database is never imported. Current legacy open/export paths refuse controlled databases. Older binaries must not be given these files. +`ControlledStore(root, owner_type, owner_id, bot_id, replica_id)` opens a dedicated owner/bot/replica database under an absolute, private root. Paths inside that root are generated from the identity. The root must not be a symlink or contain `..`; resolve platform temporary-directory aliases before passing it. Existing ancestors must be owned by the service user or root and must not be group/world-writable unless sticky, preventing an untrusted local account from replacing the validated root. The root itself is always owner-only. One process/handle may own a replica at a time. Handles inherited through `fork` raise `conflict` before touching copied locks/SQLite; create handles after forking, with a separate replica for each worker. Different replicas have separate generation IDs and cursors. A personal database is never imported. Current legacy open/export paths refuse controlled databases. Older binaries must not be given these files. -- `status()` returns `{generation, position, sealed, ready}`. Every reopen starts not ready. +- `status()` returns `{generation, position, sealed, ready, cleanup_pending}`. Every reopen starts not ready. - `seed(projections)` atomically loads a batch of current authoritative memories while the snapshot is unsealed. Retrying identical revisions is safe. No seed after snapshot completion. - `finish_snapshot(position)` durably seals a new snapshot at the PG committed snapshot watermark; does not enable queries. Repeating the same watermark succeeds; changing it after sealing conflicts. - `reconcile(generation, position)` enables queries only after both exactly match local durable status and snapshot is sealed. An unsealed snapshot raises `not_ready`; a generation/position mismatch raises `conflict` and queries remain disabled. The caller must validate the source snapshot, reconcile retained changes or rebuild a new replica before this call; equality alone does not authorize data. - `apply_batch(mutations, position)` durably and atomically applies the next sequential replay position. Empty batches can advance a position (e.g. an irrelevant event). Equal latest position is accepted only for the identical normalized batch and returns its original receipt. Older/gapped positions conflict without modifying data. Upsert/delete convenience methods accept one projection/deletion and the same position. +- `retry_cleanup()` retries physical SQLite/FTS/WAL cleanup and returns current status. Logical mutation receipts stay immutable. Every payload replacement/withdrawal atomically sets `cleanup_pending`; it remains true while a reader blocks WAL truncation or cleanup fails. A successful serialized retry clears it. Retry with bounded host backoff; backups and external copies require host retention controls. - `close()` refuses subsequent calls and drains current native operations before releasing the store lock. A new store object can reopen/reconcile the replica. Use a new replica ID for a fresh rebuild, reconcile it, atomically route new calls to it, drain/close the previous object, then retire old files under retention policy. No unsafe in-place file swap API is exposed. Each projection has `memory_id` (opaque external ID), positive integer `revision`, `scope: {kind,id}`, `kind` (`fact`, `preference`, `experience`, `method`), `text`, `source_refs: [{kind,provider,id}]`, optional RFC3339 `valid_from`/`valid_until`, and optional numeric `embedding`. Missing validity bounds are unbounded. Validity is `[valid_from, valid_until)`. Scope kind is `owner`, `entity`, `project`, `workspace`, or `user`; owner scope ID must equal the store owner ID. Scope never changes for an existing memory ID, including after deletion. To change it, withdraw that identity and create another. Projection state is implicitly active; no inferred/candidate memory is accepted automatically. @@ -41,15 +42,15 @@ Mutations are `{"op":"upsert","projection":...}` or `{"op":"delete","deletion":. ## Retrieval -`search(query, allowed_scopes, limit=10, embedding=None, max_candidates=1000, max_duration_ms=250)` returns `{hits, degraded, truncated}`. Hits include owner_type/owner_id/bot_id, memory_id, revision, scope, kind, text, source_refs, validity bounds and score. Native methods release the Python GIL. Call them through bounded host worker capacity, not unlimited async tasks. Cancellation cannot roll back an already committed mutation; retry the original position/batch. +`search(query, allowed_scopes, *, limit=10, embedding=None, max_candidates=1000, max_duration_ms=250)` returns `{hits, degraded, truncated}`. Hits include owner_type/owner_id/bot_id, memory_id, revision, scope, kind, text, source_refs, validity bounds and score. Native methods release the Python GIL. Call them through bounded host worker capacity, not unlimited async tasks. Cancellation cannot roll back an already committed mutation; retry the original position/batch. -`allowed_scopes` is required, supplied by the trusted host, and an empty list returns no hits. Candidates are filtered in SQL for scopes and current validity bounds before ranking/top-k; disabled graph/unscoped expansion cannot contribute. The existing FTS, cosine-similarity and ranking machinery are reused. FTS contributes a match signal without global BM25 statistics; only bounded authorized candidates are scored, so excluded records do not influence ranking. No request-local HNSW graph is built. The soft deadline covers lock acquisition, SQL work, vector scoring, ranking and hit projection, with cooperative checks between bounded work units; it is not a hard real-time OS scheduling SLA. Work is bounded by candidate/time limits; exhaustion reports degradation, never proof that no memory exists. Semantic lookup uses explicitly supplied vectors; no model download/inference occurs. Cortex enforces one vector dimension per store; the host must also keep the embedding model consistent. Model changes require rebuilding with new vectors. +`allowed_scopes` is required, supplied by the trusted host, and an empty list returns no hits. Candidates are filtered in SQL for scopes and coarse millisecond validity bounds, then exact RFC3339 timestamp precision is checked before ranking and again before returning hits; disabled graph/unscoped expansion cannot contribute. The existing FTS, cosine-similarity and ranking machinery are reused. FTS contributes a match signal without global BM25 statistics; only bounded authorized candidates are scored, so excluded records do not influence ranking. No request-local HNSW graph is built. The soft deadline covers lock acquisition, SQL work, vector scoring, ranking and hit projection, with cooperative checks between bounded work units; it is not a hard real-time OS scheduling SLA. Work is bounded by candidate/time limits; exhaustion reports degradation, never proof that no memory exists. Semantic lookup uses explicitly supplied vectors; no model download/inference occurs. Cortex enforces one vector dimension per store; the host must also keep the embedding model consistent. Model changes require rebuilding with new vectors. Every hit is an **internal candidate**. Before any UI/log/context exposure, the Bot must reload current PG state, validate identity/scope/source access, active/withdrawn state, exact revision and both validity bounds using server time, and emit current PG text. Resolve required defaults by exact key from PG independently of Cortex hits/readiness. Recheck validity and withdrawal at model/action boundaries. Never inject Cortex-generated prompts alongside ContextProjector. If lookup is unavailable or incomplete use the existing bounded PG fallback; authority lookup failures fail closed. ## Limits and errors -Identifiers: 1–256 UTF-8 bytes, no control characters. Owner/scope kinds are constrained as above. Text: at most 12,000 characters / 48,000 bytes. Sources: 1–16; each field uses identifier limits. Batches: at most 100 entries. Query: 1–4,000 characters and at most 64 alphanumeric terms; limit 1–100; at most 64 allowed scopes; candidates 1–10,000; time budget 1–250 ms; embeddings 1–4,096 finite floats with finite squared norm, with a consistent store-wide dimension. Parsed JSON request size is capped at 8 MiB. Duplicate memory IDs inside one batch are rejected. Unknown fields, bool/fractional/negative revisions or positions, invalid intervals and malformed vectors fail validation; source order is normalized for replay identity. +Identifiers: 1–256 UTF-8 bytes, no control characters. Owner/scope kinds are constrained as above. Text: at most 12,000 characters / 48,000 bytes. Sources: 1–16; each field uses identifier limits. Batches: at most 100 entries. Query: 1–4,000 characters and at most 64 alphanumeric terms; limit 1–100; at most 64 allowed scopes; candidates 1–10,000; time budget 1–250 ms; embeddings 1–4,096 finite floats with finite squared norm, with a consistent store-wide dimension. Python inputs must use exact built-in dict/list/string/int/float/bool/None types; custom objects and container subclasses are rejected. Conversion checks container limits before copying, rejects nesting deeper than eight levels, and enforces a conservative 8 MiB JSON-size budget incrementally. Search options are keyword-only; malformed values (including bool or overflowing bounds) raise `invalid_input`. Duplicate memory IDs inside one batch are rejected. Unknown fields, bool/fractional/negative revisions or positions, invalid intervals and malformed vectors fail validation; source order is normalized for replay identity. `ControlledError` has stable `.code`: `invalid_input`, `conflict`, `busy`, `not_ready`, `closed`, or `storage`. Stale memory revisions are receipt outcomes, not exceptions. Storage/commit ambiguity disables queries until reopen/reconciliation. No error text contains projection bodies. Python integer positions/revisions preserve precision up to SQLite's signed 64-bit maximum. @@ -77,4 +78,6 @@ store.close() The current wheel uses plaintext SQLite; private filesystem controls are mandatory. No automatic sync/export/inference/consolidation is enabled. Logical withdrawal does not promise immediate erasure from backups or already-sent model requests. See the [integration plan](design/bot-memory-integration.md) for host-owned replay, rebuild, rollout and real-PG acceptance. +Filesystem SQLite URI names (`file:...`) are rejected by the private-file guard; use a filesystem path or `:memory:`. + The existing personal API constructor is `Cortex(db_path)`; it raises `RuntimeError` for a controlled database. It has no explicit close method; release its references when no longer used. diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index 620cb20..253b428 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -171,7 +171,7 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr An acceptance subagent must be context-isolated from implementation: give it this plan and public schemas/docs, not implementation diffs or unit tests. Implementers write unit tests separately. Adversarial review is a third independent context, per repository rules. -Cortex-side automated evidence includes the independent installed-wheel acceptance suite (17 cases covering C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. +Cortex-side automated evidence includes the independent installed-wheel acceptance suite (24 cases exercising Cortex-side portions of C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. The public timing-based crash cases establish restart consistency; separate native test-only barriers deterministically terminate a real subprocess inside the transaction and after commit before cache invalidation/response. These establish process-crash behavior, not power-loss durability or a real-PG replay handshake. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. ## 9. Review, CI, rollout and evidence From b3eb34e44d870151bd4d609fba1ae5723118340f Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 12:26:23 -0400 Subject: [PATCH 11/15] docs: align replica ownership and cleanup API with implementation --- docs/design/bot-memory-integration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index 253b428..610a19d 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -13,7 +13,7 @@ The upstream Bot contract is [kol-agent PR #24](https://github.com/HEchooo/kol-a - ContextProjector remains the sole model-context composer. Raw Cortex `get_context()` output is not injected alongside it. - Workflow retains runs, checkpoints, approvals, actions, budget barriers, and recovery. Cortex receives no wallet, publishing, or execution authority. - A local revision is an object concurrency token, not a cross-repository version/hash admission rule. No shared ORM or writes to another service's database. -- Existing personal Cortex, MCP, HTTP, sync, gateway, and cloud behavior remains compatible. The new integration uses a separate controlled store and explicit opt-in; personal archives and Muse exports are never imported implicitly. +- Existing personal Cortex, MCP, HTTP, sync, gateway, and cloud API signatures remain compatible. Private-file enforcement now rejects unsupported SQLite URI names and unsafe filesystem aliases as described in the API contract. The new integration uses a separate controlled store and explicit opt-in; personal archives and Muse exports are never imported implicitly. ## 2. Evidence and reuse at the planning baseline @@ -81,7 +81,7 @@ Use a dedicated database per owner/bot and per serving-process replica. Derive p Persist a controlled-store marker. New legacy open/export/sync paths must refuse a marked controlled store, and controlled open refuses a personal store; do not rely only on caller discipline. The controlled binding exposes no general core/storage handle, legacy mutation, export, sync, or plugin access. Older binaries cannot honor a new marker, so deployment paths/access policy and rollback prohibit opening the controlled files with them. -Each replica has its own persistent replay cursor and instance ID; do not use a shared work-queue acknowledgement as proof that every replica has applied an event. P0 supports independent local replicas, not multiple processes sharing one index database. Enforce one live controlled handle per database with an in-process registry and an OS-released-on-exit lock; a second process fails with an explicit busy result. This does not replace SQLite locks or imply a distributed lease. +Each replica has its own persistent replay cursor and instance ID; do not use a shared work-queue acknowledgement as proof that every replica has applied an event. P0 supports independent local replicas, not multiple processes sharing one index database. Enforce one live controlled handle per database with an OS-released-on-exit file lock, including duplicate opens within one process; another handle or process fails with an explicit busy result. This does not replace SQLite locks or imply a distributed lease. The Bot adapter manages replay for each replica. A new/lost replica builds from current PostgreSQL state; an expired replay cursor forces rebuild. A paused or lagging replica may serve only results that survive current authoritative revalidation, otherwise the Bot uses its PostgreSQL fallback. Record bounded lag and recovery state without memory text. @@ -109,7 +109,7 @@ Logical withdrawal and physical erasure are distinct: no immediate wiping of exi ## 5. Python delivery and operational fallback (P0) -The Python `ControlledStore` binding exposes open/seed/finish_snapshot/reconcile/apply_batch/upsert/delete/search/status/close; exact dictionary schemas and limits are in the [API contract](../controlled-memory-api.md). Errors distinguish invalid scope/input, stale revision, equal-revision conflict, busy/not-ready store, and bounded retrieval degradation. Search results must not require parsing a generated prompt or tuple text to recover identity. +The Python `ControlledStore` binding exposes open/seed/finish_snapshot/reconcile/apply_batch/upsert/delete/search/status/retry_cleanup/close; exact dictionary schemas and limits are in the [API contract](../controlled-memory-api.md). Errors distinguish invalid scope/input, stale revision, equal-revision conflict, busy/not-ready store, and bounded retrieval degradation. Search results must not require parsing a generated prompt or tuple text to recover identity. P0 runtime uses a supported wheel installed in a clean environment, not an adjacent Rust checkout. Add a built-wheel smoke/integration test covering import/class name, controlled APIs, restart, and scope/delete behavior. Rust workspace CI excludes cortex-python and therefore cannot substitute for this test. Verify the host's actual Python, OS, and architecture; add the binding job to the required integration checks rather than treating a skipped release workflow as acceptance. Coordinate macOS fixes with PR #14. From 7da252d8e49393813350bf765d83b24890d25140 Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 13:01:24 -0400 Subject: [PATCH 12/15] fix: harden store creation, process locks and historical erasure --- cortex-core/Cargo.toml | 1 + cortex-core/src/controlled.rs | 421 +++++++++++++++++-- cortex-core/src/lib.rs | 9 +- cortex-core/src/storage/sqlite.rs | 270 ++++++++---- cortex-python/src/lib.rs | 48 ++- cortex-python/tests/test_controlled_store.py | 207 +++++++++ docs/controlled-memory-api.md | 16 +- docs/design/bot-memory-integration.md | 8 +- 8 files changed, 834 insertions(+), 146 deletions(-) diff --git a/cortex-core/Cargo.toml b/cortex-core/Cargo.toml index 2f1445d..de12082 100644 --- a/cortex-core/Cargo.toml +++ b/cortex-core/Cargo.toml @@ -50,3 +50,4 @@ harness = false [target.'cfg(unix)'.dependencies] libc = "0.2" +tempfile = "3" diff --git a/cortex-core/src/controlled.rs b/cortex-core/src/controlled.rs index 79b0394..45aadb3 100644 --- a/cortex-core/src/controlled.rs +++ b/cortex-core/src/controlled.rs @@ -14,7 +14,7 @@ use sha2::{Digest, Sha256}; use std::collections::{HashMap, HashSet}; use std::fs::File; use std::path::{Component, Path, PathBuf}; -use std::sync::Arc; +use std::sync::{Arc, OnceLock}; use std::time::{Duration, Instant}; use uuid::Uuid; @@ -168,11 +168,63 @@ impl Default for SearchOptions { } } +static OPEN_REPLICAS: OnceLock>> = OnceLock::new(); +struct ReplicaLease { + path: PathBuf, + pid: u32, +} +impl ReplicaLease { + fn acquire(path: &Path) -> Result { + // guard_open_process has already rejected fork children before this mutex. + let mut open = OPEN_REPLICAS + .get_or_init(|| Mutex::new(HashSet::new())) + .lock(); + if !open.insert(path.to_owned()) { + return Err(ControlledError::Busy); + } + Ok(Self { + path: path.to_owned(), + pid: std::process::id(), + }) + } +} +impl Drop for ReplicaLease { + fn drop(&mut self) { + if self.pid == std::process::id() { + if let Some(open) = OPEN_REPLICAS.get() { + open.lock().remove(&self.path); + } + } + } +} +#[cfg(unix)] +fn lock_replica(file: &File) -> Result<()> { + use std::os::fd::AsRawFd; + // POSIX process-owned record locks are not inherited through fork. In-process + // exclusion happens before opening this inode, so no duplicate FD close can + // silently release our lock. SQLite uses a different inode for its own locks. + let mut lock: libc::flock = unsafe { std::mem::zeroed() }; + lock.l_type = libc::F_WRLCK as _; + lock.l_whence = libc::SEEK_SET as _; + if unsafe { libc::fcntl(file.as_raw_fd(), libc::F_SETLK, &lock) } == -1 { + return match std::io::Error::last_os_error().raw_os_error() { + Some(libc::EAGAIN) | Some(libc::EACCES) => Err(ControlledError::Busy), + _ => Err(ControlledError::Storage), + }; + } + Ok(()) +} +#[cfg(not(unix))] +fn lock_replica(_: &File) -> Result<()> { + Err(ControlledError::InvalidInput("Unix required")) +} + struct State { storage: Option, ready: bool, closed: bool, file: Option, + lease: Option, } impl State { fn storage(&self) -> Result<&SqliteStorage> { @@ -277,7 +329,7 @@ fn private_file(path: &Path) -> Result { .create(true) .truncate(false) .mode(0o600) - .custom_flags(libc::O_NOFOLLOW) + .custom_flags(libc::O_NOFOLLOW | libc::O_NONBLOCK) .open(path) .map_err(|_| ControlledError::InvalidInput("unsafe store file"))?; let m = f.metadata().map_err(|_| ControlledError::Storage)?; @@ -295,6 +347,52 @@ fn private_file(_: &Path) -> Result { )) } +fn prepare_root(root: &Path, before_create: impl Fn(&Path)) -> Result<()> { + let mut prefix = PathBuf::new(); + for component in root.components() { + prefix.push(component); + match std::fs::symlink_metadata(&prefix) { + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + before_create(&prefix); + let mut builder = std::fs::DirBuilder::new(); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt; + builder.mode(0o700); + } + // One component at a time. mkdir never follows a precreated final + // symlink; after EEXIST we validate the actual entry, not its target. + match builder.create(&prefix) { + Ok(()) => {} + Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {} + Err(_) => return Err(ControlledError::Storage), + } + } + Err(_) => return Err(ControlledError::Storage), + Ok(_) => {} + } + let meta = std::fs::symlink_metadata(&prefix).map_err(|_| ControlledError::Storage)?; + if !meta.is_dir() { + return Err(ControlledError::InvalidInput("unsafe root entry")); + } + #[cfg(unix)] + { + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + let uid = unsafe { libc::geteuid() }; + let mode = meta.permissions().mode(); + if (meta.uid() != 0 && meta.uid() != uid) + || (mode & 0o022 != 0 && mode & 0o1000 == 0) + || (prefix == root && (meta.uid() != uid || mode & 0o077 != 0)) + { + return Err(ControlledError::InvalidInput( + "untrusted or nonprivate root", + )); + } + } + } + Ok(()) +} + fn effective(p: &Projection, now: DateTime) -> bool { p.valid_from.is_none_or(|t| t <= now) && p.valid_until.is_none_or(|t| now < t) } @@ -307,12 +405,18 @@ impl Drop for ControlledStore { if let Some(storage) = s.storage.take() { std::mem::forget(storage); } + if let Some(file) = s.file.take() { + std::mem::forget(file); + } } } } impl ControlledStore { pub fn open(root: &Path, identity: StoreIdentity) -> Result { + if !SqliteStorage::guard_open_process() { + return Err(ControlledError::Conflict("use spawn/exec after fork")); + } for x in [ &identity.owner_type, &identity.owner_id, @@ -326,46 +430,14 @@ impl ControlledStore { "absolute private root required", )); } - let mut prefix = PathBuf::new(); - for c in root.components() { - prefix.push(c); - if let Ok(m) = std::fs::symlink_metadata(&prefix) { - if m.file_type().is_symlink() { - return Err(ControlledError::InvalidInput("symlink root")); - } - #[cfg(unix)] - { - use std::os::unix::fs::{MetadataExt, PermissionsExt}; - let uid = unsafe { libc::geteuid() }; - let mode = m.permissions().mode(); - // An untrusted ancestor owner or writable non-sticky parent - // could replace the private root between validation and open. - if (m.uid() != 0 && m.uid() != uid) || (mode & 0o022 != 0 && mode & 0o1000 == 0) - { - return Err(ControlledError::InvalidInput("untrusted root ancestor")); - } - } - } - } - let mut builder = std::fs::DirBuilder::new(); - builder.recursive(true); - #[cfg(unix)] - { - use std::os::unix::fs::DirBuilderExt; - builder.mode(0o700); - } - builder.create(root).map_err(|_| ControlledError::Storage)?; - #[cfg(unix)] - { - use std::os::unix::fs::{MetadataExt, PermissionsExt}; - let m = std::fs::metadata(root).map_err(|_| ControlledError::Storage)?; - if m.uid() != unsafe { libc::geteuid() } || m.permissions().mode() & 0o077 != 0 { - return Err(ControlledError::InvalidInput("root must be owner-only")); - } - } - let path = root.join(format!("{}.db", digest(&identity)?)); + prepare_root(root, |_| {})?; + let path = root + .canonicalize() + .map_err(|_| ControlledError::Storage)? + .join(format!("{}.db", digest(&identity)?)); + let lease = ReplicaLease::acquire(&path)?; let file = private_file(&path.with_extension("lock"))?; - file.try_lock().map_err(|_| ControlledError::Busy)?; + lock_replica(&file)?; for suffix in ["", "-wal", "-shm", "-journal"] { let p = PathBuf::from(format!("{}{suffix}", path.display())); if suffix.is_empty() || p.exists() || std::fs::symlink_metadata(&p).is_ok() { @@ -393,6 +465,7 @@ impl ControlledStore { ready: false, closed: false, file: Some(file), + lease: Some(lease), }), identity, creator_pid: std::process::id(), @@ -474,6 +547,7 @@ impl ControlledStore { // no old Arc can retain SQLite or the OS lock after close returns. drop(s.storage.take()); drop(s.file.take()); + drop(s.lease.take()); Ok(()) } pub fn reconcile(&self, generation: &str, position: i64) -> Result { @@ -1144,4 +1218,269 @@ mod tests { .all(|r| matches!(r, Err(ControlledError::InvalidInput(_))))); assert_eq!(std::fs::read_dir(&target).unwrap().count(), 0); } + #[cfg(unix)] + #[test] + fn missing_root_cannot_follow_symlink_inserted_under_sticky_parent() { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let base = dir.path().canonicalize().unwrap(); + let sticky = base.join("sticky"); + std::fs::create_dir(&sticky).unwrap(); + std::fs::set_permissions(&sticky, std::fs::Permissions::from_mode(0o1777)).unwrap(); + let target = base.join("outside"); + std::fs::create_dir(&target).unwrap(); + std::fs::set_permissions(&target, std::fs::Permissions::from_mode(0o700)).unwrap(); + let root = sticky.join("initially-missing"); + let result = prepare_root(&root, |prefix| { + assert_eq!(prefix, root); + std::os::unix::fs::symlink(&target, prefix).unwrap(); + }); + assert_eq!(result.unwrap_err().code(), "invalid_input"); + assert_eq!(std::fs::read_dir(&target).unwrap().count(), 0); + } + #[test] + fn writer_probe_child() { + let Ok(path) = std::env::var("CORTEX_WRITER_PROBE_PATH") else { + return; + }; + let busy_expected = std::env::var("CORTEX_WRITER_PROBE_BUSY").unwrap() == "1"; + let conn = rusqlite::Connection::open(path).unwrap(); + conn.busy_timeout(Duration::from_millis(100)).unwrap(); + let result = conn.execute_batch("BEGIN IMMEDIATE; ROLLBACK;"); + if busy_expected { + match result { + Err(rusqlite::Error::SqliteFailure(error, _)) => assert!(matches!( + error.code, + rusqlite::ErrorCode::DatabaseBusy | rusqlite::ErrorCode::DatabaseLocked + )), + other => panic!("active writer lock was lost: {other:?}"), + } + } else { + result.unwrap(); + } + } + fn probe_writer(path: &Path, busy: bool) { + let mut child = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "controlled::tests::writer_probe_child", + "--exact", + "--nocapture", + ]) + .env("CORTEX_WRITER_PROBE_PATH", path) + .env("CORTEX_WRITER_PROBE_BUSY", if busy { "1" } else { "0" }) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .unwrap(); + let deadline = Instant::now() + Duration::from_secs(15); + while child.try_wait().unwrap().is_none() { + if Instant::now() >= deadline { + let _ = child.kill(); + let _ = child.wait(); + panic!("writer probe timed out"); + } + std::thread::sleep(Duration::from_millis(10)); + } + let output = child.wait_with_output().unwrap(); + assert!( + output.status.success(), + "writer probe failed: {} {}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + } + #[test] + fn opening_personal_or_rejecting_controlled_does_not_release_live_wal_locks() { + let dir = tempfile::tempdir().unwrap(); + let base = dir.path().canonicalize().unwrap(); + let personal_path = base.join("personal.db"); + let personal = SqliteStorage::open(personal_path.to_str().unwrap()).unwrap(); + personal + .with_controlled_conn(|conn| -> Result<()> { + conn.execute_batch("BEGIN IMMEDIATE;")?; + probe_writer(&personal_path, true); + let second = SqliteStorage::open(personal_path.to_str().unwrap()); + probe_writer(&personal_path, true); + conn.execute_batch("ROLLBACK;")?; + drop(second); + Ok(()) + }) + .unwrap(); + probe_writer(&personal_path, false); + let controlled = ControlledStore::open(&base.join("private"), identity()).unwrap(); + let state = controlled.state.lock(); + let storage = state.storage().unwrap(); + let path = storage.db_path().unwrap(); + storage + .with_controlled_conn(|conn| -> Result<()> { + conn.execute_batch("BEGIN IMMEDIATE;")?; + probe_writer(&path, true); + assert!(SqliteStorage::open(path.to_str().unwrap()).is_err()); + probe_writer(&path, true); + conn.execute_batch("ROLLBACK;")?; + Ok(()) + }) + .unwrap(); + probe_writer(&path, false); + } + #[test] + fn old_personal_history_is_scrubbed_by_checked_upgrade() { + use crate::storage::traits::StorageBackend; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().canonicalize().unwrap().join("legacy.db"); + let storage = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + let marker = "ZQXLEGACYPRIVATEHISTORY"; + let old = MemObjectBuilder::new( + MemoryTier::Episodic, + MemContent::Text((marker.to_owned() + " ").repeat(2000)), + MemSource::new("legacy"), + ) + .build(); + storage.store_memory(&old).unwrap(); + drop(storage); + let conn = rusqlite::Connection::open(&path).unwrap(); + conn.execute_batch("PRAGMA secure_delete=OFF;").unwrap(); + let _ = conn.execute_batch( + "INSERT INTO memories_fts(memories_fts,rank) VALUES('secure-delete',0);", + ); + conn.execute_batch("DROP TABLE cortex_privacy_maintenance;") + .unwrap(); + assert_eq!( + conn.query_row("PRAGMA secure_delete", [], |r| r.get::<_, i64>(0)) + .unwrap(), + 0 + ); + conn.execute( + "UPDATE memories SET content_json=?1 WHERE id=?2", + params![ + json(&MemContent::Text((marker.to_owned() + " ").repeat(2000))).unwrap(), + old.id.to_string() + ], + ) + .unwrap(); + conn.execute( + "UPDATE memories SET content_json=?1 WHERE id=?2", + params![ + json(&MemContent::Text("current replacement".into())).unwrap(), + old.id.to_string() + ], + ) + .unwrap(); + conn.execute("DELETE FROM memories WHERE id=?1", [old.id.to_string()]) + .unwrap(); + drop(conn); + let contains = |path: &Path| { + std::fs::read(path) + .unwrap_or_default() + .windows(marker.len()) + .any(|w| w.eq_ignore_ascii_case(marker.as_bytes())) + }; + assert!( + contains(&path), + "fixture must retain real legacy plaintext before upgrade" + ); + let upgraded = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + assert!(!upgraded.cleanup_pending().unwrap()); + assert!(!contains(&path)); + assert!(!contains(&PathBuf::from(format!("{}-wal", path.display())))); + } + #[test] + fn reader_probe_child() { + let Ok(path) = std::env::var("CORTEX_READER_PROBE_PATH") else { + return; + }; + let marker = std::env::var("CORTEX_READER_PROBE_MARKER").unwrap(); + let conn = + rusqlite::Connection::open_with_flags(path, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY) + .unwrap(); + conn.execute_batch("BEGIN;").unwrap(); + conn.query_row("SELECT COUNT(*) FROM memories", [], |r| r.get::<_, i64>(0)) + .unwrap(); + std::fs::write(marker, "ready").unwrap(); + let deadline = Instant::now() + Duration::from_secs(30); + while Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(20)); + } + panic!("reader probe parent did not terminate child"); + } + #[test] + fn legacy_logical_delete_tracks_blocked_physical_cleanup_and_retries() { + use crate::storage::traits::StorageBackend; + let dir = tempfile::tempdir().unwrap(); + let base = dir.path().canonicalize().unwrap(); + let path = base.join("personal.db"); + let cortex = crate::Cortex::open(path.to_str().unwrap()).unwrap(); + let memory = cortex + .ingest( + "ZQXREADBLOCKPRIVATE local private record", + "test", + None, + None, + None, + ) + .unwrap(); + let marker = base.join("reader-ready"); + struct Child(std::process::Child); + impl Drop for Child { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } + } + let child = Child( + std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "controlled::tests::reader_probe_child", + "--exact", + "--nocapture", + ]) + .env("CORTEX_READER_PROBE_PATH", &path) + .env("CORTEX_READER_PROBE_MARKER", &marker) + .stdout(std::process::Stdio::null()) + .spawn() + .unwrap(), + ); + let deadline = Instant::now() + Duration::from_secs(15); + while !marker.exists() { + assert!(Instant::now() < deadline); + std::thread::sleep(Duration::from_millis(10)); + } + cortex.delete_memory(memory.id).unwrap(); + assert!(cortex.storage.get_memory(memory.id).unwrap().is_none()); + assert!(cortex.cleanup_pending().unwrap()); + assert!(cortex.retry_cleanup().unwrap()); + drop(child); + assert!(!cortex.retry_cleanup().unwrap()); + drop(cortex); + let reopened = crate::Cortex::open(path.to_str().unwrap()).unwrap(); + assert!(!reopened.cleanup_pending().unwrap()); + } + #[test] + fn interrupted_privacy_upgrade_never_marks_completion_and_can_reopen() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().canonicalize().unwrap().join("upgrade.db"); + let storage = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + storage.with_controlled_conn(|conn| -> Result<()> { + conn.execute_batch("UPDATE cortex_privacy_maintenance SET erasure_version=0,cleanup_pending=1; + CREATE TRIGGER fail_erasure_completion BEFORE UPDATE ON cortex_privacy_maintenance + WHEN NEW.erasure_version=1 BEGIN SELECT RAISE(ABORT,'injected completion failure'); END;")?; Ok(()) + }).unwrap(); + drop(storage); + assert!(SqliteStorage::open(path.to_str().unwrap()).is_err()); + let conn = rusqlite::Connection::open(&path).unwrap(); + let (version, pending): (i64, bool) = conn + .query_row( + "SELECT erasure_version,cleanup_pending FROM cortex_privacy_maintenance WHERE id=1", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!(version, 0); + assert!(pending); + conn.execute_batch("DROP TRIGGER fail_erasure_completion;") + .unwrap(); + drop(conn); + let reopened = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + assert!(!reopened.cleanup_pending().unwrap()); + } } diff --git a/cortex-core/src/lib.rs b/cortex-core/src/lib.rs index 37bbb1a..d6073cb 100644 --- a/cortex-core/src/lib.rs +++ b/cortex-core/src/lib.rs @@ -1569,7 +1569,8 @@ impl Cortex { &mut self.working } - /// Delete a memory permanently. + /// Logically delete a memory and record retryable physical cleanup. + /// Check cleanup_pending()/retry_cleanup() before claiming local erasure. pub fn delete_memory(&self, id: Uuid) -> Result<(), CortexError> { self.storage.delete_memory(id)?; self.index.remove(&id); @@ -1604,6 +1605,12 @@ impl Cortex { self.storage.flush_durable() } + /// Whether physical SQLite cleanup remains after logical deletion/upgrade. + pub fn cleanup_pending(&self) -> Result { self.storage.cleanup_pending() } + + /// Retry physical cleanup; true means a reader or maintenance work still blocks it. + pub fn retry_cleanup(&self) -> Result { self.storage.retry_cleanup() } + /// The database file, or `None` for an in-memory instance. pub fn db_path(&self) -> Option { self.storage.db_path() diff --git a/cortex-core/src/storage/sqlite.rs b/cortex-core/src/storage/sqlite.rs index 60e2030..f90b7bc 100644 --- a/cortex-core/src/storage/sqlite.rs +++ b/cortex-core/src/storage/sqlite.rs @@ -10,6 +10,11 @@ use rusqlite::{params, Connection, OptionalExtension}; use std::sync::Arc; use uuid::Uuid; +// SQLite/native state may have copied mutexes after fork. Only exec/spawn may +// initialize a fresh process once any store has initialized this library. +#[cfg(unix)] +static SQLITE_PROCESS: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0); + /// Number of read-only connections in the pool. const READ_POOL_SIZE: usize = 4; @@ -41,6 +46,16 @@ enum DbKey<'a> { } impl SqliteStorage { + #[cfg(not(unix))] + pub(crate) fn guard_open_process() -> bool { true } + #[cfg(unix)] + pub(crate) fn guard_open_process() -> bool { + use std::sync::atomic::Ordering; + let pid = std::process::id(); + match SQLITE_PROCESS.compare_exchange(0, pid, Ordering::AcqRel, Ordering::Acquire) { + Ok(_) => true, Err(owner) => owner == pid, + } + } fn apply_read_pragmas(conn: &Connection) -> Result<(), CortexError> { conn.execute_batch( "PRAGMA cache_size=-64000; @@ -145,65 +160,82 @@ impl SqliteStorage { /// must be owner-only. Creates a new file as 0600 — SQLite gives `-wal`/`-shm` the /// main file's mode — and tightens files left group/world-readable by older releases. #[cfg(unix)] - fn restrict_db_file_mode(path: &str) -> Result<(), CortexError> { - use std::os::unix::fs::{MetadataExt, OpenOptionsExt, PermissionsExt}; - if path == ":memory:" { - return Ok(()); + fn trusted_directory_chain(path: &std::path::Path, allow_trusted_symlinks: bool) -> Result<(), CortexError> { + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + let mut prefix = std::path::PathBuf::new(); + for component in path.components() { + prefix.push(component); + let link = std::fs::symlink_metadata(&prefix).map_err(|e| CortexError::Storage(e.to_string()))?; + let uid = unsafe { libc::geteuid() }; + if link.uid() != 0 && link.uid() != uid { + return Err(CortexError::InvalidInput("untrusted database ancestor".into())); + } + let meta = if link.file_type().is_symlink() { + if !allow_trusted_symlinks { return Err(CortexError::InvalidInput("symlink database ancestor".into())); } + std::fs::metadata(&prefix).map_err(|e| CortexError::Storage(e.to_string()))? + } else { link }; + if !meta.is_dir() || (meta.uid() != 0 && meta.uid() != uid) + || (meta.permissions().mode() & 0o022 != 0 && meta.permissions().mode() & 0o1000 == 0) { + return Err(CortexError::InvalidInput("untrusted database ancestor".into())); + } } - // URI filenames have aliases and query parameters; accepting them here - // would let SQLite open a different file than the descriptor we secure. + Ok(()) + } + #[cfg(unix)] + fn restrict_db_file_mode(path: &str) -> Result { + use std::os::unix::fs::{MetadataExt, PermissionsExt}; + if path == ":memory:" { return Ok(path.to_owned()); } if path.is_empty() || path.starts_with("file:") { - return Err(CortexError::InvalidInput( - "use a filesystem path or :memory:".into(), - )); - } - let main = std::fs::OpenOptions::new() - .write(true) - .create(true) - .truncate(false) - .mode(0o600) - .custom_flags(libc::O_NOFOLLOW) - .open(path) - .map_err(|e| CortexError::Storage(e.to_string()))?; - for suffix in ["", "-wal", "-shm", "-journal"] { - let file = if suffix.is_empty() { - main.try_clone() - } else { - match std::fs::OpenOptions::new() - .read(true) - .write(true) - .custom_flags(libc::O_NOFOLLOW) - .open(format!("{path}{suffix}")) - { - Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue, - other => other, + return Err(CortexError::InvalidInput("use a filesystem path or :memory:".into())); + } + let absolute = std::env::current_dir().map_err(|e| CortexError::Storage(e.to_string()))?.join(path); + let parent = absolute.parent().ok_or_else(|| CortexError::InvalidInput("database parent".into()))?; + // Check aliases before canonicalization, then the resolved ancestry too. + // Only this user/root can change these directory entries. Sticky parents + // protect already-owned files; atomic no-clobber publication handles new names. + Self::trusted_directory_chain(parent, true)?; + let parent = parent.canonicalize().map_err(|e| CortexError::Storage(e.to_string()))?; + Self::trusted_directory_chain(&parent, false)?; + // Sticky shared ancestors are allowed, but the immediate directory must + // prevent other UIDs from racing creation of predictable WAL/SHM names. + if std::fs::metadata(&parent).map_err(|e| CortexError::Storage(e.to_string()))?.permissions().mode() & 0o022 != 0 { + return Err(CortexError::InvalidInput("database parent must not be shared-writable".into())); + } + let resolved = parent.join(absolute.file_name().ok_or_else(|| CortexError::InvalidInput("database filename".into()))?); + if matches!(std::fs::symlink_metadata(&resolved), Err(e) if e.kind() == std::io::ErrorKind::NotFound) { + // Close the new 0600 temporary descriptor BEFORE publishing its name. + // Never open/close existing SQLite artifacts outside its VFS: closing + // an unrelated -shm FD releases the process's active POSIX WAL locks. + let temporary = tempfile::Builder::new().prefix(".cortex-private-").tempfile_in(&parent) + .map_err(|e| CortexError::Storage(e.to_string()))?.into_temp_path(); + if let Err(error) = temporary.persist_noclobber(&resolved) { + if error.error.kind() != std::io::ErrorKind::AlreadyExists { + return Err(CortexError::Storage(error.error.to_string())); } } - .map_err(|e| CortexError::Storage(e.to_string()))?; - let meta = file - .metadata() - .map_err(|e| CortexError::Storage(e.to_string()))?; - if !meta.is_file() || meta.uid() != unsafe { libc::geteuid() } || meta.nlink() != 1 { - return Err(CortexError::Storage( - "database must be an owner-owned regular file".into(), - )); + } + for suffix in ["", "-wal", "-shm", "-journal"] { + let artifact = if suffix.is_empty() { resolved.clone() } else { + let mut name = resolved.as_os_str().to_os_string(); name.push(suffix); std::path::PathBuf::from(name) + }; + let before = match std::fs::symlink_metadata(&artifact) { + Err(e) if !suffix.is_empty() && e.kind() == std::io::ErrorKind::NotFound => continue, + other => other.map_err(|e| CortexError::Storage(e.to_string()))?, + }; + if !before.is_file() || before.uid() != unsafe { libc::geteuid() } || before.nlink() != 1 { + return Err(CortexError::Storage("database must be an owner-owned regular file".into())); } - file.set_permissions(std::fs::Permissions::from_mode(0o600)) + // Safe under the verified, non-replaceable directory chain and the + // file's verified owner. No unrelated descriptors are opened/closed. + std::fs::set_permissions(&artifact, std::fs::Permissions::from_mode(0o600)) .map_err(|e| CortexError::Storage(e.to_string()))?; - if file - .metadata() - .map_err(|e| CortexError::Storage(e.to_string()))? - .permissions() - .mode() - & 0o777 - != 0o600 - { - return Err(CortexError::Storage( - "database permissions are not private".into(), - )); + let after = std::fs::symlink_metadata(&artifact).map_err(|e| CortexError::Storage(e.to_string()))?; + if !after.is_file() || before.dev() != after.dev() || before.ino() != after.ino() + || before.uid() != after.uid() || after.nlink() != 1 || after.permissions().mode() & 0o777 != 0o600 { + return Err(CortexError::Storage("database file changed or is not private".into())); } } - Ok(()) + resolved.to_str().map(str::to_owned).ok_or_else(|| CortexError::InvalidInput("database path encoding".into())) } fn open_keyed(path: &str, key: Option, readers: usize) -> Result { @@ -221,14 +253,14 @@ impl SqliteStorage { DROP TRIGGER IF EXISTS memories_fts_ad; DROP TRIGGER IF EXISTS memories_fts_au; CREATE TRIGGER memories_fts_ai AFTER INSERT ON memories BEGIN - INSERT INTO memories_fts(rowid,id,content) VALUES(NEW.rowid,NEW.id,NEW.content_json); END; + INSERT INTO memories_fts(rowid,id,content) VALUES(NEW.rowid,NEW.id,json_extract(NEW.content_json,'$.Text')); END; CREATE TRIGGER memories_fts_ad AFTER DELETE ON memories BEGIN DELETE FROM memories_fts WHERE rowid=OLD.rowid; END; CREATE TRIGGER memories_fts_au AFTER UPDATE OF content_json ON memories BEGIN DELETE FROM memories_fts WHERE rowid=OLD.rowid; - INSERT INTO memories_fts(rowid,id,content) VALUES(NEW.rowid,NEW.id,NEW.content_json); END; + INSERT INTO memories_fts(rowid,id,content) VALUES(NEW.rowid,NEW.id,json_extract(NEW.content_json,'$.Text')); END; DELETE FROM memories_fts; - INSERT INTO memories_fts(rowid,id,content) SELECT rowid,id,content_json FROM memories; + INSERT INTO memories_fts(rowid,id,content) SELECT rowid,id,json_extract(content_json,'$.Text') FROM memories; COMMIT;") .map_err(|e| CortexError::Storage(e.to_string())) })?; @@ -241,8 +273,13 @@ impl SqliteStorage { readers: usize, controlled: bool, ) -> Result { + if !Self::guard_open_process() { + return Err(CortexError::InvalidInput("use spawn/exec after fork".into())); + } + #[cfg(unix)] + let private_path = Self::restrict_db_file_mode(path)?; #[cfg(unix)] - Self::restrict_db_file_mode(path)?; + let path = private_path.as_str(); let conn = Connection::open(path).map_err(|e| CortexError::Storage(e.to_string()))?; Self::apply_key(&conn, key.as_ref())?; @@ -300,24 +337,11 @@ impl SqliteStorage { ) .map_err(|e| CortexError::Storage(e.to_string()))?; - // Open read-only connections for concurrent reads - let mut read_pool = Vec::with_capacity(readers); - for _ in 0..readers { - let reader = Connection::open_with_flags( - path, - rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY - | rusqlite::OpenFlags::SQLITE_OPEN_NO_MUTEX - | rusqlite::OpenFlags::SQLITE_OPEN_URI, - ) + let had_memories: bool = conn.query_row("SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name='memories')", [], |r| r.get(0)) .map_err(|e| CortexError::Storage(e.to_string()))?; - Self::apply_key(&reader, key.as_ref())?; - Self::apply_read_pragmas(&reader)?; - read_pool.push(Mutex::new(reader)); - } - - let storage = Self { + let mut storage = Self { write_conn: Mutex::new(conn), - read_pool, + read_pool: Vec::new(), mem_cache: Mutex::new(lru::LruCache::new( std::num::NonZeroUsize::new(MEM_CACHE_SIZE).unwrap(), )), @@ -326,19 +350,91 @@ impl SqliteStorage { )), }; storage.init()?; - // Deletes are permanent: FTS5 otherwise keeps deleted terms in index segments + // Enable scrubbing of subsequent FTS changes; historical content is handled + // by the checked erasure migration below. FTS5 otherwise keeps deleted terms // until a merge (secure_delete only scrubs freed b-tree pages). Needs SQLite // 3.42+; on older builds scrub_after_delete falls back to an index rebuild. let _ = storage.write_conn.lock().execute_batch( "INSERT INTO memories_fts(memories_fts, rank) VALUES('secure-delete', 1);", ); + storage.initialize_privacy_maintenance(had_memories)?; + // Open read-only connections for concurrent reads + let mut read_pool = Vec::with_capacity(readers); + for _ in 0..readers { + let reader = Connection::open_with_flags( + path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY + | rusqlite::OpenFlags::SQLITE_OPEN_NO_MUTEX + | rusqlite::OpenFlags::SQLITE_OPEN_URI, + ) + .map_err(|e| CortexError::Storage(e.to_string()))?; + Self::apply_key(&reader, key.as_ref())?; + Self::apply_read_pragmas(&reader)?; + read_pool.push(Mutex::new(reader)); + } + + storage.read_pool = read_pool; Ok(storage) } + fn initialize_privacy_maintenance(&self, had_memories: bool) -> Result<(), CortexError> { + let conn = self.write_conn.lock(); + conn.execute_batch("CREATE TABLE IF NOT EXISTS cortex_privacy_maintenance(id INTEGER PRIMARY KEY CHECK(id=1), erasure_version INTEGER NOT NULL, cleanup_pending INTEGER NOT NULL);") + .map_err(|e| CortexError::Storage(e.to_string()))?; + conn.execute("INSERT OR IGNORE INTO cortex_privacy_maintenance(id,erasure_version,cleanup_pending) VALUES(1,?1,?2)", params![if had_memories { 0 } else { 1 }, had_memories]) + .map_err(|e| CortexError::Storage(e.to_string()))?; + // Version zero is itself pending even if a previous process crashed. + let pending = Self::retry_personal_cleanup_on_connection(&conn)?; + let version: i64 = conn.query_row("SELECT erasure_version FROM cortex_privacy_maintenance WHERE id=1", [], |r| r.get(0)).map_err(|e| CortexError::Storage(e.to_string()))?; + if pending && version == 0 { return Err(CortexError::Storage("privacy upgrade pending; quiesce readers and reopen".into())); } + Ok(()) + } + /// Logical deletion and physical file cleanup are separate outcomes. + /// True persists across restart until a successful bounded cleanup retry. + pub fn cleanup_pending(&self) -> Result { + let conn = self.write_conn.lock(); + conn.query_row("SELECT cleanup_pending OR erasure_version=0 FROM cortex_privacy_maintenance WHERE id=1", [], |r| r.get(0)) + .map_err(|e| CortexError::Storage(e.to_string())) + } + /// Retry pending cleanup. Returns true while work remains. A reader can block + /// completion; close it and retry. No files/receipts are deleted to hide failure. + pub fn retry_cleanup(&self) -> Result { + Self::retry_personal_cleanup_on_connection(&self.write_conn.lock()) + } + fn retry_personal_cleanup_on_connection(conn: &Connection) -> Result { + let (version, pending): (i64,bool) = conn.query_row("SELECT erasure_version,cleanup_pending FROM cortex_privacy_maintenance WHERE id=1", [], |r| Ok((r.get(0)?,r.get(1)?))) + .map_err(|e| CortexError::Storage(e.to_string()))?; + if version != 0 && !pending { return Ok(false); } + if version == 0 { + // Hold SQLite's exclusive locking mode across the multi-step migration, + // including VACUUM (which cannot run in an explicit transaction). + // This excludes old binaries/writers between rebuild and checkpoint. + let migrated = (|| -> rusqlite::Result { + conn.execute_batch("PRAGMA locking_mode=EXCLUSIVE; BEGIN EXCLUSIVE; + INSERT INTO memories_fts(memories_fts) VALUES('rebuild'); COMMIT; + VACUUM;")?; + if !Self::scrub_after_delete(conn) { return Ok(false); } + conn.execute("UPDATE cortex_privacy_maintenance SET erasure_version=1,cleanup_pending=0 WHERE id=1", [])?; + Ok(true) + })(); + // Restore normal sharing even after an interrupted migration attempt. + if !conn.is_autocommit() { let _ = conn.execute_batch("ROLLBACK;"); } + conn.execute_batch("PRAGMA locking_mode=NORMAL; PRAGMA schema_version;") + .map_err(|e| CortexError::Storage(e.to_string()))?; + return migrated.map(|done| !done).map_err(|e| CortexError::Storage(e.to_string())); + } + if Self::scrub_after_delete(conn) { + conn.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=0 WHERE id=1", []) + .map_err(|e| CortexError::Storage(e.to_string()))?; + return Ok(false); + } + Ok(true) + } + /// Make a committed delete permanent on disk: purge the deleted terms from the FTS /// index, then fold the WAL back into the (zero-scrubbed) database and truncate it so - /// the plaintext does not linger in old WAL frames. Best effort: a reader holding a - /// snapshot can block the checkpoint; the next delete or autocheckpoint retries. + /// the plaintext does not linger in old WAL frames. A reader holding a snapshot + /// can block completion; callers persist pending state and expose explicit retry. pub(crate) fn scrub_after_delete(conn: &Connection) -> bool { // A snapshot reader can block truncation indefinitely. Cleanup is bounded; // controlled callers persist and expose a retryable pending flag. @@ -372,6 +468,7 @@ impl SqliteStorage { } pub fn open_in_memory() -> Result { + if !Self::guard_open_process() { return Err(CortexError::InvalidInput("use spawn/exec after fork".into())); } let conn = Connection::open_in_memory().map_err(|e| CortexError::Storage(e.to_string()))?; conn.execute_batch("PRAGMA foreign_keys=ON;") .map_err(|e| CortexError::Storage(e.to_string()))?; @@ -387,6 +484,7 @@ impl SqliteStorage { )), }; storage.init()?; + storage.initialize_privacy_maintenance(false)?; Ok(storage) } @@ -986,17 +1084,16 @@ impl StorageBackend for SqliteStorage { fn delete_memory(&self, id: Uuid) -> Result<(), CortexError> { let conn = self.write_conn.lock(); - let id_str = id.to_string(); - let mut stmt = conn - .prepare_cached("DELETE FROM memories WHERE id = ?1") - .map_err(|e| CortexError::Storage(e.to_string()))?; - stmt.execute(params![id_str]) - .map_err(|e| CortexError::Storage(e.to_string()))?; - drop(stmt); - - // FTS5 synced via trigger (memories_fts_ad) - Self::scrub_after_delete(&conn); + let tx = conn.unchecked_transaction().map_err(|e| CortexError::Storage(e.to_string()))?; + tx.execute("DELETE FROM memories WHERE id=?1", [id.to_string()]).map_err(|e| CortexError::Storage(e.to_string()))?; + tx.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=1 WHERE id=1", []).map_err(|e| CortexError::Storage(e.to_string()))?; + tx.commit().map_err(|e| CortexError::Storage(e.to_string()))?; self.cache_evict(&id); + // Logical deletion is committed. Preserve its success and expose physical + // failure durably through cleanup_pending()/retry_cleanup(). + if Self::retry_personal_cleanup_on_connection(&conn).is_err() { + tracing::warn!("SQLite physical cleanup remains pending after logical deletion"); + } Ok(()) } @@ -1881,11 +1978,10 @@ impl StorageBackend for SqliteStorage { } // FTS5 synced via trigger (memories_fts_ad) + conn.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=1 WHERE id=1", []).map_err(|e| CortexError::Storage(e.to_string()))?; conn.execute_batch("COMMIT;") .map_err(|e| CortexError::Storage(e.to_string()))?; - if count > 0 { - Self::scrub_after_delete(&conn); - } + if count > 0 && Self::retry_personal_cleanup_on_connection(&conn).is_err() { tracing::warn!("SQLite physical cleanup remains pending after logical deletion"); } // Evict deleted entries from cache { let mut cache = self.mem_cache.lock(); diff --git a/cortex-python/src/lib.rs b/cortex-python/src/lib.rs index c17db04..e2e9138 100644 --- a/cortex-python/src/lib.rs +++ b/cortex-python/src/lib.rs @@ -17,11 +17,21 @@ struct PyCortex { impl PyCortex { /// Open or create a Cortex database. #[new] - fn new(db_path: &str) -> PyResult { - let inner = Cortex::open(db_path).map_err(to_py_err)?; + fn new(py: Python<'_>, db_path: &str) -> PyResult { + let inner = py + .allow_threads(|| Cortex::open(db_path)) + .map_err(to_py_err)?; Ok(Self { inner }) } + fn cleanup_pending(&self, py: Python<'_>) -> PyResult { + py.allow_threads(|| self.inner.cleanup_pending()) + .map_err(to_py_err) + } + fn retry_cleanup(&self, py: Python<'_>) -> PyResult { + py.allow_threads(|| self.inner.retry_cleanup()) + .map_err(to_py_err) + } /// Ingest a new memory from any channel. #[pyo3(signature = (text, channel, user_id=None, salience=None, embedding=None))] fn ingest( @@ -202,15 +212,15 @@ fn bounded_value( )); } if value.is_exact_instance_of::() { - return Ok(serde_json::Value::from( - value.extract::().map_err(|_| invalid())?, - )); + let number = value.extract::().map_err(|_| invalid())?; + debit(budget, number.to_string().len())?; + return Ok(serde_json::Value::from(number)); } if value.is_exact_instance_of::() { let number = value.extract::().map_err(|_| invalid())?; - return serde_json::Number::from_f64(number) - .map(serde_json::Value::Number) - .ok_or_else(invalid); + let number = serde_json::Number::from_f64(number).ok_or_else(invalid)?; + debit(budget, number.to_string().len())?; + return Ok(serde_json::Value::Number(number)); } if value.is_exact_instance_of::() { let string = value.downcast::().map_err(|_| invalid())?; @@ -312,6 +322,28 @@ struct PyControlledStore { creator_pid: u32, inner: parking_lot::Mutex>>, } +impl Drop for PyControlledStore { + fn drop(&mut self) { + let handle = self.inner.get_mut().take(); + if self.creator_pid != std::process::id() { + // Never touch copied SQLite/mutex state during child finalization. + std::mem::forget(handle); + return; + } + if let Some(handle) = handle { + // PyO3 invokes pyclass destruction with an attached Python thread; + // with_gil reuses that attachment even during interpreter teardown. + // Native destruction performs no Python work while detached. + Python::with_gil(|py| { + py.allow_threads(move || { + let _ = handle.close(); + drop(handle); + }) + }); + } + } +} + impl PyControlledStore { fn check_process(&self, py: Python<'_>) -> PyResult<()> { if self.creator_pid != std::process::id() { diff --git a/cortex-python/tests/test_controlled_store.py b/cortex-python/tests/test_controlled_store.py index 61eb5f3..31b0af7 100644 --- a/cortex-python/tests/test_controlled_store.py +++ b/cortex-python/tests/test_controlled_store.py @@ -718,6 +718,213 @@ def test_unauthorized_term_load_cannot_change_authorized_ranking(self): self.assertEqual(result["degraded"], baseline["degraded"]) self.assertEqual(result["truncated"], baseline["truncated"]) + def test_personal_fifo_is_rejected_without_blocking(self): + fifo = Path(self.temp.name).resolve() / "not-a-database.db" + os.mkfifo(fifo, 0o600) + script = r''' +import sys +from cortex_ai_memory import Cortex +try: + instance = Cortex(sys.argv[1]) +except RuntimeError: + print('refused', flush=True) +else: + raise AssertionError('personal Cortex accepted FIFO') +''' + begin = time.monotonic() + completed = subprocess.run([sys.executable, "-I", "-c", script, str(fifo)], + capture_output=True, text=True, timeout=4.0) + self.assertLess(time.monotonic() - begin, 5.0) + self.assertEqual(completed.returncode, 0, completed.stderr) + self.assertEqual(completed.stdout.strip(), "refused") + self.assertTrue(fifo.is_fifo()) + + def test_personal_owned_file_under_unsafe_ancestor_is_refused(self): + ancestor = Path(self.temp.name).resolve() / "personal-unsafe-parent" + ancestor.mkdir(mode=0o700) + database = ancestor / "personal.db" + database.write_bytes(b"") + database.chmod(0o600) + self.assertEqual(database.stat().st_uid, os.geteuid()) + before = (database.stat().st_mode, database.read_bytes()) + ancestor.chmod(0o777) + try: + with self.assertRaises(RuntimeError): + instance = memory.Cortex(str(database)) + del instance + gc.collect() + self.assertEqual((database.stat().st_mode, database.read_bytes()), before) + self.assertEqual({path.name for path in ancestor.iterdir()}, {"personal.db"}) + finally: + ancestor.chmod(0o700) + + def test_aggregate_numeric_input_budget_is_atomic(self): + store = self.ready() + vector = [1.234567890123456e-100] * 4096 + oversized = [upsert(projection("numeric-budget-%d" % i, embedding=vector)) + for i in range(100)] + self.assertGreater(len(json.dumps(oversized).encode("utf-8")), 8*1024*1024, + "fixture must exceed aggregate JSON budget") + self.error("invalid_input", store.apply_batch, oversized, 101) + self.assertEqual(store.status()["position"], 100) + self.assertEqual(self.hits(store), []) + receipt = store.upsert(projection("legitimate-vector", embedding=vector), 101) + self.assertTrue(receipt["durable"]) + self.assertEqual(self.hits(store)[0]["memory_id"], "legitimate-vector") + + def test_sticky_parent_precreated_final_symlink_cannot_escape(self): + base = Path(self.temp.name).resolve() + sticky = base / "shared-sticky" + outside = base / "outside-root" + sticky.mkdir(mode=0o700) + sticky.chmod(0o1777) + outside.mkdir(mode=0o700) + sentinel = outside / "sentinel" + sentinel.write_bytes(b"unchanged external fixture") + root = sticky / "controlled-root" + root.symlink_to(outside, target_is_directory=True) + before = (outside.stat().st_mode, sentinel.stat().st_mode, sentinel.read_bytes()) + try: + self.error("invalid_input", self.open, root=str(root)) + self.assertTrue(root.is_symlink()) + self.assertEqual({path.name for path in outside.iterdir()}, {"sentinel"}) + self.assertEqual((outside.stat().st_mode, sentinel.stat().st_mode, sentinel.read_bytes()), before) + finally: + sticky.chmod(0o700) + + def test_fork_child_liveness_does_not_retain_parent_replica_lock(self): + script = r''' +import json, os, signal, subprocess, sys +from cortex_ai_memory import ControlledStore, ControlledError +root = sys.argv[1] +s = ControlledStore(root, 'merchant', 'merchant-acceptance', 'bot-one', 'replica-one') +s.finish_snapshot(100) +r, w = os.pipe() +release_r, release_w = os.pipe() +pid = os.fork() +if pid == 0: + signal.alarm(10) + os.close(r); os.close(release_w) + codes = [] + for call in (s.status, lambda: ControlledStore(root, 'merchant', 'merchant-acceptance', 'bot-one', 'fork-new')): + try: + call(); codes.append('unexpected-success') + except ControlledError as error: + codes.append(error.code) + os.write(w, (json.dumps(codes)+'\n').encode()) + os.close(w) + os.read(release_r, 1) + os._exit(0) +os.close(w); os.close(release_r) +try: + codes = json.loads(os.fdopen(r).readline()) + assert codes == ['conflict', 'conflict'], codes + s.close() + reopened = ControlledStore(root, 'merchant', 'merchant-acceptance', 'bot-one', 'replica-one') + assert os.waitpid(pid, os.WNOHANG) == (0, 0), 'child must remain alive during reopen' + reopened.close() + fresh = "from cortex_ai_memory import ControlledStore; import sys; s=ControlledStore(sys.argv[1], 'merchant', 'merchant-acceptance', 'bot-one', 'spawn-new'); s.close(); print('fresh-opened')" + spawned = subprocess.run([sys.executable, '-I', '-c', fresh, root], capture_output=True, text=True, timeout=5) + assert spawned.returncode == 0, spawned.stderr + assert spawned.stdout.strip() == 'fresh-opened' + print(json.dumps({'inherited': codes, 'parent_reopened': True, 'fresh_spawn': True}), flush=True) +finally: + os.write(release_w, b'x'); os.close(release_w) + os.waitpid(pid, 0) +''' + completed = subprocess.run([sys.executable, "-I", "-c", script, self.root], + capture_output=True, text=True, timeout=15) + self.assertEqual(completed.returncode, 0, completed.stderr) + self.assertEqual(json.loads(completed.stdout), + {"inherited": ["conflict", "conflict"], "parent_reopened": True, "fresh_spawn": True}) + + def test_last_reference_teardown_heartbeat_and_normal_shutdown(self): + script = r''' +import gc, json, pathlib, subprocess, sys, threading, time +from cortex_ai_memory import ControlledStore +root, mode = sys.argv[1:] +s = ControlledStore(root, 'merchant', 'merchant-acceptance', 'bot-one', 'replica-one') +sc = {'kind':'owner','id':'merchant-acceptance'} +items = [{'memory_id':'drop-%d'%i,'revision':1,'scope':sc,'kind':'fact', + 'text':'orchid '+('x'*11990),'source_refs':[{'kind':'input','provider':'test','id':'drop'}]} for i in range(100)] +s.seed(items); s.finish_snapshot(100); s.reconcile(s.status()['generation'],100) +if mode == 'shutdown': + print('live-store-at-shutdown', flush=True) +else: + db = next(p for p in pathlib.Path(root).rglob('*.db') if p.read_bytes()[:16] == b'SQLite format 3\x00') + reader_code = "import sqlite3,sys; c=sqlite3.connect(sys.argv[1]+'?mode=ro',uri=True); c.execute('BEGIN'); c.execute('SELECT count(*) FROM sqlite_master').fetchone(); print('ready',flush=True); sys.stdin.readline(); c.close()" + reader = subprocess.Popen([sys.executable,'-I','-c',reader_code,db.as_uri()],stdin=subprocess.PIPE,stdout=subprocess.PIPE,text=True) + try: + assert reader.stdout.readline().strip() == 'ready' + for item in items: + item['revision'] = 2 + item['text'] = 'orchid replacement '+('y'*11980) + s.apply_batch([{'op':'upsert','projection':p} for p in items],101) + assert s.status()['cleanup_pending'] + stop = threading.Event(); started = threading.Event(); ticks = [] + def beat(): + started.set() + while not stop.is_set(): + ticks.append(time.monotonic()); time.sleep(0.0002) + thread = threading.Thread(target=beat); thread.start(); started.wait() + previous = sys.getswitchinterval() + try: + sys.setswitchinterval(10) + begin = time.monotonic(); del s; gc.collect(); end = time.monotonic() + finally: + sys.setswitchinterval(previous); stop.set(); thread.join(timeout=2) + assert end-begin < 5, ('unbounded destructor',end-begin) + progress = sum(begin < tick < end for tick in ticks) + if end-begin >= 0.005: + assert progress > 0, ('GIL blocked through substantial teardown',end-begin) + reopened = ControlledStore(root,'merchant','merchant-acceptance','bot-one','replica-one') + reopened.close() + print(json.dumps({'elapsed':end-begin,'heartbeat_ticks':progress,'reopened':True}),flush=True) + finally: + reader.communicate(input='release\n',timeout=5) +''' + for mode in ("drop", "shutdown"): + with self.subTest(mode=mode): + root = str((Path(self.temp.name) / mode).resolve()) + completed = subprocess.run([sys.executable, "-I", "-c", script, root, mode], + capture_output=True, text=True, timeout=15) + self.assertEqual(completed.returncode, 0, completed.stderr) + if mode == "drop": + self.assertTrue(json.loads(completed.stdout)["reopened"]) + print("last-reference teardown metrics: " + completed.stdout.strip(), flush=True) + else: + self.assertEqual(completed.stdout.strip(), "live-store-at-shutdown") + reopened = self.open(root=root) + self.reconcile(reopened) + self.assertEqual(len(self.hits(reopened, limit=100)), 100) + reopened.close() + + def test_lexical_matching_uses_decoded_text_not_serialization(self): + scores = [] + for index, text in enumerate(("alpha orchid", "alpha\norchid", "alpha\torchid", "alpha\\orchid")): + store = self.ready([projection(text=text, embedding=[1.0, 0.0])], replica="decoded-%d" % index) + def score(query): + hits = store.search(query, [SCOPE], limit=1, max_candidates=1, embedding=[1.0, 0.0])["hits"] + self.assertEqual(len(hits), 1) + return hits[0]["score"] + actual = score("orchid") + unrelated = score("nonexistentlexicalword") + self.assertGreater(actual, unrelated) + self.assertEqual(score("Text"), unrelated, + "serialized Text label must not boost unrelated text") + self.assertEqual(score("norchid"), unrelated, + "newline escape must not create phantom word") + self.assertEqual(score("torchid"), unrelated, + "tab escape must not create phantom word") + scores.append(actual) + for value in scores[1:]: + self.assertAlmostEqual(value, scores[0], places=6) + actual_label = self.ready([projection(text="Text orchid", embedding=[1.0, 0.0])], replica="actual-label") + match = actual_label.search("Text", [SCOPE], limit=1, max_candidates=1, embedding=[1.0, 0.0])["hits"][0] + unrelated = actual_label.search("nonexistentlexicalword", [SCOPE], limit=1, max_candidates=1, + embedding=[1.0, 0.0])["hits"][0] + self.assertGreater(match["score"], unrelated["score"]) + def test_restored_local_transaction_loss_rejects_remote_cursor(self): store = self.ready([projection()]) state = store.status() diff --git a/docs/controlled-memory-api.md b/docs/controlled-memory-api.md index abb3979..71a75de 100644 --- a/docs/controlled-memory-api.md +++ b/docs/controlled-memory-api.md @@ -5,9 +5,9 @@ Implementation contract for Cortex-side P0. Python distribution version: **2.5.0 ## Install the reviewed build The existing PyPI 2.4.0 release does not contain this API. Use the exact 2.5.0 wheel -from the reviewed PR's `controlled-wheel-*` CI artifact for your platform, or build -from the reviewed source commit. The CI covers Python 3.12 on Linux x86_64 and macOS -arm64. The Python release profile retains native symbols to avoid the reproduced macOS stripped-LINKEDIT load failure. Wheel files are platform/interpreter specific; do not install an incompatible +from the reviewed local-CI artifact manifest for your platform, or build +from the reviewed source commit. The local verification matrix targets Python 3.12 on Linux x86_64 and macOS +arm64. GitHub Actions CI is disabled under the local-only CI policy. The Python release profile retains native symbols to avoid the reproduced macOS stripped-LINKEDIT load failure. Wheel files are platform/interpreter specific; do not install an incompatible artifact or assume other targets have been verified. ```sh @@ -24,7 +24,7 @@ SHA and artifact checksum with the host's existing dependency manifest. ## Store and lifecycle -`ControlledStore(root, owner_type, owner_id, bot_id, replica_id)` opens a dedicated owner/bot/replica database under an absolute, private root. Paths inside that root are generated from the identity. The root must not be a symlink or contain `..`; resolve platform temporary-directory aliases before passing it. Existing ancestors must be owned by the service user or root and must not be group/world-writable unless sticky, preventing an untrusted local account from replacing the validated root. The root itself is always owner-only. One process/handle may own a replica at a time. Handles inherited through `fork` raise `conflict` before touching copied locks/SQLite; create handles after forking, with a separate replica for each worker. Different replicas have separate generation IDs and cursors. A personal database is never imported. Current legacy open/export paths refuse controlled databases. Older binaries must not be given these files. +`ControlledStore(root, owner_type, owner_id, bot_id, replica_id)` opens a dedicated owner/bot/replica database under an absolute, private root. Paths inside that root are generated from the identity. The root must not be a symlink or contain `..`; resolve platform temporary-directory aliases before passing it. Existing ancestors must be owned by the service user or root and must not be group/world-writable unless sticky, preventing an untrusted local account from replacing the validated root. New root components are created individually and checked without following symlinks, including a name precreated under a shared sticky parent. The root itself is always owner-only. One process/handle may own a replica at a time. Handles inherited through `fork` raise `conflict` before touching copied locks/SQLite. After any Cortex store initialization, children must exec/spawn before opening stores; a pure-fork child cannot initialize another store. Use a separate replica for each spawned worker. Process-owned locks allow the parent to close and reopen its replica while an unused fork child remains alive. Different replicas have separate generation IDs and cursors. A personal database is never imported. Current legacy open/export paths refuse controlled databases. Older binaries must not be given these files. - `status()` returns `{generation, position, sealed, ready, cleanup_pending}`. Every reopen starts not ready. - `seed(projections)` atomically loads a batch of current authoritative memories while the snapshot is unsealed. Retrying identical revisions is safe. No seed after snapshot completion. @@ -32,6 +32,7 @@ SHA and artifact checksum with the host's existing dependency manifest. - `reconcile(generation, position)` enables queries only after both exactly match local durable status and snapshot is sealed. An unsealed snapshot raises `not_ready`; a generation/position mismatch raises `conflict` and queries remain disabled. The caller must validate the source snapshot, reconcile retained changes or rebuild a new replica before this call; equality alone does not authorize data. - `apply_batch(mutations, position)` durably and atomically applies the next sequential replay position. Empty batches can advance a position (e.g. an irrelevant event). Equal latest position is accepted only for the identical normalized batch and returns its original receipt. Older/gapped positions conflict without modifying data. Upsert/delete convenience methods accept one projection/deletion and the same position. - `retry_cleanup()` retries physical SQLite/FTS/WAL cleanup and returns current status. Logical mutation receipts stay immutable. Every payload replacement/withdrawal atomically sets `cleanup_pending`; it remains true while a reader blocks WAL truncation or cleanup fails. A successful serialized retry clears it. Retry with bounded host backoff; backups and external copies require host retention controls. +- Last-reference destruction releases the GIL for native teardown. Explicit `close()` remains the deterministic lifecycle boundary, including on exception paths. - `close()` refuses subsequent calls and drains current native operations before releasing the store lock. A new store object can reopen/reconcile the replica. Use a new replica ID for a fresh rebuild, reconcile it, atomically route new calls to it, drain/close the previous object, then retire old files under retention policy. No unsafe in-place file swap API is exposed. Each projection has `memory_id` (opaque external ID), positive integer `revision`, `scope: {kind,id}`, `kind` (`fact`, `preference`, `experience`, `method`), `text`, `source_refs: [{kind,provider,id}]`, optional RFC3339 `valid_from`/`valid_until`, and optional numeric `embedding`. Missing validity bounds are unbounded. Validity is `[valid_from, valid_until)`. Scope kind is `owner`, `entity`, `project`, `workspace`, or `user`; owner scope ID must equal the store owner ID. Scope never changes for an existing memory ID, including after deletion. To change it, withdraw that identity and create another. Projection state is implicitly active; no inferred/candidate memory is accepted automatically. @@ -78,6 +79,11 @@ store.close() The current wheel uses plaintext SQLite; private filesystem controls are mandatory. No automatic sync/export/inference/consolidation is enabled. Logical withdrawal does not promise immediate erasure from backups or already-sent model requests. See the [integration plan](design/bot-memory-integration.md) for host-owned replay, rebuild, rollout and real-PG acceptance. -Filesystem SQLite URI names (`file:...`) are rejected by the private-file guard; use a filesystem path or `:memory:`. +On Unix, filesystem SQLite URI names (`file:...`) are rejected by the private-file guard; use a filesystem path or `:memory:`. Personal database parents must also have trusted ancestry (service/root ownership; no group/world write unless sticky). The immediate database directory must not be group/world-writable, including sticky shared directories; create a service-owned private child directory instead. Trusted directory aliases are resolved and revalidated; final symlinks, hardlinks, foreign-owned files and non-regular files are rejected. Permissions are prepared without opening/closing existing database or sidecar descriptors, preserving active SQLite process locks. The existing personal API constructor is `Cortex(db_path)`; it raises `RuntimeError` for a controlled database. It has no explicit close method; release its references when no longer used. + + +Personal `Cortex` retains its existing logical deletion semantics. Rust `Cortex`/`SqliteStorage` and Python `Cortex` now expose `cleanup_pending()` and `retry_cleanup()` (both return bool; true means physical work remains). Legacy deletion records pending cleanup in the same transaction as row removal, then invalidates caches/index/events regardless of cleanup contention. Logical deletion success does not assert that WAL truncation succeeded; check status and retry after readers release their snapshots. + +Existing databases without the erasure maintenance version undergo a checked, exclusive SQLite migration: rebuild current FTS data, VACUUM historical free pages, truncate WAL, then record completion. Upgrade may wait for other readers/writers and take longer for large files; quiesce other users and reopen if interrupted. Version zero remains pending through errors/crashes and is never reported as complete. New empty databases skip historical cleanup. This covers local SQLite artifacts, not backups, filesystems' old block copies, exports or content already sent elsewhere. diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index 610a19d..667e71c 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -81,7 +81,7 @@ Use a dedicated database per owner/bot and per serving-process replica. Derive p Persist a controlled-store marker. New legacy open/export/sync paths must refuse a marked controlled store, and controlled open refuses a personal store; do not rely only on caller discipline. The controlled binding exposes no general core/storage handle, legacy mutation, export, sync, or plugin access. Older binaries cannot honor a new marker, so deployment paths/access policy and rollback prohibit opening the controlled files with them. -Each replica has its own persistent replay cursor and instance ID; do not use a shared work-queue acknowledgement as proof that every replica has applied an event. P0 supports independent local replicas, not multiple processes sharing one index database. Enforce one live controlled handle per database with an OS-released-on-exit file lock, including duplicate opens within one process; another handle or process fails with an explicit busy result. This does not replace SQLite locks or imply a distributed lease. +Each replica has its own persistent replay cursor and instance ID; do not use a shared work-queue acknowledgement as proof that every replica has applied an event. P0 supports independent local replicas, not multiple processes sharing one index database. Enforce one live controlled handle per database with a process-owned POSIX file lock and an in-process registry; another handle or process fails with an explicit busy result. A process guard rejects inherited operations and new opens after a store-initialized process forks; use exec/spawn workers. Copied registry/SQLite locks are never entered in the child. This does not replace SQLite locks or imply a distributed lease. The Bot adapter manages replay for each replica. A new/lost replica builds from current PostgreSQL state; an expired replay cursor forces rebuild. A paused or lagging replica may serve only results that survive current authoritative revalidation, otherwise the Bot uses its PostgreSQL fallback. Record bounded lag and recovery state without memory text. @@ -171,13 +171,13 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr An acceptance subagent must be context-isolated from implementation: give it this plan and public schemas/docs, not implementation diffs or unit tests. Implementers write unit tests separately. Adversarial review is a third independent context, per repository rules. -Cortex-side automated evidence includes the independent installed-wheel acceptance suite (24 cases exercising Cortex-side portions of C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. The public timing-based crash cases establish restart consistency; separate native test-only barriers deterministically terminate a real subprocess inside the transaction and after commit before cache invalidation/response. These establish process-crash behavior, not power-loss durability or a real-PG replay handshake. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. +Cortex-side automated evidence includes the independent installed-wheel acceptance suite (31 cases exercising Cortex-side portions of C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. The public timing-based crash cases establish restart consistency; separate native test-only barriers deterministically terminate a real subprocess inside the transaction and after commit before cache invalidation/response. These establish process-crash behavior, not power-loss durability or a real-PG replay handshake. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. ## 9. Review, CI, rollout and evidence -For this implementation, validate local links, fences, acceptance IDs and `git diff --check`, plus the required source and installed-wheel checks. No Bot/Studio business acceptance is claimed. Keep the exact reviewed document content and head/base identifiers with raw model outputs in a traceable review record. Each model first reviews independently; cross-check only after both initial findings are saved. Fix every valid actionable finding at every severity, then repeat both reviews over the entire final diff; disputed findings require the originating reviewer to confirm closure. +For this implementation, validate local links, fences, acceptance IDs and `git diff --check`, plus the required source and installed-wheel checks. No Bot/Studio business acceptance is claimed. Keep the exact reviewed document content and head/base/tree identifiers with raw model outputs in a traceable review record. Each model first reviews independently; cross-check only after both initial findings are saved. Fix every valid actionable finding at every severity, then repeat both reviews over the entire final diff; disputed findings require the originating reviewer to confirm closure. -Before any implementation merge, require two different underlying reviewer models independently clean on the final complete head/base and all required CI on that same integration version. Recheck head/base, conflicts, findings and checks immediately before merging with exact-head constraint. Follow the authorized admin merge policy only after these gates; do not bypass failed/missing/skipped required checks. A main failure or base movement requires investigation and fresh evidence, not inherited clean conclusions. +Before any implementation merge, require two different underlying reviewer models independently clean on the final complete head/base and all required local CI on that same integration version. Recheck head/base, conflicts, findings and checks immediately before merging with exact-head constraint. Follow the authorized admin merge policy only after these gates; do not bypass failed/missing/skipped required checks. A main failure or base movement requires investigation and fresh evidence, not inherited clean conclusions. Required implementation verification includes `cargo test --workspace --exclude cortex-python --exclude cortex-wasm`, the current required Clippy/privacy jobs, installed-wheel tests, and the integration matrix above for that package. Relevant existing snapshots/golden/compatibility assertions must be checked; do not rerecord baselines to mask regressions. UI remains outside this plan; any later UI change needs desktop/mobile and affected flows. From d8c1aa465072b54e81730acd13c0989d9fce7679 Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 13:13:24 -0400 Subject: [PATCH 13/15] fix: retain partial sync records and handle coalesced directory events --- cortex-core/src/sync/oplog.rs | 17 +++++++----- cortex-core/src/sync/watcher.rs | 27 ++++++++++++++++++- cortex-core/tests/test_cache_and_perf.rs | 6 +++-- cortex-core/tests/test_cortex_full.rs | 3 ++- cortex-core/tests/test_db_file_permissions.rs | 12 +++++++++ cortex-core/tests/test_storage_advanced.rs | 6 +++-- cortex-core/tests/test_sync.rs | 15 +++++++++-- 7 files changed, 72 insertions(+), 14 deletions(-) diff --git a/cortex-core/src/sync/oplog.rs b/cortex-core/src/sync/oplog.rs index 4e7314b..425bc17 100644 --- a/cortex-core/src/sync/oplog.rs +++ b/cortex-core/src/sync/oplog.rs @@ -217,6 +217,15 @@ pub fn read_oplog( break; // EOF } + // A flush is not an atomic write: a scanner may observe a prefix while + // another process is appending. JSONL publication requires the newline. + // Leave the cursor at this record for both plaintext and encrypted tails. + if !line.ends_with('\n') { + use zeroize::Zeroize; + line.zeroize(); + break; + } + let trimmed = line.trim(); if trimmed.is_empty() { offset += bytes_read as u64; @@ -384,12 +393,8 @@ pub fn read_oplog( offset += bytes_read as u64; } Err(_) => { - // Distinguish partial write (incomplete line) from permanent corruption: - // - If the raw line ends with '\n', it's a complete but corrupt line → skip it - // - If at EOF without '\n', it's likely a partial write → don't advance, retry - // Always advance past bad lines to avoid stalling sync. - // Partial writes (no trailing \n) are also skipped — the writer - // uses flush() which guarantees complete lines on disk. + // Only newline-terminated records reach this branch. Skip complete + // corrupt records; an incomplete tail above retains its cursor. tracing::warn!("Skipping invalid oplog line at offset {}", offset); // Zeroize plaintext JSON from memory diff --git a/cortex-core/src/sync/watcher.rs b/cortex-core/src/sync/watcher.rs index efa921e..f9e5a48 100644 --- a/cortex-core/src/sync/watcher.rs +++ b/cortex-core/src/sync/watcher.rs @@ -132,10 +132,18 @@ pub fn start_watcher( /// Check whether a filesystem event is relevant (i.e., a .jsonl file from another device). fn is_relevant_event(event: &Event, my_device_id: &str) -> bool { + // FSEvents can coalesce a newly created subtree into directory/rescan + // notifications, omitting individual JSONL creates. Reuse pull_remote's + // authenticated scan; a notification is never proof of a complete oplog. + if event.need_rescan() { return true; } + let directory_event = matches!(event.kind, + EventKind::Create(notify::event::CreateKind::Folder) | + EventKind::Remove(notify::event::RemoveKind::Folder)); // Only care about creates (new oplog files) and close-write events. // Skip Modify events to avoid reading partially-written files. match event.kind { EventKind::Create(_) => {} + EventKind::Remove(notify::event::RemoveKind::Folder) => {} EventKind::Modify(notify::event::ModifyKind::Data(_)) => {} EventKind::Access(notify::event::AccessKind::Close(notify::event::AccessMode::Write)) => {} _ => return false, @@ -146,7 +154,7 @@ fn is_relevant_event(event: &Event, my_device_id: &str) -> bool { let is_jsonl = path .extension() .is_some_and(|ext| ext == "jsonl"); - if !is_jsonl { + if !is_jsonl && !directory_event { return false; } @@ -199,4 +207,21 @@ mod tests { }; assert!(!is_relevant_event(&event, "device-a")); } + #[test] + fn coalesced_remote_directory_requires_scan_but_own_directory_does_not() { + let mut event=Event::new(EventKind::Create(CreateKind::Folder)) + .add_path(PathBuf::from("/sync/devices/device-b")); + assert!(is_relevant_event(&event,"device-a")); + event.paths=vec![PathBuf::from("/sync/devices/device-a")]; + assert!(!is_relevant_event(&event,"device-a")); + event.paths.push(PathBuf::from("/sync/devices/device-b")); + assert!(is_relevant_event(&event,"device-a")); + } + #[test] + fn dropped_event_rescan_uses_existing_authenticated_pull() { + let mut event=Event::new(EventKind::Other); + event.attrs.set_flag(notify::event::Flag::Rescan); + assert!(is_relevant_event(&event,"device-a")); + } + } diff --git a/cortex-core/tests/test_cache_and_perf.rs b/cortex-core/tests/test_cache_and_perf.rs index 9fc8990..fa24451 100644 --- a/cortex-core/tests/test_cache_and_perf.rs +++ b/cortex-core/tests/test_cache_and_perf.rs @@ -240,7 +240,8 @@ fn test_retrieve_without_namespace_returns_all() { #[test] fn test_file_backed_storage_read_pool() { - let path = format!("/tmp/cortex_read_pool_{}.db", uuid::Uuid::new_v4()); + let private_dir = tempfile::tempdir().unwrap(); + let path = private_dir.path().join("memory.db").to_string_lossy().into_owned(); let cortex = Cortex::open(&path).unwrap(); // Ingest and retrieve — exercises read pool @@ -260,7 +261,8 @@ fn test_file_backed_storage_read_pool() { #[test] fn test_file_backed_concurrent_reads() { - let path = format!("/tmp/cortex_concurrent_{}.db", uuid::Uuid::new_v4()); + let private_dir = tempfile::tempdir().unwrap(); + let path = private_dir.path().join("memory.db").to_string_lossy().into_owned(); let cortex = Cortex::open(&path).unwrap(); for i in 0..20 { diff --git a/cortex-core/tests/test_cortex_full.rs b/cortex-core/tests/test_cortex_full.rs index c955d0c..b74ea6f 100644 --- a/cortex-core/tests/test_cortex_full.rs +++ b/cortex-core/tests/test_cortex_full.rs @@ -19,7 +19,8 @@ fn test_open_with_config() { ..Default::default() }; // Use a temp path - let path = format!("/tmp/cortex_test_{}.db", uuid::Uuid::new_v4()); + let private_dir = tempfile::tempdir().unwrap(); + let path = private_dir.path().join("memory.db").to_string_lossy().into_owned(); let cortex = Cortex::open_with_config(&path, config).unwrap(); // Should work normally diff --git a/cortex-core/tests/test_db_file_permissions.rs b/cortex-core/tests/test_db_file_permissions.rs index 0e99344..5cd72c2 100644 --- a/cortex-core/tests/test_db_file_permissions.rs +++ b/cortex-core/tests/test_db_file_permissions.rs @@ -90,3 +90,15 @@ fn symlink_and_hardlink_database_aliases_fail_closed() { std::fs::hard_link(&target, &link).unwrap(); assert!(Cortex::open(link.to_str().unwrap()).is_err()); } + +#[test] +fn shared_sticky_parent_requires_private_child_for_sidecar_safety() { + let dir=tempfile::tempdir().unwrap(); + std::fs::set_permissions(dir.path(),std::fs::Permissions::from_mode(0o1777)).unwrap(); + let unsafe_path=dir.path().join("unsafe.db"); + assert!(Cortex::open(unsafe_path.to_str().unwrap()).is_err()); + assert!(!unsafe_path.exists()); + let private=dir.path().join("private");std::fs::create_dir(&private).unwrap(); + std::fs::set_permissions(&private,std::fs::Permissions::from_mode(0o700)).unwrap(); + assert!(Cortex::open(private.join("memory.db").to_str().unwrap()).is_ok()); +} diff --git a/cortex-core/tests/test_storage_advanced.rs b/cortex-core/tests/test_storage_advanced.rs index ae30f51..ecaf593 100644 --- a/cortex-core/tests/test_storage_advanced.rs +++ b/cortex-core/tests/test_storage_advanced.rs @@ -178,7 +178,8 @@ fn test_store_link_upsert() { #[test] fn test_file_backed_wal_mode() { - let path = format!("/tmp/cortex_wal_test_{}.db", Uuid::new_v4()); + let private_dir = tempfile::tempdir().unwrap(); + let path = private_dir.path().join("memory.db").to_string_lossy().into_owned(); let s = SqliteStorage::open(&path).unwrap(); // Basic CRUD should work with WAL mode @@ -199,7 +200,8 @@ fn test_file_backed_wal_mode() { #[test] fn test_file_backed_read_pool_concurrent() { - let path = format!("/tmp/cortex_pool_test_{}.db", Uuid::new_v4()); + let private_dir = tempfile::tempdir().unwrap(); + let path = private_dir.path().join("memory.db").to_string_lossy().into_owned(); let s = SqliteStorage::open(&path).unwrap(); for i in 0..20 { diff --git a/cortex-core/tests/test_sync.rs b/cortex-core/tests/test_sync.rs index 18a836a..39e750e 100644 --- a/cortex-core/tests/test_sync.rs +++ b/cortex-core/tests/test_sync.rs @@ -488,11 +488,22 @@ fn test_oplog_partial_line_recovery() { hmac: None, }; let valid_line = serde_json::to_string(&op).unwrap(); - std::fs::write(&file_path, format!("{}\n{{corrupt partial", valid_line)).unwrap(); + let mut next=op.clone(); next.op_id=Uuid::new_v4(); + let next_line=serde_json::to_string(&next).unwrap(); + let split=next_line.len()/2; + std::fs::write(&file_path,format!("{}\n{}",valid_line,&next_line[..split])).unwrap(); - let (ops, _) = oplog::read_oplog(&file_path, 0, None).unwrap(); + let (ops, offset) = oplog::read_oplog(&file_path, 0, None).unwrap(); assert_eq!(ops.len(), 1); assert_eq!(ops[0].op_id, op.op_id); + assert_eq!(offset, (valid_line.len()+1) as u64); + use std::io::Write; + let mut append=std::fs::OpenOptions::new().append(true).open(&file_path).unwrap(); + writeln!(append,"{}",&next_line[split..]).unwrap();drop(append); + let (completed, final_offset)=oplog::read_oplog(&file_path,offset,None).unwrap(); + assert_eq!(completed.len(),1); + assert_eq!(completed[0].op_id,next.op_id); + assert_eq!(final_offset,(valid_line.len()+next_line.len()+2) as u64); } #[test] From a3bc77e27f6267ecf394bf7dd7071989dbad1e00 Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 13:32:46 -0400 Subject: [PATCH 14/15] fix: fence cleanup and recover interrupted oplog writes --- cortex-core/src/controlled.rs | 27 ++- cortex-core/src/storage/sqlite.rs | 197 ++++++++++++++++--- cortex-core/src/sync/oplog.rs | 19 +- cortex-core/tests/test_sync.rs | 45 +++++ cortex-python/src/lib.rs | 43 +++- cortex-python/tests/test_controlled_store.py | 32 +++ docs/controlled-memory-api.md | 4 +- docs/design/bot-memory-integration.md | 2 +- 8 files changed, 327 insertions(+), 42 deletions(-) diff --git a/cortex-core/src/controlled.rs b/cortex-core/src/controlled.rs index 45aadb3..746ec54 100644 --- a/cortex-core/src/controlled.rs +++ b/cortex-core/src/controlled.rs @@ -438,12 +438,8 @@ impl ControlledStore { let lease = ReplicaLease::acquire(&path)?; let file = private_file(&path.with_extension("lock"))?; lock_replica(&file)?; - for suffix in ["", "-wal", "-shm", "-journal"] { - let p = PathBuf::from(format!("{}{suffix}", path.display())); - if suffix.is_empty() || p.exists() || std::fs::symlink_metadata(&p).is_ok() { - drop(private_file(&p)?); - } - } + // SqliteStorage validates/prepares artifacts without opening unrelated + // descriptors; closing an existing SHM FD would release process WAL locks. let storage = SqliteStorage::open_controlled( path.to_str() .ok_or(ControlledError::InvalidInput("path encoding"))?, @@ -1324,6 +1320,25 @@ mod tests { probe_writer(&path, false); } #[test] + fn rejecting_personal_file_at_controlled_path_preserves_its_writer_lock() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().canonicalize().unwrap().join("private"); + prepare_root(&root, |_| {}).unwrap(); + let path = root.join(format!("{}.db", digest(&identity()).unwrap())); + let personal = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + personal + .with_controlled_conn(|conn| -> Result<()> { + conn.execute_batch("BEGIN IMMEDIATE;")?; + probe_writer(&path, true); + assert!(ControlledStore::open(&root, identity()).is_err()); + probe_writer(&path, true); + conn.execute_batch("ROLLBACK;")?; + Ok(()) + }) + .unwrap(); + probe_writer(&path, false); + } + #[test] fn old_personal_history_is_scrubbed_by_checked_upgrade() { use crate::storage::traits::StorageBackend; let dir = tempfile::tempdir().unwrap(); diff --git a/cortex-core/src/storage/sqlite.rs b/cortex-core/src/storage/sqlite.rs index f90b7bc..e45ab59 100644 --- a/cortex-core/src/storage/sqlite.rs +++ b/cortex-core/src/storage/sqlite.rs @@ -183,6 +183,10 @@ impl SqliteStorage { } #[cfg(unix)] fn restrict_db_file_mode(path: &str) -> Result { + Self::restrict_db_file_mode_with_hook(path, |_| {}) + } + #[cfg(unix)] + fn restrict_db_file_mode_with_hook(path: &str, mut after_metadata: impl FnMut(&std::path::Path)) -> Result { use std::os::unix::fs::{MetadataExt, PermissionsExt}; if path == ":memory:" { return Ok(path.to_owned()); } if path.is_empty() || path.starts_with("file:") { @@ -218,22 +222,42 @@ impl SqliteStorage { let artifact = if suffix.is_empty() { resolved.clone() } else { let mut name = resolved.as_os_str().to_os_string(); name.push(suffix); std::path::PathBuf::from(name) }; - let before = match std::fs::symlink_metadata(&artifact) { - Err(e) if !suffix.is_empty() && e.kind() == std::io::ErrorKind::NotFound => continue, - other => other.map_err(|e| CortexError::Storage(e.to_string()))?, - }; - if !before.is_file() || before.uid() != unsafe { libc::geteuid() } || before.nlink() != 1 { - return Err(CortexError::Storage("database must be an owner-owned regular file".into())); - } - // Safe under the verified, non-replaceable directory chain and the - // file's verified owner. No unrelated descriptors are opened/closed. - std::fs::set_permissions(&artifact, std::fs::Permissions::from_mode(0o600)) - .map_err(|e| CortexError::Storage(e.to_string()))?; - let after = std::fs::symlink_metadata(&artifact).map_err(|e| CortexError::Storage(e.to_string()))?; - if !after.is_file() || before.dev() != after.dev() || before.ino() != after.ino() - || before.uid() != after.uid() || after.nlink() != 1 || after.permissions().mode() & 0o777 != 0o600 { - return Err(CortexError::Storage("database file changed or is not private".into())); + let optional = !suffix.is_empty(); + let mut validated = false; + for _ in 0..4 { + let before = match std::fs::symlink_metadata(&artifact) { + Err(e) if optional && e.kind() == std::io::ErrorKind::NotFound => { validated = true; break; } + other => other.map_err(|e| CortexError::Storage(e.to_string()))?, + }; + if !before.is_file() || before.uid() != unsafe { libc::geteuid() } || before.nlink() != 1 { + return Err(CortexError::Storage("database must be an owner-owned regular file".into())); + } + after_metadata(&artifact); + // SQLite can remove/recreate optional sidecars as connections close. + // Revalidate pathname metadata; never open/close their descriptors. + match std::fs::set_permissions(&artifact, std::fs::Permissions::from_mode(0o600)) { + Err(e) if optional && e.kind() == std::io::ErrorKind::NotFound => continue, + other => other.map_err(|e| CortexError::Storage(e.to_string()))?, + } + let after = match std::fs::symlink_metadata(&artifact) { + Err(e) if optional && e.kind() == std::io::ErrorKind::NotFound => continue, + other => other.map_err(|e| CortexError::Storage(e.to_string()))?, + }; + if !after.is_file() || before.uid() != after.uid() || after.nlink() != 1 { + return Err(CortexError::Storage("database file changed or is not private".into())); + } + if before.dev() != after.dev() || before.ino() != after.ino() { + if optional { continue; } + return Err(CortexError::Storage("database file changed".into())); + } + if after.permissions().mode() & 0o777 != 0o600 { + return Err(CortexError::Storage("database file is not private".into())); + } + validated = true; + break; } + if !validated { return Err(CortexError::Storage("database sidecar kept changing".into())); } + } resolved.to_str().map(str::to_owned).ok_or_else(|| CortexError::InvalidInput("database path encoding".into())) } @@ -378,11 +402,17 @@ impl SqliteStorage { } fn initialize_privacy_maintenance(&self, had_memories: bool) -> Result<(), CortexError> { - let conn = self.write_conn.lock(); - conn.execute_batch("CREATE TABLE IF NOT EXISTS cortex_privacy_maintenance(id INTEGER PRIMARY KEY CHECK(id=1), erasure_version INTEGER NOT NULL, cleanup_pending INTEGER NOT NULL);") + let mut conn = self.write_conn.lock(); + // Serialize schema upgrades across personal-store handles/processes. + let tx = conn.transaction_with_behavior(rusqlite::TransactionBehavior::Immediate).map_err(|e| CortexError::Storage(e.to_string()))?; + tx.execute_batch("CREATE TABLE IF NOT EXISTS cortex_privacy_maintenance(id INTEGER PRIMARY KEY CHECK(id=1), erasure_version INTEGER NOT NULL, cleanup_pending INTEGER NOT NULL, cleanup_token TEXT NOT NULL DEFAULT '');") + .map_err(|e| CortexError::Storage(e.to_string()))?; + let has_token: bool = tx.query_row("SELECT EXISTS(SELECT 1 FROM pragma_table_info('cortex_privacy_maintenance') WHERE name='cleanup_token')", [], |r|r.get(0)) .map_err(|e| CortexError::Storage(e.to_string()))?; - conn.execute("INSERT OR IGNORE INTO cortex_privacy_maintenance(id,erasure_version,cleanup_pending) VALUES(1,?1,?2)", params![if had_memories { 0 } else { 1 }, had_memories]) + if !has_token { tx.execute_batch("ALTER TABLE cortex_privacy_maintenance ADD COLUMN cleanup_token TEXT NOT NULL DEFAULT '';").map_err(|e| CortexError::Storage(e.to_string()))?; } + tx.execute("INSERT OR IGNORE INTO cortex_privacy_maintenance(id,erasure_version,cleanup_pending) VALUES(1,?1,?2)", params![if had_memories { 0 } else { 1 }, had_memories]) .map_err(|e| CortexError::Storage(e.to_string()))?; + tx.commit().map_err(|e| CortexError::Storage(e.to_string()))?; // Version zero is itself pending even if a previous process crashed. let pending = Self::retry_personal_cleanup_on_connection(&conn)?; let version: i64 = conn.query_row("SELECT erasure_version FROM cortex_privacy_maintenance WHERE id=1", [], |r| r.get(0)).map_err(|e| CortexError::Storage(e.to_string()))?; @@ -402,7 +432,10 @@ impl SqliteStorage { Self::retry_personal_cleanup_on_connection(&self.write_conn.lock()) } fn retry_personal_cleanup_on_connection(conn: &Connection) -> Result { - let (version, pending): (i64,bool) = conn.query_row("SELECT erasure_version,cleanup_pending FROM cortex_privacy_maintenance WHERE id=1", [], |r| Ok((r.get(0)?,r.get(1)?))) + Self::retry_personal_cleanup_with_hook(conn, || {}) + } + fn retry_personal_cleanup_with_hook(conn: &Connection, after_checkpoint: impl FnOnce()) -> Result { + let (version, pending, token): (i64,bool,String) = conn.query_row("SELECT erasure_version,cleanup_pending,cleanup_token FROM cortex_privacy_maintenance WHERE id=1", [], |r| Ok((r.get(0)?,r.get(1)?,r.get(2)?))) .map_err(|e| CortexError::Storage(e.to_string()))?; if version != 0 && !pending { return Ok(false); } if version == 0 { @@ -424,9 +457,13 @@ impl SqliteStorage { return migrated.map(|done| !done).map_err(|e| CortexError::Storage(e.to_string())); } if Self::scrub_after_delete(conn) { - conn.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=0 WHERE id=1", []) + after_checkpoint(); + // A newer delete can commit between checkpoint and flag publication. + // UUID tokens never return to an older value (including after a clear). + conn.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=0 WHERE id=1 AND cleanup_token=?1", [token]) .map_err(|e| CortexError::Storage(e.to_string()))?; - return Ok(false); + return conn.query_row("SELECT cleanup_pending OR erasure_version=0 FROM cortex_privacy_maintenance WHERE id=1", [], |r|r.get(0)) + .map_err(|e| CortexError::Storage(e.to_string())); } Ok(true) } @@ -1086,7 +1123,7 @@ impl StorageBackend for SqliteStorage { let conn = self.write_conn.lock(); let tx = conn.unchecked_transaction().map_err(|e| CortexError::Storage(e.to_string()))?; tx.execute("DELETE FROM memories WHERE id=?1", [id.to_string()]).map_err(|e| CortexError::Storage(e.to_string()))?; - tx.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=1 WHERE id=1", []).map_err(|e| CortexError::Storage(e.to_string()))?; + tx.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=1,cleanup_token=?1 WHERE id=1", [Uuid::new_v4().to_string()]).map_err(|e| CortexError::Storage(e.to_string()))?; tx.commit().map_err(|e| CortexError::Storage(e.to_string()))?; self.cache_evict(&id); // Logical deletion is committed. Preserve its success and expose physical @@ -1960,12 +1997,11 @@ impl StorageBackend for SqliteStorage { return Ok(0); } let conn = self.write_conn.lock(); - conn.execute_batch("BEGIN TRANSACTION;") - .map_err(|e| CortexError::Storage(e.to_string()))?; + let tx = conn.unchecked_transaction().map_err(|e| CortexError::Storage(e.to_string()))?; let mut count = 0; { - let mut del_stmt = conn + let mut del_stmt = tx .prepare_cached("DELETE FROM memories WHERE id = ?1") .map_err(|e| CortexError::Storage(e.to_string()))?; for id in ids { @@ -1978,8 +2014,8 @@ impl StorageBackend for SqliteStorage { } // FTS5 synced via trigger (memories_fts_ad) - conn.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=1 WHERE id=1", []).map_err(|e| CortexError::Storage(e.to_string()))?; - conn.execute_batch("COMMIT;") + tx.execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=1,cleanup_token=?1 WHERE id=1", [Uuid::new_v4().to_string()]).map_err(|e| CortexError::Storage(e.to_string()))?; + tx.commit() .map_err(|e| CortexError::Storage(e.to_string()))?; if count > 0 && Self::retry_personal_cleanup_on_connection(&conn).is_err() { tracing::warn!("SQLite physical cleanup remains pending after logical deletion"); } // Evict deleted entries from cache @@ -2284,3 +2320,110 @@ mod row_parsing_tests { ); } } + +#[cfg(test)] +mod privacy_cleanup_tests { + use super::*; + use crate::storage::traits::StorageBackend; + use crate::types::{MemContent, MemObjectBuilder, MemSource, MemoryTier}; + fn memory(text: &str) -> MemObject { + MemObjectBuilder::new(MemoryTier::Episodic, MemContent::Text(text.into()), MemSource::new("test")).build() + } + #[test] + fn stale_cleanup_cannot_clear_a_newer_instances_delete() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().canonicalize().unwrap().join("race.db"); + let first = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + let second = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + let marker = "ZQXCLEANUPNEWERDELETEDPRIVATE"; + let record = memory(marker); + first.store_memory(&record).unwrap(); + first.write_conn.lock().execute("UPDATE cortex_privacy_maintenance SET cleanup_pending=1", []).unwrap(); + let reader = Connection::open_with_flags(&path, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY).unwrap(); + let conn = first.write_conn.lock(); + let pending = SqliteStorage::retry_personal_cleanup_with_hook(&conn, || { + reader.execute_batch("BEGIN;").unwrap(); + reader.query_row("SELECT COUNT(*) FROM memories", [], |r|r.get::<_,i64>(0)).unwrap(); + second.delete_memory(record.id).unwrap(); + assert!(second.cleanup_pending().unwrap()); + }).unwrap(); + assert!(pending, "older checkpoint must not clear a newer deletion"); + drop(conn); + assert!(first.cleanup_pending().unwrap()); + let contains = |p: &std::path::Path| std::fs::read(p).unwrap_or_default().windows(marker.len()).any(|w|w.eq_ignore_ascii_case(marker.as_bytes())); + assert!(contains(&path) || contains(&std::path::PathBuf::from(format!("{}-wal",path.display())))); + reader.execute_batch("ROLLBACK;").unwrap(); + drop(reader); + assert!(!first.retry_cleanup().unwrap()); + assert!(!second.cleanup_pending().unwrap()); + assert!(!contains(&path)); + assert!(!contains(&std::path::PathBuf::from(format!("{}-wal",path.display())))); + } + #[test] + fn existing_maintenance_table_acquires_cleanup_tokens_on_reopen() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().canonicalize().unwrap().join("upgrade-token.db"); + drop(SqliteStorage::open(path.to_str().unwrap()).unwrap()); + let conn = Connection::open(&path).unwrap(); + conn.execute_batch("DROP TABLE cortex_privacy_maintenance; + CREATE TABLE cortex_privacy_maintenance(id INTEGER PRIMARY KEY,erasure_version INTEGER NOT NULL,cleanup_pending INTEGER NOT NULL); + INSERT INTO cortex_privacy_maintenance VALUES(1,1,1);").unwrap(); + drop(conn); + let storage = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + assert!(!storage.cleanup_pending().unwrap()); + assert_eq!(storage.write_conn.lock().query_row("SELECT cleanup_token FROM cortex_privacy_maintenance", [], |r|r.get::<_,String>(0)).unwrap(), ""); + let record = memory("new deletion gets persistent token"); + storage.store_memory(&record).unwrap(); + storage.delete_memory(record.id).unwrap(); + let token: String = storage.write_conn.lock().query_row("SELECT cleanup_token FROM cortex_privacy_maintenance", [], |r|r.get(0)).unwrap(); + assert!(Uuid::parse_str(&token).is_ok()); + } + #[cfg(unix)] + #[test] + fn optional_sidecar_disappearance_and_replacement_are_revalidated() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().canonicalize().unwrap().join("sidecar.db"); + let storage = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + drop(storage); + let wal = std::path::PathBuf::from(format!("{}-wal", path.display())); + for replace in [false,true] { + std::fs::write(&wal,b"temporary sidecar").unwrap(); + let mut injected = false; + let result = SqliteStorage::restrict_db_file_mode_with_hook(path.to_str().unwrap(), |artifact| { + if artifact == wal && !injected { + injected = true; + std::fs::remove_file(&wal).unwrap(); + if replace { std::fs::write(&wal,b"replacement sidecar").unwrap(); } + } + }); + assert!(injected); + result.unwrap(); + if wal.exists() { std::fs::remove_file(&wal).unwrap(); } + } + } + #[test] + fn batch_maintenance_failure_rolls_back_rows_cache_and_transaction() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().canonicalize().unwrap().join("batch.db"); + let storage = SqliteStorage::open(path.to_str().unwrap()).unwrap(); + let record = memory("original cached private memory"); + storage.store_memory(&record).unwrap(); + assert!(storage.get_memory(record.id).unwrap().is_some()); + storage.write_conn.lock().execute_batch("CREATE TRIGGER fail_cleanup_state BEFORE UPDATE ON cortex_privacy_maintenance BEGIN SELECT RAISE(ABORT,'injected maintenance failure'); END;").unwrap(); + assert!(storage.delete_memories_batch(&[record.id]).is_err()); + assert!(storage.write_conn.lock().is_autocommit(), "failed batch leaked its transaction"); + let other = Connection::open(&path).unwrap(); + other.busy_timeout(std::time::Duration::from_millis(100)).unwrap(); + other.execute_batch("BEGIN IMMEDIATE; ROLLBACK;").unwrap(); + assert_eq!(other.query_row("SELECT COUNT(*) FROM memories WHERE id=?1", [record.id.to_string()], |r| r.get::<_, i64>(0)).unwrap(),1); + assert!(storage.get_memory(record.id).unwrap().is_some()); + assert!(!storage.cleanup_pending().unwrap()); + storage.write_conn.lock().execute_batch("DROP TRIGGER fail_cleanup_state;").unwrap(); + let next = memory("subsequent write commits independently"); + storage.store_memory(&next).unwrap(); + assert_eq!(other.query_row("SELECT COUNT(*) FROM memories", [], |r|r.get::<_,i64>(0)).unwrap(),2); + storage.delete_memories_batch(&[record.id,next.id]).unwrap(); + assert!(storage.get_memory(record.id).unwrap().is_none()); + assert!(storage.get_memory(next.id).unwrap().is_none()); + } +} diff --git a/cortex-core/src/sync/oplog.rs b/cortex-core/src/sync/oplog.rs index 425bc17..50a51b3 100644 --- a/cortex-core/src/sync/oplog.rs +++ b/cortex-core/src/sync/oplog.rs @@ -167,7 +167,24 @@ impl OpLogWriter { loop { let name = format!("oplog-{}-{:03}.jsonl", self.current_date, self.current_seq); let path = self.device_dir.join(&name); - if !path.exists() || fs::metadata(&path).map(|m| m.len()).unwrap_or(0) < MAX_FILE_SIZE { + // Never append an acknowledged operation onto a crash-truncated + // record. Preserve that file and rotate; readers retain its cursor + // independently and continue with the next file. + let appendable = match File::open(&path) { + Err(e) if e.kind() == std::io::ErrorKind::NotFound => true, + Err(e) => return Err(CortexError::Storage(format!("Failed to inspect oplog: {e}"))), + Ok(mut existing) => { + use std::io::{Read, Seek, SeekFrom}; + let size = existing.metadata().map_err(|e| CortexError::Storage(e.to_string()))?.len(); + if size == 0 { true } else if size >= MAX_FILE_SIZE { false } else { + existing.seek(SeekFrom::End(-1)).map_err(|e| CortexError::Storage(e.to_string()))?; + let mut last = [0u8;1]; + existing.read_exact(&mut last).map_err(|e| CortexError::Storage(e.to_string()))?; + last[0] == b'\n' + } + } + }; + if appendable { self.current_path = path; break; } diff --git a/cortex-core/tests/test_sync.rs b/cortex-core/tests/test_sync.rs index 39e750e..6100f02 100644 --- a/cortex-core/tests/test_sync.rs +++ b/cortex-core/tests/test_sync.rs @@ -1227,3 +1227,48 @@ fn test_manifest_without_hmac_is_rejected() { "an encryption manifest with its HMAC stripped must be rejected (downgrade/tamper defense)" ); } + +fn assert_oplog_restart_rotates_incomplete_tail(encrypted: bool) { + let tmp = TempDir::new().unwrap(); + let sync_dir = tmp.path().join("sync"); + let config = |device| { + let config = make_sync_config(&sync_dir, device); + if encrypted { config.with_encryption("test-only-restart-passphrase") } else { config } + }; + let cortex_a = Cortex::in_memory().unwrap(); + let make = |text: &str| { + let mut memory = MemObjectBuilder::new(MemoryTier::Episodic, MemContent::Text(text.into()), MemSource::new("test")).build(); + memory.privacy = PrivacyLevel::Shared { scope: "all".into() }; + memory + }; + let first = make("complete before torn append"); + let torn = make("crash truncated record"); + let next = make("first acknowledged operation after restart"); + let mut writer = SyncEngine::new(config("restart-device"),cortex_a.sqlite_storage()).unwrap(); + writer.record_op(SyncPayload::MemoryUpsert { memory: first.clone() }).unwrap(); + writer.record_op(SyncPayload::MemoryUpsert { memory: torn.clone() }).unwrap(); + drop(writer); + let old = oplog::list_oplog_files(&sync_dir.join("devices/restart-device")).unwrap().remove(0); + let length = std::fs::metadata(&old).unwrap().len(); + std::fs::OpenOptions::new().write(true).open(&old).unwrap().set_len(length-7).unwrap(); + let mut restarted = SyncEngine::new(config("restart-device"),cortex_a.sqlite_storage()).unwrap(); + restarted.record_op(SyncPayload::MemoryUpsert { memory: next.clone() }).unwrap(); + drop(restarted); + assert_eq!(std::fs::metadata(&old).unwrap().len(),length-7,"torn history must be preserved"); + let cortex_b = Cortex::in_memory().unwrap(); + let mut receiver = SyncEngine::new(config("receiver"),cortex_b.sqlite_storage()).unwrap(); + assert_eq!(receiver.pull_remote(cortex_b.sqlite_storage(),cortex_b.index()).unwrap(),2); + for memory in [&first,&next] { + assert_eq!(serde_json::to_value(cortex_b.storage().get_memory(memory.id).unwrap().unwrap().content).unwrap(),serde_json::to_value(&memory.content).unwrap()); + } + assert!(cortex_b.storage().get_memory(torn.id).unwrap().is_none()); + assert_eq!(receiver.pull_remote(cortex_b.sqlite_storage(),cortex_b.index()).unwrap(),0); + drop(receiver); + let mut receiver = SyncEngine::new(config("receiver"),cortex_b.sqlite_storage()).unwrap(); + assert_eq!(receiver.pull_remote(cortex_b.sqlite_storage(),cortex_b.index()).unwrap(),0); +} + +#[test] +fn test_oplog_restart_after_plaintext_partial_append() { assert_oplog_restart_rotates_incomplete_tail(false); } +#[test] +fn test_oplog_restart_after_encrypted_partial_append() { assert_oplog_restart_rotates_incomplete_tail(true); } diff --git a/cortex-python/src/lib.rs b/cortex-python/src/lib.rs index e2e9138..8d88110 100644 --- a/cortex-python/src/lib.rs +++ b/cortex-python/src/lib.rs @@ -292,6 +292,38 @@ fn bounded_value( } Err(invalid()) } +// Object positions follow the public dictionary schema. Serde struct decoding +// alone also accepts positional arrays, which are not part of this Python API. +fn controlled_object_shape(value: &serde_json::Value) -> bool { + value.as_object().is_some_and(|fields| { + fields.iter().all(|(key, value)| match key.as_str() { + "scope" | "projection" | "deletion" => controlled_object_shape(value), + "source_refs" => value + .as_array() + .is_some_and(|items| items.iter().all(controlled_object_shape)), + _ => true, + }) + }) +} +fn parse_structured( + py: Python<'_>, + value: &Bound<'_, PyAny>, + many: bool, + cap: usize, +) -> PyResult { + let value = bounded_value(py, value, 0, &mut (8 * 1024 * 1024), cap)?; + let valid = if many { + value + .as_array() + .is_some_and(|items| items.iter().all(controlled_object_shape)) + } else { + controlled_object_shape(&value) + }; + if !valid { + return Err(invalid_controlled(py)); + } + serde_json::from_value(value).map_err(|_| invalid_controlled(py)) +} fn parse_bounded( py: Python<'_>, value: &Bound<'_, PyAny>, @@ -406,7 +438,7 @@ impl PyControlledStore { } fn seed(&self, py: Python<'_>, projections: &Bound<'_, PyAny>) -> PyResult> { let s = self.handle(py)?; - let p = parse_controlled(py, projections)?; + let p = parse_structured(py, projections, true, 100)?; controlled_result(py, py.allow_threads(|| s.seed(p))) } fn finish_snapshot(&self, py: Python<'_>, position: &Bound<'_, PyAny>) -> PyResult> { @@ -432,7 +464,7 @@ impl PyControlledStore { position: &Bound<'_, PyAny>, ) -> PyResult> { let s = self.handle(py)?; - let m = parse_controlled(py, mutations)?; + let m = parse_structured(py, mutations, true, 100)?; let p = parse_controlled(py, position)?; controlled_result(py, py.allow_threads(|| s.apply_batch(m, p))) } @@ -443,7 +475,7 @@ impl PyControlledStore { position: &Bound<'_, PyAny>, ) -> PyResult> { let s = self.handle(py)?; - let m = parse_controlled(py, projection)?; + let m = parse_structured(py, projection, false, 100)?; let p = parse_controlled(py, position)?; controlled_result(py, py.allow_threads(|| s.upsert(m, p))) } @@ -454,7 +486,7 @@ impl PyControlledStore { position: &Bound<'_, PyAny>, ) -> PyResult> { let s = self.handle(py)?; - let m = parse_controlled(py, deletion)?; + let m = parse_structured(py, deletion, false, 100)?; let p = parse_controlled(py, position)?; controlled_result(py, py.allow_threads(|| s.delete(m, p))) } @@ -468,7 +500,8 @@ impl PyControlledStore { ) -> PyResult> { let s = self.handle(py)?; let query: String = parse_controlled(py, query)?; - let scopes: Vec = parse_bounded(py, allowed_scopes, 64)?; + let scopes: Vec = + parse_structured(py, allowed_scopes, true, 64)?; let mut limit: usize = 10; let mut embedding: Option> = None; let mut max_candidates: usize = 1000; diff --git a/cortex-python/tests/test_controlled_store.py b/cortex-python/tests/test_controlled_store.py index 31b0af7..56f0933 100644 --- a/cortex-python/tests/test_controlled_store.py +++ b/cortex-python/tests/test_controlled_store.py @@ -925,6 +925,38 @@ def score(query): embedding=[1.0, 0.0])["hits"][0] self.assertGreater(match["score"], unrelated["score"]) + def test_structured_objects_reject_positional_lists_without_mutation(self): + scope_list = ["owner", SCOPE["id"]] + source_list = ["input", "bot", "id"] + projected = projection(revision=2, text="orchid positional replacement") + projection_list = [projected["memory_id"], projected["revision"], projected["scope"], + projected["kind"], projected["text"], projected["source_refs"], + None, None, None] + deletion_list = ["memory-one", 2, dict(SCOPE)] + cases = [ + ("allowed_scope", lambda store: store.search("orchid", [scope_list])), + ("projection_scope", lambda store: store.upsert(dict(projected, scope=scope_list), 101)), + ("source_ref", lambda store: store.upsert(dict(projected, source_refs=[source_list]), 101)), + ("deletion_scope", lambda store: store.delete(dict(deletion(), scope=scope_list), 101)), + ("projection", lambda store: store.upsert(projection_list, 101)), + ("deletion", lambda store: store.delete(deletion_list, 101)), + ("batch_projection", lambda store: store.apply_batch([upsert(projection_list)], 101)), + ("batch_deletion", lambda store: store.apply_batch( + [{"op": "delete", "deletion": deletion_list}], 101)), + ] + for index, (name, call) in enumerate(cases): + store = self.ready([projection()], replica="dict-shape-%d" % index) + with self.subTest(case=name, check="invalid_input"): + self.error("invalid_input", call, store) + # Separate subtests retain state-integrity evidence even if the + # malformed call unexpectedly succeeds instead of raising. + with self.subTest(case=name, check="position_unchanged"): + self.assertEqual(store.status()["position"], 100) + with self.subTest(case=name, check="payload_unchanged"): + hits = self.hits(store) + self.assertEqual([(hit["memory_id"], hit["revision"], hit["text"]) for hit in hits], + [("memory-one", 1, "orchid report preference")]) + def test_restored_local_transaction_loss_rejects_remote_cursor(self): store = self.ready([projection()]) state = store.status() diff --git a/docs/controlled-memory-api.md b/docs/controlled-memory-api.md index 71a75de..b767d55 100644 --- a/docs/controlled-memory-api.md +++ b/docs/controlled-memory-api.md @@ -51,7 +51,7 @@ Every hit is an **internal candidate**. Before any UI/log/context exposure, the ## Limits and errors -Identifiers: 1–256 UTF-8 bytes, no control characters. Owner/scope kinds are constrained as above. Text: at most 12,000 characters / 48,000 bytes. Sources: 1–16; each field uses identifier limits. Batches: at most 100 entries. Query: 1–4,000 characters and at most 64 alphanumeric terms; limit 1–100; at most 64 allowed scopes; candidates 1–10,000; time budget 1–250 ms; embeddings 1–4,096 finite floats with finite squared norm, with a consistent store-wide dimension. Python inputs must use exact built-in dict/list/string/int/float/bool/None types; custom objects and container subclasses are rejected. Conversion checks container limits before copying, rejects nesting deeper than eight levels, and enforces a conservative 8 MiB JSON-size budget incrementally. Search options are keyword-only; malformed values (including bool or overflowing bounds) raise `invalid_input`. Duplicate memory IDs inside one batch are rejected. Unknown fields, bool/fractional/negative revisions or positions, invalid intervals and malformed vectors fail validation; source order is normalized for replay identity. +Identifiers: 1–256 UTF-8 bytes, no control characters. Owner/scope kinds are constrained as above. Text: at most 12,000 characters / 48,000 bytes. Sources: 1–16; each field uses identifier limits. Batches: at most 100 entries. Query: 1–4,000 characters and at most 64 alphanumeric terms; limit 1–100; at most 64 allowed scopes; candidates 1–10,000; time budget 1–250 ms; embeddings 1–4,096 finite floats with finite squared norm, with a consistent store-wide dimension. Python inputs must use exact built-in dict/list/string/int/float/bool/None types; custom objects and container subclasses are rejected. Dictionary positions (including nested scopes, source references and batch envelopes) reject positional lists. Conversion checks container limits before copying, rejects nesting deeper than eight levels, and enforces a conservative 8 MiB JSON-size budget incrementally. Search options are keyword-only; malformed values (including bool or overflowing bounds) raise `invalid_input`. Duplicate memory IDs inside one batch are rejected. Unknown fields, bool/fractional/negative revisions or positions, invalid intervals and malformed vectors fail validation; source order is normalized for replay identity. `ControlledError` has stable `.code`: `invalid_input`, `conflict`, `busy`, `not_ready`, `closed`, or `storage`. Stale memory revisions are receipt outcomes, not exceptions. Storage/commit ambiguity disables queries until reopen/reconciliation. No error text contains projection bodies. Python integer positions/revisions preserve precision up to SQLite's signed 64-bit maximum. @@ -84,6 +84,6 @@ On Unix, filesystem SQLite URI names (`file:...`) are rejected by the private-fi The existing personal API constructor is `Cortex(db_path)`; it raises `RuntimeError` for a controlled database. It has no explicit close method; release its references when no longer used. -Personal `Cortex` retains its existing logical deletion semantics. Rust `Cortex`/`SqliteStorage` and Python `Cortex` now expose `cleanup_pending()` and `retry_cleanup()` (both return bool; true means physical work remains). Legacy deletion records pending cleanup in the same transaction as row removal, then invalidates caches/index/events regardless of cleanup contention. Logical deletion success does not assert that WAL truncation succeeded; check status and retry after readers release their snapshots. +Personal `Cortex` retains its existing logical deletion semantics. Rust `Cortex`/`SqliteStorage` and Python `Cortex` now expose `cleanup_pending()` and `retry_cleanup()` (both return bool; true means physical work remains). Legacy deletion records pending cleanup in the same transaction as row removal, then invalidates caches/index/events regardless of cleanup contention. A persisted deletion token fences cleanup completion so an older retry cannot clear a newer deletion from another handle or process. Logical deletion success does not assert that WAL truncation succeeded; check status and retry after readers release their snapshots. Existing databases without the erasure maintenance version undergo a checked, exclusive SQLite migration: rebuild current FTS data, VACUUM historical free pages, truncate WAL, then record completion. Upgrade may wait for other readers/writers and take longer for large files; quiesce other users and reopen if interrupted. Version zero remains pending through errors/crashes and is never reported as complete. New empty databases skip historical cleanup. This covers local SQLite artifacts, not backups, filesystems' old block copies, exports or content already sent elsewhere. diff --git a/docs/design/bot-memory-integration.md b/docs/design/bot-memory-integration.md index 667e71c..957e4e7 100644 --- a/docs/design/bot-memory-integration.md +++ b/docs/design/bot-memory-integration.md @@ -171,7 +171,7 @@ Use synthetic memory identities/text, real temporary SQLite and real test Postgr An acceptance subagent must be context-isolated from implementation: give it this plan and public schemas/docs, not implementation diffs or unit tests. Implementers write unit tests separately. Adversarial review is a third independent context, per repository rules. -Cortex-side automated evidence includes the independent installed-wheel acceptance suite (31 cases exercising Cortex-side portions of C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. The public timing-based crash cases establish restart consistency; separate native test-only barriers deterministically terminate a real subprocess inside the transaction and after commit before cache invalidation/response. These establish process-crash behavior, not power-loss durability or a real-PG replay handshake. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. +Cortex-side automated evidence includes the independent installed-wheel acceptance suite (32 cases exercising Cortex-side portions of C01-C13 and local crash/restore parts of C16), core unit fault injection and legacy regression suites. The public timing-based crash cases establish restart consistency; separate native test-only barriers deterministically terminate a real subprocess inside the transaction and after commit before cache invalidation/response. These establish process-crash behavior, not power-loss durability or a real-PG replay handshake. C14-C18 still require the Bot/Studio adapter and real PostgreSQL; no passing local test substitutes for those. ## 9. Review, CI, rollout and evidence From 81e195a44021795c8099ef57ceb449ff45f80f8b Mon Sep 17 00:00:00 2001 From: gambletan Date: Thu, 22 Oct 2026 13:48:25 -0400 Subject: [PATCH 15/15] fix: atomically claim storage mode and recover partial writes --- cortex-core/src/controlled.rs | 26 +++++++ cortex-core/src/storage/sqlite.rs | 121 ++++++++++++++++++++---------- cortex-core/src/sync/oplog.rs | 99 ++++++++++++++++++------ cortex-core/tests/test_sync.rs | 35 +++++++++ docs/controlled-memory-api.md | 2 +- 5 files changed, 217 insertions(+), 66 deletions(-) diff --git a/cortex-core/src/controlled.rs b/cortex-core/src/controlled.rs index 746ec54..f73b953 100644 --- a/cortex-core/src/controlled.rs +++ b/cortex-core/src/controlled.rs @@ -1320,6 +1320,32 @@ mod tests { probe_writer(&path, false); } #[test] + fn simultaneous_public_openers_claim_exactly_one_database_mode() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().canonicalize().unwrap().join("private"); + prepare_root(&root, |_| {}).unwrap(); + let path = root.join(format!("{}.db", digest(&identity()).unwrap())); + let barrier = Arc::new(std::sync::Barrier::new(2)); + let other_barrier = barrier.clone(); + let other_path = path.clone(); + let personal = std::thread::spawn(move || { + other_barrier.wait(); + SqliteStorage::open(other_path.to_str().unwrap()) + }); + barrier.wait(); + let controlled = ControlledStore::open(&root, identity()); + let personal = personal.join().unwrap(); + assert_ne!( + controlled.is_ok(), + personal.is_ok(), + "exactly one public mode may initialize" + ); + drop(controlled); + drop(personal); + let conn = rusqlite::Connection::open(&path).unwrap(); + assert_eq!(conn.query_row("SELECT COUNT(*) FROM sqlite_master WHERE name IN ('cortex_controlled_marker','cortex_personal_marker')",[],|r|r.get::<_,i64>(0)).unwrap(),1); + } + #[test] fn rejecting_personal_file_at_controlled_path_preserves_its_writer_lock() { let dir = tempfile::tempdir().unwrap(); let root = dir.path().canonicalize().unwrap().join("private"); diff --git a/cortex-core/src/storage/sqlite.rs b/cortex-core/src/storage/sqlite.rs index e45ab59..2c20760 100644 --- a/cortex-core/src/storage/sqlite.rs +++ b/cortex-core/src/storage/sqlite.rs @@ -291,6 +291,42 @@ impl SqliteStorage { Ok(storage) } + fn admit_database_mode(conn: &Connection, controlled: bool, after_classification: impl FnOnce()) -> Result<(), CortexError> { + // Claim the mode before any schema initialization. Both openers must use + // this same transaction; merely inspecting a marker permits mode races. + let tx = rusqlite::Transaction::new_unchecked(conn, rusqlite::TransactionBehavior::Immediate) + .map_err(|e| CortexError::Storage(e.to_string()))?; + let exists = |name: &str| tx.query_row("SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name=?1)", [name], |r| r.get::<_,bool>(0)) + .map_err(|e| CortexError::Storage(e.to_string())); + let marked = exists("cortex_controlled_marker")?; + let personal = exists("cortex_personal_marker")?; + if (marked && personal) || (controlled && personal) || (!controlled && marked) { + return Err(CortexError::InvalidInput("database mode conflicts with requested API".into())); + } + let marker = if controlled { "cortex_controlled_marker" } else { "cortex_personal_marker" }; + if marked || personal { + let marker_is_table: bool = tx.query_row("SELECT type='table' FROM sqlite_master WHERE name=?1", [marker], |r|r.get(0)) + .map_err(|e| CortexError::Storage(e.to_string()))?; + if !marker_is_table { return Err(CortexError::InvalidInput("invalid database mode marker".into())); } + let sql = format!("SELECT COUNT(*),MIN(version),MAX(version) FROM {marker}"); + let version: (i64,Option,Option) = tx.query_row(&sql, [], |r| Ok((r.get(0)?,r.get(1)?,r.get(2)?))) + .map_err(|e| CortexError::Storage(e.to_string()))?; + if version != (1,Some(1),Some(1)) { return Err(CortexError::InvalidInput("unsupported database mode format".into())); } + } else if controlled { + let nonempty: bool = tx.query_row("SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name NOT GLOB 'sqlite_*')", [], |r|r.get(0)) + .map_err(|e| CortexError::Storage(e.to_string()))?; + if nonempty { return Err(CortexError::InvalidInput("cannot import a personal database".into())); } + } + after_classification(); + if !marked && !personal { + // The personal marker also reserves a fresh file across a crash or + // initialization failure, before memories/FTS have been created. + tx.execute_batch(&format!("CREATE TABLE {marker}(version INTEGER NOT NULL); INSERT INTO {marker} VALUES(1);")) + .map_err(|e| CortexError::Storage(e.to_string()))?; + } + tx.commit().map_err(|e| CortexError::Storage(e.to_string())) + } + fn open_keyed_mode( path: &str, key: Option, @@ -307,46 +343,7 @@ impl SqliteStorage { let conn = Connection::open(path).map_err(|e| CortexError::Storage(e.to_string()))?; Self::apply_key(&conn, key.as_ref())?; - let marked: bool = conn - .query_row( - "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name='cortex_controlled_marker')", - [], - |r| r.get(0), - ) - .map_err(|e| CortexError::Storage(e.to_string()))?; - if marked && !controlled { - return Err(CortexError::InvalidInput( - "controlled database requires ControlledStore".into(), - )); - } - if controlled && !marked { - let nonempty: bool = conn - .query_row( - "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE name NOT LIKE 'sqlite_%')", - [], - |r| r.get(0), - ) - .map_err(|e| CortexError::Storage(e.to_string()))?; - if nonempty { - return Err(CortexError::InvalidInput( - "cannot import a personal database".into(), - )); - } - conn.execute_batch("BEGIN IMMEDIATE; CREATE TABLE cortex_controlled_marker(version INTEGER NOT NULL); INSERT INTO cortex_controlled_marker VALUES(1); COMMIT;") - .map_err(|e| CortexError::Storage(e.to_string()))?; - } - if controlled { - let version: i64 = conn - .query_row("SELECT version FROM cortex_controlled_marker", [], |r| { - r.get(0) - }) - .map_err(|e| CortexError::Storage(e.to_string()))?; - if version != 1 { - return Err(CortexError::InvalidInput( - "unsupported controlled store format".into(), - )); - } - } + Self::admit_database_mode(&conn, controlled, || {})?; conn.execute_batch( "PRAGMA journal_mode=WAL; @@ -2359,6 +2356,50 @@ mod privacy_cleanup_tests { assert!(!contains(&path)); assert!(!contains(&std::path::PathBuf::from(format!("{}-wal",path.display())))); } + fn assert_mode_admission_is_serialized(first_controlled: bool) { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("mode-race.db"); + let first = Connection::open(&path).unwrap(); + let second = Connection::open(&path).unwrap(); + second.busy_timeout(std::time::Duration::ZERO).unwrap(); + let mut second_admitted = false; + SqliteStorage::admit_database_mode(&first, first_controlled, || { + second_admitted = SqliteStorage::admit_database_mode(&second, !first_controlled, || {}).is_ok(); + if second_admitted && first_controlled { + // A personal opener proceeds to initialize/populate before the + // paused controlled opener creates its marker. + second.execute_batch("CREATE TABLE personal_payload(text TEXT); INSERT INTO personal_payload VALUES('must not be imported');").unwrap(); + } + }).unwrap(); + assert!(!second_admitted,"both modes passed admission across classification"); + assert!(SqliteStorage::admit_database_mode(&second, !first_controlled, || {}).is_err()); + } + #[test] + fn personal_admission_blocks_a_competing_controlled_opener() { assert_mode_admission_is_serialized(false); } + #[test] + fn controlled_admission_blocks_a_competing_personal_opener() { assert_mode_admission_is_serialized(true); } + #[test] + fn mode_admission_rejects_user_objects_with_sqlite_like_names_and_fake_markers() { + for definition in ["CREATE TABLE sqlitex_payload(text TEXT)","CREATE VIEW sqlitex_view AS SELECT 1 AS version", "CREATE VIEW cortex_controlled_marker AS SELECT 1 AS version"] { + let conn=Connection::open_in_memory().unwrap();conn.execute_batch(definition).unwrap(); + assert!(SqliteStorage::admit_database_mode(&conn,true,||{}).is_err()); + } + } + #[test] + fn admission_reservation_survives_initialization_failure_and_rejects_conflicting_markers() { + for controlled in [false,true] { + let dir=tempfile::tempdir().unwrap();let path=dir.path().join("reserved.db"); + let conn=Connection::open(&path).unwrap(); + SqliteStorage::admit_database_mode(&conn,controlled,||{}).unwrap();drop(conn); + // Simulate process death before memories/FTS initialization. + let conn=Connection::open(&path).unwrap(); + assert!(SqliteStorage::admit_database_mode(&conn,!controlled,||{}).is_err()); + SqliteStorage::admit_database_mode(&conn,controlled,||{}).unwrap(); + let other=if controlled {"cortex_personal_marker"}else{"cortex_controlled_marker"}; + conn.execute_batch(&format!("CREATE TABLE {other}(version INTEGER NOT NULL);INSERT INTO {other} VALUES(1);")).unwrap(); + for mode in [false,true] {assert!(SqliteStorage::admit_database_mode(&conn,mode,||{}).is_err());} + } + } #[test] fn existing_maintenance_table_acquires_cleanup_tokens_on_reopen() { let dir = tempfile::tempdir().unwrap(); diff --git a/cortex-core/src/sync/oplog.rs b/cortex-core/src/sync/oplog.rs index 50a51b3..4274e64 100644 --- a/cortex-core/src/sync/oplog.rs +++ b/cortex-core/src/sync/oplog.rs @@ -115,19 +115,23 @@ impl OpLogWriter { op.hmac = Some(base64::engine::general_purpose::STANDARD.encode(hmac_bytes)); } - let mut json = serde_json::to_string(&op) - .map_err(|e| CortexError::Serialization(e.to_string()))?; + let mut json = zeroize::Zeroizing::new(serde_json::to_string(&op) + .map_err(|e| CortexError::Serialization(e.to_string()))?); - let line = if let Some(ref ctx) = self.crypto { + let line = zeroize::Zeroizing::new(if let Some(ref ctx) = self.crypto { crate::sync::crypto::encrypt_line(ctx, json.as_bytes())? } else { - json.clone() - }; + json.to_string() + }); let file = self.current_file.as_mut() .ok_or_else(|| CortexError::Storage("No open oplog file".into()))?; - writeln!(file, "{}", line) - .map_err(|e| CortexError::Storage(format!("Failed to write oplog: {}", e)))?; + if let Err(error) = writeln!(file, "{}", line.as_str()) { + // A failed write can leave an arbitrary prefix. Never acknowledge a + // later operation by appending it to that prefix, even without restart. + self.current_file.take(); + return Err(CortexError::Storage(format!("Failed to write oplog: {error}"))); + } // Zeroize plaintext JSON from memory json.zeroize(); @@ -138,8 +142,10 @@ impl OpLogWriter { /// Flush the oplog file to disk. pub fn flush(&mut self) -> Result<(), CortexError> { if let Some(ref mut file) = self.current_file { - file.flush() - .map_err(|e| CortexError::Storage(format!("Failed to flush oplog: {}", e)))?; + if let Err(error) = file.flush() { + self.current_file.take(); + return Err(CortexError::Storage(format!("Failed to flush oplog: {error}"))); + } } Ok(()) } @@ -226,22 +232,23 @@ pub fn read_oplog( const CORRUPTION_THRESHOLD: u32 = 5; // Alert if 5+ consecutive HMAC failures loop { - let mut line = String::new(); - let bytes_read = reader.read_line(&mut line) - .map_err(|e| CortexError::Storage(format!("Failed to read oplog line: {}", e)))?; - - if bytes_read == 0 { - break; // EOF - } - - // A flush is not an atomic write: a scanner may observe a prefix while - // another process is appending. JSONL publication requires the newline. - // Leave the cursor at this record for both plaintext and encrypted tails. - if !line.ends_with('\n') { - use zeroize::Zeroize; - line.zeroize(); - break; - } + let mut buffer = zeroize::Zeroizing::new(Vec::new()); + let bytes_read = reader.read_until(b'\n', &mut buffer) + .map_err(|e| CortexError::Storage(format!("Failed to read oplog line: {e}")))?; + if bytes_read == 0 { break; } + // Publication is checked in bytes: a crash may split a UTF-8 codepoint. + // All incomplete tails retain their cursor before decoding/authentication. + if !buffer.ends_with(b"\n") { break; } + let line = match std::str::from_utf8(&buffer) { + Ok(line) => line, + Err(_) => { + // A terminated malformed record follows the existing complete + // corrupt-record policy; no bytes are authenticated or applied. + tracing::warn!("Invalid UTF-8 in completed oplog record at offset {}", offset); + offset += bytes_read as u64; + continue; + } + }; let trimmed = line.trim(); if trimmed.is_empty() { @@ -629,3 +636,45 @@ mod tests { assert_eq!(parsed.op_id, op.op_id); } } + +#[cfg(all(test,unix))] +mod write_failure_tests { + use super::*; + use std::time::{Duration,Instant}; + #[test] + fn limited_writer_child() { + let Ok(root)=std::env::var("CORTEX_TEST_LIMITED_WRITER_ROOT") else { return; }; + let encrypted=std::env::var("CORTEX_TEST_LIMITED_WRITER_ENCRYPTED").unwrap()=="1"; + let crypto=encrypted.then(||std::sync::Arc::new(crate::sync::crypto::derive_key("test-only-io-failure",&crate::sync::crypto::new_encryption_manifest()).unwrap())); + let root=PathBuf::from(root); + let make=||SyncOp {op_id:Uuid::new_v4(),hlc:HlcTimestamp::new(1000,0,"io-writer"),payload:SyncPayload::MemoryDelete {id:Uuid::new_v4()},hmac:None}; + let mut writer=OpLogWriter::new(root.clone(),crypto.clone()).unwrap(); + let first=make();writer.append(first.clone()).unwrap(); + let old=writer.current_path.clone();let before=fs::metadata(&old).unwrap().len(); + let mut previous=std::mem::MaybeUninit::::uninit(); + assert_eq!(unsafe {libc::getrlimit(libc::RLIMIT_FSIZE,previous.as_mut_ptr())},0); + let previous=unsafe {previous.assume_init()}; + struct Restore {limit:libc::rlimit,handler:libc::sighandler_t} + impl Drop for Restore { fn drop(&mut self) {assert_eq!(unsafe {libc::setrlimit(libc::RLIMIT_FSIZE,&self.limit)},0);unsafe {libc::signal(libc::SIGXFSZ,self.handler);}} } + let restore=Restore {limit:previous,handler:unsafe {libc::signal(libc::SIGXFSZ,libc::SIG_IGN)}}; + let capped=libc::rlimit {rlim_cur:(before+32) as libc::rlim_t,rlim_max:previous.rlim_max}; + assert_eq!(unsafe {libc::setrlimit(libc::RLIMIT_FSIZE,&capped)},0); + assert!(writer.append(make()).is_err(),"real file-size limit must interrupt append"); + let partial=fs::metadata(&old).unwrap().len();assert!(partial>before);assert_eq!(partial,before+32); + drop(restore); + let next=make();writer.append(next.clone()).unwrap();drop(writer); + let mut ops=Vec::new();for file in list_oplog_files(&root).unwrap() {ops.extend(read_oplog(&file,0,crypto.as_deref()).unwrap().0);} + assert_eq!(ops.len(),2,"acknowledged append after same-writer recovery must survive");assert_eq!(ops[0].op_id,first.op_id);assert_eq!(ops[1].op_id,next.op_id); + } + fn run_limited_writer(encrypted:bool) { + let dir=tempfile::tempdir().unwrap(); + let mut child=std::process::Command::new(std::env::current_exe().unwrap()).args(["sync::oplog::write_failure_tests::limited_writer_child","--exact","--nocapture"]).env("CORTEX_TEST_LIMITED_WRITER_ROOT",dir.path()).env("CORTEX_TEST_LIMITED_WRITER_ENCRYPTED",if encrypted {"1"}else{"0"}).stdout(std::process::Stdio::piped()).stderr(std::process::Stdio::piped()).spawn().unwrap(); + let deadline=Instant::now()+Duration::from_secs(30); + while child.try_wait().unwrap().is_none() { if Instant::now()>=deadline {let _=child.kill();let _=child.wait();panic!("limited writer child timed out");}std::thread::sleep(Duration::from_millis(10));} + let output=child.wait_with_output().unwrap();assert!(output.status.success(),"limited child failed: {} {}",String::from_utf8_lossy(&output.stdout),String::from_utf8_lossy(&output.stderr)); + } + #[test] + fn plaintext_partial_write_failure_recovers_same_writer() {run_limited_writer(false);} + #[test] + fn encrypted_partial_write_failure_recovers_same_writer() {run_limited_writer(true);} +} diff --git a/cortex-core/tests/test_sync.rs b/cortex-core/tests/test_sync.rs index 6100f02..917297a 100644 --- a/cortex-core/tests/test_sync.rs +++ b/cortex-core/tests/test_sync.rs @@ -1272,3 +1272,38 @@ fn assert_oplog_restart_rotates_incomplete_tail(encrypted: bool) { fn test_oplog_restart_after_plaintext_partial_append() { assert_oplog_restart_rotates_incomplete_tail(false); } #[test] fn test_oplog_restart_after_encrypted_partial_append() { assert_oplog_restart_rotates_incomplete_tail(true); } + +#[test] +fn test_oplog_restart_after_torn_utf8_character() { + let tmp = TempDir::new().unwrap(); + let sync_dir = tmp.path().join("utf8-sync"); + let cortex_a = Cortex::in_memory().unwrap(); + let mut engine = SyncEngine::new(make_sync_config(&sync_dir,"utf8-writer"),cortex_a.sqlite_storage()).unwrap(); + let make = |text: &str| { + let mut m = MemObjectBuilder::new(MemoryTier::Episodic, MemContent::Text(text.into()), MemSource::new("test")).build(); + m.privacy = PrivacyLevel::Shared { scope: "all".into() };m + }; + let first=make("complete before UTF8 tail");let torn=make("before 猫 after");let next=make("next acknowledged record"); + engine.record_op(SyncPayload::MemoryUpsert {memory:first.clone()}).unwrap(); + let old=oplog::list_oplog_files(&sync_dir.join("devices/utf8-writer")).unwrap().remove(0); + let complete_offset=std::fs::metadata(&old).unwrap().len(); + engine.record_op(SyncPayload::MemoryUpsert {memory:torn.clone()}).unwrap();drop(engine); + let bytes=std::fs::read(&old).unwrap();let cut=bytes.windows("猫".len()).position(|w|w=="猫".as_bytes()).unwrap()+1; + std::fs::OpenOptions::new().write(true).open(&old).unwrap().set_len(cut as u64).unwrap(); + let (recovered,offset)=oplog::read_oplog(&old,0,None).unwrap();assert_eq!(recovered.len(),1);assert_eq!(offset,complete_offset); + let mut engine=SyncEngine::new(make_sync_config(&sync_dir,"utf8-writer"),cortex_a.sqlite_storage()).unwrap(); + engine.record_op(SyncPayload::MemoryUpsert {memory:next.clone()}).unwrap();drop(engine); + let cortex_b=Cortex::in_memory().unwrap();let mut receiver=SyncEngine::new(make_sync_config(&sync_dir,"utf8-reader"),cortex_b.sqlite_storage()).unwrap(); + assert_eq!(receiver.pull_remote(cortex_b.sqlite_storage(),cortex_b.index()).unwrap(),2); + assert!(cortex_b.storage().get_memory(first.id).unwrap().is_some());assert!(cortex_b.storage().get_memory(next.id).unwrap().is_some());assert!(cortex_b.storage().get_memory(torn.id).unwrap().is_none()); + assert_eq!(receiver.pull_remote(cortex_b.sqlite_storage(),cortex_b.index()).unwrap(),0); + assert_eq!(oplog::read_oplog(&old,complete_offset,None).unwrap().1,complete_offset); +} + +#[test] +fn test_oplog_completed_invalid_utf8_is_rejected_without_blocking_later_records() { + let tmp=TempDir::new().unwrap();let path=tmp.path().join("corrupt.jsonl"); + let make=||SyncOp {op_id:Uuid::new_v4(),hlc:HlcTimestamp::new(1000,0,"d"),payload:SyncPayload::MemoryDelete {id:Uuid::new_v4()},hmac:None}; + let first=make();let next=make();let mut bytes=serde_json::to_vec(&first).unwrap();bytes.extend_from_slice(b"\n\xff\n");bytes.extend_from_slice(&serde_json::to_vec(&next).unwrap());bytes.push(b'\n');std::fs::write(&path,&bytes).unwrap(); + let (ops,cursor)=oplog::read_oplog(&path,0,None).unwrap();assert_eq!(ops.len(),2);assert_eq!(ops[0].op_id,first.op_id);assert_eq!(ops[1].op_id,next.op_id);assert_eq!(cursor,bytes.len() as u64); +} diff --git a/docs/controlled-memory-api.md b/docs/controlled-memory-api.md index b767d55..2f82356 100644 --- a/docs/controlled-memory-api.md +++ b/docs/controlled-memory-api.md @@ -24,7 +24,7 @@ SHA and artifact checksum with the host's existing dependency manifest. ## Store and lifecycle -`ControlledStore(root, owner_type, owner_id, bot_id, replica_id)` opens a dedicated owner/bot/replica database under an absolute, private root. Paths inside that root are generated from the identity. The root must not be a symlink or contain `..`; resolve platform temporary-directory aliases before passing it. Existing ancestors must be owned by the service user or root and must not be group/world-writable unless sticky, preventing an untrusted local account from replacing the validated root. New root components are created individually and checked without following symlinks, including a name precreated under a shared sticky parent. The root itself is always owner-only. One process/handle may own a replica at a time. Handles inherited through `fork` raise `conflict` before touching copied locks/SQLite. After any Cortex store initialization, children must exec/spawn before opening stores; a pure-fork child cannot initialize another store. Use a separate replica for each spawned worker. Process-owned locks allow the parent to close and reopen its replica while an unused fork child remains alive. Different replicas have separate generation IDs and cursors. A personal database is never imported. Current legacy open/export paths refuse controlled databases. Older binaries must not be given these files. +`ControlledStore(root, owner_type, owner_id, bot_id, replica_id)` opens a dedicated owner/bot/replica database under an absolute, private root. Paths inside that root are generated from the identity. The root must not be a symlink or contain `..`; resolve platform temporary-directory aliases before passing it. Existing ancestors must be owned by the service user or root and must not be group/world-writable unless sticky, preventing an untrusted local account from replacing the validated root. New root components are created individually and checked without following symlinks, including a name precreated under a shared sticky parent. The root itself is always owner-only. One process/handle may own a replica at a time. Handles inherited through `fork` raise `conflict` before touching copied locks/SQLite. After any Cortex store initialization, children must exec/spawn before opening stores; a pure-fork child cannot initialize another store. Use a separate replica for each spawned worker. Process-owned locks allow the parent to close and reopen its replica while an unused fork child remains alive. Different replicas have separate generation IDs and cursors. A personal database is never imported. A SQLite write transaction atomically claims personal or controlled mode before initialization; the persistent marker survives an initialization failure. Current legacy open/export paths refuse controlled databases. Older binaries must not be given these files. - `status()` returns `{generation, position, sealed, ready, cleanup_pending}`. Every reopen starts not ready. - `seed(projections)` atomically loads a batch of current authoritative memories while the snapshot is unsealed. Retrying identical revisions is safe. No seed after snapshot completion.