path query: SQLite step index (path-cli 0.18.0)

CHANGELOG.md

@@ -2,6 +2,54 @@
 
 All notable changes to the Toolpath workspace are documented here.
 
+## SQLite step index for `path query` — 2026-07-21
+
+`path query` now keeps a disposable SQLite accelerator at
+`$CONFIG_DIR/index.db`: every wrapped step as a row (exactly what the
+jaq engine consumes), with generated columns over the hot fields and a
+stat-level freshness gate per document. The cache files stay canonical
+— delete the index and the next query rebuilds it lazily; a schema
+version bump rebuilds it automatically. `TOOLPATH_QUERY_NO_INDEX=1`
+bypasses it.
+
+What it accelerates (real 97-doc / 114 MB / 46k-step cache, medians;
+"rayon" = 0.17.0's parallel scan):
+
+- **Absorbed counts** — a filter that is exactly `length` or
+  `map(select(P)) | length` with a recognized `P` becomes
+  `SELECT count(*)`: 0.90 s → **30 ms** (~30×; no step is parsed at
+  all). Recognized `P` atoms: `.dead_end` (and `| not`),
+  `.step.actor == "…"`, `.step.actor | startswith("…")`,
+  `.path.meta.source == "…"`, and `and`-conjunctions of those.
+- **Predicated scans** — a leading `map(select(P))` / `.[] | select(P)`
+  prefilters rows in SQL before parsing: `map(select(.step.actor |
+  startswith("agent:"))) | length` 0.98 s → 169 ms (~6×; 310 ms under
+  rayon alone).
+- Unrecognized/slurp queries are unchanged (they ride 0.17.0's
+  parallel scan); a predicated stream lands at parity with it
+  (row-fetch + per-row parse ≈ parallel whole-file parse).
+
+Correctness: recognition is exact (whole-predicate or nothing) and the
+prefilter feeds the *unchanged* original filter, so pruned rows are
+ones the filter's own first stage would drop — the integration suite
+asserts index-served output is byte-identical to the no-index path
+across every plan, staleness self-healing included. Content-scoped
+queries (`--kind`/`--project`/`--project-under`) bypass the index.
+
+- **`path-cli`** (0.18.0):
+  - `query/index.rs`: the store — WAL, per-thread connections,
+    `(mtime_ns, size)` stamps, write-through from `cache::write_cached`
+    (sync/import keep the index warm), purge on `p cache rm`, orphan
+    pruning on full scans, version-gated self-rebuild. One-time build
+    cost on first query: ~1.7 s for the 114 MB cache; index size ~250 MB
+    (rows + hot-field indexes).
+  - `query/plan.rs`: `RowPredicate` — conservative recognizer for the
+    leading-`select` predicate and the absorbable-count shape, with an
+    in-code `matches` mirror used when a stale doc is reparsed.
+  - `TOOLPATH_QUERY_EXPLAIN=1` now also prints the index strategy
+    (`count absorbed into SQL (dead_end=true)`, `serving fresh docs,
+    prefilter …`, or `off`).
+
 ## Parallel `path query` execution — 2026-08-06
 
 `path query` now evaluates cache documents on a thread pool. For the

CLAUDE.md

@@ -217,7 +217,7 @@ Format references for the agent on-disk formats live at `docs/agents/formats/` 
 - Interactive pickers: `p import <provider>` auto-launches a fuzzy picker when TTY and no `--session`. Backend: external `fzf` if present, else the embedded skim picker (`embedded-picker` default feature, `crates/path-cli/src/skim_picker.rs`). Multi-select produces a `Graph`; single-select a `Path`. No usable backend falls back to most-recent (with `--project`) or prints the manual recipe. `p list <provider> --format tsv` is the machine-readable surface; the trailing column carries `first_user_message`.
 - `path share` is the one-shot `p import <harness> | p export pathbase`: probes installed harnesses, aggregates sessions into one picker ranking current-directory sessions first; `--harness`/`--session`/`--project` skip the picker; pathbase flags match `p export pathbase`. When the sync manifest shows the picked session unchanged (`sync::fresh_cache_id`), it uploads the cached doc instead of re-deriving. Uploads carry the same full derivation as local projection — no egress stripping.
 - `path resume <input>` is the inverse: accepts a Pathbase URL, `owner/repo/slug` shorthand, local file, or cache id; validates a single agent-bearing `Path`; opens a harness picker (pre-selecting `path.meta.source` when installed; `--harness` skips); projects the session into the harness's on-disk layout under `-C/--cwd` (default: shell cwd) and `execvp`'s the harness's resume command (spawn-and-wait on Windows).
-- `path query` plans before it scans: `crates/path-cli/src/query/plan.rs` classifies the jaq filter into `PerFileStream` (element-wise, print as you go), `Decompose` (algebraic aggregation with a derived combine), or `Slurp` (the always-correct whole-array fallback). Recognition is conservative so **the planner never changes an answer**; `query/filter.rs` tests assert streamed output equals slurp byte-for-byte. Execution is parallel (`mod.rs::execute_plan`/`for_each_file`): `PerFileStream`/`Decompose` run the whole per-file pipeline (parse → wrap → filter → render/pack) on rayon workers in chunks — partials cross threads as compact JSON bytes since jaq `Val`s are `Rc`-based — while `Slurp` parallelizes only parse/wrap; output stays byte-identical to a sequential scan (ordering, warnings, error precedence), and the emscripten build stays fully sequential. `TOOLPATH_QUERY_EXPLAIN=1` prints the chosen plan; there is no user-facing flag. Caveat: a streamed top-N matches slurp's ranking, but boundary ties may resolve to different rows.
+- `path query` plans before it scans: `crates/path-cli/src/query/plan.rs` classifies the jaq filter into `PerFileStream` (element-wise, print as you go), `Decompose` (algebraic aggregation with a derived combine), or `Slurp` (the always-correct whole-array fallback). Recognition is conservative so **the planner never changes an answer**; `query/filter.rs` tests assert streamed output equals slurp byte-for-byte. Execution is parallel (`mod.rs::execute_plan`/`for_each_file`): `PerFileStream`/`Decompose` run the whole per-file pipeline (parse → wrap → filter → render/pack) on rayon workers in chunks — partials cross threads as compact JSON bytes since jaq `Val`s are `Rc`-based — while `Slurp` parallelizes only parse/wrap; output stays byte-identical to a sequential scan (ordering, warnings, error precedence), and the emscripten build stays fully sequential. A disposable SQLite step index at `$CONFIG_DIR/index.db` (`query/index.rs`; wrapped-step rows + generated hot-field columns, stat-gated per doc, write-through from `cache::write_cached`, purged by `p cache rm`, rebuilt on schema-version bump) accelerates two shapes: a filter that is exactly `length`/`map(select(P))|length` with a recognized `P` (`plan.rs::RowPredicate`: `.dead_end`, actor eq/startswith, `.path.meta.source` eq, `and`-conjunctions) collapses to `SELECT count(*)`, and a recognized leading `select` prefilters rows in SQL before parsing. Recognition is whole-predicate-or-nothing and the unchanged original filter still runs, so answers never change (integration tests assert byte-equality vs `TOOLPATH_QUERY_NO_INDEX=1`); content-scoped queries (`--kind`/`--project`/`--project-under`) bypass the index. `TOOLPATH_QUERY_EXPLAIN=1` prints the chosen plan and index strategy; there is no user-facing flag. Caveat: a streamed top-N matches slurp's ranking, but boundary ties may resolve to different rows.
 
 ### Cache sync
 

Cargo.toml

@@ -37,7 +37,7 @@ toolpath-github = { version = "0.6.0", path = "crates/toolpath-github" }
 toolpath-dot = { version = "0.5.0", path = "crates/toolpath-dot" }
 toolpath-md = { version = "0.7.0", path = "crates/toolpath-md" }
 toolpath-pi = { version = "0.6.1", path = "crates/toolpath-pi" }
-path-cli = { version = "0.17.0", path = "crates/path-cli" }
+path-cli = { version = "0.18.0", path = "crates/path-cli" }
 pathbase-client = { version = "0.2.0", path = "crates/pathbase-client" }
 
 reqwest = { version = "0.13", default-features = false, features = ["blocking", "json", "rustls"] }

crates/path-cli/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "path-cli"
-version = "0.17.0"
+version = "0.18.0"
 edition.workspace = true
 license.workspace = true
 repository = "https://github.com/empathic/toolpath"

crates/path-cli/src/cache.rs

@@ -83,6 +83,13 @@ pub(crate) fn write_cached(id: &str, doc: &Graph, force: bool) -> Result<PathBuf
         std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600))
             .with_context(|| format!("chmod 0600 {}", path.display()))?;
     }
+    // Write-through to the query step index, so sync/import keep it warm
+    // and the next query serves this doc without reparsing it. Best-effort:
+    // the index is a disposable accelerator and self-heals via its stat gate.
+    #[cfg(not(target_os = "emscripten"))]
+    if let Err(e) = crate::query::index::index_written_doc(id, doc, &path) {
+        eprintln!("warning: query index not updated for {id}: {e:#}");
+    }
     Ok(path)
 }
 

crates/path-cli/src/cmd_cache.rs

@@ -75,6 +75,10 @@ fn run_rm(id: &str) -> Result<()> {
     if let Err(e) = crate::sync::evict_cache_id(id) {
         eprintln!("warning: sync manifest not updated: {e}");
     }
+    #[cfg(not(target_os = "emscripten"))]
+    if let Err(e) = crate::query::index::purge_id(id) {
+        eprintln!("warning: query index not updated: {e}");
+    }
     eprintln!("Removed {id}");
     Ok(())
 }