(Claude) Run one pass at a time per cache file

CONTRIBUTING.md

@@ -623,7 +623,12 @@ The tool implements a SQLite-based caching system for performance:
   turns through a fair cross-process queue (`write_queue.py`), so none gives
   up on a sibling. Workers write only rows: an existing search index is
   caught up once at the end of the pass, by the parent, not inside each
-  worker's write transaction. Against other processes connections wait 30 s
+  worker's write transaction. One pass at a time per cache file: a pass
+  holds a lock (`<cache>.pass-lock`, `pass_lock.py`) for its whole run,
+  catch-up included, so a second `claude-code-log` or a `serve --watch`
+  tick over the same archive waits for it instead of contending
+  (`CLAUDE_CODE_LOG_PASS_LOCK=<seconds>` bounds the wait, `=0` disables
+  the lock). Against other processes connections wait 30 s
   (`CLAUDE_CODE_LOG_CACHE_TIMEOUT` overrides, in seconds), and a project
   locked out is retried after the pool rather than failed. A pass compacts
   an oversized WAL before it starts and truncates the WAL when it ends. A

claude_code_log/converter.py

@@ -6584,14 +6584,26 @@ def process_projects_hierarchy(
     it, so a caller can tell a complete pass from one that lost projects
     (the CLI exits non-zero on the latter).
     """
+    from .pass_lock import pass_lock
     from .search import deferred_index_maintenance
 
+    # One pass at a time per cache file (see `pass_lock`): a second pass,
+    # or a `serve --watch` tick beside a CLI run, waits here instead of
+    # competing for the write lock with a queue of its own. Taken before
+    # the lease and the WAL trim, released after the index catch-up and
+    # the checkpoint below, so the waiting pass never contends with the
+    # serial catch-up and starts on a current index and a reset log.
+    lock = (
+        pass_lock(get_cache_db_path(projects_path))
+        if use_cache
+        else contextlib.nullcontext()
+    )
     lease = (
         connection_lease(get_cache_db_path(projects_path))
         if use_cache
         else contextlib.nullcontext()
     )
-    with lease:
+    with lock, lease:
         if use_cache:
             # Inside the lease, so the connection that recovers the log
             # is the one the pass then keeps.

claude_code_log/pass_lock.py

@@ -0,0 +1,215 @@
+"""One hierarchy pass per cache file.
+
+Inside a pass, the project workers share a fair write queue
+(`write_queue`), so none of them gives up on a sibling. Two *passes* over
+the same cache — a second ``claude-code-log`` run started while one is
+going, or ``serve --watch`` ticking while a CLI run refreshes the same
+archive — share nothing: each is, to the other, one tight-loop writer,
+and SQLite's lock polling is unfair to outsiders (a writer that just
+committed tends to re-acquire before a sleeper wakes), so the second pass
+can lose projects to ``database is locked`` however patient its own
+workers are. It is also the realistic way two refreshes of one project
+overlap and clear each other's ``refresh_pending``.
+
+So a pass takes the **pass lock** first: an exclusive transaction on a
+sidecar SQLite file next to the cache (``<cache>.pass-lock``), held until
+the pass — its WAL trim, its conversions, its checkpoint — is over.
+SQLite supplies the cross-platform advisory lock and releases it with
+the holder's process, so a killed pass leaves nothing stale to clean up.
+Nothing else opens that file. A second pass waits, says so after a
+moment, and then runs — usually to find the work done. The wait is
+unbounded by default; ``CLAUDE_CODE_LOG_PASS_LOCK=<seconds>`` bounds it
+(a pass that runs out of patience fails with `PassLockTimeout`), and
+``=0`` disables the lock altogether.
+
+Re-entrant within a process — process-wide, not per thread: a pass
+started while this process already holds the lock does not wait on
+itself. Nothing nests today (``serve`` finishes its startup pass before
+its watch thread starts ticking), so this is a guard against a future
+caller, not something the server relies on. The TUI's writes and a
+single-file ``convert_jsonl_to`` are short transactions outside any
+pass; they keep the ordinary busy timeout and are not gated here.
+
+The lock degrades rather than fails: when the lock file cannot be
+created (a read-only directory), is not a SQLite file (junk written
+there), or cannot be written (another user's file — a read-only
+``BEGIN EXCLUSIVE`` excludes nobody, so holding one would only pretend),
+the pass says so, names the file, and runs without the lock.
+"""
+
+from __future__ import annotations
+
+import os
+import sqlite3
+import threading
+import time
+from contextlib import contextmanager
+from pathlib import Path
+from typing import Callable, Dict, Iterator, Optional, Tuple
+
+#: ``0``/``off`` disables the pass lock; a number bounds the wait in seconds.
+ENV_PASS_LOCK = "CLAUDE_CODE_LOG_PASS_LOCK"
+
+#: Seconds of waiting before the pass says what it is waiting for.
+DEFAULT_NOTICE_AFTER = 2.0
+
+_POLL_SECONDS = 0.25
+
+
+class PassLockTimeout(RuntimeError):
+    """Another pass held the cache for longer than this one may wait."""
+
+
+def pass_lock_path(db_path: Path) -> Path:
+    """The lock file for ``db_path``: beside it, ``<name>.pass-lock``.
+
+    Keyed on the resolved path, so a cache reached through a symlink has
+    the same lock as the cache reached directly.
+    """
+    db_path = Path(db_path).resolve()
+    return db_path.with_name(db_path.name + ".pass-lock")
+
+
+def _wait_budget() -> Tuple[bool, Optional[float]]:
+    """``(enabled, seconds)``: seconds is None for an unbounded wait."""
+    raw = os.getenv(ENV_PASS_LOCK, "")
+    value = raw.strip().lower()
+    if not value:
+        # Unset, or set to nothing (`VAR= command`): the default, like every
+        # other CLAUDE_CODE_LOG_* switch.
+        return True, None
+    if value in ("off", "false", "no"):
+        return False, None
+    try:
+        seconds = float(value)
+    except ValueError:
+        raise ValueError(
+            f"{ENV_PASS_LOCK}={raw!r}: expected a number of seconds, or 0 to "
+            "disable the pass lock"
+        ) from None
+    if seconds <= 0:
+        return False, None
+    return True, seconds
+
+
+# This process's held locks: lock file -> (connection, re-entry depth).
+_holders: Dict[str, Tuple[sqlite3.Connection, int]] = {}
+_guard = threading.Lock()
+
+
+def pass_lock_held(db_path: Path) -> bool:
+    """Whether this process holds the pass lock for ``db_path``'s cache.
+
+    For tests pinning what runs under the lock; a disabled or degraded
+    lock is never "held".
+    """
+    with _guard:
+        return str(pass_lock_path(db_path)) in _holders
+
+
+def _acquire(
+    path: Path,
+    budget: Optional[float],
+    notice_after: float,
+    announce: Callable[[str], None],
+) -> sqlite3.Connection:
+    from .cache import is_busy_database_error
+
+    # `timeout=0`: SQLite's own busy handler would wait too, but silently;
+    # polling here is what lets the pass say what it is waiting for.
+    conn = sqlite3.connect(path, timeout=0, isolation_level=None)
+    started = time.monotonic()
+    announced = False
+    try:
+        while True:
+            try:
+                conn.execute("BEGIN EXCLUSIVE")
+                # A file this process cannot write opens read-only, and a
+                # read-only exclusive transaction takes no lock at all. One
+                # write inside the transaction makes that surface here, as
+                # `attempt to write a readonly database`; it is rolled back
+                # with the lock, so the file stays empty.
+                conn.execute("PRAGMA user_version = 1")
+                return conn
+            except sqlite3.OperationalError as exc:
+                if not is_busy_database_error(exc):
+                    raise
+            waited = time.monotonic() - started
+            if budget is not None and waited >= budget:
+                raise PassLockTimeout(
+                    f"another claude-code-log pass has held {path} for the "
+                    f"{budget:g} s this one was allowed to wait "
+                    f"({ENV_PASS_LOCK}); nothing was converted"
+                )
+            if not announced and waited >= notice_after:
+                announce(
+                    "another claude-code-log pass is running on this cache "
+                    f"({path}); waiting for it to finish"
+                )
+                announced = True
+            remaining = _POLL_SECONDS if budget is None else budget - waited
+            time.sleep(max(0.0, min(_POLL_SECONDS, remaining)))
+    except BaseException:
+        conn.close()
+        raise
+
+
+@contextmanager
+def pass_lock(
+    db_path: Path,
+    *,
+    notice_after: float = DEFAULT_NOTICE_AFTER,
+    announce: Callable[[str], None] = print,
+) -> Iterator[None]:
+    """Hold the pass lock for ``db_path``'s cache for the scope.
+
+    Waits for another process's pass to finish (announcing that after
+    ``notice_after`` seconds), re-enters freely within this process, and
+    is a no-op when disabled by the environment — or when the lock file
+    cannot be used (not creatable, not a database, not writable), which
+    is announced rather than allowed to fail a pass that could otherwise
+    run.
+    """
+    enabled, budget = _wait_budget()
+    if not enabled:
+        yield
+        return
+    path = pass_lock_path(db_path)
+    key = str(path)
+
+    with _guard:
+        held = _holders.get(key)
+        if held is not None:
+            _holders[key] = (held[0], held[1] + 1)
+    if held is not None:
+        try:
+            yield
+        finally:
+            with _guard:
+                conn, depth = _holders[key]
+                _holders[key] = (conn, depth - 1)
+        return
+
+    try:
+        conn = _acquire(path, budget, notice_after, announce)
+    except sqlite3.DatabaseError as exc:
+        # OperationalError (cannot create or write the file) and its parent
+        # DatabaseError (`file is not a database`): without the lock, not
+        # without the pass.
+        announce(
+            f"Warning: could not take the pass lock at {path} ({exc}); "
+            "running without it"
+        )
+        yield
+        return
+    with _guard:
+        _holders[key] = (conn, 1)
+    try:
+        yield
+    finally:
+        with _guard:
+            _holders.pop(key, None)
+        try:
+            conn.rollback()
+        finally:
+            conn.close()

dev-docs/application_model.md

@@ -215,6 +215,41 @@ in the parent) is the open lever if that worst case ever matters. A
 single file or directory conversion (`convert_jsonl_to` with no pass
 around it) still re-indexes inside the write, where nothing competes
 for the lock. `CLAUDE_CODE_LOG_SEARCH_AUTO_INDEX=0` switches both off.
+The queue is per pass, so two *passes* over one cache — a second CLI
+run, or a `serve --watch` tick while a CLI run refreshes the same
+archive — are, to each other, one tight-loop writer each, and the
+unfair polling above applies between them (it is also the realistic
+way two refreshes of one project overlap and clear each other's
+`refresh_pending`). So `process_projects_hierarchy` first takes the
+**pass lock** ([`pass_lock.py`](../claude_code_log/pass_lock.py)): an
+exclusive transaction on a sidecar SQLite file beside the cache
+(`<cache>.pass-lock`), held around the WAL trim, the conversions, the
+index catch-up and the checkpoint, and released with the holder's
+process (nothing stale to clean after a kill). A second pass waits,
+announces what it is waiting for after 2 s, then runs and usually
+finds the work done;
+`CLAUDE_CODE_LOG_PASS_LOCK=<seconds>` bounds the wait (then
+`PassLockTimeout`, nothing converted) and `=0` disables the lock.
+Re-entrant within a process (process-wide, not per thread), though
+nothing nests today: `serve` finishes its startup pass before its watch
+thread ticks. The lock is keyed on the resolved cache path, so a
+symlinked cache shares it. It degrades rather than fails: a lock file
+that cannot be created (read-only directory), is not SQLite (junk), or
+cannot be written (another user's; a read-only `BEGIN EXCLUSIVE`
+excludes nobody, which one rolled-back write inside the transaction
+makes surface) is announced by name and the pass runs without the
+lock. Outside the gate, by design: the TUI's writes and a
+single-file `convert_jsonl_to` are short transactions with the
+ordinary busy timeout, as before. `test_pass_lock.py` pins the
+waiting, the bound, the switch, the three degrade cases, the symlink,
+two concurrent passes serialising, and that the lock is held through
+the WAL trim, the index catch-up and the checkpoint
+(`pass_lock.pass_lock_held`, recorded from inside each).
+Measured on the 32 largest projects of a real archive (every file
+re-saved, two passes started together at `--jobs 8`): without the lock
+each pass took ~872 s and the pair re-saved 1.4x the files; with it the
+first ran alone in 552 s and the second, after waiting, found every
+project current in 0.2 s.
 `serve` takes the same `--jobs` as the main command. A pool-level failure (e.g. a library caller without the
 `if __name__ == "__main__"` guard `spawn` needs) degrades to inline
 sequential processing with a warning rather than aborting.

docs/live-updates.md

@@ -114,6 +114,13 @@ page notices on its next poll.
 Raise `--quiet-period` if conversions feel too frequent on a large
 project; lower `--interval` if you want changes noticed sooner.
 
+A `serve --watch` or `watch --all-projects` tick and a plain
+`claude-code-log` run over the same archive take turns rather than
+racing for the cache: whichever starts second waits for the first to
+finish (and says so after a couple of seconds), then runs — usually to
+find the work already done. `CLAUDE_CODE_LOG_PASS_LOCK=<seconds>` bounds
+that wait; `CLAUDE_CODE_LOG_PASS_LOCK=0` turns the lock off.
+
 Switching `--theme` rewrites every page in the new theme on the first
 tick — including a combined page that `--combined no` would otherwise
 leave alone, so its session pages never link back into a page of the

test/test_cache_sqlite_integrity.py

@@ -699,6 +699,12 @@ class TestConnectionCensus:
             1,
             "mode=ro reader: one SELECT across projects, then close",
         ),
+        ("pass_lock.py", "_acquire"): (
+            1,
+            "the sidecar `<cache>.pass-lock` file, not the cache database: "
+            "holds `BEGIN EXCLUSIVE` as the pass lock; its one write (a "
+            "`user_version` probe that surfaces a read-only file) is rolled back",
+        ),
     }
 
     @staticmethod