(Claude) Render what the cache refresh committed, and write no entry rows

CONTRIBUTING.md

@@ -647,7 +647,9 @@ The tool implements a SQLite-based caching system for performance:
   locked out is retried after the pool rather than failed. A pass compacts
   an oversized WAL before it starts and truncates the WAL when it ends. A
   refresh that fails part-way leaves `projects.refresh_pending` set, and the
-  next run refreshes that project in full. A date-filtered run refreshes the
+  next run refreshes that project in full. Only the refresh writes entry
+  rows: a run renders what its refresh committed, so lines written to a
+  session while it renders appear on the next run. A date-filtered run refreshes the
   cache as an unfiltered one would and leaves the files it wrote from its
   window marked stale, so the next unfiltered run re-renders them. Its
   refresh therefore loads whole files when they changed and the incremental

claude_code_log/cache.py

@@ -1577,13 +1577,7 @@ def load_cached_entries(self, jsonl_path: Path) -> Optional[List[TranscriptEntry
         if file_id is None:
             return None
 
-        with self._get_connection() as conn:
-            rows = conn.execute(
-                "SELECT content FROM messages WHERE file_id = ? ORDER BY timestamp NULLS LAST",
-                (file_id,),
-            ).fetchall()
-
-        return [self._deserialize_entry(row) for row in rows]
+        return self._entries_for_file(file_id, None, None)
 
     def load_cached_entries_filtered(
         self, jsonl_path: Path, from_date: Optional[str], to_date: Optional[str]
@@ -1600,28 +1594,69 @@ def load_cached_entries_filtered(
         if file_id is None:
             return None
 
-        # The renderer's reading of the window: ``filter_messages_by_date``
-        # makes the exact cut after this load, so the two must agree on
-        # what a bound means.
-        from .utils import parse_date_bounds
+        return self._entries_for_file(file_id, from_date, to_date)
 
-        from_dt, to_dt = parse_date_bounds(from_date, to_date)
+    def load_snapshot_entries(
+        self,
+        jsonl_path: Path,
+        from_date: Optional[str] = None,
+        to_date: Optional[str] = None,
+    ) -> Optional[List[TranscriptEntry]]:
+        """The entries the cache holds for ``jsonl_path``, however the file
+        has moved since they were written.
+
+        This is the render's read. A run refreshes the cache, then renders
+        what the refresh committed: a session file can grow in between,
+        and parsing it again here would put lines in the pages that the
+        session rows and aggregates do not count. Lines that land after the
+        refresh are the next refresh's. So the file's size, mtime and
+        sub-agent fingerprint are not consulted; only the content version
+        is, since rows of another version may lack what the renderer reads.
+
+        None when the cache holds no usable rows for the file: it appeared
+        after the refresh, which leaves it to the next run.
+        """
+        if self._project_id is None:
+            return None
+
+        with self._get_connection() as conn:
+            row = conn.execute(
+                "SELECT id, content_version FROM cached_files"
+                " WHERE project_id = ? AND file_name = ?",
+                (self._project_id, jsonl_path.name),
+            ).fetchone()
+
+        if row is None or row["content_version"] != content_schema_version():
+            return None
+        return self._entries_for_file(row["id"], from_date, to_date)
 
-        # Build query with SQL-based filtering
+    def _entries_for_file(
+        self, file_id: int, from_date: Optional[str], to_date: Optional[str]
+    ) -> List[TranscriptEntry]:
+        """A cached file's entries, in timestamp order, within the window if
+        one is given."""
         sql = "SELECT content FROM messages WHERE file_id = ?"
         params: List[Any] = [file_id]
 
-        if from_dt:
-            # Same format as stored timestamps (``_normalize_timestamp``)
-            from_bound = from_dt.strftime("%Y-%m-%dT%H:%M:%SZ")
-            # Include entries with NULL timestamp (like summaries) OR within date range
-            sql += " AND (timestamp IS NULL OR timestamp >= ?)"
-            params.append(from_bound)
-
-        if to_dt:
-            to_bound = to_dt.strftime("%Y-%m-%dT%H:%M:%SZ")
-            sql += " AND (timestamp IS NULL OR timestamp <= ?)"
-            params.append(to_bound)
+        if from_date or to_date:
+            # The renderer's reading of the window: ``filter_messages_by_date``
+            # makes the exact cut after this load, so the two must agree on
+            # what a bound means.
+            from .utils import parse_date_bounds
+
+            from_dt, to_dt = parse_date_bounds(from_date, to_date)
+
+            if from_dt:
+                # Same format as stored timestamps (``_normalize_timestamp``)
+                from_bound = from_dt.strftime("%Y-%m-%dT%H:%M:%SZ")
+                # Include entries with NULL timestamp (like summaries) OR within date range
+                sql += " AND (timestamp IS NULL OR timestamp >= ?)"
+                params.append(from_bound)
+
+            if to_dt:
+                to_bound = to_dt.strftime("%Y-%m-%dT%H:%M:%SZ")
+                sql += " AND (timestamp IS NULL OR timestamp <= ?)"
+                params.append(to_bound)
 
         sql += " ORDER BY timestamp NULLS LAST"
 

claude_code_log/converter.py

@@ -480,10 +480,17 @@ def load_transcript(
     _loaded_files: Optional[set[Path]] = None,
     _meta_maps: Optional[dict[Path, dict[str, str]]] = None,
     entry_store: "Optional[ParsedEntryStore]" = None,
+    snapshot: bool = False,
 ) -> list[TranscriptEntry]:
     """Load and parse JSONL transcript file, using cache if available.
 
     Args:
+        snapshot: The render's load, after this run's cache refresh: serve
+            the entries the refresh committed (``load_snapshot_entries``),
+            even when the file has grown since, and never parse the file
+            or write the cache. A file the cache has no rows for yet is
+            left to the next run. Without a cache manager it changes
+            nothing: there are no rows to disagree with.
         entry_store: Optional per-conversion store of entries this
             conversion has already materialised (``entry_store.py``).
             Consulted ahead of the cache, which it saves a
@@ -514,6 +521,15 @@ def load_transcript(
             if not silent:
                 print(f"Loading {jsonl_path} from this run's parsed entries...")
             return held
+    if snapshot and cache_manager is not None:
+        committed = cache_manager.load_snapshot_entries(jsonl_path, from_date, to_date)
+        if committed is None:
+            if not silent:
+                print(f"Skipping {jsonl_path}: not in the cache yet")
+            return []
+        if not silent:
+            print(f"Loading {jsonl_path} from cache...")
+        return committed
     # Try to load from cache first
     if cache_manager is not None:
         # Use filtered loading if date parameters are provided
@@ -1337,6 +1353,7 @@ def load_directory_transcripts(
     silent: bool = False,
     persist_sidecar: bool = False,
     sidecar_required: bool = False,
+    snapshot: bool = False,
 ) -> tuple[list[TranscriptEntry], SessionTree]:
     """Load all JSONL transcript files from a directory and combine them.
 
@@ -1352,6 +1369,7 @@ def load_directory_transcripts(
     ``sidecar_required`` makes a failed sidecar write raise on
     contention or drop the sidecar otherwise, instead of only being
     logged: the cache refresh passes it, see ``_persist_session_sidecar``.
+    ``snapshot`` is the render's load: see ``load_transcript``.
     """
     all_messages: list[TranscriptEntry] = []
 
@@ -1369,7 +1387,12 @@ def load_directory_transcripts(
     with load_batch:
         for jsonl_file in jsonl_files:
             messages = load_transcript(
-                jsonl_file, cache_manager, from_date, to_date, silent
+                jsonl_file,
+                cache_manager,
+                from_date,
+                to_date,
+                silent,
+                snapshot=snapshot,
             )
             all_messages.extend(messages)
 
@@ -1595,6 +1618,7 @@ def _load_stale_session_transcripts(
         trunk_files,
         silent,
         entry_store=entry_store,
+        snapshot=True,
     )
 
 
@@ -1651,6 +1675,7 @@ def _load_sessions_partial(
     reelect_uuids: Optional[set[str]] = None,
     native_junction_uuids: Optional[set[str]] = None,
     entry_store: "Optional[ParsedEntryStore]" = None,
+    snapshot: bool = False,
 ) -> tuple[list[TranscriptEntry], SessionTree]:
     """Load the given trunk files into a faithful partial (entries, tree).
 
@@ -1694,6 +1719,7 @@ def _load_sessions_partial(
                     None,
                     silent,
                     entry_store=entry_store,
+                    snapshot=snapshot,
                 )
             )
 
@@ -3411,7 +3437,7 @@ def _stream_paginated_conversion(
             if {f.name for f in trunk_files} != needed_files:
                 return None
             page_entries, page_tree = _load_sessions_partial(
-                input_path, cache_manager, sidecar, trunk_files, silent
+                input_path, cache_manager, sidecar, trunk_files, silent, snapshot=True
             )
             page_entries = deduplicate_messages(page_entries)
 
@@ -4046,6 +4072,7 @@ def convert_jsonl_to(
             # A full unfiltered load is the moment the cross-session
             # sidecar can be (re)derived for the session-scoped path.
             persist_sidecar=(from_date is None and to_date is None),
+            snapshot=True,
         )
 
         # Get working directories from cache
@@ -5449,8 +5476,10 @@ def generate_single_session_file(
     # Ensure fresh cache
     ensure_fresh_cache(input_path, cache_manager, silent=True)
 
-    # Load messages from JSONL files
-    messages, _session_tree = load_directory_transcripts(input_path, cache_manager)
+    # Render what the refresh committed (see ``load_transcript``).
+    messages, _session_tree = load_directory_transcripts(
+        input_path, cache_manager, snapshot=True
+    )
 
     # Collect all known session IDs: from loaded messages + cache metadata
     all_session_ids: set[str] = {

claude_code_log/tui.py

@@ -1856,9 +1856,12 @@ def _ensure_session_file(
                 # Load from cache for archived sessions
                 messages = self.cache_manager.load_session_entries(session_id)
             else:
-                # Load from JSONL files for current sessions
+                # Refresh first, then render what the refresh committed
+                # (see ``load_transcript``): the cache may date from when
+                # the TUI started, and a session is still being written.
+                ensure_fresh_cache(self.project_path, self.cache_manager, silent=True)
                 messages, _tree = load_directory_transcripts(
-                    self.project_path, self.cache_manager, silent=True
+                    self.project_path, self.cache_manager, silent=True, snapshot=True
                 )
             if not messages:
                 return None

dev-docs/application_model.md

@@ -281,6 +281,10 @@ HTML exports honour `--theme` / `$CLAUDE_CODE_LOG_THEME` (§ 2.1):
 `html_theme` (not `theme`, which is Textual's own UI theme), and
 `_ensure_session_file` renders with it — regenerating a session page
 left in the other theme, under the same `session-{id}.html` name.
+Before it renders a live session, `_ensure_session_file` refreshes the
+cache (`ensure_fresh_cache`), then renders what that refresh committed
+(§ 2.3): the cache the TUI read at start may predate the session's
+newest lines.
 
 The "archive" action is interesting: it moves a session's source JSONL
 out of `~/.claude/projects/` while keeping the cache row intact. The
@@ -379,6 +383,33 @@ project-wide and boolean: two processes refreshing the same project at
 once can still clear each other's, as they could already overwrite each
 other's session rows.
 
+**A run renders what its refresh committed.** Only the refresh writes
+entry rows; the render reads them and writes none. A session file can
+grow between the two, since sessions are written to while a run converts
+them. The render used to find that file's rows stale, parse it again and
+write its rows itself, stamped with the file's new size and mtime, while
+the session rows and aggregates kept the refresh's counts. The run's
+pages then showed lines its session rows did not count, and the cache
+called the file current, so only `refresh_pending` (above) made the next
+run look at it again. Now every render-side load passes `snapshot=True`
+to `load_transcript`, through `load_directory_transcripts` and
+`_load_sessions_partial`: Phase 1b (§ 2.12), the streaming page loads
+(§ 2.13), Phase 2, `generate_single_session_file` and the TUI's export.
+In that mode the loader serves `CacheManager.load_snapshot_entries`,
+the rows the cache holds at the current content version whatever the
+file's stamp says now, or a parsed-entry store hit (§ 2.16), which is
+verified against the file and so describes the same rows. It never
+parses the file and never writes the cache. A file with no rows yet
+(created after the refresh) is skipped. Lines that land after the
+refresh are the next refresh's: that run finds the file modified, as
+nothing restamped it. `--no-cache` and single-file mode have no rows and
+parse as before. The hierarchy's no-cache-data fallback and Codex's
+wholesale render load and write their rows and session rows from one
+load, so they keep doing both. The TUI refreshes right before it exports
+a session, because the cache it would otherwise render may date from its
+start. `test/test_render_snapshot.py` grows a file right after the
+refresh, on each of these paths.
+
 **Corruption is recovered by discarding, not repairing.** A damaged
 cache file is unusable and, left alone, permanently so: `apply_migration`
 records only a migration that completed, so a `CREATE INDEX` that hits a
@@ -1463,6 +1494,12 @@ states nobody sees; without the cap a long unbroken stream would never
 surface. `tick()` (poll once) is split from `run()` (wait) and the clock
 is injectable, so tests drive ticks by hand against a fake clock.
 
+A tick stamps the files before it calls the conversion, and the
+conversion renders what its own refresh committed (§ 2.3). So a line
+written while a tick renders is in none of that tick's output, and it is
+a change for the next tick, which refreshes and shows it: one tick late,
+never lost.
+
 **What makes a tick cheap** is §2.12's session-scoped path, which is
 reachable here only because `ensure_fresh_cache_detailed` reports *how*
 it refreshed (`CacheRefresh.NONE`/`INCREMENTAL`/`FULL`). Phase 1b's
@@ -1631,7 +1668,9 @@ reason it is a parameter rather than a memo:
   `(size, mtime_ns)` against the stamp captured *before* the parse. A
   stamp taken before the parse can only be older than the entries
   describe, so a file that grew mid-parse declines to the cache rather
-  than serving a list its stamp misdescribes. The stamp also carries the
+  than serving a list its stamp misdescribes. A file that grew after the
+  refresh declines too, and the render then reads the rows the refresh
+  committed (§ 2.3), not the file. The stamp also carries the
   file's **sub-agent fingerprint** (`cache.subagents_fingerprint`, which
   covers the `agent-*.jsonl` transcripts as well as the sidecars, scoped
   for an agent file as its cache row is — `PendingStamp` snapshots before