Skip to content

perf(sessions): skip unchanged host bookkeeping writes on idle passes - #3079

Merged
ScriptedAlchemy merged 10 commits into
masterfrom
devin/1791140000-idle-history-commits-nothing
Oct 5, 2026
Merged

ScriptedAlchemy merged 10 commits into
masterfrom
devin/1791140000-idle-history-commits-nothing

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • An idle history pass no longer commits changes to the session stores. Before this change, each pass rewrote every provider's host-coverage://<provider>/v1 row and the Kimi and Pi discovery frontiers, even when nothing had changed. Each idle project pass made 9 WAL commits, plus about the same number on user-sessions.db. The daemon runs this pass every minute according to HISTORY_IDLE_RECHECK_INTERVAL.
  • The new revise_host_record in runtime/source.rs writes a record only when its byte_offset or file_id changes. Readers check only these fields. A real change still updates mtime, allowing a record to move byte_offset backward past the monotonic cursor guard.

Motivation

Refs #2521, Refs #2658.

I measured the path again after #2777 and #2948. The capture path already reads only the objects named by a commit. In load_current_mutations, only sources with complete snapshot absence reach SourceObjectCoverageV1::Complete.

I then inspected an idle daemon with no active ingestion, using an isolated profile and project. Temporary SQL probes, which have since been removed, and parse_offsets snapshots showed which rows changed each minute. Both stores rewrote host-coverage://{claude,cursor,kimi,opencode,pi}/v1 and host-frontier://{kimi,pi}/discovery/v1, even though only mtime increased. Every commit adds WAL frames that readers and checkpoints continue to reread while the daemon is idle. #2658 identifies this idle traffic as the reason that whole-daemon bytes per message cannot be measured.

Changes

  • crates/tracedecay-sessions/src/runtime/source.rs: persist_host_provider_coverage now uses the new revise_host_record, which skips unchanged records.
  • hosts/kimi.rs and hosts/pi.rs: discovery frontier writes now use revise_host_record. Reaching the end on a frontier that is already reset no longer writes.
  • ingest/project_provider.rs and ingest/user_provider.rs: remove the Codex callers' stored_coverage != Some(coverage) guard. The shared write now performs this check for every provider.
  • crates/tracedecay-session-runtime/tests/session_store_read_cost.rs: adds idle_history_pass_commits_nothing. The test runs the production project ingest pass twice over an empty host home on a registered store. It verifies that the first pass commits and that the second appends no WAL commit frames.

Test plan

  • bash scripts/require-exact-test.sh cargo test -p tracedecay-session-runtime --features test-helpers --test session_store_read_cost -- idle_history_pass_commits_nothing --exact: 1 passed. Without the tracedecay-sessions change, it fails with left: Some(9) right: Some(0).
  • cargo test -p tracedecay-sessions --lib -- runtime::hosts runtime::ingest runtime::source: 353 passed.
  • cargo clippy -p tracedecay-sessions -p tracedecay-session-runtime --all-targets --features tracedecay-session-runtime/test-helpers -- -D warnings: clean. cargo fmt --all -- --check: clean.
  • Tested manually on a real daemon built from this branch with an isolated HOME/TRACEDECAY_DATA_DIR and an enrolled project. Before the change, the daemon rewrote coverage and frontier rows every idle minute. After the change, parse_offsets and both stores' WAL mxFrame remained byte-identical for 130 s, covering two idle passes.

Not covered: this PR does not remeasure operator-scale read volume, reported as 2 TB in 3 h in #2521. It does not change the per-message store-growth target in #2658.

Checklist

  • CHANGELOG.md: not updated; release-please generates it from commits
  • No secrets, credentials, or .env files included
  • Breaking changes documented (none)

Link to Devin session: https://app.devin.ai/sessions/868821e98d0743dc83648344702f3479
Open in Devin Desktop: https://app.devin.ai/desktop/session/868821e98d0743dc83648344702f3479?variant=devin
Requested by: @ScriptedAlchemy


Devin Review

The daemon reruns the history pass every idle minute. Each pass rewrote every provider's host-coverage row and the Kimi and Pi discovery frontiers with a bumped mtime even when nothing changed: nine commits per idle project pass, plus the same on the user store, each adding WAL frames that readers and checkpoints reread.

revise_host_record now writes such a record only when its byte_offset or file_id changes; mtime stays the revision that lets a changed record move backwards past the monotonic cursor guard. The Codex callers' own unchanged-coverage guard is gone, since the shared write owns it.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@changeset-bot

changeset-bot Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 848a48f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@ScriptedAlchemy
ScriptedAlchemy merged commit b9d82b3 into master Oct 5, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant