Repository navigation
A daily sweep takes a decaying source over its max_items back to it, lowest scores first - #21
Merged
Merged
Conversation
…lowest scores first Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
gedaiu
force-pushed
the
feat/source-decay-sweep
branch
from
October 22, 2026 10:39
c388bdb to
25f9472
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slice 4 of 5 of RC-003 (a source may decay), on top of #20. A source that sets
decayis now kept to itsmax_items.What is in it.
decay::sources_over_capanddecay::plan_eviction: for a source overmax_items, take the2 x excessoldest live documents that are at leastmin_age_daysold, score them with Lore's score (document name as key, text length, retrievals, half-life fromdocument_salienceor the source, stale derived from the last retrieval), and pick theexcesslowest. Equal scores keep the older first.decay_sweep.rs: deletes each chosen document the way a person deletes one (so it can be restored), removes it from the full-text index, and writes onedecay.evictaudit row with no actor.decay_sweepjob per source over its cap withenqueue_unless_queued, anddispatchruns it. The job has the default priority, so it waits behind extraction.docs/design/sources.mddescribes it.Memorynow owns its key and carries a text length, andis_staleis the one staleness rule.Tests. A unit test for the selection and one for staleness; a store test builds a source with five documents (one too young, one recently read, one with a topic key) and pins which two go; a server test runs the sweep against a real database and search index and pins the deletion, the index removal, the audit row and that a second sweep does nothing.
Size. 527 lines, about 230 of them tests. Fine for the fork; to be split before it goes upstream.
Checked locally on the rebased branch:
cargo fmt --check,cargo clippy --workspace --all-targets -D warnings,cargo test --workspace(1688 passed, 0 failed).🤖 Generated with Claude Code