Skip to content

Keep PostStore records when a sibling index evicts - #57

Open
Pitchfork-and-Torch wants to merge 1 commit into
xai-org:mainfrom
Pitchfork-and-Torch:cursor/thunder-index-eviction-5421
Open

Keep PostStore records when a sibling index evicts#57
Pitchfork-and-Torch wants to merge 1 commit into
xai-org:mainfrom
Pitchfork-and-Torch:cursor/thunder-index-eviction-5421

Conversation

@Pitchfork-and-Torch

Copy link
Copy Markdown

Bug

PostStore::insert_posts_internal indexes an original video in both original_posts_by_user and video_posts_by_user. When either deque hits MAX_POSTING_LIST_SIZE (5000), the oldest id is popped and posts.remove runs unconditionally.

get_all_posts_by_users / get_videos_by_users look up CompactPost by that id. If the video list overflows first — a user who retweets a lot of video, which is the common fill path — the original is still on the original index, but the canonical record is gone. filter_map drops it. The viewer following that author no longer sees a post that is still inside the original posting list and inside retention.

This is not the same-second insert drop (#48). It is a cross-index consistency hole on capacity eviction.

Fix

Only remove from posts when the author no longer indexes the id in original, secondary, or video. Overflow of one list leaves the other list servable. An original evicted from its only index still leaves posts.

Tests

  • Video-list overflow after one original video plus 4999 video retweets: original stays in posts and is returned by get_all_posts_by_users; it is not returned by get_videos_by_users.
  • Original-only overflow still drops the evicted id from posts.

Standalone harness (same insert/get control flow, cap 4): buggy path drops the still-indexed original; fixed path serves it; single-index overflow still GC's.

thunder/ has no Cargo manifest in this snapshot, so the crate tests cannot run here.

Lane: thunder/posts/post_store.rs only. Based on xai-org/x-algorithm main (c65aa179).

Capacity eviction from video_posts_by_user (or original/secondary)
unconditionally removed the CompactPost. An original video is indexed
in both lists, so overflowing the video list with retweets dropped a
still-indexed original from serving.

Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants