Skip to content

Proposal: keep the newest feed items when a news category fills - #177

Open
mishan wants to merge 1 commit into
jhalter:masterfrom
mishan:feed-keep-newest-items
Open

mishan wants to merge 1 commit into
jhalter:masterfrom
mishan:feed-keep-newest-items

Conversation

@mishan

@mishan mishan commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

I'd like to propose a change to what happens when a feed category reaches the 65,535-byte article-list limit. The current behavior is documented and deliberate, so I've put this up as a proposal. I'm happy to rework it, or drop it if you'd prefer the current design.

What happens today. Items are imported oldest first, and the import stops at the limit. For a long-lived feed, such as a GitHub releases feed, that means:

  • once the category fills, new releases never appear, and the category keeps showing the oldest ones;
  • validators aren't saved while unseen items remain, so every later load downloads and parses the whole feed to import nothing.

The docs suggest splitting a large source across categories, but a single feed can't be split, and the category stops updating without anyone noticing.

What this PR does instead. When an addition wouldn't fit, the oldest imported articles are removed until it does:

  • removed items stay recorded as seen, so they aren't imported again;
  • local posts, and imported articles that have replies, are never removed;
  • if those alone leave no room, the import stops as it does today.

A prerequisite. Pruning relies on the feed state's IDs naming imported articles, so new articles, local posts included, are now numbered above every ID the feed has used. Without that, deleting the newest article let the next post reuse its ID, and pruning could then remove a user's post.

Alternatives I considered and would be glad to switch to:

  • a per-feed MaxArticles setting instead of pruning only at the limit;
  • pruning only when the operator opts in.

Tests:

  • the existing limit test, now asserting the newest item is kept and validators advance;
  • local posts and a replied-to import surviving a prune;
  • a local post not reusing a deleted import's ID.

Feed items are imported oldest first, and once the category's article
list reached the 65,535-byte protocol limit the import stopped. Nothing
was pruned, so every newer item was dropped from then on, and because
validators are not saved while unseen items remain, every later load
re-downloaded and re-parsed the whole feed to import nothing.

When an addition would not fit, remove the oldest imported articles until
it does. Removed items stay in the feed state as seen, so they are not
imported again. Locally posted articles and imported articles with
replies are never removed; if they alone leave no room, the import stops
as before.

Pruning relies on the feed state's article IDs naming imported
articles, so new articles, local posts included, are now numbered above
every ID the feed has imported under. Before, deleting the newest
article let the next post reuse its ID.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant