Repository navigation
Conversation
Feed items are imported oldest first, and once the category's article list reached the 65,535-byte protocol limit the import stopped. Nothing was pruned, so every newer item was dropped from then on, and because validators are not saved while unseen items remain, every later load re-downloaded and re-parsed the whole feed to import nothing. When an addition would not fit, remove the oldest imported articles until it does. Removed items stay in the feed state as seen, so they are not imported again. Locally posted articles and imported articles with replies are never removed; if they alone leave no room, the import stops as before. Pruning relies on the feed state's article IDs naming imported articles, so new articles, local posts included, are now numbered above every ID the feed has imported under. Before, deleting the newest article let the next post reuse its ID.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I'd like to propose a change to what happens when a feed category reaches the 65,535-byte article-list limit. The current behavior is documented and deliberate, so I've put this up as a proposal. I'm happy to rework it, or drop it if you'd prefer the current design.
What happens today. Items are imported oldest first, and the import stops at the limit. For a long-lived feed, such as a GitHub releases feed, that means:
The docs suggest splitting a large source across categories, but a single feed can't be split, and the category stops updating without anyone noticing.
What this PR does instead. When an addition wouldn't fit, the oldest imported articles are removed until it does:
A prerequisite. Pruning relies on the feed state's IDs naming imported articles, so new articles, local posts included, are now numbered above every ID the feed has used. Without that, deleting the newest article let the next post reuse its ID, and pruning could then remove a user's post.
Alternatives I considered and would be glad to switch to:
MaxArticlessetting instead of pruning only at the limit;Tests: