fix: read table changefeeds by paging the database feed - #286
Open
RobertSigmundsson wants to merge 1 commit into
Open
RobertSigmundsson wants to merge 1 commit into
RobertSigmundsson wants to merge 1 commit into
Conversation
SurrealDB 3.2 applies LIMIT to the database-wide feed before it keeps one table's entries, so on a busy database a short or empty page of SHOW CHANGES FOR TABLE does not mean the table's history is exhausted. All three readers relied on that: the barrier search stopped early and semantic_link failed with "semantic source barrier was not found within the bounded changefeed scan"; the quick source check (LIMIT 10) missed neuron/synapse changes that sat behind other tables' writes and accepted a stale token; the paged source check stopped at an empty page. Add _show_table_changes, which pages SHOW CHANGES FOR DATABASE and keeps the table's changes, so a short result really means the end of the feed, and use it in all three places. Live tests put 10-30 other-table writes ahead of the change each reader must see; they fail on the old code and pass now.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SHOW CHANGES FOR TABLE ... LIMIT nreturning n entries of that table._show_table_changes(table, since, limit)pagesSHOW CHANGES FOR DATABASEand keeps the table's changes, so a short result really means the end of the feed. The barrier search, the quick source check and the paged source check all use it.Why
On SurrealDB 3.2.3,
LIMITinSHOW CHANGES FOR TABLEis applied to the database-wide feed before the table's entries are kept: after 200 writes to tablex, a write toyonly shows up withLIMIT 250. The three readers insemantic_source_revision.pyassume the limit counts the table's own entries. On a busy database that breaks all three: the barrier search takes a short page as the end of the feed, so semantic_link stops with "semantic source barrier was not found within the bounded changefeed scan (3 rows)"; the quick source check (LIMIT 10) misses neuron/synapse changes queued behind other tables' writes and accepts a stale token; the paged source check stops at an empty page. On a quiet database the marker sits in the first page, which is why this doesn't show in small tests.Test plan
pytest tests/ -m "not stress" -n autopasses locally.ruff check src/ tests/clean.mypy src/ --ignore-missing-importsclean.tests/integration/test_surrealdb_semantic_source_revision.py(live SurrealDB 3.2.3,SURREALDB_URL): three new tests put 10-30 other-table writes ahead of the change each reader must see. Onv3.12.0they fail ("DID NOT RAISE SemanticSourceChangedError", "semantic source barrier was not present in its changefeed"); with the fix they pass, together with the existing changefeed tests (34 passed).v3.12.0with the message above; with the fix it completes (three pauses across the 600 s budget, each resume passing the source check).The existing unit fake models per-table feeds, so it now overrides
_show_table_changes; the database-feed paging itself is covered by the live tests.Verified by
@RobertSigmundsson.