Skip to content

fix: read table changefeeds by paging the database feed - #286

Open
RobertSigmundsson wants to merge 1 commit into
acidkill:mainfrom
RobertSigmundsson:fix/changefeed-table-limit
Open

RobertSigmundsson wants to merge 1 commit into
acidkill:mainfrom
RobertSigmundsson:fix/changefeed-table-limit

Conversation

@RobertSigmundsson

Copy link
Copy Markdown
Contributor

Summary

  • Semantic-link source fences no longer rely on SHOW CHANGES FOR TABLE ... LIMIT n returning n entries of that table.
  • New _show_table_changes(table, since, limit) pages SHOW CHANGES FOR DATABASE and keeps the table's changes, so a short result really means the end of the feed. The barrier search, the quick source check and the paged source check all use it.
  • Adds three live-SurrealDB regression tests and a CHANGELOG entry.

Why

On SurrealDB 3.2.3, LIMIT in SHOW CHANGES FOR TABLE is applied to the database-wide feed before the table's entries are kept: after 200 writes to table x, a write to y only shows up with LIMIT 250. The three readers in semantic_source_revision.py assume the limit counts the table's own entries. On a busy database that breaks all three: the barrier search takes a short page as the end of the feed, so semantic_link stops with "semantic source barrier was not found within the bounded changefeed scan (3 rows)"; the quick source check (LIMIT 10) misses neuron/synapse changes queued behind other tables' writes and accepts a stale token; the paged source check stops at an empty page. On a quiet database the marker sits in the first page, which is why this doesn't show in small tests.

Test plan

  • pytest tests/ -m "not stress" -n auto passes locally.
  • ruff check src/ tests/ clean.
  • mypy src/ --ignore-missing-imports clean.
  • tests/integration/test_surrealdb_semantic_source_revision.py (live SurrealDB 3.2.3, SURREALDB_URL): three new tests put 10-30 other-table writes ahead of the change each reader must see. On v3.12.0 they fail ("DID NOT RAISE SemanticSourceChangedError", "semantic source barrier was not present in its changefeed"); with the fix they pass, together with the existing changefeed tests (34 passed).
  • A copy of a large, busy database: semantic_link fails on v3.12.0 with the message above; with the fix it completes (three pauses across the 600 s budget, each resume passing the source check).

The existing unit fake models per-table feeds, so it now overrides _show_table_changes; the database-feed paging itself is covered by the live tests.

Verified by

@RobertSigmundsson.

SurrealDB 3.2 applies LIMIT to the database-wide feed before it keeps one
table's entries, so on a busy database a short or empty page of SHOW CHANGES
FOR TABLE does not mean the table's history is exhausted. All three readers
relied on that: the barrier search stopped early and semantic_link failed
with "semantic source barrier was not found within the bounded changefeed
scan"; the quick source check (LIMIT 10) missed neuron/synapse changes that
sat behind other tables' writes and accepted a stale token; the paged source
check stopped at an empty page.

Add _show_table_changes, which pages SHOW CHANGES FOR DATABASE and keeps the
table's changes, so a short result really means the end of the feed, and use
it in all three places. Live tests put 10-30 other-table writes ahead of the
change each reader must see; they fail on the old code and pass now.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant