fix: fingerprint the fiber census page as SurrealDB stores it - #283
Open
RobertSigmundsson wants to merge 1 commit into
Open
RobertSigmundsson wants to merge 1 commit into
RobertSigmundsson wants to merge 1 commit into
Conversation
The SDK sends None as NONE and SurrealDB drops NONE-valued keys, nested ones included, so a census page fingerprinted with its empty fields never matched the page read back. Every page holding a fiber with an empty summary, last_ghost_shown_at or metadata value failed validation and merge stopped with "fiber census staged page is invalid" - on a real brain merge never ran. Encode the page without None-valued keys; decode_fiber already defaults them.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
None-valued keys (recursively) before it is fingerprinted and stored, i.e. exactly as SurrealDB stores it.Why
After upgrading to 3.12.0 the first consolidation run stopped at merge every time ("fiber census staged page is invalid"), and because a
failedprogress record blocks the other strategies, enrich, mature and semantic_link came backblocked. The census fingerprints the encoded page with itsNonefields, but the Python SDK (2.0.0, within the pinnedsurrealdb>=2.0.0,<3.0.0) sendsNoneas NONE and SurrealDB drops NONE-valued keys from the stored record, nested ones insidemetadataincluded. The page read back no longer hashes topage_fingerprint, sovalidate_pagerejects it. Almost every fiber has an emptysummaryorlast_ghost_shown_at, so merge never gets past the first page. Encoding the page the way it will come back fixes the mismatch;decode_fiberalready defaults every missing key, so nothing else changes.Test plan
pytest tests/ -m "not stress" -n autopasses locally.ruff check src/ tests/clean.mypy src/ --ignore-missing-importsclean.tests/integration/test_fiber_census_none_fields_db.py(live SurrealDB 3.2.3,SMEM_TEST_SURREALDB_URL): three fibers withsummary=Noneand aNonemetadata value, then a real merge run - fails onv3.12.0with the message above, passes with the fix.Note: a run that already failed on 3.12.0 keeps pages fingerprinted the old way and cannot resume even after the fix - its
consolidation_progressrow andconsolidation_fiber_censusrows have to be deleted. Maybe worth a line in the release notes, or a CLI way to abandon a failed run.Verified by
@RobertSigmundsson.