Skip to content

fix: fingerprint the fiber census page as SurrealDB stores it - #283

Open
RobertSigmundsson wants to merge 1 commit into
acidkill:mainfrom
RobertSigmundsson:fix/fiber-census-none-fields
Open

RobertSigmundsson wants to merge 1 commit into
acidkill:mainfrom
RobertSigmundsson:fix/fiber-census-none-fields

Conversation

@RobertSigmundsson

Copy link
Copy Markdown
Contributor

Summary

  • Merge no longer stops with "fiber census staged page is invalid" when a fiber has an empty field.
  • The census page is encoded without None-valued keys (recursively) before it is fingerprinted and stored, i.e. exactly as SurrealDB stores it.
  • Adds a live-SurrealDB regression test and a CHANGELOG entry.

Why

After upgrading to 3.12.0 the first consolidation run stopped at merge every time ("fiber census staged page is invalid"), and because a failed progress record blocks the other strategies, enrich, mature and semantic_link came back blocked. The census fingerprints the encoded page with its None fields, but the Python SDK (2.0.0, within the pinned surrealdb>=2.0.0,<3.0.0) sends None as NONE and SurrealDB drops NONE-valued keys from the stored record, nested ones inside metadata included. The page read back no longer hashes to page_fingerprint, so validate_page rejects it. Almost every fiber has an empty summary or last_ghost_shown_at, so merge never gets past the first page. Encoding the page the way it will come back fixes the mismatch; decode_fiber already defaults every missing key, so nothing else changes.

Test plan

  • pytest tests/ -m "not stress" -n auto passes locally.
  • ruff check src/ tests/ clean.
  • mypy src/ --ignore-missing-imports clean.
  • tests/integration/test_fiber_census_none_fields_db.py (live SurrealDB 3.2.3, SMEM_TEST_SURREALDB_URL): three fibers with summary=None and a None metadata value, then a real merge run - fails on v3.12.0 with the message above, passes with the fix.
  • Full-size brain restored from a dump: merge, enrich, mature and semantic_link all complete (semantic_link resumed twice across the 600 s budget).

Note: a run that already failed on 3.12.0 keeps pages fingerprinted the old way and cannot resume even after the fix - its consolidation_progress row and consolidation_fiber_census rows have to be deleted. Maybe worth a line in the release notes, or a CLI way to abandon a failed run.

Verified by

@RobertSigmundsson.

The SDK sends None as NONE and SurrealDB drops NONE-valued keys, nested ones
included, so a census page fingerprinted with its empty fields never matched
the page read back. Every page holding a fiber with an empty summary,
last_ghost_shown_at or metadata value failed validation and merge stopped
with "fiber census staged page is invalid" - on a real brain merge never ran.

Encode the page without None-valued keys; decode_fiber already defaults them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant