Support upgrading tables to format version 3#3623
Draft
moomindani wants to merge 2 commits into
Draft
Conversation
Allow upgrade_table_version(3) and initialize next-row-id to 0 on upgrade, per the spec's Row Lineage for Upgraded Tables. Persisting the upgraded metadata still requires V3 metadata serialization (apache#3551). Closes apache#3622
moomindani
force-pushed
the
moomindani/v3-upgrade
branch
from
4f3e899 to
44631e0
Compare
Support upgrading via the format-version table property in set_properties, matching TableMetadata.replaceProperties in Java: the property triggers UpgradeFormatVersionUpdate instead of being stored. Add tests for parity with TestTableMetadata and TestRowLineageMetadata: direct v1-to-v3 upgrade, same-version no-op, and property-based upgrade.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3622 (part of #1551)
Rationale for this change
Allow upgrading existing tables to format version 3:
upgrade_table_version(3)is no longer rejected, andnext-row-idis initialized to 0 on upgrade, following the spec's "Row Lineage for Upgraded Tables" (existing snapshots are left unmodified).Draft until #3551 merges: persisting the upgraded metadata requires V3 metadata serialization, which #3551 enables. Until then the updated integration test (
test_upgrade_table_version) fails at the metadata write. I'll rebase and mark this ready for review once #3551 lands.Are these changes tested?
Yes. A new unit test (
test_update_metadata_upgrade_to_v3) covers the upgrade producingTableMetadataV3withnext-row-id=0including re-validation, and thetest_upgrade_table_versionintegration test is updated to upgrade to v3 (currently blocked on #3551 as noted above).Are there any user-facing changes?
Yes: v2 tables can be upgraded to v3 via
upgrade_table_version(3)once V3 metadata serialization (#3551) is available.This pull request and its description were written by Claude Fable 5.