Skip to content

[core] Harden and slim FM index storage - #9450

Merged
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/fix-fm-index-format
Aug 28, 2026
Merged

[core] Harden and slim FM index storage#9450
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/fix-fm-index-format

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Purpose

  • Freeze the FM V1 checksum contract as IEEE CRC32 over stored bytes plus the compression ID.
  • Align dense-query handling with Milvus: keep selective FM locate, but decline expensive intervals so the normal data path scans source values.
  • Remove duplicated verification-value pages, their configuration, feature flag, reader/writer paths, and dead byte-pattern matcher.
  • Avoid repeated physical reads with zero or undersized caches, including block-wise null bitmap scans.
  • Cross-check container and partition feature flags, refresh the unpublished V1 golden fixture, and update the FM documentation.

Storage impact

For uncompressed or hard-to-compress text, the index no longer carries roughly the full source string bytes plus 4 bytes per row and page metadata. The golden fixture shrinks from 1,926 to 1,860 bytes for 14 source bytes across four rows.

Tests

  • mvn -pl paimon-common -DwildcardSuites=none -Dtest=FMGlobalIndexTest,FMIndexSuffixArrayTest test
  • 25 tests passed; checkstyle, spotless, and enforcer passed.

@utafrali utafrali left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR is well-structured and the test coverage for the new paths is solid. The one blocking concern is the undocumented format break: switching from CRC32C to IEEE CRC32 without incrementing VERSION makes old and new files silently incompatible, which is fixable with a single constant change and a clear error message in the footer reader.

@JingsongLi JingsongLi changed the title [core] Harden FM index storage and reads [core] Harden and slim FM index storage Aug 28, 2026
@JingsongLi
JingsongLi merged commit 052adc3 into apache:master Aug 28, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants