[core] Validate global-index schema compatibility before reader and coverage - #9402
[core] Validate global-index schema compatibility before reader and coverage#9402QuakeWang wants to merge 1 commit into
Conversation
…overage Global indexes are serialized with the indexed field types from their build schema, while readers use the current table schema. Reusing an incompatible index can miss matches, and counting it in coverage can skip the required data scan. Persist the build schema ID in global-index metadata and compare indexed field types before reader grouping and coverage. Fail closed for legacy metadata and preserve the field across serializers and row-id reassignment. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
We should introduce schemaId to IndexManifestEntry, just like DataFileMeta.
|
Agree. I will persist the nullable schema ID on IndexManifestEntry and perform compatibility filtering on manifest entries. To preserve the build-time identity across long-running Flink/Spark index builds, I plan to carry schemaId in IndexFileMeta only as committable transport, copy it to the top-level IndexManifestEntry field in ManifestEntryChanges, and restore it to the transport when an entry is read for delete, copy, or recommit flows. The index manifest will persist it only on IndexManifestEntry, not inside GlobalIndexMeta. Legacy entries will remain null and fail closed. Does this match your intent? |
I think it is OK. |
2 participants
Purpose
Global indexes are written with serializers derived from the build schema, while readers use the current table schema. After an indexed field type evolves, reusing an old index can miss matches, and counting that index in coverage can incorrectly skip the required data scan.
This change records the build schema ID in global-index metadata and centrally compares the indexed fields' logical types before reader grouping and coverage calculation. Legacy metadata without a build schema ID fails closed, and the new field is preserved across builders, serializers, and row-ID reassignment.
Tests