fix(parquet): keep mask execution off pages a RowGroups did not load - #10649
Draft
alanprot wants to merge 1 commit into
Draft
fix(parquet): keep mask execution off pages a RowGroups did not load#10649alanprot wants to merge 1 commit into
alanprot wants to merge 1 commit into
Conversation
`ParquetRecordBatchReader::try_new_with_row_groups` builds its read plan with `RowSelectionPolicy::default()` (`Auto`) and no loaded row ranges. For a fragmented selection `Auto` resolves to mask execution, and `MaskCursor` then decodes every row a chunk covers before applying the mask. A caller whose `RowGroups` holds only the pages its `RowSelection` touches - the pattern `RowSelection::scan_ranges` exists to support - therefore fails with "Invalid offset in sparse column chunk data". `ParquetPushDecoder` is unaffected because `RowGroupReaderBuilder` calls `prepare_selection_for_page_skipping`, but that is not reachable from this constructor. The regression dates to apache#8733, which introduced the mask strategy and the guard together; this constructor was never wired into it and has not been touched since apache#4307. Add `RowGroups::offset_index`, defaulting to `None`, for implementations to surface the offset index they pruned with, and derive the loaded row ranges from it inside `try_new_with_row_groups`. The derivation is conservative: a caller that loaded more pages than the selection touches still gets a subset of what it actually loaded, so confining mask execution to those ranges is always safe. `RowSelection::loaded_row_ranges` holds the derivation, and `loaded_row_ranges_for_projection` in the push decoder now delegates to it. No public types are added and no signatures change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tentative fix for #10648