Load and cache Parquet page indexes only for the columns and row groups a scan needs

[bharath-techie]

Is your feature request related to a problem or challenge?

DataFusion can skip loading the Parquet page index when row-group statistics show that page pruning cannot help. However, when page pruning is useful, it loads and decodes the complete column-index and offset-index for all the column of the row group.

For wide files, this cost is disproportionate to the query. A query filtering on 2 columns and projecting 5 out of 400 generally needs:

• Column indexes for predicate columns, for page pruning.
• Offset indexes for physical columns involved in pruning, row-selection application, and decoding.

Indexes for the remaining columns are fetched, decoded, and retained without benefiting the query.

We measured this while implementing scoped page-index loading downstream in OpenSearch:

opensearch-project/OpenSearch#22254

On a wide, one-billion-row textbench dataset:

Metadata	Memory
Complete ParquetMetaData with page indexes	~1750 MB
Footer metadata	~157 MB
Required offset indexes	~158 MB
Required column indexes	~20-30 MB

The required metadata was approximately 335–345 MB instead of 1750 MB. Exact values depend on the schema and workload.

So, raising this feature request to see if the wider community can benefit from these changes.

Describe the solution you'd like

Current behavior

The Parquet opener initially requests metadata with PageIndexPolicy::Skip and prunes row groups using footer statistics.

should_load_page_index then checks whether:

• A page-pruning predicate exists.
• At least one surviving row group is not fully matched.
• At least one predicate column has column-index and offset-index locations.

This already avoids many unnecessary page-index loads.

When indexes are required, load_page_index runs ParquetMetaDataReader with PageIndexPolicy::Optional. Arrow then fetches and decodes complete page-index matrices. DataFusion cannot request particular columns or row groups through this API.

FileMetadataCache stores one Arc<ParquetMetaData> per file. An entry is treated as either:

• Footer-only, when column_index() or offset_index() is None.
• Fully indexed, when both are Some.

The deferred load currently bypasses this cache, as described in #23978. Caching the complete index would avoid repeated I/O but could substantially increase resident memory for wide files.

Proposed behavior

After footer-based row-group pruning, derive the page-index entries required by the remaining scan:

• Load column indexes only for Parquet leaf columns referenced by page-pruning predicates.
• Load offset indexes only for physical columns needed by page pruning, row-selection application, pushed-down filtering, or decoding.
• Do not decode indexes for pruned row groups.
• Do not decode indexes for fully matched row groups unless another row selection or read requirement needs them.

The opener has the required information at this point:

• Surviving and fully matched row groups.
• Page-pruning predicate columns.
• Decoder projection.
• Columns decoded by pushed-down row filters.

This also allows row-group scoping, which is difficult to implement safely in an external reader factory because DataFusion determines the final access plan after obtaining metadata.

If an entry is absent or optional decoding fails, DataFusion should preserve its conservative behavior and scan the affected data without page pruning.

Separate fetch from decode

Selecting entries for decoding should not require issuing one object-store request per entry.

The implementation should separate:

1. The logical selection of row-group and column indexes.
2. The physical byte ranges fetched to satisfy that selection.

The appropriate strategy depends on storage characteristics. Local files can benefit from narrow reads, while remote object stores generally prefer fewer and larger requests.

For example, the OpenSearch implementation uses different strategies by storage type:

• For local storage, it fetches ranges covering only selected column chunks.
• For remote storage backed by Foyer, shard warmup stores the complete column-index and offset-index regions under exact range keys. Query-time loading requests those same complete regions, producing cache hits without remote I/O.

But decodes are done only for selected entries.

This preserves the main CPU and memory benefits even when reducing fetched bytes is not worthwhile.

A DataFusion implementation could initially use a simple per-reader policy:

  enum PageIndexFetchPolicy {
      Exact,
      Coalesce {
          max_gap: usize,
          target_size: usize,
      },
      WholeIndexRegion,
  }

The exact API is open for discussion. The important requirement is that the logical decode selection remains independent from range coalescing.

Longer term, the object-store or reader implementation could provide its preferred policy.

Scoped caching

• The current metadata cache can be extended to contain decoded page-index entries at a finer granularity than the complete file index.

• A cache entry needs to identify at least:

  • File identity, including freshness information.
  • Index type: column index or offset index.
  • Parquet physical leaf-column index.
  • Row-group index, unless entries are deliberately grouped across row groups.

Arrow support

Current arrow-rs main has no public Parquet API for decoding a selected set of page-index entries.

ParquetMetaDataReader and ParquetMetaDataPushDecoder support separate policies for column and offset indexes, but each applies to the whole file. Their parsers produce complete dense matrices.

Relevant arrow-rs issues include:

• [Parquet] How to decode part columns of ColumnIndex arrow-rs#9609: decode page indexes only for selected columns.
• Pass options to Parquet metadata readers arrow-rs#8643: metadata projection and progressive loading.
• Explore changing the form of the Parquet page indexes in ParquetMetaData arrow-rs#8818: optional or sparse page-index entries.

A clean integration likely requires changes in arrow-rs as well to support scoped page-index decoding.

Possible implementation stages

1. Add an arrow-rs API that accepts independent column-index and offset-index selections, optionally restricted to selected row groups.
2. Initially fetch the existing covering page-index range, but decode only selected entries. This provides the main CPU and memory benefit without introducing additional object-store requests.
3. In the DataFusion opener, derive the required columns and row groups after footer-statistics pruning.
4. Pass the scoped indexes to page pruning and Parquet decoding, with conservative fallback when a required index is unavailable.
5. Add bounded caching for decoded page-index entries. This could extend the existing metadata cache or use a separate cache associated with the cached footer metadata.
6. Add storage-aware range fetching so readers can choose between exact ranges, coalesced ranges, and the complete index region.
7. Add metrics and benchmarks

I am happy to contribute / help on the DataFusion implementation.
I might be missing internal details that might make some of this tricky and I'd love to hear feedback from community.

Describe alternatives you've considered

Additional context

Relevant issues

• Reduce page metadata loading to only what is necessary for query execution in ParquetOpen #16200 describes the general goal of reading page metadata only for required row groups and columns. This proposal adds independent column-index and offset-index selections, storage-aware fetching, incremental caching, and downstream implementation evidence.

Downstream custom implementation

For reference, In the downstream , OpenSearch implemented scoped loading without changing DataFusion, we implemented workaround as follows :

1. A custom ParquetFileReaderFactory returns footer metadata with selected page indexes attached.
2. A physical optimizer rule replaces the reader factory installed by ParquetFormat so the custom reader receives predicate and projection columns.
3. Because ParquetMetaData expects dense [row_group][column] matrices, unrequested column-index cells are populated with ColumnIndexMetaData::NONE.
4. Offset indexes have no equivalent missing-cell representation, requiring synthetic entries for unrequested cells - This is a hacky workaround.
5. The listing-table path cannot safely scope by row group because DataFusion determines the final row-group access plan after invoking the reader factory.
6. Selective decoding uses the deprecated read_columns_indexes and read_offset_indexes APIs available in the older Parquet 58.3 version.

Those subset decoder APIs were removed from arrow-rs in apache/arrow-rs#10035, so this workaround has no upgrade path to current Arrow.

[bharath-techie]

Tagging @alamb @adriangb @etseidl @zhuqi-lucas who might have already investigated this for #16200

[etseidl]

Thanks @bharath-techie. This is timely as I've just submitted apache/arrow-rs#10653 which will help with enabling selective population of the page indexes.

[adriangb]

Very interested in this work! Work decoding wide parquet schemas is one of our primary performance constraints.

[alamb]

I think we should likely start with the appropriate API in the arrow-rs parquet crate (MetadataPushDecoder) so we can specify what columns to read (I don't think there is any way to do that now).

Once we have the API in arrow, I think it will be straightforward to integrate it into DataFusion

I did a recent PR here to avoid reloading the page index when it wasn't present. I think we could extend the same idea

• Skip page index load (and ParquetMetaData clone) when the file has no page index #24150

[bharath-techie]

@alamb @etseidl, one challenge I faced was assembling ParquetMetaData from granularly cached page-index entries.

The metadata cache continues to serve the file and row-group metadata through a shared Arc<ParquetMetaData>. The scoped caches separately store column-index and offset-index entries. For each query, I need to combine the shared footer metadata with only the page indexes required by that query.

There were two difficulties:

1. Attaching page indexes requires an owned ParquetMetaData. Because the footer remains shared by the metadata cache, Arc::try_unwrap fails and the fallback is to clone the complete metadata. This copies
   every RowGroupMetaData and ColumnChunkMetaData just to replace the page-index fields.

2. ParquetColumnIndex and ParquetOffsetIndex were dense [row_group][column] matrices. A scoped index therefore required placeholders for unrequested entries. Column indexes had ColumnIndexMetaData::NONE, but offset indexes had no equivalent, so I had to construct conservative synthetic offset-index entries.

#10653 addresses the second problem by changing both matrices to contain Option<T>. This should remove the need for synthetic placeholders and allow missing entries to fall back conservatively.

The first problem remains. Even with optional page-index cells, combining a shared cached footer with query-specific indexes still requires cloning ParquetMetaData before calling into_builder().

I prototyped storing row_groups behind an Arc, which makes this clone share the expensive row-group and column-chunk metadata. The public API remains unchanged, and attaching indexes becomes inexpensive because only the outer metadata and page-index fields are rebuilt.

Would this be a reasonable follow-up to #10653, or would you prefer page indexes to be passed separately from the shared footer metadata so that rebuilding ParquetMetaData is unnecessary?

[alamb]

Would this be a reasonable follow-up to #10653, or would you prefer page indexes to be passed separately from the shared footer metadata so that rebuilding ParquetMetaData is unnecessary?

What if we made retrieving the indexes as a trait rahter than always directly using ParquetMetaData 🤔

[bharath-techie]

Yes thanks @alamb , I think traits will work. I tried out a quick POC [thanks to Claude ] with your suggestion,
Arrow-rs : bharath-techie/arrow-rs@10789f7#diff-65155cd086aaf6f9ca8293a91d353e85d5662eb83cf7c5a52c630ac034ba6ecf

Datafusion : 91bc27a

Please let me know if something like this might work or you are envisioning it differently

[alamb]

The page index provider trick is a neat one for Arrow -- I think that looks reasonable to me

If we are going to add it to arrow we should probably make the API a little more general (e.g. something so that we can do the same trick for on-demand bloom filters. Maybe data ranges as well 🤔 )