Skip to content

Core, ORC: surface struct-level null counts for content stats - #17844

Open
bujjibabukatta wants to merge 1 commit into
apache:mainfrom
bujjibabukatta:fix/#17463
Open

Core, ORC: surface struct-level null counts for content stats#17844
bujjibabukatta wants to merge 1 commit into
apache:mainfrom
bujjibabukatta:fix/#17463

Conversation

@bujjibabukatta

Copy link
Copy Markdown

Summary

Proposes one direction for #17463 (open investigation issue, no
consensus yet): let ORC surface struct-level null counts it already
computes, and let V4 content stats store them.

Root Cause

  • ORC: ColumnStatistics[] includes real null-count data for
    struct columns, but StatsColumnsVisitor.record() only ever added
    children's field IDs — the struct's own ID was discarded.
  • V4 stats: fieldStatsStruct() excluded all nested types
    (struct, map, list) uniformly via isNestedType(), so structs had
    no way to store counts even when a writer could supply them.

Read side is unaffected — InclusiveMetricsEvaluator already does a
generic field-ID lookup with no leaf/struct distinction.

Fix

  • OrcMetrics.javaStatsColumnsVisitor.record() now also adds
    the struct's own field ID via ORCSchemaUtil.icebergID(record).
    Bounds unaffected (struct ColumnStatistics matches no
    instanceof branch in fromOrcMin/fromOrcMax).
  • StatsUtil.javafieldStatsStruct() returns a reduced
    value_count/null_value_count-only struct for StructType
    instead of null. Map/list still excluded as before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant