Add list_sum expression#8676
Draft
mhk197 wants to merge 2 commits into
Draft
Conversation
Design for a vortex.list.sum scalar function: survey of DuckDB/DataFusion/ ClickHouse/Polars semantics, reuse of the grouped-sum machinery (GroupedAccumulator), SQL NULL-on-empty semantics for engine pushdown parity, NaN handling via NumericalAggregateOpts, and version-gated DataFusion integration (array_sum absent from the pinned DF 54). Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Implements the vortex.list.sum scalar function per the design doc: sums each list's elements through the grouped aggregate machinery (GroupedAccumulator<Sum>) so widening, overflow, and NaN semantics match sum() exactly, then nulls out empty and all-null lists to follow SQL SUM semantics (matching DuckDB list_sum and DataFusion array_sum). NaN handling is controlled by NumericalAggregateOpts, defaulting to skip_nans like sum(). Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Merging this PR will improve performance by 12.41%
Performance Changes
Tip Curious why this is faster? Comment Comparing 
Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
vortex.list.sumscalar function: the sum of each row's list elements, one output row per input row — the equivalent of DuckDB'slist_sum()and DataFusion'sarray_sum(). It is the simplest member of the list-aggregate family (list_min/list_max/list_avg/…) and establishes the vtable shape, typing rule, and semantics decisions the rest of the family can reuse.A full design doc is included at
vortex-array/src/scalar_fn/fns/list_sum/design.md, covering a survey of DuckDB / DataFusion / ClickHouse / Polars semantics and the decision points. The headline decision: SQLSUMsemantics per list (null, empty, and all-null lists yield null; null elements are skipped), chosen because both engines Vortex pushes expressions down to (DuckDB, DataFusion) agree on it, and a pushed-downlist_sumreturning0where the engine returnsNULLwould silently change query results.