Repository navigation
Add blog post: TopK per partition with enable_window_topn #219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e8f899a
a408273
972751b
5bcf8cd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,377 @@ | ||
| --- | ||
| layout: post | ||
| title: Top N Per Group with PartitionedTopKExec in Apache DataFusion | ||
| date: 2026-11-13 | ||
| author: Vadim Lazuk | ||
| categories: [performance] | ||
| --- | ||
|
|
||
| <!-- | ||
| {% comment %} | ||
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to you under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| {% endcomment %} | ||
| --> | ||
|
|
||
| [TOC] | ||
|
|
||
| *[Vadim Lazuk](https://github.com/Inf1mum)* | ||
|
|
||
| "Find the top N rows within each group" is one of the most common shapes in | ||
| analytics: the five best-selling products in each category, the three most | ||
| recent events per device, the two highest-revenue rows per store. In SQL this | ||
| is usually written with a ranking window function and an outer filter: | ||
|
|
||
| ```sql | ||
| SELECT * FROM ( | ||
| SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, * | ||
| FROM sales | ||
| ) WHERE rn <= 5; | ||
| ``` | ||
|
|
||
| Until recently, DataFusion evaluated this query by sorting the *entire* input | ||
| to compute the window function, and then throwing almost all of the sorted | ||
| output away. DataFusion 55 recognizes the pattern: with | ||
| [datafusion.optimizer.enable_window_topn] enabled, the full sort is replaced | ||
| by a new `PartitionedTopKExec` operator that keeps only the top N rows per | ||
| partition in a bounded heap. | ||
|
|
||
| The rest of this post shows how to enable the feature, what changes in the | ||
| plan, how the operator works, and where it helps and where it does not. | ||
|
|
||
| [@alamb]: https://github.com/alamb | ||
| [@SubhamSinghal]: https://github.com/SubhamSinghal | ||
| [datafusion.optimizer.enable_window_topn]: https://datafusion.apache.org/user-guide/configs.html | ||
| [#6899]: https://github.com/apache/datafusion/issues/6899 | ||
|
|
||
| ## The "Top N Per Group" Pattern | ||
|
|
||
| The query above computes the top five rows per `category`. The same shape | ||
| covers the latest events per device, the highest-paid employees per | ||
| department, and many other screens and reports. It is also query 8 of the | ||
| [h2o group-by benchmark] ("top 2 rows by value per group"), which DataFusion | ||
| tracks as part of its effort to stay competitive on [db-benchmark][#13586]. | ||
|
|
||
| [h2o group-by benchmark]: https://github.com/h2oai/db-benchmark | ||
| [#13586]: https://github.com/apache/datafusion/issues/13586 | ||
|
|
||
| What matters for the optimizer is the ratio: the filter keeps a *small, fixed | ||
| number of rows per group* (`rn <= K`), while the input per group is large. | ||
| When that holds, almost all the work of a sort-based plan is discarded. | ||
|
|
||
| ## The Default Plan Sorts Everything | ||
|
|
||
| A ranking window function must see each partition's rows as one contiguous, | ||
| ordered run. The straightforward way to guarantee that is to sort the whole | ||
| input by `(partition keys, order keys)` and evaluate the window over the | ||
| sorted stream. With the optimization disabled, the physical plan for the | ||
| query above is (abridged): | ||
|
|
||
| ```text | ||
| ProjectionExec | ||
| FilterExec: rn <= 5 | ||
| BoundedWindowAggExec: ROW_NUMBER() PARTITION BY [category] ORDER BY [revenue DESC] | ||
| SortExec: expr=[category ASC, revenue DESC] | ||
| DataSourceExec | ||
| ``` | ||
|
|
||
| The plan is correct, but every stage does work on rows the filter later | ||
| throws away: | ||
|
|
||
| 1. **`SortExec` sorts the entire dataset by `(category, revenue)`.** The | ||
| window function needs each partition's rows contiguous and in `ORDER BY` | ||
| order, and a full sort is the only way to guarantee that here. It is a | ||
| blocking, spill-capable sort of all N rows: O(N log N) time, O(N) memory, | ||
| with excess rows written to disk and merged back once the memory pool | ||
| runs out. | ||
|
|
||
| 2. **`BoundedWindowAggExec` computes `ROW_NUMBER` for every row.** It walks | ||
| the sorted stream and numbers each partition's rows 1, 2, 3, ... It holds | ||
| state only for the partition currently open, hence "bounded", but it | ||
| still touches all N rows, including the ones about to be discarded. | ||
|
|
||
| 3. **`FilterExec` keeps only the rows with `rn <= 5`.** Everything the sort | ||
| and the window did for the other rows is wasted here. | ||
|
|
||
| With 10 million rows, 1,000 partitions, and `K = 3`, the plan sorts all 10 | ||
| million rows to keep 3,000 — 0.03% of the input. The surviving fraction is | ||
| roughly `K × P / N`, and it shrinks toward zero as the partition key's | ||
| cardinality grows. | ||
|
|
||
| ## How It Works | ||
|
|
||
| `PartitionedTopKExec` reads *unsorted* input and keeps one bounded top-K heap | ||
| per distinct partition key, instead of sorting the dataset. Rows are | ||
| converted to Arrow's row format with a single shared `RowConverter` and | ||
| grouped by their partition key. Each group's rows flow into that group's | ||
| heap, which holds the best K rows according to the `ORDER BY` and evicts the | ||
| rest as new rows arrive. All heaps share a single `MemoryReservation`, so the | ||
| operator reports its memory usage to DataFusion's memory pool as one unit | ||
| [#21479], [#23096]. | ||
|
|
||
| The overall cost changes from O(N log N) time and O(N) memory to: | ||
|
|
||
| - **O(N log K)** time — each row is compared against a heap of size K. | ||
| - **O(K × P × row_size)** memory — K retained rows for each of the P | ||
| distinct partitions. | ||
|
|
||
| For example, with `fetch=3` and input: | ||
|
|
||
| ```text | ||
| store | revenue | ||
| ------|-------- | ||
| A | 100 | ||
| B | 50 | ||
| A | 200 | ||
| B | 150 | ||
| A | 300 | ||
| A | 400 | ||
| ``` | ||
|
|
||
| the operator keeps two heaps: `store=A` retains `{400, 300, 200}` and evicts | ||
| `100`; `store=B` retains `{150, 50}`. It emits the retained rows in | ||
| `(partition_keys, order_keys)` order — `A/400, A/300, A/200, B/150, B/50` — | ||
| which is exactly the contiguous ordered run the window operator needs to | ||
| assign `ROW_NUMBER` values 1..3 per partition. | ||
|
|
||
| ### What Each Ranking Function Has to Retain | ||
|
|
||
| The three ranking functions differ in what a correct implementation must keep | ||
| per partition: | ||
|
|
||
| - `ROW_NUMBER` numbers rows 1, 2, 3, ... with no ties, so exactly K rows | ||
| per partition suffice. | ||
| - `RANK` gives tied rows the same rank and skips subsequent ranks, so | ||
| `WHERE rk <= K` must keep K rows *plus any additional rows tied with the | ||
| row at the boundary*. The operator therefore pairs its heap with a sibling | ||
| ties list, and a query can legitimately return more than K rows per | ||
| partition when ties straddle the boundary — that is what `RANK` means. | ||
| - `DENSE_RANK` gives tied rows the same rank without gaps, which requires | ||
| tracking distinct order-by values rather than just rows. | ||
|
|
||
| ### Emission Order and the Window Operator | ||
|
|
||
| Window functions do not require the *partition keys* to be globally sorted — | ||
| only each partition's rows must be contiguous and ordered among themselves. | ||
| `PartitionedTopKExec` emits rows grouped by partition and ordered by the | ||
| `ORDER BY` keys within each partition, which is the weaker guarantee the | ||
| window operator needs, and avoids paying for a global sort of the partition | ||
| keys. The `BoundedWindowAggExec` above the operator then runs in `Sorted` | ||
| mode: it only walks the retained rows and writes the rank column. | ||
|
|
||
| Work on the main branch since DataFusion 55 goes further: [#26133] emits the | ||
| rank column directly from `PartitionedTopKExec`, allowing the window operator | ||
| to be removed from the plan entirely. | ||
|
|
||
| [#21479]: https://github.com/apache/datafusion/pull/21479 | ||
| [#23096]: https://github.com/apache/datafusion/pull/23096 | ||
| [#26133]: https://github.com/apache/datafusion/pull/26133 | ||
|
|
||
| ## Using It in DataFusion 55 | ||
|
|
||
| The optimization ships in DataFusion 55.0.0 behind the | ||
| `datafusion.optimizer.enable_window_topn` configuration flag, which is | ||
| **disabled by default**. Enable it with: | ||
|
|
||
| ```sql | ||
| SET datafusion.optimizer.enable_window_topn = true; | ||
| ``` | ||
|
|
||
| With the flag on, the same query produces (abridged): | ||
|
|
||
| ```text | ||
| ProjectionExec | ||
| BoundedWindowAggExec: ROW_NUMBER() PARTITION BY [category] ORDER BY [revenue DESC] | ||
| RepartitionExec: partitioning=Hash([category], ...) | ||
| PartitionedTopKExec: fn=row_number, fetch=5, partition=[category], order=[revenue DESC] | ||
| DataSourceExec | ||
| ``` | ||
|
|
||
| The `SortExec` and the `FilterExec` are gone. `PartitionedTopKExec` sits | ||
| directly below the window operator and feeds it only the surviving rows, | ||
| already grouped and ordered. | ||
|
|
||
| ### Which Queries the Rewrite Applies To | ||
|
|
||
| The physical optimizer rule behind this is called `WindowTopN`. It fires when | ||
| all of the following hold: | ||
|
|
||
| - The window function is `ROW_NUMBER` or `RANK` (in DataFusion 55; see | ||
| [Status and Roadmap](#status-and-roadmap) for `DENSE_RANK`). | ||
| - The window has a `PARTITION BY` clause. A global top-N without partitioning | ||
| is already handled efficiently by `SortExec` with a fetch limit. | ||
| - For `RANK`: the window has a non-empty `ORDER BY` clause (with an empty | ||
| `ORDER BY`, every row ties at rank 1 and there is nothing to prune). | ||
| - The filter compares the ranking column to an integer literal using one of | ||
| `rn <= K`, `rn < K`, `K >= rn`, or `K > rn`. | ||
|
|
||
| The rule derives the number of rows to retain from the predicate: `rn <= 5` | ||
| retains 5 rows per partition, and `rn < 5` retains 4. | ||
|
|
||
| If any of these conditions fail, the rule does not fire and the sort-based | ||
| plan above is used. The flag switches only this rewrite on and off; it | ||
| affects nothing else. To disable it again: | ||
|
|
||
| ```sql | ||
| SET datafusion.optimizer.enable_window_topn = false; | ||
| ``` | ||
|
|
||
| ## Benchmarks: Where It Helps, and Where It Does Not | ||
|
|
||
| When the operator was introduced, it was measured on the h2o group-by shape — | ||
| 10 million rows, top-2 rows per partition — across partition counts from 100 | ||
| to 100,000 [#21479]. The full input is processed either way; only the sort is | ||
| replaced by the heaps. | ||
|
|
||
| | Scenario | Enabled | Disabled | Speedup | | ||
| | --------------------------------- | ------: | -------: | ----------- | | ||
| | 100 partitions (100K rows each) | 43 ms | 174 ms | **4.0x** | | ||
| | 1K partitions (10K rows each) | 71 ms | 146 ms | **2.1x** | | ||
| | 10K partitions (1K rows each) | 619 ms | 128 ms | **0.2x** (regression) | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. would be good to come up with golden partition num number and also the reason why regression happens
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Measured this on a released build (55.2.0, exact h2o group-by Q8 shape, 10M rows, K=2, min of 4 runs) — two things turned up:
Reworking the section now with the fresh table and fixing the "follow-up work on main" sentence. |
||
| | 100K partitions (100 rows each) | 4368 ms | 135 ms | **0.03x** (regression) | | ||
|
|
||
| *Timings reported in [#21479] when the operator was introduced — on a build | ||
| that predates [#23096], which shares the row encoder and memory reservation | ||
| across partitions and is included in the 55.0.0 release. Hardware and | ||
| absolute numbers will vary, but the shape of the curve is the point.* | ||
|
|
||
| ### Fresh numbers on a released build | ||
|
|
||
| I re-ran the same shape on DataFusion **55.2.0**: the exact h2o group-by Q8 | ||
| query, 10M rows, `K = 2`, in-memory data, min of 4 runs on a single desktop | ||
| machine. Compare the *shape*, not the absolutes, with the table above: | ||
|
|
||
| | Partitions | Rows/partition | Enabled | Disabled | Enabled/Disabled | | ||
| | ---------: | -------------: | -------: | --------: | ---------------- | | ||
| | 100 | 100,000 | 79 ms | 342 ms | 0.23x (4.3x faster) | | ||
| | 1,000 | 10,000 | 92 ms | 610 ms | 0.15x (6.6x faster) | | ||
| | 5,000 | 2,000 | 124 ms | 848 ms | 0.15x (6.8x faster) | | ||
| | 10,000 | 1,000 | 159 ms | 885 ms | 0.18x (5.6x faster) | | ||
| | 20,000 | 500 | 205 ms | 791 ms | 0.26x (3.9x faster) | | ||
| | 50,000 | 200 | 416 ms | 500 ms | 0.83x (≈ parity) | | ||
| | 60,000 | ~167 | 454 ms | 289 ms | 1.57x (regression) | | ||
| | 100,000 | 100 | 699 ms | 530 ms | 1.32x (regression) | | ||
| | 150,000 | ~67 | 944 ms | 566 ms | 1.67x (regression) | | ||
|
|
||
| The crossover sits between ~50K and ~60K partitions — roughly **150–200 rows | ||
| per partition** at `K = 2`. Past that the rewrite loses ground for two | ||
| reasons: | ||
|
|
||
| - **Per-partition overhead grows linearly with P.** Every row probes the | ||
| hash map of P partition states; every distinct key owns a heap and a | ||
| hash-map entry; the emit step sorts all P keys once, O(P log P). The | ||
| enabled plan costs roughly 5–6 µs per distinct partition key, while the | ||
| sort baseline stays roughly flat in P. The regression starts where the two | ||
| curves cross. | ||
| - **`PartitionedTopKExec` has no spill path.** SortExec degrades gracefully: | ||
| excess data goes to disk. The heaps must hold O(K × P × row_size) in | ||
| memory, and when the memory reservation cannot grow, the query fails | ||
| rather than spills. The window operator on top pays ~4 µs per partition | ||
| (417 µs at P=100 → 418 ms at P=100K) even though it only touches the 2×P | ||
| surviving rows. | ||
|
|
||
| Per-partition bookkeeping grows linearly with the number of distinct keys, | ||
| while the sort it replaces costs O(N log N) regardless of P. The rewrite | ||
| wins while each partition still holds enough rows for the heaps to pay for | ||
| themselves, and loses when they don't. That is why the feature is behind a | ||
| flag rather than always-on; the [configuration documentation][configs-doc] | ||
| reaches the same conclusion: the optimization helps with low-cardinality | ||
| window partition keys, while high-cardinality keys can regress both memory | ||
| usage and runtime. | ||
|
|
||
| [configs-doc]: https://datafusion.apache.org/user-guide/configs.html | ||
|
|
||
| Both follow-up optimizations from the [#21479] era have since shipped: | ||
| [#23096] (shared encoder and memory reservation) is part of the 55.0.0 | ||
| release, and [#24591] reduced `RANK` memory from O(input) to O(K + ties). | ||
|
|
||
| [#24591]: https://github.com/apache/datafusion/pull/24591 | ||
|
|
||
| ## Status and Roadmap | ||
|
|
||
| As of DataFusion 55.0.0: | ||
|
|
||
| - `ROW_NUMBER` and `RANK` are supported, opt-in via | ||
| `datafusion.optimizer.enable_window_topn`. | ||
| - `DENSE_RANK` support has landed on the main branch | ||
| ([#23869]) and is expected in an upcoming release. | ||
| - The optimization is **not yet enabled by default**. Work to flip the | ||
| default is tracked in the [enable-by-default EPIC][#24608] | ||
| ([#27197] is the change itself, open at the time of writing). | ||
|
|
||
| [#23869]: https://github.com/apache/datafusion/pull/23869 | ||
| [#24608]: https://github.com/apache/datafusion/issues/24608 | ||
| [#27197]: https://github.com/apache/datafusion/pull/27197 | ||
|
|
||
| ## Remaining Work | ||
|
|
||
| There is active work to extend and harden the optimization; good places to | ||
| jump in: | ||
|
|
||
| - [Further "per partition" top-k optimizations][#24060] — the umbrella for | ||
| the next round of improvements. | ||
| - [Support filters with embedded projections][#21596] — widen the set of | ||
| queries the rule recognizes. | ||
| - [Memory-efficient dedup on wide payloads][#23600] — the `ROW_NUMBER() = 1` | ||
| / `DISTINCT ON` dedup pattern. | ||
| - [Utf8View buffers kept alive by retained rows][#26381] — memory footprint | ||
| when string columns are present. | ||
|
|
||
| [#24060]: https://github.com/apache/datafusion/issues/24060 | ||
| [#21596]: https://github.com/apache/datafusion/issues/21596 | ||
| [#23600]: https://github.com/apache/datafusion/issues/23600 | ||
| [#26381]: https://github.com/apache/datafusion/issues/26381 | ||
|
|
||
| ## Acknowledgements | ||
|
|
||
| This feature was implemented by [@SubhamSinghal] across a long series of | ||
| patches, with reviews from [@2010YOUY01] and [@kosiew]. [@jayzhan211] drove | ||
| the memory optimizations, and further contributions came from | ||
| [@saadtajwar], [@stuhood], [@viirya], and [@zhuqi-lucas]. The original | ||
| feature request was filed in 2023 ([#6899]), and the draft this post is based | ||
| on was written by [@alamb] for the DataFusion 55 announcement. | ||
|
|
||
| [@2010YOUY01]: https://github.com/2010YOUY01 | ||
| [@kosiew]: https://github.com/kosiew | ||
| [@jayzhan211]: https://github.com/jayzhan211 | ||
| [@saadtajwar]: https://github.com/saadtajwar | ||
| [@stuhood]: https://github.com/stuhood | ||
| [@viirya]: https://github.com/viirya | ||
| [@zhuqi-lucas]: https://github.com/zhuqi-lucas | ||
|
|
||
| ## Get Involved | ||
|
|
||
| - **Try it out**: run your "top N per group" queries with | ||
| `datafusion.optimizer.enable_window_topn = true` on DataFusion 55 and | ||
| compare `EXPLAIN ANALYZE` output. If you hit the high-cardinality cliff, | ||
| please report it — those numbers drive the remaining optimization work. | ||
| - **Work on a [good first issue](https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)**, | ||
| or pick up one of the [remaining work](#remaining-work) items above. | ||
| - **File issues or join the conversation**: [GitHub](https://github.com/apache/datafusion/) | ||
| for bugs and feature requests, [Slack or Discord](https://datafusion.apache.org/contributor-guide/communication.html) | ||
| for discussion. | ||
| - Learn more by visiting the [DataFusion](https://datafusion.apache.org/index.html) project page. | ||
|
|
||
| ## References | ||
|
|
||
| - Original feature request (2023): [#6899] | ||
| - Introducing `PartitionedTopKExec` and the `WindowTopN` rule (`ROW_NUMBER`): [#21479] | ||
| - `RANK` support: [#22885](https://github.com/apache/datafusion/pull/22885) | ||
| - `DENSE_RANK` support (main): [#23869] | ||
| - Shared encoder and memory reservation across partitions: [#23096] | ||
| - `RANK` memory from O(input) to O(K + ties): [#24591] | ||
| - Emit the rank column from the operator (main): [#26133] | ||
| - Enable-by-default EPIC: [#24608] / [#27197] | ||
| - This post's ticket: [#27233](https://github.com/apache/datafusion/issues/27233) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would put this into beginning
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed