Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
377 changes: 377 additions & 0 deletions content/blog/2026-11-13-topk-per-partition.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,377 @@
---
layout: post
title: Top N Per Group with PartitionedTopKExec in Apache DataFusion
date: 2026-11-13
author: Vadim Lazuk
categories: [performance]
---

<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->

[TOC]

*[Vadim Lazuk](https://github.com/Inf1mum)*

"Find the top N rows within each group" is one of the most common shapes in
analytics: the five best-selling products in each category, the three most
recent events per device, the two highest-revenue rows per store. In SQL this
is usually written with a ranking window function and an outer filter:

```sql
SELECT * FROM (
SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, *
FROM sales
) WHERE rn <= 5;
```

Until recently, DataFusion evaluated this query by sorting the *entire* input
to compute the window function, and then throwing almost all of the sorted
output away. DataFusion 55 recognizes the pattern: with
[datafusion.optimizer.enable_window_topn] enabled, the full sort is replaced
by a new `PartitionedTopKExec` operator that keeps only the top N rows per
partition in a bounded heap.

The rest of this post shows how to enable the feature, what changes in the
plan, how the operator works, and where it helps and where it does not.

[@alamb]: https://github.com/alamb
[@SubhamSinghal]: https://github.com/SubhamSinghal
[datafusion.optimizer.enable_window_topn]: https://datafusion.apache.org/user-guide/configs.html
[#6899]: https://github.com/apache/datafusion/issues/6899

## The "Top N Per Group" Pattern

The query above computes the top five rows per `category`. The same shape
covers the latest events per device, the highest-paid employees per
department, and many other screens and reports. It is also query 8 of the
[h2o group-by benchmark] ("top 2 rows by value per group"), which DataFusion
tracks as part of its effort to stay competitive on [db-benchmark][#13586].

[h2o group-by benchmark]: https://github.com/h2oai/db-benchmark
[#13586]: https://github.com/apache/datafusion/issues/13586

What matters for the optimizer is the ratio: the filter keeps a *small, fixed
number of rows per group* (`rn <= K`), while the input per group is large.
When that holds, almost all the work of a sort-based plan is discarded.

## The Default Plan Sorts Everything

A ranking window function must see each partition's rows as one contiguous,
ordered run. The straightforward way to guarantee that is to sort the whole
input by `(partition keys, order keys)` and evaluate the window over the
sorted stream. With the optimization disabled, the physical plan for the
query above is (abridged):

```text
ProjectionExec
FilterExec: rn <= 5
BoundedWindowAggExec: ROW_NUMBER() PARTITION BY [category] ORDER BY [revenue DESC]
SortExec: expr=[category ASC, revenue DESC]
DataSourceExec
```

The plan is correct, but every stage does work on rows the filter later
throws away:

1. **`SortExec` sorts the entire dataset by `(category, revenue)`.** The
window function needs each partition's rows contiguous and in `ORDER BY`
order, and a full sort is the only way to guarantee that here. It is a
blocking, spill-capable sort of all N rows: O(N log N) time, O(N) memory,
with excess rows written to disk and merged back once the memory pool
runs out.

2. **`BoundedWindowAggExec` computes `ROW_NUMBER` for every row.** It walks
the sorted stream and numbers each partition's rows 1, 2, 3, ... It holds
state only for the partition currently open, hence "bounded", but it
still touches all N rows, including the ones about to be discarded.

3. **`FilterExec` keeps only the rows with `rn <= 5`.** Everything the sort
and the window did for the other rows is wasted here.

With 10 million rows, 1,000 partitions, and `K = 3`, the plan sorts all 10
million rows to keep 3,000 — 0.03% of the input. The surviving fraction is
roughly `K × P / N`, and it shrinks toward zero as the partition key's
cardinality grows.

## How It Works

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put this into beginning

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


`PartitionedTopKExec` reads *unsorted* input and keeps one bounded top-K heap
per distinct partition key, instead of sorting the dataset. Rows are
converted to Arrow's row format with a single shared `RowConverter` and
grouped by their partition key. Each group's rows flow into that group's
heap, which holds the best K rows according to the `ORDER BY` and evicts the
rest as new rows arrive. All heaps share a single `MemoryReservation`, so the
operator reports its memory usage to DataFusion's memory pool as one unit
[#21479], [#23096].

The overall cost changes from O(N log N) time and O(N) memory to:

- **O(N log K)** time — each row is compared against a heap of size K.
- **O(K × P × row_size)** memory — K retained rows for each of the P
distinct partitions.

For example, with `fetch=3` and input:

```text
store | revenue
------|--------
A | 100
B | 50
A | 200
B | 150
A | 300
A | 400
```

the operator keeps two heaps: `store=A` retains `{400, 300, 200}` and evicts
`100`; `store=B` retains `{150, 50}`. It emits the retained rows in
`(partition_keys, order_keys)` order — `A/400, A/300, A/200, B/150, B/50` —
which is exactly the contiguous ordered run the window operator needs to
assign `ROW_NUMBER` values 1..3 per partition.

### What Each Ranking Function Has to Retain

The three ranking functions differ in what a correct implementation must keep
per partition:

- `ROW_NUMBER` numbers rows 1, 2, 3, ... with no ties, so exactly K rows
per partition suffice.
- `RANK` gives tied rows the same rank and skips subsequent ranks, so
`WHERE rk <= K` must keep K rows *plus any additional rows tied with the
row at the boundary*. The operator therefore pairs its heap with a sibling
ties list, and a query can legitimately return more than K rows per
partition when ties straddle the boundary — that is what `RANK` means.
- `DENSE_RANK` gives tied rows the same rank without gaps, which requires
tracking distinct order-by values rather than just rows.

### Emission Order and the Window Operator

Window functions do not require the *partition keys* to be globally sorted —
only each partition's rows must be contiguous and ordered among themselves.
`PartitionedTopKExec` emits rows grouped by partition and ordered by the
`ORDER BY` keys within each partition, which is the weaker guarantee the
window operator needs, and avoids paying for a global sort of the partition
keys. The `BoundedWindowAggExec` above the operator then runs in `Sorted`
mode: it only walks the retained rows and writes the rank column.

Work on the main branch since DataFusion 55 goes further: [#26133] emits the
rank column directly from `PartitionedTopKExec`, allowing the window operator
to be removed from the plan entirely.

[#21479]: https://github.com/apache/datafusion/pull/21479
[#23096]: https://github.com/apache/datafusion/pull/23096
[#26133]: https://github.com/apache/datafusion/pull/26133

## Using It in DataFusion 55

The optimization ships in DataFusion 55.0.0 behind the
`datafusion.optimizer.enable_window_topn` configuration flag, which is
**disabled by default**. Enable it with:

```sql
SET datafusion.optimizer.enable_window_topn = true;
```

With the flag on, the same query produces (abridged):

```text
ProjectionExec
BoundedWindowAggExec: ROW_NUMBER() PARTITION BY [category] ORDER BY [revenue DESC]
RepartitionExec: partitioning=Hash([category], ...)
PartitionedTopKExec: fn=row_number, fetch=5, partition=[category], order=[revenue DESC]
DataSourceExec
```

The `SortExec` and the `FilterExec` are gone. `PartitionedTopKExec` sits
directly below the window operator and feeds it only the surviving rows,
already grouped and ordered.

### Which Queries the Rewrite Applies To

The physical optimizer rule behind this is called `WindowTopN`. It fires when
all of the following hold:

- The window function is `ROW_NUMBER` or `RANK` (in DataFusion 55; see
[Status and Roadmap](#status-and-roadmap) for `DENSE_RANK`).
- The window has a `PARTITION BY` clause. A global top-N without partitioning
is already handled efficiently by `SortExec` with a fetch limit.
- For `RANK`: the window has a non-empty `ORDER BY` clause (with an empty
`ORDER BY`, every row ties at rank 1 and there is nothing to prune).
- The filter compares the ranking column to an integer literal using one of
`rn <= K`, `rn < K`, `K >= rn`, or `K > rn`.

The rule derives the number of rows to retain from the predicate: `rn <= 5`
retains 5 rows per partition, and `rn < 5` retains 4.

If any of these conditions fail, the rule does not fire and the sort-based
plan above is used. The flag switches only this rewrite on and off; it
affects nothing else. To disable it again:

```sql
SET datafusion.optimizer.enable_window_topn = false;
```

## Benchmarks: Where It Helps, and Where It Does Not

When the operator was introduced, it was measured on the h2o group-by shape —
10 million rows, top-2 rows per partition — across partition counts from 100
to 100,000 [#21479]. The full input is processed either way; only the sort is
replaced by the heaps.

| Scenario | Enabled | Disabled | Speedup |
| --------------------------------- | ------: | -------: | ----------- |
| 100 partitions (100K rows each) | 43 ms | 174 ms | **4.0x** |
| 1K partitions (10K rows each) | 71 ms | 146 ms | **2.1x** |
| 10K partitions (1K rows each) | 619 ms | 128 ms | **0.2x** (regression) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be good to come up with golden partition num number and also the reason why regression happens

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Measured this on a released build (55.2.0, exact h2o group-by Q8 shape, 10M rows, K=2, min of 4 runs) — two things turned up:

  • The published table predates #23096 (shared encoder/reservation), which is already in the 55.0.0 changelog — so the 10K-partition regression doesn't reproduce on released 55.x: the rewrite is ~5.6x faster there. The old numbers are real for the #21479-era build, so the post keeps them labeled "as measured in #21479" and adds a fresh sweep.
  • On 55.2.0 the crossover sits between ~50K and ~60K partitions — roughly 150–200 rows per partition at K=2. The reason the regression happens beyond that: the enabled-plan cost grows linearly with P (~5–6 µs per distinct partition key — a hash probe into P heap states per row, one heap + hash-map entry per key, an O(P log P) emit sort), while the sort it replaces stays roughly flat in P. And unlike SortExec, PartitionedTopKExec has no spill path (memory is O(K×P×row_size)), so it can't degrade gracefully the way the sort does. The window operator on top adds its own ~4 µs/partition on top of that.

Reworking the section now with the fresh table and fixing the "follow-up work on main" sentence.

| 100K partitions (100 rows each) | 4368 ms | 135 ms | **0.03x** (regression) |

*Timings reported in [#21479] when the operator was introduced — on a build
that predates [#23096], which shares the row encoder and memory reservation
across partitions and is included in the 55.0.0 release. Hardware and
absolute numbers will vary, but the shape of the curve is the point.*

### Fresh numbers on a released build

I re-ran the same shape on DataFusion **55.2.0**: the exact h2o group-by Q8
query, 10M rows, `K = 2`, in-memory data, min of 4 runs on a single desktop
machine. Compare the *shape*, not the absolutes, with the table above:

| Partitions | Rows/partition | Enabled | Disabled | Enabled/Disabled |
| ---------: | -------------: | -------: | --------: | ---------------- |
| 100 | 100,000 | 79 ms | 342 ms | 0.23x (4.3x faster) |
| 1,000 | 10,000 | 92 ms | 610 ms | 0.15x (6.6x faster) |
| 5,000 | 2,000 | 124 ms | 848 ms | 0.15x (6.8x faster) |
| 10,000 | 1,000 | 159 ms | 885 ms | 0.18x (5.6x faster) |
| 20,000 | 500 | 205 ms | 791 ms | 0.26x (3.9x faster) |
| 50,000 | 200 | 416 ms | 500 ms | 0.83x (≈ parity) |
| 60,000 | ~167 | 454 ms | 289 ms | 1.57x (regression) |
| 100,000 | 100 | 699 ms | 530 ms | 1.32x (regression) |
| 150,000 | ~67 | 944 ms | 566 ms | 1.67x (regression) |

The crossover sits between ~50K and ~60K partitions — roughly **150–200 rows
per partition** at `K = 2`. Past that the rewrite loses ground for two
reasons:

- **Per-partition overhead grows linearly with P.** Every row probes the
hash map of P partition states; every distinct key owns a heap and a
hash-map entry; the emit step sorts all P keys once, O(P log P). The
enabled plan costs roughly 5–6 µs per distinct partition key, while the
sort baseline stays roughly flat in P. The regression starts where the two
curves cross.
- **`PartitionedTopKExec` has no spill path.** SortExec degrades gracefully:
excess data goes to disk. The heaps must hold O(K × P × row_size) in
memory, and when the memory reservation cannot grow, the query fails
rather than spills. The window operator on top pays ~4 µs per partition
(417 µs at P=100 → 418 ms at P=100K) even though it only touches the 2×P
surviving rows.

Per-partition bookkeeping grows linearly with the number of distinct keys,
while the sort it replaces costs O(N log N) regardless of P. The rewrite
wins while each partition still holds enough rows for the heaps to pay for
themselves, and loses when they don't. That is why the feature is behind a
flag rather than always-on; the [configuration documentation][configs-doc]
reaches the same conclusion: the optimization helps with low-cardinality
window partition keys, while high-cardinality keys can regress both memory
usage and runtime.

[configs-doc]: https://datafusion.apache.org/user-guide/configs.html

Both follow-up optimizations from the [#21479] era have since shipped:
[#23096] (shared encoder and memory reservation) is part of the 55.0.0
release, and [#24591] reduced `RANK` memory from O(input) to O(K + ties).

[#24591]: https://github.com/apache/datafusion/pull/24591

## Status and Roadmap

As of DataFusion 55.0.0:

- `ROW_NUMBER` and `RANK` are supported, opt-in via
`datafusion.optimizer.enable_window_topn`.
- `DENSE_RANK` support has landed on the main branch
([#23869]) and is expected in an upcoming release.
- The optimization is **not yet enabled by default**. Work to flip the
default is tracked in the [enable-by-default EPIC][#24608]
([#27197] is the change itself, open at the time of writing).

[#23869]: https://github.com/apache/datafusion/pull/23869
[#24608]: https://github.com/apache/datafusion/issues/24608
[#27197]: https://github.com/apache/datafusion/pull/27197

## Remaining Work

There is active work to extend and harden the optimization; good places to
jump in:

- [Further "per partition" top-k optimizations][#24060] — the umbrella for
the next round of improvements.
- [Support filters with embedded projections][#21596] — widen the set of
queries the rule recognizes.
- [Memory-efficient dedup on wide payloads][#23600] — the `ROW_NUMBER() = 1`
/ `DISTINCT ON` dedup pattern.
- [Utf8View buffers kept alive by retained rows][#26381] — memory footprint
when string columns are present.

[#24060]: https://github.com/apache/datafusion/issues/24060
[#21596]: https://github.com/apache/datafusion/issues/21596
[#23600]: https://github.com/apache/datafusion/issues/23600
[#26381]: https://github.com/apache/datafusion/issues/26381

## Acknowledgements

This feature was implemented by [@SubhamSinghal] across a long series of
patches, with reviews from [@2010YOUY01] and [@kosiew]. [@jayzhan211] drove
the memory optimizations, and further contributions came from
[@saadtajwar], [@stuhood], [@viirya], and [@zhuqi-lucas]. The original
feature request was filed in 2023 ([#6899]), and the draft this post is based
on was written by [@alamb] for the DataFusion 55 announcement.

[@2010YOUY01]: https://github.com/2010YOUY01
[@kosiew]: https://github.com/kosiew
[@jayzhan211]: https://github.com/jayzhan211
[@saadtajwar]: https://github.com/saadtajwar
[@stuhood]: https://github.com/stuhood
[@viirya]: https://github.com/viirya
[@zhuqi-lucas]: https://github.com/zhuqi-lucas

## Get Involved

- **Try it out**: run your "top N per group" queries with
`datafusion.optimizer.enable_window_topn = true` on DataFusion 55 and
compare `EXPLAIN ANALYZE` output. If you hit the high-cardinality cliff,
please report it — those numbers drive the remaining optimization work.
- **Work on a [good first issue](https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)**,
or pick up one of the [remaining work](#remaining-work) items above.
- **File issues or join the conversation**: [GitHub](https://github.com/apache/datafusion/)
for bugs and feature requests, [Slack or Discord](https://datafusion.apache.org/contributor-guide/communication.html)
for discussion.
- Learn more by visiting the [DataFusion](https://datafusion.apache.org/index.html) project page.

## References

- Original feature request (2023): [#6899]
- Introducing `PartitionedTopKExec` and the `WindowTopN` rule (`ROW_NUMBER`): [#21479]
- `RANK` support: [#22885](https://github.com/apache/datafusion/pull/22885)
- `DENSE_RANK` support (main): [#23869]
- Shared encoder and memory reservation across partitions: [#23096]
- `RANK` memory from O(input) to O(K + ties): [#24591]
- Emit the rank column from the operator (main): [#26133]
- Enable-by-default EPIC: [#24608] / [#27197]
- This post's ticket: [#27233](https://github.com/apache/datafusion/issues/27233)