Repository navigation
Conversation
…nd robin The post follows a Parquet compaction job (read a daily partition, repartition(50), write) on a table with more than 200 columns nested up to seven levels deep. It explains why Comet's hash-based round robin made the shuffle write slow on nested data, the two experiments that followed (a faster hash and placement by batch), and the positional round robin that Comet has offered since 1.1.0. Compaction time goes from 80 minutes on Spark to 28 minutes on Comet with positional round robin. Adds three SVG figures: results, the job's stages, and row placement under the three strategies.
- Open with testing Comet on real-world use cases beyond standard benchmarks, with nested TPC-H as the first step, and link to it. - Explain that compaction is a heavily I/O-bound background job that covers many datasets and is typically very expensive in resources. - Add a benchmark setup section. Spark is an internal fork of Spark 4.0, since the job is a real customer workload. Comet is 1.3.0-SNAPSHOT.
Abhishek-Mohan
left a comment
There was a problem hiding this comment.
Thanks for writing this up. I checked the technical content against Comet main, Spark 4.0 source and the linked PRs, and the code, config, Spark and microbenchmark claims hold up well, including the placement simulation numbers.
Most of what I'd want before publishing is already on your Draft status list (peak memory definition, cluster, write path); my comments on those just add specifics. The rest are factual tightenings, plus a few nits, each as its own comment.
|
|
||
| Compaction rewrites the many small files of a table partition as a few large ones. It runs in the background across many datasets of a data lake, and it is typically very expensive in terms of resources. For plain Parquet tables it is often a single line of Spark, and at its center is the round robin shuffle behind `repartition(n)`. | ||
|
|
||
| On a table with more than 200 columns, nested up to seven levels deep, Comet compacted a 200 GB daily partition in 70 minutes, against 80 minutes for Spark. The bottleneck was the shuffle write, where Comet's round robin hashed every value of every row. With positional round robin, which places rows by their position and copies them in contiguous groups, Comet compacts the same partition in 28 minutes. That is 2.9x faster than Spark, with 6.4x less memory. |
There was a problem hiding this comment.
The "6.4x less memory" in the opening paragraph depends on a metric the post doesn't define yet (see my comment on the Results table). I'd keep it out of the intro until the metric is pinned down, and lead with the time result, which is solid.
| - **Memory**: peak memory is the mean over all executors. | ||
|
|
||
| ## Results | ||
|
|
||
| | | Execution time | Peak memory, mean over executors | |
There was a problem hiding this comment.
Agree with the "Peak memory" item in the Draft status. Before this ships, the table needs to say:
- which metric this is: JVM heap used, off-heap, or process memory (RSS);
- whether it is per executor or summed across executors (770 GB as a mean per executor would be an unusually large executor);
- how much heap and off-heap memory each engine was configured with.
If it is a JVM heap metric, it misses Comet's off-heap memory, so the Spark-vs-Comet comparison would not be like for like. And if Spark ran with a large heap, its peak mostly reflects the configured size rather than what the job needs.
| | Comet, positional round robin | **28 min (2.9x)** | **120 GB (6.4x less)** | | ||
|
|
||
| - Positional round robin cut Comet's time from 70 to 28 minutes, 2.5x on its own, since it is the only setting that differs between the two Comet runs. | ||
| - Both Comet runs use about a sixth of Spark's memory. We have not profiled where Spark's memory goes, but its sort before the round robin is the likely cause of most of it. |
There was a problem hiding this comment.
Since this is unprofiled, I'd soften it or drop it. The sort before repartition fills execution memory up to its limit and then spills, so it can push Spark's peak up to its configured budget, but whether it explains most of the 770 GB depends on the metric and memory settings asked about above.
| ## Benchmark setup | ||
|
|
||
| - **Workload**: a real customer compaction job, `spark.read.parquet(inputPath).repartition(50).write.parquet(outputPath)`, over one daily partition. | ||
| - **Data**: 1,200 Parquet files and 200 GB in total, with more than 200 columns nested up to seven levels deep. The job writes 50 Parquet files. | ||
| - **Spark**: an internal fork of Spark 4.0. This is a real customer workload, so we tested it on the Spark fork it runs on in production. | ||
| - **Comet**: 1.3.0-SNAPSHOT, a development build ahead of the 1.3.0 release. Positional round robin itself has been available since [Comet 1.1.0](https://datafusion.apache.org/blog/2026/10/01/datafusion-comet-1.1.0/). | ||
| - **Comet settings**: `spark.comet.shuffle.native.partitioning.roundrobin.enabled=true` in both Comet runs, plus `spark.comet.shuffle.native.partitioning.roundrobin.positional.enabled=true` in the positional run. Nothing else differs between the two Comet runs. | ||
| - **Memory**: peak memory is the mean over all executors. |
There was a problem hiding this comment.
Agree with the "Cluster" and "Write path" items in the Draft status. To match the setup detail in the nested TPC-H post, could this section also give:
- number of executors, cores per executor, heap and off-heap sizes;
- the settings shared by both engines;
- how many runs per configuration, and whether the figures are best, median or mean;
- the Comet commit of the 1.3.0-SNAPSHOT build (no need to rerun on a release, but the shuffle writer has changed since 1.2.0);
- anything the internal Spark fork changes on this path, and confirmation that
spark.sql.execution.sortBeforeRepartitionwas left at its defaulttrue, since line 220 relies on it; - which Parquet writer ran.
spark.comet.write.parquet.enabledis a testing-only config, off by default, so I assume Spark's writer in all three runs. That matters for the 2.9x versus Spark, not for the 2.5x between the two Comet runs.
|
|
||
| Compaction rewrites the many small files of a table partition as a few large ones. It runs in the background across many datasets of a data lake, and it is typically very expensive in terms of resources. For plain Parquet tables it is often a single line of Spark, and at its center is the round robin shuffle behind `repartition(n)`. | ||
|
|
||
| On a table with more than 200 columns, nested up to seven levels deep, Comet compacted a 200 GB daily partition in 70 minutes, against 80 minutes for Spark. The bottleneck was the shuffle write, where Comet's round robin hashed every value of every row. With positional round robin, which places rows by their position and copies them in contiguous groups, Comet compacts the same partition in 28 minutes. That is 2.9x faster than Spark, with 6.4x less memory. |
There was a problem hiding this comment.
Small scoping point: "Comet's round robin hashed every value" is only true once spark.comet.shuffle.native.partitioning.roundrobin.enabled is set. With defaults, Comet hands round robin to its JVM shuffle, which converts batches to rows and runs Spark's sort. Something like "Comet's round robin in the native shuffle hashed every value" would avoid readers thinking this is the default behavior.
|
|
||
| ### Copying runs instead of rows | ||
|
|
||
| Keeping neighboring rows together also removes most of the gather. The writer records `(batch, start, len)` runs instead of one `(batch, row)` pair per row, and builds each output batch by slicing and concatenating runs. That is one bulk copy per buffer per run, instead of a walk through every column for every row. A run that covers a whole buffered batch passes through without a copy. Arrow's IPC writer writes only the visible window of a sliced array for every type except `Utf8View` and `BinaryView`, which it writes with their shared data buffers in full, so a schema with a view type falls back to hash placement. |
There was a problem hiding this comment.
Nit: "A run that covers a whole buffered batch passes through without a copy" needs one more condition. The no-copy path in RunIterator also requires the run to be exactly batch_size rows, so a shorter buffered batch (for example after a filter) is still concatenated with the runs after it.
| Keeping neighboring rows together also removes most of the gather. The writer records `(batch, start, len)` runs instead of one `(batch, row)` pair per row, and builds each output batch by slicing and concatenating runs. That is one bulk copy per buffer per run, instead of a walk through every column for every row. A run that covers a whole buffered batch passes through without a copy. Arrow's IPC writer writes only the visible window of a sliced array for every type except `Utf8View` and `BinaryView`, which it writes with their shared data buffers in full, so a schema with a view type falls back to hash placement. | |
| Keeping neighboring rows together also removes most of the gather. The writer records `(batch, start, len)` runs instead of one `(batch, row)` pair per row, and builds each output batch by slicing and concatenating runs. That is one bulk copy per buffer per run, instead of a walk through every column for every row. A run that covers a whole, full-size buffered batch passes through without a copy. Arrow's IPC writer writes only the visible window of a sliced array for every type except `Utf8View` and `BinaryView`, which it writes with their shared data buffers in full, so a schema with a view type falls back to hash placement. |
|
|
||
| ### Copying runs instead of rows | ||
|
|
||
| Keeping neighboring rows together also removes most of the gather. The writer records `(batch, start, len)` runs instead of one `(batch, row)` pair per row, and builds each output batch by slicing and concatenating runs. That is one bulk copy per buffer per run, instead of a walk through every column for every row. A run that covers a whole buffered batch passes through without a copy. Arrow's IPC writer writes only the visible window of a sliced array for every type except `Utf8View` and `BinaryView`, which it writes with their shared data buffers in full, so a schema with a view type falls back to hash placement. |
There was a problem hiding this comment.
Nit: "for every type except Utf8View and BinaryView" is broader than Arrow's behavior. Arrow's IPC writer also serializes the full child array of a sliced ListView / LargeListView (get_list_view_array_buffers). Spark types never become list views in this shuffle, so it doesn't affect Comet, but scoping the sentence to the types that can reach the shuffle would keep it accurate.
| The figure follows one map task with 18 rows and three output partitions. Hashing scatters neighboring rows. Vectorized round robin keeps each batch together, so a retry that frames the rows as 4, 8 and 6 instead of 6, 6 and 6 moves rows 4 and 5. Positional round robin counts rows, so the retry places every row the same way. | ||
|
|
||
| - **Rows, not batches.** A group that one batch leaves part-way through is finished by the next. Placement depends on row order alone, not on how the rows are framed or when the writer spills. Tests feed the same 1,000 rows through five different framings, and through forced spills, and check that every row lands in the same partition. | ||
| - **Group size.** By default `groupRows` is `clamp(batchSize / numPartitions, 64, batchSize)`, which is 163 rows for 8,192-row batches into 50 partitions. Each task then cycles through all the partitions about once per batch, and within one task no two partitions differ by more than `groupRows` rows. The 64-row floor keeps runs from getting too short when there are many partitions, at the cost that a task then needs several batches to cycle through them all. The driver resolves the group size and stores it with the shuffle dependency, so a task that reruns after `spark.comet.batchSize` changes still uses the original size. |
There was a problem hiding this comment.
Nit: the 64-row floor has a visible cost beyond "several batches to cycle through them all": with few rows per map task and many partitions, some output files can end up empty. #6095 gives an example: ten map tasks of one 8,192-row batch each into 1,000 partitions leave 311 empty. It doesn't affect repartition(50), so a short clause is enough, or leave it to the native shuffle guide.
|
|
||
| ## Get involved | ||
|
|
||
| Thanks to [@andygrove] for native round robin and positional round robin, to [@comphead] for the experiments in #5449 and #5450, and to [@sunchao] and [@mbutrovich] for the reviews that shaped its retry safety and defaults. |
There was a problem hiding this comment.
Nit: @sunchao also proposed the core idea, counting rows across batches and sending fixed-size groups to the next partition, in #5449, which @andygrove then picked up. It might be worth crediting that, not only the reviews.
|
|
||
| The input is one daily partition of a production event table, 1,200 Parquet files and 200 GB in total. The output is 50 files. | ||
|
|
||
| The table is wide and deep. It has more than 200 columns, nested up to seven levels deep. Parquet stores each leaf field as its own column. Arrow gives each level of nesting its own array with its own validity bitmap, and each list adds an offsets buffer. Even a small nested column is several arrays: |
There was a problem hiding this comment.
Nit: could you double-check "more than 200 columns"? #5449 describes what looks like the same workload as "~194 nested columns".
|
Thanks @Abhishek-Mohan for the review |
Rendered version: https://comphead.github.io/datafusion-site/blog/2026/11/15/comet-parquet-compaction/. This is a GitHub Pages preview built from this branch with the site's ASF Pelican build, so the preview URL has the same shape as production.
Adds a blog post on Parquet compaction with Apache DataFusion Comet, and on how positional round robin (apache/datafusion-comet#6095) made it 2.9x faster than Spark.
The post covers:
spark.read.parquet(...).repartition(50).write.parquet(...)runs. The map stage scans, partitions rows round robin and writes the shuffle. The reduce stage reads the shuffle and writes one Parquet file per task.Files:
content/blog/2026-11-15-comet-parquet-compaction.mdcontent/images/comet-parquet-compaction/: three SVG figures (results, the job's stages, and row placement under the three strategies). They embed the same Lato subset as the nested TPC-H charts.Draft status
To confirm before review:
Checks:
typospasses.