From 04a0502d48676302bbd880cc8e7fd58c8e9f73f1 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 12 Nov 2026 06:18:53 -0700 Subject: [PATCH] Add Apache DataFusion Ballista 56.0.0 release blog post (draft) Co-Authored-By: Claude Opus 5.5 --- .../2026-11-30-datafusion-ballista-56.0.0.md | 306 ++++++++++++++++++ .../tpch_q18_memory_smallest.svg | 75 +++++ .../tpch_q18_memory_time.svg | 119 +++++++ .../tpch_sf1000_total.svg | 42 +++ 4 files changed, 542 insertions(+) create mode 100644 content/blog/2026-11-30-datafusion-ballista-56.0.0.md create mode 100644 content/images/datafusion-ballista-56.0.0/tpch_q18_memory_smallest.svg create mode 100644 content/images/datafusion-ballista-56.0.0/tpch_q18_memory_time.svg create mode 100644 content/images/datafusion-ballista-56.0.0/tpch_sf1000_total.svg diff --git a/content/blog/2026-11-30-datafusion-ballista-56.0.0.md b/content/blog/2026-11-30-datafusion-ballista-56.0.0.md new file mode 100644 index 00000000..d010ae6b --- /dev/null +++ b/content/blog/2026-11-30-datafusion-ballista-56.0.0.md @@ -0,0 +1,306 @@ +--- +layout: post +title: Apache DataFusion Ballista 56.0.0 Released +date: 2026-11-30 +author: pmc +categories: [release] +--- + + + + +[TOC] + +We are pleased to announce version [56.0.0] of [Apache DataFusion Ballista]. Ballista is a distributed query +execution engine that enhances [Apache DataFusion] by enabling parallel execution of workloads across multiple +nodes. + +[56.0.0]: https://github.com/apache/datafusion-ballista/blob/56.0.0/docs/source/changelog/56.0.0.md +[Apache DataFusion Ballista]: https://datafusion.apache.org/ballista/ +[Apache DataFusion]: https://datafusion.apache.org + +This release contains 189 commits from 20 contributors. It builds on [55.0.0], which made adaptive query +execution (AQE) the default, and this cycle went into three things: making Ballista faster, making AQE's +decisions better, and making the scheduler and executors behave correctly when things go wrong. + +The highlights: + +- **Faster**: Ballista now runs our 1 TB benchmark derived from TPC-H 2.2x faster than Apache Spark by default, + up from 1.79x in 55.0.0, and 3.4x faster with a new, experimental pipelined execution mode. +- **Less memory**: on the most memory-intensive query in that benchmark, Ballista finished in less executor + memory than Spark or Spark with Apache DataFusion Comet, and ran just as fast in 8 GiB as in 64 GiB. +- **Better adaptive plans**: AQE now sees distinct counts from shuffles, measures a join's output before + broadcasting it, runs a repeated subquery once, and plans submitted physical plans too. +- **More robust**: graceful shutdown, a standalone result service, and a set of fixes that turn silent wrong + results and hung jobs into correct results or clear errors. + +[55.0.0]: /blog/2026/10/03/datafusion-ballista-55.0.0/ + +## Performance + +We benchmarked Ballista against [Apache Spark] 4.1.3, and against Spark 4.1.3 with [Apache DataFusion Comet] +1.1.0, on TPC-H at scale factor 1000, with the data stored as Parquet on S3. All three ran on the same Kubernetes +cluster with 32 executors, each with 8 vCPU and 64 GiB of memory, and each query time is the mean of two runs +including the cold first run. + + + +Ballista ran the 22 queries in 405.0 seconds, down from the 502.4 seconds we reported for 55.0.0 on the same +cluster. That is 2.2x faster than Spark's 898.2 seconds, and Ballista was faster on 20 of the 22 queries, with +the largest gaps on Q1 (4.9x), Q18 (3.8x) and Q17 (3.7x). + +In the 55.0.0 announcement Comet was 1.10x faster than Ballista overall. Ballista is now 1.12x faster than +Comet's 455.1 seconds, though Comet still wins 13 of the 22 queries, and by more than 2x on Q14 and Q20. With the +experimental pipelined mode described below, Ballista ran the suite in 260.9 seconds: 3.4x faster than Spark and +1.7x faster than Comet, and faster than Spark on every query. + +These are not tuned comparisons. Spark and Comet ran 16 tasks on each executor's 8 vCPU, while Ballista ran 8, +and Comet had an extra 32 GiB of off-heap memory per executor. The Spark and Comet results come from an earlier +run with separate scripts and their own phrasing of the queries, so they are not an exact like-for-like +comparison. The [benchmarking guide] has the full configuration, the per-query results, and columns for +[DataFusion Distributed] and Trino. The scripts that ran the Ballista columns are now in the repository, in +[`benchmarks/cluster`], and can run all five engines on your own Kubernetes cluster. + +[Apache Spark]: https://spark.apache.org/ +[Apache DataFusion Comet]: https://datafusion.apache.org/comet/ +[benchmarking guide]: https://datafusion.apache.org/ballista/contributors-guide/benchmarking/index.html +[DataFusion Distributed]: https://github.com/datafusion-contrib/datafusion-distributed +[`benchmarks/cluster`]: https://github.com/apache/datafusion-ballista/tree/main/benchmarks/cluster + +### Where the time went + +Most of the gain comes from moving and reading less data: + +- **zstd shuffle compression by default.** Shuffles are usually bound by the bytes they move. For the + `lineitem` columns that Q5 shuffles, zstd at level 1 stores about 8.2 bytes per row where LZ4 stored about 16, + for a little more CPU. `ballista.shuffle.compression.codec` still accepts `lz4` and `none`. +- **One broadcast read per executor.** Every task of a broadcast hash join used to fetch and decode the build + side on its own. In TPC-H SF100 Q2, a 16-task stage made 16,384 fetch requests and decoded the same table 16 + times. The first task on an executor now fetches it, and the other tasks share it. +- **Semi-join reduction.** DataFusion applies a selective filter on a small dimension table, such as a single + `nation`, only at the join that names it. In Q7 that meant joining and shuffling 182 million `lineitem` rows + before the filter dropped 92% of them. A new rule also semi-joins the filtered dimension below the joins in + between, so the rows it would drop are removed before the shuffle. +- **Balanced scans.** DataFusion splits a scan's files into groups before any filter is applied, so when the data + is clustered by the filtered column, as with date-partitioned directories, the files that survive pruning end + up in a few groups and a few tasks do all the reading. The scheduler now prunes files from their statistics and + spreads the remaining files evenly across tasks. +- **Smaller requests.** Partition locations list each executor once instead of copying its metadata into every + location, scheduler connections no longer wait on Nagle's algorithm, remote shuffle fetches stream instead of + buffering whole blocks, and object stores built for S3 are cached. + +### Pipelined execution (experimental) + +Ballista has always run a query stage by stage: each stage writes its output to shuffle files, and the next stage +starts when they are complete. That is robust and lets AQE replan between stages, but each stage boundary costs +time, which matters most for interactive queries. + +56.0.0 adds an opt-in pipelined mode. The scheduler places every task of a job up front, and stages stream +record batches to each other over Arrow Flight instead of through shuffle files. Results stream to the client and +to Flight SQL as they are produced. Joins are planned by size, since there is no measured input to adapt to. + +Pipelined execution is experimental. It is enabled per session with `ballista.execution.mode = pipelined` and +needs push-based scheduling. Pipelined jobs are planned statically, so AQE does not apply to them, and a failed +task fails the whole job rather than being retried. The default mode, now called `staged`, is unchanged. The +[pipelined execution guide] describes when to use it. + +[pipelined execution guide]: https://datafusion.apache.org/ballista/user-guide/pipelined-execution.html + +## Memory usage + +Executor memory is what a cluster is sized for, and paid for: each executor reserves it for the whole job, +whether it uses it or not. So as well as how fast Ballista is, we wanted to know how little memory it needs. + +We ran TPC-H Q18 at scale factor 1000 on a smaller cluster: two servers with 32 CPU cores and 125 GiB of memory +each, running 2 executors of 16 CPUs, with the data on local disk. Q18 aggregates all of `lineitem` by order key, +about 1.5 billion groups, and joins the result back to `orders`, `customer` and `lineitem`, which makes it the most +memory-intensive query in TPC-H. Each engine ran it at 64 GiB per executor and then at smaller and smaller sizes, +until the query no longer finished without an executor running out of memory. Spark and Comet were each run with +several divisions of executor memory between the heap, off-heap memory and overhead. Ballista ran with its +default settings. + + + +Ballista finished in 8 GiB per executor. Spark needed 12 GiB in its best configuration, with half of its memory +off-heap, and 18 GiB with all of it on the heap. Spark with [Comet 1.2.0] needed 20 GiB, with three quarters +off-heap. + + + +Ballista's time barely changed as its memory shrank: 133 seconds at 64 GiB and 132 seconds at 8 GiB. Since +55.0.0, executors size their memory pool from the container's limit by default and spill to disk under +pressure, so Ballista ran the query in whatever memory it was given. Spark slowed by 17% on the way to its +smallest size. Comet, at about 150 seconds, was faster than Spark at every size it completed but needed the most +memory. Every Comet failure was an executor exceeding its container's limit rather than an error from Comet's +memory pool, consistent with the native memory outside the pool that [Comet 1.1.0 began to measure]. + +This is one query on one cluster, and each size ran once, so a size close to an engine's limit could go either +way on another run. Below 8 GiB, Ballista executors ran out of memory and were restarted. At 6 GiB the query +still finished, at about a third of its usual speed, but at 5 GiB and below it stopped making progress instead of +failing ([#2856]). The [memory usage page] has every run and how to reproduce them. + +[Comet 1.2.0]: /blog/2026/11/02/datafusion-comet-1.2.0/ +[Comet 1.1.0 began to measure]: /blog/2026/10/01/datafusion-comet-1.1.0/ +[memory usage page]: https://datafusion.apache.org/ballista/contributors-guide/benchmarking/memory.html +[#2856]: https://github.com/apache/datafusion-ballista/issues/2856 + +Executors are also easier to size and debug. They report their memory use in heartbeats and log how much memory +they have allocated, including memory the pool does not track, and when a task runs out of memory, the error +names the operators holding the most memory. Spill files now go to the executor's work directory, whose free +space sets the spill limit by default. Executors also raise their open file limit at startup: a task holds a +file for each shuffle partition it reads or writes, and with the common limit of 1024, TPC-H at 128 partitions +or more failed with `Too many open files`. + +## Adaptive query execution + +AQE, the default since 55.0.0, replans a query between stages using statistics from the stages that have +completed. This release gives it better statistics and lets it do more with them. + +### Better statistics, safer broadcasts + +A shuffle used to carry row counts and sizes but no distinct counts, so after a shuffle DataFusion estimated an +inner join at the size of its smaller input. For a join on a foreign key that can be wrong by the join's whole +fan-out. In Q21 at SF1000, a join estimated at 399,809 rows produced 151,688,609, AQE broadcast it, and the job +failed. + +Two changes fix this. Shuffle writers now sketch the distinct count of each partitioning key with HyperLogLog +(`ballista.shuffle.collect_distinct_counts`), so AQE estimates joins after a shuffle from distinct counts as +well as row counts. And when a join's build side is itself an unexecuted join whose size is only an estimate, +AQE now runs it as its own stage and measures it before deciding to broadcast it +(`ballista.optimizer.stage_build_side`). + +### Doing less work + +- **Exchange reuse.** When a query reads the same subplan more than once, such as a common table expression + joined to itself or Q15's revenue view, AQE now runs its stage once and lets every reader share the output + (`ballista.planner.reuse_exchange.enabled`, on by default). +- **Physical plans through AQE.** Clients that build their own physical plans and submit them directly, for + example with their own operators between DataFusion repartitions, used to get the static planner. They now go + through AQE, like SQL and DataFrame queries. +- **Coalescing to one partition.** With `ballista.planner.coalesce.enabled`, a stage whose whole input fits in + one target-sized partition is now coalesced all the way down to a single task, as Spark does for final + aggregates, small joins and the sorts below an `ORDER BY`. +- **Skew joins (opt-in).** With `ballista.planner.skew_join.enabled`, AQE splits a skewed partition of a + partitioned join into smaller tasks and replicates the matching partition of the other side, so one hot key + no longer serializes the join on a single task. +- **Extension rules.** Projects that embed the scheduler can add their own rules to adaptive planning. + +The release also lays the groundwork for distributed runtime filters: executors can now build Bloom, IN-list, +and min/max summaries of a join's build side and apply them to the probe side. Placing these filters in query +plans will follow in a future release. + +### Correctness + +Much of the AQE work in this cycle was making replanning safe. A stage cancelled by a replan is now retired and +its late tasks discarded, a stage that is rolled back discards its partial outputs before it runs again, a +failed stage transition no longer drops the stage from the graph, and intermediate shuffle files are freed when +an adaptive job succeeds. Aggregates above joins whose strategy is still undecided keep their partial and final +phases, and semi joins with volatile expressions are no longer moved. + +## Robustness and operations + +### Shutting down cleanly + +The scheduler, executors and the Docker images now handle `SIGTERM`, so `docker stop` and Kubernetes pod +termination reach Ballista instead of waiting out the grace period. Executors finish their running tasks before +they exit. The scheduler stops accepting new jobs, over gRPC and Flight SQL alike, lets its active jobs finish, +and then drains its connections, all within `--graceful-shutdown-timeout-seconds`. + +### A standalone result service + +Clients that cannot reach the executors, as is common on Kubernetes, used to fetch results through a proxy +embedded in the scheduler, where every result byte competed with planning and scheduling. The new +`ballista-result-service` is a stateless Arrow Flight service that forwards each fetch to the executor holding +the partition, and can be scaled on its own behind a load balancer. The embedded proxy is deprecated and will +be removed in 57.0.0. + +### Errors instead of wrong answers + +Several plans could previously return wrong results with no error. They now either return correct results or +fail clearly: + +- A range repartition in a submitted physical plan was dropped, so a distributed global sort returned partitions + that were sorted but overlapping. It is now planned as a shuffle. +- When a stage is split between tasks, a leaf operator the scheduler did not know how to restrict to a task's + partitions ran all of them, so some partitions were returned twice and others not at all. The scheduler now + fails the job, and projects with custom operators can teach it how to restrict them. +- A task whose status update carried metrics the scheduler could not merge had its status dropped, and under + pull scheduling, tasks that did not fit in one poll response never ran and the job hung. Both are fixed, and an + unreadable shuffle block is now reported as a fetch failure so the stage that wrote it reruns. +- Pull-based executors no longer take on more partitions than they have vcores. + +The TPC-H SF10 and TPC-DS SF1 suites now run against every code change before it merges, through a merge +queue, in addition to the high-availability chaos tests added in 55.0.0. + +## Python + +The Python client has been rebuilt on DataFusion Python's foreign function interface. `BallistaSessionContext` +now returns a standard `datafusion.SessionContext` with a Ballista query planner installed, so every operation +that executes a plan, including `collect`, `to_pandas`, `write_parquet` and `EXPLAIN ANALYZE`, runs on the cluster +without a wrapper in between. Python UDFs are supported, as is S3, and a pure-Python client is available as the +`python-client` extra for environments where the native client does not fit. + +## Other changes + +- `DataFrame::cache()` materializes a DataFrame on the cluster, and later queries read it back. +- `DESCRIBE` works on Ballista tables. +- `ComposedLogicalExtensionCodec` combines logical extension codecs, so a project can use several. +- The `spark-compat` feature has been removed. +- S3 credentials no longer appear in debug and trace logs. + +## Upgrading + +The wire format between the scheduler and executors has changed (protocol version 4), so they must be upgraded +together. The Python client's API changed as described above, and several scheduler and executor APIs gained +fields. The [upgrade guide] lists every change and what to do about it. + +[upgrade guide]: https://datafusion.apache.org/ballista/upgrading/56.0.0.html + +## Thank You + + + +Thank you to all 20 contributors, including Andy Grove, Marko Milenković, Ville Brofeldt, Richard Durkee, Noah +Kusaba, Samuel James, Akshay Chitneni, and everyone who contributed code, reviews, bug reports, and feedback. +See the [changelog] for the complete list of changes and contributors. + +[changelog]: https://github.com/apache/datafusion-ballista/blob/56.0.0/docs/source/changelog/56.0.0.md diff --git a/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_smallest.svg b/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_smallest.svg new file mode 100644 index 00000000..04352802 --- /dev/null +++ b/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_smallest.svg @@ -0,0 +1,75 @@ + + + + Ballista finished in 8 GiB per executor, Spark needed 12 GiB and Spark + Comet 20 GiB + Smallest executor memory at which TPC-H q18 (SF1000) finished with no executor out of memory, and its time there. + + + + + + + + + + + Ballista + Spark, on-heap + Spark, on-heap, 2x overhead + Spark, 50% off-heap + Spark + Comet, 50% off-heap + Spark + Comet, 75% off-heap + Spark + Comet, 75% off-heap, 2x overhead + + 8 GiB · 132 s + 18 GiB · 336 s + 12 GiB · 406 s + 12 GiB · 387 s + 24 GiB · 160 s + 20 GiB · 150 s + 22 GiB · 174 s + + Derived from TPC-H, not an official TPC-H result. 2 executors of 16 CPUs. Each size is a fresh cluster; mean of 2 runs. + diff --git a/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_time.svg b/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_time.svg new file mode 100644 index 00000000..68882f11 --- /dev/null +++ b/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_time.svg @@ -0,0 +1,119 @@ + + + + Time for TPC-H q18 as executor memory shrinks + Query time in seconds for each engine in its configuration with the smallest size. Lower is better; lines stop at that size. + + + Ballista + + Spark + + Spark + Comet + + largest size that failed or needed an executor restarted + + + + + + + + + 0 + 100 + 200 + 300 + 400 + + + 0 + 8 + 16 + 24 + 32 + 40 + 48 + 56 + 64 + Executor memory (GiB) + + + + + + + + + + + + + + + + + + + + + + + + + + + + Spark + 330 s at 64 GiB + + Spark + Comet + 145 s at 64 GiB + + Ballista + 133 s at 64 GiB + + Derived from TPC-H, not an official TPC-H result. 2 executors of 16 CPUs. Each size is a fresh cluster; mean of 2 runs. + diff --git a/content/images/datafusion-ballista-56.0.0/tpch_sf1000_total.svg b/content/images/datafusion-ballista-56.0.0/tpch_sf1000_total.svg new file mode 100644 index 00000000..40523103 --- /dev/null +++ b/content/images/datafusion-ballista-56.0.0/tpch_sf1000_total.svg @@ -0,0 +1,42 @@ + + + + 2.2x faster than Apache Spark by default, 3.4x with pipelined execution + Total time for the 22 queries of a 1 TB (SF1000) benchmark derived from TPC-H. Lower is better. + + + + + + + Apache Spark 4.1.3 + Ballista staged (default) + Ballista pipelined (experimental) + + 898 s + 405 s · 2.2x faster + 261 s · 3.4x faster + + Derived from TPC-H, not an official TPC-H result. 32 executors with 8 vCPUs each. Mean of 2 runs including the cold first run. +