diff --git a/content/blog/2026-11-30-datafusion-ballista-56.0.0.md b/content/blog/2026-11-30-datafusion-ballista-56.0.0.md
new file mode 100644
index 00000000..d010ae6b
--- /dev/null
+++ b/content/blog/2026-11-30-datafusion-ballista-56.0.0.md
@@ -0,0 +1,306 @@
+---
+layout: post
+title: Apache DataFusion Ballista 56.0.0 Released
+date: 2026-11-30
+author: pmc
+categories: [release]
+---
+
+
+
+
+[TOC]
+
+We are pleased to announce version [56.0.0] of [Apache DataFusion Ballista]. Ballista is a distributed query
+execution engine that enhances [Apache DataFusion] by enabling parallel execution of workloads across multiple
+nodes.
+
+[56.0.0]: https://github.com/apache/datafusion-ballista/blob/56.0.0/docs/source/changelog/56.0.0.md
+[Apache DataFusion Ballista]: https://datafusion.apache.org/ballista/
+[Apache DataFusion]: https://datafusion.apache.org
+
+This release contains 189 commits from 20 contributors. It builds on [55.0.0], which made adaptive query
+execution (AQE) the default, and this cycle went into three things: making Ballista faster, making AQE's
+decisions better, and making the scheduler and executors behave correctly when things go wrong.
+
+The highlights:
+
+- **Faster**: Ballista now runs our 1 TB benchmark derived from TPC-H 2.2x faster than Apache Spark by default,
+ up from 1.79x in 55.0.0, and 3.4x faster with a new, experimental pipelined execution mode.
+- **Less memory**: on the most memory-intensive query in that benchmark, Ballista finished in less executor
+ memory than Spark or Spark with Apache DataFusion Comet, and ran just as fast in 8 GiB as in 64 GiB.
+- **Better adaptive plans**: AQE now sees distinct counts from shuffles, measures a join's output before
+ broadcasting it, runs a repeated subquery once, and plans submitted physical plans too.
+- **More robust**: graceful shutdown, a standalone result service, and a set of fixes that turn silent wrong
+ results and hung jobs into correct results or clear errors.
+
+[55.0.0]: /blog/2026/10/03/datafusion-ballista-55.0.0/
+
+## Performance
+
+We benchmarked Ballista against [Apache Spark] 4.1.3, and against Spark 4.1.3 with [Apache DataFusion Comet]
+1.1.0, on TPC-H at scale factor 1000, with the data stored as Parquet on S3. All three ran on the same Kubernetes
+cluster with 32 executors, each with 8 vCPU and 64 GiB of memory, and each query time is the mean of two runs
+including the cold first run.
+
+
+
+Ballista ran the 22 queries in 405.0 seconds, down from the 502.4 seconds we reported for 55.0.0 on the same
+cluster. That is 2.2x faster than Spark's 898.2 seconds, and Ballista was faster on 20 of the 22 queries, with
+the largest gaps on Q1 (4.9x), Q18 (3.8x) and Q17 (3.7x).
+
+In the 55.0.0 announcement Comet was 1.10x faster than Ballista overall. Ballista is now 1.12x faster than
+Comet's 455.1 seconds, though Comet still wins 13 of the 22 queries, and by more than 2x on Q14 and Q20. With the
+experimental pipelined mode described below, Ballista ran the suite in 260.9 seconds: 3.4x faster than Spark and
+1.7x faster than Comet, and faster than Spark on every query.
+
+These are not tuned comparisons. Spark and Comet ran 16 tasks on each executor's 8 vCPU, while Ballista ran 8,
+and Comet had an extra 32 GiB of off-heap memory per executor. The Spark and Comet results come from an earlier
+run with separate scripts and their own phrasing of the queries, so they are not an exact like-for-like
+comparison. The [benchmarking guide] has the full configuration, the per-query results, and columns for
+[DataFusion Distributed] and Trino. The scripts that ran the Ballista columns are now in the repository, in
+[`benchmarks/cluster`], and can run all five engines on your own Kubernetes cluster.
+
+[Apache Spark]: https://spark.apache.org/
+[Apache DataFusion Comet]: https://datafusion.apache.org/comet/
+[benchmarking guide]: https://datafusion.apache.org/ballista/contributors-guide/benchmarking/index.html
+[DataFusion Distributed]: https://github.com/datafusion-contrib/datafusion-distributed
+[`benchmarks/cluster`]: https://github.com/apache/datafusion-ballista/tree/main/benchmarks/cluster
+
+### Where the time went
+
+Most of the gain comes from moving and reading less data:
+
+- **zstd shuffle compression by default.** Shuffles are usually bound by the bytes they move. For the
+ `lineitem` columns that Q5 shuffles, zstd at level 1 stores about 8.2 bytes per row where LZ4 stored about 16,
+ for a little more CPU. `ballista.shuffle.compression.codec` still accepts `lz4` and `none`.
+- **One broadcast read per executor.** Every task of a broadcast hash join used to fetch and decode the build
+ side on its own. In TPC-H SF100 Q2, a 16-task stage made 16,384 fetch requests and decoded the same table 16
+ times. The first task on an executor now fetches it, and the other tasks share it.
+- **Semi-join reduction.** DataFusion applies a selective filter on a small dimension table, such as a single
+ `nation`, only at the join that names it. In Q7 that meant joining and shuffling 182 million `lineitem` rows
+ before the filter dropped 92% of them. A new rule also semi-joins the filtered dimension below the joins in
+ between, so the rows it would drop are removed before the shuffle.
+- **Balanced scans.** DataFusion splits a scan's files into groups before any filter is applied, so when the data
+ is clustered by the filtered column, as with date-partitioned directories, the files that survive pruning end
+ up in a few groups and a few tasks do all the reading. The scheduler now prunes files from their statistics and
+ spreads the remaining files evenly across tasks.
+- **Smaller requests.** Partition locations list each executor once instead of copying its metadata into every
+ location, scheduler connections no longer wait on Nagle's algorithm, remote shuffle fetches stream instead of
+ buffering whole blocks, and object stores built for S3 are cached.
+
+### Pipelined execution (experimental)
+
+Ballista has always run a query stage by stage: each stage writes its output to shuffle files, and the next stage
+starts when they are complete. That is robust and lets AQE replan between stages, but each stage boundary costs
+time, which matters most for interactive queries.
+
+56.0.0 adds an opt-in pipelined mode. The scheduler places every task of a job up front, and stages stream
+record batches to each other over Arrow Flight instead of through shuffle files. Results stream to the client and
+to Flight SQL as they are produced. Joins are planned by size, since there is no measured input to adapt to.
+
+Pipelined execution is experimental. It is enabled per session with `ballista.execution.mode = pipelined` and
+needs push-based scheduling. Pipelined jobs are planned statically, so AQE does not apply to them, and a failed
+task fails the whole job rather than being retried. The default mode, now called `staged`, is unchanged. The
+[pipelined execution guide] describes when to use it.
+
+[pipelined execution guide]: https://datafusion.apache.org/ballista/user-guide/pipelined-execution.html
+
+## Memory usage
+
+Executor memory is what a cluster is sized for, and paid for: each executor reserves it for the whole job,
+whether it uses it or not. So as well as how fast Ballista is, we wanted to know how little memory it needs.
+
+We ran TPC-H Q18 at scale factor 1000 on a smaller cluster: two servers with 32 CPU cores and 125 GiB of memory
+each, running 2 executors of 16 CPUs, with the data on local disk. Q18 aggregates all of `lineitem` by order key,
+about 1.5 billion groups, and joins the result back to `orders`, `customer` and `lineitem`, which makes it the most
+memory-intensive query in TPC-H. Each engine ran it at 64 GiB per executor and then at smaller and smaller sizes,
+until the query no longer finished without an executor running out of memory. Spark and Comet were each run with
+several divisions of executor memory between the heap, off-heap memory and overhead. Ballista ran with its
+default settings.
+
+
+
+Ballista finished in 8 GiB per executor. Spark needed 12 GiB in its best configuration, with half of its memory
+off-heap, and 18 GiB with all of it on the heap. Spark with [Comet 1.2.0] needed 20 GiB, with three quarters
+off-heap.
+
+
+
+Ballista's time barely changed as its memory shrank: 133 seconds at 64 GiB and 132 seconds at 8 GiB. Since
+55.0.0, executors size their memory pool from the container's limit by default and spill to disk under
+pressure, so Ballista ran the query in whatever memory it was given. Spark slowed by 17% on the way to its
+smallest size. Comet, at about 150 seconds, was faster than Spark at every size it completed but needed the most
+memory. Every Comet failure was an executor exceeding its container's limit rather than an error from Comet's
+memory pool, consistent with the native memory outside the pool that [Comet 1.1.0 began to measure].
+
+This is one query on one cluster, and each size ran once, so a size close to an engine's limit could go either
+way on another run. Below 8 GiB, Ballista executors ran out of memory and were restarted. At 6 GiB the query
+still finished, at about a third of its usual speed, but at 5 GiB and below it stopped making progress instead of
+failing ([#2856]). The [memory usage page] has every run and how to reproduce them.
+
+[Comet 1.2.0]: /blog/2026/11/02/datafusion-comet-1.2.0/
+[Comet 1.1.0 began to measure]: /blog/2026/10/01/datafusion-comet-1.1.0/
+[memory usage page]: https://datafusion.apache.org/ballista/contributors-guide/benchmarking/memory.html
+[#2856]: https://github.com/apache/datafusion-ballista/issues/2856
+
+Executors are also easier to size and debug. They report their memory use in heartbeats and log how much memory
+they have allocated, including memory the pool does not track, and when a task runs out of memory, the error
+names the operators holding the most memory. Spill files now go to the executor's work directory, whose free
+space sets the spill limit by default. Executors also raise their open file limit at startup: a task holds a
+file for each shuffle partition it reads or writes, and with the common limit of 1024, TPC-H at 128 partitions
+or more failed with `Too many open files`.
+
+## Adaptive query execution
+
+AQE, the default since 55.0.0, replans a query between stages using statistics from the stages that have
+completed. This release gives it better statistics and lets it do more with them.
+
+### Better statistics, safer broadcasts
+
+A shuffle used to carry row counts and sizes but no distinct counts, so after a shuffle DataFusion estimated an
+inner join at the size of its smaller input. For a join on a foreign key that can be wrong by the join's whole
+fan-out. In Q21 at SF1000, a join estimated at 399,809 rows produced 151,688,609, AQE broadcast it, and the job
+failed.
+
+Two changes fix this. Shuffle writers now sketch the distinct count of each partitioning key with HyperLogLog
+(`ballista.shuffle.collect_distinct_counts`), so AQE estimates joins after a shuffle from distinct counts as
+well as row counts. And when a join's build side is itself an unexecuted join whose size is only an estimate,
+AQE now runs it as its own stage and measures it before deciding to broadcast it
+(`ballista.optimizer.stage_build_side`).
+
+### Doing less work
+
+- **Exchange reuse.** When a query reads the same subplan more than once, such as a common table expression
+ joined to itself or Q15's revenue view, AQE now runs its stage once and lets every reader share the output
+ (`ballista.planner.reuse_exchange.enabled`, on by default).
+- **Physical plans through AQE.** Clients that build their own physical plans and submit them directly, for
+ example with their own operators between DataFusion repartitions, used to get the static planner. They now go
+ through AQE, like SQL and DataFrame queries.
+- **Coalescing to one partition.** With `ballista.planner.coalesce.enabled`, a stage whose whole input fits in
+ one target-sized partition is now coalesced all the way down to a single task, as Spark does for final
+ aggregates, small joins and the sorts below an `ORDER BY`.
+- **Skew joins (opt-in).** With `ballista.planner.skew_join.enabled`, AQE splits a skewed partition of a
+ partitioned join into smaller tasks and replicates the matching partition of the other side, so one hot key
+ no longer serializes the join on a single task.
+- **Extension rules.** Projects that embed the scheduler can add their own rules to adaptive planning.
+
+The release also lays the groundwork for distributed runtime filters: executors can now build Bloom, IN-list,
+and min/max summaries of a join's build side and apply them to the probe side. Placing these filters in query
+plans will follow in a future release.
+
+### Correctness
+
+Much of the AQE work in this cycle was making replanning safe. A stage cancelled by a replan is now retired and
+its late tasks discarded, a stage that is rolled back discards its partial outputs before it runs again, a
+failed stage transition no longer drops the stage from the graph, and intermediate shuffle files are freed when
+an adaptive job succeeds. Aggregates above joins whose strategy is still undecided keep their partial and final
+phases, and semi joins with volatile expressions are no longer moved.
+
+## Robustness and operations
+
+### Shutting down cleanly
+
+The scheduler, executors and the Docker images now handle `SIGTERM`, so `docker stop` and Kubernetes pod
+termination reach Ballista instead of waiting out the grace period. Executors finish their running tasks before
+they exit. The scheduler stops accepting new jobs, over gRPC and Flight SQL alike, lets its active jobs finish,
+and then drains its connections, all within `--graceful-shutdown-timeout-seconds`.
+
+### A standalone result service
+
+Clients that cannot reach the executors, as is common on Kubernetes, used to fetch results through a proxy
+embedded in the scheduler, where every result byte competed with planning and scheduling. The new
+`ballista-result-service` is a stateless Arrow Flight service that forwards each fetch to the executor holding
+the partition, and can be scaled on its own behind a load balancer. The embedded proxy is deprecated and will
+be removed in 57.0.0.
+
+### Errors instead of wrong answers
+
+Several plans could previously return wrong results with no error. They now either return correct results or
+fail clearly:
+
+- A range repartition in a submitted physical plan was dropped, so a distributed global sort returned partitions
+ that were sorted but overlapping. It is now planned as a shuffle.
+- When a stage is split between tasks, a leaf operator the scheduler did not know how to restrict to a task's
+ partitions ran all of them, so some partitions were returned twice and others not at all. The scheduler now
+ fails the job, and projects with custom operators can teach it how to restrict them.
+- A task whose status update carried metrics the scheduler could not merge had its status dropped, and under
+ pull scheduling, tasks that did not fit in one poll response never ran and the job hung. Both are fixed, and an
+ unreadable shuffle block is now reported as a fetch failure so the stage that wrote it reruns.
+- Pull-based executors no longer take on more partitions than they have vcores.
+
+The TPC-H SF10 and TPC-DS SF1 suites now run against every code change before it merges, through a merge
+queue, in addition to the high-availability chaos tests added in 55.0.0.
+
+## Python
+
+The Python client has been rebuilt on DataFusion Python's foreign function interface. `BallistaSessionContext`
+now returns a standard `datafusion.SessionContext` with a Ballista query planner installed, so every operation
+that executes a plan, including `collect`, `to_pandas`, `write_parquet` and `EXPLAIN ANALYZE`, runs on the cluster
+without a wrapper in between. Python UDFs are supported, as is S3, and a pure-Python client is available as the
+`python-client` extra for environments where the native client does not fit.
+
+## Other changes
+
+- `DataFrame::cache()` materializes a DataFrame on the cluster, and later queries read it back.
+- `DESCRIBE` works on Ballista tables.
+- `ComposedLogicalExtensionCodec` combines logical extension codecs, so a project can use several.
+- The `spark-compat` feature has been removed.
+- S3 credentials no longer appear in debug and trace logs.
+
+## Upgrading
+
+The wire format between the scheduler and executors has changed (protocol version 4), so they must be upgraded
+together. The Python client's API changed as described above, and several scheduler and executor APIs gained
+fields. The [upgrade guide] lists every change and what to do about it.
+
+[upgrade guide]: https://datafusion.apache.org/ballista/upgrading/56.0.0.html
+
+## Thank You
+
+
+
+Thank you to all 20 contributors, including Andy Grove, Marko Milenković, Ville Brofeldt, Richard Durkee, Noah
+Kusaba, Samuel James, Akshay Chitneni, and everyone who contributed code, reviews, bug reports, and feedback.
+See the [changelog] for the complete list of changes and contributors.
+
+[changelog]: https://github.com/apache/datafusion-ballista/blob/56.0.0/docs/source/changelog/56.0.0.md
diff --git a/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_smallest.svg b/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_smallest.svg
new file mode 100644
index 00000000..04352802
--- /dev/null
+++ b/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_smallest.svg
@@ -0,0 +1,75 @@
+
diff --git a/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_time.svg b/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_time.svg
new file mode 100644
index 00000000..68882f11
--- /dev/null
+++ b/content/images/datafusion-ballista-56.0.0/tpch_q18_memory_time.svg
@@ -0,0 +1,119 @@
+
diff --git a/content/images/datafusion-ballista-56.0.0/tpch_sf1000_total.svg b/content/images/datafusion-ballista-56.0.0/tpch_sf1000_total.svg
new file mode 100644
index 00000000..40523103
--- /dev/null
+++ b/content/images/datafusion-ballista-56.0.0/tpch_sf1000_total.svg
@@ -0,0 +1,42 @@
+