Skip to content

fix(scheduler): fix split UNION ALL stages over custom sources or with merging branches - #2852

Open
NoahKusaba wants to merge 21 commits into
apache:mainfrom
NoahKusaba:feat/empty-unknown-leaf
Open

NoahKusaba wants to merge 21 commits into
apache:mainfrom
NoahKusaba:feat/empty-unknown-leaf

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Follows up #2679 / #2721. Found through andygrove's UNION ALL repro on #2638 (#2638 (comment)).

Related: #2854. This PR makes those queries correct by running the stage as one task; #2854 can stay open for the planner fix that would restore its parallelism.

Rationale for this change

When a UNION ALL stage is split between tasks, each task restricts the branches it doesn't own to no partitions. Two things went wrong:

  1. A union over a custom source, such as Iceberg, failed, because a leaf the scheduler can't restrict fails the job unless the task runs all of it.
  2. Under the adaptive planner, some rows came back more than once when a branch merges its input into one partition (ORDER BY, a LIMIT branch, an aggregate without GROUP BY), on any source. Such a branch keeps its partition however it is restricted, so every task ran it.
Split-stage query main this PR
UNION ALL over Iceberg fails ✓
UNION ALL … ORDER BY, a LIMIT branch, or aggregates without GROUP BY, adaptive planner wrong rows ✓, as one task
The same, static planner ✓ ✓

What changes are included in this PR?

  • A leaf a task runs none of becomes an empty leaf, a zero-partition MemorySourceConfig (an EmptyExec decodes with one partition). No resolver needed, so standalone mode works too.
  • One check, branch_splits, decides whether a union branch can be split: can it be cut to none of its partitions and to its first alone?
    • The static planner keeps a branch inline only if it passes. A branch over a one-partition input it can't restrict, such as an Iceberg scan or a one-partition shuffle, now stays inline, since only the task that owns that partition runs it; one over several partitions keeps its own stage, as on main. All 99 TPC-DS static plans match main's.
    • Binding packs a stage with a failing branch into one task, ignoring max_partitions_per_task and taking a vcore per partition up to what the executor has free, and logs it at info, naming the branch. It judges with the configured resolver, which JobInfoCache now carries, and keeps the answer on the RunningStage for later binds.
  • EmptyExec and PlaceholderRowExec are cut to the task's partitions. The adaptive planner replaces an empty stage with an EmptyExec of several partitions; a one-partition PlaceholderRowExec under a round-robin repartition now gives its row once.
  • Restriction still refuses a branch that keeps its partitions, as a backstop should binding and restriction ever disagree.

Are these changes tested?

Yes. Unit tests cover the empty leaf and its serialization, cutting down EmptyExec and PlaceholderRowExec, the static planner's stages (including andygrove's UnionExec[scan(3), PayloadExec(2)] repro), which unions can be split, binding with and without a resolver, and a split union over a custom leaf launched through prepare_task_definition.

split_union.rs compares 18 UNION ALL queries with DataFusion under both planners at 1 and 2 partitions per task. Without the one-task binding, the adaptive cases fail.

An earlier revision also passed the Iceberg case end to end (#2638) and 184 split-stage union queries under both planners; not rerun on this one.

Follow-ups

Are there any user-facing changes?

  • A split union over a custom source succeeds.
  • Under the adaptive planner, a split union with a merging branch returns the right rows, as one task, with less parallelism.
  • In a split stage, with or without a union, a one-partition PlaceholderRowExec under a round-robin RepartitionExec gives its row once, not once per task.

The 56.0.0 upgrade guide, extending-components.md and the Iceberg guide are updated.

A task that owns none of a leaf's partitions, such as one running another
branch of a UnionExec, never executes it, so the leaf can be replaced by a
zero-partition EmptyExec whatever it is. Leaves the scheduler doesn't know,
such as a DataSourceExec over a custom source, no longer fail such a task,
and need no resolver for it, which the standalone scheduler can't take.

The static planner's dry run now empties these branches too, so a union
branch over an UnresolvedShuffleExec stays in the union's stage instead of
getting one of its own.
…unts

The leaf a task runs none of was an EmptyExec, which DataFusion's proto
codec decodes with one partition, so the union on the executor ran the
wrong branch. Use a zero-partition MemorySourceConfig, as the EmptyExec
and PlaceholderRowExec case already did, through one helper.

Also refuse a UnionExec branch that keeps another partition count after
restriction, such as one that collapses its input through a SortExec under
the adaptive planner, instead of running the wrong branch.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 11, 2026
Check for a leaf the task runs none of at the top of the walk, for every
leaf, instead of after the resolver for unknown leaves and separately for
EmptyExec and PlaceholderRowExec. A broadcast reader in a union branch
another task owns is now emptied too, rather than kept whole.

Also: document the union branch check on restrict_plan_to_partitions, word
its error without a plural for one partition, and round-trip the
serialization test through Ballista's codec.
…down to a task's partitions

The adaptive planner replaces a stage that gives no rows with an EmptyExec
of that stage's partition count. Restriction returned it unchanged, so a
union branch kept its full count and the new union check failed the job.
Each leaf now keeps one partition for each of the task's partitions in its
own range, or none.

Adds an end-to-end test of split unions under both planners, and notes in
the upgrade guide that a union branch that collapses to one partition now
fails under the adaptive planner instead of returning wrong rows.
…e workaround

The executor runs every partition of a task's plan, so a branch that keeps
partitions the task wasn't given runs them too. The error now names the
query shapes the adaptive planner leaves in a union's stage, and the static
planner as a workaround; the upgrade guide adds the LIMIT shapes.
…, and tighten the tests

A shuffle reader or scan the task runs none of now restricts itself as it
did before this branch, which a test from main expects of a range reader.
Only a leaf Ballista does not list becomes an empty in-memory leaf.

Tests:
- check a multi-partition EmptyExec branch keeps its share through
  serialization, which the fix depends on;
- check the union branch check on a SortExec, the shape the adaptive planner
  produces;
- drop a resolver case the empty-leaf rule made redundant, and compare the
  unknown leaf's error in a union with its error alone;
- describe nodes as `Name(partitions)` instead of tuples;
- run both slices of the round-robin case through the repartition.
…y comments

A split UNION ALL with a branch that merges its input affects anyone on the
adaptive planner, not only users with nodes of their own, so it gets its own
heading in the 56.0.0 upgrade guide, with the workaround. Also reflow a doc
comment and use describe() for the last union assertion.
@NoahKusaba

Copy link
Copy Markdown
Contributor Author

cc @andygrove, I think you probably understand this part of the codebase best.
The actual change itself is pretty minimal, but there is alot of doc / test noise.

NoahKusaba added a commit to NoahKusaba/datafusion-ballista that referenced this pull request Nov 14, 2026
…pins

- Mark the docs page and the crate docs experimental.
- Document that a UNION ALL on a stage the adaptive planner splits can fail
  (a union of Iceberg scans, apache#2852) or return rows more than once (branches
  that merge their input, such as under ORDER BY, apache#2854), and that the
  static planner avoids both.
- Test a split-stage UNION ALL of Iceberg scans under both planners.
- Pin Ballista to the last commit on main before DataFusion 56, which
  iceberg-datafusion doesn't support yet, and iceberg-datafusion to its
  latest main. iceberg-rust stays at the revision iceberg-datafusion pins.
NoahKusaba added a commit to NoahKusaba/datafusion-ballista that referenced this pull request Nov 14, 2026
Since apache#2721 the scheduler refuses a stage split over a scan it can't
restrict, so the guard keeps a query from failing only on a busy
cluster rather than from returning duplicate rows. Ballista main now
needs Rust 1.95 too. Say what apache#2852 does in the union limitation.
@andygrove andygrove mentioned this pull request Nov 15, 2026
7 of 15 tasks

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into this. Turning the adaptive planner's duplicate rows into an error is a real improvement, and the empty leaf needing no resolver is a nice touch. I left a few comments inline. The main one is that the static planner now keeps a multi-partition custom leaf inline in a union stage, which fails in tasks that work on main.

Since this changes stage shape for static-planner unions and the per-task plans for every split union stage, could you add the run-tpcds label so the TPC-DS SF1 suite runs at 1 and 4 partitions per task? Several TPC-DS queries are union-heavy (q2, q5, q49, q75, q77, q80). An EXPLAIN diff for one of them, before and after, would show the stage-count change. It would also help to know how many TPC-DS queries hit the new union-branch error under the adaptive planner with ballista.scheduler.max_partitions_per_task = 1.

Comment thread ballista/client/tests/split_union.rs Resolved
… leaf

The union-branch error no longer points every user at the adaptive planner.
It names the cause it finds: a broadcast reader, a branch that merges its
input into one partition, or a node that sets its own partition count, which
is pointed at the resolver alone.

Add a scheduler test that ships a split union over an unknown leaf through
`launch_tasks` with no resolver, and an end-to-end test that an adaptive
`UNION ALL ... ORDER BY` split one partition per task fails with that error.
Update the stale doc comments on `restricts_to_nothing` and
`select_output_partitions`.
… again

The stage planner kept any branch the rewriter could restrict to no
partitions in the union's stage. Since a leaf the scheduler doesn't know
became an empty leaf, that took in a custom scan of several partitions,
and a task owning part of it then failed. The planner now also requires
the branch to restrict to its first partition alone, which such a leaf
fails, so it gets a stage of its own as on main. A one-partition unknown
leaf stays inline.

An unresolved shuffle passes that check through a planner-only resolver,
as the reader it becomes restricts to any slice, so a branch fed by a
shuffle still stays in the union's stage.
…failing

The adaptive planner can leave a union branch that keeps partitions however
it is restricted, such as one that merges its input into one partition, in
the union's stage. A task given part of that stage would also run the branch,
so preparing it failed, but only when the stage happened to be split, which
depends on free task slots.

Binding now packs such a stage into one task, as it does a stage that
collapses its input, so each branch runs once and the query succeeds. The
task ignores max_partitions_per_task and takes one vcore per partition, up to
what its executor has free. Binding judges a stage with the resolver its
tasks are restricted with, carried on the job's cache entry, so a node that
resolver restricts doesn't pack the stage. The task-time error stays as a
backstop.
marvelefe pushed a commit to marvelefe/datafusion-ballista that referenced this pull request Nov 16, 2026
* feat(iceberg): scan Iceberg tables in Ballista

* refactor(iceberg): rename the crate to ballista-iceberg

* feat(iceberg): pin catalog-backed tables when a query is planned

register_iceberg(state) wraps the session's catalogs so that DataFusion's
async table lookup, which runs once per table a query names, loads each
catalog-backed Iceberg table and returns it at that version. Every scan of
a table in one query reads the same version, and the logical codec no longer
loads tables, so it no longer blocks the client or needs a multi-threaded
runtime.

* feat(iceberg): mount Iceberg catalogs on the scheduler for SQL clients

Flight SQL clients send SQL that the scheduler plans, so they hold no
catalog. iceberg_session_builder builds the scheduler's sessions with
configured catalogs mounted and readied with register_iceberg, so their
tables are pinned while the scheduler plans.

register_iceberg now wraps the codecs itself and leaves an already-ready
session unchanged, so register_iceberg_codecs is gone. The docs now say a
separately deployed scheduler and executors set the Iceberg codecs as their
codec overrides, and that a view keeps the version from when it was
created.

* fix(iceberg): refuse metadata tables with a clear error

A metadata table such as t$snapshots reached the inner codec, which failed
with an internal error suggesting a DataFusion bug. The logical codec now
refuses it as not supported on Ballista yet.

* feat(iceberg): wrap the scheduler's own session builder

iceberg_session_builder now takes the scheduler's existing session builder
and wraps it, so a scheduler that sets up its sessions itself (for example
to register object stores) keeps that setup. Also explains the dead_code
allow on the shared test helpers.

* docs(iceberg): say why PinnedTable isn't a static table provider

* feat(iceberg): take and return the scheduler's SessionBuilder shape

iceberg_session_builder now takes and returns Arc<dyn Fn>, the shape
SchedulerConfig stores, so an existing builder passes straight in and the
result goes straight to with_override_session_builder. The Flight SQL docs
now link to mounting Iceberg catalogs.

* fix(iceberg): keep the session ID in register_iceberg

SessionStateBuilder::new_from_existing gives a session a new ID, so
register_iceberg now keeps the original, as Ballista's own
share_file_statistics_cache does. Also makes the SessionBuilder alias
private: ballista-scheduler already exports one.

* Frame Iceberg payloads with an "iceberg" prefix

Payloads that aren't Iceberg's now pass to the inner codec untouched,
so codecs for other table formats that follow the same convention can
be stacked in any order. Document the convention in the user guide.

* cleanup

* Move ballista-iceberg out of the root workspace, add with_iceberg

ballista-iceberg is now its own workspace, excluded from Ballista's like
python/, with its own Cargo.lock and CI workflow. Ballista's root
Cargo.lock matches main again, so a DataFusion upgrade that iceberg-rust
doesn't support yet doesn't block Ballista's build or release.

SchedulerConfigExt::with_iceberg (the `scheduler` feature) sets the
Iceberg codecs and mounts catalogs on the scheduler's sessions, and
ExecutorProcessConfigExt::with_iceberg (the `executor` feature) sets the
codecs on an executor. Both wrap whatever the config already has. They
replace iceberg_session_builder.

* Limit the Iceberg workflow's token to reading contents

* Compose the Iceberg codecs after the session's codecs

* Fall back to Ballista's codecs in IcebergLogicalCodec::after

* Correct the Iceberg docs on how the codecs are added

* Re-pin a view's Iceberg tables each time a query names it

* Move view re-pinning to a follow-up PR

Reverts 9025298 so it can be reviewed on its own. Views keep the version
from when they were created, documented as a limitation.

* Pin ballista-iceberg's Ballista crates to a commit on main

Depend on the Ballista crates at a commit that includes apache#2752 instead of
the crates beside it, and stop running the workflow on changes under
ballista/, so a dependency bump in Ballista can't break this build.

* Refuse to encode an Iceberg scan with more than one partition

Ballista restricts each task to its partitions only for the scans it
knows, so each task would read every partition of an Iceberg scan and
return duplicate rows. Fail loudly once a dependency bump starts
splitting scans.

* Document the catalog wrapper, the partition guard and the Ballista pin

* Format ballista-iceberg's Cargo.toml with taplo

* Send Iceberg tables without their metadata

A table's metadata grows with each snapshot, and every scan of it carried
a copy, so a self-join on a table with a long history could exceed the
gRPC message limit. Send only the metadata file's location: the table
cache keeps the extra read to once per version.

* Say when the Iceberg table cache rebuilds a version again

* Document that encrypted Iceberg tables aren't supported

* Document calling with_iceberg once, with every catalog

* Correct which operators above an Iceberg scan run in parallel

* Pin ballista-iceberg to a Ballista commit with apache#2721

* Mark Iceberg support experimental, document split-union limits, bump pins

- Mark the docs page and the crate docs experimental.
- Document that a UNION ALL on a stage the adaptive planner splits can fail
  (a union of Iceberg scans, apache#2852) or return rows more than once (branches
  that merge their input, such as under ORDER BY, apache#2854), and that the
  static planner avoids both.
- Test a split-stage UNION ALL of Iceberg scans under both planners.
- Pin Ballista to the last commit on main before DataFusion 56, which
  iceberg-datafusion doesn't support yet, and iceberg-datafusion to its
  latest main. iceberg-rust stays at the revision iceberg-datafusion pins.

* Correct the multi-partition guard's reason and a stale Rust version note

Since apache#2721 the scheduler refuses a stage split over a scan it can't
restrict, so the guard keeps a query from failing only on a busy
cluster rather than from returning duplicate rows. Ballista main now
needs Rust 1.95 too. Say what apache#2852 does in the union limitation.

* Say a split UNION fails too, not only UNION ALL
…leaf

# Conflicts:
#	docs/source/upgrading/56.0.0.md
…backstop

- Planning and binding now ask the same question through `branch_splits`:
  can restriction give a task none of a union branch and part of it. Binding
  used to try only none, so under the adaptive planner a branch over a leaf
  of several partitions that nothing can restrict failed at task time; its
  stage now runs as one task.
- The union-branch error is a backstop that binding makes unreachable, so it
  states the invariant instead of classifying causes.
- Binding returns early with nothing pending and runs the union check only
  when the pending partitions don't fit in one slice anyway.
- JobInfoCache's resolver is crate-private.
- The Iceberg guide no longer lists split unions as a limitation.
- Share one sort fixture, `sorted_into_one_partition`, between the
  restriction and binding tests.
- Shorten the unsplittable-union cases and move the resolver cases into a
  test of their own.
- Drop planner predicate tests the stage-planning tests already cover, and
  reuse `project_leading_columns`.
- Name the backstop test for what it guards, and compare the empty leaf by
  its full display.
- Say why each group of end-to-end queries is there.
The split check now returns the first branch that can't be split rather
than a bool, so binding can log it at info level with a pointer to the
resolver, which a node of your own needs to let the stage be split. The
shape test asserts which branch each case finds.
The planner checked a union branch with a resolver that treated an
unresolved shuffle as the reader it becomes, which kept a node over a
shuffle in the union's stage and changed 7 TPC-DS static plans. Neither
fix needs that, so the planner now asks without a resolver, as andygrove
suggested, and every static plan matches main again. The cache test
goes back to main's stage numbers.
…tition shuffle

- The docs said a union stage with a branch nothing can restrict runs as
  one task; that is the adaptive planner's behaviour, while the static
  planner gives the branch a stage of its own.
- The resolver guide still said a split stage holding an unknown leaf
  fails; a task that runs all or none of it is fine.
- A node over a one-partition shuffle stays in the union's stage, like a
  one-partition custom leaf; the planner test now says so.
@NoahKusaba

NoahKusaba commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review. TPC-DS SF1 (tpcgen-cli), standalone, 16 partitions, prefer_hash_join=false as in CI, main vs this branch:

  • Static planner: all 99 distributed plans match main's.
  • Adaptive planner, max_partitions_per_task = 1: all 19 union queries match DataFusion and none fails. One stage runs as one task: q49's three ranked branches, 3 partitions. Main also gets all 19 right, since q49's UNION drops the duplicates; the bug needs a UNION ALL with a merging branch, as in split_union.rs.
  • run-tpcds: I can't add labels here, could you? It runs only the static planner, whose plans don't change.

I'll try to re-review this again tomorrow if I have time.

@NoahKusaba NoahKusaba changed the title fix(scheduler): restrict a leaf a task runs none of to an empty leaf fix(scheduler): fix split UNION ALL stages over custom sources or with merging branches Nov 17, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @NoahKusaba, this came together really well. Running the stage as one task instead of failing means these queries now just work whether or not the stage is split, and #2854 can be about parallelism alone. I'm glad you went back to the strict planner check too. All 99 TPC-DS static plans matching main is a great result. I traced the new binding path for push and pull. The vcore accounting lines up with what the executor acquires and releases, and a retry of a packed task packs again. The upgrade note explains the trade-off clearly. One small optional suggestion inline. LGTM.

Comment thread ballista/scheduler/src/cluster/mod.rs Resolved
`bind_one` walked the stage plan, restricting every union branch twice,
on each bind of a stage that doesn't fit in one slice. The answer
depends only on the plan and the job's resolver, both fixed for a
running stage, so the first bind that needs it stores it on
`RunningStage` and later binds reuse it. A rerun builds a new running
stage and so starts over.

Leaves a TODO to do the same for `stage_has_input_collapse`, and makes
the upgrade guide say which stage runs as one task.
@NoahKusaba

Copy link
Copy Markdown
Contributor Author

Thanks for the review, addressed all comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation run-tpcds

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants