Repository navigation
fix(scheduler): fix split UNION ALL stages over custom sources or with merging branches - #2852
NoahKusaba wants to merge 21 commits into
Conversation
A task that owns none of a leaf's partitions, such as one running another branch of a UnionExec, never executes it, so the leaf can be replaced by a zero-partition EmptyExec whatever it is. Leaves the scheduler doesn't know, such as a DataSourceExec over a custom source, no longer fail such a task, and need no resolver for it, which the standalone scheduler can't take. The static planner's dry run now empties these branches too, so a union branch over an UnresolvedShuffleExec stays in the union's stage instead of getting one of its own.
…unts The leaf a task runs none of was an EmptyExec, which DataFusion's proto codec decodes with one partition, so the union on the executor ran the wrong branch. Use a zero-partition MemorySourceConfig, as the EmptyExec and PlaceholderRowExec case already did, through one helper. Also refuse a UnionExec branch that keeps another partition count after restriction, such as one that collapses its input through a SortExec under the adaptive planner, instead of running the wrong branch.
Check for a leaf the task runs none of at the top of the walk, for every leaf, instead of after the resolver for unknown leaves and separately for EmptyExec and PlaceholderRowExec. A broadcast reader in a union branch another task owns is now emptied too, rather than kept whole. Also: document the union branch check on restrict_plan_to_partitions, word its error without a plural for one partition, and round-trip the serialization test through Ballista's codec.
…down to a task's partitions The adaptive planner replaces a stage that gives no rows with an EmptyExec of that stage's partition count. Restriction returned it unchanged, so a union branch kept its full count and the new union check failed the job. Each leaf now keeps one partition for each of the task's partitions in its own range, or none. Adds an end-to-end test of split unions under both planners, and notes in the upgrade guide that a union branch that collapses to one partition now fails under the adaptive planner instead of returning wrong rows.
…e workaround The executor runs every partition of a task's plan, so a branch that keeps partitions the task wasn't given runs them too. The error now names the query shapes the adaptive planner leaves in a union's stage, and the static planner as a workaround; the upgrade guide adds the LIMIT shapes.
…, and tighten the tests A shuffle reader or scan the task runs none of now restricts itself as it did before this branch, which a test from main expects of a range reader. Only a leaf Ballista does not list becomes an empty in-memory leaf. Tests: - check a multi-partition EmptyExec branch keeps its share through serialization, which the fix depends on; - check the union branch check on a SortExec, the shape the adaptive planner produces; - drop a resolver case the empty-leaf rule made redundant, and compare the unknown leaf's error in a union with its error alone; - describe nodes as `Name(partitions)` instead of tuples; - run both slices of the round-robin case through the repartition.
…y comments A split UNION ALL with a branch that merges its input affects anyone on the adaptive planner, not only users with nodes of their own, so it gets its own heading in the 56.0.0 upgrade guide, with the workaround. Also reflow a doc comment and use describe() for the last union assertion.
|
cc @andygrove, I think you probably understand this part of the codebase best. |
…pins - Mark the docs page and the crate docs experimental. - Document that a UNION ALL on a stage the adaptive planner splits can fail (a union of Iceberg scans, apache#2852) or return rows more than once (branches that merge their input, such as under ORDER BY, apache#2854), and that the static planner avoids both. - Test a split-stage UNION ALL of Iceberg scans under both planners. - Pin Ballista to the last commit on main before DataFusion 56, which iceberg-datafusion doesn't support yet, and iceberg-datafusion to its latest main. iceberg-rust stays at the revision iceberg-datafusion pins.
Since apache#2721 the scheduler refuses a stage split over a scan it can't restrict, so the guard keeps a query from failing only on a busy cluster rather than from returning duplicate rows. Ballista main now needs Rust 1.95 too. Say what apache#2852 does in the union limitation.
andygrove
left a comment
There was a problem hiding this comment.
Thanks for digging into this. Turning the adaptive planner's duplicate rows into an error is a real improvement, and the empty leaf needing no resolver is a nice touch. I left a few comments inline. The main one is that the static planner now keeps a multi-partition custom leaf inline in a union stage, which fails in tasks that work on main.
Since this changes stage shape for static-planner unions and the per-task plans for every split union stage, could you add the run-tpcds label so the TPC-DS SF1 suite runs at 1 and 4 partitions per task? Several TPC-DS queries are union-heavy (q2, q5, q49, q75, q77, q80). An EXPLAIN diff for one of them, before and after, would show the stage-count change. It would also help to know how many TPC-DS queries hit the new union-branch error under the adaptive planner with ballista.scheduler.max_partitions_per_task = 1.
… leaf The union-branch error no longer points every user at the adaptive planner. It names the cause it finds: a broadcast reader, a branch that merges its input into one partition, or a node that sets its own partition count, which is pointed at the resolver alone. Add a scheduler test that ships a split union over an unknown leaf through `launch_tasks` with no resolver, and an end-to-end test that an adaptive `UNION ALL ... ORDER BY` split one partition per task fails with that error. Update the stale doc comments on `restricts_to_nothing` and `select_output_partitions`.
… again The stage planner kept any branch the rewriter could restrict to no partitions in the union's stage. Since a leaf the scheduler doesn't know became an empty leaf, that took in a custom scan of several partitions, and a task owning part of it then failed. The planner now also requires the branch to restrict to its first partition alone, which such a leaf fails, so it gets a stage of its own as on main. A one-partition unknown leaf stays inline. An unresolved shuffle passes that check through a planner-only resolver, as the reader it becomes restricts to any slice, so a branch fed by a shuffle still stays in the union's stage.
…failing The adaptive planner can leave a union branch that keeps partitions however it is restricted, such as one that merges its input into one partition, in the union's stage. A task given part of that stage would also run the branch, so preparing it failed, but only when the stage happened to be split, which depends on free task slots. Binding now packs such a stage into one task, as it does a stage that collapses its input, so each branch runs once and the query succeeds. The task ignores max_partitions_per_task and takes one vcore per partition, up to what its executor has free. Binding judges a stage with the resolver its tasks are restricted with, carried on the job's cache entry, so a node that resolver restricts doesn't pack the stage. The task-time error stays as a backstop.
* feat(iceberg): scan Iceberg tables in Ballista * refactor(iceberg): rename the crate to ballista-iceberg * feat(iceberg): pin catalog-backed tables when a query is planned register_iceberg(state) wraps the session's catalogs so that DataFusion's async table lookup, which runs once per table a query names, loads each catalog-backed Iceberg table and returns it at that version. Every scan of a table in one query reads the same version, and the logical codec no longer loads tables, so it no longer blocks the client or needs a multi-threaded runtime. * feat(iceberg): mount Iceberg catalogs on the scheduler for SQL clients Flight SQL clients send SQL that the scheduler plans, so they hold no catalog. iceberg_session_builder builds the scheduler's sessions with configured catalogs mounted and readied with register_iceberg, so their tables are pinned while the scheduler plans. register_iceberg now wraps the codecs itself and leaves an already-ready session unchanged, so register_iceberg_codecs is gone. The docs now say a separately deployed scheduler and executors set the Iceberg codecs as their codec overrides, and that a view keeps the version from when it was created. * fix(iceberg): refuse metadata tables with a clear error A metadata table such as t$snapshots reached the inner codec, which failed with an internal error suggesting a DataFusion bug. The logical codec now refuses it as not supported on Ballista yet. * feat(iceberg): wrap the scheduler's own session builder iceberg_session_builder now takes the scheduler's existing session builder and wraps it, so a scheduler that sets up its sessions itself (for example to register object stores) keeps that setup. Also explains the dead_code allow on the shared test helpers. * docs(iceberg): say why PinnedTable isn't a static table provider * feat(iceberg): take and return the scheduler's SessionBuilder shape iceberg_session_builder now takes and returns Arc<dyn Fn>, the shape SchedulerConfig stores, so an existing builder passes straight in and the result goes straight to with_override_session_builder. The Flight SQL docs now link to mounting Iceberg catalogs. * fix(iceberg): keep the session ID in register_iceberg SessionStateBuilder::new_from_existing gives a session a new ID, so register_iceberg now keeps the original, as Ballista's own share_file_statistics_cache does. Also makes the SessionBuilder alias private: ballista-scheduler already exports one. * Frame Iceberg payloads with an "iceberg" prefix Payloads that aren't Iceberg's now pass to the inner codec untouched, so codecs for other table formats that follow the same convention can be stacked in any order. Document the convention in the user guide. * cleanup * Move ballista-iceberg out of the root workspace, add with_iceberg ballista-iceberg is now its own workspace, excluded from Ballista's like python/, with its own Cargo.lock and CI workflow. Ballista's root Cargo.lock matches main again, so a DataFusion upgrade that iceberg-rust doesn't support yet doesn't block Ballista's build or release. SchedulerConfigExt::with_iceberg (the `scheduler` feature) sets the Iceberg codecs and mounts catalogs on the scheduler's sessions, and ExecutorProcessConfigExt::with_iceberg (the `executor` feature) sets the codecs on an executor. Both wrap whatever the config already has. They replace iceberg_session_builder. * Limit the Iceberg workflow's token to reading contents * Compose the Iceberg codecs after the session's codecs * Fall back to Ballista's codecs in IcebergLogicalCodec::after * Correct the Iceberg docs on how the codecs are added * Re-pin a view's Iceberg tables each time a query names it * Move view re-pinning to a follow-up PR Reverts 9025298 so it can be reviewed on its own. Views keep the version from when they were created, documented as a limitation. * Pin ballista-iceberg's Ballista crates to a commit on main Depend on the Ballista crates at a commit that includes apache#2752 instead of the crates beside it, and stop running the workflow on changes under ballista/, so a dependency bump in Ballista can't break this build. * Refuse to encode an Iceberg scan with more than one partition Ballista restricts each task to its partitions only for the scans it knows, so each task would read every partition of an Iceberg scan and return duplicate rows. Fail loudly once a dependency bump starts splitting scans. * Document the catalog wrapper, the partition guard and the Ballista pin * Format ballista-iceberg's Cargo.toml with taplo * Send Iceberg tables without their metadata A table's metadata grows with each snapshot, and every scan of it carried a copy, so a self-join on a table with a long history could exceed the gRPC message limit. Send only the metadata file's location: the table cache keeps the extra read to once per version. * Say when the Iceberg table cache rebuilds a version again * Document that encrypted Iceberg tables aren't supported * Document calling with_iceberg once, with every catalog * Correct which operators above an Iceberg scan run in parallel * Pin ballista-iceberg to a Ballista commit with apache#2721 * Mark Iceberg support experimental, document split-union limits, bump pins - Mark the docs page and the crate docs experimental. - Document that a UNION ALL on a stage the adaptive planner splits can fail (a union of Iceberg scans, apache#2852) or return rows more than once (branches that merge their input, such as under ORDER BY, apache#2854), and that the static planner avoids both. - Test a split-stage UNION ALL of Iceberg scans under both planners. - Pin Ballista to the last commit on main before DataFusion 56, which iceberg-datafusion doesn't support yet, and iceberg-datafusion to its latest main. iceberg-rust stays at the revision iceberg-datafusion pins. * Correct the multi-partition guard's reason and a stale Rust version note Since apache#2721 the scheduler refuses a stage split over a scan it can't restrict, so the guard keeps a query from failing only on a busy cluster rather than from returning duplicate rows. Ballista main now needs Rust 1.95 too. Say what apache#2852 does in the union limitation. * Say a split UNION fails too, not only UNION ALL
…leaf # Conflicts: # docs/source/upgrading/56.0.0.md
…backstop - Planning and binding now ask the same question through `branch_splits`: can restriction give a task none of a union branch and part of it. Binding used to try only none, so under the adaptive planner a branch over a leaf of several partitions that nothing can restrict failed at task time; its stage now runs as one task. - The union-branch error is a backstop that binding makes unreachable, so it states the invariant instead of classifying causes. - Binding returns early with nothing pending and runs the union check only when the pending partitions don't fit in one slice anyway. - JobInfoCache's resolver is crate-private. - The Iceberg guide no longer lists split unions as a limitation.
- Share one sort fixture, `sorted_into_one_partition`, between the restriction and binding tests. - Shorten the unsplittable-union cases and move the resolver cases into a test of their own. - Drop planner predicate tests the stage-planning tests already cover, and reuse `project_leading_columns`. - Name the backstop test for what it guards, and compare the empty leaf by its full display. - Say why each group of end-to-end queries is there.
The split check now returns the first branch that can't be split rather than a bool, so binding can log it at info level with a pointer to the resolver, which a node of your own needs to let the stage be split. The shape test asserts which branch each case finds.
The planner checked a union branch with a resolver that treated an unresolved shuffle as the reader it becomes, which kept a node over a shuffle in the union's stage and changed 7 TPC-DS static plans. Neither fix needs that, so the planner now asks without a resolver, as andygrove suggested, and every static plan matches main again. The cache test goes back to main's stage numbers.
…tition shuffle - The docs said a union stage with a branch nothing can restrict runs as one task; that is the adaptive planner's behaviour, while the static planner gives the branch a stage of its own. - The resolver guide still said a split stage holding an unknown leaf fails; a task that runs all or none of it is fine. - A node over a one-partition shuffle stays in the union's stage, like a one-partition custom leaf; the planner test now says so.
|
Thanks for the review. TPC-DS SF1 (
I'll try to re-review this again tomorrow if I have time. |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @NoahKusaba, this came together really well. Running the stage as one task instead of failing means these queries now just work whether or not the stage is split, and #2854 can be about parallelism alone. I'm glad you went back to the strict planner check too. All 99 TPC-DS static plans matching main is a great result. I traced the new binding path for push and pull. The vcore accounting lines up with what the executor acquires and releases, and a retry of a packed task packs again. The upgrade note explains the trade-off clearly. One small optional suggestion inline. LGTM.
`bind_one` walked the stage plan, restricting every union branch twice, on each bind of a stage that doesn't fit in one slice. The answer depends only on the plan and the job's resolver, both fixed for a running stage, so the first bind that needs it stores it on `RunningStage` and later binds reuse it. A rerun builds a new running stage and so starts over. Leaves a TODO to do the same for `stage_has_input_collapse`, and makes the upgrade guide say which stage runs as one task.
|
Thanks for the review, addressed all comment. |
Which issue does this PR close?
Follows up #2679 / #2721. Found through andygrove's
UNION ALLrepro on #2638 (#2638 (comment)).Related: #2854. This PR makes those queries correct by running the stage as one task; #2854 can stay open for the planner fix that would restore its parallelism.
Rationale for this change
When a
UNION ALLstage is split between tasks, each task restricts the branches it doesn't own to no partitions. Two things went wrong:ORDER BY, aLIMITbranch, an aggregate withoutGROUP BY), on any source. Such a branch keeps its partition however it is restricted, so every task ran it.UNION ALLover IcebergUNION ALL … ORDER BY, aLIMITbranch, or aggregates withoutGROUP BY, adaptive plannerWhat changes are included in this PR?
MemorySourceConfig(anEmptyExecdecodes with one partition). No resolver needed, so standalone mode works too.branch_splits, decides whether a union branch can be split: can it be cut to none of its partitions and to its first alone?max_partitions_per_taskand taking a vcore per partition up to what the executor has free, and logs it atinfo, naming the branch. It judges with the configured resolver, whichJobInfoCachenow carries, and keeps the answer on theRunningStagefor later binds.EmptyExecandPlaceholderRowExecare cut to the task's partitions. The adaptive planner replaces an empty stage with anEmptyExecof several partitions; a one-partitionPlaceholderRowExecunder a round-robin repartition now gives its row once.Are these changes tested?
Yes. Unit tests cover the empty leaf and its serialization, cutting down
EmptyExecandPlaceholderRowExec, the static planner's stages (including andygrove'sUnionExec[scan(3), PayloadExec(2)]repro), which unions can be split, binding with and without a resolver, and a split union over a custom leaf launched throughprepare_task_definition.split_union.rscompares 18UNION ALLqueries with DataFusion under both planners at 1 and 2 partitions per task. Without the one-task binding, the adaptive cases fail.An earlier revision also passed the Iceberg case end to end (#2638) and 184 split-stage union queries under both planners; not rerun on this one.
Follow-ups
stage_has_input_collapseonRunningStage, as this PR does the union check: it walks the stage plan on every bind (TODO inbind_one).Are there any user-facing changes?
PlaceholderRowExecunder a round-robinRepartitionExecgives its row once, not once per task.The 56.0.0 upgrade guide,
extending-components.mdand the Iceberg guide are updated.