Skip to content

refactor(core): share job setup and status handling in DistributedQueryExec - #2884

Open
marvelefe wants to merge 2 commits into
apache:mainfrom
marvelefe:main
Open

marvelefe wants to merge 2 commits into
apache:mainfrom
marvelefe:main

Conversation

@marvelefe

@marvelefe marvelefe commented Nov 17, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Closes #2598.

Rationale for this change

execute_query_pull and execute_query_push duplicated connection setup, job-success handling and status messages. The Substrait example had a third copy of the status loop. This is a refactor; pull and push are both kept, as agreed in the issue thread.

What changes are included in this PR?

  • distributed_query.rs: adds connect_scheduler, finish_successful_job, log_job_wait and job_failed. PipelinedFetch::for_job bundles the fetch settings, which removes the too_many_arguments allows. fetch_partition takes PipelinedFetch.
  • distributed_query.rs and mod.rs: adds the public poll_job_until_successful helper and exports SchedulerClient.
  • standalone-substrait.rs: replaces the private poll_job with the helper.
  • Tests: scripted scheduler covering success (timing gauges), failure and polling for pull and push.

Are there any user-facing changes?

No behavior change for DistributedQueryExec. The new public helper is additive. The Substrait example now sends gRPC metadata interceptor headers (empty by default) and returns an error for pipelined jobs.

Verification

  • Lint: Lint, Clippy, Check for unused dependencies, Check Cargo.toml formatting, Check generated config docs are in sync, Check vendored DataFusion proto is in sync, cargo doc (CI, passing)
  • distributed_query and ballista-core tests: test linux crates, macos test (CI, passing)
  • Substrait example run (--features ballista-scheduler/substrait): runs end to end, returns 8 rows
  • ballista client tests with ballista.client.pull=false (default): test linux ballista (CI), and locally
  • ballista client tests with ballista.client.pull=true: passes locally

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @marvelefe, this is a really nice cleanup. The pull and push loops are much easier to follow now, and I checked that the error and log messages, poll timing, and fetch settings all come through unchanged. The new FakeScheduler tests are great too. One thing I'd like to see before approving is inline on poll_job_until_successful, and the rest are small. If it's easy, could you also run the ballista client tests once with ballista.client.pull=true and the Substrait example end to end, and tick those boxes in the description?

..
}),
) if !pipelined_output_locations.is_empty() => {
return Err(DataFusionError::NotImplemented(format!(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When this sees a running pipelined job it returns an error, but the job is still running on the cluster. Since nobody reads the final stage, it holds its vcores until the producer hits ballista.pipelined.registration_timeout_seconds. Could we call cancel_job before returning in the Running case? FakeScheduler already records cancellations, so poll_job_until_successful_rejects_pipelined_jobs could assert on scheduler.cancelled() too.

///
/// Pipelined jobs are not supported here, because their results stream while
/// the job runs. Use [DistributedQueryExec] to read them.
pub async fn poll_job_until_successful(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small question on the new public surface. poll_job_until_successful only accepts the intercepted SchedulerClient, which is why the Substrait example had to switch to with_interceptor. Would it be worth making it generic over the client's transport, so embedders with their own SchedulerGrpcClient<T> can use it too? Happy either way if you'd rather keep it simple for now.

io_retries_times,
io_retry_wait_time_ms,
registration_timeout,
..fetch.clone()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiny one. This arm uses ..fetch.clone() but it breaks right after, so ..fetch should compile just like the Successful arm. Same at line 1217.

Some(job_status::Status::Failed(err)) => {
return Err(job_failed(&job_id, err.error));
}
Some(job_status::Status::Successful(successful)) => return Ok(successful),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal, but the Substrait example used to log Job ... finished executing in ..., and this helper doesn't log on success. Could it log here like finish_successful_job does?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deduplicate execute_query_pull and execute_query_push in DistributedQueryExec

2 participants