Repository navigation
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks @marvelefe, this is a really nice cleanup. The pull and push loops are much easier to follow now, and I checked that the error and log messages, poll timing, and fetch settings all come through unchanged. The new FakeScheduler tests are great too. One thing I'd like to see before approving is inline on poll_job_until_successful, and the rest are small. If it's easy, could you also run the ballista client tests once with ballista.client.pull=true and the Substrait example end to end, and tick those boxes in the description?
| .. | ||
| }), | ||
| ) if !pipelined_output_locations.is_empty() => { | ||
| return Err(DataFusionError::NotImplemented(format!( |
There was a problem hiding this comment.
When this sees a running pipelined job it returns an error, but the job is still running on the cluster. Since nobody reads the final stage, it holds its vcores until the producer hits ballista.pipelined.registration_timeout_seconds. Could we call cancel_job before returning in the Running case? FakeScheduler already records cancellations, so poll_job_until_successful_rejects_pipelined_jobs could assert on scheduler.cancelled() too.
| /// | ||
| /// Pipelined jobs are not supported here, because their results stream while | ||
| /// the job runs. Use [DistributedQueryExec] to read them. | ||
| pub async fn poll_job_until_successful( |
There was a problem hiding this comment.
Small question on the new public surface. poll_job_until_successful only accepts the intercepted SchedulerClient, which is why the Substrait example had to switch to with_interceptor. Would it be worth making it generic over the client's transport, so embedders with their own SchedulerGrpcClient<T> can use it too? Happy either way if you'd rather keep it simple for now.
| io_retries_times, | ||
| io_retry_wait_time_ms, | ||
| registration_timeout, | ||
| ..fetch.clone() |
There was a problem hiding this comment.
Tiny one. This arm uses ..fetch.clone() but it breaks right after, so ..fetch should compile just like the Successful arm. Same at line 1217.
| Some(job_status::Status::Failed(err)) => { | ||
| return Err(job_failed(&job_id, err.error)); | ||
| } | ||
| Some(job_status::Status::Successful(successful)) => return Ok(successful), |
There was a problem hiding this comment.
Not a big deal, but the Substrait example used to log Job ... finished executing in ..., and this helper doesn't log on success. Could it log here like finish_successful_job does?
…and improve success logging
Which issue does this PR close?
Closes #2598.
Rationale for this change
execute_query_pullandexecute_query_pushduplicated connection setup, job-success handling and status messages. The Substrait example had a third copy of the status loop. This is a refactor; pull and push are both kept, as agreed in the issue thread.What changes are included in this PR?
distributed_query.rs: addsconnect_scheduler,finish_successful_job,log_job_waitandjob_failed.PipelinedFetch::for_jobbundles the fetch settings, which removes thetoo_many_argumentsallows.fetch_partitiontakesPipelinedFetch.distributed_query.rsandmod.rs: adds the publicpoll_job_until_successfulhelper and exportsSchedulerClient.standalone-substrait.rs: replaces the privatepoll_jobwith the helper.Are there any user-facing changes?
No behavior change for
DistributedQueryExec. The new public helper is additive. The Substrait example now sends gRPC metadata interceptor headers (empty by default) and returns an error for pipelined jobs.Verification
Lint,Clippy,Check for unused dependencies,Check Cargo.toml formatting,Check generated config docs are in sync,Check vendored DataFusion proto is in sync,cargo doc(CI, passing)distributed_queryandballista-coretests:test linux crates,macos test(CI, passing)--features ballista-scheduler/substrait): runs end to end, returns 8 rowsballistaclient tests withballista.client.pull=false(default):test linux ballista(CI), and locallyballistaclient tests withballista.client.pull=true: passes locally