Repository navigation
fix(scheduler): fail an AQE job that loses a finished stage's output instead of hanging - #2887
abstractdog wants to merge 1 commit into
Conversation
…instead of hanging When an executor holding the shuffle output of a finished stage is lost, or a reduce task fails to fetch that output, the adaptive graph turned the stage back into a running one to rerun it. Its planner had already dropped the stage when it finished, though, so the first rerun task to report failed with "Can't find active stage to update stage outputs". The task manager drops the whole status update on that error, and the rolled-back reader stage is never rebuilt either, so the job hung with no running tasks until cancelled (apache#2539). Rerunning a finished stage under AQE is not supported yet (apache#1986), so fail the job instead, with a message that points to retrying the query or to the static planner, which recovers lost output: - reset_stages_on_lost_executor returns an error when the lost executor held output a finished stage's readers still need; the task manager already fails the job on that error. - A fetch failure that would resubmit a finished stage fails that stage, and with it the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andygrove
left a comment
There was a problem hiding this comment.
Thanks @abstractdog, this is a nice fix. The root-cause write-up and the log excerpt make the hang really easy to follow. Failing the job with a message that points at the static planner is much better than hanging forever. The new tests fail without the change, which is great to see. A couple of small suggestions inline, none of them blocking.
| for stage_id in mem::take(&mut updated_stages.resubmit_successful_stages) { | ||
| updated_stages | ||
| .failed_stages | ||
| .insert(stage_id, cannot_rerun_finished_stage(stage_id)); |
There was a problem hiding this comment.
Small wording thing. On the fetch-failure path the job error ends up as Job failed due to stage 0 failed: the output of finished stage 0 was lost .... Stage 0 actually succeeded. It was the reader stage whose task failed. Would it read better to key this entry on the reader stage, or to adjust the message so it doesn't say stage 0 failed?
| let task = graph.pop_next_task(&b.id)?.unwrap(); | ||
| let status = mock_failed_task(task, fetch_partition_failure("a", 0, 0)); | ||
|
|
||
| graph.update_task_status(&b, vec![status], 4, 4)?; |
There was a problem hiding this comment.
Could this also assert that update_task_status returns a QueryStageSchedulerEvent::JobRunningFailed? That event is what actually aborts the job and cancels its tasks. Checking it would guard the "no longer hangs" part directly, since the status alone could be set without the event going out.
DISCLAIMER: mostly generated code, not yet checked by myself
Which issue does this PR close?
Part of #1986. Addresses the hang reported in #2539, which this turns into a job failure; see below for what it leaves open.
Rationale for this change
With AQE, a job hangs forever once it loses the shuffle output of a stage that has already finished. I hit this on
main(6ee64d6) running TPC-DS SF10000 on 12 executors, with push-staged scheduling: several executors were evicted mid-query, and q40 then sat with no running tasks until a client-side timeout cancelled it 27 minutes later. The scheduler log matches #2539:So the hang does not depend on pull-staged mode.
The cause: when a stage finishes, the
AdaptivePlannerdrops it fromrunnable_stage_cacheandrunnable_stage_output. Losing that stage's output, through a lost executor (reset_stages_on_lost_executor) or a reduce task'sFetchPartitionError, makes the graph callrerun_successful_stage. That turns the stage back into a running one in the graph only. The first rerun task to report then fails inupdate_exchange_locationswith "Can't find active stage to update stage outputs". The task manager drops the whole status update on that error, so the stage never completes. Even without that error,update_stage_progressonly adds stages the graph does not have yet, so the rolled-back reader stage would never be rebuilt.Rerunning a finished stage under AQE is the missing recovery that #1986 tracks. Until it exists, failing the job with a clear message is better than hanging it: the client gets an error it can act on, and the job releases its resources.
What changes are included in this PR?
reset_stages_on_lost_executorreturns an error when the lost executor held output that a finished stage's readers still need.TaskManager::executor_lostalready fails the job when this returns an error.processing_stages_updateturns each finished stage that a fetch failure would resubmit into a failed stage, so the job fails through the usual path.ballista.planner.adaptive.enabledtofalseto use the static planner, which recovers lost output.AdaptiveExecutionGraph::rerun_successful_stagestays, as theExecutionGraphtrait requires it, but the adaptive graph no longer calls it.aqe/test/job_failure.rsfor both paths.losing_a_shared_stage_output_rolls_back_the_stage_onceinaqe/test/reuse_exchange.rsrecorded the old behaviour; it is renamed tolosing_a_shared_stage_output_fails_the_rollbackand now expects the error. All three fail without the change.Not addressed here: actually rerunning the stage (#1986), and the stale tasks #2539 reports in pull-staged mode, which a rolled-back stage cannot cancel.
Are there any user-facing changes?
Yes. An AQE job that loses the output of a finished stage now fails with the error above instead of hanging. The AQE limitations in the tuning guide say so. Losing only the output of running stages is handled as before.
🤖 Generated with Claude Code