Repository navigation
fix(executor): retain private default TempDir for executor lifetime - #2775
aetherxeg-source wants to merge 4 commits into
Conversation
Fixes apache#2767. Applies the SHA-256-qualified 12-file R4 source patch at the pinned Apache main 1aad53c; removes the staging patch. Source patch SHA-256: 838a6f3b447f62a0a7cc523e021f76be69d1dcfe17c130078c47e58361a80508
|
Hi Ballista maintainers, This PR addresses #2767 and is currently at HEAD The targeted regression suite passed 13/13 tests in the linked qualification run, along with cargo check, Clippy, and formatting checks. I noticed that the upstream PR currently shows the The signed individual ICLA was submitted to the ASF Secretary on November 4; formal acknowledgement is still pending. I'm happy to address any technical feedback, particularly around the writer-lifetime and shutdown behavior. Thank you, |
andygrove
left a comment
There was a problem hiding this comment.
Thanks for picking this up! I think the fix for #2767 can be a lot smaller though. The issue is just that the TempDir guard gets dropped right after we read its path, so holding onto it in start_executor_process_inner until the function returns should give us a 0700 directory for the executor's whole lifetime and remove it on exit.
The lease registry adds a process-wide mutex that every shuffle write goes through, and it lets shuffle writer structs hold leases for as long as anything keeps a reference to the plan. If one of those outlives the task, shutdown waits out the graceful timeout and then keeps the directory around, which brings back the leftover-files problem from the issue. I'd rather not take on that global state in ballista-core to cover a writer that races process exit.
Could we shrink this down to keeping the TempDir alive (plus the explicit 0700 set_permissions, which is a nice touch) and one test that checks the directory exists with 0700 after startup? If there's a concrete case where a late writer still matters after that, I'm happy to look at it as a follow-up with a reproducer. Could you also update the description to follow the PR template?
|
@andygrove Thanks for the clear review. I've updated PR #2775 at
Local WSL2 validation: 101/101 executor library tests passed, Clippy with Would you mind taking another look when convenient? The upstream Actions runs for this commit still show Thanks again for the guidance. |
|
@andygrove Following up on the narrowed fix for #2767, now at The revised PR contains a one-file change: keep the default The revised candidate passed 101/101 local executor library tests, Clippy ( Would you or another Ballista maintainer be able to re-review the reduced patch and approve the pending CI runs, if appropriate? I will address any specific findings on this same PR. Thank you for the earlier guidance. AETHER X GLOBAL |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @aetherxeg-source, this is exactly the shape I was hoping for. Keeping the guard alive for the executor's lifetime fixes #2767 without any new global state, and the explicit 0700 is a nice belt-and-braces touch. I've approved the CI runs, so once they're green this is good to go. One tiny optional suggestion inline.
| return Err(BallistaError::Configuration( | ||
| "Unable to bind work dir".to_string(), | ||
| )); | ||
| let (work_dir, _default_work_dir_owner) = match opt.work_dir.clone() { |
There was a problem hiding this comment.
Tiny suggestion. This binding is the whole fix, and renaming it to _ would quietly drop the guard again without the test noticing. Could you add a short comment here, something like "keep the guard alive until the executor exits, or the directory is deleted at startup"?
Which issue does this PR close?
Closes #2767.
Rationale for this change
When no
--work-diris configured,start_executor_process_innerpreviously converted a freshly createdTempDirto a path string and immediately dropped its owner. The private directory was consequently removed before normal executor startup completed. Later shuffle activity could recreate the path with process-default permissions and without automatic cleanup.What changes are included in this PR?
TempDirowner in scope for the async executor lifecycle, so the automatically created directory remains present and is cleaned up when the function exits.0700permissions for the default directory..await, private permissions, and deletion on owner drop.WorkDirActivity/WorkDirLeaseinfrastructure and shuffle-writer changes, in response to the maintainer's review. Late-writer races, if separately reproducible, are out of scope.Are there any user-facing changes?
An executor running without an explicit work directory now retains a private
0700work directory while it is running and removes it on normal process-function exit. This does not introduce a public API or protocol change.Validation
Local Linux/WSL2 qualification on 2026-11-06, from the exact proposed minimal working-tree candidate (upstream base
1aad53ce9b1a25c18d4f143a07be2995740191fb):cargo test -p ballista-executor --lib -- --test-threads=2: 101 passed, 0 failed, including the new default work-directory regression test.cargo clippy -p ballista-executor --lib --tests -- -D warnings: PASS.cargo fmt --all -- --check: PASS.git diff --check: PASS./healthzPASS; executor registration and/readyzPASS; default directory still present with mode 0700 after registration; executor clean exit 0; directory removed after exit; finalPOST_REGISTRATION_QUALIFICATION=PASS.The post-registration check was an external local Python integration harness, not a committed Rust test and not upstream GitHub Actions CI. Upstream CI and maintainer acceptance remain pending; no test here claims to cover unrelated detached-writer races or all platforms.
Contribution
AETHER X GLOBAL (
@aetherxeg-source).