Skip to content

fix(executor): retain private default TempDir for executor lifetime - #2775

Open
aetherxeg-source wants to merge 4 commits into
apache:mainfrom
AETHERXGLOBAL:fix/2767-private-workdir-writer-quiescence
Open

aetherxeg-source wants to merge 4 commits into
apache:mainfrom
AETHERXGLOBAL:fix/2767-private-workdir-writer-quiescence

Conversation

@aetherxeg-source

@aetherxeg-source aetherxeg-source commented Nov 4, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Closes #2767.

Rationale for this change

When no --work-dir is configured, start_executor_process_inner previously converted a freshly created TempDir to a path string and immediately dropped its owner. The private directory was consequently removed before normal executor startup completed. Later shuffle activity could recreate the path with process-default permissions and without automatic cleanup.

What changes are included in this PR?

  • Keep the TempDir owner in scope for the async executor lifecycle, so the automatically created directory remains present and is cleaned up when the function exits.
  • Explicitly enforce Unix 0700 permissions for the default directory.
  • Leave caller-supplied work directories caller-owned, with no automatic deletion or permission changes.
  • Add one focused Unix regression unit test covering owner lifetime across an .await, private permissions, and deletion on owner drop.
  • Remove the previously proposed process-wide WorkDirActivity/WorkDirLease infrastructure and shuffle-writer changes, in response to the maintainer's review. Late-writer races, if separately reproducible, are out of scope.

Are there any user-facing changes?

An executor running without an explicit work directory now retains a private 0700 work directory while it is running and removes it on normal process-function exit. This does not introduce a public API or protocol change.

Validation

Local Linux/WSL2 qualification on 2026-11-06, from the exact proposed minimal working-tree candidate (upstream base 1aad53ce9b1a25c18d4f143a07be2995740191fb):

  • cargo test -p ballista-executor --lib -- --test-threads=2: 101 passed, 0 failed, including the new default work-directory regression test.
  • cargo clippy -p ballista-executor --lib --tests -- -D warnings: PASS.
  • cargo fmt --all -- --check: PASS.
  • git diff --check: PASS.
  • Standalone executor binary startup smoke test against an unavailable scheduler: default work directory present during startup with mode 0700, removed on process exit. The expected scheduler-connect failure is not treated as a successful scheduler registration.
  • Live, local Scheduler + Executor integration: scheduler /healthz PASS; executor registration and /readyz PASS; default directory still present with mode 0700 after registration; executor clean exit 0; directory removed after exit; final POST_REGISTRATION_QUALIFICATION=PASS.

The post-registration check was an external local Python integration harness, not a committed Rust test and not upstream GitHub Actions CI. Upstream CI and maintainer acceptance remain pending; no test here claims to cover unrelated detached-writer races or all platforms.

Contribution

AETHER X GLOBAL (@aetherxeg-source).

Fixes apache#2767.

Applies the SHA-256-qualified 12-file R4 source patch at the pinned Apache main
1aad53c; removes the staging patch.

Source patch SHA-256: 838a6f3b447f62a0a7cc523e021f76be69d1dcfe17c130078c47e58361a80508
@aetherxeg-source

Copy link
Copy Markdown
Author

Hi Ballista maintainers,

This PR addresses #2767 and is currently at HEAD 62543d90754a0e77e3d8a765ed82d8ccde9306ce.

The targeted regression suite passed 13/13 tests in the linked qualification run, along with cargo check, Clippy, and formatting checks.

I noticed that the upstream PR currently shows the Process check, but not the Rust CI suite. If a maintainer needs to approve CI execution for an external contributor, could you please enable it when convenient?

The signed individual ICLA was submitted to the ASF Secretary on November 4; formal acknowledgement is still pending.

I'm happy to address any technical feedback, particularly around the writer-lifetime and shutdown behavior.

Thank you,
Ahmed Younis
AETHER X GLOBAL

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up! I think the fix for #2767 can be a lot smaller though. The issue is just that the TempDir guard gets dropped right after we read its path, so holding onto it in start_executor_process_inner until the function returns should give us a 0700 directory for the executor's whole lifetime and remove it on exit.

The lease registry adds a process-wide mutex that every shuffle write goes through, and it lets shuffle writer structs hold leases for as long as anything keeps a reference to the plan. If one of those outlives the task, shutdown waits out the graceful timeout and then keeps the directory around, which brings back the leftover-files problem from the issue. I'd rather not take on that global state in ballista-core to cover a writer that races process exit.

Could we shrink this down to keeping the TempDir alive (plus the explicit 0700 set_permissions, which is a nice touch) and one test that checks the directory exists with 0700 after startup? If there's a concrete case where a late writer still matters after that, I'm happy to look at it as a follow-up with a reproducer. Could you also update the description to follow the PR template?

@aetherxeg-source aetherxeg-source changed the title fix(executor): keep private workdirs alive until shuffle writers quiesce fix(executor): retain private default TempDir for executor lifetime Nov 6, 2026
@aetherxeg-source

Copy link
Copy Markdown
Author

@andygrove Thanks for the clear review. I've updated PR #2775 at 09ecf8caa322c29ec57c2e9e1f7bebbffd987e93 to address your feedback:

  • Reduced the final diff against the Apache base to one file (executor_process.rs), removing the WorkDirActivity/WorkDirLease registry and shuffle-writer changes.
  • Retained the default TempDir guard for the executor function's lifetime and explicitly set Unix permissions to 0700, leaving user-specified directories untouched.
  • Added one focused regression test for directory lifetime, 0700 permissions, and cleanup.
  • Updated the PR title and description to match the repository template and the revised scope.

Local WSL2 validation: 101/101 executor library tests passed, Clippy with -D warnings, formatting, and git diff --check passed. An additional local Scheduler + Executor integration check confirmed that the directory still exists with 0700 permissions after registration/readiness and is removed after clean executor exit. That integration check is external to the PR and does not substitute for upstream CI.

Would you mind taking another look when convenient? The upstream Actions runs for this commit still show action_required; if maintainer approval is needed to run CI for the fork, could you please enable it?

Thanks again for the guidance.

@aetherxeg-source

Copy link
Copy Markdown
Author

@andygrove Following up on the narrowed fix for #2767, now at 09ecf8caa322c29ec57c2e9e1f7bebbffd987e93.

The revised PR contains a one-file change: keep the default TempDir guard alive for the executor's lifetime, enforce Unix 0700, and add a focused lifetime/cleanup regression. The earlier process-wide lease registry and shuffle-writer changes are gone, and the description now follows the PR template.

The revised candidate passed 101/101 local executor library tests, Clippy (-D warnings), cargo fmt --check, and a local Scheduler + Executor lifecycle check. Those results do not replace upstream CI. On GitHub, the Process check succeeded, but the Rust/build workflows for this commit remain action_required; the existing changes-requested review predates the revision.

Would you or another Ballista maintainer be able to re-review the reduced patch and approve the pending CI runs, if appropriate? I will address any specific findings on this same PR.

Thank you for the earlier guidance.

AETHER X GLOBAL

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @aetherxeg-source, this is exactly the shape I was hoping for. Keeping the guard alive for the executor's lifetime fixes #2767 without any new global state, and the explicit 0700 is a nice belt-and-braces touch. I've approved the CI runs, so once they're green this is good to go. One tiny optional suggestion inline.

return Err(BallistaError::Configuration(
"Unable to bind work dir".to_string(),
));
let (work_dir, _default_work_dir_owner) = match opt.work_dir.clone() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiny suggestion. This binding is the whole fix, and renaming it to _ would quietly drop the guard again without the test noticing. Could you add a short comment here, something like "keep the guard alive until the executor exits, or the directory is deleted at startup"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Executor's default work dir is deleted at startup and recreated without TempDir's permissions

2 participants