Skip to content

Create temp_dir if missing and fail all ranks together instead of deadlocking - #10

Merged
Sophia Wen (hfwen0502) merged 1 commit into
mainfrom
fix-temp-dir-hang
Aug 27, 2026
Merged

Sophia Wen (hfwen0502) merged 1 commit into
mainfrom
fix-temp-dir-hang

Conversation

@hfwen0502

Copy link
Copy Markdown
Member

Summary

_make_sbd_dir passed temp_dir straight to tempfile.mkdtemp, which does not create parent directories. Passing a temp_dir (the --temp_dir option) that points at a path which does not exist yet therefore raised FileNotFoundError on rank 0.

Because that failure happens before the mpi_comm.bcast that hands the created directory to the other ranks, ranks 1..N block in the broadcast forever. Under MPI the whole job hangs with no output instead of erroring; a single-rank run just surfaces the traceback.

Fix

  • Create temp_dir and any missing parents on rank 0 (Path(temp_dir).mkdir(parents=True, exist_ok=True)) before mkdtemp, so a caller may point --temp_dir at a fresh scratch location.
  • Wrap rank 0's setup and broadcast its outcome as (ok, payload). On failure every rank raises RuntimeError together rather than deadlocking in the broadcast.

No new imports (Path is already imported); no behavior change when temp_dir already exists or is omitted (the omitted case still routes through tempfile.gettempdir()).

Verification

Reproduced and verified with the real function under mpirun -np 2:

  • Before: non-existent --temp_dir → rank 0 raises, rank 1 blocks in bcast; job killed by mpirun --timeout at 8s.
  • After, missing dir: both ranks return the same, existing directory; exit 0 in ~1.3s.
  • After, uncreatable dir (parent is a regular file): both ranks raise RuntimeError together with a clear message; exit fast, no deadlock.

…dlocking

_make_sbd_dir handed temp_dir straight to tempfile.mkdtemp, which does not
create parent directories. A temp_dir (the --temp_dir option) pointing at a
path that did not exist yet therefore raised FileNotFoundError on rank 0. That
happened before the broadcast that hands the created directory to the other
ranks, so the others blocked in the broadcast forever and the whole job hung
with no output instead of erroring.

Create temp_dir and any missing parents on rank 0 before mkdtemp, and wrap the
setup so its outcome is broadcast: on failure every rank raises RuntimeError
together rather than deadlocking.
@hfwen0502
Sophia Wen (hfwen0502) merged commit 7c7cc09 into main Aug 27, 2026
6 checks passed
@hfwen0502
Sophia Wen (hfwen0502) deleted the fix-temp-dir-hang branch August 27, 2026 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant