Create temp_dir if missing and fail all ranks together instead of deadlocking - #10
Merged
Merged
Conversation
…dlocking _make_sbd_dir handed temp_dir straight to tempfile.mkdtemp, which does not create parent directories. A temp_dir (the --temp_dir option) pointing at a path that did not exist yet therefore raised FileNotFoundError on rank 0. That happened before the broadcast that hands the created directory to the other ranks, so the others blocked in the broadcast forever and the whole job hung with no output instead of erroring. Create temp_dir and any missing parents on rank 0 before mkdtemp, and wrap the setup so its outcome is broadcast: on failure every rank raises RuntimeError together rather than deadlocking.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_make_sbd_dirpassedtemp_dirstraight totempfile.mkdtemp, which does not create parent directories. Passing atemp_dir(the--temp_diroption) that points at a path which does not exist yet therefore raisedFileNotFoundErroron rank 0.Because that failure happens before the
mpi_comm.bcastthat hands the created directory to the other ranks, ranks 1..N block in the broadcast forever. Under MPI the whole job hangs with no output instead of erroring; a single-rank run just surfaces the traceback.Fix
temp_dirand any missing parents on rank 0 (Path(temp_dir).mkdir(parents=True, exist_ok=True)) beforemkdtemp, so a caller may point--temp_dirat a fresh scratch location.(ok, payload). On failure every rank raisesRuntimeErrortogether rather than deadlocking in the broadcast.No new imports (
Pathis already imported); no behavior change whentemp_diralready exists or is omitted (the omitted case still routes throughtempfile.gettempdir()).Verification
Reproduced and verified with the real function under
mpirun -np 2:--temp_dir→ rank 0 raises, rank 1 blocks inbcast; job killed bympirun --timeoutat 8s.RuntimeErrortogether with a clear message; exit fast, no deadlock.