Repository navigation
Conversation
|
test:mpich/ch4/ofi |
sonjahapp
left a comment
sonjahapp
left a comment
There was a problem hiding this comment.
Thanks for this PR!
Just found a few minor typo fixes for the testlist. 😉
Thanks for the catch. I will fix them in the next push. By the way, do you think we can remove the PMI atexit hook completely now we have the CVAR option? |
Hmm... I would keep it. It leaves freedom to the device to enable more "fine grained" solutions for re-init. The new CVAR is quite the "hammer" since it postpones almost the entire MPI cleanup to an atexit handler. For example, in the psp device we offer the ability to finalize most parts in |
04a2a79 to
641af69
Compare
|
test:mpich/ch4/ofi |
|
I think now with the new CVAR set in the tests, on re-init the Perhaps the comm bootstrap hook needs to check if the shmem segment is already there and then reuse it in case of re-init instead of attempting to init it again? |
There is no shmem reinitialization if the finalize is delayed. I am not sure what is causing the pmix test failures yet. |
|
test:mpich/ch4/ofi |
|
test:mpich/ch4/ofi |
This reverts commit cc91750.
This reverts commit 338496d.
Add this cvar as a work-around for sessions re-init. There is no good solution to synchronize shared memory over re-init now that we switched to total asynchronous initialization. Consider the following case: * Process A and B both initialized shm and both re-initialize. * B re-init shm while A delays. * B send a message to A via shm. * A (still in the previous epoch) either discards the message or error. * A re-init. * A never receives the message B send. Delay the final finalize via atexit avoids the re-init altogether.
In case device runtime also register atexit hooks during MPID_Init, register finalize_atexit at the end of Init so finalize_atexit runs first, before potential device atexit hooks.
Putting MPII_Finalize in atexit obviates the need for pmi atexit hook.
Since **atexit is also used in mpir_init.c, move it to src/mpi/errhan/errnames.txt instead of src/pmi/errnames.txt.
|
test:mpich/ch4/ofi |
Pull Request Description
As cc91750#r3758517489 show, we can't rely on local init_counter to collectively resolve conflicts between session re-init epochs. In fact, after a full-day's effort, I concluded there is no way to create a solution for
We just can't have a collective synchronization when we can't establish collective assumptions.
This PR reverts the previous attempts and add
MPIR_CVAR_FINALIZE_ATEXITto workaround. When the cvar is set, we delay the finalMPII_Finalizetoatexit, avoiding the necessity of re-initialize both shm and netmod.Author Checklist
Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
Commits are self-contained and do not do two things at once.
Commit message is of the form:
module: short descriptionCommit message explains what's in the commit.
Whitespace checker. Warnings test. Additional tests via comments.
For non-Argonne authors, check contribution agreement.
If necessary, request an explicit comment from your companies PR approval manager.