Conversation
A test file that dies from an uncaught exception (a failing `@test`, an `MPIError`, ...) exits nonzero, and `MPI.Init`'s atexit hook then skips `MPI.Finalize()`, correctly, since it is collective. But the failing rank still runs its object finalizers afterwards, and `free(::Win)`, `free(::Comm)` and `close(::FileHandle)` all make collective MPI calls, gated only on `!Finalized()` -- which is false on this path. So the failing rank *enters* a collective while its peers are blocked somewhere else entirely. Every process is still alive, so there is no dead process for the launcher to notice, and the job hangs until the CI job-level `timeout-minutes: 20` expires. The run is then reported as a job timeout rather than as a named failing test, with no information about the other test files. Julia runs atexit hooks before object finalizers, so an atexit hook is the right place to call `MPI_Abort`: it preempts both the blocked peers and the finalizer deadlock. Load it into every test process with `julia -L` rather than editing each test file, since only 26 of the 50 test files include `common.jl` and a new test file would otherwise have to remember to opt in. Note that launchers truncate the abort code to 8 bits when reporting it as the job's exit status, so the errcode has to be sanitized: passing a raw exit code of 256 makes mpiexec exit 0, turning a hard failure into a green run. Verified on both MPICH and Open MPI. Also correct usage.md and external.md, which both described the atexit hook as calling `MPI.Finalize` when Julia exits, omitting the exit-code condition that causes all of the above.
Contributor
Author
|
The failing test is probably cured by #982. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Failing self-tests exit via an exception. Depending on the MPI implementation, this may lead to a hang, which leads to a time-out in Github CI, which leads to a
cancelledclassification instead of an error.This PR calls
MPI_Abortwhen a test fails by adding a respectiveatexithook.This is done only for the self-tests. It might be useful to do the same for all MPI applications – it doesn't really make sense to just exit the Unix process, we should either call
MPI_FinalizeorMPI_Abortin either case, and if we exit without the app callingMPI_Finalizethe we should probably abort. But that's a different issue.