Description
When worker_main fails while constructing GenerationExecutorWorker on a non-leader rank, that rank returns while rank 0 can remain blocked in an MPI collective that the failed rank can no longer enter. The executor process then remains wedged instead of failing promptly, so the serving worker may require manual termination.
Proposed fix
Abort the MPI communicator from a non-leader rank when worker initialization raises. Rank 0 retains the existing behavior of reporting the initialization exception through worker_init_status_queue.
Validation
- Focused unit regression covering the non-leader failure path.
- Real 4-rank MPI integration test with rank 3 constructor failure: rank 3 invoked
MPI_Abort, and the MPI job exited with code 1 before a 20-second timeout.
This was originally observed with disaggregated TensorRT-LLM serving, where a rank initialization failure otherwise left the node-level worker wedged.
Description
When
worker_mainfails while constructingGenerationExecutorWorkeron a non-leader rank, that rank returns while rank 0 can remain blocked in an MPI collective that the failed rank can no longer enter. The executor process then remains wedged instead of failing promptly, so the serving worker may require manual termination.Proposed fix
Abort the MPI communicator from a non-leader rank when worker initialization raises. Rank 0 retains the existing behavior of reporting the initialization exception through
worker_init_status_queue.Validation
MPI_Abort, and the MPI job exited with code 1 before a 20-second timeout.This was originally observed with disaggregated TensorRT-LLM serving, where a rank initialization failure otherwise left the node-level worker wedged.