refactor(mpi): channel persistent requests - #268
Open
dssgabriel wants to merge 5 commits into
Open
Conversation
- Forward-declare Channel as a communication-space template defaulting to MpiSpace. - Define the existing implementation as the Channel<MpiSpace> specialization, keeping Channel<> MPI-specific when NCCL is enabled. - Document that MPI is the only currently provided Channel specialization. Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
- Replace the distinct send and receive Request vectors with one contiguous vector of raw persistent MPI requests and corresponding status storage. - Initialize, start, and wait on persistent request storage directly, avoiding merged temporary arrays on every communication cycle. - Keep completed persistent requests allocated for reuse and release them exactly once when their owning Channel is destroyed. - Transfer request ownership during Channel moves and report MPI operation failures through the existing error-handling path. Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
- Register channel operations once and verify cycle-specific data across repeated start/wait cycles. - Cover move construction and move assignment across all supported scalar types. - Ensure registered Views outlive their Channel and verify the move-only type contract. Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
- Document persistent request reuse and the buffer-access rules between start/wait cycles. - Require registered buffer allocations and the borrowed communicator to remain valid until Channel destruction. - Describe persistent request release, move ownership, and inactive destruction and assignment preconditions. Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
Replace the fixed one-million-double case with a Cartesian range covering payloads from 1 B to 8 MiB and one to sixteen send/receive operation pairs. This exposes both latency-sensitive request-management overhead and bandwidth-dominated behavior while showing how performance scales with the number of registered operations. Register operations once before timing, retain distinct send and receive buffers for their full lifetime, and report both processed bytes and registered operation counts. Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
dssgabriel
force-pushed
the
refactor/mpi-channel-persistent-requests
branch
from
July 30, 2026 17:26
fdb767b to
ddbf5f4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Specialize the MPI
Channelimplementation to manage persistent MPI requests directly as rawMPI_Requesthandles.This removes temporary request containers and handle copies from
start()andwait(), ensures persistent requests are released correctly, and preserves registered operations across Channel reuse and moves.Channelto use raw persistent MPI requests #265Additional context
The previous implementation stored persistent requests in separate send and receive vectors using the non-persistent
Request<MpiSpace>abstraction.Every communication cycle then created temporary containers and copied or moved request handles before calling the MPI collective request APIs.
It also did not release the persistent requests with
MPI_Request_free.The specialized implementation stores send and receive requests in a single registration-ordered vector of raw persistent handles.
It calls
MPI_StartallandMPI_Waitalldirectly on that vector, reuses a preallocated status vector, and releases the requests when the Channel is destroyed or replaced through move assignment.Changes
List:
Channel<MpiSpace>for the MPI backend.MPI_Requesthandles.MPI_Request_free.Performance impact
The benchmark registers all persistent send and receive operations once, outside the timed loop, and measures repeated
Channel::start()/Channel::wait()cycles.It covers:
For a fair comparison, the new benchmark source was applied unchanged to clean snapshots of the
developbranch (which only contained the previous fixed-size Channel benchmark).Both snapshots were built in Release mode using the same Spack environment:
Representative mean results (on my laptop):
developThis branch is faster in all configurations.
Configurations up to 32 KiB averaged a 40.7% reduction in median time.
As expected, the relative improvement decreases for larger payloads, as communication time comes to dominate the request-management overhead.
The
developrun also reported unfreed inactive persistent requests during MPI finalization.Those diagnostics occurred outside the timed region and confirm the persistent-request lifetime issue that this refactor addresses.
Checklist