Skip to content

refactor(mpi): channel persistent requests - #268

Open
dssgabriel wants to merge 5 commits into
kokkos:developfrom
dssgabriel:refactor/mpi-channel-persistent-requests
Open

refactor(mpi): channel persistent requests#268
dssgabriel wants to merge 5 commits into
kokkos:developfrom
dssgabriel:refactor/mpi-channel-persistent-requests

Conversation

@dssgabriel

@dssgabriel dssgabriel commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Description

Specialize the MPI Channel implementation to manage persistent MPI requests directly as raw MPI_Request handles.

This removes temporary request containers and handle copies from start() and wait(), ensures persistent requests are released correctly, and preserves registered operations across Channel reuse and moves.

Additional context

The previous implementation stored persistent requests in separate send and receive vectors using the non-persistent Request<MpiSpace> abstraction.

Every communication cycle then created temporary containers and copied or moved request handles before calling the MPI collective request APIs.
It also did not release the persistent requests with MPI_Request_free.

The specialized implementation stores send and receive requests in a single registration-ordered vector of raw persistent handles.
It calls MPI_Startall and MPI_Waitall directly on that vector, reuses a preallocated status vector, and releases the requests when the Channel is destroyed or replaced through move assignment.

Changes

  • Affected areas: mpi, unit tests, perf tests
  • Breaking change(s)? no

List:

  • Specialize Channel<MpiSpace> for the MPI backend.
  • Store persistent send and receive operations together as raw MPI_Request handles.
  • Start and wait on the registered requests without allocating temporary work vectors.
  • Retain completion statuses in a reusable vector.
  • Release persistent requests with MPI_Request_free.
  • Transfer registered requests safely during Channel move construction and move assignment.
  • Document Channel ownership, lifetime, registration-order, and reuse semantics.
  • Cover host and default execution spaces, repeated communication cycles, and Channel moves.
  • Expand the Channel point-to-point benchmark across payload sizes and registered operation counts.

Performance impact

The benchmark registers all persistent send and receive operations once, outside the timed loop, and measures repeated Channel::start()/Channel::wait() cycles.

It covers:

  • Payload sizes from 1 B to 8 MiB.
  • 1, 2, 4, 8, and 16 send/receive operation pairs.
  • 45 configurations in total.

For a fair comparison, the new benchmark source was applied unchanged to clean snapshots of the develop branch (which only contained the previous fixed-size Channel benchmark).

Both snapshots were built in Release mode using the same Spack environment:

  • GCC 15.3
  • MPICH 5.0 (2 ranks pinned to separate CPU cores)
  • Kokkos 5.0 (OpenMP backend, 1 thread/rank)

Representative mean results (on my laptop):

Configuration This PR develop Time reduction
1 B, 1 pair 0.836 µs 1.441 µs 42.0%
1 B, 8 pairs 3.406 µs 6.167 µs 44.8%
4 KiB, 16 pairs 11.609 µs 20.459 µs 43.3%
8 MiB, 16 pairs 20.636 ms 21.289 ms 3.1%

This branch is faster in all configurations.
Configurations up to 32 KiB averaged a 40.7% reduction in median time.
As expected, the relative improvement decreases for larger payloads, as communication time comes to dominate the request-management overhead.

The develop run also reported unfreed inactive persistent requests during MPI finalization.
Those diagnostics occurred outside the timed region and confirm the persistent-request lifetime issue that this refactor addresses.

Checklist

  • Tests are up-to-date
  • Documentation is up-to-date

- Forward-declare Channel as a communication-space template defaulting
to MpiSpace.
- Define the existing implementation as the Channel<MpiSpace>
specialization, keeping Channel<> MPI-specific when NCCL is enabled.
- Document that MPI is the only currently provided Channel
specialization.

Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
- Replace the distinct send and receive Request vectors with one
contiguous vector of raw persistent MPI requests and corresponding
status storage.
- Initialize, start, and wait on persistent request storage directly,
avoiding merged temporary arrays on every communication cycle.
- Keep completed persistent requests allocated for reuse and release
them exactly once when their owning Channel is destroyed.
- Transfer request ownership during Channel moves and report MPI
operation failures through the existing error-handling path.

Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
@dssgabriel
dssgabriel requested a review from nicoleavans July 30, 2026 17:13
@dssgabriel dssgabriel self-assigned this Jul 30, 2026
@dssgabriel dssgabriel added C-enhancement Category: an enhancement T-performance Topic: something related to performance A-unit-tests Area: KokkosComm unit tests A-perf-tests Area: KokkosComm performance tests A-mpi Area: KokkosComm MPI backend implementation labels Jul 30, 2026
- Register channel operations once and verify cycle-specific data across
repeated start/wait cycles.
- Cover move construction and move assignment across all supported
scalar types.
- Ensure registered Views outlive their Channel and verify the move-only
type contract.

Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
- Document persistent request reuse and the buffer-access rules between
start/wait cycles.
- Require registered buffer allocations and the borrowed communicator to
remain valid until Channel destruction.
- Describe persistent request release, move ownership, and inactive
destruction and assignment preconditions.

Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
Replace the fixed one-million-double case with a Cartesian range
covering payloads from 1 B to 8 MiB and one to sixteen send/receive
operation pairs. This exposes both latency-sensitive request-management
overhead and bandwidth-dominated behavior while showing how performance
scales with the number of registered operations. Register operations
once before timing, retain distinct send and receive buffers for their
full lifetime, and report both processed bytes and registered operation
counts.

Signed-off-by: Gabriel Dos Santos <gabriel.dossantos@cea.fr>
@dssgabriel
dssgabriel force-pushed the refactor/mpi-channel-persistent-requests branch from fdb767b to ddbf5f4 Compare July 30, 2026 17:26
@dssgabriel
dssgabriel requested a review from cwpearson July 31, 2026 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-mpi Area: KokkosComm MPI backend implementation A-perf-tests Area: KokkosComm performance tests A-unit-tests Area: KokkosComm unit tests C-enhancement Category: an enhancement T-performance Topic: something related to performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rework Channel to use raw persistent MPI requests

1 participant