Skip to content

Add RIKYU support for MHDTurbulence - #144

Draft
william-dawson wants to merge 1 commit into
RIKEN-RCCS:developfrom
william-dawson:add-mhdturbulence-rikyu
Draft

Add RIKYU support for MHDTurbulence#144
william-dawson wants to merge 1 commit into
RIKEN-RCCS:developfrom
william-dawson:add-mhdturbulence-rikyu

Conversation

@william-dawson

Copy link
Copy Markdown
Contributor

What

  • RIKYU rows in list.csv: 1, 4, and 8 GPUs (1×1, 1×4, 2×4), 10-min budget, as-shipped config.
  • RIKYU build/run cases: nvhpc-hpcx-cuda13/26.5; build.sh produces one binary per rank count (Simulation.x.t{1,4,8}) because ntiles is a compile-time parameter; run.sh selects by nodes × numproc_node and launches through a per-rank GPU wrapper.
  • Two portability patches (grep-guarded in build.sh, no-ops once upstream):
    • random_seed_put_size.patchrandom_seed(PUT=) needs the compiler's seed size (8 for gfortran); physically inert (rrv=0).
    • acc_init_before_mpi.patch — initialize the CUDA context before MPI_Init.
  • Launch wrapper (RIKYU run.sh): CUDA_VISIBLE_DEVICES=$OMPI_COMM_WORLD_LOCAL_RANK per rank.

Why the patch + wrapper are required together

Without them, multi-node GPU runs hang silently in the first BoundaryCondition: UCX fails the device-buffer MPI_ISEND with cannot find remote protocol for: inter-node tag_send(multi) from cuda/GPU, and the code waits forever in MPI_WAITALL (the app sets MPI_ERRORS_RETURN and never checks the ISEND return). The context must exist before MPI_Init and each rank must pin exactly one GPU at launch; either part alone still fails. Excluding CUDA transports is not an alternative — hpcx-2.50 has no host-staging path for device buffers (MPI_INIT aborts, "no copy across memory types transport").

Validation (2026-09-03, Rikyu)

  • 8 GPUs / 2 nodes: completed clean, 13,764 steps, dt bit-identical (14 digits) to the 1-GPU run; wall 57.1 s vs 115.6 s at 1 GPU.
  • Patches originate from william-dawson/MHDTurbulence branch gfortran-port (commits d9a19d9, 4db3adb); candidate upstream PRs to cfcanaoj/MHDTurbulence.

Draft for review of the build/run structure and the patch-application pattern.

- Add RIKYU rows, build, and run support (nvhpc-hpcx-cuda13/26.5)
- Add patches fixing gfortran random_seed PUT size and CUDA-context-before-MPI
- Add the per-rank CUDA_VISIBLE_DEVICES wrapper needed for multi-GPU launch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant