Skip to content

Point petsc-gmres at the official per-site CX_input directory #224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yoshifuminakamura merged 1 commit into from
Sep 28, 2026
Merged

Point petsc-gmres at the official per-site CX_input directory #224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions programs/petsc-gmres/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,16 +52,22 @@ the same format as PETSc's own `MatView`, so no separate conversion step
(e.g. `mtx2petsc`) is needed. The `MatLoad` call in `GMRES-PETSc.cpp`
reads it directly.

Pre-staged locations (same convention as this repo's `ffb` and
Pre-staged at the per-site `CX_input` local-file directory (the shared
staging convention for this deployment: RIKYU `/data1/rkp00015/CX_input`,
R-CCS Cloud `/lvs0/rccs-nghpcadu/CX_input`, Fugaku
`/vol0500/share/ra250029/CX_input`), like this repo's `ffb` and
`LQCD_dw_solver`, which pre-stage their own — much larger — source
archives at a fixed per-system path rather than fetching them at
build/run time):
archives rather than fetching them at build/run time:

| system | path |
|---|---|
| RIKYU | `/data1/rkp00015/benchkit-data/petsc-gmres/stokes2.dat` |
| Fugaku | `/vol0500/data/ra250029/benchkit-data/petsc-gmres/stokes2.dat` — use the `/vol0500` mount alias, not `/vol0005`, in an actual path; `/vol0005` is the volume name but doesn't resolve from a compute-node job. Covered by this app's `FJ` queue.csv `GFSCACHE` declaration (`/vol0002:/vol0003:/vol0004:/vol0005`), so no per-app `-x` handling is needed |
| RC_DGXSP | `/home/users/william.dawson/benchkit-data/petsc-gmres/stokes2.dat` (this system has no separate group-storage tier, so — unlike RIKYU/Fugaku — this lives under `$HOME`) |
| RIKYU | `/data1/rkp00015/CX_input/petsc-gmres/stokes2.dat` |
| Fugaku | `/vol0500/share/ra250029/CX_input/petsc-gmres/stokes2.dat` — use the `/vol0500` mount alias, not `/vol0005`, in an actual path; `/vol0005` is the volume name but doesn't resolve from a compute-node job. Covered by this app's `FJ` queue.csv `GFSCACHE` declaration (`/vol0002:/vol0003:/vol0004:/vol0005`), so no per-app `-x` handling is needed |
| RC_DGXSP | `/lvs0/rccs-nghpcadu/CX_input/petsc-gmres/stokes2.dat` — `/lvs0` is mounted and readable on the ng-dgx compute nodes (verified with a compute-node read test) |

Every copy has SHA-256
`01971e1fb37268a195ff3d96c7800d3f7cf6a37c2ddfe397825c62b3b592bdaf` —
check a fresh copy against it before use.

To re-stage on a system: copy the `stokes2.dat` file (3,092,468,720 bytes)
from an existing location above. To regenerate from scratch, run the
Expand Down
13 changes: 7 additions & 6 deletions programs/petsc-gmres/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ touch .run_marker

case "${system}" in
RIKYU)
DATA="${BK_PETSC_GMRES_MATRIX:-/data1/rkp00015/benchkit-data/petsc-gmres/stokes2.dat}"
DATA="${BK_PETSC_GMRES_MATRIX:-/data1/rkp00015/CX_input/petsc-gmres/stokes2.dat}"
module load nvhpc-hpcx/26.3
# One MPI rank per GPU: each rank gets a distinct GPU via
# CUDA_VISIBLE_DEVICES (set inside mpirun so OMPI_COMM_WORLD_LOCAL_RANK
Expand All @@ -72,7 +72,7 @@ case "${system}" in
# this was staged) is what actually resolves from a compute-node job;
# found by testing the real run.sh in a real job, not by trusting the
# canonical-looking path a filesystem tool reported.
DATA="${BK_PETSC_GMRES_MATRIX:-/vol0500/data/ra250029/benchkit-data/petsc-gmres/stokes2.dat}"
DATA="${BK_PETSC_GMRES_MATRIX:-/vol0500/share/ra250029/CX_input/petsc-gmres/stokes2.dat}"
module load lang/tcsds-1.2.43
module load LLVM/llvmorg-22.1.0
mpiexec -n "${n_ranks}" \
Expand All @@ -91,10 +91,11 @@ case "${system}" in
fi
;;
RC_DGXSP)
# GPU run (1 rank/GPU) -- see build.sh. This system has no separate
# group-storage tier (see README.md), so the data lives under $HOME
# like everything else here.
DATA="${BK_PETSC_GMRES_MATRIX:-/home/users/william.dawson/benchkit-data/petsc-gmres/stokes2.dat}"
# GPU run (1 rank/GPU) -- see build.sh. /lvs0 group storage IS mounted
# and readable on the ng-dgx compute nodes (verified with a compute-node
# read test), so the data lives in the site's CX_input like everywhere
# else.
DATA="${BK_PETSC_GMRES_MATRIX:-/lvs0/rccs-nghpcadu/CX_input/petsc-gmres/stokes2.dat}"
source /etc/profile.d/modules.sh
module load system/ng-dgx nvhpc-hpcx
mpirun -np "${n_ranks}" \
Expand Down
Loading