Skip to content

Commit 4c0e834

Browse files
[code:petsc-gmres] Stage stokes2.dat at the per-site CX_input paths
All three systems now default to the shared CX_input local-file directory: /data1/rkp00015/CX_input (RIKYU), /vol0500/share/ra250029/CX_input (Fugaku), /lvs0/rccs-nghpcadu/CX_input (RC_DGXSP -- /lvs0 verified readable from ng-dgx compute nodes, replacing the deleted $HOME copy). Record the stokes2.dat SHA-256 in the README.
1 parent aaadb53 commit 4c0e834

2 files changed

Lines changed: 19 additions & 12 deletions

File tree

‎programs/petsc-gmres/README.md‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -52,16 +52,22 @@ the same format as PETSc's own `MatView`, so no separate conversion step
5252
(e.g. `mtx2petsc`) is needed. The `MatLoad` call in `GMRES-PETSc.cpp`
5353
reads it directly.
5454

55-
Pre-staged locations (same convention as this repo's `ffb` and
55+
Pre-staged at the per-site `CX_input` local-file directory (the shared
56+
staging convention for this deployment: RIKYU `/data1/rkp00015/CX_input`,
57+
R-CCS Cloud `/lvs0/rccs-nghpcadu/CX_input`, Fugaku
58+
`/vol0500/share/ra250029/CX_input`), like this repo's `ffb` and
5659
`LQCD_dw_solver`, which pre-stage their own — much larger — source
57-
archives at a fixed per-system path rather than fetching them at
58-
build/run time):
60+
archives rather than fetching them at build/run time:
5961

6062
| system | path |
6163
|---|---|
62-
| RIKYU | `/data1/rkp00015/benchkit-data/petsc-gmres/stokes2.dat` |
63-
| Fugaku | `/vol0500/data/ra250029/benchkit-data/petsc-gmres/stokes2.dat` — use the `/vol0500` mount alias, not `/vol0005`, in an actual path; `/vol0005` is the volume name but doesn't resolve from a compute-node job. Covered by this app's `FJ` queue.csv `GFSCACHE` declaration (`/vol0002:/vol0003:/vol0004:/vol0005`), so no per-app `-x` handling is needed |
64-
| RC_DGXSP | `/home/users/william.dawson/benchkit-data/petsc-gmres/stokes2.dat` (this system has no separate group-storage tier, so — unlike RIKYU/Fugaku — this lives under `$HOME`) |
64+
| RIKYU | `/data1/rkp00015/CX_input/petsc-gmres/stokes2.dat` |
65+
| Fugaku | `/vol0500/share/ra250029/CX_input/petsc-gmres/stokes2.dat` — use the `/vol0500` mount alias, not `/vol0005`, in an actual path; `/vol0005` is the volume name but doesn't resolve from a compute-node job. Covered by this app's `FJ` queue.csv `GFSCACHE` declaration (`/vol0002:/vol0003:/vol0004:/vol0005`), so no per-app `-x` handling is needed |
66+
| RC_DGXSP | `/lvs0/rccs-nghpcadu/CX_input/petsc-gmres/stokes2.dat` — `/lvs0` is mounted and readable on the ng-dgx compute nodes (verified with a compute-node read test) |
67+
68+
Every copy has SHA-256
69+
`01971e1fb37268a195ff3d96c7800d3f7cf6a37c2ddfe397825c62b3b592bdaf` —
70+
check a fresh copy against it before use.
6571

6672
To re-stage on a system: copy the `stokes2.dat` file (3,092,468,720 bytes)
6773
from an existing location above. To regenerate from scratch, run the

‎programs/petsc-gmres/run.sh‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ touch .run_marker
4747

4848
case "${system}" in
4949
RIKYU)
50-
DATA="${BK_PETSC_GMRES_MATRIX:-/data1/rkp00015/benchkit-data/petsc-gmres/stokes2.dat}"
50+
DATA="${BK_PETSC_GMRES_MATRIX:-/data1/rkp00015/CX_input/petsc-gmres/stokes2.dat}"
5151
module load nvhpc-hpcx/26.3
5252
# One MPI rank per GPU: each rank gets a distinct GPU via
5353
# CUDA_VISIBLE_DEVICES (set inside mpirun so OMPI_COMM_WORLD_LOCAL_RANK
@@ -71,7 +71,7 @@ case "${system}" in
7171
# this was staged) is what actually resolves from a compute-node job;
7272
# found by testing the real run.sh in a real job, not by trusting the
7373
# canonical-looking path a filesystem tool reported.
74-
DATA="${BK_PETSC_GMRES_MATRIX:-/vol0500/data/ra250029/benchkit-data/petsc-gmres/stokes2.dat}"
74+
DATA="${BK_PETSC_GMRES_MATRIX:-/vol0500/share/ra250029/CX_input/petsc-gmres/stokes2.dat}"
7575
module load lang/tcsds-1.2.43
7676
module load LLVM/llvmorg-22.1.0
7777
mpiexec -n "${n_ranks}" \
@@ -90,10 +90,11 @@ case "${system}" in
9090
fi
9191
;;
9292
RC_DGXSP)
93-
# GPU run (1 rank/GPU) -- see build.sh. This system has no separate
94-
# group-storage tier (see README.md), so the data lives under $HOME
95-
# like everything else here.
96-
DATA="${BK_PETSC_GMRES_MATRIX:-/home/users/william.dawson/benchkit-data/petsc-gmres/stokes2.dat}"
93+
# GPU run (1 rank/GPU) -- see build.sh. /lvs0 group storage IS mounted
94+
# and readable on the ng-dgx compute nodes (verified with a compute-node
95+
# read test), so the data lives in the site's CX_input like everywhere
96+
# else.
97+
DATA="${BK_PETSC_GMRES_MATRIX:-/lvs0/rccs-nghpcadu/CX_input/petsc-gmres/stokes2.dat}"
9798
source /etc/profile.d/modules.sh
9899
module load system/ng-dgx nvhpc-hpcx
99100
mpirun -np "${n_ranks}" \

0 commit comments

Comments
 (0)