Initial software set

[heatherkellyucl]

Large usage MD codes

• LAMMPS + variants: Install Request: LAMMPS with Python in version of 7 Feb 2024 or later  rcps-buildscripts#575
• CP2K: Install Request: CP2K v2023.1 (latest version) [IN05824831] rcps-buildscripts#511 (CPU done, GPU in progress)
• GROMACS + variants: Install Request: GROMACS 2023 (latest update) including CPU, GPU and + Plumed versions [IN05756873] [IN05807314][IN05844418] rcps-buildscripts#508 (plumed only available as variant for 2023.x, plumed for 2024 is not ready yet)
• CASTEP: Install Request: CASTEP 2x [IN05839181] rcps-buildscripts#514 Pull request for initial spec at Create Castep package spack/spack#41230 - is now merged.
• ORCA (binary, but needs a working OpenMPI 4.1.x): Install Request: ORCA 5 rcps-buildscripts#474 (Done a 5.x and 6.x)
• NAMD 2.14 CPU done, 3.0.0 issues with charmpp for CPU and GPU, 2.14 GPU requires CUDA 11.
• Check whether we want to put VASP in atm or leave it out (lots of use, but we have a recent install).

Myriad-flavoured packages

[Initial stack done spack/spack#49]
Samtools, BCFTools, HTSlib: UCL-ARC/rcps-buildscripts#532

Samtools, Bwa, Bedtools, Vcftools, picard, GATK (requested by Biosciences) https://ucldata.atlassian.net/browse/AHS-139

Alphafold (would be nice to not have to tell people to go and track down the working containers): UCL-ARC/rcps-buildscripts#529 UCL-ARC/rcps-buildscripts#463 [Not doing until Spack 0.22]

HDF5
Netcdf
BEAST 2.7: UCL-ARC/rcps-buildscripts#498

The list of user-requested software builds is at https://github.com/UCL-RITS/rcps-buildscripts/issues

Second set

Amber
Cellranger: UCL-ARC/rcps-buildscripts#499
Have a look at dependencies of SRAtoolkit: UCL-ARC/rcps-buildscripts#543
Hammock
AFNI
Quantum Espresso
NWChem

Out of scope for initial set

Python
R

GPU builds

Only Myriad and Young have GPUs. Young's GPUs are AMD-cpu nodes while Myriad's are Intel. (GPUs themselves are Nvidia). Young only has A100s, Myriad has A100s and some older GPUs, of which the oldest will be retired soonish.

GPU builds are dependent on the version of the CUDA drivers being updated on the GPU nodes - not worth putting any GPU builds in the buildcache until after that has happened. UCL-ARC/rcps-buildscripts#528 and UCL-ARC/rcps-buildscripts#517 and https://ucldata.atlassian.net/browse/ARI-254. Done.

Work cannot begin on Kathleen and Michael until https://ucldata.atlassian.net/browse/ARI-37 is complete (including devtoolset-11 and rh-python38 in the image). Done.

[heatherkellyucl]

FSL in Spack patches out the job submission capability.
Freesurfer doesn't depend on Matlab but uses it.

[heatherkellyucl]

We were considering compilers but Spack is about to change how it deals with them so probably worth waiting.

[heatherkellyucl]

We want to test that the OpenMPI we get from Spack is reasonable performance-wise on the OmniPath clusters.

So far we have not specified any fabrics and Spack lets OpenMPI set options as auto in that case.

We should check performance of that version against one specifying some likely options (and can test against the vader-using mpi/openmpi/4.1.1/gnu-4.9.2 too - if it performs worse than that, we have a problem!)

Benchmarks: https://mvapich.cse.ohio-state.edu/benchmarks/

Can see available variants with spack info --all openmpi

[krishnakumarg1984]

Everything is working finally, and have been going fine for the last 2 days.

Currently, building gromacs@2023.1%gcc@12.2.0 ^openmpi@4.1.5 schedulers=sge %gcc@12.2.0.

The complete provenance is:

gromacs@2023.1%gcc@12.2.0~blas~cp2k~cuda~cycle_subcounters~double+hwloc~ipo~lapack~mdrun_only+mpi~nosuffix~opencl+openmp~relaxed_double_precision+shared~sycl build_system=cmake build_type=Release generator=make openmp_max_threads=none arch=linux-rhel7-broadwell
    ^cmake@3.26.3%gcc@12.2.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-rhel7-broadwell
        ^ncurses@6.4%gcc@12.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel7-broadwell
        ^openssl@1.1.1t%gcc@12.2.0~docs~shared build_system=generic certs=mozilla arch=linux-rhel7-broadwell
            ^ca-certificates-mozilla@2023-01-10%gcc@12.2.0 build_system=generic arch=linux-rhel7-broadwell
    ^fftw@3.3.10%gcc@12.2.0+mpi~openmp~pfft_patches build_system=autotools precision=double,float arch=linux-rhel7-broadwell
    ^gmake@4.4.1%gcc@12.2.0~guile build_system=autotools arch=linux-rhel7-broadwell
    ^hwloc@2.9.1%gcc@12.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rhel7-broadwell
        ^libpciaccess@0.17%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
            ^libtool@2.4.7%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
            ^util-macros@1.19.3%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
        ^libxml2@2.10.3%gcc@12.2.0~python build_system=autotools arch=linux-rhel7-broadwell
            ^libiconv@1.17%gcc@12.2.0 build_system=autotools libs=shared,static arch=linux-rhel7-broadwell
            ^xz@5.4.1%gcc@12.2.0~pic build_system=autotools libs=shared,static arch=linux-rhel7-broadwell
        ^pkgconf@1.9.5%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
    ^openmpi@4.1.5%gcc@12.2.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~orterunprefix+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=none schedulers=sge arch=linux-rhel7-broadwell
        ^numactl@2.0.14%gcc@12.2.0 build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-rhel7-broadwell
            ^autoconf@2.69%gcc@12.2.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-rhel7-broadwell
            ^automake@1.16.5%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
            ^m4@1.4.19%gcc@12.2.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rhel7-broadwell
                ^diffutils@3.9%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
                ^libsigsegv@2.14%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
        ^openssh@9.3p1%gcc@12.2.0+gssapi build_system=autotools arch=linux-rhel7-broadwell
            ^krb5@1.20.1%gcc@12.2.0+shared build_system=autotools arch=linux-rhel7-broadwell
                ^bison@3.8.2%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
                ^gettext@0.21.1%gcc@12.2.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-rhel7-broadwell
                    ^tar@1.34%gcc@12.2.0 build_system=autotools zip=pigz arch=linux-rhel7-broadwell
                        ^pigz@2.7%gcc@12.2.0 build_system=makefile arch=linux-rhel7-broadwell
                        ^zstd@1.5.5%gcc@12.2.0+programs build_system=makefile compression=none libs=shared,static arch=linux-rhel7-broadwell
            ^libedit@3.1-20210216%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
            ^libxcrypt@4.4.33%gcc@12.2.0~obsolete_api build_system=autotools arch=linux-rhel7-broadwell
        ^perl@5.36.0%gcc@12.2.0+cpanm+open+shared+threads build_system=generic arch=linux-rhel7-broadwell
            ^berkeley-db@18.1.40%gcc@12.2.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rhel7-broadwell
            ^bzip2@1.0.8%gcc@12.2.0~debug~pic+shared build_system=generic arch=linux-rhel7-broadwell
            ^gdbm@1.23%gcc@12.2.0 build_system=autotools arch=linux-rhel7-broadwell
                ^readline@8.2%gcc@12.2.0 build_system=autotools patches=bbf97f1 arch=linux-rhel7-broadwell
        ^pmix@4.2.3%gcc@12.2.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-rhel7-broadwell
            ^libevent@2.1.12%gcc@12.2.0+openssl build_system=autotools arch=linux-rhel7-broadwell
        ^zlib@1.2.13%gcc@12.2.0+optimize+pic+shared build_system=makefile arch=linux-rhel7-broadwell

[heatherkellyucl]

Looks like the gromacs variants we would have, based on our current ones, are these. Or for 2022.5 we could just do the bolded plumed variants as we can't build those yet for 2023.

We normally include the double-precision versions with the non-cuda builds.

cuda_arch=80 is the A100s. Ignore the cuda variants for the first go-round. (Would only be added on Myriad and Young, and on Young it'd need to be built on a GPU node as those are AMD).

gromacs@2022.5 %gcc@12.2.0 +double
gromacs@2022.5 %gcc@12.2.0 +double +plumed
gromacs@2022.5 %gcc@12.2.0 +cuda cuda_arch=80
gromacs@2022.5 %gcc@12.2.0 +cuda cuda_arch=80 +plumed

gromacs@2023.1 %gcc@12.2.0 +double
gromacs@2023.1 %gcc@12.2.0 +cuda cuda_arch=80

[krishnakumarg1984]

Ok. Happy to do just the bolded ones for now.

[heatherkellyucl]

Not just the bolded ones, gromacs@2023.1 %gcc@12.2.0 +double is fine, and should be related to what you were already doing, just with +double.

[heatherkellyucl]

@krishnakumarg1984 When you have installed some packages, spack find -ldf will show you a tree.

kut6cpb gcc@12.2.0%gcc
y5z2x63     binutils@2.40%gcc
rrab7ki         pkgconf@1.8.0%gcc
ehpjj76     diffutils@3.8%gcc
uaerdna         libiconv@1.16%gcc
pokguyu     gawk@5.1.1%gcc
qw2elyk         libsigsegv@2.13%gcc
jtrpaef         readline@8.1%gcc
3yevelt     gmake@4.4.1%gcc
scvmoox     gmp@6.2.1%gcc
gphegdo         autoconf@2.69%gcc
3fsyut7         automake@1.16.5%gcc
wiuk7k4         m4@1.4.19%gcc
5a6s7ss     libtool@2.4.7%gcc
t6o5yxx     mpc@1.2.1%gcc
43lsbjo     mpfr@4.1.0%gcc
rmhkp3h         autoconf-archive@2022.02.11%gcc
che7ezb     perl@5.36.0%gcc
6pqhjod         berkeley-db@18.1.40%gcc
mq5csuf         bzip2@1.0.8%gcc
afgghmq         gdbm@1.23%gcc
5xsqgvu     texinfo@6.5%gcc
rp5rvar         gettext@0.21.1%gcc
lfr5oag             libxml2@2.10.3%gcc
sqh3ckt             tar@1.30%gcc
omwksxg                 pigz@2.7%gcc
7ibfohu             xz@5.4.1%gcc
6nrbosy         ncurses@6.3%gcc
gsyujwx     zlib@1.2.13%gcc
myexac2     zstd@1.5.2%gcc

afgghmq gdbm@1.23%gcc
jtrpaef     readline@8.1%gcc
6nrbosy         ncurses@6.3%gcc
rrab7ki             pkgconf@1.8.0%gcc

etc since you'll get one for every individual package.

[heatherkellyucl]

I'm pretty sure it says this somewhere else in here, but for a site called envtest and an environment called compilers, then these are created during concretisation and list the specs and hashes for that environment. (And the concretization is shown on the command line as it does it).

$sites_root/envtest/spack/var/spack/environments/compilers/spack.yaml
$sites_root/envtest/spack/var/spack/environments/compilers/spack.lock

There's a .spack-env in there too. We're creating an environment in our site: https://spack.readthedocs.io/en/latest/environments.html

[krishnakumarg1984]

@heatherkellyucl Done. Finally got it working.

[+] /scratch/home/uccagop/apps/spack/0.20/krishna_spacksite_michael_ucl/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-12.2.0/fftw-3.3.10-3y2q7r6hhiqhzofiawmfxdmem72y2ikp
==> Installing gromacs-2023.1-iqsepjm7yg5q2dpbfbr622xvwph4rgxu
==> No binary for gromacs-2023.1-iqsepjm7yg5q2dpbfbr622xvwph4rgxu found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ee/eef2bb4a6cb6314cf9da47f26df2a0d27af4bf7b3099723d43601073ab0a42f4.tar.gz
==> Ran patch() for gromacs
==> gromacs: Executing phase: 'cmake'
==> gromacs: Executing phase: 'build'
==> gromacs: Executing phase: 'install'
==> gromacs: Successfully installed gromacs-2023.1-iqsepjm7yg5q2dpbfbr622xvwph4rgxu
  Stage: 24.40s.  Cmake: 1m 9.11s.  Build: 8m 5.01s.  Install: 10.59s.  Post-install: 3.34s.  Total: 9m 53.94s
[+] /scratch/home/uccagop/apps/spack/0.20/krishna_spacksite_michael_ucl/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-12.2.0/gromacs-2023.1-iqsepjm7yg5q2dpbfbr622xvwph4rgxu
==> Updating view at /scratch/home/uccagop/apps/spack/0.20/krishna_spacksite_michael_ucl/spack/var/spack/environments/krishna_spackenv_michael_ucl/.spack-env/view
[krishna_spackenv_michael_ucl] [uccagop@login11 hpc-spack]$

==> Installed packages
-- linux-rhel7-broadwell / gcc@12.2.0 ---------------------------
iqsepjm gromacs@2023.1
incrgec     cmake@3.26.3
6nrbosy         ncurses@6.3
x5spvhy         openssl@1.1.1t
76n2l7a             ca-certificates-mozilla@2023-01-10
3y2q7r6     fftw@3.3.10
dvrmwg4     gmake@4.4.1
hmy42iw     hwloc@2.9.1
ionflx6         libpciaccess@0.17
5a6s7ss             libtool@2.4.7
v26d2xt             util-macros@1.19.3
lfr5oag         libxml2@2.10.3
uaerdna             libiconv@1.16
7ibfohu             xz@5.4.1
rrab7ki         pkgconf@1.8.0
mu7mzua     openmpi@4.1.5
l6vxr43         numactl@2.0.14
gphegdo             autoconf@2.69
3fsyut7             automake@1.16.5
wiuk7k4             m4@1.4.19
ehpjj76                 diffutils@3.8
qw2elyk                 libsigsegv@2.13
n35h3ms         openssh@9.3p1
p3ujrfd             krb5@1.20.1
yoyydto                 bison@3.8.2
rp5rvar                 gettext@0.21.1
sqh3ckt                     tar@1.30
omwksxg                         pigz@2.7
upkqxh2             libedit@3.1-20210216
tnk52b6             libxcrypt@4.4.33
che7ezb         perl@5.36.0
6pqhjod             berkeley-db@18.1.40
mq5csuf             bzip2@1.0.8
afgghmq             gdbm@1.23
jtrpaef                 readline@8.1
hnw45un         pmix@4.2.3
jftho3y             libevent@2.1.12
gsyujwx         zlib@1.2.13

==> 1 installed package

Do we have any standardised way of testing the installed gromacs?

[giordano]

For future record: as discussed in #49 (comment) we decided to use gcc@13.1.0 as base compiler, but it turned out that cuda@12.2.0 is incompatible with this version of GCC (nvcc throws an error if you attempt to use that as base compiler), which for example is a problem for CP2K (spec I'm currently experimenting with: 'cp2k@2023.1 +cuda +mpi +openmp ~sirius cuda_arch=80', based on https://github.com/UCL-RITS/rcps-buildscripts/blob/db896111408d215aec55e820cb250bc576387b21/cp2k-7.1_install).

We will try to use nvhpc@23.3 as compiler for CUDA packages, and gcc@13.1.0 for the rest and see how that goes.

Update (2024-02-28): we ran into too many problems with nvhpc (e.g. spack/spack#42879 and spack/spack-packages#953). Also, nvhpc can't be used to compile all packages, there are many known incompatibilities (e.g. python%nvhpc won't even concretize due to known conflicts). Next step is try to compile most of our stack with gcc@13.1.0, and then try to use gcc@12.3.0 for the CUDA packages (hoping we don't run again into the ICEs in the vectoriser mentioned at #49 (comment)).

Update (2024-03-04): gcc@12.3.0 has fixed the ICEs we ran into with gcc@12.2.0, so we should be good with using it as the only compiler for the entire stack.

[heatherkellyucl]

LAMMPS (GNU, CPU) is going to look like this, as they specify ALL the packages individually where we'd been using presets/most.cmake for the recent releases.

+mpi +openmp are on by default, but I figured since we were specifying everything else it would be worth saying so.

lammps@20240207.1 +mpi +python +amoeba +asphere +bocs +body +bpm +brownian +cg-dna +cg-spica +class2 +colloid +colvars +compress +coreshell +dielectric +diffraction +dipole +dpd-basic +dpd-meso +dpd-react +dpd-smooth +drude +eff +electrode +extra-compute +extra-dump +extra-fix +extra-molecule +extra-pair +fep +granular +interlayer +kspace +lepton +machdyn +manybody +mc +meam +mesont +misc +ml-iap +ml-pod +ml-snap +mofff +molecule +openmp +opt +orient +peri +phonon +plugin +poems +qeq +reaction +reaxff +replica +rigid +shock +sph +spin +srd +tally +uef +voronoi +yaff

Only thing missing that we had mentioned in the last install was quip, which isn't an option.

For CUDA, build with GNU. Same packages. +cuda cuda_arch=80. Needed doing on GPU node last time.

For Intel, we do one version with +user-intel and one without.

[heatherkellyucl]

CASTEP, ORCA, NAMD are all manual-download packages.

You can set a mirror or source_cache to be checked and they should be found in it, or you can specify exactly the location of each package (can be in environment's .yaml) like

https://spack.readthedocs.io/en/latest/packages_yaml.html#assigning-package-attributes

You can assign class-level attributes in the configuration.

packages:
 mpileaks:
   package_attributes:
     # Override existing attributes
     url: http://www.somewhereelse.com/mpileaks-1.0.tar.gz
     # ... or add new ones
     x: 1

Attributes set this way will be accessible to any method executed in the package.py file (e.g. the install() method). Values for these attributes may be any value parseable by yaml.

We have the pkg-store so setting that as a source mirror should work.