Replace HDF5 and NetCDF functions here and in all downstream codes by ISO C bindings

[krystophny]

Codes that need only HDF5 will not need NetCDF, and no-one will need Fortran packages for the two. One could also think about doing this in lazy-fortran orga.

[krystophny]

Automated triage pass (Claude), grounded in the repository at fe3f76e. Verify before acting.

Purpose

libneo currently depends on the Fortran bindings of HDF5 and NetCDF (libhdf5-fortran, netcdf-fortran). Unlike the C libraries, these ship compiler-specific .mod files and a compiler-specific ABI, so a system install built with gfortran-13 is unusable from ifx, flang-new or nvfortran. libneo already pays for this in full: cmake/DetectDependencies.cmake:34-114 runs a bespoke check_fortran_dependency() compile probe against use hdf5 and use netcdf, and on failure falls back to a source superbuild of HDF5 1.14.5, NetCDF-C 4.9.2 and NetCDF-Fortran 4.6.1 (cmake/BuildExternalDependencies.cmake:42,144,190). should_force_fetch_deps() makes that fallback unconditional for NVIDIA/PGI (DetectDependencies.cmake:18-25), which is exactly the compiler used by make nvfortran.

Binding the C APIs through iso_c_binding removes the compiler-ABI coupling entirely: libnetcdf/libhdf5 are stable C shared objects present on essentially every HPC system and in every distro. The payoff is a faster, more reliable configure step on MPCDF/Viper and in CI, one fewer source superbuild, and a path for flang-new (needed for the Enzyme work in #276) that does not require rebuilding two Fortran packages. It also makes #268 ("Make HDF5 Fortran an optional dependency") moot: there is no HDF5-Fortran left to make optional.

Current state

Two wrapper modules own almost all of the traffic:

File	Lines	Wraps
src/hdf5_tools/hdf5_tools.F90	1977	use hdf5, use h5lt
src/hdf5_tools/hdf5_tools_f2003.f90	205	HDF5 F2003 interface
src/nctools/nctools_module.f90	855	use netcdf

but there is also substantial direct use that bypasses them. rg -c 'nf90_' over src/ gives: nctools_module.f90 (129), src/field/mgrid.f90 (57), src/magfie/coil_tools.f90 (55), src/field/boozmn_file.f90 (47), src/field/boozer_chartmap_io.f90 (46), src/boozer_chartmap.f90 (42), src/coordinates/libneo_chartmap_vmec_generator.f90 (26), src/coordinates/libneo_coordinates_validator.f90 (25), src/coordinates/libneo_coordinates_file_detection.f90 (14), src/coordinates/libneo_coordinates_chartmap.f90 (10), src/vmec_support/new_vmec_allocation_stuff.f90 (4). Tests add ~350 more call sites (test/source/test_chartmap_validator.f90 alone has 159).

The good news is that the used API subset is small and closed. The complete set of HDF5 Fortran entry points in src/ + test/ is 46 procedures (h5open_f, h5fcreate_f, h5fopen_f, h5fclose_f, h5gcreate_f/h5gopen_f/h5gclose_f/h5gn_members_f/h5gget_info_by_name_f/h5gget_obj_info_idx_f, h5dcreate_f/h5dopen_f/h5dread_f/h5dwrite_f/h5dget_space_f/h5dset_extent_f/h5dclose_f, h5screate_simple_f/h5sselect_hyperslab_f/h5sclose_f, h5tcreate_f/h5tcopy_f/h5tinsert_f/h5tset_size_f/h5tget_size_f/h5tclose_f, h5pcreate_f/h5pset_chunk_f/h5pclose_f, h5lexists_f/h5ldelete_f/h5ocopy_f/h5iis_valid_f/h5aexists_by_name_f/h5eset_auto_f/h5fget_obj_count_f/h5close_f, plus 9 h5lt* HL calls). The named constants are just 15: H5F_ACC_RDONLY_F, H5F_ACC_RDWR_F, H5F_ACC_TRUNC_F, H5F_OBJ_ALL_F, H5P_DATASET_CREATE_F, H5S_SELECT_SET_F, H5S_UNLIMITED_F, H5T_COMPOUND_F, H5T_FORTRAN_S1, H5T_NATIVE_DOUBLE, H5T_NATIVE_INTEGER, H5_INTEGER_KIND, HID_T, HSIZE_T, SIZE_T. On the NetCDF side it is 24 nf90_* procedures and 8 constants.

Typical call site that must keep working byte-for-byte (src/nctools/nctools_module.f90:549-557):

  subroutine nc_get_double_2(ncid, name, var)
    integer :: ncid
    character(len=*) :: name
    double precision, dimension(:,:), intent(out) :: var
    integer :: varid

    call nf90_check(nf90_inq_varid(ncid, name, varid))
    call nf90_check(nf90_get_var(ncid, varid, var))
  end subroutine nc_get_double_2

and the HDF5 hyperslab path (src/hdf5_tools/hdf5_tools.F90:534-541), which is the hardest case because it mixes HSIZE_T offsets with Fortran array ordering:

    call h5screate_simple_f(rank, dims, memspace, h5error)
    call h5dget_space_f(dsetid, dspaceid, h5error)
    call h5sselect_hyperslab_f(dspaceid, H5S_SELECT_SET_F, offsetd, countd, h5error)
    call h5dwrite_f(dsetid, H5T_NATIVE_INTEGER, value, dims, h5error, memspace, dspaceid)

Reproduction / evidence

The dependency problem is directly observable:

cd /path/to/libneo
cmake --preset default 2>&1 | grep -A2 'NetCDF-Fortran\|system HDF5'
# On a box where nf-config points at a gfortran-built netcdf-fortran and
# CMAKE_Fortran_COMPILER=ifx you get:
#   -> ABI incompatible, will build from source
# and the configure step then clones netcdf-c v4.9.2 + netcdf-fortran v4.6.1
# and untars hdf5-1.14.5 (cmake/BuildExternalDependencies.cmake:42,144,190).
FC=nvfortran cmake --preset default   # unconditionally superbuilds, DetectDependencies.cmake:18-25

Verification recipe for the finished work (this is a refactor, so the criterion is bit-identical output):

make && cd build && ctest --output-on-failure
# baseline artefacts to freeze BEFORE the change:
h5dump -pBH build/test/<hdf5 artefact from test_hdf5_tools> > /tmp/before.h5.txt
ncdump -h  build/test/<chartmap artefact>.nc                > /tmp/before.nc.txt
# after the change, the same two dumps must be byte-identical:
diff /tmp/before.h5.txt /tmp/after.h5.txt && diff /tmp/before.nc.txt /tmp/after.nc.txt

The two dedicated regression tests already exist and are registered: test/CMakeLists.txt:155,267 (test_hdf5_tools.x / test_hdf5_tools) and test/CMakeLists.txt:241,288 (test_nctools_nc_get3.x / test_nctools_nc_get3).

The three landmines

These are the reason a naive port silently corrupts files, and they must be handled explicitly:

1. Dimension order. NetCDF-Fortran reverses dimension order relative to the C API: in nf90_def_var the fastest-varying dimension comes first, in nc_def_var it comes last. The same reversal applies to start/count. A shim over nc_* must reverse dimids, start and count vectors.
2. Index base. nf90_get_var(..., start=[1,1,1]) is 1-based (src/field/boozer_chartmap_io.f90:140-144); nc_get_vara_double is 0-based. Subtract one.
3. HDF5 dataspace order. h5screate_simple_f(rank, dims, ...) takes dims in Fortran order and the wrapper reverses them before calling H5Screate_simple, so a Fortran (n1,n2) array appears in h5dump/h5py as shape [n2,n1]. Reproducing this reversal is mandatory or every existing .h5 file written by NEO-2/KAMEL becomes unreadable by its Python consumers.

Behaviour (BDD)

Scenario: NetCDF C bindings produce a byte-identical chartmap file
  Given the chartmap writer in src/field/boozer_chartmap_io.f90 and a reference
        *.chartmap.nc produced at fe3f76e with netcdf-fortran 4.6.1
  When libneo is rebuilt with the nctools C-binding backend and the same
        chartmap is regenerated
  Then "ncdump -h" of the new file is identical to the reference header
   And every double variable matches the reference to exactly 0 ULP

Scenario: Hyperslab read through the C binding matches the Fortran binding
  Given the fixture written by test/source/test_hdf5_tools.f90
  When h5_get with offset_par/count_par is called via h5_get_double_4_hyperslab
        (src/hdf5_tools/hdf5_tools.F90:1172)
  Then the returned 4-D block equals the Fortran-binding result elementwise
   And h5py reading the same dataset reports shape reversed relative to the
        Fortran declaration, as it does today

Scenario: Configure succeeds without netcdf-fortran installed
  Given a container built from setup/debian.sh with libnetcdff-dev removed
  When "cmake --preset default && cmake --build --preset default" runs
  Then configure does not print "will build from source"
   And no NetCDF-Fortran FetchContent clone is performed
   And "ctest" passes with zero failures

Scenario: nvfortran no longer triggers the source superbuild
  Given FC=nvfortran and system libhdf5 / libnetcdf (C only)
  When "make nvfortran" runs
  Then cmake/BuildExternalDependencies.cmake is not entered for HDF5 or NetCDF
   And "make test-nvfortran" passes

Implementation plan

This is not one PR. Split it into the atomic issues below; the ordering is a hard PR dependency chain because of the downstream gate in ci/downstreams.

PR 1 — NetCDF C interface layer (libneo only, no call-site changes).

1. Create src/nctools/netcdf_c_interface.f90: module netcdf_c_interface, use, intrinsic :: iso_c_binding, bind(c) interfaces for the C subset actually needed — nc_create, nc_open, nc_close, nc_redef, nc_enddef, nc_def_dim, nc_def_var, nc_def_grp, nc_inq_ncid, nc_inq_varid, nc_inq_dimid, nc_inq_dimlen, nc_inq_var, nc_inq_attlen, nc_inq_atttype, nc_put_att_text, nc_put_att_double, nc_get_att_text, nc_get_att_double, nc_put_vara_double, nc_put_vara_int, nc_put_vara_text, nc_get_vara_double, nc_get_vara_int, nc_get_vara_text, nc_strerror. Types: integer(c_int) for ncid/varid/status, integer(c_size_t) for start/count, character(kind=c_char) strings.
2. Create src/nctools/netcdf_shim.f90: module netcdf (same module name as netcdf-fortran, so no call site changes anywhere) re-exporting nf90_open/create/close/def_dim/def_var/def_grp/enddef/redef/inq_varid/inq_dimid/inq_ncid/inquire_dimension/inquire_variable/inquire_attribute/put_att/get_att/put_var/get_var/strerror and the constants nf90_noerr, nf90_nowrite, nf90_clobber, nf90_global, nf90_double, nf90_int, nf90_max_name, nf90_max_var_dims. Apply the reversal + index-base conversion here and nowhere else.
3. Edit src/nctools/CMakeLists.txt: add both new sources to add_library(nctools ...), and replace the whole 45-line netcdf::netcdff / nf-config --flibs cascade with pkg_check_modules(NETCDF REQUIRED netcdf) (C only) plus target_link_libraries(nctools PUBLIC ${NETCDF_LIBRARIES}).
4. Add test/source/test_netcdf_shim.f90: round-trip 1-D/2-D/3-D double, int, and text variables plus a group and an attribute; assert nf90_get_var with explicit start/count returns the same slab as a full read. Register in test/CMakeLists.txt alongside test_nctools_nc_get3 (line 241) with add_test(NAME test_netcdf_shim ...) near line 288.
5. Add test/scripts/test_netcdf_layout_compat.py: open a chartmap written by the new code with netCDF4 and assert variable shapes and dimension names match the committed reference header.

PR 2 — HDF5 C interface layer.
6. Create src/hdf5_tools/hdf5_c_interface.f90: bind(c) interfaces for H5open/H5close, H5Fcreate/H5Fopen/H5Fclose/H5Fget_obj_count, H5Gcreate2/H5Gopen2/H5Gclose/H5Gget_info_by_name, H5Dcreate2/H5Dopen2/H5Dread/H5Dwrite/H5Dget_space/H5Dset_extent/H5Dclose, H5Screate_simple/H5Sselect_hyperslab/H5Sclose, H5Tcreate/H5Tcopy/H5Tinsert/H5Tset_size/H5Tget_size/H5Tclose, H5Pcreate/H5Pset_chunk/H5Pclose, H5Lexists/H5Ldelete/H5Ocopy/H5Iis_valid/H5Aexists_by_name/H5Eset_auto2, and the HL entry points H5LTmake_dataset_double/_int/_string, H5LTread_dataset_double/_int/_string, H5LTset_attribute_double/_int/_string, H5LTget_attribute_int. Define integer, parameter :: HID_T = c_int64_t, HSIZE_T = c_int64_t, SIZE_T = c_size_t.
7. Create src/hdf5_tools/hdf5_shim.f90: module hdf5 and module h5lt exposing the 46 *_f names with the current argument lists, so hdf5_tools.F90 (1977 lines) and hdf5_tools_f2003.f90 are not rewritten. The native type ids H5T_NATIVE_DOUBLE, H5T_NATIVE_INTEGER, H5T_FORTRAN_S1 must be initialised at h5open_f time from the C globals (they are not link-time constants in C either — resolve them via H5T_NATIVE_DOUBLE_g etc. or via H5Tcopy of the predefined ids exposed by a 3-line C shim in src/hdf5_tools/hdf5_native_types.c; adding that C file is acceptable, enable_language(C) is already set at src/hdf5_tools/CMakeLists.txt:6).
8. Edit src/hdf5_tools/CMakeLists.txt: add the new sources; change find_package(HDF5 COMPONENTS C HL Fortran Fortran_HL REQUIRED) (line ~17) to find_package(HDF5 COMPONENTS C HL REQUIRED); drop hdf5::hdf5_fortran and hdf5::hdf5_hl_fortran from target_link_libraries.
9. Extend test/source/test_hdf5_tools.f90 with a compound-type case and a h5_get_double_4_hyperslab case, and add test/scripts/test_hdf5_layout_compat.py asserting h5py sees the historically expected (reversed) shapes.

PR 3 — CMake and CI cleanup.
10. Edit cmake/DetectDependencies.cmake: delete the check_fortran_dependency("HDF5-Fortran", ...) and check_fortran_dependency("NetCDF-Fortran", ...) probes (lines 43-49, 87-93), replace nf-config/nc-config discovery with pkg_check_modules(netcdf), and delete NEED_BUILD_NETCDF's "NetCDF depends on HDF5" coupling at lines 111-113.
11. Edit cmake/BuildExternalDependencies.cmake: remove the NetCDF-Fortran v4.6.1 block (line ~190) and the HDF5_BUILD_FORTRAN option from the HDF5 block (line ~42).
12. Edit .github/workflows/main.yml:50 and setup/debian.sh:5: drop libnetcdff-dev. Keep libnetcdf-dev and libhdf5-dev. This is the single clearest proof the migration worked.

PR 4+ — downstreams, one PR per repo, gated in the order given by ci/downstreams. Each downstream that does use netcdf / use hdf5 picks up libneo's shim modules automatically if it links LIBNEO::nctools / LIBNEO::hdf5_tools; the work is removing its own find_package(NetCDF COMPONENTS Fortran). Affected files: SIMPLE/src/netcdf_results_output.f90, SIMPLE/src/netcdf_orbit_output.f90, SIMPLE/src/field/field_boozer_chartmap.f90, SIMPLE/src/wall/chartmap_metadata.f90; MEPHIT/src/mephit_{conf,util,iter,mesh,pert,post,test,flr2}.f90; NEO-2/COMMON/{collop_compute,binarysplit_mod,neo_sub,collision_operator_mems}.f90 and NEO-2/{NEO-2-QL,NEO-2-PAR}/propagator.f90; rabe/src/netcdf/netcdf.f90 (note rabe already has a module named netcdf — resolve the clash before PR 1 lands); NEO-RT/POTATO/SRC/neo2_reader.f90. KAMEL/common/hdf5_tools/hdf5_tools.f90 is a stale private copy of libneo's file — fold this into itpplasma/KAMEL#54 rather than porting it twice.

Acceptance criteria

• rg -l 'use hdf5$|use h5lt$|use netcdf$' src/ test/ resolves only to modules defined inside libneo; no external Fortran binding package is referenced.
• find_package(HDF5 ... Fortran ...) and nf-config appear nowhere in cmake/ or any CMakeLists.txt.
• libnetcdff-dev is removed from .github/workflows/main.yml and setup/debian.sh, and CI is green.
• make && cd build && ctest --output-on-failure passes with zero failures, including test_hdf5_tools and test_nctools_nc_get3.
• New tests test_netcdf_shim and the extended test_hdf5_tools are registered in test/CMakeLists.txt and pass.
• ncdump -h and h5dump -pBH of every artefact written under build/test/ are byte-identical to the pre-change baselines; all double data matches to 0 ULP (this is a refactor — no tolerance is granted).
• test/source/test_boozer_converter_vs_simple.f90 and test/scripts/test_ascot5_compare.py still pass unchanged.
• The full ci/downstreams gate (SIMPLE, NEO-2, MEPHIT, KAMEL, rabe blocking) is green on the libneo PR.
• FC=nvfortran make nvfortran && make test-nvfortran succeeds without entering cmake/BuildExternalDependencies.cmake.

References

• The HDF5 C API and its hid_t/herr_t/hsize_t typedefs, and the H5LT high-level API, are specified in the HDF Group's HDF5 Reference Manual (HDF5 1.14.x) — this is the authoritative contract for step 6.
• The dimension-order inversion between the Fortran and C NetCDF interfaces is documented in Unidata's NetCDF Fortran 90 Interface Guide (Rew, Davis, Emmerson, Davies, Hartnett, Heimbigner); see the "Dimensions" and "Variables" sections. It is the single most common source of silent corruption in ports of this kind.
• Prior art for calling these C libraries directly from Fortran: the h5fortran project (Michael Hirsch) wraps the HDF5 C library from modern Fortran; VMEC++ (github.com/proximafusion/vmecpp) dropped the Fortran NetCDF bindings in favour of the C/C++ ones for exactly the ABI-portability reason argued here. SIMPLE and ASCOT5 both consume the C libraries in their non-Fortran layers already.
• The compiler-ABI motivation connects to the flang-new requirement in Prepare VMEC/Boozer interpolation kernels for Enzyme autodifferentiation #276 (Enzyme AD); Enzyme itself: Moses & Churavy, Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients, NeurIPS 2020.

Scope and risk

Large: roughly 3 libneo PRs plus one per downstream, and it should be split into atomic issues along the PR boundaries above. The dominant risk is silent on-disk layout change — dimension order, index base, and string null-termination — which is why every acceptance criterion is a byte-identity check against a frozen baseline rather than a numerical tolerance. Secondary risk: the module-name shadowing trick (libneo defining module netcdf) breaks any downstream that links both libneo and a real netcdf-fortran; rabe already defines its own netcdf module in rabe/src/netcdf/netcdf.f90 and must be reconciled first.

Related: this supersedes #268 (making HDF5-Fortran optional becomes vacuous), overlaps itpplasma/KAMEL#54 (KAMEL's private hdf5_tools copy), and unblocks part of #276.