Summary
With #2796, ROCm device-architecture detection changes from a constant ("gfx942") to a live device query (cp.cuda.runtime.getDeviceProperties(0)). The process-pool compile workers (GT4PY_BUILD_JOBS_MODE=process) were never taught the ROCm side of the arch-forwarding workaround that exists for CUDA, so on ROCm every pool worker now probes the GPU itself — and when that probe fails, the build silently falls back to the compiler-default architecture and produces a binary that cannot run on the actual device.
Raised in review of #2796: #2796 (comment). This is the concrete ROCm instance of the general contract problem tracked in #2694.
Mechanism
For CUDA, _pool_worker_initializer (src/gt4py/next/otf/runners.py) deliberately keeps workers off the GPU: the main process resolves the arch once, ships it to each worker as CUDAARCHS, and sets CUDA_VISIBLE_DEVICES="" so no build step in the worker creates a CUDA context on the device the parent may be actively using (introduced in #2679).
_detect_cuda_archs() is explicitly CUDA-only (runners.py:108) and returns None on ROCm, so ROCm workers get neither a HIPARCHS export nor device hiding. Each worker's build then reaches get_device_arch() (src/gt4py/next/otf/compilation/common.py) with HIPARCHS unset and calls getDeviceProperties(0) itself. Two consequences:
- GPU interference. Every worker creates a HIP context on the parent's GPU just to read the arch — exactly what the CUDA workaround exists to avoid while the parent is running kernels.
- Silent wrong-arch fallback. If the query fails in a worker (device saturated — e.g.
cudaErrorMemoryAllocation under many concurrent test processes — or devices masked in the environment), _query_device_arch() only warns and returns None. The result is cached for the worker's lifetime (functools.cache). CMake is then invoked without -DCMAKE_HIP_ARCHITECTURES (build_systems/cmake.py:get_cmake_device_arch_option), and the DaCe path skips its compiler.cuda.hip_arch override (runners/dace/workflow/common.py), so the code object is built for the ROCm compiler's default gfx* target instead of the present device.
Failure mode
On a ROCm machine with GT4PY_BUILD_JOBS_MODE=process (the default where process pools are enabled), whenever the worker-side device probe fails:
- The only diagnostic is a
UserWarning inside the worker process ("Could not determine the HIP device architecture: ...") — easy to miss, and the build then succeeds for the wrong architecture.
- At first kernel launch the program fails at runtime. Observed behaviours for a gfx ISA mismatch on ROCm: a
"no kernel image is available"-class error, or — worse — a segfault inside libamdhip64 at hipLaunchKernel with no clean Python-level error at all (observed with ROCm 7.2 on gfx1103 when a fat binary lacked the device's ISA).
This is precisely the class of "no matching code object" failure that #2796 sets out to fix for the main process, reintroduced in pool workers. Before #2796 the ROCm branch was a constant, so workers never touched the device and the behaviour, while hard-coded, was deterministic.
Suggested fix
Minimal, symmetric to CUDA: resolve the arch once in the main process for ROCm too, and have _pool_worker_initializer export HIPARCHS (already honoured by get_device_arch()) and hide the devices from workers (HIP_VISIBLE_DEVICES=""/ROCR_VISIBLE_DEVICES="" as the ROCm counterpart of CUDA_VISIBLE_DEVICES="").
Long term, the clean solution is the contract change discussed in #2694 / the TODO in _pool_worker_initializer: GPU backends receive the target architecture as an explicit compile-step argument, and no build step performs device detection.
References
Summary
With #2796, ROCm device-architecture detection changes from a constant (
"gfx942") to a live device query (cp.cuda.runtime.getDeviceProperties(0)). The process-pool compile workers (GT4PY_BUILD_JOBS_MODE=process) were never taught the ROCm side of the arch-forwarding workaround that exists for CUDA, so on ROCm every pool worker now probes the GPU itself — and when that probe fails, the build silently falls back to the compiler-default architecture and produces a binary that cannot run on the actual device.Raised in review of #2796: #2796 (comment). This is the concrete ROCm instance of the general contract problem tracked in #2694.
Mechanism
For CUDA,
_pool_worker_initializer(src/gt4py/next/otf/runners.py) deliberately keeps workers off the GPU: the main process resolves the arch once, ships it to each worker asCUDAARCHS, and setsCUDA_VISIBLE_DEVICES=""so no build step in the worker creates a CUDA context on the device the parent may be actively using (introduced in #2679)._detect_cuda_archs()is explicitly CUDA-only (runners.py:108) and returnsNoneon ROCm, so ROCm workers get neither aHIPARCHSexport nor device hiding. Each worker's build then reachesget_device_arch()(src/gt4py/next/otf/compilation/common.py) withHIPARCHSunset and callsgetDeviceProperties(0)itself. Two consequences:cudaErrorMemoryAllocationunder many concurrent test processes — or devices masked in the environment),_query_device_arch()only warns and returnsNone. The result is cached for the worker's lifetime (functools.cache). CMake is then invoked without-DCMAKE_HIP_ARCHITECTURES(build_systems/cmake.py:get_cmake_device_arch_option), and the DaCe path skips itscompiler.cuda.hip_archoverride (runners/dace/workflow/common.py), so the code object is built for the ROCm compiler's defaultgfx*target instead of the present device.Failure mode
On a ROCm machine with
GT4PY_BUILD_JOBS_MODE=process(the default where process pools are enabled), whenever the worker-side device probe fails:UserWarninginside the worker process ("Could not determine the HIP device architecture: ...") — easy to miss, and the build then succeeds for the wrong architecture."no kernel image is available"-class error, or — worse — a segfault insidelibamdhip64athipLaunchKernelwith no clean Python-level error at all (observed with ROCm 7.2 on gfx1103 when a fat binary lacked the device's ISA).This is precisely the class of "no matching code object" failure that #2796 sets out to fix for the main process, reintroduced in pool workers. Before #2796 the ROCm branch was a constant, so workers never touched the device and the behaviour, while hard-coded, was deterministic.
Suggested fix
Minimal, symmetric to CUDA: resolve the arch once in the main process for ROCm too, and have
_pool_worker_initializerexportHIPARCHS(already honoured byget_device_arch()) and hide the devices from workers (HIP_VISIBLE_DEVICES=""/ROCR_VISIBLE_DEVICES=""as the ROCm counterpart ofCUDA_VISIBLE_DEVICES="").Long term, the clean solution is the contract change discussed in #2694 / the TODO in
_pool_worker_initializer: GPU backends receive the target architecture as an explicit compile-step argument, and no build step performs device detection.References