Build failure with pip

[elliottslaughter]

This is similar to #988 except I am building with pip install . instead of make.

The build fails with:

  *** Building project with Ninja...
  2025-05-01 10:56:38,507 - scikit_build_core - DEBUG - RUNENV - changes since last run only:
  
  2025-05-01 10:56:38,507 - scikit_build_core - INFO - RUN: miniforge3/envs/build-legate-2025-04-25/bin/cmake --build legate/arch-linux-py-cuda-release/skbuild_core -v
  Change Dir: 'legate/arch-linux-py-cuda-release/skbuild_core'
 
  Run Build Command(s): /usr/bin/ninja-build -v -j 143
  ninja: error: build.ninja:12992: multiple outputs aren't (yet?) supported by depslog; bring this up on the mailing list if it affects you

Using the same environment and software versions as #988 (comment)

Using Legate main 27f4b34

Ninja version, to save you the trouble of checking the environment:

$ ninja --version                                                          
1.12.1

[manopapad]

@Jacobfaib ?

[Jacobfaib]

@elliottslaughter can you attach $LEGATE_DIR/$LEGATE_ARCH/skbuild_core/wheel.log?

[elliottslaughter]

I cannot find any log files at all under skbuild_core. Could it be named something else?

$ find arch-linux-py-cuda-release/skbuild_core -name '*.log'
$ ls arch-linux-py-cuda-release/skbuild_core
build.ninja     CMakeInit.txt          CPackSourceConfig.cmake  legate_python-config.cmake          LEGATE_RAPIDS.cmake
cmake           cmake_install.cmake    CPM_modules              legate_python-config-version.cmake  python
CMakeCache.txt  compile_commands.json  cpm-package-lock.cmake   legate_python-dependencies.cmake    rapids-cmake
CMakeFiles      CPackConfig.cmake      _deps                    legate_python-targets.cmake

[elliottslaughter]

I'm not sure what happened, but a clean build produced a wheel.log file. Attached below. Note some paths have been redacted with ....

wheel.log

[manopapad]

@elliottslaughter this is still affecting you?

[elliottslaughter]

I haven't tried since my last comment, but I haven't heard anything on this thread that would lead me to believe my issue has been fixed in the meantime.

It's less urgent now since binary packages have been updated, but overall still seems like a good idea to get the source builds fixed.

[manopapad]

I tried installing the environment from #988, then doing:

./configure --with-cuda --with-openmp --with-python
pip install .

I couldn't reproduce this error, but I did see the following

      [498/659] ccache /home/mpapadakis/miniforge3/envs/bug-988/bin/x86_64-conda-linux-gnu-c++ -DBS_THREAD_POOL_ENABLE_PAUSE=1 -DCPPTRACE_DEMANGLE_WITH_CXXABI -DCPPTRACE_GET_SYMBOLS_WITH_LIBDWARF -DCPPTRACE_HAS_CXX_EXCEPTION_TYPE -DCPPTRACE_HAS_DLADDR1 -DCPPTRACE_UNWIND_WITH_UNWIND -DCUFILE_FOUND -DFMT_HEADER_ONLY=1 -DKVIKIO_CUDA_FOUND -DKVIKIO_CUFILE_BATCH_API_FOUND -DKVIKIO_CUFILE_FOUND -DKVIKIO_CUFILE_STREAM_API_FOUND -DKVIKIO_LIBCURL_FOUND -DLIBCUDACXX_ENABLE_HOST_NVFP16=1 -DNDEBUG -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DUSE_CUDA -D_FORTIFY_SOURCE=2 -Dlegate_EXPORTS -I/home/mpapadakis/temp/legate.internal/src/cpp -I/home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/python/legate_cpp/cpp/include/legate -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cccl-src/thrust/thrust/cmake/../.. -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cccl-src/libcudacxx/lib/cmake/libcudacxx/../../../include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cccl-src/cub/cub/cmake/../.. -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-src/runtime -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-src/runtime/mappers -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-build/runtime -isystem /usr/local/cuda/targets/x86_64-linux/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/mdspan-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/span-src/include -isystem /home/mpapadakis/miniforge3/envs/bug-988/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/fmt-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/argparse-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/kvikio-src/cpp/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/bs_thread_pool-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/nvtx3-src/c/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/hdf5_vfd_gds-src/src -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/hdf5_vfd_gds-build/src -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/highfive-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cpptrace-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cpptrace-build/include -O3 -O3 -DNDEBUG -std=gnu++17 -fPIC -fdiagnostics-color=always -UTHRUST_DEVICE_SYSTEM -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP -Wall -Wextra -Werror -Walloca -Wdeprecated -Wimplicit-fallthrough -fdiagnostics-show-template-tree -Wignored-qualifiers -Wmissing-field-initializers -pedantic -Wsign-compare -Wshadow -Wswitch-enum -Walloc-zero -Wundef -Wtsan -Wenum-conversion -Wpacked -O3 -fstack-protector-strong -fopenmp -MD -MT python/legate_cpp/cpp/CMakeFiles/legate.dir/legate/data/physical_store.cc.o -MF python/legate_cpp/cpp/CMakeFiles/legate.dir/legate/data/physical_store.cc.o.d -o python/legate_cpp/cpp/CMakeFiles/legate.dir/legate/data/physical_store.cc.o -c /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/physical_store.cc
      FAILED: python/legate_cpp/cpp/CMakeFiles/legate.dir/legate/data/physical_store.cc.o
      ccache /home/mpapadakis/miniforge3/envs/bug-988/bin/x86_64-conda-linux-gnu-c++ -DBS_THREAD_POOL_ENABLE_PAUSE=1 -DCPPTRACE_DEMANGLE_WITH_CXXABI -DCPPTRACE_GET_SYMBOLS_WITH_LIBDWARF -DCPPTRACE_HAS_CXX_EXCEPTION_TYPE -DCPPTRACE_HAS_DLADDR1 -DCPPTRACE_UNWIND_WITH_UNWIND -DCUFILE_FOUND -DFMT_HEADER_ONLY=1 -DKVIKIO_CUDA_FOUND -DKVIKIO_CUFILE_BATCH_API_FOUND -DKVIKIO_CUFILE_FOUND -DKVIKIO_CUFILE_STREAM_API_FOUND -DKVIKIO_LIBCURL_FOUND -DLIBCUDACXX_ENABLE_HOST_NVFP16=1 -DNDEBUG -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DUSE_CUDA -D_FORTIFY_SOURCE=2 -Dlegate_EXPORTS -I/home/mpapadakis/temp/legate.internal/src/cpp -I/home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/python/legate_cpp/cpp/include/legate -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cccl-src/thrust/thrust/cmake/../.. -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cccl-src/libcudacxx/lib/cmake/libcudacxx/../../../include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cccl-src/cub/cub/cmake/../.. -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-src/runtime -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-src/runtime/mappers -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-build/runtime -isystem /usr/local/cuda/targets/x86_64-linux/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/mdspan-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/span-src/include -isystem /home/mpapadakis/miniforge3/envs/bug-988/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/fmt-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/argparse-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/kvikio-src/cpp/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/bs_thread_pool-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/nvtx3-src/c/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/hdf5_vfd_gds-src/src -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/hdf5_vfd_gds-build/src -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/highfive-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cpptrace-src/include -isystem /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/cpptrace-build/include -O3 -O3 -DNDEBUG -std=gnu++17 -fPIC -fdiagnostics-color=always -UTHRUST_DEVICE_SYSTEM -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP -Wall -Wextra -Werror -Walloca -Wdeprecated -Wimplicit-fallthrough -fdiagnostics-show-template-tree -Wignored-qualifiers -Wmissing-field-initializers -pedantic -Wsign-compare -Wshadow -Wswitch-enum -Walloc-zero -Wundef -Wtsan -Wenum-conversion -Wpacked -O3 -fstack-protector-strong -fopenmp -MD -MT python/legate_cpp/cpp/CMakeFiles/legate.dir/legate/data/physical_store.cc.o -MF python/legate_cpp/cpp/CMakeFiles/legate.dir/legate/data/physical_store.cc.o.d -o python/legate_cpp/cpp/CMakeFiles/legate.dir/legate/data/physical_store.cc.o -c /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/physical_store.cc
      In file included from /home/mpapadakis/miniforge3/envs/bug-988/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/vector:62,
                       from /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/inline_allocation.h:13,
                       from /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/buffer.h:9,
                       from /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/physical_store.h:11,
                       from /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/physical_store.cc:7:
      In static member function 'static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = legion_dimension_kind_t; _Up = legion_dimension_kind_t; bool _IsMove = false]',
          inlined from '_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = legion_dimension_kind_t*; _OI = legion_dimension_kind_t*]' at /home/mpapadakis/miniforge3/envs/bug-988/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/stl_algobase.h:506:30,
          inlined from '_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = legion_dimension_kind_t*; _OI = legion_dimension_kind_t*]' at /home/mpapadakis/miniforge3/envs/bug-988/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/stl_algobase.h:533:42,
          inlined from '_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = __gnu_cxx::__normal_iterator<legion_dimension_kind_t*, vector<legion_dimension_kind_t> >; _OI = legion_dimension_kind_t*]' at /home/mpapadakis/miniforge3/envs/bug-988/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/stl_algobase.h:540:31,
          inlined from '_OI std::copy(_II, _II, _OI) [with _II = __gnu_cxx::__normal_iterator<legion_dimension_kind_t*, vector<legion_dimension_kind_t> >; _OI = legion_dimension_kind_t*]' at /home/mpapadakis/miniforge3/envs/bug-988/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/stl_algobase.h:633:7,
          inlined from 'Legion::DeferredBuffer<T, DIM, COORD_T, CHECK_BOUNDS> Legion::OutputRegion::create_buffer(Legion::Point<DIM, COORD_T>&, Legion::FieldID, const T*, bool) [with T = cuda::std::__4::complex<double>; int DIM = 1; COORD_T = long long int; bool CHECK_BOUNDS = false]' at /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-src/runtime/legion/legion.inl:16739:16,
          inlined from 'legate::Buffer<T, DIM> legate::PhysicalStore::create_output_buffer(legate::Point<DIM>&, bool) const [with T = cuda::std::__4::complex<double>; int DIM = 1]' at /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/physical_store.inl:165:8,
          inlined from 'legate::TaskLocalBuffer legate::{anonymous}::CreateBuffer::operator()(const legate::PhysicalStore&, const legate::Type&, const Legion::DomainPoint&, bool) const [with legate::Type::Code CODE = legate::Type::Code::COMPLEX128; int DIM = 1]' at /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/physical_store.cc:42:16:
      /home/mpapadakis/miniforge3/envs/bug-988/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/stl_algobase.h:437:30: error: 'void* __builtin_memcpy(void*, const void*, long unsigned int)' forming offset 4 is out of the bounds [0, 4] of object 'ord' with type 'std::array<legion_dimension_kind_t, 1>' [-Werror=array-bounds=]
        437 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
            |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-src/runtime/legion.h:10544,
                       from /home/mpapadakis/temp/legate.internal/src/cpp/legate/utilities/typedefs.h:13,
                       from /home/mpapadakis/temp/legate.internal/src/cpp/legate/type/types.h:13,
                       from /home/mpapadakis/temp/legate.internal/src/cpp/legate/data/buffer.h:10:
      /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-src/runtime/legion/legion.inl: In function 'legate::TaskLocalBuffer legate::{anonymous}::CreateBuffer::operator()(const legate::PhysicalStore&, const legate::Type&, const Legion::DomainPoint&, bool) const [with legate::Type::Code CODE = legate::Type::Code::COMPLEX128; int DIM = 1]':
      /home/mpapadakis/temp/legate.internal/arch-linux-py-cuda-release/skbuild_core/_deps/legion-src/runtime/legion/legion.inl:16735:38: note: 'ord' declared here
      16735 |       std::array<DimensionKind, DIM> ord;
            |                                      ^~~

@Jacobfaib could you try to reproduce?

Maybe we can try doing this in a container.

[Jacobfaib]

This is a bug in GCC 13 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111273.

I don't think there's anything Legate can do, maybe Legion can do some massaging to help the compiler realize the error of its ways.

template<typename T, int DIM, typename COORD_T, bool CHECK_BOUNDS>
DeferredBuffer<T,DIM,COORD_T,CHECK_BOUNDS> OutputRegion::create_buffer(
                                              const Point<DIM,COORD_T> &extents,
                                              FieldID field_id,
                                              const T *initial_value /*= NULL*/,
                                              bool return_buffer /*= false*/)
{
  // ...
  std::vector<DimensionKind> ordering;
  // ...
  std::array<DimensionKind, DIM> ord;
#ifdef DEBUG_LEGION
  assert(ordering.size() == DIM);
#endif
  std::copy(ordering.begin(), ordering.end(), ord.begin()); 
  // ^^^ gcc is complaining about this line
}

I notice for example that the assert() is if-def-ed out. Maybe we just leave it in unconditionally (checking the size is trivially cheap anyway). That should help to optimizer realize that they are the same size. cc @lightsighter

[manopapad]

Getting back to the original build issue, presumably this issue

  ninja: error: build.ninja:12992: multiple outputs aren't (yet?) supported by depslog; bring this up on the mailing list if it affects you

has been fixed as of ninja 1.10 (see ninja-build/ninja#1184).

@elliottslaughter looking into the error message, it appears that the binary used is /usr/bin/ninja-build, could you please check the version of that? It's possible this is less recent than 1.10, and for some reason it's taking priority over the 1.12 ninja from conda.

[Jacobfaib]

https://gitlab.com/StanfordLegion/legion/-/merge_requests/1775 to fix the -Warray-bounds issue upstream