Skip to content

[Common/TE] Replace hardcoded CUDA paths with CUDAToolkit discovery - #3215

Merged
alogfans merged 1 commit into
mainfrom
czh/te-cuda-headers
Aug 3, 2026
Merged

[Common/TE] Replace hardcoded CUDA paths with CUDAToolkit discovery#3215
alogfans merged 1 commit into
mainfrom
czh/te-cuda-headers

Conversation

@caozhanhao

@caozhanhao caozhanhao commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Description

When multiple CUDA toolkits are installed, CMake may discover one toolkit while common.cmake still adding include and library search paths from the hardcoded /usr/local/cuda location.

This issue was identified on a machine with both CUDA 12.8 and CUDA 13.0 installed, where /usr/local/cuda points to CUDA 12.8. CUDA 13.0 was selected for the build using the following environment configuration:

export CUDA_HOME=/usr/local/cuda-13.0
export PATH="${CUDA_HOME}/bin:${PATH}"

With a clean build directory, CMake finds /usr/local/cuda-13.0/bin/nvcc through PATH and correctly selects CUDA 13.0. However, common.cmake still adds /usr/local/cuda/include to the include search path. As a result, CUDA 12.8 headers take precedence over the headers from the selected CUDA 13.0 toolkit.

Consequently, CUDA 13.0's nvcc emits host-side launch code using the CUDA 13.0 interface, but header resolution selects CUDA 12.8's runtime headers. This interface mismatch results in errors such as:

error: macro "__cudaLaunch" passed 2 arguments, but takes just 1

This is a pre-existing issue exposed by #3206, which decouples PG from PyTorch and builds it directly as a CMake target. Previously, PG's .cu sources were compiled separately through PyTorch's CUDAExtension and therefore did not inherit the top-level CMake include paths. After #3206, these sources become part of the top-level CMake build, exposing the existing toolkit mismatch.

This PR updates the shared CUDA configuration in common.cmake and the affected TE and TENT targets to consistently use the CUDA toolkit selected by FindCUDAToolkit:

  • Replace hardcoded /usr/local/cuda include and library paths with CUDAToolkit_INCLUDE_DIRS and CUDAToolkit_LIBRARY_DIR.
  • Link the main TE target against the imported CUDA::cuda_driver and CUDA::cudart targets.
  • Prefer the selected CUDA toolkit when locating TENT's cufile.h and libcufile.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Verified builds in both configurations: this PR alone and this PR combined with #3206.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit run --all-files and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

Codex was used to assist in implementing this PR. All code changes have been thoroughly reviewed and verified by me in person.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@alogfans alogfans left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alogfans
alogfans merged commit 3dabe12 into main Aug 3, 2026
52 of 57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants