[Common/TE] Replace hardcoded CUDA paths with CUDAToolkit discovery - #3215
Merged
Conversation
caozhanhao
requested review from
00fish0,
alogfans,
chestnut-Q,
doujiang24,
dtcccc and
staryxchen
as code owners
July 31, 2026 06:16
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When multiple CUDA toolkits are installed, CMake may discover one toolkit while
common.cmakestill adding include and library search paths from the hardcoded/usr/local/cudalocation.This issue was identified on a machine with both CUDA 12.8 and CUDA 13.0 installed, where
/usr/local/cudapoints to CUDA 12.8. CUDA 13.0 was selected for the build using the following environment configuration:With a clean build directory, CMake finds
/usr/local/cuda-13.0/bin/nvccthroughPATHand correctly selects CUDA 13.0. However,common.cmakestill adds/usr/local/cuda/includeto the include search path. As a result, CUDA 12.8 headers take precedence over the headers from the selected CUDA 13.0 toolkit.Consequently, CUDA 13.0's
nvccemits host-side launch code using the CUDA 13.0 interface, but header resolution selects CUDA 12.8's runtime headers. This interface mismatch results in errors such as:This is a pre-existing issue exposed by #3206, which decouples PG from PyTorch and builds it directly as a CMake target. Previously, PG's
.cusources were compiled separately through PyTorch'sCUDAExtensionand therefore did not inherit the top-level CMake include paths. After #3206, these sources become part of the top-level CMake build, exposing the existing toolkit mismatch.This PR updates the shared CUDA configuration in
common.cmakeand the affected TE and TENT targets to consistently use the CUDA toolkit selected byFindCUDAToolkit:/usr/local/cudainclude and library paths withCUDAToolkit_INCLUDE_DIRSandCUDAToolkit_LIBRARY_DIR.CUDA::cuda_driverandCUDA::cudarttargets.cufile.handlibcufile.Module
mooncake-transfer-engine)mooncake-store)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Verified builds in both configurations: this PR alone and this PR combined with #3206.
Checklist
./scripts/code_format.shpre-commit run --all-filesand all hooks passAI Assistance Disclosure
Codex was used to assist in implementing this PR. All code changes have been thoroughly reviewed and verified by me in person.