fix(docker): restore libcudnn.so.9 in :cuda image - #18
Merged
Conversation
The published :main-cuda image fails on `import torch` because the second `uv sync` (project-install pass) reconciles against the lockfile, sees nvidia-cudnn-cu13 dist-info already present from the cu130 torch install, and silently strips libcudnn.so.9 off disk while keeping the dist-info shell — leaving torch's _C extension unable to load. Two surgical changes to Dockerfile.cuda: 1. Pin the cu130 install URL explicitly (was cu124, which was masking a silent fallback to cu130) and add --extra-index-url for non-torch transitives. 2. Replace the second `uv sync` (which was the only step that could un-install nvidia-cudnn) with `uv pip install --no-deps -e .`. We only need the project itself in the venv at that point; deps are already correctly laid down by step 1 + the cu130 install. Verified end-to-end on this machine: - libcudnn.so.9 is present at /app/.venv/lib/python3.11/site-packages/nvidia/cudnn/lib/ - `import torch` succeeds; torch.cuda.is_available() is True - GPU matmul (2048x2048) runs on RTX 5000 Ada, cuda:0 - Full nvidia profile stack boots cleanly: app /health=200, labeler+trainer workers report `CUDA available: True` with the GPU visible Image size unchanged at 13.6 GB local; PR #8 (devel→runtime base) can land on top of this. Closes #17. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
oldhero5
added a commit
that referenced
this pull request
May 25, 2026
Cuts the local image from 13.6 GB to 8.69 GB (~36% / 4.9 GB saved). The published compressed size will drop accordingly. We ship prebuilt CUDA torch wheels and never invoke nvcc, so the devel base's CUDA SDK was pure dead weight. Verified locally on this machine (RTX 5000 Ada): - Image builds clean - `import torch` succeeds; torch.cuda.is_available() is True - 2048x2048 matmul on cuda:0 works (no JIT/nvcc requirement surfaced) Builds on top of #18 which fixed the underlying libcudnn install — this is now just the one-line base swap the issue scoped. Closes #8. Co-authored-by: oldhero5 <oldhero5@proton.me> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The published `oldhero5/waldo:main-cuda` image fails on `import torch` with `ImportError: libcudnn.so.9: cannot open shared object file`. Verified locally on this branch: now imports cleanly, torch sees the GPU, GPU matmul runs.
Root cause
`Dockerfile.cuda` had two `uv sync` steps with a `uv pip install ... torch torchvision` between them. The second `uv sync` reconciled against the lockfile, saw `nvidia-cudnn-cu13` dist-info already present (from the cu130 torch install), and stripped `libcudnn.so.9` off disk while keeping the dist-info shell. `torch._C` then failed to load at import time.
Fix
Test plan
Unblocks #8 (devel→runtime base swap), which can now land as a one-line follow-up.
Closes #17.
🤖 Generated with Claude Code