Skip to content

feat: ship with CUDA provider libs and preload CUDA runtime on load - #10

Merged
talmago merged 2 commits into
mainfrom
talmago/cuda_build
Aug 5, 2026
Merged

talmago merged 2 commits into
mainfrom
talmago/cuda_build

Conversation

@talmago

@talmago talmago commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Summary

  • Bundle ORT CUDA provider plugins (providers_shared / providers_cuda) into CUDA-profile wheels only after auditwheel, so libonnxruntime_providers_cuda.so is found next to the vendored runtime (CPU wheels unchanged).
  • Add nvidia-cublas (and onnxruntime-gpu[cuda,cudnn]) to the cuda extra so CUDA 13 user libs are installable via pip install "fast-lightonocr[cuda]".
  • When runtime_kwargs selects execution_provider="cuda", call onnxruntime.preload_dlls(cuda=True, cudnn=True) before native session create so pip NVIDIA libs resolve without LD_LIBRARY_PATH. CPU loads skip this path entirely.
  • Reorganize the Python README install / source-build docs (CPU vs CUDA).

Test plan

  • Colab: BUILD_PROFILE=cuda pip install -v ".[cuda]" from this branch; wheel contains provider libs in fast_lightonocr.libs
  • Colab: LightOnOCR.from_pretrained(..., runtime_kwargs={"execution_provider": "cuda"}) runs OCR
  • Colab: nvidia-smi sampling during process() shows ~2 GiB / non-zero util
  • Local: default CPU pip install -v ".[cpu]" still works (no provider bundling / no preload)
  • Confirm BUILD_PROFILE=cpu wheel RECORD is unchanged vs pre-PR behavior

talmago added 2 commits August 5, 2026 15:58
Inject providers_shared/cuda after auditwheel (CPU wheels untouched) and pull in runtime CUDA 13 deps via the cuda extra.
Call `onnxruntime.preload_dlls` only when execution_provider is cuda
and reorganize the Python README install/source-build sections.
@talmago
talmago merged commit e8672b3 into main Aug 5, 2026
1 check passed
@talmago
talmago deleted the talmago/cuda_build branch August 17, 2026 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant