Skip to content

installation-guide.md

Latest commit

 

History

History
208 lines (142 loc) · 8.62 KB

installation-guide.md

File metadata and controls

208 lines (142 loc) · 8.62 KB
Raw

(installation-guide)= (containers)=

Installation Guide

There are multiple ways to install and run TensorRT LLM. The options below are ordered from simplest to most involved. Before installing, check the Supported Hardware page to ensure your GPU is compatible.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Nightly releases are development builds identified by a .dev segment in the version. They provide early access to the latest changes, but their quality is not guaranteed and they may contain bugs or regressions.

Option 1: Pre-built Release Container

Pre-built TensorRT LLM releases are available as container images on NGC. This is the simplest way to obtain TensorRT LLM.

Replace x.y.z with the desired version tag. The available tags on NGC include both regular and .dev nightly releases.

docker pull nvcr.io/nvidia/tensorrt-llm/release:x.y.z

docker run --rm -it --ipc host --gpus all --ulimit memlock=-1 --ulimit stack=67108864 -p 8000:8000 nvcr.io/nvidia/tensorrt-llm/release:x.y.z

{{container_tag_admonition}}

View the source commit recorded in the image by running the following inside the container:

printenv TRT_LLM_GIT_COMMIT

Sanity check the installation by running the following inside the container:

python3 -c "import tensorrt_llm"

(linux)=

Option 2: Install on Linux via pip

Note: The TensorRT LLM wheel on PyPI is built with the public PyTorch package. This version may be incompatible with the NVIDIA NGC PyTorch container, which uses a different PyTorch build. If you are using the NGC PyTorch container, follow Install in an NGC PyTorch container to install a wheel built specifically for that container.

Tested on Ubuntu 24.04.

Install prerequisites

Before the pre-built Python wheel can be installed via pip, a few prerequisites must be put into place:

Install CUDA Toolkit 13.4 following the CUDA Installation Guide for Linux and make sure CUDA_HOME environment variable is properly set.

The cuda-compat-13-4 package may be required depending on your system's NVIDIA GPU driver version. For additional information, refer to the CUDA Forward Compatibility.

# By default, the PyTorch CUDA 13.0 package is installed. Install the PyTorch CUDA 13.2 package instead to better align with the CUDA version used to build the TensorRT LLM wheels.
pip3 install torch==2.14.0 torchvision --index-url https://download.pytorch.org/whl/cu132

sudo apt-get -y install libopenmpi-dev

# Optional step: Only required for disagg-serving
sudo apt-get -y install libzmq3-dev
Instead of manually installing the prerequisites as described
above, it is also possible to use the pre-built [TensorRT LLM Develop container
image hosted on NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/devel)
(see [here](containers) for information on container tags).

Install pre-built TensorRT LLM wheel

Once all prerequisites are in place, TensorRT LLM can be installed as follows:

Before installing the latest version, uninstall any previous CUTLASS DSL installation as described in the CUTLASS DSL installation guide:

pip3 uninstall nvidia-cutlass-dsl nvidia-cutlass-dsl-libs-base \
    nvidia-cutlass-dsl-libs-core nvidia-cutlass-dsl-libs-cu12 nvidia-cutlass-dsl-libs-cu13
pip3 install --ignore-installed pip setuptools wheel && pip3 install tensorrt_llm

Install a nightly release

Nightly wheels are published to a dedicated package index. Query that index to list compatible versions, and use --extra-index-url when installing to keep PyPI available for dependency resolution.

List the nightly releases compatible with the current environment:

pip3 index versions tensorrt_llm --pre \
    --index-url https://pypi.nvidia.com/trtllm_nightly/

Choose a version from the list and install it explicitly:

pip3 install "tensorrt_llm==<version-from-the-list>" \
    --extra-index-url https://pypi.nvidia.com/trtllm_nightly/

If needed, browse the raw nightly wheel index to inspect builds for other Python versions or platforms.

Install in an NGC PyTorch container

Query the appropriate index and replace <version-from-the-list> with the desired version.

Install a regular release

Download the wheel from that index before installing it to avoid selecting a PyPI wheel with the same version.

pip3 index versions tensorrt_llm --pre \
    --index-url https://pypi.nvidia.com/trtllm_ngcpytorch/

TRTLLM_WHEEL_DIR=$(mktemp -d)
pip3 download --no-deps "tensorrt_llm==<version-from-the-list>" \
    --index-url https://pypi.nvidia.com/trtllm_ngcpytorch/ \
    --dest "$TRTLLM_WHEEL_DIR"
pip3 install "$TRTLLM_WHEEL_DIR"/tensorrt_llm-*.whl

Install a nightly release

pip3 index versions tensorrt_llm --pre \
    --index-url https://pypi.nvidia.com/trtllm_ngcpytorch_nightly/

pip3 install "tensorrt_llm==<version-from-the-list>" \
    --extra-index-url https://pypi.nvidia.com/trtllm_ngcpytorch_nightly/

Sanity check

View the source commit recorded in the wheel metadata:

pip3 show --verbose tensorrt_llm | grep "Source Commit"

Run a quick start example on supported environment:

    :language: python
    :linenos:

Known limitations

There are some known limitations when you pip install the pre-built TensorRT LLM wheel package.

  1. MPI in the Slurm environment

    If you encounter an error while running TensorRT LLM in a Slurm-managed cluster, you need to reconfigure the MPI installation to work with Slurm. The setup method depends on your Slurm configuration, please check with your admin. This is not TensorRT LLM specific, but rather a general MPI+Slurm issue.

    The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM support. This usually happens
when OMPI was not configured --with-slurm and we weren't able
to discover a SLURM installation in the usual places.

  2. Prevent Open MPI 5 from rejecting a long hostname when spawning workers

    Where the MPI implementation is Open MPI 5 -- which the release container now provides, since its base image ships it -- a singleton MPI_Comm_spawn fails when the hostname is too long, despite being a perfectly valid hostname. This is an upstream Open MPI issue; the fix (open-mpi/ompi#14398) has been merged and will ship in a future Open MPI release. TensorRT LLM spawns one such worker per MpiPoolSession, so on a host with a long name, a Kubernetes pod for instance, startup fails with:

    mpi4py.MPI.Exception: MPI_ERR_UNKNOWN: unknown error

    Until that fix is available, export a shorter PMIX_HOSTNAME in the environment TensorRT LLM starts from. PMIx only uses the value to identify the node during the handshake, so it does not have to resolve; any short string will do:

    export PMIX_HOSTNAME=trtllm-node

  3. Prevent pip from replacing existing PyTorch installation

    On certain systems, particularly Ubuntu 22.04, users installing TensorRT LLM would find that their existing, CUDA 13.x compatible PyTorch installation (e.g., torch==2.9.0+cu130) was being uninstalled by pip. It was then replaced by a CUDA 12.y version (torch==2.9.0), causing the TensorRT LLM installation to be unusable and leading to runtime errors.

    The solution is to create a pip constraints file, locking torch to the currently installed version. Here is an example of how this can be done manually:

    CURRENT_TORCH_VERSION=$(python3 -c "import torch; print(torch.__version__)")
echo "torch==$CURRENT_TORCH_VERSION" > /tmp/torch-constraint.txt
pip3 install --ignore-installed pip setuptools wheel && pip3 install tensorrt_llm -c /tmp/torch-constraint.txt

Option 3: Build from Source

For developers who wish to modify, customize, or contribute to TensorRT LLM, see Build from Source.