Add RHEL / manylinux build tutorial by nv-rinig · Pull Request #157 · triton-inference-server/tutorials

nv-rinig · 2026-07-17T18:20:09Z

Summary

Adds a step-by-step tutorial under Build_Guide/RHEL_Manylinux/ for building Triton Inference Server on RHEL 8 / manylinux from 100% public sources, reproducing the release's build.py --target-platform=rhel artifacts. Also adds a top-level Build Guide index entry.

The rhel target is experimental / not officially supported; the tutorial is flagged as a community example throughout.

What the tutorial covers

Reconstructing the CUDA/cuDNN/TensorRT base image from nvidia/cuda:*-cudnn-devel-rockylinux8
Building the onnxruntime, pytorch, and python backends
Verifying end-to-end: server loads each backend and returns correct inference; wheels are verifiably manylinux (auditwheel show)

Files

Build_Guide/RHEL_Manylinux/README.md — the tutorial
Build_Guide/RHEL_Manylinux/Dockerfile.base.rhel — reconstructed public base image
Build_Guide/RHEL_Manylinux/Dockerfile.pytorch.rhel — public prebuilt-PyTorch image (build.py --image=pytorch input)
Build_Guide/RHEL_Manylinux/Dockerfile.pytorch-runtime.rhel — small completion layer so the PyTorch backend can serve
README.md — top-level Build Guide index entry

Test plan

Follow the tutorial end-to-end on a Linux x86-64 host with Docker; server serves add_py, add_onnx, and add_torch returning [11, 22, 33, 44].
auditwheel show build/install/python/tritonserver-*.whl reports manylinux_2_XX_x86_64.
docker run tritonserver:latest ldd --version reports glibc 2.28 (EL8).

mc-nv · 2026-07-21T16:44:18Z

+FROM nvidia/cuda:${CUDA_IMAGE_TAG}
+
+# Empty = newest TensorRT for the base CUDA; pin to the release, e.g. 10.16.1.11-1.cuda13.2 (26.05).
+ARG TENSORRT_VERSION=


I believe it would be beneficial to have a default value, unless it's not added intentionally.

empty means newest, also i think this also depends on the release version the user specifies so i'd rather avoid a default value

I found logic later don't you want to make it consistent and keep input next to it's processing directive?

i mean we can move this down but it's purely cosmetic right, and i've always put ARGs at the top. i feel the tensorrt installation should be in the layer its sitting in

Prefer to follow convention of Dockerfile where default values declared as manifest and later referred as part of the layer declaration.

mc-nv · 2026-07-21T17:28:39Z

+RUN printf '#!/bin/sh\nexec /usr/bin/python3.12 -m pip "$@"\n' > /usr/local/bin/pip && \
+    chmod +x /usr/local/bin/pip


Questionable installation of the python, can we use virtual environment instead of mocking system defautls?

this isn't installing python, it's just allowing us to use the python3.12 pip instead of the default 3.6 the image ships with. using a virtualenv wouldn't be discoverable on the path, and i'm not too sure how else we should handle this. it's purely used for the wheels

I would suggest to use following declaration instead:

RUN dnf install python3.12; python3.12 -m venv /opt/venv-triton ENV PATH=/opt/venv-triton/bin:$PATH

by doing this, we'd have this inside the generated container after build.py - i would personally prefer to reduce blast radius bc i haven't tested what that effect could be

If you decide to stick with system binary then probably have something like this is more straight forward considering that python3.12-pip package was installed with previous instructions.

RUN ln -s /usr/bin/pip3 /usr/local/bin/pip

Walks through building Triton Inference Server on RHEL 8 / manylinux from 100% public sources, reproducing the `build.py --target-platform=rhel` path that ships the release's manylinux artifacts. Covers: - Reconstructing the CUDA/cuDNN/TensorRT base image from `nvidia/cuda:*-rockylinux8` - Building the onnxruntime, pytorch, and python backends - Verifying the resulting server serves correct inference for each backend and that the wheels are verifiably manylinux (`auditwheel show`) The `build.py rhel` target is experimental and not an officially supported build path.

greptile-apps · 2026-07-22T23:28:42Z

Greptile Summary

This PR adds a community tutorial under Build_Guide/RHEL_Manylinux/ for building Triton Inference Server on RHEL 8 / manylinux entirely from public sources, plus a top-level index entry in README.md. Previously-flagged blocking issues (torch version mismatch between the two PyTorch Dockerfiles, and the hardcoded 3.12.3 pyenv patch path) are addressed: both Dockerfiles now default TORCH_VERSION=2.13.0, and the runtime Dockerfile uses a ln -sfn … | head -1 symlink to resolve the actual provisioned patch dynamically.

Dockerfile.base.rhel correctly uses ${LD_LIBRARY_PATH:+:...} to avoid the previously-flagged trailing-colon issue and isolates auditwheel/patchelf in a venv so the pyenv Python reshuffle during build.py can't orphan them.
Dockerfile.pytorch.rhel backfills stub MKL .so files and the LICENSE/codegen paths that build.py's hardcoded docker cp steps require but a public torch wheel doesn't ship.
Dockerfile.pytorch-runtime.rhel wires the NCCL, cuSPARSELt, and nvshmem runtime directories into ldconfig; the nvshmem entry warrants a quick spot-check (see inline comment) to confirm nvidia-nvshmem-cu12 is present as a torch 2.13.0 pip dependency.

Confidence Score: 5/5

Safe to merge; this is a documentation and tutorial-only addition with no changes to existing code paths, and the two previously-reported build-breaking defects are now fixed.

All five changed files are new tutorial content (Dockerfiles + Markdown). The two defects flagged in prior review rounds — the torch version mismatch across the two PyTorch Dockerfiles and the hardcoded pyenv Python patch path — are addressed in this revision. The remaining open item (the nvshmem ldconfig path) is a speculative runtime concern that is safe to follow up on without blocking the merge.

Dockerfile.pytorch-runtime.rhel deserves a quick check that pip show nvidia-nvshmem-cu12 succeeds in the installed torch environment, so the ldconfig nvshmem path it writes is actually backed by real libraries at server load time.

Important Files Changed

Filename	Overview
Build_Guide/RHEL_Manylinux/Dockerfile.base.rhel	Reconstructs the EL8 CUDA/TensorRT base image from public sources; correctly uses ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} to avoid the previously-flagged trailing-colon issue, and isolates auditwheel in a venv to survive the pyenv Python reshuffle
Build_Guide/RHEL_Manylinux/Dockerfile.pytorch.rhel	Builds the manylinux libtorch source image with stub MKL/.so backfills; TORCH_VERSION default now matches Dockerfile.pytorch-runtime.rhel (2.13.0), but the ls -d symlink creation still lacks head -1 (previously flagged)
Build_Guide/RHEL_Manylinux/Dockerfile.pytorch-runtime.rhel	Completes the serving image by adding torch into the pyenv Python; previously-flagged hardcoded patch version and version-mismatch issues are addressed, but ldconfig writes an nvshmem path that may not exist if nvidia-nvshmem is not a torch pip dependency
Build_Guide/RHEL_Manylinux/README.md	Well-structured tutorial with clear prerequisites, version-update guidance, and end-to-end verification steps; prominently warns this is a community/experimental path
README.md	Adds Build Guide index entry and fixes exhaustive typo; minimal, safe change

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Host: Linux x86-64 + Docker] --> B

    subgraph Step1["Step 1: Build Base Image"]
        B[Dockerfile.base.rhel\nnvidia/cuda:*-rockylinux8\n+ EPEL/PowerTools\n+ gcc-toolset-13\n+ TensorRT\n+ auditwheel venv] --> C[triton-manylinux-base:example]
    end

    subgraph Step2["Step 2 (optional): PyTorch Image"]
        D[Dockerfile.pytorch.rhel\nquay.io/pypa/manylinux_2_28_x86_64\n+ torch wheel\n+ stub MKL/.so files] --> E[triton-manylinux-pytorch:example]
        E --> F[Push to localhost:5000 registry]
    end

    subgraph Step3["Step 3: build.py"]
        G[build.py\n--target-platform=rhel\n--image=base,...\n--image=pytorch,...\n--backend=onnxruntime/pytorch/python]
        C --> G
        F --> G
        G --> H[tritonserver Docker image\n+ build/install/ artifacts]
    end

    subgraph Step4["Step 4: Completion + Verify"]
        I[Dockerfile.pytorch-runtime.rhel\nFROM tritonserver\n+ symlink pyenv 3.12\n+ pip install torch\n+ ldconfig nvshmem/nccl/cusparselt]
        H --> I
        I --> J[tritonserver-pytorch:example]
        H --> K[Extract manylinux .whl\nfrom tritonserver_builder]
        K --> L[auditwheel show\n→ manylinux_2_27_x86_64]
        J --> M[GPU inference:\nadd_torch returns 11,22,33,44]
        H --> N[CPU inference:\nadd_py + add_onnx return 11,22,33,44]
    end

    B --> G
    Step1 --> Step2
    Step2 --> Step3
    Step3 --> Step4

_{Reviews (8): Last reviewed commit: "Address remaining Dockerfile.base.rhel r..." | Re-trigger Greptile}

Softens the terse "public torch wheel ships no torchvision" note into a clearer statement of scope: this example intentionally builds without torchvision, and wiring it up from public sources is not something the tutorial covers or verifies.

- Dockerfile.base.rhel: fix LD_LIBRARY_PATH trailing-colon that would let the dynamic loader treat $CWD as a library search path if the base image had LD_LIBRARY_PATH unset. Uses greptile's suggested ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} idiom -- only adds the separator when the base value is non-empty. Also restore the accidentally-removed `ARG TENSORRT_VERSION=` declaration so --build-arg TENSORRT_VERSION=... continues to be honored. - Dockerfile.pytorch.rhel: soften the file-header comment about torchvision -- previously implied the public wheel doesn't ship torchvision at all, which isn't accurate. Match the README's framing that this tutorial simply doesn't cover the torchvision path and it's untested here. - README.md: guard the manylinux wheel extraction's `rm -f build/install/python/*-linux_x86_64.whl` with a `find ... | grep -q .` check on the just-copied manylinux wheels. Without the guard, an empty extract (e.g. build.py's repair step didn't fire) would silently delete the un-repaired originals and leave the user with no wheels at all.

…lout Closes the greptile findings on the two pytorch Dockerfiles + adds a top-of-README callout so a reader targeting a non-26.05 Triton release knows every version pin to update and where to look them up. - Dockerfile.pytorch.rhel: change `ARG TORCH_VERSION=` (empty default, pip resolved "latest at cu132" at build time) to `ARG TORCH_VERSION= 2.13.0`, matching Dockerfile.pytorch-runtime.rhel's default. A user following the README's Step 2 without passing --build-arg TORCH_VERSION now gets the same torch version in both images by default. In-sync comment added on each side referencing the other. - Dockerfile.pytorch-runtime.rhel: stop hardcoding the Python 3.12 patch in downstream paths. Add a symlink step that points /opt/pyenv_build/versions/3.12 at whichever 3.12.<patch> build.py actually provisioned; the pip install and ld.so.conf.d steps below use the 3.12 alias. A future Triton release that provisions 3.12.4 (or later) works without editing this Dockerfile. - README.md: add a top-of-file `> [!IMPORTANT]` callout listing every version pin a reader must update to target a different Triton release, and linking NVIDIA's Framework Support Matrix and Triton release notes as the canonical sources of truth. Explicitly calls out that TORCH_VERSION lives in both Dockerfiles and mismatches ABI-break at server load rather than build time.

When TORCH_VERSION was empty-defaulted, the ${:+} guard was needed so pip wouldn't see an invalid `torch==` requirement. Now that TORCH_VERSION defaults to 2.13.0, the guard is dead code that also lets `--build-arg TORCH_VERSION=` silently install latest torch in the wheel image -- while Dockerfile.pytorch-runtime.rhel's plain `torch==${TORCH_VERSION}` errors on the same empty override. Fixes the wheel/runtime asymmetry greptile flagged: both Dockerfiles now use `"torch==${TORCH_VERSION}"` and both error identically on an empty pin. The tutorial's compatibility-matrix callout in the README is the intended path for users targeting different releases.

Closes the two open threads from PR #157 review: - Move `ARG TENSORRT_VERSION=` from the top of the Dockerfile down to immediately before the TensorRT RUN block that consumes it, matching the Dockerfile convention of colocating a build-arg with the layer that reads it. - Simplify the /usr/local/bin/pip shim from a heredoc-generated wrapper script to a symlink at /usr/bin/pip3.12 (which the python3.12-pip package installed earlier already provides). Reviewer's suggested `pip3` would resolve to python3.6 via Rocky 8's alternatives; pip3.12 is the explicit binary from the installed package and guarantees the target interpreter.

nv-rinig force-pushed the feat/rhel-manylinux-tutorial branch from 75e6cbf to ca5c042 Compare July 17, 2026 18:26

nv-rinig changed the title ~~Add RHEL / manylinux build tutorial (community example)~~ Add RHEL / manylinux build tutorial Jul 17, 2026

nv-rinig force-pushed the feat/rhel-manylinux-tutorial branch 2 times, most recently from e8b2469 to d2c5331 Compare July 17, 2026 19:34

mc-nv reviewed Jul 21, 2026

View reviewed changes

Comment thread Build_Guide/RHEL_Manylinux/Dockerfile.pytorch.rhel

nv-rinig force-pushed the feat/rhel-manylinux-tutorial branch 3 times, most recently from 5ad7cac to bac31e5 Compare July 21, 2026 19:23

nv-rinig force-pushed the feat/rhel-manylinux-tutorial branch from bac31e5 to e22412e Compare July 21, 2026 19:47

greptile-apps Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread Build_Guide/RHEL_Manylinux/Dockerfile.base.rhel

Comment thread Build_Guide/RHEL_Manylinux/Dockerfile.pytorch.rhel Outdated

nv-rinig force-pushed the feat/rhel-manylinux-tutorial branch from f86b846 to 71f32c1 Compare July 23, 2026 17:14

nv-rinig added 4 commits July 23, 2026 10:54

		RUN printf '#!/bin/sh\nexec /usr/bin/python3.12 -m pip "$@"\n' > /usr/local/bin/pip && \
		chmod +x /usr/local/bin/pip

Uh oh!

Conversation

nv-rinig commented Jul 17, 2026

Summary

Test plan

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nv-rinig Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nv-rinig Jul 21, 2026 •

edited

Loading

greptile-apps Bot commented Jul 22, 2026 •

edited

Loading