native-onnx

Rust ONNX inference using your system's ONNX Runtime, CUDA, and TensorRT. Load a model, supply typed tensors, and choose a required backend or ordered fallback. No Python runtime or automatic native-library downloads.

Public API guide · OnnxSession methods

Why use it?

• Less repeated work: reuse input/output buffers; GPU paths reuse pinned host and device storage. Eligible fixed-shape models can use CUDA graph replay.
• Faster startup after compilation: compile a TensorRT model once, then load its embedded engine on later launches.
• Explicit execution: require CUDA/TensorRT or allow fallback to CPU, with recorded failure reasons and checked tensor names, types, shapes, and lengths.

Compute speed comes from ONNX Runtime and its native providers. This crate reduces buffer setup and allocation work; it does not promise a universal speedup over ort, C++, or Python. Benchmark your model and hardware.

Install the native libraries first

Requires Rust 1.88+ and a matching native runtime for your process architecture. You must install these system libraries yourself:

Execution	Required installation
CPU	ONNX Runtime 1.27+ shared library (C API 27); no NVIDIA libraries needed
CUDA	ONNX Runtime 1.27+ with CUDA EP; standard GPU builds require CUDA 13.x (13.0+), cuDNN 9.x for CUDA 13, and a compatible driver (at least R580; the cuDNN/TensorRT build may require a newer driver)
TensorRT	The GPU stack above plus the TensorRT version required by your ORT build. The validated baseline is TensorRT 10.15.1 (ABI 10); a newer incompatible major version will not work
CoreML	macOS 12+ and ONNX Runtime 1.27+ with CoreML EP

GPU dependency versions must match the installed ORT build, not merely exceed a number. Custom CUDA 12 builds need their matching CUDA/cuDNN/TensorRT libraries. Release tests passed on Linux/RTX 3060 with observed library versions ORT 1.29.1, CUDA runtime 13.1, cuDNN 9.26, and TensorRT 10.15.1. These are test results, not universal dependency minimums. macOS and Windows still need native validation. Check the cuDNN support matrix for the exact driver requirements of the package you install. See ORT installation, CUDA requirements, and TensorRT compatibility.

No runtime path is required in Rust. ONNX Runtime, CUDA, and TensorRT are loaded from your system. If their libraries are already discoverable by the platform loader, no environment configuration is needed. For an installation in custom directories, set the search paths before launching your application:

# Replace these directories with your actual installation paths.
export LD_LIBRARY_PATH=/opt/onnxruntime/lib:/usr/local/cuda/lib64:/opt/cudnn/lib:/opt/tensorrt/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
# Optional: select a particular ONNX Runtime library instead of loader discovery.
export ORT_DYLIB_PATH=/opt/onnxruntime/lib/libonnxruntime.so

On Windows, add library folders to PATH; on macOS, use DYLD_LIBRARY_PATH for custom directories. The loader looks for onnxruntime.dll, libonnxruntime.dylib, or libonnxruntime.so. The first ORT initialization selects one library for the whole process, so there is no per-session runtime_path option. Missing/incompatible libraries return installation guidance; print errors with eprintln!("{error:#}") to include the native loader's cause.

Basic example

[dependencies]
native-onnx = "0.2.0"

For a model with an FP32 input named x, shape [1, 16]:

use native_onnx::{OnnxOptions, OnnxSession, Result, TensorView};

fn main() -> Result<()> {
    let mut model = OnnxSession::load("model.onnx", OnnxOptions::cpu())?;
    let values = [1.0_f32; 16];
    let input = TensorView::f32("x", &[1, 16], &values);

    for output in model.inference(&[input])? {
        println!("{}: {:?}", output.name, output.shape);
    }
    Ok(())
}

Replace the path, name, shape, and values for your model. model.info() exposes its specifications. Input storage is contiguous and row-major; preprocessing and postprocessing belong in your application. Each OnnxSession owns one active ONNX Runtime session plus its provider-fallback state and reusable I/O buffers.

Select a GPU and compile once

use native_onnx::{
    Backend, BackendSelection, CompileOptions, CompileTarget, OnnxOptions,
    OnnxSession, TensorRtOptions, TensorView,
};

let config = TensorRtOptions {
    fp16: true,
    ..TensorRtOptions::default()
};
let compile = CompileOptions::new(CompileTarget::TensorRt(config.clone()));
let values = [1.0_f32; 16];
let input = TensorView::f32("x", &[1, 16], &values);

// Run once. The destination must not already exist.
OnnxSession::compile("model.onnx", "model.ctx.onnx", compile, &[input])?;

// Later launches start here; the original model is not needed.
let options = OnnxOptions {
    backend: BackendSelection::Require(Backend::TensorRt),
    tensorrt: Some(config),
    ..OnnxOptions::default()
};
let mut model = OnnxSession::load("model.ctx.onnx", options)?;
let outputs = model.inference(&[input])?;
# Ok::<(), native_onnx::Error>(())

Compilation requires an explicit target and its settings: CompileTarget::TensorRt(config), CompileTarget::Cuda(config), or CompileTarget::Cpu. CompileOptions::new(target) also exposes threading and dimension overrides. It has no default target or fallback; the artifact is validated with your inputs before saving. The returned CompileReport identifies the backend and saved format.

For loading/inference, use Require(Backend::Cuda) for CUDA. Use Auto(vec![Backend::TensorRt, Backend::Cuda, Backend::Cpu]) to permit fallback. backend() reports the primary provider; in automatic mode some nodes can run on CPU. fallback_events() reports whole-session failures. Require rejects CPU node fallback.

TensorRT export currently requires one partition. CPU/CUDA compilation produces an optimized ONNX graph. Compiled contexts require their native provider; they cannot fall back to a CPU engine. Recompile after changing the model, GPU, or incompatible runtime/provider settings. Raw TensorRT .engine files are not accepted.

Reuse buffers and configure execution

API / option	Purpose
inference(inputs)	Borrow inputs and return owned CPU outputs
inference_into(inputs, outputs)	Fill caller-owned CPU outputs, matched by name
input_mut(name), run(), output(name)	Reuse fixed-shape buffers; inputs start at zero
info()	Input/output names, element types, and resolved shapes
is_prepared()	Whether persistent fixed-shape I/O storage is available
cuda_graph_enabled()	Whether the active session is configured for graph capture/replay
backend(), fallback_events()	Primary provider and chronological whole-session failures
end_profiling()	Finish active-session profiling and return its file path
CudaOptions, TensorRtOptions	Device, precision, CUDA graphs, and TensorRT profiles
dimension_overrides	Resolve named dynamic dimensions
parallel_execution, inter_threads	Enable/configure inter-operator CPU parallelism

CUDA graphs default to true for CUDA and TensorRT. Capture requires fixed, positive I/O shapes, sequential execution, and BackendSelection::Require or a compiled EPContext. Auto placement stays graph-free to preserve CPU partitioning and fallback. Check model.cuda_graph_enabled(); set cuda_graph: false to opt out. TensorRT defaults to FP32 with TF32 allowed; unset NVIDIA_TF32_OVERRIDE, or set it to 0 with tf32: false. Dynamic or unknown-rank I/O uses ordinary inference. Dense numeric and boolean tensors are supported; strings, sequences, and sparse I/O are not.

Public inputs and outputs use contiguous, row-major CPU storage, including on GPU backends. Tensor/TensorBuffer own data; TensorView/TensorData borrow it; TensorViewMut/TensorDataMut borrow it for mutation. FP32, FP64, FP16, BF16, signed/unsigned 8/16/32/64-bit integers, and Boolean tensors are supported. Use the appropriate TensorData variant for non-FP32 input; no implicit conversion occurs.

TensorSpec.shape uses None for unknown rank, Some(vec![]) for a scalar, and None inside the dimension vector for a dynamic size. Supply every input exactly once; names, dtypes, shapes, and element counts are checked. Owned inference outputs remain valid after the next run. Prepared output borrows end before another mutable session operation; copy with to_owned() to retain them.

use native_onnx::{OnnxOptions, OnnxSession};

let mut model = OnnxSession::load("model.onnx", OnnxOptions::cpu())?;
if model.is_prepared() {
    model.input_mut("x")?.as_f32_mut()?.fill(1.0);
    model.run()?;
    println!("{:?}", model.output("y")?.as_f32()?);
}
# Ok::<(), native_onnx::Error>(())

Input buffers start at zero and retain values until changed. Borrowing an input for mutation invalidates previous outputs. run() must succeed before output(). inference_into() avoids creating caller-visible output allocations on the prepared path; dynamic execution first creates owned results and copies into your buffers. Neither path promises zero internal allocations or zero CPU/GPU transfers.

Common defaults: one intra-operator thread, one inter-operator thread, sequential execution, no dimension overrides, and profiling off. intra_threads: 0 lets ORT choose; inter_threads only applies with parallel_execution: true. Set profiling: Some("profiles/run".into()) before loading to collect an ORT trace (create the parent directory first). Provider-specific None options use that provider's defaults. Options for unused providers are ignored.

TensorRT defaults to device 0, a 4 GiB workspace on 64-bit targets, builder level 3, automatic auxiliary streams, and sparsity allowed. FP16/TF32 affect internal arithmetic, not I/O types. Dynamic shape profiles require all three of min_shapes, opt_shapes, max_shapes, using ORT syntax such as images:1x3x640x640. All public fields and error contracts are documented in the API reference.

Independent inference threads

Calls are synchronous. OnnxSession is neither Send nor Sync: construct, use, and drop one inside each worker. Multiple CUDA/TensorRT sessions can enable CUDA graphs on the same GPU. Each session owns a nonblocking stream used for transfers and inference; a mixed TensorRT/CUDA session shares its stream internally.

The engine coordinates native setup, first-run warm-up/capture, and teardown. Ordinary inference and completed graph replays share access and may run concurrently. There is no lifetime reservation for a GPU. Initialization includes TensorRT plugin registration, so callers do not need their own initialization mutex.

This coordination covers this crate's sessions in the current process. External CUDA/ORT users must avoid incompatible work during capture, or disable graphs. Concurrent streams do not guarantee a speedup on a saturated GPU. Native provider crashes cannot be recovered as Rust errors.

use native_onnx::{Backend, BackendSelection, OnnxOptions, OnnxSession, Result, TensorView};

let workers: Vec<_> = ["first.onnx", "second.onnx"].into_iter().map(|path| {
    std::thread::spawn(move || -> Result<()> {
        let mut model = OnnxSession::load(path, OnnxOptions {
            backend: BackendSelection::Require(Backend::TensorRt),
            ..OnnxOptions::default()
        })?;
        let outputs = model.inference(&[TensorView::f32("x", &[1, 16], &[1.; 16])])?;
        println!("{} outputs", outputs.len());
        Ok(())
    })
}).collect();
for worker in workers {
    worker.join().expect("worker panicked")?;
}
# Ok::<(), native_onnx::Error>(())

Additional providers use Backend::Custom; CUDA/TensorRT dispatches must instead use their built-in variants and typed options. Enable the openvino feature for OpenVINO configuration and install its matching native provider separately.

The pinned ort rc.13 dependency has an exit-time CUDA teardown bug. This crate retains one global environment until process exit as a workaround; model sessions and buffers are still released when dropped.

Development

With native libraries available through the system loader or ORT_DYLIB_PATH:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo publish --dry-run
# On a GPU host with the matching native libraries:
# The YOLO threading tests need the original model and a TensorRT-compatible artifact:
# export NATIVE_ONNX_YOLO_MODEL=/path/to/yolo11m.trt-fp16.onnx
# export NATIVE_ONNX_CUDA_YOLO_MODEL=/path/to/yolo11m.onnx
cargo test --release --all-features --tests -- --ignored --test-threads=1

The crate archive includes only engine source, build script, README, license, and Cargo metadata. Tests stay in Git; local examples, experiments, and model assets are ignored. Local development examples use their own ignored examples/Cargo.toml manifest; run SAM3 with cargo run --release --manifest-path examples/Cargo.toml --bin sam3.

Changes in 0.2

OnnxRuntime → OnnxSession; Compilation → CompileReport. Old aliases and per-session runtime_path fields are removed. CUDA/TensorRT graphs now default on for eligible strict sessions, each session owns its stream, and native setup is serialized inside the engine for independent inference workers.

Licensed under Apache-2.0. See LICENSE-APACHE for the full text.