diff --git a/.gitignore b/.gitignore index b3c22c2..4bbf412 100644 --- a/.gitignore +++ b/.gitignore @@ -57,3 +57,8 @@ bench/cache/ # graphify local knowledge-graph artifact (per-developer, regenerated via `graphify update .`) graphify-out/ + +# TensorRT runtime artifacts (from --verify-runtime / trtexec smoke runs) +*.engine +*.plan +*.trt diff --git a/CHANGELOG.md b/CHANGELOG.md index c1af6f2..0cdae7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,117 @@ All notable changes to this project are documented here. Format loosely follows [Keep a Changelog](https://keepachangelog.com). -## [Unreleased] +## [1.1.0] - 2026-07-22 + +### Added +- Real-TensorRT smoke validation (2026-07-22): the 7-model corpus ran + against genuine TensorRT 10.3.0 (official NGC `tensorrt:24.08-py3` + container, RTX 4050 Laptop GPU) using the installed wheel — 5 genuine + engine builds (including the `--fix` output and the Reshape-INT64 + regression model), 2 genuine parser failures (SequenceEmpty, custom + domain without plugin), and zero disagreements between + `--verify-runtime` and independent direct `trtexec` runs. New + repo-owned runner: `scripts/real-smoke-container.sh` + + `scripts/real_tensorrt_smoke.py`; evidence in + `REAL_TENSORRT_VALIDATION_REPORT.md` and + `bench/real_tensorrt_smoke_results.json`. Recorded TensorRT-10.3 + trtexec behavior: dynamic models without shape flags are auto-overridden + to 1x1x1x1 (warning), not rejected. +- Release-readiness pass (2026-07-22): a recorded trtexec parser/build + *failure* now demotes an otherwise-`likely` verdict to `unverified` + (runtime evidence against the model is never hidden behind a clean + static prediction); fixers that report changes without modifying the + model are rejected; plugin checker findings without a `rule_id` get a + namespaced `PLUGIN-` fallback; `--fix` before/after identity now + includes the owning graph scope; `bench/score.py --json` emits a + machine-readable summary; `scripts/package-smoke.sh` installs the built + wheel into a fresh venv and exercises the CLI from outside the repo; + `SECURITY_REVIEW.md` documents the reviewed surfaces and trust model. +- **Four-state verdict model.** `AnalysisReport.verdict` is now one of + `blocked` / `unverified` / `likely` / `verified` (`trtcheck.Verdict`). + `unverified` is new: no known blocker, but unresolved conditions remain. + The boolean `conversion_likely` survives as a deprecated compatibility + property (`verdict != blocked`). Exit codes unchanged by default + (`1` on blocked); new `--fail-on unverified` tightens the gate. +- **Stable rule ids + report schema 2.0.** Every finding carries + `rule_id` (e.g. `TRT-OP-UNSUPPORTED`, `TRT-DTYPE-UINT8-INPUT`), + `confidence` (high/medium/low), `verify_required`, `target_trt`, and + `graph_scope`. The registry is documented in `docs/rules.md` and pinned + by a stability test. All 1.x JSON keys are preserved. +- **Honest uncertainty for unknown operators.** Default-domain ops absent + from the support matrix now emit `TRT-OP-UNCLASSIFIED` (info, + aggregated per op type) instead of silently passing; custom-domain ops + emit `TRT-OP-CUSTOM-DOMAIN` unless the domain is declared plugin-backed + via `--plugin-domain` / `AnalyzerConfig.plugin_domains`. +- **Conditional operator support (matrix schema 2.0).** Matrix entries may + carry evidence-backed `conditions` (`attribute_allowed`, + `constant_input_max`, scoped by `applies_to` TRT versions). Violations + emit `TRT-OP-CONDITION`; statically unresolvable conditions emit + `TRT-OP-CONDITION-UNRESOLVED` (unverified). First converted operators, + sourced from the upstream onnx-tensorrt table (retrieved 2026-07-22): + TopK (`sorted=1` required, `K < 3840`) and Resize (nearest/linear only, + restricted `coordinate_transformation_mode`, no antialias). `Clip` + added to the matrix (TRT 10.x supported per the same source; 8.x left + unknown). +- **Optional runtime verification.** `--verify-runtime` runs + `trtexec --onnx=MODEL` (list-args, timeout, captured output; no shell) + and records status/version/command metadata in the report. Only a + successful build upgrades the verdict to `verified`; a missing trtexec, + timeout, or failure leaves the static verdict untouched. Statuses: + success / parser_failure / build_failure / missing_trtexec / timeout. +- **Audited `--fix` pipeline.** `--fix` now analyzes with the selected + `--target-trt`, applies fixers, re-analyzes with the same target, and + reports findings resolved / remaining / introduced (keyed by rule id + + node identity). `--format json` emits a machine-readable fix summary. + Structurally invalid input models are refused. +- Bench harness three-way vocabulary: predictions may be `unverified`, + which is excluded from the blocker confusion matrix and reported as + coverage (never counted as success). New corpus fixtures: + `topk_unsorted`, `custom_domain`, `reshape_int64_shape`. +- `docs/rules.md` (rule registry) and + `docs/design/analysis-verdicts-and-fix-safety.md` (invariants, trust + model, transactional fixing). + +### Fixed +- **`Int64ToInt32Fixer` no longer corrupts models whose INT64 initializers + feed schema-required-INT64 inputs** (`Reshape` shape, `Slice` + starts/ends, ...). The old "fits in INT32" rule produced models that + passed the shallow checker but failed strict type inference. The fixer + is now use-aware: it converts only when every use (including nested + subgraph captures) is at an allowlisted INT32-compatible position, and + refuses shadowed names, signature tensors, custom-domain consumers, + overflow, and dead initializers. It also no longer retypes an + initializer that doubles as a graph input (that silently changed the + model's public signature). +- **Fixers are transactional.** Every fixer (built-in or plugin) runs + against an isolated deep-copy candidate that is committed only after it + passes `onnx.checker.check_model(full_check=True)` (basic check for + external-data models). A fixer that crashes mid-mutation, returns + malformed records, mutates without declaring it, or emits an invalid + model cannot affect the output; one failed fixer no longer prevents + later fixers from running. Plugin tracebacks are hidden unless + `TRTCHECK_DEBUG=1`. +- **`DropDropoutFixer` respects training mode.** Opset >= 12 Dropouts are + removed only when `training_mode` is absent or statically false + (initializer or Constant, unambiguous across scopes); true, dynamic, + computed, or ambiguous training modes are left alone. Opset <= 6 + requires `is_test=1`. +- Resize matrix prose corrected against current upstream onnx-tensorrt + docs: cubic mode and antialiasing are *not* supported in TRT 10.x (the + old notes claimed both were added in 10.0). + +### Changed +- Console/HTML headlines use conservative verdict wording ("LIKELY -- + static analysis found no known blocker" instead of "LIKELY TO + CONVERT"; "CONVERSION BLOCKED" instead of "CONVERSION WILL FAIL"), and + both reporters gained a Rule column plus the TRT target in the header. +- `bench/predict.py` runs the full report (dropping `--severity + critical`) because verdicts require the INFO-level uncertainty + findings; it maps schema-2.0 verdicts with a 1.x fallback. +- The exit code is computed from the unfiltered report: `--severity` + affects display only. + +_Also in this release (earlier unreleased entries):_ ### Fixed - The README / case-study `--fix` walkthrough is reproducible again. The diff --git a/CLAUDE.md b/CLAUDE.md index 919b743..9512808 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -349,13 +349,18 @@ trtcheck model.onnx --format json --output report.json trtcheck model.onnx --format html --output report.html # Filter by severity -trtcheck model.onnx --severity critical # Only show blockers +trtcheck model.onnx --severity critical # Only show blockers (display-only; exit code uses the full report) trtcheck model.onnx --severity warning # Critical + warnings # Diff two ONNX files (before/after fix) trtcheck model_v1.onnx model_v2.onnx --diff +# Strict CI gate / plugin domains / runtime verification +trtcheck model.onnx --fail-on unverified +trtcheck model.onnx --plugin-domain com.example +trtcheck model.onnx --verify-runtime # optional; needs trtexec + # Version info trtcheck --version ``` @@ -366,7 +371,7 @@ trtcheck --version ╔══════════════════════════════════════════════════════════════════════╗ ║ trtcheck Report ║ ║ model.onnx → TensorRT 10.3 ║ -║ Status: CONVERSION WILL FAIL (1 critical, 2 warnings) ║ +║ Status: CONVERSION BLOCKED (1 critical, 2 warnings) ║ ╚══════════════════════════════════════════════════════════════════════╝ ┌──────────┬────────┬─────────────┬────────────────────────┬─────────────────────────────┐ diff --git a/OVERNIGHT_REPORT.md b/OVERNIGHT_REPORT.md new file mode 100644 index 0000000..fcf1d73 --- /dev/null +++ b/OVERNIGHT_REPORT.md @@ -0,0 +1,198 @@ +# Overnight hardening report — trtcheck + +Branch: `claude/trtcheck-hardening` (off `main` @ 374fe48). Nothing pushed, +published, tagged, or released. Date: 2026-07-22. + +## 1. Executive summary + +trtcheck is now a release-candidate static pre-flight checker **and safe +repair tool** for ONNX → TensorRT deployment. The three transformation +soundness bugs (INT64 schema corruption, non-transactional fixers, Dropout +training-mode) are fixed with regression proofs; analysis is honest about +what it cannot know (four-state verdict, per-finding confidence, no +silently-ignored operators); diagnostics are CI-stable (rule-id registry, +JSON schema 2.0 that is a strict superset of 1.x); the compatibility matrix +can express evidence-backed conditional support; and an optional, fully +isolated `trtexec` runtime-verification path exists. All 445 tests, strict +mypy, black, and isort pass. + +## 2. Baseline + +Environment: Linux 6.8, repo venv (`.venv`, Python 3.10), onnx 1.21.0, +click 8.4.1, numpy 2.4.6. No TensorRT, no GPU, no Docker. Network access +was available (used only to retrieve the upstream onnx-tensorrt operator +table for evidence). + +Baseline commands and results (clean tree at 374fe48): + +- `./scripts/run-tests.sh -q` → **380 passed, 1 skipped** (benchmark + opt-in), 1.64 s +- `.venv/bin/mypy trtcheck/ --strict` → **Success: no issues in 25 files** +- `.venv/bin/black --check .` → clean; `.venv/bin/isort --check-only .` → clean + +## 3. Major architectural changes + +1. **Verdict lattice** (`trtcheck/types.py`): `Verdict.{BLOCKED, + UNVERIFIED, LIKELY, VERIFIED}` derived from findings; + `conversion_likely` kept as a deprecated property. Invariant: no + operator disappears silently — everything is classified, flagged as + uncertain, or explicitly declared plugin-backed. +2. **Stable diagnostics** (schema 2.0): every `Issue` carries `rule_id`, + `confidence`, `verify_required`, `target_trt`, `graph_scope`. + Rule ids live in `remediation_db.json` + checker constants; the + registry (`docs/rules.md`) is pinned by + `tests/test_verdicts.py::test_rule_id_registry_is_stable`. +3. **Transactional fixer pipeline** (`trtcheck/fixers/run_fixers`): + per-fixer deep-copy candidates, committed only after ONNX validation at + the strongest level the input model itself passes (full → basic → none; + the CLI refuses to fix models that fail even basic). +4. **Conditional capability data** (matrix schema 2.0): per-operator + `conditions` with `applies_to` version scoping and `evidence` + metadata; evaluators for `attribute_allowed` and `constant_input_max`; + unknown kinds and unresolvable conditions fail toward *unverified*, + never toward *pass*. +5. **Isolated runtime verification** (`trtcheck/runtime_verify.py`): + list-args subprocess, timeout, truncated output capture, five distinct + failure states; only a successful build sets `VERIFIED`, and never over + a static `BLOCKED`. +6. **Audited `--fix`**: analyze(target) → transactional fix → validate → + re-analyze(same target) → resolved/remaining/introduced diff by + `(rule_id, node, operator)`; JSON summary via `--format json`. + +## 4. Bugs reproduced and fixed + +| Bug | Reproduction | Fix | +|---|---|---| +| `Int64ToInt32Fixer` corrupted models whose INT64 initializer feeds a schema-required-INT64 input (e.g. `Reshape` shape) | `tests/test_fixers_int64_schema.py::test_blind_conversion_breaks_full_validation` proves the old rewrite passes the shallow checker and fails `full_check=True` | Use-aware conversion: every use (incl. nested subgraph captures) must be at an allowlisted INT32-independent position (Gather/GatherElements/ScatterElements indices, Cast/Shape/Size input); refuses shadowing, signature tensors, custom domains, overflow, dead/empty initializers. No speculative Casts. | +| Fixers could leave partial mutations after a crash; a crashed plugin fixer's edits leaked into `--fix` output | `tests/test_fixers_transactional.py` (`_MutateThenCrash`, `_EmitsInvalidModel`, `_UndeclaredMutation`, malformed returns) | `run_fixers()` transaction; failures recorded as `FixFailure`; later fixers still run; tracebacks only with `TRTCHECK_DEBUG=1` | +| `DropDropoutFixer` removed Dropouts regardless of `training_mode` | `tests/test_fixers_dropout.py::TestDropoutTrainingMode` (true/dynamic/computed/ambiguous kept; absent/static-false removed; opset-6 `is_test`) | Static resolution of opset semantics + `training_mode` through initializers/Constant nodes with scope-ambiguity refusal | +| Unknown default-domain and custom-domain operators silently passed | `tests/test_verdicts.py` | `TRT-OP-UNCLASSIFIED` / `TRT-OP-CUSTOM-DOMAIN` info findings (aggregated per op type), `verify_required=True` → verdict `unverified`; `--plugin-domain` opt-out | +| `--fix` ignored `--target-trt` and never compared before/after | `tests/test_cli_fix.py` | Target-aware audited pipeline (above) | +| Matrix prose for Resize contradicted current upstream docs (claimed cubic + antialias supported in 10.0) | upstream onnx-tensorrt `operators.md`, retrieved 2026-07-22 | Notes corrected; conditions added; drift-checked | +| `mobilenetv2` scored `unverified` — `Clip` missing from the matrix | bench run in this session | `Clip` added: TRT 10.x supported (official docs), 8.x left `unknown` (no evidence claimed) | + +## 5. Files changed (56 files, +3547/−351) + +- **Core types/analysis**: `trtcheck/types.py`, `analyzer.py`, + `remediation.py`, `checkers/operator_support.py` (rewritten), + `checkers/{precision,control_flow}.py` (scope threading) +- **Fixers**: `fixers/__init__.py` (transactional pipeline), + `fixers/int64_to_int32.py` (rewritten), `fixers/drop_dropout.py` +- **CLI/runtime**: `cli.py` (verdicts, `--fail-on`, `--plugin-domain`, + `--verify-runtime`, audited `--fix`), new `runtime_verify.py` +- **Data**: `data/operator_matrix.json` + `data/remediation_db.json` + (schema 2.0), `tools/build_operator_matrix.py` (source of truth updated), + regenerated `docs/operators/*` +- **Bench**: `bench/predict.py`, `bench/score.py`, `bench/manifest.yaml`, + `bench/outcomes.json`, new fixtures via `tests/fixtures/generate_broken.py` +- **Tests**: 6 new suites (`test_fixers_int64_schema`, + `test_fixers_transactional`, `test_verdicts`, `test_conditions`, + `test_cli_fix`, `test_runtime_verify`) + targeted updates +- **Docs**: `usage.md`, `fixers.md`, new `rules.md`, new + `design/analysis-verdicts-and-fix-safety.md`, `design/plugin-sdk.md`, + `README.md`, `SCORECARD.md`, `CHANGELOG.md`, case study, SVG wording, + `CLAUDE.md`, `RELEASE_NOTES_DRAFT.md` + +## 6. Public API / schema changes and migration + +- JSON report: `schema_version: "2.0"`. **Superset of 1.x** — every old + key including `conversion_likely` is still emitted. Migrate consumers + from `conversion_likely` to `verdict`; filter findings on `rule_id`. +- `Issue` constructor: new fields all default → third-party checkers + unchanged. +- `apply_all()` keeps its signature; it is now transactional under the + hood. New `run_fixers()` additionally reports `FixFailure`s. +- Exit codes unchanged by default (1 = blocked). `--severity` no longer + influences the exit code. `--fix` now refuses invalid input models and + declines unsound INT64 conversions it previously performed — + intentional soundness changes, documented in the changelog. + +## 7. Tests and final check results + +Added ~65 tests across the six new suites plus updated existing suites. + +Final results (this session, exact commands): + +- `./scripts/run-tests.sh -q` → **445 passed, 1 skipped** (~1.7 s) +- `.venv/bin/mypy trtcheck/ --strict` → **Success: no issues in 26 files** +- `.venv/bin/black --check .` / `.venv/bin/isort --check-only .` → clean + +## 8. Runtime / TensorRT checks actually performed + +**None.** No TensorRT, `trtexec`, or GPU exists in this environment. The +runtime-verification module is tested exclusively through mocked +subprocess calls; the scorecard states explicitly that ground truth is +documented TRT behavior, not live builds. To verify for real: +`trtcheck model.onnx --verify-runtime` on a machine with TensorRT, or the +`bench/README.md` GPU protocol (`trtexec --onnx=` per manifest +entry, recorded into `outcomes.json` under the `trtexec` key). + +Network evidence used: the upstream onnx-tensorrt operator table +(https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md, +retrieved 2026-07-22) for TopK/Resize/Clip entries. + +## 9. Remaining limitations / deferred work + +- Conditional-support data covers 2 operators (TopK, Resize) by design — + the evaluation infrastructure exists; converting more operators is + data-entry work with the same evidence discipline. +- `graph_scope` is populated where the owning graph is cheaply known + (operator support, initializer precision, control flow); dynamic-shape + and graph-input findings leave it empty (top-level by construction). +- The dtype columns of the upstream table (per-op supported dtypes) are + not yet modeled as conditions. +- Real trtexec leg of the scorecard: hardware-only, procedure documented. +- The GitHub Action still summarizes via `conversion_likely` (works, but + could surface the four-state verdict in the PR comment). + +## 10. Suggested next version and release checklist + +Suggested: **v1.1.0** (additive schema, behavior corrections). See +`RELEASE_NOTES_DRAFT.md` for the draft notes and the pre-release +checklist. Do not release from this branch without the checklist. + +## 11. Local commits (oldest first) + +1. `1d97144` feat: stable rule IDs, four-state verdict model, schema-aware + fixers, transactional fix pipeline +2. `7240911` test: regression suites for schema-aware INT64, transactional + fixing, dropout training-mode, verdicts, conditions, --fix CLI, runtime verify +3. `02b702e` feat(bench): three-way outcome vocabulary with unverified + coverage metrics +4. `c26b051` docs: verdict model, rule registry, fix-safety design doc, + honest scorecard +5. (final) docs/assets/report commit containing this file + +## 12. Five-minute interview explanation + +**Architecture.** trtcheck is a plugin-composable static analyzer: a thin +`Analyzer` walks an ONNX protobuf (including every nested If/Loop/Scan +subgraph) through independent `Checker`s that each return typed `Issue`s; +pure `Reporter`s render the aggregate; `Fixer`s are the only components +allowed to rewrite the graph, and they run inside a transactional pipeline +that validates every candidate before committing it. Checkers never +format, reporters never analyze, fixers never guess. + +**Hardest correctness issue.** The INT64 auto-fixer. TensorRT prefers +INT32, so "downcast every in-range INT64 initializer" looks obviously +right — and passes ONNX's default checker. But ONNX type constraints are +positional: `Reshape`'s shape input *requires* INT64, and elementwise ops +bind one type variable across both operands, so retyping a single operand +breaks the model in ways only strict type inference catches. The fix is a +whole-model use index (subgraphs can capture outer-scope names) plus an +allowlist of input positions whose type variable binds only that input. +Everything else — shadowed names, signature tensors, custom-domain +consumers — is refused. The design lesson: a graph rewrite is only as +safe as your model of every consumer's schema, and the honest fallback is +refusal, not a speculative Cast. + +**Trust model.** Input models are untrusted (CI runs on arbitrary +protobufs): parse errors become domain errors, model-derived text is +sanitized before terminals and HTML, traversal depth is bounded. Plugins +are semi-trusted: a crashed checker becomes a visible "missing coverage" +finding, and a crashed fixer physically cannot corrupt output because it +only ever touched a discarded copy. And trtcheck does not trust *itself*: +static analysis reports four verdicts, where "unverified" is a +first-class answer meaning "I checked and I cannot know" — only a real +TensorRT build, run explicitly and recorded with its environment, is +allowed to claim "verified". diff --git a/PR_BODY_DRAFT.md b/PR_BODY_DRAFT.md new file mode 100644 index 0000000..0d7b66e --- /dev/null +++ b/PR_BODY_DRAFT.md @@ -0,0 +1,49 @@ +# PR draft: honest verdicts, safe fixes, release readiness + +> Draft only — no PR has been opened. Branches: +> `claude/trtcheck-hardening` (5 commits) + `claude/trtcheck-release-readiness` +> (verification & hardening of the hardening), targeting `main`. + +## What this PR does + +1. **Four-state verdicts** (`blocked` / `unverified` / `likely` / + `verified`). Unknown and custom-domain operators no longer pass + silently; a failed real trtexec run can no longer hide behind a clean + static prediction. `conversion_likely` stays as a deprecated alias. +2. **Stable diagnostics, JSON schema 2.0** — per-finding `rule_id`, + `confidence`, `verify_required`, `target_trt`, `graph_scope`; the + registry (docs/rules.md) is pinned by a stability test; every 1.x JSON + key is preserved. +3. **Safe, transactional `--fix`** — per-fixer isolated candidates + validated with strict type/shape inference; use-aware INT64 + conversion (the Reshape shape-input corruption is a pinned regression); + training-mode-aware Dropout removal; before/after findings diff with + the same `--target-trt`. +4. **Conditional operator support** — evidence-backed per-op conditions + (TopK, Resize) sourced from the upstream onnx-tensorrt table. +5. **Optional runtime verification** — `--verify-runtime` (trtexec, + list-args, timeout); only a real successful build yields `verified`. +6. **Evidence discipline** — bench harness with three-way outcomes and + unverified coverage (never counted as success), machine-readable + summary, refreshed honest SCORECARD; package smoke test from a fresh + venv; SECURITY_REVIEW.md. + +## Compatibility + +- Public API additive; JSON 2.0 is a superset of 1.x. +- Exit codes unchanged by default; `--severity` is now display-only. +- Deliberate behavior changes (documented in CHANGELOG): unclassified / + custom ops report `unverified`; `--fix` refuses invalid inputs and + declines previously-unsound INT64 conversions. + +## Test plan + +- [x] 456 tests, 1 opt-in skip (`./scripts/run-tests.sh`) +- [x] `mypy trtcheck/ --strict` clean; black + isort clean +- [x] `python -m build` + `twine check` pass +- [x] `scripts/package-smoke.sh` — wheel install in fresh venv, CLI + analyze / JSON / fix / missing-trtexec paths +- [x] Real `trtexec` smoke — TensorRT 10.3.0 (NGC `24.08-py3` container, + RTX 4050): 7-model corpus, 5 genuine builds, 2 genuine failures, + 0 wrapper/direct disagreements, installed wheel used throughout. + Evidence: `REAL_TENSORRT_VALIDATION_REPORT.md` diff --git a/README.md b/README.md index 1a71941..39e3d0e 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,11 @@ cd trtcheck pip install -e ".[dev]" ``` -Python 3.10+. No platform dependencies beyond `onnx` itself. +Python 3.10–3.13, `onnx >= 1.15, < 2.0`. No platform dependencies beyond +`onnx` itself — analysis needs no TensorRT, no GPU. Modeled TensorRT +targets: 8.0, 8.6, 10.0, 10.3; each operator entry carries its own +evidence level (official documentation, inferred, or unknown — see +[`docs/rules.md`](docs/rules.md) and the operator pages). ## Quick start @@ -44,7 +48,7 @@ $ trtcheck model.onnx ``` ``` -CONVERSION WILL FAIL — 1 critical, 0 warning +CONVERSION BLOCKED — 1 critical, 0 warning CRITICAL input Input Input 'input' has dtype UINT8; TensorRT accepts only FP32, FP16, INT32, or INT8 as graph inputs. @@ -55,8 +59,14 @@ CRITICAL input Input Input 'input' has dtype UINT8; TensorRT Estimated fix time: 15–30 minutes. ``` -The exit code is `1` when conversion is expected to fail and `0` -otherwise, so the same command works unchanged as a CI gate. +Every report carries one of four verdicts — **blocked** (a known critical +incompatibility), **unverified** (no known blocker, but unresolved +conditions: unclassified or custom-domain operators, conditional support +that static analysis cannot settle), **likely** (all static checks passed +— a prediction, not a guarantee), and **verified** (an optional real +`trtexec` build succeeded via `--verify-runtime`). The exit code is `1` +on `blocked` and `0` otherwise (`--fail-on unverified` tightens the CI +gate), so the same command works unchanged as a CI gate. [`docs/case-studies/uint8-input.md`](docs/case-studies/uint8-input.md) walks this exact case end to end, including the `--fix` rewrite that turns it into a passing graph. Verdict accuracy is measured: @@ -85,7 +95,7 @@ fix, before an engine build is attempted. | Checker | Catches | |---|---| -| **operator support** | Ops missing or partial in the target TRT version (e.g. `SequenceEmpty`, `GroupNormalization` on TRT 8.x) | +| **operator support** | Ops missing or partial in the target TRT version (e.g. `SequenceEmpty`, `GroupNormalization` on TRT 8.x); documented conditional-support rules (e.g. TopK `sorted=0`, cubic `Resize`); honest `unverified` findings for operators the matrix does not classify and for custom-domain ops that need a TRT plugin | | **precision** | `UINT8` / `INT64` / `FLOAT64` / `STRING` / `BFLOAT16` graph inputs, `INT64` weights, and `FLOAT64` introduced by a `Cast` or `Constant` anywhere in the graph | | **dynamic shapes** | Two or more symbolic input dims, including dynamic dims encoded as a concrete `-1` | | **control flow** | `Loop` with runtime trip count, nested `Loop`, `If`, `Scan` | @@ -97,15 +107,19 @@ includes a specific remediation. Not "this is bad" — what to change, where. ## What it auto-fixes -Pass `--fix` to apply built-in safe rewrites in place. Use `--dry-run` -to preview them first. +`--fix` runs an audited, **transactional** pipeline: every fixer works on +an isolated candidate copy, the result must pass full ONNX validation +(strict type/shape inference) before it is kept, and the report shows +which findings were resolved, which remain, and whether any were +introduced. A fixer that crashes — including a third-party plugin — cannot +leave a half-rewritten model. Use `--dry-run` to preview. | Fixer | Rewrites | |---|---| | **`uint8_input`** | Promotes a `UINT8` graph input to `FLOAT` and drops the redundant downstream `Cast` | -| **`int64_to_int32`** | Casts `INT64` initializers to `INT32` when every value is in range | +| **`int64_to_int32`** | Casts `INT64` initializers to `INT32` only when every use is at a schema position that accepts INT32 (e.g. `Gather` indices) — never `Reshape`/`Slice` shape inputs, which require INT64 | | **`float64_to_float32`** | Casts `FLOAT64` initializers to `FLOAT32` when no value is NaN, infinite, or out of FP32 range | -| **`drop_dropout`** | Removes `Dropout` nodes and rewires consumers (skips nodes whose `mask` output is used) | +| **`drop_dropout`** | Removes `Dropout` nodes that are provably in inference mode (`training_mode` absent or statically false; mask unused) | | **`upsample_to_resize`** | Rewrites leftover deprecated `Upsample` nodes as `Resize` on opset-13+ graphs (nearest / linear) | ```bash @@ -119,20 +133,31 @@ Refuses to overwrite the input or an existing output unless you pass ## Measured accuracy The [`bench/`](bench/) harness scores trtcheck's verdicts against a -corpus with known conversion outcomes. Latest run, v1.0.0 against the -TRT 10.3 matrix at the CI gate configuration: - -| Corpus | Precision | Recall | Total wall time | -|---|---|---|---| -| 9 models: 3 from the ONNX Model Zoo, 6 bundled fixtures | 1.000 | 1.000 | 2.0 s | - -Nine models is a small corpus and the failure cases are synthetic, so -read this as "the checks do what they claim on known patterns", not as a -field-accuracy estimate. [`SCORECARD.md`](SCORECARD.md) has the -per-model table, the methodology, and the false negative the first run -caught (it became the `loop_runtime_trip_count` critical check). To -grow the corpus, add a model with a known outcome to -[`bench/manifest.yaml`](bench/manifest.yaml) and open a PR. +corpus with known conversion outcomes. Latest run against the TRT 10.3 +matrix: + +| Corpus | Blocker precision | Blocker recall | Unverified coverage | Total wall time | +|---|---|---|---|---| +| 12 models: 3 from the ONNX Model Zoo, 9 bundled fixtures | 1.000 | 1.000 | 0.250 | 2.3 s | + +`unverified` predictions are never counted as successes — they are +reported separately, split by ground truth. Twelve models is a small +corpus and the failure cases are synthetic, so read this as "the checks +do what they claim on known patterns", not as a field-accuracy estimate. +For the scorecard corpus, ground truth is documented TRT behavior, not a +live `trtexec` run. Separately, the runtime-verification integration was +smoke-tested against **real TensorRT 10.3.0** (official NGC container, 7 +representative fixtures: 5 genuine engine builds, 2 genuine parser +failures, full agreement between `--verify-runtime` and direct `trtexec`) +— see [`REAL_TENSORRT_VALIDATION_REPORT.md`](REAL_TENSORRT_VALIDATION_REPORT.md). +That validates the integration path and those cases, not universal model +compatibility. +[`SCORECARD.md`](SCORECARD.md) has the per-model table, the methodology, +and what each run caught (the first run's false negative became the +`loop_runtime_trip_count` critical check; this run exposed a `Clip` +coverage gap in the matrix). To grow the corpus, add a model with a +known outcome to [`bench/manifest.yaml`](bench/manifest.yaml) and open a +PR. ## How it compares @@ -142,7 +167,7 @@ grow the corpus, add a model with a known outcome to | Time to a verdict | seconds | minutes (builds a real engine) | manual inspection | | Fix suggestions | per-finding remediation + `--fix` rewrites | no | no | | CI integration | exit code, JSON, GitHub Action | scriptable, needs a GPU runner | no | -| Verdict strength | predicts the build outcome | proves it | n/a | +| Verdict strength | predicts the build outcome (and says so: four-state verdict with explicit uncertainty; optional `--verify-runtime` runs trtexec when available) | proves it | n/a | Use them together. Polygraphy building an engine is the ground truth; if you have the GPU and the minutes, run it. Netron is for eyeballing @@ -171,11 +196,21 @@ trtcheck model.onnx --severity critical # compare two versions of a model (before / after a fix) trtcheck before.onnx after.onnx --diff -# auto-fix simple issues +# auto-fix simple issues (transactional; reports resolved/remaining findings) trtcheck model.onnx --fix --output model_fixed.onnx + +# strict CI gate: also fail on unresolved conditions +trtcheck model.onnx --fail-on unverified + +# optional: verify with a real TensorRT build (needs trtexec) +trtcheck model.onnx --verify-runtime ``` -Exit code is `1` if conversion is unlikely to succeed, `0` otherwise. +Exit code is `1` on a `blocked` verdict, `0` otherwise; `--fail-on +unverified` also fails on unresolved conditions. Findings carry stable +rule ids (`TRT-OP-UNSUPPORTED`, `TRT-DTYPE-UINT8-INPUT`, ...) for CI +filtering — see [`docs/rules.md`](docs/rules.md) and +[`docs/usage.md`](docs/usage.md). Full CLI reference: `trtcheck --help`. @@ -224,7 +259,7 @@ repo. Full template at | Input | Default | Purpose | |---|---|---| -| `version` | `1.0.0` | trtcheck PyPI version to install | +| `version` | `1.1.0` | trtcheck PyPI version to install | | `target-trt` | `10.3` | `--target-trt` value | | `severity` | `warning` | `--severity` filter | | `fail-on` | `critical` | Exit policy: `critical`, `warning`, or `never` | diff --git a/REAL_TENSORRT_VALIDATION_REPORT.md b/REAL_TENSORRT_VALIDATION_REPORT.md new file mode 100644 index 0000000..6ad4fc7 --- /dev/null +++ b/REAL_TENSORRT_VALIDATION_REPORT.md @@ -0,0 +1,186 @@ +# Real TensorRT validation report + +Date: 2026-07-22. Branch: `claude/trtcheck-ngc-runtime-validation` +(supersedes the BLOCKED attempt recorded on +`claude/trtcheck-real-tensorrt-validation`). + +## Status: **COMPLETE — real runtime smoke passed** + +Real runtime integration was smoke-tested on **TensorRT 10.3.0** using 7 +representative generated/public fixtures inside the official NGC +container. This validates the verification integration and the selected +cases, **not universal model compatibility**. + +## 1. Environment + +| Item | Value | +|---|---| +| Host OS | Zorin OS 17.3 (Ubuntu 22.04 base), Linux 6.8.0-106-generic, x86_64 | +| GPU | NVIDIA GeForce RTX 4050 Laptop GPU (6 GB) | +| NVIDIA driver | 580.126.20 | +| Docker | Engine 29.3.0 (native, default context) | +| NVIDIA Container Toolkit | **not installed** — see GPU-access strategy below | +| Container image | `nvcr.io/nvidia/tensorrt:24.08-py3` | +| Image digest | `sha256:9507e5f248fc61a5b2c985ce6e386ecf2576c3d96112ceb38cf88b240d4ca072` (14.6 GB local) | +| TensorRT (in container) | **10.3.0** (`trtexec` at `/opt/tensorrt/bin/trtexec`, banner `TensorRT v100300`) | +| Container Python | 3.10 | +| trtcheck under test | **installed wheel** `trtcheck-1.0.0-py3-none-any.whl` (never editable) | + +### Why this image + +The repository's support targets are TensorRT 8.0 / 8.6 / 10.0 / 10.3. +Newer NGC images carry TensorRT versions (10.16, 11.0) that are **not** +repository targets, and claiming them would violate the evidence policy. +`24.08-py3` is the official NGC release that ships **exactly TensorRT +10.3**, matching the default target the whole matrix is scored against. + +### GPU-access strategy (no host changes) + +`sudo` on this host requires an interactive password, so the NVIDIA +Container Toolkit could not be installed. Instead of modifying the host, +GPU access uses manual passthrough — strictly less invasive: + +- `--device /dev/nvidia0 /dev/nvidiactl /dev/nvidia-uvm /dev/nvidia-uvm-tools` +- read-only mounts of the driver's user-space libraries only + (`libcuda*`, `libcudadebugger*`, `libnvidia-ml*`, `libnvidia-cfg*`, + `libnvidia-nvvm*`, `libnvidia-ptxjitcompiler*`, `libnvidia-gpucomp*`) + staged into a temp dir, exposed via `LD_LIBRARY_PATH=/nvlibs` +- read-only mount of the host `nvidia-smi` binary (NGC images don't ship it) + +Verified end-to-end before the corpus: `nvidia-smi` sees the RTX 4050 +inside the container and a real engine build PASSED. **Zero host +packages/configuration were installed or changed** (nothing to roll back). +`scripts/real-smoke-container.sh` automates all of this and prefers +`--gpus all` automatically on hosts where the toolkit exists. + +## 2. Corpus and results + +Runner: `scripts/real-smoke-container.sh` → `scripts/real_tensorrt_smoke.py`. +Machine-readable results: [`bench/real_tensorrt_smoke_results.json`](bench/real_tensorrt_smoke_results.json). +Per model: static analysis (target 10.3), `trtcheck --verify-runtime +--trtexec `, and an **independent direct trtexec run**, all with +timeouts and bounded output capture. + +| Model | Static verdict | Wrapper runtime | Direct trtexec | Agree | Expected | +|---|---|---|---|---|---| +| clean_minimal | likely | success → **verified** | build PASSED (~6 s) | yes | yes | +| squeezenet1_1 (public zoo) | likely | success → **verified** | build PASSED | yes | yes | +| sequence_empty | **blocked** | parser_failure → stays blocked | parser FAILED | yes | yes | +| fully_dynamic | unverified | success → verified¹ | build PASSED¹ | yes | yes¹ | +| custom_domain | unverified | parser_failure → stays **unverified** | parser FAILED | yes | yes | +| uint8_input after `--fix` | likely (was blocked) | success → **verified** | build PASSED | yes | yes | +| reshape_int64_shape | likely | success → **verified** | build PASSED | yes | yes | + +**Summary: 7/7 run, 5 genuine engine builds, 2 genuine parser failures, +0 wrapper/direct disagreements, 0 unexpected outcomes.** + +¹ See dynamic-shape analysis below. + +Key invariants confirmed with real execution: + +- static analysis alone never yields `verified`; only the real trtexec + success path set it; +- real parser failures were classified `parser_failure` (not lost, not + misread from incidental log text) and never upgraded the verdict; +- the custom-domain model did **not** become verified without its plugin + — trtexec's parser rejects it and trtcheck stays `unverified`; +- the `--fix` output of the UINT8 fixture — produced by the installed + wheel inside the container — genuinely builds an engine; +- the Reshape INT64 regression model builds as-is, and the fixer's + refusal to convert its shape initializer was re-confirmed in-container; +- trtexec is invoked as an argument list with a timeout; the recorded + commands round-trip exactly. + +## 3. Dynamic-shape / profile analysis + +Observed, version-specific tool behavior on **TensorRT 10.3 trtexec**: a +model whose input is fully dynamic (`[batch, channels, h, w]`) does *not* +fail without shape flags — trtexec warns +`Dynamic dimensions required for input: input, but no shapes were +provided. Automatically overriding shape to: 1x1x1x1` and builds a +degenerate engine. With an explicit profile +(`--minShapes=input:1x1x8x8 --optShapes=input:1x3x64x64 +--maxShapes=input:2x3x128x128`) the build also passes (5.2 s). + +Consequences, honestly stated: + +- a missing profile is **not** misclassified as an unsupported-operator + failure — it isn't a failure at all on this trtexec version; +- trtcheck's static `TRT-SHAPE-PROFILE-MISSING` (unverified) finding + remains the right warning: the no-profile "success" builds an engine + fixed at 1×1×1×1, which is not a usable dynamic deployment; +- the wrapper reports `verified` for that no-profile success because a + build genuinely succeeded in this environment; the auto-override + warning is preserved in the recorded output tail. This nuance is + listed under limitations. + +This is recorded as TensorRT-10.3-specific evidence only; no +generalization to other versions and no matrix changes were made from it. + +## 4. Defects discovered + +- **trtcheck product defects: none.** All wrapper classifications matched + independent trtexec behavior; JSON stayed schema-valid (2.0) under real + runs; the installed wheel worked end-to-end in-container. +- **Smoke-runner defects (fixed in `scripts/real_tensorrt_smoke.py` + before the final run):** (1) it parsed trtcheck JSON from a truncated + output tail; (2) it mis-extracted the TensorRT version because + `trtexec --version` exits non-zero on this build; (3) its initial + no-profile expectation for dynamic models didn't match real TRT 10.3 + behavior. These are test-harness fixes, not product changes, and the + runner is deterministic and repo-owned. + +## 5. Installed-wheel validation + +The corpus ran the **installed console script** (`pip install +/wheel/trtcheck-1.0.0-py3-none-any.whl` inside the container, repo +mounted read-only): packaged matrix/remediation data loaded, entry point +worked, explicit `--trtexec` configuration worked, JSON reports remained +schema 2.0 with rule ids throughout. + +## 6. Cleanup + +- No engines were saved (`--saveEngine` never passed); container work + dirs live under `/tmp` outside the repo; staged driver libs are removed + by the wrapper's trap. +- Temporary containers were `--rm` (none remain). +- `.gitignore` covers `*.engine`, `*.plan`, `*.trt`. +- The NGC image (14.6 GB) is retained in the local Docker cache — 150+ GB + remain free. Optional removal: `docker rmi nvcr.io/nvidia/tensorrt:24.08-py3`. +- Unrelated images/containers untouched. No host packages or + configuration were changed (nothing to back up or roll back). + +## 7. Limitations + +- Bounded smoke: 7 models, one TensorRT version (10.3.0), one GPU + (RTX 4050 Laptop). This validates the verification integration and the + selected cases — it is not a compatibility benchmark, and results are + not generalized to TensorRT 8.x/10.0 or other hardware. +- `verified` means "trtexec parsed and built an engine in this + environment"; for dynamic models without profiles, TRT 10.3 builds a + degenerate 1×1×1×1 engine (see §3) — the static unverified findings + remain the actionable signal for dynamic deployments. +- The public-model leg used the already-cached, SHA-256-verified + squeezenet1_1 from the ONNX Model Zoo. + +## 8. Reproduction + +```bash +# one-time: docker pull nvcr.io/nvidia/tensorrt:24.08-py3 +python -m build # produce dist/trtcheck-*.whl +scripts/real-smoke-container.sh # runs the whole corpus, prints the summary +# results JSON is written to the printed temp dir; the committed copy is +# bench/real_tensorrt_smoke_results.json +``` + +On a host with the NVIDIA Container Toolkit the script uses `--gpus all` +automatically; otherwise it falls back to the no-root manual passthrough +described above. + +## 9. Release recommendation + +The one named external check from `RELEASE_READINESS_REPORT.md` — a real +trtexec smoke with at least one genuine success and one genuine failure — +has now passed, with full wrapper/direct agreement, using the installed +wheel. **Recommendation: release v1.1.0** (after the routine version-bump +and changelog-roll checklist in `RELEASE_NOTES_DRAFT.md`). diff --git a/RELEASE_NOTES_DRAFT.md b/RELEASE_NOTES_DRAFT.md new file mode 100644 index 0000000..c0cf815 --- /dev/null +++ b/RELEASE_NOTES_DRAFT.md @@ -0,0 +1,62 @@ +# trtcheck v1.1.0 — release notes + +Theme: **honest verdicts, safe fixes.** + +## Highlights + +- **Four-state verdicts.** Reports now conclude `blocked`, `unverified`, + `likely`, or `verified` instead of a single boolean. Unknown operators + and custom-domain ops no longer disappear silently — they make the + verdict `unverified`, with per-op findings. `--fail-on unverified` + turns that into a CI failure; `--plugin-domain` declares plugin-backed + domains. +- **Stable rule ids (report schema 2.0).** Filter CI on + `TRT-OP-UNSUPPORTED`, `TRT-DTYPE-UINT8-INPUT`, `TRT-CONTROL-LOOP-*`, + ... Registry in `docs/rules.md`, guarded by a stability test. All 1.x + JSON keys (including `conversion_likely`) are still emitted. +- **Safe `--fix`.** The fix pipeline is transactional (a crashing fixer — + including third-party plugins — cannot leave a half-rewritten model), + validates with strict type/shape inference, honors `--target-trt`, and + reports findings resolved / remaining / introduced. Two correctness + fixes ship with it: + - `int64_to_int32` no longer corrupts `Reshape`/`Slice`-style + schema-required-INT64 inputs; it converts only provably safe uses. + - `drop_dropout` no longer removes Dropouts whose `training_mode` is + true, dynamic, or unresolvable. +- **Conditional operator support.** The matrix can now express documented + per-operator conditions (first: TopK `sorted=1` / `K < 3840`, Resize + mode + coordinate-transform + antialias restrictions, sourced from the + upstream onnx-tensorrt table). +- **Optional runtime verification.** `trtcheck model.onnx + --verify-runtime` runs `trtexec` when available and upgrades the + verdict to `verified` on a successful build; a recorded parser/build + *failure* demotes an otherwise-clean report to `unverified`. Static + analysis still needs no TensorRT, no GPU. + +## Runtime evidence + +Runtime verification was smoke-tested on TensorRT 10.3.0 using 7 +representative generated/public fixtures in the official NGC container +(5 genuine builds, 2 genuine failures, full wrapper/trtexec agreement, +installed wheel). This validates the verification integration and the +selected cases, not universal model compatibility. + +## Compatibility + +- Public API: additive. `conversion_likely` is deprecated but present. +- JSON schema 2.0 is a superset of 1.x; consumers should migrate from + `conversion_likely` to `verdict`. +- Exit codes unchanged by default. `--severity` no longer influences the + exit code (it was display-only in intent; now in behavior too). +- Behavior change: models with unclassified/custom operators previously + reported clean; they now report `unverified` findings (INFO severity, + exit 0 unless `--fail-on unverified`). +- Behavior change: `--fix` refuses structurally invalid input models and + converts fewer INT64 initializers than before (only provably safe + uses). This is deliberate: the removed conversions were unsound. + +## Install + +```bash +pip install trtcheck==1.1.0 +``` diff --git a/RELEASE_READINESS_REPORT.md b/RELEASE_READINESS_REPORT.md new file mode 100644 index 0000000..beeca4c --- /dev/null +++ b/RELEASE_READINESS_REPORT.md @@ -0,0 +1,187 @@ +# Release readiness report — trtcheck + +Date: 2026-07-22. + +## 1. Executive status: **COMPLETE** + +All critical hardening claims were independently re-verified from the +repository state; four real gaps found during the audit were fixed with +regression tests; the package installs and works from a fresh venv outside +the source tree; security review is documented; no claim in the repo +depends on unavailable hardware evidence. Remote state untouched. + +## 2. Branches and commits + +- Starting point: `claude/trtcheck-hardening` @ `01b3f43` + (5 commits on top of `main` @ `374fe48`) — clean tree, confirmed. +- Work branch: `claude/trtcheck-release-readiness`, created from that HEAD. + Hardening commits were not rewritten, squashed, or rebased. + +## 3. Environment + +Linux 6.8.0-106-generic; repo venv Python **3.13.9** (Anaconda build; note: +mypy checks against `python_version = "3.10"` per pyproject, and the +package declares 3.10–3.13); onnx 1.21.0, click 8.4.1, numpy 2.4.6, +rich 15.0.0. **TensorRT: not installed. trtexec: not on PATH. GPU: not +used.** Network was available (used only for pip installs of dev tools). + +## 4. Claims independently verified + +| Handoff claim | Verification | +|---|---| +| 5 commits on main@374fe48, clean tree | `git log/status` — confirmed | +| 445 passed / 1 skipped; mypy strict, black, isort clean | Re-run at baseline — reproduced exactly | +| Reshape INT64 regression pinned; blind cast breaks full validation | `tests/test_fixers_int64_schema.py` proves the failure mode and the refusal; shared/mixed consumers, nested captures, shadowing, custom domains, graph-boundary tensors, overflow, empty and dead initializers all covered | +| Transactional fixers | Adversarial fixtures: mutate-then-raise, emit-invalid, undeclared mutation, malformed return — none can affect output; failures identify the fixer; later fixers run | +| Dropout training-mode safety | absent/false/true/dynamic/computed/ambiguous + is_test + mask + cross-scope cases all tested | +| Four-state verdicts + precedence | blocked > verified > unverified > likely tested, incl. combined conditions; static analysis cannot set `verified` (only the runtime path flips `runtime_verified`); missing trtexec is a controlled status | +| Stable rule ids / schema 2.0 | Registry pinned by test; every emitted finding carries an id; 1.x keys preserved; rendering deterministic (new test) | +| Unknown/custom ops cannot pass silently | Fixtures per category; `--plugin-domain` opt-out tested | +| Target-aware `--fix` diff, safe writes, exit codes | `tests/test_cli_fix.py`; invalid outputs never written; overwrite guards tested | +| Runtime verification isolation | list-args, timeout, output truncation, five failure states — mocked tests | +| Packaging data files, entry points | Verified in built artifacts + external smoke | + +## 5. Defects discovered during this audit (and fixed) + +1. **Runtime failure could hide behind static `likely`.** A recorded + trtexec parser/build failure left the verdict `likely`. Now demoted to + `unverified` (statuses that merely could not run — missing binary, + timeout, spawn error — leave the static verdict). Unit + CLI tests. +2. **Lying fix records.** A fixer returning `FixApplied` records without + modifying the model had its records committed. Now rejected with an + explicit `FixFailure`; later fixers unaffected. Test added. +3. **Blank plugin rule ids.** Plugin checker findings without a `rule_id` + reached reports empty. Now assigned a namespaced `PLUGIN-` + fallback. Test added. +4. **Diff identity ignored graph scope.** Same-named nodes in different + subgraphs could alias in `--fix` before/after comparisons. + `Issue.identity()` now includes `graph_scope`. Test added. + +## 6. Test results (final) + +- `./scripts/run-tests.sh` → **456 passed, 1 skipped** (opt-in benchmark), ~1.5 s +- `mypy trtcheck/ --strict` → clean (26 files) +- `black --check .` / `isort --check-only .` → clean +- No linter (ruff/flake8) is configured for this repo; none was added. + +## 7. Packaging results + +- Stale untracked `build/`, `dist/`, `*.egg-info` removed; rebuilt with + `python -m build` → `trtcheck-1.0.0.tar.gz` + `trtcheck-1.0.0-py3-none-any.whl` +- `twine check dist/*` → PASSED for both artifacts +- sdist verified to include `LICENSE`, `README.md`, `trtcheck/data/*.json` +- **External smoke (`scripts/package-smoke.sh`, added this pass): PASS** — + fresh venv in `/tmp`, wheel (non-editable) install, then from outside the + repo: import + version, packaged data files, `trtcheck --help`, + `python -m trtcheck --version`, analyze a generated model (console + + schema-2.0 JSON), `--fix` producing a fully-valid model, and + missing-trtexec behavior under an empty `PATH`. + +## 8. Security / privacy findings + +See `SECURITY_REVIEW.md`. Summary: no secrets, personal paths, employer +identifiers, or large/generated binaries in tracked files; `yaml.safe_load` +only; no pickle/eval/shell=True/archive extraction; subprocess use is +list-args with timeouts; downloads are https + SHA-256 verified; model- +derived text is sanitized for terminal and HTML; plugin trust boundary +documented. No security-critical fixes were required this pass. + +## 9. Runtime evidence + +**Real smoke passed (2026-07-22).** On branch +`claude/trtcheck-ngc-runtime-validation`, the 7-model corpus ran against +genuine TensorRT **10.3.0** (official NGC container +`nvcr.io/nvidia/tensorrt:24.08-py3`, RTX 4050 Laptop GPU, driver +580.126.20) using the installed wheel: 5 genuine engine builds, 2 genuine +parser failures, 0 wrapper/direct-trtexec disagreements. Full evidence: +`REAL_TENSORRT_VALIDATION_REPORT.md` + +`bench/real_tensorrt_smoke_results.json`. This validates the verification +integration and the selected cases, not universal model compatibility. + +(An earlier same-day host-level attempt on +`claude/trtcheck-real-tensorrt-validation` was BLOCKED — no host TensorRT +existed; the container route above resolved it without any host changes. +Reproduce with `scripts/real-smoke-container.sh`.) + +## 10. Benchmark evidence and limitations + +12-model corpus (3 ONNX Model Zoo + 9 deterministic bundled fixtures); +blocker precision/recall 1.000 with unverified coverage 0.250; unverified +predictions are never counted as success. Ground truth is **documented +TRT behavior, not live builds** — stated in `SCORECARD.md` and the README. +Machine-readable summary: `bench/summary.json` (new `score.py --json`). +Fixtures are generated by `tests/fixtures/generate_broken.py` +(deterministic, byte-stable — verified by regeneration). + +## 11. Public API / schema changes (this branch, on top of hardening) + +- `AnalysisReport.verdict`: new demotion rule for recorded runtime + failures (documented in docstring, docs/usage.md, design doc). +- `Issue.identity()` returns a 4-tuple (adds `graph_scope`) — internal + diffing helper, not part of the JSON schema. +- `run_fixers` rejects claimed-but-absent changes (new failure mode + string; `FixFailure` shape unchanged). +- `bench/score.py` gains `--json`; `ScoreResult.to_dict()`. +- JSON schema remains 2.0 — no key added or removed this pass. + +## 12. Migration instructions + +For 1.x JSON consumers: switch `conversion_likely` → `verdict`; filter +findings on `rule_id` (`docs/rules.md`). All 1.x keys still emitted. +CI gates that must treat unresolved conditions as failures: add +`--fail-on unverified`. No Python API removals. + +## 13. Files changed (this branch: 16 files, +400/−13) + +`trtcheck/types.py`, `trtcheck/analyzer.py`, `trtcheck/fixers/__init__.py` +(audit fixes); `bench/score.py`, `bench/summary.json`; +`scripts/package-smoke.sh`; tests (`test_verdicts`, `test_runtime_verify`, +`test_fixers_transactional`, `test_plugins_module`, `test_bench_score`); +docs (`usage.md`, design doc, README, CHANGELOG, RELEASE_NOTES_DRAFT); +new `SECURITY_REVIEW.md`, `PR_BODY_DRAFT.md`, this report. + +## 14. Local commits (this branch) + +1. `a335d6f` fix: close four audit gaps from independent hardening review +2. `2921dbe` chore: package smoke script (fresh-venv wheel install + CLI exercise) +3. (branch HEAD) docs: security review, PR draft, release readiness report, README/CHANGELOG updates — the commit containing this file; see `git log -1` + +(Plus the five untouched hardening commits beneath.) + +## 15. Version recommendation + +**1.1.0.** Additive public API and JSON schema (2.0 is a superset of 1.x); +deliberate, documented behavior corrections (unclassified ops → +unverified; sounder `--fix` refusals; runtime-failure demotion). Not a +patch (user-visible behavior changes); not 2.0.0 (nothing removed, no key +repurposed). Bump `trtcheck/__init__.py.__version__` and +`pyproject.toml` together at release time — do not bump on this branch. + +## 16. Remaining external actions + +1. ~~Run the real trtexec smoke on a TensorRT machine~~ — **DONE** + (2026-07-22, NGC TensorRT 10.3 container; see + `REAL_TENSORRT_VALIDATION_REPORT.md`). No external checks remain. +2. Push branches, open the PR (draft body in `PR_BODY_DRAFT.md`), let CI + run on all supported Pythons. +3. Optional: refresh `assets/demo.svg` from live CLI output (text updated; + layout not re-rendered). + +## 17. Commands for later (do not run automatically) + +```bash +# inspect +git log --oneline main..claude/trtcheck-release-readiness +git diff main...claude/trtcheck-release-readiness + +# push + PR +git push -u origin claude/trtcheck-hardening claude/trtcheck-release-readiness +gh pr create --base main --head claude/trtcheck-release-readiness \ + --title "Honest verdicts, safe fixes, release readiness" \ + --body-file PR_BODY_DRAFT.md + +# release (after merge + version bump + CHANGELOG roll) +git tag -a v1.1.0 -m "trtcheck 1.1.0" +git push origin v1.1.0 +python -m build && python -m twine check dist/* && python -m twine upload dist/* +``` diff --git a/SCORECARD.md b/SCORECARD.md index 502b4f2..854f591 100644 --- a/SCORECARD.md +++ b/SCORECARD.md @@ -1,65 +1,82 @@ # trtcheck validation scorecard -Measured accuracy of trtcheck's conversion verdicts against the +Measured accuracy of trtcheck's **static** verdicts against the `bench/manifest.yaml` corpus. Produced by the `bench/` harness; raw predictions in [`bench/outcomes.json`](bench/outcomes.json). -- **trtcheck version:** 1.0.0 -- **Target:** TensorRT 10.3, `--severity critical` (the CI gate configuration) -- **Corpus:** 9 models — 3 public (ONNX Model Zoo), 6 bundled fixtures -- **Run date:** 2026-07-02 -- **Hardware:** none. Static analysis only; total wall time for all 9 models: **2.0 s**. +- **trtcheck version:** 1.0.0 + unreleased verdict-model changes (branch `claude/trtcheck-hardening`) +- **Target:** TensorRT 10.3, full report (verdict-based; the old `--severity critical` gate is no longer used because verdicts need the INFO-level uncertainty findings) +- **Corpus:** 12 models — 3 public (ONNX Model Zoo), 9 bundled fixtures +- **Run date:** 2026-07-22 +- **Hardware:** none. Static analysis only; total wall time for all 12 models: **2.3 s** (per-model times below include Python interpreter startup). -## Results +Since the verdict model landed, predictions are three-way: `fail` +(blocked), `convert` (likely/verified), and `unverified` (no known +blocker, unresolved conditions). **Unverified predictions are never +counted as successes** — they are reported as coverage, split by ground +truth. + +## Results (blocker confusion matrix, unverified excluded) | | expected: fail | expected: convert | |---|---|---| -| **trtcheck: fail** | 3 (TP) | 0 (FP) | -| **trtcheck: convert** | 0 (FN) | 6 (TN) | +| **trtcheck: fail** | 4 (TP) | 0 (FP) | +| **trtcheck: convert** | 0 (FN) | 5 (TN) | | Metric | Value | |---|---| -| Precision | **1.000** | -| Recall | **1.000** | -| F1 | **1.000** | +| Blocker precision | **1.000** | +| Blocker recall | **1.000** | +| Blocker F1 | **1.000** | +| Unverified coverage | **0.250** (3 of 12: 1 real failure, 2 that actually convert) | ## Per-model outcomes | Model | Source | Expected | trtcheck | Time | |---|---|---|---|---| -| resnet50_v2 | ONNX Model Zoo | convert | convert | 0.32 s | -| mobilenetv2_1_0 | ONNX Model Zoo | convert | convert | 0.24 s | -| squeezenet1_1 | ONNX Model Zoo | convert | convert | 0.19 s | -| bundled_sequence_empty | fixture | fail | fail | 0.18 s | -| bundled_uint8_input | fixture | fail | fail | 0.22 s | -| bundled_int64_weights | fixture | convert | convert | 0.22 s | -| bundled_control_flow_loop | fixture | fail | fail | 0.22 s | -| bundled_fully_dynamic | fixture | convert | convert | 0.19 s | -| bundled_clean_minimal | fixture | convert | convert | 0.20 s | +| resnet50_v2 | ONNX Model Zoo | convert | convert | 0.29 s | +| mobilenetv2_1_0 | ONNX Model Zoo | convert | convert | 0.21 s | +| squeezenet1_1 | ONNX Model Zoo | convert | convert | 0.16 s | +| bundled_sequence_empty | fixture | fail | fail | 0.17 s | +| bundled_uint8_input | fixture | fail | fail | 0.21 s | +| bundled_int64_weights | fixture | convert | convert | 0.20 s | +| bundled_control_flow_loop | fixture | fail | fail | 0.17 s | +| bundled_fully_dynamic | fixture | convert | **unverified** | 0.17 s | +| bundled_clean_minimal | fixture | convert | convert | 0.21 s | +| bundled_topk_unsorted | fixture | fail | fail | 0.15 s | +| bundled_custom_domain | fixture | fail | **unverified** | 0.15 s | +| bundled_reshape_int64_shape | fixture | convert | convert | 0.16 s | ## What this run caught -The first pass of this harness scored 8/9: `bundled_control_flow_loop` -was a false negative. The checker flagged a Loop trip count fed from a -graph input — a pattern TensorRT always rejects at engine build — as -WARNING, so the `--severity critical` gate waved it through. That -finding became the `loop_runtime_trip_count` critical check (a trip -count *computed* inside the graph, which TRT may still shape-infer, -stays a warning). The harness exists precisely to surface this class of -misclassification. +- The honesty change immediately exposed a **matrix coverage gap**: + `mobilenetv2_1_0` came back `unverified` because `Clip` (35 nodes) was + absent from the operator matrix. `Clip` is now classified for TRT 10.x + from the upstream onnx-tensorrt table (8.x left `unknown` — no evidence + in hand), and the model classifies cleanly again. That is exactly the + loop the unverified verdict exists to drive. +- `bundled_topk_unsorted` (TopK `sorted=0`) is caught by the new + conditional-support rules (`TRT-OP-CONDITION`), not by an operator-level + blanket status. +- `bundled_custom_domain` is *expected: fail* (no TensorRT plugin exists + for it) and trtcheck reports `unverified` — honest: static analysis + cannot know whether a plugin is installed in the deployment environment. + With `--fail-on unverified`, a CI gate still fails it. +- `bundled_fully_dynamic` is `unverified` because a usable engine needs an + optimization profile trtcheck cannot see; the manifest labels it + `convert` since a build with profiles succeeds. ## Honest limitations -- **Small corpus.** Nine models. The three public models are - well-behaved classifiers; the failure cases are synthetic fixtures - built to exhibit specific TRT failure modes. Treat these numbers as - "the checks do what they claim on known patterns," not as a +- **Small corpus.** Twelve models; the failure cases are synthetic + fixtures built to exhibit specific TRT failure modes. Treat these + numbers as "the checks do what they claim on known patterns," not a field-accuracy estimate. - **Ground truth is the manifest, not live `trtexec`.** The `expected` - labels encode documented TRT behavior. The harness supports a second - leg — running `trtexec` on GPU hardware and recording drift between - the manifest and reality — which has not been run yet. See - [`bench/README.md`](bench/README.md) for the GPU protocol. + labels encode documented TRT behavior. No TensorRT/GPU run was + performed for this scorecard; the harness supports recording real + `trtexec` outcomes and reporting drift (see `bench/README.md`), and + `trtcheck --verify-runtime` now does the same per-model. - **One TRT target.** Scored against the 10.3 operator matrix only. ## Reproduce @@ -70,6 +87,9 @@ python bench/predict.py # writes bench/outcomes.json python bench/score.py --outcomes bench/outcomes.json ``` +Environment for the numbers above: Linux 6.8, Python 3.10 venv, +onnx 1.21.0, no GPU, no TensorRT installed. + ## Expand the corpus Add entries to `bench/manifest.yaml` (stable URL + `expected` outcome + diff --git a/SECURITY_REVIEW.md b/SECURITY_REVIEW.md new file mode 100644 index 0000000..41bb24b --- /dev/null +++ b/SECURITY_REVIEW.md @@ -0,0 +1,49 @@ +# Security review — release-readiness pass (2026-07-22) + +Scope: the `claude/trtcheck-release-readiness` branch, all tracked files +plus build output. This is a project self-review, not an external audit. +Vulnerability reporting instructions live in [SECURITY.md](SECURITY.md). + +## Threat model in one paragraph + +trtcheck parses **untrusted ONNX protobufs** (CI runs it on arbitrary PR +artifacts), loads **semi-trusted entry-point plugins** the user chose to +install, and optionally shells out to a **user-provided trtexec**. It has +no network access at analysis time, no credentials, and writes only files +the user names. + +## Reviewed surfaces and findings + +| Surface | Status | +|---|---| +| Secrets / credentials in tracked files & build output | **Clean.** Only CI `${{ secrets.GITHUB_TOKEN }}` references and API-token *variable names* in `action/post_comment.py` (token comes from the workflow env, never stored). | +| Personal / employer material | **Clean.** No local paths, usernames, private hostnames, or employer identifiers in tracked files. Author name/public email in `pyproject.toml`/`LICENSE` are intentional. | +| Large or generated binaries | **Clean.** Largest tracked file is 24 KB; the only binaries are nine deterministic test fixtures (<100 KB total) regenerated by `tests/fixtures/generate_broken.py`. `dist/`, `build/`, `*.egg-info`, `bench/cache/` are gitignored. | +| Untrusted model parsing | `onnx.load` failures are wrapped into a domain error (no raw tracebacks); a size cap (`--max-model-size`, default 500 MB) bounds memory; subgraph traversal is depth-bounded (256) against stack-exhaustion inputs. | +| Terminal / HTML injection from model-derived text | Node names, producer strings, and filenames are stripped of control chars, ANSI escapes, and bidi overrides (`trtcheck/_text.py`) before console output, and additionally HTML-escaped in the HTML reporter; `docs_link` hrefs are restricted to http(s). Covered by `tests/test_reporters_security.py`. | +| Subprocess use | Only `trtcheck/runtime_verify.py` (trtexec) and the bench/predict harness. Both use list-args (`shell=False` by construction), timeouts, and captured, truncated output. No user string ever reaches a shell. Covered by `tests/test_runtime_verify.py`. | +| YAML / pickle / archive handling | `yaml.safe_load` only (bench/tools); no `pickle`, no `eval`/`exec`, no `tarfile`/`zipfile` extraction anywhere in the package. | +| Network fetches | `bench/fetch.py` (opt-in, dev-only) downloads over https with SHA-256 verification against the manifest; `tools/check_matrix_drift.py` fetches one fixed https URL with a timeout. The installed package performs no network I/O. | +| Path handling | Output paths are user-supplied CLI arguments; the CLI refuses to overwrite existing files without `--force` and refuses to overwrite the `--fix` input file. No path is derived from model content. | +| Plugin trust boundary | Entry-point plugins are semi-trusted *code the user installed* — loading a plugin is arbitrary code execution by design, same as any pip install. Within that boundary: plugin checker crashes are contained and surfaced as findings (missing coverage is visible), plugin fixers run inside the transactional pipeline and cannot corrupt output, plugin tracebacks are hidden unless `TRTCHECK_DEBUG=1`, and unnamespaced plugin findings get a `PLUGIN-` rule-id fallback. | +| Dependency ranges | `onnx>=1.15,<2.0`, `click>=8.1,<9.0`, `rich>=13.5,<16.0` — upper-bounded majors, no known-vulnerable pins. Dev-only extras are not installed with the package. | + +## Fixes made during this pass + +- None security-critical. (The transactional-fixer and verdict-honesty + fixes on this branch are correctness/robustness work; they also shrink + the blast radius of a malicious plugin fixer.) + +## Accepted risks + +- Installing a third-party trtcheck plugin executes its code at import + time — inherent to the entry-point mechanism; documented in the plugin + SDK design doc. +- `--verify-runtime` executes whatever binary `--trtexec` points at (or + the first `trtexec` on PATH). That is the feature; users control PATH. +- `onnx`/protobuf parsing bugs are upstream; the size cap and error + wrapping reduce, not eliminate, exposure to hostile protobufs. + +## Reporting + +See [SECURITY.md](SECURITY.md) for how to report a vulnerability. diff --git a/action.yml b/action.yml index 28f0a85..7705a32 100644 --- a/action.yml +++ b/action.yml @@ -10,7 +10,7 @@ inputs: version: description: 'Pinned trtcheck PyPI version' required: false - default: '1.0.0' + default: '1.1.0' target-trt: description: 'TensorRT version to check against (e.g. 10.3)' required: false diff --git a/assets/banner.svg b/assets/banner.svg index f9a947a..5ba96a8 100644 --- a/assets/banner.svg +++ b/assets/banner.svg @@ -1,6 +1,6 @@ - + trtcheck - Masthead for trtcheck. The wordmark, the line "Static pre-flight checks for ONNX to TensorRT conversion.", and a terminal panel running trtcheck on model.onnx: a red CONVERSION WILL FAIL verdict beside the opset, node count, and finding counts. + Masthead for trtcheck. The wordmark, the line "Static pre-flight checks for ONNX to TensorRT conversion.", and a terminal panel running trtcheck on model.onnx: a red CONVERSION BLOCKED verdict beside the opset, node count, and finding counts. @@ -20,7 +20,7 @@ $ trtcheck model.onnx - CONVERSION WILL FAIL + CONVERSION BLOCKED file: model.onnx opset: 17 · nodes: 2 · 1 critical, 0 warning diff --git a/assets/demo.svg b/assets/demo.svg index b8009e0..82caf95 100644 --- a/assets/demo.svg +++ b/assets/demo.svg @@ -1,6 +1,6 @@ - + trtcheck demo - A ten-second looping terminal session in three beats. First, trtcheck model.onnx returns a red CONVERSION WILL FAIL verdict and a one-row table describing a UINT8 graph input. Second, trtcheck --fix promotes the input to FLOAT and drops a redundant Cast, writing fixed.onnx. Third, a re-run returns a green LIKELY TO CONVERT verdict with zero critical, warning, and info findings. + A ten-second looping terminal session in three beats. First, trtcheck model.onnx returns a red CONVERSION BLOCKED verdict and a one-row table describing a UINT8 graph input. Second, trtcheck --fix promotes the input to FLOAT and drops a redundant Cast, writing fixed.onnx. Third, a re-run returns a green NO KNOWN BLOCKERS verdict with zero critical, warning, and info findings. @@ -24,7 +24,7 @@ - CONVERSION WILL FAIL + CONVERSION BLOCKED file: model.onnx opset: 17 · nodes: 2 · 1 critical, 0 warning @@ -82,7 +82,7 @@ - LIKELY TO CONVERT + NO KNOWN BLOCKERS 0 critical · 0 warning · 0 info diff --git a/bench/manifest.yaml b/bench/manifest.yaml index ac23fb0..10a08c2 100644 --- a/bench/manifest.yaml +++ b/bench/manifest.yaml @@ -66,3 +66,24 @@ models: source: tests/fixtures/clean_minimal.onnx expected: convert reason: none + + - name: bundled_topk_unsorted + source: tests/fixtures/failing/topk_unsorted.onnx + expected: fail + reason: operator_support + + # No TensorRT plugin exists for com.example::FancyCustomOp, so a real + # trtexec parse fails. trtcheck predicts "unverified" (honest uncertainty), + # which scores as unverified coverage, never as a pass. + - name: bundled_custom_domain + source: tests/fixtures/custom_domain.onnx + expected: fail + reason: operator_support + + # The fixer-safety regression model: valid ONNX, converts fine. Guards + # against trtcheck --fix corrupting Reshape shape inputs and against the + # analyzer over-flagging INT64 shape initializers. + - name: bundled_reshape_int64_shape + source: tests/fixtures/reshape_int64_shape.onnx + expected: convert + reason: none diff --git a/bench/outcomes.json b/bench/outcomes.json index 00361a9..e46a6a1 100644 --- a/bench/outcomes.json +++ b/bench/outcomes.json @@ -22,10 +22,19 @@ "trtcheck": "fail" }, "bundled_fully_dynamic": { - "trtcheck": "convert" + "trtcheck": "unverified" }, "bundled_clean_minimal": { "trtcheck": "convert" + }, + "bundled_topk_unsorted": { + "trtcheck": "fail" + }, + "bundled_custom_domain": { + "trtcheck": "unverified" + }, + "bundled_reshape_int64_shape": { + "trtcheck": "convert" } } } diff --git a/bench/predict.py b/bench/predict.py index 1dd0926..cab5ba7 100644 --- a/bench/predict.py +++ b/bench/predict.py @@ -1,9 +1,8 @@ """Run trtcheck against every manifest entry and write an outcomes file. This is the trtcheck leg of the validation harness: for each model in -bench/manifest.yaml it invokes the CLI with ``--format json --severity -critical`` (the CI gate configuration) and records the verdict as -``convert`` or ``fail``. The result feeds bench/score.py. +bench/manifest.yaml it invokes the CLI with ``--format json`` and records the verdict as +``convert``, ``unverified``, or ``fail``. The result feeds bench/score.py. URL-sourced entries are read from bench/cache/ -- run bench/fetch.py first. Bundled-fixture entries are read in place, so the pipeline works @@ -35,7 +34,20 @@ def resolve_model_path(entry: dict[str, Any], root: Path) -> Path: def verdict_from_report(report: dict[str, Any]) -> str: - """Map trtcheck's JSON report to the outcomes vocabulary.""" + """Map trtcheck's JSON report to the outcomes vocabulary. + + Schema 2.x reports carry a four-state ``verdict``; it maps to three + outcome buckets: blocked -> "fail", unverified -> "unverified", + likely/verified -> "convert". Schema 1.x reports (no ``verdict`` key) + fall back to the boolean ``conversion_likely``. + """ + verdict = report.get("verdict") + if verdict is not None: + if verdict == "blocked": + return "fail" + if verdict == "unverified": + return "unverified" + return "convert" return "convert" if report["conversion_likely"] else "fail" @@ -57,7 +69,7 @@ def predict( f"{entry['name']}: {model} not found -- run bench/fetch.py first?" ) proc = subprocess.run( - [*cmd, str(model), "--format", "json", "--severity", "critical"], + [*cmd, str(model), "--format", "json"], capture_output=True, text=True, ) diff --git a/bench/real_tensorrt_smoke_results.json b/bench/real_tensorrt_smoke_results.json new file mode 100644 index 0000000..94a9395 --- /dev/null +++ b/bench/real_tensorrt_smoke_results.json @@ -0,0 +1,187 @@ +{ + "environment": { + "date": "2026-07-22", + "host_os": "Zorin OS 17.3 (Ubuntu 22.04 base), Linux 6.8.0-106-generic x86_64", + "gpu": "NVIDIA GeForce RTX 4050 Laptop GPU (6 GB)", + "nvidia_driver": "580.126.20", + "container_image": "nvcr.io/nvidia/tensorrt:24.08-py3", + "image_digest": "sha256:9507e5f248fc61a5b2c985ce6e386ecf2576c3d96112ceb38cf88b240d4ca072", + "tensorrt": "10.3.0", + "gpu_access": "manual passthrough (--device /dev/nvidia* + RO driver-lib mounts); no host changes", + "trtcheck_wheel": "trtcheck-1.0.0-py3-none-any.whl (installed, non-editable)", + "scope_note": "bounded smoke of the verification integration and selected cases, not universal model compatibility" + }, + "generated_by": "scripts/real_tensorrt_smoke.py", + "tensorrt_version_line": "TensorRT v100300", + "target_trt": "10.3", + "reshape_int64_fixer_refusal": { + "fixes_applied": [] + }, + "models": [ + { + "name": "clean_minimal", + "model": "clean_minimal.onnx", + "expected_direct_outcome": "build_success", + "static_verdict": "likely", + "static_rule_ids": [], + "verify_runtime": { + "status": "success", + "verdict_after": "verified", + "runtime_verified": true, + "elapsed_s": 5.61 + }, + "direct_trtexec": { + "outcome": "build_success", + "returncode": 0, + "elapsed_s": 5.39, + "diagnostic": "[07/22/2026-10:21:59] [W] * GPU compute time is unstable, with coefficient of variance = 59.0572%.\n[07/22/2026-10:21:59] [W] If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.\n" + }, + "wrapper_agrees_with_direct": true, + "matched_expectation": true + }, + { + "name": "squeezenet1_1_public", + "model": "squeezenet1_1.onnx", + "expected_direct_outcome": "build_success", + "static_verdict": "likely", + "static_rule_ids": [ + "TRT-DTYPE-INT64-WEIGHTS", + "TRT-OPSET-OLD" + ], + "verify_runtime": { + "status": "success", + "verdict_after": "verified", + "runtime_verified": true, + "elapsed_s": 15.31 + }, + "direct_trtexec": { + "outcome": "build_success", + "returncode": 0, + "elapsed_s": 15.13, + "diagnostic": "[07/22/2026-10:22:30] [W] * GPU compute time is unstable, with coefficient of variance = 8.79046%.\n[07/22/2026-10:22:30] [W] If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.\n" + }, + "wrapper_agrees_with_direct": true, + "matched_expectation": true + }, + { + "name": "sequence_empty", + "model": "sequence_empty.onnx", + "expected_direct_outcome": "parser_failure", + "static_verdict": "blocked", + "static_rule_ids": [ + "TRT-DTYPE-INT64-WEIGHTS", + "TRT-OP-UNSUPPORTED" + ], + "verify_runtime": { + "status": "parser_failure", + "verdict_after": "blocked", + "runtime_verified": false, + "elapsed_s": 2.25 + }, + "direct_trtexec": { + "outcome": "parser_failure", + "returncode": 1, + "elapsed_s": 2.02, + "diagnostic": "10:22:34] [E] [TRT] ModelImporter.cpp:951: --- End node ---\n[07/22/2026-10:22:34] [E] [TRT] ModelImporter.cpp:954: ERROR: onnxOpCheckers.cpp:981 In function checkSequenceAt:\n[8] false\n[07/22/2026-10:22:34] [E] Failed to parse onnx file\n[07/22/2026-10:22:34] [E] Parsing model failed\n[07/22/2026-10:22:34] [E] Failed to create engine from model or file.\n[07/22/2026-10:22:34] [E] Engine set up failed\n" + }, + "wrapper_agrees_with_direct": true, + "matched_expectation": true + }, + { + "name": "fully_dynamic", + "model": "fully_dynamic.onnx", + "expected_direct_outcome": "build_success", + "static_verdict": "unverified", + "static_rule_ids": [ + "TRT-SHAPE-PROFILE-MISSING" + ], + "verify_runtime": { + "status": "success", + "verdict_after": "verified", + "runtime_verified": true, + "elapsed_s": 5.55 + }, + "direct_trtexec": { + "outcome": "build_success", + "returncode": 0, + "elapsed_s": 5.39, + "diagnostic": "[07/22/2026-10:22:42] [W] Dynamic dimensions required for input: input, but no shapes were provided. Automatically overriding shape to: 1x1x1x1\n[07/22/2026-10:22:45] [W] * GPU compute time is unstable, with coefficient of variance = 79.5852%.\n[07/22/2026-10:22:45] [W] If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.\n" + }, + "wrapper_agrees_with_direct": true, + "matched_expectation": true, + "direct_trtexec_with_profile": { + "outcome": "build_success", + "returncode": 0, + "elapsed_s": 5.36, + "profile": "--minShapes=input:1x1x8x8 --optShapes=input:1x3x64x64 --maxShapes=input:2x3x128x128" + } + }, + { + "name": "custom_domain", + "model": "custom_domain.onnx", + "expected_direct_outcome": "parser_failure", + "static_verdict": "unverified", + "static_rule_ids": [ + "TRT-OP-CUSTOM-DOMAIN" + ], + "verify_runtime": { + "status": "parser_failure", + "verdict_after": "unverified", + "runtime_verified": false, + "elapsed_s": 2.23 + }, + "direct_trtexec": { + "outcome": "parser_failure", + "returncode": 1, + "elapsed_s": 2.07, + "diagnostic": "] ModelImporter.cpp:954: ERROR: onnxOpCheckers.cpp:781 In function checkFallbackPluginImporter:\n[6] creator && \"Plugin not found, are the plugin name, version, and namespace correct?\"\n[07/22/2026-10:22:55] [E] Failed to parse onnx file\n[07/22/2026-10:22:55] [E] Parsing model failed\n[07/22/2026-10:22:55] [E] Failed to create engine from model or file.\n[07/22/2026-10:22:55] [E] Engine set up failed\n" + }, + "wrapper_agrees_with_direct": true, + "matched_expectation": true + }, + { + "name": "uint8_fixed_via_fix", + "model": "uint8_fixed.onnx", + "expected_direct_outcome": "build_success", + "static_verdict": "likely", + "static_rule_ids": [], + "verify_runtime": { + "status": "success", + "verdict_after": "verified", + "runtime_verified": true, + "elapsed_s": 6.43 + }, + "direct_trtexec": { + "outcome": "build_success", + "returncode": 0, + "elapsed_s": 6.4, + "diagnostic": "vice transfers for the inputs rather than GPU Compute and the GPU may be under-utilized.\n[07/22/2026-10:23:08] [W] Add --noDataTransfers flag to disable data transfers.\n[07/22/2026-10:23:08] [W] * GPU compute time is unstable, with coefficient of variance = 95.7458%.\n[07/22/2026-10:23:08] [W] If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.\n" + }, + "wrapper_agrees_with_direct": true, + "matched_expectation": true + }, + { + "name": "reshape_int64_shape", + "model": "reshape_int64_shape.onnx", + "expected_direct_outcome": "build_success", + "static_verdict": "likely", + "static_rule_ids": [ + "TRT-DTYPE-INT64-WEIGHTS" + ], + "verify_runtime": { + "status": "success", + "verdict_after": "verified", + "runtime_verified": true, + "elapsed_s": 5.45 + }, + "direct_trtexec": { + "outcome": "build_success", + "returncode": 0, + "elapsed_s": 5.18, + "diagnostic": "[07/22/2026-10:23:19] [W] * GPU compute time is unstable, with coefficient of variance = 66.0441%.\n[07/22/2026-10:23:19] [W] If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.\n" + }, + "wrapper_agrees_with_direct": true, + "matched_expectation": true + } + ] +} \ No newline at end of file diff --git a/bench/score.py b/bench/score.py index 18e3ca2..3f3aec5 100644 --- a/bench/score.py +++ b/bench/score.py @@ -36,6 +36,11 @@ _REPO_ROOT = Path(__file__).resolve().parent.parent _VALID_OUTCOMES = {"convert", "fail"} +# trtcheck predictions may additionally be "unverified": no known blocker but +# unresolved conditions remain. Unverified predictions are excluded from the +# blocker confusion matrix and reported as coverage instead -- an unverified +# call is neither a caught failure nor a clean bill of health. +_VALID_PREDICTIONS = _VALID_OUTCOMES | {"unverified"} @dataclass @@ -48,11 +53,46 @@ class ScoreResult: false_negative: int = 0 skipped: list[str] = field(default_factory=list) drift: list[str] = field(default_factory=list) + # Entries trtcheck declined to classify (prediction == "unverified"), + # split by what the ground truth says they actually do. + unverified_on_fail: list[str] = field(default_factory=list) + unverified_on_convert: list[str] = field(default_factory=list) @property def total(self) -> int: return self.true_positive + self.false_positive + self.true_negative + self.false_negative + @property + def unverified_total(self) -> int: + return len(self.unverified_on_fail) + len(self.unverified_on_convert) + + def to_dict(self) -> dict[str, Any]: + """Machine-readable summary (written by --json).""" + return { + "true_positive": self.true_positive, + "false_positive": self.false_positive, + "true_negative": self.true_negative, + "false_negative": self.false_negative, + "scored": self.total, + "blocker_precision": self.precision, + "blocker_recall": self.recall, + "blocker_f1": self.f1, + "unverified_on_fail": list(self.unverified_on_fail), + "unverified_on_convert": list(self.unverified_on_convert), + "unverified_coverage": self.unverified_coverage, + "skipped": list(self.skipped), + "drift": list(self.drift), + } + + @property + def unverified_coverage(self) -> float: + """Fraction of all classified-or-unverified entries trtcheck declined + to classify. High coverage with low blocker recall means the tool is + honest but not yet informative; low coverage with high recall is the + goal.""" + denom = self.total + self.unverified_total + return self.unverified_total / denom if denom else 0.0 + @property def precision(self) -> float: denom = self.true_positive + self.false_positive @@ -94,13 +134,21 @@ def score( continue trtcheck_pred = pred_block["trtcheck"] - if trtcheck_pred not in _VALID_OUTCOMES: + if trtcheck_pred not in _VALID_PREDICTIONS: raise ValueError(f"outcomes['{name}'].trtcheck has invalid value {trtcheck_pred!r}") trtexec_pred = pred_block.get("trtexec") if trtexec_pred and trtexec_pred != expected: result.drift.append(name) + if trtcheck_pred == "unverified": + # Never counted as success: tracked separately, both ways. + if expected == "fail": + result.unverified_on_fail.append(name) + else: + result.unverified_on_convert.append(name) + continue + # "fail" is the positive class if trtcheck_pred == "fail" and expected == "fail": result.true_positive += 1 @@ -123,9 +171,19 @@ def format_report(s: ScoreResult) -> str: lines.append(f" true negative: {s.true_negative} (trtcheck=convert, expected=convert)") lines.append(f" false negative:{s.false_negative} (trtcheck=convert, expected=fail)") lines.append("") - lines.append(f" precision: {s.precision:.3f}") - lines.append(f" recall: {s.recall:.3f}") - lines.append(f" f1: {s.f1:.3f}") + lines.append(f" blocker precision: {s.precision:.3f}") + lines.append(f" blocker recall: {s.recall:.3f}") + lines.append(f" blocker f1: {s.f1:.3f}") + if s.unverified_total: + lines.append("") + lines.append( + f" unverified: {s.unverified_total} " + f"(coverage {s.unverified_coverage:.3f}; " + f"{len(s.unverified_on_fail)} were real failures, " + f"{len(s.unverified_on_convert)} actually convert)" + ) + for name in s.unverified_on_fail + s.unverified_on_convert: + lines.append(f" - {name}") if s.skipped: lines.append("") lines.append(f" skipped (no prediction): {len(s.skipped)}") @@ -153,6 +211,13 @@ def main(argv: list[str] | None = None) -> int: required=True, help="Path to outcomes.json produced by the validation runner.", ) + parser.add_argument( + "--json", + type=Path, + default=None, + metavar="PATH", + help="Also write the summary as machine-readable JSON to PATH.", + ) args = parser.parse_args(argv) try: @@ -170,6 +235,9 @@ def main(argv: list[str] | None = None) -> int: result = score(manifest, outcomes) print(format_report(result)) + if args.json is not None: + args.json.write_text(json.dumps(result.to_dict(), indent=2) + "\n") + print(f"wrote {args.json}") return 0 diff --git a/bench/summary.json b/bench/summary.json new file mode 100644 index 0000000..d3444f0 --- /dev/null +++ b/bench/summary.json @@ -0,0 +1,19 @@ +{ + "true_positive": 4, + "false_positive": 0, + "true_negative": 6, + "false_negative": 0, + "scored": 10, + "blocker_precision": 1.0, + "blocker_recall": 1.0, + "blocker_f1": 1.0, + "unverified_on_fail": [ + "bundled_custom_domain" + ], + "unverified_on_convert": [ + "bundled_fully_dynamic" + ], + "unverified_coverage": 0.16666666666666666, + "skipped": [], + "drift": [] +} diff --git a/docs/case-studies/uint8-input.md b/docs/case-studies/uint8-input.md index b96606f..645b4d7 100644 --- a/docs/case-studies/uint8-input.md +++ b/docs/case-studies/uint8-input.md @@ -46,25 +46,28 @@ $ trtcheck tests/fixtures/failing/uint8_input.onnx ``` ``` -╭─────────────────── trtcheck report ───────────────────╮ -│ CONVERSION WILL FAIL │ -│ file: tests/fixtures/failing/uint8_input.onnx │ -│ opset: 17 producer: trtcheck-fixtures nodes: 2 │ -│ 1 critical 0 warning 0 info │ -╰───────────────────────────────────────────────────────╯ +╭──────────────────────── trtcheck report ────────────────────────╮ +│ CONVERSION BLOCKED -- known critical incompatibilities │ +│ file: tests/fixtures/failing/uint8_input.onnx target: TRT 10.3 │ +│ opset: 17 producer: trtcheck-fixtures nodes: 2 │ +│ 1 critical 0 warning 0 info │ +╰─────────────────────────────────────────────────────────────────╯ Detected issues -┏━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ -┃ Severity ┃ Node ┃ Operator ┃ Issue ┃ Fix ┃ -┡━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ -│ CRITICAL │ input │ Input │ Input 'input' has │ Move the UINT8 → │ -│ │ │ │ dtype UINT8; │ FLOAT32 conversion │ -│ │ │ │ TensorRT accepts only │ (and normalization) │ -│ │ │ │ FP32, FP16, INT32, or │ into your │ -│ │ │ │ INT8 as graph inputs. │ preprocessing │ -│ │ │ │ │ pipeline rather than │ -│ │ │ │ │ the model body. │ -└──────────┴───────┴──────────┴───────────────────────┴───────────────────────┘ +┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ +┃ Severity ┃ Rule ┃ Node ┃ Operator ┃ Issue ┃ Fix ┃ +┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ +│ CRITICAL │ TRT-DTYPE-UINT8-INPUT │ input │ Input │ Input 'input' │ Apply the │ +│ │ │ │ │ has dtype │ UINT8 -> │ +│ │ │ │ │ UINT8. │ FLOAT32 │ +│ │ │ │ │ TensorRT │ conversion │ +│ │ │ │ │ inputs must │ (and │ +│ │ │ │ │ be FLOAT32, │ normalization)│ +│ │ │ │ │ FLOAT16, │ in your │ +│ │ │ │ │ INT32, or │ preprocessing │ +│ │ │ │ │ INT8. │ pipeline. │ +└──────────┴───────────────────────┴───────┴──────────┴───────────────┴───────────────┘ Estimated fix time: 15-30 minutes. +Address critical issues first; warnings can often wait. ``` `echo $?` → `1`. Exits non-zero so CI can fail the PR. @@ -84,6 +87,9 @@ $ trtcheck tests/fixtures/failing/uint8_input.onnx \ [uint8_input] promote input 'input' from UINT8 to FLOAT and drop the redundant Cast node 'cast_1' +verdict: blocked -> likely (TensorRT 10.3); 1 finding(s) resolved, +0 remaining, 0 introduced + 1 fix(es) applied. Wrote model_fixed.onnx. ``` @@ -98,12 +104,12 @@ What changed in the graph: Re-running trtcheck against the rewritten file: ``` -╭─────────────────── trtcheck report ───────────────────╮ -│ LIKELY TO CONVERT │ -│ file: model_fixed.onnx │ -│ opset: 17 producer: trtcheck-fixtures nodes: 1 │ -│ 0 critical 0 warning 0 info │ -╰───────────────────────────────────────────────────────╯ +╭──────────────────────── trtcheck report ────────────────────────╮ +│ LIKELY -- static analysis found no known blocker │ +│ file: model_fixed.onnx target: TensorRT 10.3 │ +│ opset: 17 producer: trtcheck-fixtures nodes: 1 │ +│ 0 critical 0 warning 0 info │ +╰─────────────────────────────────────────────────────────────────╯ No issues detected. ``` diff --git a/docs/design/analysis-verdicts-and-fix-safety.md b/docs/design/analysis-verdicts-and-fix-safety.md new file mode 100644 index 0000000..d219a52 --- /dev/null +++ b/docs/design/analysis-verdicts-and-fix-safety.md @@ -0,0 +1,147 @@ +# Design: analysis verdicts and fix safety + +Status: implemented (schema 2.0, unreleased). This document records the +invariants the verdict model and the fixer pipeline are built on, so future +changes can be checked against them. + +## Trust model + +trtcheck runs on **untrusted input** (arbitrary ONNX protobufs in CI) and +loads **semi-trusted extensions** (entry-point plugins the user installed). + +- Model-derived text (node names, producer strings, filenames) is sanitized + before reaching a terminal or an HTML document (`trtcheck/_text.py`, + reporter-level escaping). +- Built-in checker crashes propagate: they are trtcheck bugs and must fail + tests. Third-party checker crashes are isolated into a + `TRT-PLUGIN-CHECKER-ERROR` finding — a crashed checker means *missing + coverage*, which must be visible in the report, not swallowed. +- Third-party fixers get no more trust than built-ins: every fixer runs + inside the same transaction (below). +- Runtime verification shells out to `trtexec` with list-args (no shell), + a timeout, and captured/truncated output. + +## The verdict lattice + +A single boolean ("conversion likely") could not express the difference +between "we checked and found nothing" and "we could not check". Schema 2.0 +uses four states: + +``` +BLOCKED > UNVERIFIED > LIKELY < VERIFIED +``` + +- `BLOCKED`: >= 1 critical finding. Runtime success never overrides a + static blocker — a contradiction there means the matrix is wrong and must + be surfaced, not papered over. +- `UNVERIFIED`: no critical, but >= 1 finding with `verify_required=True`. + Sources: unclassified operators, custom domains without a declared + plugin, partial support, unresolvable conditional-support rules, + crashed plugin checkers. +- `LIKELY`: no critical, nothing unresolved. Static prediction only. +- `VERIFIED`: `LIKELY`/`UNVERIFIED` + a successful `trtexec` parse/build + for the user's actual environment. Only the runtime path sets it. +- A recorded runtime *failure* (parser or build) demotes an + otherwise-`LIKELY` report to `UNVERIFIED`; verification that merely could + not run (missing trtexec, timeout, spawn error) leaves the static verdict + untouched. + +**Invariant:** no operator disappears silently. Every node is either +classified by the matrix, covered by an explicit uncertainty finding +(`TRT-OP-UNCLASSIFIED` / `TRT-OP-CUSTOM-DOMAIN`), or explicitly declared +plugin-backed by the user (`--plugin-domain`). + +The deprecated `conversion_likely` boolean survives as +`verdict != BLOCKED` for 1.x JSON consumers. + +## Evidence and confidence + +Findings carry `confidence` (`high` = documented/tested, `medium` = +heuristic with known gaps, `low` = uncertainty marker) and matrix +conditional-support entries carry an `evidence` object +(`status`: `official_docs` | `empirically_verified` | `inferred` | +`unknown`, plus `source` URL and retrieval date). The drift checker +(`tools/check_matrix_drift.py`) compares the matrix against the upstream +onnx-tensorrt operator table on a schedule. + +## Conditional support (matrix schema 2.x) + +`operator_matrix.json` entries may carry `conditions`: + +- `attribute_allowed` — a node attribute must be in an allowed set + (`default_ok` covers the absent-attribute case); +- `constant_input_max` — an input must, when statically constant, be a + scalar int <= `max_value`; when it is runtime-dynamic the condition is + *unresolvable*, which produces an unverified finding rather than a pass + or a guess. + +Each condition evaluates to pass / violated (`TRT-OP-CONDITION`, severity +from the data) / unresolved (`TRT-OP-CONDITION-UNRESOLVED`, +`verify_required`). Unknown condition kinds (data newer than code) evaluate +to unresolved, never to pass. `applies_to` scopes a condition to the TRT +versions the evidence actually covers. + +## Transactional fixing + +`run_fixers()` (trtcheck/fixers/__init__.py) enforces: + +1. The input model is never mutated (outer deep copy). +2. Each fixer runs against a fresh deep copy of the **last valid model**. +3. A candidate is committed only if the fixer returned well-formed + `FixApplied` records *and* the candidate passes validation. +4. Any failure — exception (even after mutating), malformed return, + invalid candidate, undeclared mutation, or *claimed fixes with no actual + change* — discards the candidate and records a `FixFailure`. Later + fixers still run on the last valid state. + +### Validation levels + +A candidate is held to the strongest bar the *input* model meets: + +- `full` — `onnx.checker.check_model(full_check=True)` (strict type/shape + inference). This is the level that catches dtype rewrites the shallow + checker misses (the Reshape INT64 regression). +- `basic` — structural check only; used for external-data models (full + inference cannot read payloads from an in-memory proto) and inputs that + fail full inference for pre-existing reasons. +- `none` — input fails even the basic check. Library callers may still run + fixers; the CLI refuses to `--fix` such models. + +### Schema-aware INT64 conversion + +`Int64ToInt32Fixer` converts an initializer only when **every** use, across +all nested subgraphs, is at an input position whose ONNX type constraint +admits `tensor(int32)` *independently of other inputs/outputs* (allowlist: +Gather/GatherElements/ScatterElements `indices`, Cast/Shape/Size data +input). It refuses shadowed names, signature tensors (graph inputs and +outputs), custom-domain consumers, unknown positions (Reshape `shape`, +Slice `starts`, ...), overflow, and dead initializers. It never inserts +speculative Cast nodes to force a conversion through. + +### Dropout removal + +Removal requires provable inference mode: opset <= 6 `is_test=1`; +opset 7–11 unconditionally inference; opset >= 12 `training_mode` absent or +resolvable to a static scalar `False` (initializer or Constant, unambiguous +across scopes). True / dynamic / computed / ambiguous → refuse. + +## The --fix pipeline + +``` +analyze(target) -> run_fixers (transactional) -> validate -> re-analyze(same target) + -> diff findings by (rule_id, node_name, operator) -> resolved/remaining/introduced + -> write only a validated candidate, never over the input +``` + +Dry-run performs everything except the write. `--format json` emits the +full machine-readable summary. + +## Extension points + +- Checkers/fixers/reporters via entry points (`docs/design/plugin-sdk.md`). + New Issue fields all default, so 1.x-era plugin checkers keep working; + their findings simply carry empty `rule_id`. +- New conditional-support kinds: add an evaluator in + `checkers/operator_support.py`; unknown kinds are already fail-safe. +- Runtime verification is deliberately isolated in + `trtcheck/runtime_verify.py`; static analysis never imports TensorRT. diff --git a/docs/design/plugin-sdk.md b/docs/design/plugin-sdk.md index d2f9df0..f1c48e1 100644 --- a/docs/design/plugin-sdk.md +++ b/docs/design/plugin-sdk.md @@ -202,3 +202,20 @@ The v1.0 release blocks on all of: installable via `pip install -e .` and shows up in `--list-plugins`. - The CHANGELOG calls out the public API surface and the migration story. + +## Schema 2.0 compatibility (unreleased) + +- `Issue` gained `rule_id`, `confidence`, `verify_required`, `target_trt`, + and `graph_scope` — all with defaults, so existing plugin checkers keep + constructing `Issue` unchanged. Give your findings a namespaced rule id + (e.g. `MYPLUGIN-...`) so CI consumers can filter them; the `TRT-` prefix + is reserved for built-ins. +- Plugin **fixers** now run inside the transactional pipeline: you receive + a private candidate copy of the model; your changes are kept only if you + return a well-formed `list[FixApplied]` and the candidate passes ONNX + validation. Raising an exception can no longer corrupt the output model + — but it will be reported to the user, so still refuse rather than raise + where you can. Tracebacks are only shown with `TRTCHECK_DEBUG=1`. +- A crashed plugin **checker** now surfaces as a `TRT-PLUGIN-CHECKER-ERROR` + finding with `verify_required=True`, which makes the report verdict + `unverified` — missing coverage is visible, not silent. diff --git a/docs/fixers.md b/docs/fixers.md index f7a8a86..62b04a4 100644 --- a/docs/fixers.md +++ b/docs/fixers.md @@ -1,19 +1,49 @@ # Auto-fixers -`trtcheck --fix` runs a pipeline of conservative ONNX rewrites that target -the most common TensorRT conversion failures. Each fixer either applies -the rewrite cleanly or refuses; nothing is half-rewritten. +`trtcheck --fix` runs an **audited, transactional** pipeline of conservative +ONNX rewrites targeting common TensorRT conversion failures: + +1. analyze the input for the selected `--target-trt`; +2. run every fixer against an isolated deep-copy candidate — a fixer that + crashes, returns malformed records, mutates without declaring it, or + produces an ONNX-invalid model has its changes discarded (and the + failure reported), while later fixers still run; +3. validate the result with `onnx.checker.check_model(full_check=True)` + where the input supports it (external-data models fall back to the + basic check); +4. re-analyze with the same target and report findings **resolved / + remaining / introduced**, keyed by stable rule id + node identity; +5. write only a validated model, never over the input file. + +Nothing is ever half-rewritten, and no speculative `Cast` nodes are +inserted to make a fix "succeed". When a rewrite is not provably safe, +the fixer skips it (skips are logged at INFO level on the +`trtcheck.fixers` logger). + +See `docs/design/analysis-verdicts-and-fix-safety.md` for the invariants. ## Built-in fixers ### `int64_to_int32` -Casts INT64 initializers down to INT32. Refuses if any value is outside -the INT32 range. - -> **Why:** TensorRT casts INT64 to INT32 at engine build time anyway. -> Doing it at fix time surfaces overflow as a clear refusal rather than a -> silent build-time failure. +Casts an INT64 initializer to INT32 **only when every use of it — across +all nested subgraphs — is at an input position whose ONNX schema accepts +INT32 independently of the operator's other inputs**: `Gather` / +`GatherElements` / `ScatterElements` indices, and the data input of +`Cast` / `Shape` / `Size`. + +Refuses, among others: + +- consumers whose schema requires INT64 (`Reshape` shape, `Slice` + starts/ends/axes, `Pad` pads, `Tile` repeats, ...) — converting those + produces a model that passes the shallow checker but fails strict type + inference; +- elementwise consumers (`Add`, `Mul`, ...) — int32 is an allowed dtype + there, but the type variable binds both operands, so retyping one + breaks the model; +- initializers that shadow a graph input/output (signature change), + names defined in multiple scopes, custom-domain consumers, values + outside INT32 range, empty or unconsumed initializers. ### `float64_to_float32` @@ -24,45 +54,61 @@ empty initializers, or any value exceeding FP32 range. Promotes a `UINT8` graph input to `FLOAT` when its only consumer is a `Cast(to=FLOAT)`. Removes the redundant Cast and rewires downstream -nodes. - -> Refuses for any other UINT8 consumption pattern -- the right rewrite -> depends on what the model expected the UINT8 to mean (raw bytes, -> normalized image, indices). +nodes. Refuses any other UINT8 consumption pattern. ### `drop_dropout` -Removes Dropout nodes and rewires their data consumers. Refuses if the -Dropout's mask output is referenced anywhere. +Removes a Dropout node **only when it is provably in inference mode**: + +- opset >= 12: the optional `training_mode` input is absent, or resolves + to a static scalar `False` (initializer or Constant node, unambiguous + across scopes). `True`, runtime-fed, computed, or ambiguous + training_mode → the node is left alone; +- opset 7–11: Dropout has no training switch (inference = identity); +- opset <= 6: only with `is_test=1`. -> TensorRT folds Dropout out of the engine anyway; removing it up front -> keeps the diagnostic report and any visualisation tooling cleaner. +Also refuses when the mask output is referenced anywhere, when the data +output is captured by another scope, or when rewiring would change the +graph signature. ### `upsample_to_resize` Rewrites leftover deprecated `Upsample` ops to `Resize` (mode `nearest` -or `linear`) on opset-13+ graphs -- the shape some exporters still emit -even though the op stopped being legal after opset 9. Refuses below -opset 13: the 4-input Resize form with empty `roi`/`sizes` placeholders -only validates from 13, so a conformant opset-9 model needs a -whole-model opset bump (`onnx.version_converter`) first, then a re-run. +or `linear`) on opset-13+ graphs. Refuses below opset 13 (the 4-input +Resize form only validates from 13; run +`onnx.version_converter.convert_version` first). ## How to invoke ```bash -# preview what would change -trtcheck model.onnx --fix --dry-run --output model.fixed.onnx +# preview what would change (nothing is written) +trtcheck model.onnx --fix --dry-run -# actually write the fixed file +# write the fixed file, report before/after findings trtcheck model.onnx --fix --output model.fixed.onnx + +# machine-readable fix summary +trtcheck model.onnx --fix --output model.fixed.onnx --format json ``` -`--fix` requires `--output` (unless `--dry-run` is set) and refuses to -overwrite the input file. Use `--force` to overwrite an existing output. +`--fix` requires `--output` (unless `--dry-run`), refuses to overwrite the +input file, refuses structurally invalid input models, and honors +`--target-trt` for both the before and after analysis. Use `--force` to +overwrite an existing output. Exit code follows the *fixed* model's +verdict (`--fail-on` applies). + +## Third-party fixers + +Plugin fixers (entry point `trtcheck.fixers`) run inside the same +transaction as built-ins: a crashing or misbehaving plugin is reported on +stderr and cannot corrupt the output model. `TRTCHECK_DEBUG=1` includes +the traceback. ## Adding a fixer Each fixer lives in `trtcheck/fixers/`, implements the `Fixer` protocol (`fix(model) -> list[FixApplied]`), and is registered in -`default_fixers()`. The TDD discipline is the same as the checkers: -write a failing test first, then implement the minimum that passes. +`default_fixers()`. Contract: mutate the model you are handed (the +pipeline gives you a private candidate copy), return one `FixApplied` per +change, and skip anything not unambiguously safe. Write the failing test +first. diff --git a/docs/index.md b/docs/index.md index cbd8c96..0a7bbfe 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,6 +21,8 @@ invoking `trtexec`. - [Install](install.md) -- `pip install trtcheck` - [Usage](usage.md) -- CLI flags, examples, CI integration - [Fixers](fixers.md) -- what `--fix` rewrites and when it refuses +- [Rule registry](rules.md) -- stable finding ids for CI filtering +- [Design: verdicts & fix safety](design/analysis-verdicts-and-fix-safety.md) -- invariants and trust model - [Operators](operators/index.md) -- per-operator TensorRT support matrix ## What it checks diff --git a/docs/operators/Clip.md b/docs/operators/Clip.md new file mode 100644 index 0000000..0581504 --- /dev/null +++ b/docs/operators/Clip.md @@ -0,0 +1,14 @@ +# Clip + +## TensorRT support + +| Version | Status | +| --- | --- | +| 8.0 | unknown | +| 8.6 | unknown | +| 10.0 | supported | +| 10.3 | supported | + +## Notes + +Supported in TRT 10.x (FP32/FP16/BF16) per onnx-tensorrt docs, retrieved 2026-07-22. 8.x status not verified here, left unknown. diff --git a/docs/operators/Resize.md b/docs/operators/Resize.md index cca1ea4..60b85bf 100644 --- a/docs/operators/Resize.md +++ b/docs/operators/Resize.md @@ -11,8 +11,9 @@ ## Notes -Only nearest and linear modes pre-10.0. Cubic added in 10.0. +Only nearest and linear modes; cubic is not supported (onnx-tensorrt docs, retrieved 2026-07-22). ## Limitations -- antialias attribute not supported before TRT 10.0. +- Antialiasing (antialias=1) is not supported. +- coordinate_transformation_mode limited to half_pixel, pytorch_half_pixel, tf_half_pixel_for_nn, asymmetric, align_corners. diff --git a/docs/operators/index.md b/docs/operators/index.md index 68ff76d..6ab5dbe 100644 --- a/docs/operators/index.md +++ b/docs/operators/index.md @@ -2,7 +2,7 @@ Per-operator TensorRT support against versions 8.0, 8.6, 10.0, 10.3. -`100` operators tracked. +`101` operators tracked. - [Abs](Abs.md) - [Add](Add.md) @@ -12,6 +12,7 @@ Per-operator TensorRT support against versions 8.0, 8.6, 10.0, 10.3. - [AveragePool](AveragePool.md) - [BatchNormalization](BatchNormalization.md) - [Cast](Cast.md) +- [Clip](Clip.md) - [Concat](Concat.md) - [Constant](Constant.md) - [ConstantOfShape](ConstantOfShape.md) diff --git a/docs/rules.md b/docs/rules.md new file mode 100644 index 0000000..2fe7782 --- /dev/null +++ b/docs/rules.md @@ -0,0 +1,69 @@ +# Rule registry + +Every finding trtcheck emits carries a stable, machine-readable `rule_id`. +Renaming or removing an id is a breaking change guarded by +`tests/test_verdicts.py::test_rule_id_registry_is_stable`; additions are +backward-compatible. + +Per-issue metadata (JSON schema 2.0): `rule_id`, `severity`, `category`, +`node_name`, `operator`, `graph_scope` (owning graph, where known), +`target_trt`, `confidence` (`high` / `medium` / `low`), `verify_required` +(true when the finding needs runtime or manual verification), `remediation`, +`docs_link`. + +## Operator support + +| Rule | Severity | Meaning | +|---|---|---| +| `TRT-OP-UNSUPPORTED` | critical | Operator documented as not supported for the target TRT version. | +| `TRT-OP-PARTIAL` | warning | Partial support with documented limitations; verify against your export. | +| `TRT-OP-UNCLASSIFIED` | info (verify) | Operator not classified in trtcheck's matrix — no evidence either way. One finding per op type. | +| `TRT-OP-CUSTOM-DOMAIN` | info (verify) | Custom-domain op; needs a TensorRT plugin. Suppress with `--plugin-domain`. | +| `TRT-OP-CONDITION` | per-condition | A documented conditional-support rule is violated (e.g. TopK `sorted=0`, Resize `mode=cubic`). | +| `TRT-OP-CONDITION-UNRESOLVED` | info (verify) | A conditional-support rule cannot be settled statically (e.g. runtime-dynamic TopK `K`). | + +## Precision / dtype + +| Rule | Severity | +|---|---| +| `TRT-DTYPE-UINT8-INPUT` | critical | +| `TRT-DTYPE-STRING` | critical | +| `TRT-DTYPE-FP64` | warning | +| `TRT-DTYPE-INT64-INPUT` | warning | +| `TRT-DTYPE-INT64-WEIGHTS` | warning | +| `TRT-DTYPE-BF16` | warning (verify) | + +## Shapes and control flow + +| Rule | Severity | +|---|---| +| `TRT-SHAPE-PROFILE-MISSING` | warning (verify) | +| `TRT-CONTROL-LOOP-RUNTIME-TRIP` | critical | +| `TRT-CONTROL-LOOP-DYNAMIC-TRIP` | warning (verify) | +| `TRT-CONTROL-LOOP-NESTED` | critical | +| `TRT-CONTROL-IF-SHAPE-MISMATCH` | critical | +| `TRT-CONTROL-IF-UNVERIFIED` | warning (verify) | +| `TRT-CONTROL-SCAN` | warning (verify) | + +## Graph structure + +| Rule | Severity | +|---|---| +| `TRT-GRAPH-NO-OUTPUT` | critical | +| `TRT-GRAPH-INPUT-UNTYPED` | critical | +| `TRT-GRAPH-DUP-NODE-NAME` | warning | +| `TRT-GRAPH-ISOLATED-NODE` | warning | +| `TRT-GRAPH-LARGE-CONSTANT` | info | +| `TRT-GRAPH-EXTERNAL-DATA` | critical | +| `TRT-GRAPH-ALIASING` | warning | +| `TRT-OPSET-OLD` | info | + +## Plugins + +| Rule | Severity | +|---|---| +| `TRT-PLUGIN-CHECKER-ERROR` | warning (verify) — a third-party checker crashed; its coverage is missing from this report. | + +Severities for remediation-DB rules come from +`trtcheck/data/remediation_db.json`; the table above summarizes, the JSON is +authoritative. diff --git a/docs/usage.md b/docs/usage.md index 65c9175..01e71fd 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -6,8 +6,26 @@ trtcheck model.onnx ``` -Exit code is `1` if conversion is unlikely, `0` otherwise. Wire that into -CI to catch regressions at PR time. +trtcheck is a **static** analyzer: it predicts, it does not guarantee. Every +report carries one of four verdicts: + +| Verdict | Meaning | +|---|---| +| `blocked` | At least one known-critical incompatibility for the target TensorRT version. | +| `unverified` | No known blocker, but unresolved conditions remain: operators the support matrix does not classify, custom-domain ops that need a TensorRT plugin, or conditional support that cannot be settled statically (e.g. a runtime-dynamic TopK `K`). | +| `likely` | Every static check passed with nothing unresolved. Still a prediction — say "static analysis found no known blocker", never "guaranteed to convert". | +| `verified` | An optional real TensorRT build (`--verify-runtime`, requires `trtexec`) parsed the model and built an engine in *your* environment. | + +## Exit codes + +| Code | When | +|---|---| +| `0` | Verdict is `likely`, `verified`, or (by default) `unverified` | +| `1` | Verdict is `blocked`; with `--fail-on unverified`, also on `unverified`. Also fatal CLI errors. | +| `2` | Usage errors (bad flags/arguments) | + +The exit code is computed from the **full** report: `--severity` trims the +displayed findings but never upgrades the verdict. ## Common flags @@ -15,21 +33,27 @@ CI to catch regressions at PR time. # target a specific TensorRT version trtcheck model.onnx --target-trt 8.6 -# machine-readable output for CI +# machine-readable output for CI (schema 2.0: verdict, rule_id, confidence) trtcheck model.onnx --format json --output report.json +# strict CI gate: also fail on unresolved/unverified conditions +trtcheck model.onnx --fail-on unverified + +# declare that a TensorRT plugin implements a custom domain +trtcheck model.onnx --plugin-domain com.mycompany.ops + # self-contained HTML report trtcheck model.onnx --format html --output report.html -# filter to blockers only -trtcheck model.onnx --severity critical - # compare two ONNX files trtcheck before.onnx after.onnx --diff --format html --output diff.html -# auto-fix simple cases (INT64 indices, UINT8 inputs after Cast, etc.) -trtcheck model.onnx --fix --dry-run --output fixed.onnx +# safe auto-fix (see docs/fixers.md): preview, then write +trtcheck model.onnx --fix --dry-run trtcheck model.onnx --fix --output fixed.onnx + +# optional runtime verification with a real TensorRT build +trtcheck model.onnx --verify-runtime --verify-timeout 900 ``` ## All flags @@ -39,16 +63,76 @@ trtcheck model.onnx --fix --output fixed.onnx | `--target-trt` | `10.3` | TensorRT version to check against | | `--format` | `console` | `console`, `json`, `html`, or a plugin reporter name | | `--output` | (stdout) | Write the report to this path | -| `--severity` | `info` | Minimum severity to include | +| `--severity` | `info` | Minimum severity to *display* (never changes the exit code) | +| `--fail-on` | `blocked` | `blocked` or `unverified`: which verdict fails the run | +| `--plugin-domain DOMAIN` | (none) | Declare a custom domain as plugin-backed; repeatable | | `--diff` | off | Compare two ONNX files | | `--force` | off | Allow `--output` to overwrite existing files | | `--max-model-size` | `500` | Refuse to load ONNX files larger than this (MB) | -| `--fix` | off | Apply built-in and plugin fixers; writes to `--output` | +| `--fix` | off | Run the audited fix pipeline; writes to `--output` | | `--dry-run` | off | With `--fix`, print changes without writing | +| `--verify-runtime` | off | Run `trtexec --onnx=MODEL` after static analysis | +| `--trtexec PATH` | (PATH lookup) | Explicit trtexec executable | +| `--verify-timeout` | `600` | Seconds before the trtexec run is killed | | `--list-plugins` | off | Print discovered checkers, fixers, and reporters, then exit | | `--disable-plugin NAME` | (none) | Exclude a checker, fixer, or reporter by name; repeatable | | `-h`, `--help` | | Full CLI reference | +Set `TRTCHECK_DEBUG=1` to include tracebacks for third-party plugin +failures (hidden by default). + +## Examples + +### A model with a custom operator + +```text +$ trtcheck detector.onnx +│ UNVERIFIED -- no known blocker, unresolved conditions remain │ +... +│ INFO │ TRT-OP-CUSTOM-DOMAIN │ <3 nodes> │ com.acme::DeformConv │ ... +$ echo $? # 0 by default +$ trtcheck detector.onnx --fail-on unverified; echo $? # 1 +$ trtcheck detector.onnx --plugin-domain com.acme # finding suppressed +``` + +### Safe fixing + +```text +$ trtcheck model.onnx --fix --output fixed.onnx + [int64_to_int32] cast initializer 'indices' from INT64 to INT32 ... + [drop_dropout] removed Dropout node 'drop' ... + +verdict: blocked -> likely (TensorRT 10.3); 2 finding(s) resolved, 0 remaining, 0 introduced + +2 fix(es) applied. Wrote fixed.onnx. +``` + +Machine-readable summary: add `--format json` to the `--fix` invocation. + +### Runtime verification (needs TensorRT + usually a GPU) + +```text +$ trtcheck model.onnx --verify-runtime +runtime verification: success -- trtexec parsed the model and built an engine +│ VERIFIED -- TensorRT runtime build succeeded │ +``` + +If `trtexec` is missing, times out, or cannot be spawned, the static +verdict is kept. If trtexec *ran and failed* (parser or engine-build +failure), an otherwise-`likely` report is demoted to `unverified` — runtime +evidence against the model is never hidden behind a clean static +prediction. Either way, the metadata (status, command, version, output +tails) is recorded in the JSON report under `runtime_verification`. + +## JSON schema + +Reports are schema `2.0`. Every 1.x key is still present (including the +deprecated boolean `conversion_likely`); new keys include `schema_version`, +`verdict`, `target_trt`, `runtime_verified`, `runtime_verification`, and +per-issue `rule_id` / `confidence` / `verify_required` / `target_trt` / +`graph_scope`. Filter CI on `rule_id` — the registry in +[docs/rules.md](rules.md) is covered by a stability test. + ## CI integration A composite GitHub Action ships in this repo. See diff --git a/pyproject.toml b/pyproject.toml index 02c6264..43a9d75 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "trtcheck" -version = "1.0.0" +version = "1.1.0" description = "Static pre-flight checker for ONNX -> TensorRT conversion." readme = "README.md" requires-python = ">=3.10" diff --git a/scripts/package-smoke.sh b/scripts/package-smoke.sh new file mode 100755 index 0000000..dd079dc --- /dev/null +++ b/scripts/package-smoke.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# Package smoke test: install the built wheel into a FRESH venv outside the +# repository and exercise the public surface from there. Catches packaging +# bugs editable installs hide (missing data files, broken entry points). +# +# Usage: scripts/package-smoke.sh [path-to-wheel] +# Default wheel: the newest dist/trtcheck-*.whl in the repo. +set -euo pipefail + +REPO="$(cd "$(dirname "$0")/.." && pwd)" +WHEEL="${1:-$(ls -t "$REPO"/dist/trtcheck-*.whl | head -1)}" +[ -f "$WHEEL" ] && WHEEL="$(readlink -f "$WHEEL")" + +WORK="$(mktemp -d "${TMPDIR:-/tmp}/trtcheck-smoke.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT +cd "$WORK" # everything below runs OUTSIDE the repository + +python3 -m venv venv +# Keep the smoke honest: no repo paths, no dev env leakage. +env -u PYTHONPATH -u AMENT_PREFIX_PATH ./venv/bin/pip -q install "$WHEEL" + +run() { env -u PYTHONPATH -u AMENT_PREFIX_PATH "$@"; } + +echo "== import + version" +run ./venv/bin/python -c "import trtcheck; print('trtcheck', trtcheck.__version__)" + +echo "== packaged data files" +run ./venv/bin/python - <<'PY' +from importlib import resources +import json +for name in ("operator_matrix.json", "remediation_db.json"): + data = json.loads(resources.files("trtcheck.data").joinpath(name).read_text()) + assert data.get("schema_version"), name +print("data files load: ok") +PY + +echo "== console entry point --help" +run ./venv/bin/trtcheck --help > /dev/null +run ./venv/bin/python -m trtcheck --version + +echo "== generate a model and analyze (console + json)" +run ./venv/bin/python - <<'PY' +import numpy as np, onnx +from onnx import TensorProto, helper, numpy_helper +inp = helper.make_tensor_value_info("input", TensorProto.FLOAT, [10, 4]) +out = helper.make_tensor_value_info("output", TensorProto.FLOAT, [3, 4]) +idx = numpy_helper.from_array(np.array([0, 1, 2], dtype=np.int64), name="indices") +gather = helper.make_node("Gather", ["input", "indices"], ["g"], name="g0", axis=0) +drop = helper.make_node("Dropout", ["g"], ["d"], name="drop") +ident = helper.make_node("Identity", ["d"], ["output"], name="ident") +graph = helper.make_graph([gather, drop, ident], "m", [inp], [out], initializer=[idx]) +model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) +model.ir_version = 8 +onnx.save(model, "smoke.onnx") +print("wrote smoke.onnx") +PY +run ./venv/bin/trtcheck smoke.onnx | head -5 +run ./venv/bin/trtcheck smoke.onnx --format json --output report.json +run ./venv/bin/python - <<'PY' +import json +report = json.load(open("report.json")) +assert report["schema_version"] == "2.0" +assert report["verdict"] in ("blocked", "unverified", "likely", "verified") +assert all(i["rule_id"] for i in report["issues"]) +print("json report: ok, verdict =", report["verdict"]) +PY + +echo "== safe fix mode" +run ./venv/bin/trtcheck smoke.onnx --fix --output fixed.onnx | tail -3 +run ./venv/bin/python -c "import onnx; onnx.checker.check_model(onnx.load('fixed.onnx'), full_check=True); print('fixed model fully valid')" + +echo "== missing-verifier behavior (empty PATH)" +run env PATH="" ./venv/bin/trtcheck smoke.onnx --verify-runtime --format json --output verify.json 2> verify.err || true +grep -q '"status": "missing_trtexec"' verify.json +echo "missing trtexec handled: ok" + +echo +echo "PACKAGE SMOKE: PASS ($WHEEL)" diff --git a/scripts/real-smoke-container.sh b/scripts/real-smoke-container.sh new file mode 100755 index 0000000..c488b39 --- /dev/null +++ b/scripts/real-smoke-container.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# Run the real-TensorRT smoke corpus inside an official NGC TensorRT +# container, using the INSTALLED trtcheck wheel (never editable mode). +# +# GPU access strategy: prefers the NVIDIA Container Toolkit (--gpus all) +# when Docker advertises the nvidia runtime; otherwise falls back to +# manual passthrough (--device /dev/nvidia* + read-only mounts of the +# driver's user-space libraries), which requires no root and changes +# nothing on the host. +# +# Usage: scripts/real-smoke-container.sh [IMAGE] [WHEEL] +# IMAGE default: nvcr.io/nvidia/tensorrt:24.08-py3 (TensorRT 10.3 — +# matches the repository's 10.3 support target) +# WHEEL default: newest dist/trtcheck-*.whl +set -euo pipefail + +REPO="$(cd "$(dirname "$0")/.." && pwd)" +IMAGE="${1:-nvcr.io/nvidia/tensorrt:24.08-py3}" +WHEEL="${2:-$(ls -t "$REPO"/dist/trtcheck-*.whl | head -1)}" +WHEEL="$(readlink -f "$WHEEL")" +OUT="$(mktemp -d "${TMPDIR:-/tmp}/trtcheck-realsmoke.XXXXXX")" +trap 'rm -rf "$OUT/nvlibs"' EXIT +echo "image: $IMAGE" +echo "wheel: $WHEEL" +echo "outdir: $OUT" + +GPU_ARGS=() +if docker info 2>/dev/null | grep -q "Runtimes:.*nvidia"; then + GPU_ARGS+=(--gpus all) +else + echo "nvidia runtime not configured -- using manual GPU passthrough" + mkdir -p "$OUT/nvlibs" + # Only the driver-side libraries the CUDA/TensorRT stack dlopens; the + # container keeps its own CUDA runtime. Read-only. + ( cd /usr/lib/x86_64-linux-gnu && cp -a \ + libcuda.so* libcudadebugger.so* libnvidia-ml.so* libnvidia-cfg.so* \ + libnvidia-nvvm.so* libnvidia-ptxjitcompiler.so* libnvidia-gpucomp.so* \ + "$OUT/nvlibs/" 2>/dev/null ) + for dev in /dev/nvidia0 /dev/nvidiactl /dev/nvidia-uvm /dev/nvidia-uvm-tools; do + [ -e "$dev" ] && GPU_ARGS+=(--device "$dev") + done + GPU_ARGS+=(-v "$OUT/nvlibs:/nvlibs:ro" -e LD_LIBRARY_PATH=/nvlibs) + # NGC images do not ship nvidia-smi; the toolkit normally injects it. + [ -x /usr/bin/nvidia-smi ] && GPU_ARGS+=(-v /usr/bin/nvidia-smi:/usr/bin/nvidia-smi:ro) +fi + +SQUEEZENET_ARG="" +if [ -f "$REPO/bench/cache/squeezenet1_1.onnx" ]; then + SQUEEZENET_ARG="--squeezenet /repo/bench/cache/squeezenet1_1.onnx" +fi + +docker run --rm "${GPU_ARGS[@]}" \ + -v "$REPO:/repo:ro" \ + -v "$WHEEL:/wheel/$(basename "$WHEEL"):ro" \ + -v "$OUT:/out" \ + -w /out \ + "$IMAGE" \ + bash -lc " + set -euo pipefail + nvidia-smi --query-gpu=name,driver_version --format=csv,noheader + TRTEXEC=\$(command -v trtexec || ls /usr/src/tensorrt/bin/trtexec /opt/tensorrt/bin/trtexec 2>/dev/null | head -1) + echo \"trtexec: \$TRTEXEC\" + \$TRTEXEC --version 2>&1 | tail -1 || true + pip install --quiet /wheel/*.whl + trtcheck --version + python3 /repo/scripts/real_tensorrt_smoke.py \ + --trtexec \"\$TRTEXEC\" --fixtures /repo/tests/fixtures --out /out \ + $SQUEEZENET_ARG + " +echo +echo "results: $OUT/real_tensorrt_smoke_results.json" diff --git a/scripts/real_tensorrt_smoke.py b/scripts/real_tensorrt_smoke.py new file mode 100755 index 0000000..4dc4b75 --- /dev/null +++ b/scripts/real_tensorrt_smoke.py @@ -0,0 +1,274 @@ +#!/usr/bin/env python3 +"""Real-TensorRT smoke runner: the bounded 7-model corpus, executed against a +genuine trtexec. + +Designed to run INSIDE a TensorRT container (or any machine with trtexec) +with the *installed* trtcheck wheel — never editable mode. It runs, per +model: static analysis, ``trtcheck --verify-runtime``, and an independent +direct ``trtexec`` invocation, then records agreement. The dynamic-shape +fixture additionally gets a with-profiles trtexec leg. + +Usage (see scripts/real-smoke-container.sh for the container wrapper): + + real_tensorrt_smoke.py --trtexec /path/to/trtexec \ + --fixtures /tests/fixtures --out /out [--timeout 600] + +Writes ``real_tensorrt_smoke_results.json`` into --out. Engines are never +saved (--saveEngine is not passed); all scratch stays under --out. +""" + +from __future__ import annotations + +import argparse +import json +import re +import shutil +import subprocess +import sys +import time +from pathlib import Path + +_TAIL = 1500 + +# (name, fixture-relative path, expected direct-trtexec outcome) +CORPUS = [ + ("clean_minimal", "clean_minimal.onnx", "build_success"), + ("squeezenet1_1_public", "SQUEEZENET", "build_success"), + ("sequence_empty", "failing/sequence_empty.onnx", "parser_failure"), + # TensorRT 10.3 trtexec does NOT fail on a dynamic model without shape + # flags: it warns and auto-overrides every unspecified dynamic dim to 1 + # (observed 2026-07-22; version-specific tool behavior). The explicit + # with-profile leg below covers the real dynamic path. + ("fully_dynamic", "failing/fully_dynamic.onnx", "build_success"), + ("custom_domain", "custom_domain.onnx", "parser_failure"), + ("uint8_fixed_via_fix", "FIXED", "build_success"), + ("reshape_int64_shape", "reshape_int64_shape.onnx", "build_success"), +] + +# Explicit optimization profile for the fully_dynamic fixture +# (input 'input', rank 4, all dims symbolic). +DYN_PROFILE = [ + "--minShapes=input:1x1x8x8", + "--optShapes=input:1x3x64x64", + "--maxShapes=input:2x3x128x128", +] + + +def _run(cmd: list[str], timeout: int) -> dict: + start = time.monotonic() + try: + proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False) + rc: int | None = proc.returncode + out, err, note = proc.stdout, proc.stderr, "" + except subprocess.TimeoutExpired: + rc, out, err, note = None, "", "", f"timeout after {timeout}s" + return { + "command": " ".join(cmd), + "returncode": rc, + "elapsed_s": round(time.monotonic() - start, 2), + # Full stdout is kept for JSON parsing but never written to the + # results file; records carry only the bounded tails. + "stdout_full": out, + "stdout_tail": out[-_TAIL:], + "stderr_tail": err[-_TAIL:], + "note": note, + } + + +def _classify_trtexec(res: dict) -> str: + if res["returncode"] is None: + return "timeout" + combined = (res["stdout_tail"] + res["stderr_tail"]).lower() + if res["returncode"] == 0: + return "build_success" + parser_markers = ( + "failed to parse onnx", + "modelimporter", + "onnx2trt", + "could not parse", + "in function importmodel", + "invalidnode", + "getplugincreator could not find plugin", + ) + if any(m in combined for m in parser_markers): + return "parser_failure" + return "build_failure" + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--trtexec", required=True) + ap.add_argument("--fixtures", type=Path, required=True) + ap.add_argument("--out", type=Path, required=True) + ap.add_argument("--target-trt", default="10.3") + ap.add_argument("--timeout", type=int, default=600) + ap.add_argument( + "--squeezenet", + type=Path, + default=None, + help="Path to a cached public squeezenet ONNX (skipped if absent).", + ) + args = ap.parse_args() + args.out.mkdir(parents=True, exist_ok=True) + + trtcheck = shutil.which("trtcheck") + assert trtcheck, "installed trtcheck console script not found on PATH" + ver = _run([args.trtexec, "--version"], 60) + # trtexec --version exits non-zero on some builds; the banner still + # carries "[TensorRT vNNNNN]". + match = re.search(r"TensorRT v\d+", ver["stdout_full"] + ver["stderr_tail"]) + tensorrt_version = match.group(0) if match else "unknown" + + # Produce the --fix corpus entry with the installed wheel. + fixed = args.out / "uint8_fixed.onnx" + fixed.unlink(missing_ok=True) + fix = _run( + [ + trtcheck, + str(args.fixtures / "failing/uint8_input.onnx"), + "--fix", + "--output", + str(fixed), + "--format", + "json", + ], + args.timeout, + ) + assert fixed.exists(), f"--fix did not write the fixed model: {fix['stderr_tail']}" + + # Reshape regression: the int64 fixer must refuse. + refuse = subprocess.run( + [ + trtcheck, + str(args.fixtures / "reshape_int64_shape.onnx"), + "--fix", + "--dry-run", + "--format", + "json", + ], + capture_output=True, + text=True, + check=False, + ) + refuse_fixes = [f["fixer"] for f in json.loads(refuse.stdout)["fixes_applied"]] + assert "int64_to_int32" not in refuse_fixes, "int64 fixer must refuse Reshape shape input" + + results: dict = { + "generated_by": "scripts/real_tensorrt_smoke.py", + "tensorrt_version_line": tensorrt_version, + "target_trt": args.target_trt, + "reshape_int64_fixer_refusal": {"fixes_applied": refuse_fixes}, + "models": [], + } + + for name, rel, expected in CORPUS: + if rel == "FIXED": + model = fixed + elif rel == "SQUEEZENET": + if not args.squeezenet or not args.squeezenet.exists(): + results["models"].append( + {"name": name, "skipped": "no cached public model provided"} + ) + continue + model = args.squeezenet + else: + model = args.fixtures / rel + + static = _run( + [trtcheck, str(model), "--target-trt", args.target_trt, "--format", "json"], + args.timeout, + ) + static_report = ( + json.loads(static["stdout_full"]) if static["returncode"] is not None else {} + ) + + verify = _run( + [ + trtcheck, + str(model), + "--target-trt", + args.target_trt, + "--verify-runtime", + "--trtexec", + args.trtexec, + "--verify-timeout", + str(args.timeout), + "--format", + "json", + ], + args.timeout + 60, + ) + verify_report = ( + json.loads(verify["stdout_full"]) if verify["returncode"] is not None else {} + ) + rv = verify_report.get("runtime_verification") or {} + + direct = _run([args.trtexec, f"--onnx={model}"], args.timeout) + direct_outcome = _classify_trtexec(direct) + + wrapper_status = rv.get("status", "missing") + # Agreement: the wrapper's classification must match the independent run. + agree = { + "build_success": wrapper_status == "success", + "parser_failure": wrapper_status == "parser_failure", + "build_failure": wrapper_status == "build_failure", + "timeout": wrapper_status == "timeout", + }.get(direct_outcome, False) + + entry = { + "name": name, + "model": model.name, + "expected_direct_outcome": expected, + "static_verdict": static_report.get("verdict"), + "static_rule_ids": sorted({i["rule_id"] for i in static_report.get("issues", [])}), + "verify_runtime": { + "status": wrapper_status, + "verdict_after": verify_report.get("verdict"), + "runtime_verified": verify_report.get("runtime_verified"), + "elapsed_s": verify["elapsed_s"], + }, + "direct_trtexec": { + "outcome": direct_outcome, + "returncode": direct["returncode"], + "elapsed_s": direct["elapsed_s"], + "diagnostic": (direct["stderr_tail"] or direct["stdout_tail"])[-400:], + }, + "wrapper_agrees_with_direct": agree, + "matched_expectation": direct_outcome == expected, + } + + if name == "fully_dynamic": + with_profile = _run([args.trtexec, f"--onnx={model}", *DYN_PROFILE], args.timeout) + entry["direct_trtexec_with_profile"] = { + "outcome": _classify_trtexec(with_profile), + "returncode": with_profile["returncode"], + "elapsed_s": with_profile["elapsed_s"], + "profile": " ".join(DYN_PROFILE), + } + + results["models"].append(entry) + print( + f"{name:22s} static={entry['static_verdict']!s:10s} " + f"wrapper={wrapper_status:15s} direct={direct_outcome:15s} " + f"agree={agree} expected_ok={entry['matched_expectation']}" + ) + + out_file = args.out / "real_tensorrt_smoke_results.json" + out_file.write_text(json.dumps(results, indent=2) + "\n") + print(f"wrote {out_file}") + + ran = [m for m in results["models"] if "skipped" not in m] + successes = [m for m in ran if m["direct_trtexec"]["outcome"] == "build_success"] + failures = [ + m for m in ran if m["direct_trtexec"]["outcome"] in ("parser_failure", "build_failure") + ] + disagreements = [m["name"] for m in ran if not m["wrapper_agrees_with_direct"]] + print( + f"\nsummary: {len(ran)} run, {len(successes)} genuine builds, " + f"{len(failures)} genuine failures, disagreements: {disagreements or 'none'}" + ) + return 1 if disagreements or not successes or not failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/fixtures/custom_domain.onnx b/tests/fixtures/custom_domain.onnx new file mode 100644 index 0000000..1f3aa7f Binary files /dev/null and b/tests/fixtures/custom_domain.onnx differ diff --git a/tests/fixtures/failing/topk_unsorted.onnx b/tests/fixtures/failing/topk_unsorted.onnx new file mode 100644 index 0000000..9ec66e4 Binary files /dev/null and b/tests/fixtures/failing/topk_unsorted.onnx differ diff --git a/tests/fixtures/generate_broken.py b/tests/fixtures/generate_broken.py index 229df77..ef695c8 100644 --- a/tests/fixtures/generate_broken.py +++ b/tests/fixtures/generate_broken.py @@ -218,11 +218,65 @@ def create_control_flow_loop() -> onnx.ModelProto: return _make_model(graph) +# -- New corpus entries (verdict-model era) ----------------------------------- + + +def create_topk_unsorted() -> onnx.ModelProto: + """TopK with sorted=0: TensorRT 10.x rejects it (onnx-tensorrt docs). + + Exercises the conditional-support rule TRT-OP-CONDITION. + """ + x = helper.make_tensor_value_info("x", TensorProto.FLOAT, [100]) + vals = helper.make_tensor_value_info("vals", TensorProto.FLOAT, [3]) + idxs = helper.make_tensor_value_info("idxs", TensorProto.INT64, [3]) + k = numpy_helper.from_array(np.array([3], dtype=np.int64), name="k") + topk = helper.make_node("TopK", ["x", "k"], ["vals", "idxs"], name="topk_0", axis=0, sorted=0) + graph = helper.make_graph([topk], "topk_unsorted", [x], [vals, idxs], initializer=[k]) + return _make_model(graph) + + +def create_custom_domain() -> onnx.ModelProto: + """A custom-domain op with no TensorRT plugin declared. + + trtcheck must report this as UNVERIFIED (rule TRT-OP-CUSTOM-DOMAIN), not + silently clean and not a hard blocker. Without a plugin, a real trtexec + parse fails, so the corpus labels it expected: fail. + """ + x = helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 4]) + y = helper.make_tensor_value_info("y", TensorProto.FLOAT, [1, 4]) + node = helper.make_node("FancyCustomOp", ["x"], ["y"], name="c0", domain="com.example") + graph = helper.make_graph([node], "custom_domain", [x], [y]) + model = helper.make_model( + graph, + producer_name="trtcheck-fixtures", + opset_imports=[helper.make_opsetid("", OPSET), helper.make_opsetid("com.example", 1)], + ) + model.ir_version = IR_VERSION + return model + + +def create_reshape_int64_shape() -> onnx.ModelProto: + """Valid model whose INT64 initializer is Reshape's shape input. + + The P0 fixer-safety regression: converting that initializer to INT32 + breaks the ONNX schema. trtcheck --fix must leave it alone, and the + analyzer must not report the model as blocked. + """ + x = helper.make_tensor_value_info("x", TensorProto.FLOAT, [2, 6]) + y = helper.make_tensor_value_info("y", TensorProto.FLOAT, [3, 4]) + shape = numpy_helper.from_array(np.array([3, 4], dtype=np.int64), name="new_shape") + reshape = helper.make_node("Reshape", ["x", "new_shape"], ["y"], name="reshape_0") + graph = helper.make_graph([reshape], "reshape_int64_shape", [x], [y], initializer=[shape]) + return _make_model(graph) + + # -- Driver ------------------------------------------------------------------ _CLEAN: dict[str, callable] = { "clean_minimal.onnx": create_clean_minimal, + "custom_domain.onnx": create_custom_domain, + "reshape_int64_shape.onnx": create_reshape_int64_shape, } _FAILING: dict[str, callable] = { @@ -231,6 +285,7 @@ def create_control_flow_loop() -> onnx.ModelProto: "fully_dynamic.onnx": create_fully_dynamic, "uint8_input.onnx": create_uint8_input, "control_flow_loop.onnx": create_control_flow_loop, + "topk_unsorted.onnx": create_topk_unsorted, } diff --git a/tests/fixtures/reshape_int64_shape.onnx b/tests/fixtures/reshape_int64_shape.onnx new file mode 100644 index 0000000..3098296 Binary files /dev/null and b/tests/fixtures/reshape_int64_shape.onnx differ diff --git a/tests/test_bench_predict.py b/tests/test_bench_predict.py index 8f1accb..4d9b019 100644 --- a/tests/test_bench_predict.py +++ b/tests/test_bench_predict.py @@ -24,6 +24,12 @@ def test_resolve_bundled_entry_is_repo_relative() -> None: def test_verdict_mapping() -> None: + # Schema 2.x: four-state verdict maps to three outcome buckets. + assert verdict_from_report({"verdict": "blocked"}) == "fail" + assert verdict_from_report({"verdict": "unverified"}) == "unverified" + assert verdict_from_report({"verdict": "likely"}) == "convert" + assert verdict_from_report({"verdict": "verified"}) == "convert" + # Schema 1.x fallback. assert verdict_from_report({"conversion_likely": True}) == "convert" assert verdict_from_report({"conversion_likely": False}) == "fail" diff --git a/tests/test_bench_score.py b/tests/test_bench_score.py index 6133038..46ae0e5 100644 --- a/tests/test_bench_score.py +++ b/tests/test_bench_score.py @@ -146,3 +146,62 @@ def test_cli_errors_on_bad_outcomes_shape(self, tmp_path: Path) -> None: outcomes_path.write_text(json.dumps({"predictions": []})) # list, not dict rc = main(["--manifest", str(manifest_path), "--outcomes", str(outcomes_path)]) assert rc == 2 + + +class TestUnverifiedPredictions: + def test_unverified_is_never_counted_as_success(self) -> None: + manifest = _manifest(("real_fail", "fail"), ("real_convert", "convert")) + outcomes = { + "real_fail": {"trtcheck": "unverified"}, + "real_convert": {"trtcheck": "unverified"}, + } + r = score(manifest, outcomes) + # Nothing lands in the confusion matrix -- and nothing reads as a pass. + assert r.total == 0 + assert r.unverified_on_fail == ["real_fail"] + assert r.unverified_on_convert == ["real_convert"] + assert r.unverified_total == 2 + assert r.unverified_coverage == 1.0 + + def test_unverified_coverage_mixes_with_classified(self) -> None: + manifest = _manifest(("a", "fail"), ("b", "convert"), ("c", "fail")) + outcomes = { + "a": {"trtcheck": "fail"}, + "b": {"trtcheck": "convert"}, + "c": {"trtcheck": "unverified"}, + } + r = score(manifest, outcomes) + assert r.true_positive == 1 and r.true_negative == 1 + assert r.unverified_coverage == pytest.approx(1 / 3) + assert "unverified" in format_report(r) + + def test_bogus_prediction_value_still_raises(self) -> None: + manifest = _manifest(("a", "fail")) + with pytest.raises(ValueError): + score(manifest, {"a": {"trtcheck": "maybe"}}) + + +def test_score_to_dict_and_json_flag(tmp_path: Path) -> None: + manifest = _manifest(("a", "fail"), ("b", "convert"), ("c", "fail")) + outcomes = { + "a": {"trtcheck": "fail"}, + "b": {"trtcheck": "convert"}, + "c": {"trtcheck": "unverified"}, + } + r = score(manifest, outcomes) + d = r.to_dict() + assert d["blocker_precision"] == 1.0 + assert d["blocker_recall"] == 1.0 + assert d["unverified_on_fail"] == ["c"] + assert d["unverified_coverage"] == pytest.approx(1 / 3) + + # CLI round trip + mpath = tmp_path / "manifest.yaml" + opath = tmp_path / "outcomes.json" + jpath = tmp_path / "summary.json" + import yaml + + mpath.write_text(yaml.safe_dump({"models": manifest})) + opath.write_text(json.dumps({"predictions": outcomes})) + assert main(["--manifest", str(mpath), "--outcomes", str(opath), "--json", str(jpath)]) == 0 + assert json.loads(jpath.read_text())["scored"] == 2 diff --git a/tests/test_cli_fix.py b/tests/test_cli_fix.py new file mode 100644 index 0000000..e703cbc --- /dev/null +++ b/tests/test_cli_fix.py @@ -0,0 +1,174 @@ +"""CLI tests for the audited --fix pipeline and verdict-driven exit codes.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import numpy as np +import onnx +import pytest +from click.testing import CliRunner +from onnx import TensorProto, helper, numpy_helper + +from trtcheck.cli import main + + +@pytest.fixture() +def runner() -> CliRunner: + return CliRunner() + + +def _save(model: onnx.ModelProto, path: Path) -> Path: + onnx.save(model, str(path)) + return path + + +def _fixable_model() -> onnx.ModelProto: + """Gather with INT64 indices (safe to convert) plus an inference Dropout.""" + inp = helper.make_tensor_value_info("input", TensorProto.FLOAT, [10, 4]) + out = helper.make_tensor_value_info("output", TensorProto.FLOAT, [3, 4]) + idx = numpy_helper.from_array(np.array([0, 1, 2], dtype=np.int64), name="indices") + gather = helper.make_node("Gather", ["input", "indices"], ["g"], name="g0", axis=0) + drop = helper.make_node("Dropout", ["g"], ["d"], name="drop") + ident = helper.make_node("Identity", ["d"], ["output"], name="ident") + graph = helper.make_graph([gather, drop, ident], "m", [inp], [out], initializer=[idx]) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + return model + + +def _clean_model() -> onnx.ModelProto: + inp = helper.make_tensor_value_info("input", TensorProto.FLOAT, [1, 4]) + out = helper.make_tensor_value_info("output", TensorProto.FLOAT, [1, 4]) + relu = helper.make_node("Relu", ["input"], ["output"], name="r") + graph = helper.make_graph([relu], "m", [inp], [out]) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + return model + + +class TestFixPipeline: + def test_successful_fix_writes_and_reports_deltas( + self, runner: CliRunner, tmp_path: Path + ) -> None: + src = _save(_fixable_model(), tmp_path / "in.onnx") + dst = tmp_path / "out.onnx" + result = runner.invoke(main, [str(src), "--fix", "--output", str(dst)]) + assert result.exit_code == 0, result.output + assert dst.exists() + assert "int64_to_int32" in result.output + assert "drop_dropout" in result.output + assert "verdict:" in result.output and "resolved" in result.output + fixed = onnx.load(str(dst)) + onnx.checker.check_model(fixed, full_check=True) + assert not any(n.op_type == "Dropout" for n in fixed.graph.node) + + def test_fix_uses_selected_target(self, runner: CliRunner, tmp_path: Path) -> None: + src = _save(_fixable_model(), tmp_path / "in.onnx") + dst = tmp_path / "out.onnx" + result = runner.invoke( + main, [str(src), "--fix", "--target-trt", "8.6", "--output", str(dst)] + ) + assert result.exit_code == 0, result.output + assert "TensorRT 8.6" in result.output + + def test_fix_json_summary_is_machine_readable(self, runner: CliRunner, tmp_path: Path) -> None: + src = _save(_fixable_model(), tmp_path / "in.onnx") + dst = tmp_path / "out.onnx" + result = runner.invoke(main, [str(src), "--fix", "--format", "json", "--output", str(dst)]) + assert result.exit_code == 0, result.output + payload = json.loads(result.stdout) + assert payload["target_trt"] == "10.3" + assert payload["schema_version"] == "2.0" + assert {f["fixer"] for f in payload["fixes_applied"]} >= {"int64_to_int32"} + assert isinstance(payload["resolved"], list) + assert isinstance(payload["remaining"], list) + assert isinstance(payload["introduced"], list) + assert payload["verdict_before"] in ("blocked", "unverified", "likely") + assert payload["verdict_after"] in ("blocked", "unverified", "likely") + + def test_dry_run_writes_nothing(self, runner: CliRunner, tmp_path: Path) -> None: + src = _save(_fixable_model(), tmp_path / "in.onnx") + result = runner.invoke(main, [str(src), "--fix", "--dry-run"]) + assert result.exit_code == 0, result.output + assert "dry run" in result.output + assert list(tmp_path.glob("*.onnx")) == [src] + + def test_noop_fix_reports_and_writes_nothing(self, runner: CliRunner, tmp_path: Path) -> None: + src = _save(_clean_model(), tmp_path / "in.onnx") + dst = tmp_path / "out.onnx" + result = runner.invoke(main, [str(src), "--fix", "--output", str(dst)]) + assert result.exit_code == 0, result.output + assert "no fixes applied" in result.output + assert not dst.exists() + + def test_invalid_input_model_is_refused(self, runner: CliRunner, tmp_path: Path) -> None: + model = _clean_model() + model.graph.node[0].input[0] = "missing_tensor" + src = _save(model, tmp_path / "bad.onnx") + result = runner.invoke(main, [str(src), "--fix", "--dry-run"]) + assert result.exit_code != 0 + assert "failed ONNX validation" in result.output + + def test_crashing_plugin_fixer_cannot_corrupt_output( + self, runner: CliRunner, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + from trtcheck import cli as cli_mod + from trtcheck.fixers import FixApplied + + class EvilFixer: + name = "evil" + + def fix(self, model: onnx.ModelProto) -> list[FixApplied]: + for node in model.graph.node: + node.op_type = "Bogus" + raise RuntimeError("kaboom") + + monkeypatch.setattr(cli_mod, "load_plugins", lambda: ([], [EvilFixer()], [])) + src = _save(_fixable_model(), tmp_path / "in.onnx") + dst = tmp_path / "out.onnx" + result = runner.invoke(main, [str(src), "--fix", "--output", str(dst)]) + assert result.exit_code == 0, result.output + assert "kaboom" in result.output or "evil" in result.output # warned on stderr + fixed = onnx.load(str(dst)) + assert not any(n.op_type == "Bogus" for n in fixed.graph.node) + onnx.checker.check_model(fixed, full_check=True) + + def test_refuses_to_overwrite_input(self, runner: CliRunner, tmp_path: Path) -> None: + src = _save(_fixable_model(), tmp_path / "in.onnx") + result = runner.invoke(main, [str(src), "--fix", "--output", str(src)]) + assert result.exit_code != 0 + assert "refusing to overwrite the input file" in result.output + + +class TestExitCodes: + def test_unverified_model_exits_zero_by_default( + self, runner: CliRunner, tmp_path: Path + ) -> None: + model = _clean_model() + model.graph.node[0].op_type = "TotallyNovelOp" + src = _save(model, tmp_path / "m.onnx") + result = runner.invoke(main, [str(src)]) + assert result.exit_code == 0, result.output + assert "UNVERIFIED" in result.output + + def test_fail_on_unverified_exits_one(self, runner: CliRunner, tmp_path: Path) -> None: + model = _clean_model() + model.graph.node[0].op_type = "TotallyNovelOp" + src = _save(model, tmp_path / "m.onnx") + result = runner.invoke(main, [str(src), "--fail-on", "unverified"]) + assert result.exit_code == 1 + + def test_severity_filter_does_not_change_exit_code( + self, runner: CliRunner, tmp_path: Path + ) -> None: + """--severity trims the display only; hiding the unverified INFO + finding must not flip a --fail-on unverified failure into a pass.""" + model = _clean_model() + model.graph.node[0].op_type = "TotallyNovelOp" + src = _save(model, tmp_path / "m.onnx") + result = runner.invoke( + main, [str(src), "--severity", "critical", "--fail-on", "unverified"] + ) + assert result.exit_code == 1 diff --git a/tests/test_conditions.py b/tests/test_conditions.py new file mode 100644 index 0000000..10532b2 --- /dev/null +++ b/tests/test_conditions.py @@ -0,0 +1,108 @@ +"""Conditional-support rule evaluation (matrix schema 2.x `conditions`).""" + +from __future__ import annotations + +import numpy as np +import onnx +from onnx import TensorProto, helper, numpy_helper + +from trtcheck.checkers.operator_support import ( + RULE_CONDITION, + RULE_CONDITION_UNRESOLVED, + OperatorSupportChecker, +) +from trtcheck.types import Confidence, Severity + + +def _topk_model(*, sorted_attr: int | None = None, k_value: int | None = 3, k_dynamic=False): + inputs = [helper.make_tensor_value_info("x", TensorProto.FLOAT, [100])] + inits = [] + if k_dynamic: + inputs.append(helper.make_tensor_value_info("k", TensorProto.INT64, [1])) + else: + inits.append(numpy_helper.from_array(np.array([k_value], dtype=np.int64), name="k")) + kwargs = {} if sorted_attr is None else {"sorted": sorted_attr} + topk = helper.make_node("TopK", ["x", "k"], ["vals", "idxs"], name="tk", axis=0, **kwargs) + outs = [ + onnx.ValueInfoProto(name="vals"), + onnx.ValueInfoProto(name="idxs"), + ] + graph = helper.make_graph([topk], "m", inputs, outs, initializer=inits) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + return model + + +def _resize_model(**attrs): + x = helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 3, 4, 4]) + scales = numpy_helper.from_array( + np.array([1.0, 1.0, 2.0, 2.0], dtype=np.float32), name="scales" + ) + node = helper.make_node("Resize", ["x", "", "scales"], ["y"], name="rz", **attrs) + graph = helper.make_graph( + [node], "m", [x], [onnx.ValueInfoProto(name="y")], initializer=[scales] + ) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + return model + + +def _issues(model, target="10.3"): + return OperatorSupportChecker(target_trt=target).check(model) + + +class TestTopKConditions: + def test_default_sorted_and_small_constant_k_pass(self) -> None: + issues = _issues(_topk_model()) + assert all(i.rule_id not in (RULE_CONDITION,) for i in issues) + + def test_sorted_zero_is_a_violation(self) -> None: + issues = _issues(_topk_model(sorted_attr=0)) + hit = next(i for i in issues if i.rule_id == RULE_CONDITION) + assert hit.severity is Severity.CRITICAL + assert "sorted" in hit.message + assert hit.confidence is Confidence.HIGH # official-docs evidence + assert hit.docs_link and "onnx-tensorrt" in hit.docs_link + + def test_k_above_limit_is_a_violation(self) -> None: + issues = _issues(_topk_model(k_value=5000)) + hit = next(i for i in issues if i.rule_id == RULE_CONDITION) + assert "3840" in hit.message + assert hit.severity is Severity.CRITICAL + + def test_dynamic_k_is_unresolved_not_violated(self) -> None: + issues = _issues(_topk_model(k_dynamic=True)) + assert all(i.rule_id != RULE_CONDITION for i in issues) + hit = next(i for i in issues if i.rule_id == RULE_CONDITION_UNRESOLVED) + assert hit.severity is Severity.INFO + assert hit.verify_required is True + + def test_conditions_do_not_apply_to_other_targets(self) -> None: + # The evidence tracks TRT 10.x; an 8.6 target must not inherit it. + issues = _issues(_topk_model(sorted_attr=0), target="8.6") + assert all(i.rule_id not in (RULE_CONDITION, RULE_CONDITION_UNRESOLVED) for i in issues) + + +class TestResizeConditions: + def test_cubic_mode_is_a_violation(self) -> None: + issues = _issues(_resize_model(mode="cubic")) + hit = next(i for i in issues if i.rule_id == RULE_CONDITION) + assert hit.severity is Severity.CRITICAL + assert "cubic" in hit.message or "mode" in hit.message + + def test_nearest_mode_passes(self) -> None: + issues = _issues(_resize_model(mode="nearest")) + assert all(i.rule_id != RULE_CONDITION for i in issues) + + def test_antialias_is_a_violation(self) -> None: + issues = _issues(_resize_model(mode="linear", antialias=1)) + assert any(i.rule_id == RULE_CONDITION and "antialias" in i.message for i in issues) + + def test_unsupported_coord_transform_is_a_violation(self) -> None: + issues = _issues( + _resize_model(mode="linear", coordinate_transformation_mode="tf_crop_and_resize") + ) + assert any( + i.rule_id == RULE_CONDITION and "coordinate_transformation_mode" in i.message + for i in issues + ) diff --git a/tests/test_data_files.py b/tests/test_data_files.py index 34c06ec..2ee4bab 100644 --- a/tests/test_data_files.py +++ b/tests/test_data_files.py @@ -29,7 +29,7 @@ def remediation() -> dict: class TestOperatorMatrix: def test_top_level_fields(self, matrix: dict) -> None: - assert matrix["schema_version"] == "1.0" + assert matrix["schema_version"] == "2.0" assert set(matrix["target_trt_versions"]) == _EXPECTED_VERSIONS assert isinstance(matrix["operators"], dict) @@ -64,7 +64,7 @@ def test_sequence_ops_uniformly_unsupported(self, matrix: dict) -> None: class TestRemediationDb: def test_top_level_fields(self, remediation: dict) -> None: - assert remediation["schema_version"] == "1.0" + assert remediation["schema_version"] == "2.0" assert isinstance(remediation["issues"], dict) def test_minimum_entry_count(self, remediation: dict) -> None: diff --git a/tests/test_entry_points.py b/tests/test_entry_points.py index 09cb3f0..6223bb3 100644 --- a/tests/test_entry_points.py +++ b/tests/test_entry_points.py @@ -45,4 +45,4 @@ def test_python_m_trtcheck_runs_against_clean_fixture() -> None: cwd=str(_REPO), ) assert result.returncode == 0, result.stderr - assert "convert" in result.stdout.lower() + assert "no known blocker" in result.stdout.lower() diff --git a/tests/test_fixers_dropout.py b/tests/test_fixers_dropout.py index 6ddef9b..b1bcd19 100644 --- a/tests/test_fixers_dropout.py +++ b/tests/test_fixers_dropout.py @@ -96,3 +96,110 @@ def test_dropout_feeding_graph_output(self) -> None: def test_clean_model_emits_no_fixes(self, clean_model: onnx.ModelProto) -> None: _, applied = apply_all(clean_model, [DropDropoutFixer()]) assert applied == [] + + +def _dropout12(training_value=None, training_input=None, opset: int = 17): + """Dropout with explicit ratio + training_mode inputs (opset 12+ form). + + training_value: None (no third input), True/False (bool initializer), or + the string "dynamic" (wired to a graph input) / "computed" (wired to a + non-Constant node output). + """ + import numpy as np + from onnx import numpy_helper + + inputs = [helper.make_tensor_value_info("input", TensorProto.FLOAT, [1, 4])] + inits = [] + nodes = [] + drop_inputs = ["input"] + if training_value is not None or training_input is not None: + ratio = numpy_helper.from_array(np.array(0.5, dtype=np.float32), name="ratio") + inits.append(ratio) + drop_inputs += ["ratio"] + if training_input == "dynamic": + inputs.append(helper.make_tensor_value_info("tm", TensorProto.BOOL, [])) + drop_inputs += ["tm"] + elif training_input == "computed": + nodes.append(helper.make_node("Not", ["flag"], ["tm"], name="mk_tm")) + inputs.append(helper.make_tensor_value_info("flag", TensorProto.BOOL, [])) + drop_inputs += ["tm"] + else: + tm = numpy_helper.from_array(np.array(training_value, dtype=bool), name="tm") + inits.append(tm) + drop_inputs += ["tm"] + nodes.append(helper.make_node("Dropout", drop_inputs, ["d"], name="drop")) + nodes.append(helper.make_node("Identity", ["d"], ["output"], name="ident")) + graph = helper.make_graph( + nodes, + "m", + inputs, + [helper.make_tensor_value_info("output", TensorProto.FLOAT, [1, 4])], + initializer=inits, + ) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", opset)]) + model.ir_version = 8 + return model + + +class TestDropoutTrainingMode: + def test_absent_training_mode_is_removed(self) -> None: + _new, applied = apply_all(_dropout12(), [DropDropoutFixer()]) + assert len(applied) == 1 + + def test_static_false_training_mode_is_removed(self) -> None: + new_model, applied = apply_all(_dropout12(training_value=False), [DropDropoutFixer()]) + assert len(applied) == 1 + assert not any(n.op_type == "Dropout" for n in new_model.graph.node) + onnx.checker.check_model(new_model, full_check=True) + + def test_static_true_training_mode_is_kept(self) -> None: + new_model, applied = apply_all(_dropout12(training_value=True), [DropDropoutFixer()]) + assert applied == [] + assert any(n.op_type == "Dropout" for n in new_model.graph.node) + + def test_dynamic_training_mode_is_kept(self) -> None: + _new, applied = apply_all(_dropout12(training_input="dynamic"), [DropDropoutFixer()]) + assert applied == [] + + def test_computed_training_mode_is_kept(self) -> None: + _new, applied = apply_all(_dropout12(training_input="computed"), [DropDropoutFixer()]) + assert applied == [] + + def test_constant_node_false_training_mode_is_removed(self) -> None: + cst = helper.make_node( + "Constant", + [], + ["tm"], + name="cst", + value=helper.make_tensor("tmv", TensorProto.BOOL, [], [False]), + ) + import numpy as np + from onnx import numpy_helper + + ratio = numpy_helper.from_array(np.array(0.5, dtype=np.float32), name="ratio") + drop = helper.make_node("Dropout", ["input", "ratio", "tm"], ["d"], name="drop") + ident = helper.make_node("Identity", ["d"], ["output"], name="ident") + graph = helper.make_graph( + [cst, drop, ident], + "m", + [helper.make_tensor_value_info("input", TensorProto.FLOAT, [1, 4])], + [helper.make_tensor_value_info("output", TensorProto.FLOAT, [1, 4])], + initializer=[ratio], + ) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + _new, applied = apply_all(model, [DropDropoutFixer()]) + assert len(applied) == 1 + + def test_opset6_is_test_zero_is_kept(self) -> None: + drop = helper.make_node("Dropout", ["input"], ["d"], name="drop", is_test=0, ratio=0.5) + ident = helper.make_node("Identity", ["d"], ["output"], name="ident") + graph = helper.make_graph( + [drop, ident], + "m", + [helper.make_tensor_value_info("input", TensorProto.FLOAT, [1, 4])], + [helper.make_tensor_value_info("output", TensorProto.FLOAT, [1, 4])], + ) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 6)]) + _new, applied = apply_all(model, [DropDropoutFixer()]) + assert applied == [] diff --git a/tests/test_fixers_int64_schema.py b/tests/test_fixers_int64_schema.py new file mode 100644 index 0000000..396c255 --- /dev/null +++ b/tests/test_fixers_int64_schema.py @@ -0,0 +1,190 @@ +"""Schema-awareness regression tests for Int64ToInt32Fixer. + +The headline regression: ONNX ``Reshape`` requires its ``shape`` input to be +INT64. The old fixer converted every in-range INT64 initializer, producing a +model that passes the shallow ``onnx.checker.check_model()`` but fails +``full_check=True`` (strict type inference). These tests prove the failure +mode exists and that the use-aware fixer refuses it. +""" + +from __future__ import annotations + +import numpy as np +import onnx +import pytest +from onnx import TensorProto, helper, numpy_helper + +from trtcheck.fixers import apply_all +from trtcheck.fixers.int64_to_int32 import Int64ToInt32Fixer + + +def _reshape_model() -> onnx.ModelProto: + """A valid model whose INT64 initializer is Reshape's shape input.""" + inp = helper.make_tensor_value_info("x", TensorProto.FLOAT, [2, 6]) + out = helper.make_tensor_value_info("y", TensorProto.FLOAT, [3, 4]) + shape = numpy_helper.from_array(np.array([3, 4], dtype=np.int64), name="new_shape") + reshape = helper.make_node("Reshape", ["x", "new_shape"], ["y"], name="r") + graph = helper.make_graph([reshape], "m", [inp], [out], initializer=[shape]) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + return model + + +class TestReshapeRegression: + def test_original_model_is_fully_valid(self) -> None: + onnx.checker.check_model(_reshape_model(), full_check=True) + + def test_blind_conversion_breaks_full_validation(self) -> None: + """Proof of the failure mode: manually cast the shape tensor to INT32 + (what the old fixer did). The shallow check passes; full type + inference rejects the model.""" + model = _reshape_model() + init = model.graph.initializer[0] + arr = numpy_helper.to_array(init) + init.CopyFrom(numpy_helper.from_array(arr.astype(np.int32), name=init.name)) + + onnx.checker.check_model(model) # shallow check: no complaint + with pytest.raises(Exception): + onnx.checker.check_model(model, full_check=True) + + def test_fixer_refuses_reshape_shape_input(self) -> None: + model = _reshape_model() + fixed, applied = apply_all(model, [Int64ToInt32Fixer()]) + assert applied == [] + init = next(i for i in fixed.graph.initializer if i.name == "new_shape") + assert init.data_type == TensorProto.INT64 + onnx.checker.check_model(fixed, full_check=True) + + +class TestMixedAndNestedUses: + def test_shared_initializer_with_mixed_consumers_is_refused(self) -> None: + """One initializer feeding both a safe position (Gather indices) and an + unsafe one (Reshape shape): every use must be safe, so refuse.""" + inp = helper.make_tensor_value_info("x", TensorProto.FLOAT, [4, 4]) + out_g = helper.make_tensor_value_info("g_out", TensorProto.FLOAT, [2, 4]) + out_r = helper.make_tensor_value_info("r_out", TensorProto.FLOAT, [2, 8]) + shared = numpy_helper.from_array(np.array([2, 8], dtype=np.int64), name="shared") + gather = helper.make_node("Gather", ["x", "shared"], ["g_out"], name="g", axis=0) + reshape = helper.make_node("Reshape", ["x", "shared"], ["r_out"], name="r") + graph = helper.make_graph( + [gather, reshape], "m", [inp], [out_g, out_r], initializer=[shared] + ) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + onnx.checker.check_model(model, full_check=True) + + fixed, applied = apply_all(model, [Int64ToInt32Fixer()]) + assert applied == [] + assert fixed.graph.initializer[0].data_type == TensorProto.INT64 + + def test_outer_initializer_used_unsafely_inside_subgraph_is_refused(self) -> None: + """A top-level INT64 initializer captured by a Reshape inside an If + branch: the nested use must be seen and must veto the conversion.""" + shape64 = numpy_helper.from_array(np.array([4], dtype=np.int64), name="cap_shape") + then_g = helper.make_graph( + [helper.make_node("Reshape", ["data", "cap_shape"], ["t_out"], name="rs")], + "then_body", + [], + [helper.make_tensor_value_info("t_out", TensorProto.FLOAT, [4])], + ) + else_g = helper.make_graph( + [helper.make_node("Identity", ["data"], ["e_out"], name="eid")], + "else_body", + [], + [helper.make_tensor_value_info("e_out", TensorProto.FLOAT, [2, 2])], + ) + cond = helper.make_node( + "Constant", + [], + ["cond"], + name="cond", + value=helper.make_tensor("c", TensorProto.BOOL, [], [True]), + ) + ifnode = helper.make_node( + "If", ["cond"], ["if_out"], name="i", then_branch=then_g, else_branch=else_g + ) + vi = onnx.ValueInfoProto() + vi.name = "if_out" + g = helper.make_graph( + [cond, ifnode], + "root", + [helper.make_tensor_value_info("data", TensorProto.FLOAT, [2, 2])], + [vi], + initializer=[shape64], + ) + model = helper.make_model(g, opset_imports=[helper.make_opsetid("", 17)]) + + _fixed, applied = apply_all(model, [Int64ToInt32Fixer()]) + assert applied == [], "nested Reshape use must veto the conversion" + + def test_custom_domain_consumer_is_refused(self) -> None: + idx = numpy_helper.from_array(np.array([0, 1], dtype=np.int64), name="idx") + node = helper.make_node("MyOp", ["idx"], ["y"], name="c", domain="com.example") + vi = onnx.ValueInfoProto() + vi.name = "y" + g = helper.make_graph([node], "m", [], [vi], initializer=[idx]) + model = helper.make_model( + g, + opset_imports=[helper.make_opsetid("", 17), helper.make_opsetid("com.example", 1)], + ) + _fixed, applied = apply_all(model, [Int64ToInt32Fixer()]) + assert applied == [] + + def test_shadowed_name_across_scopes_is_refused(self) -> None: + """The same name defined as an initializer at two scopes: which one a + nested consumer sees is scope-dependent, so the fixer must refuse.""" + outer = numpy_helper.from_array(np.array([0], dtype=np.int64), name="dup") + inner = numpy_helper.from_array(np.array([1], dtype=np.int64), name="dup") + then_g = helper.make_graph( + [helper.make_node("Gather", ["data", "dup"], ["t_out"], name="g2", axis=0)], + "then_body", + [], + [helper.make_tensor_value_info("t_out", TensorProto.FLOAT, [1])], + initializer=[inner], + ) + else_g = helper.make_graph( + [helper.make_node("Identity", ["data"], ["e_out"], name="eid")], + "else_body", + [], + [helper.make_tensor_value_info("e_out", TensorProto.FLOAT, [2])], + ) + cond = helper.make_node( + "Constant", + [], + ["cond"], + name="cond", + value=helper.make_tensor("c", TensorProto.BOOL, [], [True]), + ) + ifnode = helper.make_node( + "If", ["cond"], ["if_out"], name="i", then_branch=then_g, else_branch=else_g + ) + gather = helper.make_node("Gather", ["data", "dup"], ["g_out"], name="g1", axis=0) + vi = onnx.ValueInfoProto() + vi.name = "if_out" + vo = onnx.ValueInfoProto() + vo.name = "g_out" + g = helper.make_graph( + [gather, cond, ifnode], + "root", + [helper.make_tensor_value_info("data", TensorProto.FLOAT, [2])], + [vi, vo], + initializer=[outer], + ) + model = helper.make_model(g, opset_imports=[helper.make_opsetid("", 17)]) + + _fixed, applied = apply_all(model, [Int64ToInt32Fixer()]) + assert applied == [] + + def test_unconsumed_initializer_is_refused(self) -> None: + dead = numpy_helper.from_array(np.array([1], dtype=np.int64), name="dead") + ident = helper.make_node("Identity", ["x"], ["y"], name="id") + g = helper.make_graph( + [ident], + "m", + [helper.make_tensor_value_info("x", TensorProto.FLOAT, [1])], + [helper.make_tensor_value_info("y", TensorProto.FLOAT, [1])], + initializer=[dead], + ) + model = helper.make_model(g, opset_imports=[helper.make_opsetid("", 17)]) + _fixed, applied = apply_all(model, [Int64ToInt32Fixer()]) + assert applied == [] diff --git a/tests/test_fixers_roundtrip.py b/tests/test_fixers_roundtrip.py index 49b025f..56eb879 100644 --- a/tests/test_fixers_roundtrip.py +++ b/tests/test_fixers_roundtrip.py @@ -89,21 +89,21 @@ def _float64_initializer_shadows_input() -> onnx.ModelProto: return model -def test_int64_fixer_updates_shadowed_input_dtype() -> None: +def test_int64_fixer_refuses_initializer_shadowing_graph_input() -> None: + """An initializer that also appears in graph.input is part of the model's + public signature. The schema-aware fixer refuses to retype it (converting + would silently change the input contract for callers feeding that input).""" model = _int64_initializer_shadows_input() # Premise: the input model is genuinely valid under full type inference. assert_valid_model(model) fixed, applied = apply_all(model, default_fixers()) - assert [a.fixer for a in applied] == ["int64_to_int32"] - - # The initializer was cast... - new_init = next(i for i in fixed.graph.initializer if i.name == "wi") - assert new_init.data_type == TensorProto.INT32 - # ...and the shadowing graph input must be retyped to match, or full type - # inference rejects the model. - new_input = next(i for i in fixed.graph.input if i.name == "wi") - assert new_input.type.tensor_type.elem_type == TensorProto.INT32 + assert all(a.fixer != "int64_to_int32" for a in applied) + + unchanged = next(i for i in fixed.graph.initializer if i.name == "wi") + assert unchanged.data_type == TensorProto.INT64 + still_input = next(i for i in fixed.graph.input if i.name == "wi") + assert still_input.type.tensor_type.elem_type == TensorProto.INT64 assert_valid_model(fixed) diff --git a/tests/test_fixers_subgraph.py b/tests/test_fixers_subgraph.py index dc720a8..07af911 100644 --- a/tests/test_fixers_subgraph.py +++ b/tests/test_fixers_subgraph.py @@ -35,9 +35,11 @@ def test_empty_int64_initializer_does_not_crash() -> None: def test_int64_initializer_inside_if_branch_is_fixed() -> None: + """Descent proof: an INT64 initializer buried in an If branch, used only at + an INT32-compatible position (Gather indices), is converted.""" buried = numpy_helper.from_array(np.array([1, 2, 3], dtype=np.int64), name="buried_idx") then_g = helper.make_graph( - [helper.make_node("Identity", ["buried_idx"], ["t_out"], name="id")], + [helper.make_node("Gather", ["data", "buried_idx"], ["t_out"], name="id", axis=0)], "then_body", [], [_undef_out("t_out")], @@ -67,12 +69,17 @@ def test_int64_initializer_inside_if_branch_is_fixed() -> None: ifnode = helper.make_node( "If", ["cond"], ["if_out"], name="i", then_branch=then_g, else_branch=else_g ) - g = helper.make_graph([cond, ifnode], "root", [], [_undef_out("if_out")]) + g = helper.make_graph( + [cond, ifnode], + "root", + [helper.make_tensor_value_info("data", TensorProto.FLOAT, [10])], + [_undef_out("if_out")], + ) model = helper.make_model(g, opset_imports=[helper.make_opsetid("", 17)]) new_model, applied = apply_all(model, [Int64ToInt32Fixer()]) - assert applied, "fixer must descend into the If branch and cast the buried INT64 weight" + assert applied, "fixer must descend into the If branch and cast the buried INT64 indices" # locate the buried initializer in the rewritten model's then_branch if_node = next(n for n in new_model.graph.node if n.op_type == "If") then_branch = next(a.g for a in if_node.attribute if a.name == "then_branch") @@ -84,7 +91,7 @@ def test_default_fixer_pipeline_handles_subgraphs_without_error() -> None: """The full --fix pipeline must run cleanly over a model with subgraphs.""" buried = numpy_helper.from_array(np.array([4, 5], dtype=np.int64), name="bidx") then_g = helper.make_graph( - [helper.make_node("Identity", ["bidx"], ["t_out"], name="id")], + [helper.make_node("Gather", ["data", "bidx"], ["t_out"], name="id", axis=0)], "then_body", [], [_undef_out("t_out")], @@ -114,7 +121,12 @@ def test_default_fixer_pipeline_handles_subgraphs_without_error() -> None: ifnode = helper.make_node( "If", ["cond"], ["if_out"], name="i", then_branch=then_g, else_branch=else_g ) - g = helper.make_graph([cond, ifnode], "root", [], [_undef_out("if_out")]) + g = helper.make_graph( + [cond, ifnode], + "root", + [helper.make_tensor_value_info("data", TensorProto.FLOAT, [10])], + [_undef_out("if_out")], + ) model = helper.make_model(g, opset_imports=[helper.make_opsetid("", 17)]) _new, applied = apply_all(model, default_fixers()) # must not raise diff --git a/tests/test_fixers_transactional.py b/tests/test_fixers_transactional.py new file mode 100644 index 0000000..787b59b --- /dev/null +++ b/tests/test_fixers_transactional.py @@ -0,0 +1,166 @@ +"""Transactional guarantees of the fixer pipeline (run_fixers). + +A fixer -- built-in or third-party -- must not be able to leave a partial +mutation in the output model, no matter how it fails: raising mid-mutation, +returning malformed records, mutating without declaring it, or producing an +ONNX-invalid model. +""" + +from __future__ import annotations + +import numpy as np +import onnx +from onnx import TensorProto, helper, numpy_helper + +from trtcheck.fixers import FixApplied, run_fixers +from trtcheck.fixers.drop_dropout import DropDropoutFixer + + +def _valid_model() -> onnx.ModelProto: + inp = helper.make_tensor_value_info("x", TensorProto.FLOAT, [3]) + out = helper.make_tensor_value_info("y", TensorProto.FLOAT, [3]) + relu = helper.make_node("Relu", ["x"], ["mid"], name="relu") + drop = helper.make_node("Dropout", ["mid"], ["d"], name="drop") + ident = helper.make_node("Identity", ["d"], ["y"], name="ident") + graph = helper.make_graph([relu, drop, ident], "m", [inp], [out]) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + return model + + +class _MutateThenCrash: + """Adversarial fixer: renames every node, THEN raises.""" + + name = "mutate_then_crash" + + def fix(self, model: onnx.ModelProto) -> list[FixApplied]: + for node in model.graph.node: + node.name = "CORRUPTED_" + node.name + model.graph.node[0].op_type = "TotallyBogusOp" + raise RuntimeError("boom after mutation") + + +class _MalformedReturn: + name = "malformed_return" + + def fix(self, model: onnx.ModelProto) -> object: + model.graph.node[0].name = "MUTATED_ANYWAY" + return {"not": "a list of FixApplied"} + + +class _UndeclaredMutation: + """Claims it did nothing but actually mutated the candidate.""" + + name = "undeclared_mutation" + + def fix(self, model: onnx.ModelProto) -> list[FixApplied]: + model.graph.node[0].name = "SNEAKY_EDIT" + return [] + + +class _EmitsInvalidModel: + """Declares a fix but breaks the graph (dangling output reference).""" + + name = "emits_invalid" + + def fix(self, model: onnx.ModelProto) -> list[FixApplied]: + model.graph.node[-1].input[0] = "no_such_tensor" + return [FixApplied(fixer=self.name, target="y", description="broke the graph")] + + +def _node_names(model: onnx.ModelProto) -> list[str]: + return [n.name for n in model.graph.node] + + +class TestTransactionalPipeline: + def test_mutate_then_crash_leaves_no_trace(self) -> None: + model = _valid_model() + outcome = run_fixers(model, [_MutateThenCrash()]) + assert outcome.applied == [] + assert len(outcome.failures) == 1 + assert outcome.failures[0].fixer == "mutate_then_crash" + assert "RuntimeError" in outcome.failures[0].reason + assert not any(n.startswith("CORRUPTED_") for n in _node_names(outcome.model)) + onnx.checker.check_model(outcome.model, full_check=True) + + def test_failed_fixer_does_not_block_later_fixers(self) -> None: + model = _valid_model() + outcome = run_fixers(model, [_MutateThenCrash(), DropDropoutFixer()]) + assert [f.fixer for f in outcome.failures] == ["mutate_then_crash"] + assert [a.fixer for a in outcome.applied] == ["drop_dropout"] + assert not any(n.op_type == "Dropout" for n in outcome.model.graph.node) + assert not any(n.startswith("CORRUPTED_") for n in _node_names(outcome.model)) + onnx.checker.check_model(outcome.model, full_check=True) + + def test_malformed_return_is_rejected_and_mutation_discarded(self) -> None: + model = _valid_model() + outcome = run_fixers(model, [_MalformedReturn()]) # type: ignore[list-item] + assert outcome.applied == [] + assert [f.fixer for f in outcome.failures] == ["malformed_return"] + assert "MUTATED_ANYWAY" not in _node_names(outcome.model) + + def test_undeclared_mutation_is_discarded(self) -> None: + model = _valid_model() + outcome = run_fixers(model, [_UndeclaredMutation()]) + assert outcome.applied == [] + assert outcome.failures == [] + assert "SNEAKY_EDIT" not in _node_names(outcome.model) + + def test_invalid_candidate_is_discarded(self) -> None: + model = _valid_model() + outcome = run_fixers(model, [_EmitsInvalidModel()]) + assert outcome.applied == [] + assert [f.fixer for f in outcome.failures] == ["emits_invalid"] + assert "produced an invalid model" in outcome.failures[0].reason + onnx.checker.check_model(outcome.model, full_check=True) + + def test_input_model_is_never_mutated(self) -> None: + model = _valid_model() + before = model.SerializeToString() + run_fixers(model, [_MutateThenCrash(), DropDropoutFixer(), _EmitsInvalidModel()]) + assert model.SerializeToString() == before + + def test_validation_level_recorded(self) -> None: + model = _valid_model() + outcome = run_fixers(model, [DropDropoutFixer()]) + assert outcome.validation == "full" + + +def test_external_data_model_gets_basic_validation(tmp_path, monkeypatch) -> None: + """External-data initializers cannot be read by full inference from an + in-memory proto; the pipeline must degrade to the basic check, not crash.""" + # onnx.checker resolves external-data paths against the CWD for in-memory + # protos; give it a real payload file so the basic check can pass. + (tmp_path / "weights.bin").write_bytes(b"\x00" * 12) + monkeypatch.chdir(tmp_path) + model = _valid_model() + ext = numpy_helper.from_array(np.zeros(3, dtype=np.float32), name="w_ext") + ext.data_location = onnx.TensorProto.EXTERNAL + ext.ClearField("raw_data") + entry = ext.external_data.add() + entry.key = "location" + entry.value = "weights.bin" + model.graph.initializer.append(ext) + + outcome = run_fixers(model, [DropDropoutFixer()]) + assert outcome.validation == "basic" + assert [a.fixer for a in outcome.applied] == ["drop_dropout"] + + +class _ClaimsWithoutMutation: + """Reports a fix but leaves the model untouched: the record is a lie.""" + + name = "claims_without_mutation" + + def fix(self, model: onnx.ModelProto) -> list[FixApplied]: + return [FixApplied(fixer=self.name, target="x", description="did nothing")] + + +def test_claimed_fixes_without_mutation_are_rejected() -> None: + model = _valid_model() + outcome = run_fixers(model, [_ClaimsWithoutMutation(), DropDropoutFixer()]) + assert all(a.fixer != "claims_without_mutation" for a in outcome.applied) + assert [f.fixer for f in outcome.failures] == ["claims_without_mutation"] + assert "did not modify" in outcome.failures[0].reason + # The honest fixer after it still runs. + assert [a.fixer for a in outcome.applied] == ["drop_dropout"] diff --git a/tests/test_plugins_module.py b/tests/test_plugins_module.py index 674ea92..1e53d9e 100644 --- a/tests/test_plugins_module.py +++ b/tests/test_plugins_module.py @@ -63,3 +63,40 @@ def test_in_tree_reporters_still_satisfy_protocol(self) -> None: from trtcheck.reporters.json import JSONReporter assert isinstance(JSONReporter(), Reporter) + + +def test_plugin_checker_findings_get_namespaced_rule_id_fallback() -> None: + """A plugin finding without a rule_id must not reach the report blank -- + it gets a PLUGIN- fallback so CI filters always have an id.""" + import onnx + from onnx import TensorProto, helper + + from trtcheck.analyzer import Analyzer, AnalyzerConfig + from trtcheck.types import CheckCategory, Issue, Severity + + class NoIdChecker: + name = "my checker!" + + def check(self, model: onnx.ModelProto) -> list[Issue]: + return [ + Issue( + severity=Severity.INFO, + category=CheckCategory.GRAPH_STRUCTURE, + node_name="n", + operator="X", + message="plugin finding", + remediation="none", + ) + ] + + analyzer = Analyzer(AnalyzerConfig(discover_entry_point_plugins=False)) + analyzer.checkers.append(NoIdChecker()) + + inp = helper.make_tensor_value_info("x", TensorProto.FLOAT, [1]) + out = helper.make_tensor_value_info("y", TensorProto.FLOAT, [1]) + graph = helper.make_graph([helper.make_node("Relu", ["x"], ["y"], name="r")], "m", [inp], [out]) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + + report = analyzer.analyze_model(model) + finding = next(i for i in report.issues if i.message == "plugin finding") + assert finding.rule_id == "PLUGIN-MY-CHECKER" diff --git a/tests/test_reporters.py b/tests/test_reporters.py index 1ebef0b..bdc80b1 100644 --- a/tests/test_reporters.py +++ b/tests/test_reporters.py @@ -331,5 +331,5 @@ def test_render_diff_distinguishes_passing_from_failing(self) -> None: after = self._build("after.onnx", with_issues=False) # 0 critical -> pass out = HTMLReporter().render_diff(before, after) # Each side surfaces its verdict. - assert "conversion will fail" in out.lower() - assert "likely to convert" in out.lower() + assert "conversion blocked" in out.lower() + assert "no known blocker" in out.lower() diff --git a/tests/test_runtime_verify.py b/tests/test_runtime_verify.py new file mode 100644 index 0000000..d29584b --- /dev/null +++ b/tests/test_runtime_verify.py @@ -0,0 +1,145 @@ +"""Runtime verification module -- all subprocess behavior mocked. + +No test here requires TensorRT, a GPU, or network access. +""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import onnx +import pytest +from click.testing import CliRunner +from onnx import TensorProto, helper + +from trtcheck import runtime_verify +from trtcheck.cli import main +from trtcheck.runtime_verify import RuntimeStatus, verify_model + + +class _FakeProc: + def __init__(self, returncode: int, stdout: str = "", stderr: str = "") -> None: + self.returncode = returncode + self.stdout = stdout + self.stderr = stderr + + +def _patch_run(monkeypatch: pytest.MonkeyPatch, proc: _FakeProc) -> dict: + seen: dict = {} + + def fake_run(cmd, **kwargs): # noqa: ANN001 + seen["cmd"] = cmd + seen["kwargs"] = kwargs + return proc + + monkeypatch.setattr(runtime_verify.subprocess, "run", fake_run) + return seen + + +class TestVerifyModel: + def test_missing_trtexec(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: None) + result = verify_model("model.onnx") + assert result.status is RuntimeStatus.MISSING_TRTEXEC + assert not result.verified + + def test_success(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: "/usr/bin/trtexec") + seen = _patch_run( + monkeypatch, + _FakeProc(0, stdout="[I] TensorRT version: 10.3.0\n[I] Engine built"), + ) + result = verify_model("model.onnx", timeout_s=42) + assert result.status is RuntimeStatus.SUCCESS + assert result.verified + assert result.trtexec_version and "TensorRT" in result.trtexec_version + # No shell, list-args invocation, timeout honored. + assert seen["cmd"][0] == "/usr/bin/trtexec" + assert seen["cmd"][1] == "--onnx=model.onnx" + assert seen["kwargs"]["timeout"] == 42 + assert "shell" not in seen["kwargs"] or seen["kwargs"]["shell"] is False + + def test_parser_failure(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: "/usr/bin/trtexec") + _patch_run( + monkeypatch, + _FakeProc(1, stderr="[E] ModelImporter.cpp: Failed to parse ONNX model"), + ) + result = verify_model("model.onnx") + assert result.status is RuntimeStatus.PARSER_FAILURE + + def test_build_failure(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: "/usr/bin/trtexec") + _patch_run(monkeypatch, _FakeProc(1, stderr="[E] Error: out of workspace memory")) + result = verify_model("model.onnx") + assert result.status is RuntimeStatus.BUILD_FAILURE + + def test_timeout(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: "/usr/bin/trtexec") + + def raise_timeout(cmd, **kwargs): # noqa: ANN001 + raise subprocess.TimeoutExpired(cmd, kwargs.get("timeout", 0)) + + monkeypatch.setattr(runtime_verify.subprocess, "run", raise_timeout) + result = verify_model("model.onnx", timeout_s=1) + assert result.status is RuntimeStatus.TIMEOUT + + def test_output_tails_are_truncated(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: "/usr/bin/trtexec") + _patch_run(monkeypatch, _FakeProc(0, stdout="x" * 100_000)) + result = verify_model("model.onnx") + assert len(result.stdout_tail) <= 2000 + + +class TestCliIntegration: + def _clean_path(self, tmp_path: Path) -> Path: + inp = helper.make_tensor_value_info("input", TensorProto.FLOAT, [1, 4]) + out = helper.make_tensor_value_info("output", TensorProto.FLOAT, [1, 4]) + relu = helper.make_node("Relu", ["input"], ["output"], name="r") + graph = helper.make_graph([relu], "m", [inp], [out]) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + model.ir_version = 8 + path = tmp_path / "m.onnx" + onnx.save(model, str(path)) + return path + + def test_successful_runtime_verification_yields_verified( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: "/usr/bin/trtexec") + _patch_run(monkeypatch, _FakeProc(0, stdout="TensorRT version 10.3")) + result = CliRunner().invoke( + main, [str(self._clean_path(tmp_path)), "--verify-runtime", "--format", "json"] + ) + assert result.exit_code == 0, result.output + assert '"verdict": "verified"' in result.stdout + assert '"runtime_verified": true' in result.stdout + + def test_missing_trtexec_leaves_static_verdict( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: None) + result = CliRunner().invoke( + main, [str(self._clean_path(tmp_path)), "--verify-runtime", "--format", "json"] + ) + assert result.exit_code == 0, result.output + assert '"verdict": "likely"' in result.stdout + assert '"status": "missing_trtexec"' in result.stdout + + def test_runtime_failure_demotes_likely_to_unverified( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A recorded parser/build failure must not hide behind a clean + static prediction: the verdict drops to unverified.""" + monkeypatch.setattr(runtime_verify.shutil, "which", lambda _: "/usr/bin/trtexec") + _patch_run( + monkeypatch, + _FakeProc(1, stderr="[E] ModelImporter.cpp: Failed to parse ONNX model"), + ) + result = CliRunner().invoke( + main, [str(self._clean_path(tmp_path)), "--verify-runtime", "--format", "json"] + ) + assert result.exit_code == 0, result.output + assert '"verdict": "unverified"' in result.stdout + assert '"status": "parser_failure"' in result.stdout diff --git a/tests/test_verdicts.py b/tests/test_verdicts.py new file mode 100644 index 0000000..b121f23 --- /dev/null +++ b/tests/test_verdicts.py @@ -0,0 +1,277 @@ +"""Verdict model, stable rule ids, and honest-uncertainty findings.""" + +from __future__ import annotations + +import json + +import onnx +from onnx import TensorProto, helper + +from trtcheck import remediation +from trtcheck.analyzer import Analyzer, AnalyzerConfig +from trtcheck.checkers.operator_support import ( + RULE_CONDITION, + RULE_CONDITION_UNRESOLVED, + RULE_CUSTOM_DOMAIN, + RULE_PARTIAL, + RULE_UNCLASSIFIED, + RULE_UNSUPPORTED, +) +from trtcheck.reporters.json import JSONReporter +from trtcheck.types import REPORT_SCHEMA_VERSION, AnalysisReport, Confidence, Severity, Verdict + +_ANALYZER = Analyzer(AnalyzerConfig(discover_entry_point_plugins=False)) + +# The complete public rule-id registry. Renaming or removing an id here is a +# BREAKING change for CI consumers filtering on rule_id -- this test is the +# tripwire. Additions are fine. +_DOCUMENTED_RULE_IDS = { + "TRT-DTYPE-INT64-WEIGHTS", + "TRT-DTYPE-INT64-INPUT", + "TRT-DTYPE-UINT8-INPUT", + "TRT-DTYPE-BF16", + "TRT-DTYPE-FP64", + "TRT-DTYPE-STRING", + "TRT-SHAPE-PROFILE-MISSING", + "TRT-GRAPH-NO-OUTPUT", + "TRT-GRAPH-ISOLATED-NODE", + "TRT-GRAPH-DUP-NODE-NAME", + "TRT-GRAPH-LARGE-CONSTANT", + "TRT-GRAPH-EXTERNAL-DATA", + "TRT-GRAPH-INPUT-UNTYPED", + "TRT-GRAPH-ALIASING", + "TRT-CONTROL-LOOP-RUNTIME-TRIP", + "TRT-CONTROL-LOOP-DYNAMIC-TRIP", + "TRT-CONTROL-LOOP-NESTED", + "TRT-CONTROL-IF-SHAPE-MISMATCH", + "TRT-CONTROL-IF-UNVERIFIED", + "TRT-CONTROL-SCAN", + "TRT-OPSET-OLD", + "TRT-OP-UNSUPPORTED", + "TRT-OP-PARTIAL", +} +_CHECKER_OWNED_IDS = { + RULE_UNSUPPORTED, + RULE_PARTIAL, + RULE_UNCLASSIFIED, + RULE_CUSTOM_DOMAIN, + RULE_CONDITION, + RULE_CONDITION_UNRESOLVED, +} + + +def test_rule_id_registry_is_stable() -> None: + assert remediation.rule_ids() == frozenset(_DOCUMENTED_RULE_IDS) + # Checker-owned ids never collide with DB-owned ones (except the two the + # operator checker shares with legacy DB entries by design). + overlap = _CHECKER_OWNED_IDS & remediation.rule_ids() + assert overlap == {RULE_UNSUPPORTED, RULE_PARTIAL} + + +def _model_with_node(node: helper.NodeProto, opsets=None) -> onnx.ModelProto: + inp = helper.make_tensor_value_info("input", TensorProto.FLOAT, [1, 4]) + vi = onnx.ValueInfoProto() + vi.name = node.output[0] + graph = helper.make_graph([node], "m", [inp], [vi]) + model = helper.make_model(graph, opset_imports=opsets or [helper.make_opsetid("", 17)]) + model.ir_version = 8 + return model + + +class TestVerdicts: + def test_clean_model_is_likely(self, clean_model: onnx.ModelProto) -> None: + report = _ANALYZER.analyze_model(clean_model) + assert report.verdict is Verdict.LIKELY + assert report.conversion_likely is True + + def test_critical_issue_means_blocked(self, uint8_input_model: onnx.ModelProto) -> None: + report = _ANALYZER.analyze_model(uint8_input_model) + assert report.verdict is Verdict.BLOCKED + assert report.conversion_likely is False + + def test_unknown_default_domain_op_means_unverified(self) -> None: + node = helper.make_node("TotallyNovelOp", ["input"], ["y"], name="n") + report = _ANALYZER.analyze_model(_model_with_node(node)) + assert report.verdict is Verdict.UNVERIFIED + finding = next(i for i in report.issues if i.rule_id == RULE_UNCLASSIFIED) + assert finding.severity is Severity.INFO + assert finding.verify_required is True + assert finding.confidence is Confidence.LOW + + def test_custom_domain_op_means_unverified(self) -> None: + node = helper.make_node("PluginOp", ["input"], ["y"], name="n", domain="com.acme") + model = _model_with_node( + node, + opsets=[helper.make_opsetid("", 17), helper.make_opsetid("com.acme", 1)], + ) + report = _ANALYZER.analyze_model(model) + assert report.verdict is Verdict.UNVERIFIED + finding = next(i for i in report.issues if i.rule_id == RULE_CUSTOM_DOMAIN) + assert "com.acme" in finding.operator + # Not a guaranteed blocker: honest uncertainty, not a critical. + assert finding.severity is Severity.INFO + + def test_declared_plugin_domain_suppresses_finding(self) -> None: + node = helper.make_node("PluginOp", ["input"], ["y"], name="n", domain="com.acme") + model = _model_with_node( + node, + opsets=[helper.make_opsetid("", 17), helper.make_opsetid("com.acme", 1)], + ) + analyzer = Analyzer( + AnalyzerConfig(discover_entry_point_plugins=False, plugin_domains=["com.acme"]) + ) + report = analyzer.analyze_model(model) + assert all(i.rule_id != RULE_CUSTOM_DOMAIN for i in report.issues) + + def test_unclassified_findings_aggregate_per_op_type(self) -> None: + n1 = helper.make_node("NovelOp", ["input"], ["y1"], name="n1") + n2 = helper.make_node("NovelOp", ["y1"], ["y2"], name="n2") + inp = helper.make_tensor_value_info("input", TensorProto.FLOAT, [1]) + vi = onnx.ValueInfoProto() + vi.name = "y2" + graph = helper.make_graph([n1, n2], "m", [inp], [vi]) + model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]) + report = _ANALYZER.analyze_model(model) + unclassified = [i for i in report.issues if i.rule_id == RULE_UNCLASSIFIED] + assert len(unclassified) == 1 + assert "2 nodes" in unclassified[0].node_name + + def test_runtime_verified_upgrades_to_verified(self) -> None: + report = AnalysisReport( + filename="f", + onnx_ir_version="8", + opset_version=17, + producer="p", + total_nodes=1, + runtime_verified=True, + ) + assert report.verdict is Verdict.VERIFIED + + def test_runtime_verified_never_overrides_blocked( + self, uint8_input_model: onnx.ModelProto + ) -> None: + report = _ANALYZER.analyze_model(uint8_input_model) + report.runtime_verified = True + assert report.verdict is Verdict.BLOCKED + + +class TestJsonSchema: + def test_report_carries_schema_version_verdict_and_rule_ids( + self, uint8_input_model: onnx.ModelProto + ) -> None: + report = _ANALYZER.analyze_model(uint8_input_model) + payload = json.loads(JSONReporter().render(report)) + assert payload["schema_version"] == REPORT_SCHEMA_VERSION + assert payload["verdict"] == "blocked" + assert payload["target_trt"] == "10.3" + # Deprecated 1.x keys are still present for old consumers. + assert payload["conversion_likely"] is False + issue = payload["issues"][0] + for key in ("rule_id", "confidence", "verify_required", "target_trt", "graph_scope"): + assert key in issue + assert issue["rule_id"].startswith("TRT-") + + def test_every_emitted_issue_has_a_rule_id( + self, sequence_empty_model: onnx.ModelProto, control_flow_loop_model: onnx.ModelProto + ) -> None: + for model in (sequence_empty_model, control_flow_loop_model): + report = _ANALYZER.analyze_model(model) + assert report.issues + for issue in report.issues: + assert issue.rule_id, f"issue without rule_id: {issue.message}" + assert issue.target_trt == "10.3" + + +class TestVerdictPrecedence: + """Precedence when multiple conditions coexist: + BLOCKED > VERIFIED > UNVERIFIED > LIKELY, with runtime *failures* + demoting an otherwise-clean report to UNVERIFIED.""" + + def _report(self, **kwargs) -> AnalysisReport: + base = dict( + filename="f", onnx_ir_version="8", opset_version=17, producer="p", total_nodes=1 + ) + base.update(kwargs) + return AnalysisReport(**base) + + def _verify_issue(self) -> "onnx.ModelProto": + from trtcheck.types import CheckCategory, Issue + + return Issue( + severity=Severity.INFO, + category=CheckCategory.OPERATOR_SUPPORT, + node_name="n", + operator="X", + message="m", + remediation="r", + verify_required=True, + ) + + def test_blocked_beats_runtime_verified_and_unresolved(self) -> None: + from trtcheck.types import CheckCategory, Issue + + crit = Issue( + severity=Severity.CRITICAL, + category=CheckCategory.OPERATOR_SUPPORT, + node_name="n", + operator="X", + message="m", + remediation="r", + ) + report = self._report( + issues=[crit, self._verify_issue()], + runtime_verified=True, + runtime_verification={"status": "success"}, + ) + assert report.verdict is Verdict.BLOCKED + + def test_verified_beats_unresolved_conditions(self) -> None: + report = self._report( + issues=[self._verify_issue()], + runtime_verified=True, + runtime_verification={"status": "success"}, + ) + assert report.verdict is Verdict.VERIFIED + + def test_runtime_parser_failure_demotes_likely_to_unverified(self) -> None: + report = self._report(runtime_verification={"status": "parser_failure"}) + assert report.verdict is Verdict.UNVERIFIED + + def test_runtime_build_failure_demotes_likely_to_unverified(self) -> None: + report = self._report(runtime_verification={"status": "build_failure"}) + assert report.verdict is Verdict.UNVERIFIED + + def test_unavailable_runtime_leaves_static_verdict(self) -> None: + for status in ("missing_trtexec", "timeout", "error"): + report = self._report(runtime_verification={"status": status}) + assert report.verdict is Verdict.LIKELY, status + + def test_issue_identity_includes_graph_scope(self) -> None: + from trtcheck.types import CheckCategory, Issue + + a = Issue( + severity=Severity.INFO, + category=CheckCategory.OPERATOR_SUPPORT, + node_name="n", + operator="X", + message="m", + remediation="r", + rule_id="TRT-X", + graph_scope="then_body", + ) + b = Issue( + severity=Severity.INFO, + category=CheckCategory.OPERATOR_SUPPORT, + node_name="n", + operator="X", + message="m", + remediation="r", + rule_id="TRT-X", + graph_scope="else_body", + ) + assert a.identity() != b.identity() + + def test_json_rendering_is_deterministic(self, uint8_input_model: onnx.ModelProto) -> None: + r1 = _ANALYZER.analyze_model(uint8_input_model) + r2 = _ANALYZER.analyze_model(uint8_input_model) + assert JSONReporter().render(r1) == JSONReporter().render(r2) diff --git a/tools/build_operator_matrix.py b/tools/build_operator_matrix.py index 5b40377..42247e3 100644 --- a/tools/build_operator_matrix.py +++ b/tools/build_operator_matrix.py @@ -11,6 +11,15 @@ VERSIONS = ["8.0", "8.6", "10.0", "10.3"] +# Evidence pointer for conditional-support entries sourced from the upstream +# onnx-tensorrt operator table (tracks TensorRT 10.x on its main branch). +_ONNX_TRT_DOCS = "https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md" +_ONNX_TRT_EVIDENCE = { + "status": "official_docs", + "source": _ONNX_TRT_DOCS, + "retrieved": "2026-07-22", +} + # Status shorthand to keep the source readable: # S = supported, P = partial, N = not_supported, U = unknown _STATUS = {"S": "supported", "P": "partial", "N": "not_supported", "U": "unknown"} @@ -61,6 +70,13 @@ def expand(*codes: str) -> dict[str, str]: "notes": "Added in TRT 8.6 as a plugin, native in 10.0+.", }, "Softmax": {"support": expand("S", "S", "S", "S")}, + "Clip": { + "support": expand("U", "U", "S", "S"), + "notes": ( + "Supported in TRT 10.x (FP32/FP16/BF16) per onnx-tensorrt docs, " + "retrieved 2026-07-22. 8.x status not verified here, left unknown." + ), + }, "LogSoftmax": {"support": expand("S", "S", "S", "S")}, # Normalization "BatchNormalization": {"support": expand("S", "S", "S", "S")}, @@ -136,8 +152,69 @@ def expand(*codes: str) -> dict[str, str]: # Resize / Upsample "Resize": { "support": expand("P", "P", "S", "S"), - "notes": "Only nearest and linear modes pre-10.0. Cubic added in 10.0.", - "limitations": ["antialias attribute not supported before TRT 10.0."], + "notes": ( + "Only nearest and linear modes; cubic is not supported " + "(onnx-tensorrt docs, retrieved 2026-07-22)." + ), + "limitations": [ + "Antialiasing (antialias=1) is not supported.", + "coordinate_transformation_mode limited to half_pixel, pytorch_half_pixel, " + "tf_half_pixel_for_nn, asymmetric, align_corners.", + ], + "conditions": [ + { + "id": "resize-mode", + "applies_to": ["10.0", "10.3"], + "kind": "attribute_allowed", + "attribute": "mode", + "allowed_values": ["nearest", "linear"], + "default_ok": True, + "severity": "critical", + "message": ( + "TensorRT supports only Resize modes 'nearest' and 'linear' " + "(cubic is rejected)." + ), + "remediation": ( + "Re-export with mode=nearest or mode=linear, or implement cubic " + "resize as a plugin." + ), + "evidence": _ONNX_TRT_EVIDENCE, + }, + { + "id": "resize-coord-transform", + "applies_to": ["10.0", "10.3"], + "kind": "attribute_allowed", + "attribute": "coordinate_transformation_mode", + "allowed_values": [ + "half_pixel", + "pytorch_half_pixel", + "tf_half_pixel_for_nn", + "asymmetric", + "align_corners", + ], + "default_ok": True, + "severity": "critical", + "message": ( + "TensorRT supports Resize coordinate_transformation_mode in " + "{half_pixel, pytorch_half_pixel, tf_half_pixel_for_nn, asymmetric, " + "align_corners} only." + ), + "remediation": "Re-export with a supported coordinate_transformation_mode.", + "evidence": _ONNX_TRT_EVIDENCE, + }, + { + "id": "resize-no-antialias", + "applies_to": ["10.0", "10.3"], + "kind": "attribute_allowed", + "attribute": "antialias", + "allowed_values": [0], + "default_ok": True, + "severity": "critical", + "message": "TensorRT does not support antialiased Resize (antialias=1).", + "remediation": "Re-export with antialias=0.", + "evidence": _ONNX_TRT_EVIDENCE, + }, + ], }, "Upsample": { "support": expand("S", "S", "S", "S"), @@ -191,7 +268,37 @@ def expand(*codes: str) -> dict[str, str]: "ConstantOfShape": {"support": expand("S", "S", "S", "S")}, "ArgMax": {"support": expand("S", "S", "S", "S")}, "ArgMin": {"support": expand("S", "S", "S", "S")}, - "TopK": {"support": expand("S", "S", "S", "S")}, + "TopK": { + "support": expand("S", "S", "S", "S"), + "conditions": [ + { + "id": "topk-sorted-required", + "applies_to": ["10.0", "10.3"], + "kind": "attribute_allowed", + "attribute": "sorted", + "allowed_values": [1], + "default_ok": True, + "severity": "critical", + "message": ( + "TensorRT requires TopK attribute sorted=1 (ONNX default). " + "sorted=0 is rejected." + ), + "remediation": "Re-export with sorted=1, or sort outside the model.", + "evidence": _ONNX_TRT_EVIDENCE, + }, + { + "id": "topk-k-max-3840", + "applies_to": ["10.0", "10.3"], + "kind": "constant_input_max", + "input_index": 1, + "max_value": 3839, + "severity": "critical", + "message": "TensorRT requires the TopK K input to be less than 3840.", + "remediation": ("Reduce K below 3840, or restructure the selection into chunks."), + "evidence": _ONNX_TRT_EVIDENCE, + }, + ], + }, "NonZero": {"support": expand("P", "S", "S", "S")}, "NonMaxSuppression": { "support": expand("P", "S", "S", "S"), @@ -223,8 +330,8 @@ def build_matrix() -> dict[str, Any]: module-level ``OPERATORS`` source of truth. """ return { - "schema_version": "1.0", - "last_updated": "2026-05-21", + "schema_version": "2.0", + "last_updated": "2026-07-22", "target_trt_versions": list(VERSIONS), "operators": copy.deepcopy(OPERATORS), } diff --git a/trtcheck/__init__.py b/trtcheck/__init__.py index cf66233..ea24ed0 100644 --- a/trtcheck/__init__.py +++ b/trtcheck/__init__.py @@ -1,16 +1,27 @@ """trtcheck -- static pre-flight checker for ONNX -> TensorRT conversion.""" from trtcheck.analyzer import Analyzer, AnalyzerConfig, analyze -from trtcheck.types import AnalysisReport, CheckCategory, Issue, Severity +from trtcheck.types import ( + REPORT_SCHEMA_VERSION, + AnalysisReport, + CheckCategory, + Confidence, + Issue, + Severity, + Verdict, +) -__version__ = "1.0.0" +__version__ = "1.1.0" __all__ = [ "Analyzer", "AnalyzerConfig", "analyze", "AnalysisReport", "CheckCategory", + "Confidence", "Issue", "Severity", + "Verdict", + "REPORT_SCHEMA_VERSION", "__version__", ] diff --git a/trtcheck/analyzer.py b/trtcheck/analyzer.py index 18de9aa..2e635d5 100644 --- a/trtcheck/analyzer.py +++ b/trtcheck/analyzer.py @@ -14,7 +14,7 @@ from trtcheck.checkers.graph_structure import GraphStructureChecker from trtcheck.checkers.operator_support import OperatorSupportChecker from trtcheck.checkers.precision import PrecisionChecker -from trtcheck.types import AnalysisReport, CheckCategory, Issue, Severity +from trtcheck.types import AnalysisReport, CheckCategory, Confidence, Issue, Severity # Names of the checkers trtcheck ships. A crash in one of these is a bug in # trtcheck and must surface loudly; only *third-party plugin* checkers are @@ -24,6 +24,12 @@ ) +def _slug(name: str) -> str: + """Uppercase A-Z0-9 slug of a plugin name for rule-id fallbacks.""" + cleaned = "".join(c if c.isalnum() else "-" for c in name.upper()).strip("-") + return cleaned or "UNNAMED" + + def safe_load(path: Path | str) -> onnx.ModelProto: """``onnx.load`` with parse failures surfaced as a clean ``ValueError``. @@ -48,6 +54,10 @@ class AnalyzerConfig: max_model_size_mb: int = 500 # refuse to load files larger than this discover_entry_point_plugins: bool = True disable_plugins: list[str] = field(default_factory=list) + # Custom ONNX domains the user declares as backed by an installed + # TensorRT plugin. Ops in these domains stop producing + # TRT-OP-CUSTOM-DOMAIN unverified findings. + plugin_domains: list[str] = field(default_factory=list) class Analyzer: @@ -64,6 +74,7 @@ def _build_checkers(self) -> list[Checker]: OperatorSupportChecker( matrix_path=self.config.matrix_path, target_trt=self.config.target_trt, + plugin_domains=self.config.plugin_domains, ), DynamicShapeChecker(), ControlFlowChecker(target_trt=self.config.target_trt), @@ -104,7 +115,15 @@ def analyze_model( all_issues.extend(checker.check(model)) continue try: - all_issues.extend(checker.check(model)) + plugin_issues = checker.check(model) + # A plugin finding without a rule id gets a namespaced + # fallback so CI filters on rule_id always have something + # stable to match, and the plugin origin stays visible. + fallback = f"PLUGIN-{_slug(name)}" + for issue in plugin_issues: + if not issue.rule_id: + issue.rule_id = fallback + all_issues.extend(plugin_issues) except Exception as exc: all_issues.append( Issue( @@ -115,9 +134,19 @@ def analyze_model( message=(f"plugin {name!r} raised {exc.__class__.__name__}: {exc}"), remediation=("Disable the plugin with --disable-plugin or uninstall it."), docs_link=None, + rule_id="TRT-PLUGIN-CHECKER-ERROR", + confidence=Confidence.LOW, + verify_required=True, ) ) + # Every finding in this report was made against the same TRT target; + # stamp it on issues whose checker didn't (precision/graph checks are + # target-independent but the report they land in is not). + for issue in all_issues: + if issue.target_trt is None: + issue.target_trt = self.config.target_trt + # Sort: critical first, then warning, then info. Stable. all_issues.sort(key=lambda i: Severity.rank(i.severity)) @@ -125,6 +154,7 @@ def analyze_model( (o.version for o in model.opset_import if o.domain in ("", "ai.onnx")), default=0 ) report = AnalysisReport( + target_trt=self.config.target_trt, filename=filename, onnx_ir_version=str(model.ir_version), opset_version=opset, diff --git a/trtcheck/checkers/control_flow.py b/trtcheck/checkers/control_flow.py index 3a8fa41..88069a8 100644 --- a/trtcheck/checkers/control_flow.py +++ b/trtcheck/checkers/control_flow.py @@ -56,13 +56,15 @@ def check(self, model: onnx.ModelProto) -> list[Issue]: # "runtime trip count" warnings for outer-scope constants. initializer_names = {init.name for init, _ in iter_initializers(model.graph)} graph_input_names = {inp.name for inp in model.graph.input} - for node, _graph in iter_nodes(model.graph): + for node, graph in iter_nodes(model.graph): if node.op_type == "Loop": - issues.extend(self._check_loop(node, initializer_names, graph_input_names)) + issues.extend( + self._check_loop(node, initializer_names, graph_input_names, graph.name) + ) elif node.op_type == "If": - issues.append(self._check_if(node)) + issues.append(self._check_if(node, graph.name)) elif node.op_type == "Scan": - issues.append(self._scan_warning(node)) + issues.append(self._scan_warning(node, graph.name)) return issues # -- Loop -------------------------------------------------------------- @@ -72,6 +74,7 @@ def _check_loop( node: onnx.NodeProto, initializer_names: set[str], graph_input_names: set[str], + graph_scope: str, ) -> list[Issue]: issues: list[Issue] = [] name = node.name or "" @@ -95,6 +98,7 @@ def _check_loop( else f"computed value '{trip_input}'" ) + " as its trip count", + graph_scope=graph_scope, ) ) @@ -107,28 +111,31 @@ def _check_loop( node_name=name, operator="Loop", prefix=f"Loop '{node.name}' contains a nested Loop in its body", + graph_scope=graph_scope, ) ) return issues # -- If --------------------------------------------------------------- - def _check_if(self, node: onnx.NodeProto) -> Issue: + def _check_if(self, node: onnx.NodeProto, graph_scope: str) -> Issue: return remediation.make_issue( "if_detected_unverified", node_name=node.name or "", operator="If", prefix=f"If '{node.name}' detected", + graph_scope=graph_scope, ) # -- Scan ------------------------------------------------------------- - def _scan_warning(self, node: onnx.NodeProto) -> Issue: + def _scan_warning(self, node: onnx.NodeProto, graph_scope: str) -> Issue: return remediation.make_issue( "scan_dynamic_length", node_name=node.name or "", operator="Scan", prefix=f"Scan '{node.name}' detected", + graph_scope=graph_scope, ) diff --git a/trtcheck/checkers/operator_support.py b/trtcheck/checkers/operator_support.py index ea6e906..646c689 100644 --- a/trtcheck/checkers/operator_support.py +++ b/trtcheck/checkers/operator_support.py @@ -3,26 +3,44 @@ For each node in the graph, find the operator in operator_matrix.json and emit an Issue at the appropriate severity: - - not_supported -> CRITICAL - - partial -> WARNING - - supported -> nothing - - unknown -> nothing (info-level, never blocking) - -Custom-domain ops are silently skipped: the matrix only describes the -default ONNX domain, and emitting a finding for every custom plugin would -be noise. + - not_supported -> CRITICAL (rule TRT-OP-UNSUPPORTED) + - partial -> WARNING (rule TRT-OP-PARTIAL, needs verification) + - supported -> nothing, unless a conditional-support rule fires + - unknown / absent from the matrix -> INFO (rule TRT-OP-UNCLASSIFIED, + needs verification). An operator trtcheck has no evidence about must + not silently pass as clean. + +Custom-domain operators (rule TRT-OP-CUSTOM-DOMAIN) always need a TensorRT +plugin; they are reported as unverified findings unless the caller +explicitly declares the domain as plugin-backed via ``plugin_domains``. + +Conditional support (schema 2.x matrices) is expressed per operator as a +``conditions`` list. Each condition either passes, fires a violation +(rule TRT-OP-CONDITION), or cannot be resolved statically and produces an +unverified finding (rule TRT-OP-CONDITION-UNRESOLVED). """ from __future__ import annotations import json +from collections import Counter from pathlib import Path -from typing import Any +from typing import Any, Iterable import onnx -from trtcheck._graph import iter_nodes -from trtcheck.types import CheckCategory, Issue, Severity +from trtcheck._graph import iter_nodes, iter_subgraphs +from trtcheck.types import CheckCategory, Confidence, Issue, Severity + +# Stable rule ids this checker owns (the remediation DB owns the rest). +RULE_UNSUPPORTED = "TRT-OP-UNSUPPORTED" +RULE_PARTIAL = "TRT-OP-PARTIAL" +RULE_UNCLASSIFIED = "TRT-OP-UNCLASSIFIED" +RULE_CUSTOM_DOMAIN = "TRT-OP-CUSTOM-DOMAIN" +RULE_CONDITION = "TRT-OP-CONDITION" +RULE_CONDITION_UNRESOLVED = "TRT-OP-CONDITION-UNRESOLVED" + +_DEFAULT_DOMAINS = ("", "ai.onnx") def _default_matrix_path() -> Path: @@ -36,6 +54,7 @@ def __init__( self, matrix_path: Path | str | None = None, target_trt: str = "10.3", + plugin_domains: Iterable[str] = (), ) -> None: path = Path(matrix_path) if matrix_path else _default_matrix_path() with open(path) as f: @@ -45,30 +64,57 @@ def __init__( raise ValueError(f"target_trt={target_trt!r} not in matrix versions {sorted(valid)}") self._target = target_trt self._ops: dict[str, dict[str, Any]] = self._matrix["operators"] + self._plugin_domains = set(plugin_domains) def check(self, model: onnx.ModelProto) -> list[Issue]: issues: list[Issue] = [] + # Scalar-int constants (initializers + Constant nodes) across every + # scope, used to evaluate constant-input conditions statically. + constants = _collect_scalar_constants(model) + # Aggregated uncertainty findings: one per distinct op_type (or + # domain/op_type pair), not one per node -- a transformer with 400 + # unclassified nodes of the same op should read as one finding. + unclassified: Counter[str] = Counter() + custom: Counter[tuple[str, str]] = Counter() + # Walk the top-level graph AND every If/Loop/Scan subgraph body: an # unsupported op inside a branch still blocks the TensorRT build. - for node, _graph in iter_nodes(model.graph): - # Skip custom domains -- the matrix only describes "" / "ai.onnx". - if node.domain and node.domain not in ("", "ai.onnx"): + for node, graph in iter_nodes(model.graph): + if node.domain and node.domain not in _DEFAULT_DOMAINS: + if node.domain not in self._plugin_domains: + custom[(node.domain, node.op_type)] += 1 continue entry = self._ops.get(node.op_type) if entry is None: - continue # Unknown op -- don't blow up on third-party ops + unclassified[node.op_type] += 1 + continue status = entry["support"].get(self._target, "unknown") if status == "not_supported": - issues.append(self._make_issue(node, entry, Severity.CRITICAL)) + issues.append(self._make_issue(node, entry, Severity.CRITICAL, graph.name)) elif status == "partial": - issues.append(self._make_issue(node, entry, Severity.WARNING)) + issues.append(self._make_issue(node, entry, Severity.WARNING, graph.name)) + elif status == "unknown": + unclassified[node.op_type] += 1 + if status in ("supported", "partial"): + issues.extend(self._check_conditions(node, entry, graph.name, constants)) + + issues.extend( + self._unclassified_issue(op, count) for op, count in sorted(unclassified.items()) + ) + issues.extend( + self._custom_domain_issue(domain, op, count) + for (domain, op), count in sorted(custom.items()) + ) return issues + # -- support-status findings ------------------------------------------ + def _make_issue( self, node: onnx.NodeProto, entry: dict[str, Any], severity: Severity, + graph_scope: str, ) -> Issue: op = node.op_type notes = entry.get("notes", "") @@ -80,6 +126,7 @@ def _make_issue( "Replace with an equivalent supported op, write a TRT plugin, " "or remove from the graph if it is dead." ) + rule_id, confidence, verify = RULE_UNSUPPORTED, Confidence.HIGH, False else: limitations = entry.get("limitations", []) lim_str = f" Limitations: {'; '.join(limitations)}." if limitations else "" @@ -92,6 +139,7 @@ def _make_issue( "Check the operator-specific limitations and validate against your " "exported attribute set." ) + rule_id, confidence, verify = RULE_PARTIAL, Confidence.MEDIUM, True return Issue( severity=severity, category=CheckCategory.OPERATOR_SUPPORT, @@ -100,4 +148,205 @@ def _make_issue( message=message, remediation=remediation, docs_link=entry.get("github_issue"), + rule_id=rule_id, + confidence=confidence, + verify_required=verify, + target_trt=self._target, + graph_scope=graph_scope, + ) + + def _unclassified_issue(self, op: str, count: int) -> Issue: + plural = "s" if count != 1 else "" + return Issue( + severity=Severity.INFO, + category=CheckCategory.OPERATOR_SUPPORT, + node_name=f"<{count} node{plural}>", + operator=op, + message=( + f"Operator '{op}' ({count} node{plural}) is not classified in " + f"trtcheck's support matrix for TensorRT {self._target}. It may " + "convert fine, but static analysis has no evidence either way." + ), + remediation=( + "Verify with a real TensorRT parse (trtexec --onnx=model.onnx) " + "or check the onnx-tensorrt supported-operators list for this op." + ), + docs_link="https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md", + rule_id=RULE_UNCLASSIFIED, + confidence=Confidence.LOW, + verify_required=True, + target_trt=self._target, + ) + + def _custom_domain_issue(self, domain: str, op: str, count: int) -> Issue: + plural = "s" if count != 1 else "" + return Issue( + severity=Severity.INFO, + category=CheckCategory.OPERATOR_SUPPORT, + node_name=f"<{count} node{plural}>", + operator=f"{domain}::{op}", + message=( + f"Operator '{op}' ({count} node{plural}) lives in custom domain " + f"'{domain}'. TensorRT needs a plugin that implements it; trtcheck " + "cannot verify plugin availability statically." + ), + remediation=( + "If a TensorRT plugin for this domain is installed, declare it " + "with --plugin-domain to suppress this finding; otherwise " + "implement/register the plugin before converting." + ), + docs_link=( + "https://docs.nvidia.com/deeplearning/tensorrt/latest/" + "inference-library/extending-custom-layers.html" + ), + rule_id=RULE_CUSTOM_DOMAIN, + confidence=Confidence.LOW, + verify_required=True, + target_trt=self._target, + ) + + # -- conditional support ---------------------------------------------- + + def _check_conditions( + self, + node: onnx.NodeProto, + entry: dict[str, Any], + graph_scope: str, + constants: dict[str, int | None], + ) -> list[Issue]: + issues: list[Issue] = [] + for cond in entry.get("conditions", []): + applies = cond.get("applies_to") + if applies is not None and self._target not in applies: + continue + kind = cond.get("kind") + if kind == "attribute_allowed": + verdict = _eval_attribute_allowed(node, cond) + elif kind == "constant_input_max": + verdict = _eval_constant_input_max(node, cond, constants) + else: + # Unknown condition kind: matrix is ahead of the code. Treat as + # unresolvable rather than silently passing. + verdict = "unresolved" + if verdict == "pass": + continue + issues.append(self._condition_issue(node, entry, cond, verdict, graph_scope)) + return issues + + def _condition_issue( + self, + node: onnx.NodeProto, + entry: dict[str, Any], + cond: dict[str, Any], + verdict: str, + graph_scope: str, + ) -> Issue: + op = node.op_type + evidence = cond.get("evidence", {}) + docs = evidence.get("source") or entry.get("github_issue") + if verdict == "violated": + severity = Severity(cond.get("severity", "warning")) + message = ( + f"Operator '{op}' violates a TensorRT {self._target} support " + f"condition: {cond.get('message', cond.get('id', 'condition'))}" + ) + rule_id = RULE_CONDITION + confidence = ( + Confidence.HIGH if evidence.get("status") == "official_docs" else Confidence.MEDIUM + ) + verify = False + else: # unresolved + severity = Severity.INFO + message = ( + f"Operator '{op}' has a TensorRT {self._target} support condition " + f"that static analysis cannot resolve: " + f"{cond.get('message', cond.get('id', 'condition'))}" + ) + rule_id = RULE_CONDITION_UNRESOLVED + confidence = Confidence.LOW + verify = True + return Issue( + severity=severity, + category=CheckCategory.OPERATOR_SUPPORT, + node_name=node.name or f"", + operator=op, + message=message, + remediation=cond.get("remediation") + or "Verify with a real TensorRT parse (trtexec --onnx=model.onnx).", + docs_link=docs, + rule_id=rule_id, + confidence=confidence, + verify_required=verify, + target_trt=self._target, + graph_scope=graph_scope, ) + + +def _collect_scalar_constants(model: onnx.ModelProto) -> dict[str, int | None]: + """Map tensor name -> scalar int value for every single-element integer + initializer or Constant node output in the model. Value is None when the + tensor is constant but not a readable scalar int.""" + from onnx import numpy_helper + + out: dict[str, int | None] = {} + for graph in iter_subgraphs(model.graph): + for init in graph.initializer: + out[init.name] = _scalar_int(numpy_helper, init) + for node in graph.node: + if node.op_type == "Constant" and node.output: + tensor = next( + (a.t for a in node.attribute if a.name == "value" and a.HasField("t")), + None, + ) + out[node.output[0]] = _scalar_int(numpy_helper, tensor) if tensor else None + return out + + +def _scalar_int(numpy_helper: Any, tensor: onnx.TensorProto) -> int | None: + try: + arr = numpy_helper.to_array(tensor) + except Exception: + return None + if arr.size != 1 or arr.dtype.kind not in ("i", "u"): + return None + return int(arr.reshape(())) + + +def _eval_attribute_allowed(node: onnx.NodeProto, cond: dict[str, Any]) -> str: + """'pass' | 'violated'. Checks a node attribute against an allowed set.""" + attr_name = cond["attribute"] + allowed = cond["allowed_values"] + attr = next((a for a in node.attribute if a.name == attr_name), None) + if attr is None: + return "pass" if cond.get("default_ok", True) else "violated" + if attr.type == onnx.AttributeProto.INT: + return "pass" if attr.i in allowed else "violated" + if attr.type == onnx.AttributeProto.STRING: + return "pass" if attr.s.decode(errors="replace") in allowed else "violated" + # Attribute exists but has a type this condition can't compare: be honest. + return "unresolved" + + +def _eval_constant_input_max( + node: onnx.NodeProto, cond: dict[str, Any], constants: dict[str, int | None] +) -> str: + """'pass' | 'violated' | 'unresolved'. + + The input at ``input_index`` must, when statically constant, hold a scalar + integer <= ``max_value``. A non-constant (runtime) input cannot be checked + statically -> unresolved. + """ + idx = cond["input_index"] + if idx >= len(node.input) or not node.input[idx]: + # Optional input absent: nothing to violate. + return "pass" + name = node.input[idx] + if name not in constants: + return "unresolved" + value = constants[name] + if value is None: + return "unresolved" + max_value = cond.get("max_value") + if max_value is not None and value > max_value: + return "violated" + return "pass" diff --git a/trtcheck/checkers/precision.py b/trtcheck/checkers/precision.py index b9d6e74..5dc7f7f 100644 --- a/trtcheck/checkers/precision.py +++ b/trtcheck/checkers/precision.py @@ -90,7 +90,7 @@ def _check_initializers(self, graph: onnx.GraphProto) -> list[Issue]: issues: list[Issue] = [] # Walk subgraph initializers too -- an INT64/DOUBLE weight buried in an # If/Loop/Scan body is just as much of a conversion problem. - for init, _owner in iter_initializers(graph): + for init, owner in iter_initializers(graph): mapping = _INIT_DTYPES.get(init.data_type) if mapping is None: continue @@ -101,6 +101,7 @@ def _check_initializers(self, graph: onnx.GraphProto) -> list[Issue]: node_name=init.name, operator="Initializer", prefix=f"Initializer '{init.name}' has dtype {token}", + graph_scope=owner.name, ) ) return issues diff --git a/trtcheck/cli.py b/trtcheck/cli.py index 6f3d8c5..1cb2c33 100644 --- a/trtcheck/cli.py +++ b/trtcheck/cli.py @@ -11,39 +11,34 @@ from trtcheck import __version__ from trtcheck.analyzer import _BUILTIN_CHECKER_NAMES, Analyzer, AnalyzerConfig, safe_load -from trtcheck.fixers import FixApplied, apply_all, default_fixers -from trtcheck.plugins import Fixer, Reporter, load_plugins +from trtcheck.fixers import ( + default_fixers, + run_fixers, + validate_model, + validation_level_for, +) +from trtcheck.plugins import Reporter, load_plugins from trtcheck.reporters.console import ConsoleReporter from trtcheck.reporters.html import HTMLReporter from trtcheck.reporters.json import JSONReporter -from trtcheck.types import AnalysisReport, Severity +from trtcheck.types import REPORT_SCHEMA_VERSION, AnalysisReport, Severity, Verdict _FORMATS = ["console", "json", "html"] _SEVERITIES = ["critical", "warning", "info"] _KNOWN_TARGETS = ["8.0", "8.6", "10.0", "10.3"] +_FAIL_ON = ["blocked", "unverified"] -class _SafePluginFixer: - """Isolate a third-party fixer the way the analyzer isolates plugin - checkers: a crash is reported on stderr and skipped, never a traceback. - A partial mutation from a crashed fixer is caught downstream by the - onnx.checker validation before anything is written.""" - - def __init__(self, inner: Fixer) -> None: - self._inner = inner - self.name = getattr(inner, "name", inner.__class__.__name__) - - def fix(self, model: onnx.ModelProto) -> list[FixApplied]: - try: - return self._inner.fix(model) - except Exception as exc: - click.echo( - f"warning: plugin fixer {self.name!r} raised " - f"{exc.__class__.__name__}: {exc}; skipped " - "(--disable-plugin to silence)", - err=True, - ) - return [] +# Exit codes (documented in docs/usage.md): +# 0 verdict is LIKELY or VERIFIED (or UNVERIFIED unless --fail-on unverified) +# 1 verdict is BLOCKED, or a fatal CLI error +# 2 usage error (Click) +def _exit_code(report: AnalysisReport, fail_on: str) -> int: + if report.verdict is Verdict.BLOCKED: + return 1 + if fail_on == "unverified" and report.verdict is Verdict.UNVERIFIED: + return 1 + return 0 def _plugin_reporters(disable_plugins: tuple[str, ...] = ()) -> dict[str, Reporter]: @@ -95,6 +90,9 @@ def _filter_issues(report: AnalysisReport, minimum: str) -> AnalysisReport: producer=report.producer, total_nodes=report.total_nodes, issues=kept, + target_trt=report.target_trt, + runtime_verified=report.runtime_verified, + runtime_verification=report.runtime_verification, ) return filtered @@ -219,6 +217,51 @@ def _emit(text: str, output_path: Path | None, force: bool = False) -> None: metavar="NAME", help="Exclude a plugin by its name. May be passed multiple times.", ) +@click.option( + "--plugin-domain", + "plugin_domains", + multiple=True, + metavar="DOMAIN", + help=( + "Declare a custom ONNX domain as backed by an installed TensorRT " + "plugin, suppressing its TRT-OP-CUSTOM-DOMAIN findings." + ), +) +@click.option( + "--fail-on", + type=click.Choice(_FAIL_ON), + default="blocked", + show_default=True, + help=( + "Exit non-zero on this verdict or worse: 'blocked' fails only on " + "known blockers; 'unverified' also fails when unresolved conditions " + "remain." + ), +) +@click.option( + "--verify-runtime", + is_flag=True, + default=False, + help=( + "After static analysis, run 'trtexec --onnx=MODEL' to verify with a " + "real TensorRT build. Requires trtexec (and usually a GPU)." + ), +) +@click.option( + "--trtexec", + "trtexec_path", + type=click.Path(path_type=str), + default=None, + help="Path to the trtexec executable (default: search PATH).", +) +@click.option( + "--verify-timeout", + type=int, + default=600, + show_default=True, + metavar="SECONDS", + help="Timeout for the trtexec run started by --verify-runtime.", +) def main( models: tuple[Path, ...], target_trt: str, @@ -232,6 +275,11 @@ def main( dry_run: bool, list_plugins: bool, disable_plugins: tuple[str, ...], + plugin_domains: tuple[str, ...], + fail_on: str, + verify_runtime: bool, + trtexec_path: str | None, + verify_timeout: int, ) -> None: """Run trtcheck against one or two ONNX models. @@ -260,7 +308,18 @@ def main( path = models[0] if fix_mode: - _run_fix(path, output, force, dry_run, max_model_size, disable_plugins) + _run_fix( + path, + output, + force, + dry_run, + max_model_size, + disable_plugins, + target_trt=target_trt, + fmt=fmt, + plugin_domains=plugin_domains, + fail_on=fail_on, + ) return if not path.exists(): @@ -271,19 +330,43 @@ def main( target_trt=target_trt, max_model_size_mb=max_model_size, disable_plugins=list(disable_plugins), + plugin_domains=list(plugin_domains), ) ) try: report = analyzer.analyze_path(path) except ValueError as exc: raise click.ClickException(str(exc)) from exc - report = _filter_issues(report, severity) - text = _render(report, fmt, color=output is None, disable_plugins=disable_plugins) + if verify_runtime: + _attach_runtime_verification(report, path, trtexec_path, verify_timeout) + + display = _filter_issues(report, severity) + + text = _render(display, fmt, color=output is None, disable_plugins=disable_plugins) _emit(text, output, force=force) - if not report.conversion_likely: - sys.exit(1) + # Exit code comes from the UNfiltered report: --severity only trims the + # display, it must not upgrade an unverified model to a passing one. + sys.exit(_exit_code(report, fail_on)) + + +def _attach_runtime_verification( + report: AnalysisReport, path: Path, trtexec_path: str | None, timeout_s: int +) -> None: + """Run trtexec against `path` and fold the outcome into `report`. + + VERIFIED is only ever set on a successful build with no static blocker; + a runtime failure or an unavailable trtexec leaves the static verdict + untouched (the metadata still records what happened). + """ + from trtcheck.runtime_verify import verify_model + + result = verify_model(path, trtexec_path=trtexec_path, timeout_s=timeout_s) + report.runtime_verification = result.to_dict() + if result.verified and report.verdict is not Verdict.BLOCKED: + report.runtime_verified = True + click.echo(f"runtime verification: {result.status.value} -- {result.detail}", err=True) def _run_diff( @@ -354,7 +437,15 @@ def _run_fix( dry_run: bool, max_model_size: int, disable_plugins: tuple[str, ...] = (), + *, + target_trt: str = "10.3", + fmt: str = "console", + plugin_domains: tuple[str, ...] = (), + fail_on: str = "blocked", ) -> None: + """The --fix pipeline: analyze -> fix transactionally -> validate -> + re-analyze with the same target -> report resolved/remaining/new findings. + """ if not path.exists(): raise click.ClickException(f"ONNX file not found: {path}") size_mb = path.stat().st_size / (1024 * 1024) @@ -368,47 +459,116 @@ def _run_fix( # who forgets --output is told immediately rather than after the report. if not dry_run and output is None: raise click.ClickException("--fix requires --output to write the fixed model") + if output is not None: + if output.resolve() == path.resolve(): + raise click.ClickException( + "refusing to overwrite the input file; choose a different --output" + ) + if not dry_run and output.exists() and not force: + raise click.ClickException( + f"refusing to overwrite existing file: {output} (use --force)" + ) try: model = safe_load(path) except ValueError as exc: raise click.ClickException(str(exc)) from exc - # Built-ins first, then discovered plugin fixers (isolated so a broken - # plugin can't take down the run), minus anything disabled by name. + + # --fix only operates on structurally valid ONNX: fixers correct TensorRT + # incompatibilities, not broken protobufs. + try: + validate_model(model, level="basic") + except Exception as exc: + raise click.ClickException( + f"input model failed ONNX validation; --fix needs a valid model: {exc}" + ) from exc + + analyzer = Analyzer( + AnalyzerConfig( + target_trt=target_trt, + max_model_size_mb=max_model_size, + disable_plugins=list(disable_plugins), + plugin_domains=list(plugin_domains), + ) + ) + before = analyzer.analyze_model(model, filename=str(path)) + + # Built-ins first, then discovered plugin fixers, minus anything disabled + # by name. run_fixers() is transactional: a fixer that crashes or emits an + # invalid model has its changes discarded and later fixers still run. _, plugin_fixers, _ = load_plugins() disabled = set(disable_plugins) fixers = [ - f - for f in [*default_fixers(), *(_SafePluginFixer(pf) for pf in plugin_fixers)] - if getattr(f, "name", "") not in disabled + f for f in [*default_fixers(), *plugin_fixers] if getattr(f, "name", "") not in disabled ] - new_model, applied = apply_all(model, fixers) + outcome = run_fixers(model, fixers) - if not applied: - click.echo("no fixes applied -- model unchanged") - return + for failure in outcome.failures: + click.echo(f"warning: fixer {failure.fixer!r}: {failure.reason}", err=True) - for fix in applied: - click.echo(f" [{fix.fixer}] {fix.description}") + after = analyzer.analyze_model( + outcome.model, filename=str(output) if output else f"{path} (fixed)" + ) + before_ids = {i.identity() for i in before.issues} + after_ids = {i.identity() for i in after.issues} + resolved = [i for i in before.issues if i.identity() not in after_ids] + remaining = [i for i in after.issues if i.identity() in before_ids] + introduced = [i for i in after.issues if i.identity() not in before_ids] + + if fmt == "json": + payload = { + "schema_version": REPORT_SCHEMA_VERSION, + "target_trt": target_trt, + "input": str(path), + "output": str(output) if output else None, + "dry_run": dry_run, + "validation": outcome.validation, + "fixes_applied": [f.to_dict() for f in outcome.applied], + "fixer_failures": [f.to_dict() for f in outcome.failures], + "resolved": [i.to_dict() for i in resolved], + "remaining": [i.to_dict() for i in remaining], + "introduced": [i.to_dict() for i in introduced], + "verdict_before": before.verdict.value, + "verdict_after": after.verdict.value, + } + click.echo(json.dumps(payload, indent=2)) + else: + if not outcome.applied and not outcome.failures: + click.echo("no fixes applied -- model unchanged") + for fix in outcome.applied: + click.echo(f" [{fix.fixer}] {fix.description}") + click.echo( + f"\nverdict: {before.verdict.value} -> {after.verdict.value} " + f"(TensorRT {target_trt}); " + f"{len(resolved)} finding(s) resolved, {len(remaining)} remaining, " + f"{len(introduced)} introduced" + ) + for issue in introduced: + click.echo(f" introduced: [{issue.rule_id}] {issue.message}") + + if not outcome.applied: + # Nothing changed; never write an output file that is byte-identical + # in content but pretends to be "fixed". + if not dry_run: + click.echo("nothing to write -- no fixer made a change", err=True) + sys.exit(_exit_code(after, fail_on)) if dry_run: - click.echo(f"\n{len(applied)} fix(es) would be applied (dry run).") - return + if fmt != "json": + click.echo(f"\n{len(outcome.applied)} fix(es) would be applied (dry run).") + sys.exit(_exit_code(after, fail_on)) - if output is None: # unreachable: guarded above, but keep mypy + -O happy - raise click.ClickException("--fix requires --output to write the fixed model") - if output.resolve() == path.resolve(): - raise click.ClickException( - "refusing to overwrite the input file; choose a different --output" - ) - if output.exists() and not force: - raise click.ClickException(f"refusing to overwrite existing file: {output} (use --force)") + assert output is not None # guarded above + # run_fixers validated every committed candidate; validate once more at + # the write boundary as a belt-and-braces invariant. try: - onnx.checker.check_model(new_model) - except Exception as exc: # onnx ValidationError et al. + validate_model(outcome.model, level=outcome.validation) + except Exception as exc: raise click.ClickException(f"applying fixes produced an invalid ONNX model: {exc}") from exc - onnx.save(new_model, str(output)) - click.echo(f"\n{len(applied)} fix(es) applied. Wrote {output}.") + onnx.save(outcome.model, str(output)) + if fmt != "json": + click.echo(f"\n{len(outcome.applied)} fix(es) applied. Wrote {output}.") + sys.exit(_exit_code(after, fail_on)) def _print_plugin_listing(target_trt: str, max_model_size: int, disable_plugins: list[str]) -> None: diff --git a/trtcheck/data/operator_matrix.json b/trtcheck/data/operator_matrix.json index 17fb858..86febd1 100644 --- a/trtcheck/data/operator_matrix.json +++ b/trtcheck/data/operator_matrix.json @@ -1,6 +1,6 @@ { - "schema_version": "1.0", - "last_updated": "2026-05-21", + "schema_version": "2.0", + "last_updated": "2026-07-22", "target_trt_versions": [ "8.0", "8.6", @@ -146,6 +146,15 @@ "10.3": "supported" } }, + "Clip": { + "support": { + "8.0": "unknown", + "8.6": "unknown", + "10.0": "supported", + "10.3": "supported" + }, + "notes": "Supported in TRT 10.x (FP32/FP16/BF16) per onnx-tensorrt docs, retrieved 2026-07-22. 8.x status not verified here, left unknown." + }, "LogSoftmax": { "support": { "8.0": "supported", @@ -576,9 +585,80 @@ "10.0": "supported", "10.3": "supported" }, - "notes": "Only nearest and linear modes pre-10.0. Cubic added in 10.0.", + "notes": "Only nearest and linear modes; cubic is not supported (onnx-tensorrt docs, retrieved 2026-07-22).", "limitations": [ - "antialias attribute not supported before TRT 10.0." + "Antialiasing (antialias=1) is not supported.", + "coordinate_transformation_mode limited to half_pixel, pytorch_half_pixel, tf_half_pixel_for_nn, asymmetric, align_corners." + ], + "conditions": [ + { + "id": "resize-mode", + "applies_to": [ + "10.0", + "10.3" + ], + "kind": "attribute_allowed", + "attribute": "mode", + "allowed_values": [ + "nearest", + "linear" + ], + "default_ok": true, + "severity": "critical", + "message": "TensorRT supports only Resize modes 'nearest' and 'linear' (cubic is rejected).", + "remediation": "Re-export with mode=nearest or mode=linear, or implement cubic resize as a plugin.", + "evidence": { + "status": "official_docs", + "source": "https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md", + "retrieved": "2026-07-22" + } + }, + { + "id": "resize-coord-transform", + "applies_to": [ + "10.0", + "10.3" + ], + "kind": "attribute_allowed", + "attribute": "coordinate_transformation_mode", + "allowed_values": [ + "half_pixel", + "pytorch_half_pixel", + "tf_half_pixel_for_nn", + "asymmetric", + "align_corners" + ], + "default_ok": true, + "severity": "critical", + "message": "TensorRT supports Resize coordinate_transformation_mode in {half_pixel, pytorch_half_pixel, tf_half_pixel_for_nn, asymmetric, align_corners} only.", + "remediation": "Re-export with a supported coordinate_transformation_mode.", + "evidence": { + "status": "official_docs", + "source": "https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md", + "retrieved": "2026-07-22" + } + }, + { + "id": "resize-no-antialias", + "applies_to": [ + "10.0", + "10.3" + ], + "kind": "attribute_allowed", + "attribute": "antialias", + "allowed_values": [ + 0 + ], + "default_ok": true, + "severity": "critical", + "message": "TensorRT does not support antialiased Resize (antialias=1).", + "remediation": "Re-export with antialias=0.", + "evidence": { + "status": "official_docs", + "source": "https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md", + "retrieved": "2026-07-22" + } + } ] }, "Upsample": { @@ -754,7 +834,48 @@ "8.6": "supported", "10.0": "supported", "10.3": "supported" - } + }, + "conditions": [ + { + "id": "topk-sorted-required", + "applies_to": [ + "10.0", + "10.3" + ], + "kind": "attribute_allowed", + "attribute": "sorted", + "allowed_values": [ + 1 + ], + "default_ok": true, + "severity": "critical", + "message": "TensorRT requires TopK attribute sorted=1 (ONNX default). sorted=0 is rejected.", + "remediation": "Re-export with sorted=1, or sort outside the model.", + "evidence": { + "status": "official_docs", + "source": "https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md", + "retrieved": "2026-07-22" + } + }, + { + "id": "topk-k-max-3840", + "applies_to": [ + "10.0", + "10.3" + ], + "kind": "constant_input_max", + "input_index": 1, + "max_value": 3839, + "severity": "critical", + "message": "TensorRT requires the TopK K input to be less than 3840.", + "remediation": "Reduce K below 3840, or restructure the selection into chunks.", + "evidence": { + "status": "official_docs", + "source": "https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md", + "retrieved": "2026-07-22" + } + } + ] }, "NonZero": { "support": { diff --git a/trtcheck/data/remediation_db.json b/trtcheck/data/remediation_db.json index bb856ad..c6e6252 100644 --- a/trtcheck/data/remediation_db.json +++ b/trtcheck/data/remediation_db.json @@ -1,177 +1,246 @@ { - "schema_version": "1.0", + "schema_version": "2.0", "last_updated": "2026-05-21", "issues": { "int64_weights": { + "rule_id": "TRT-DTYPE-INT64-WEIGHTS", "category": "precision", "severity": "warning", "summary": "INT64 tensors detected in graph constants or weights.", "explanation": "TensorRT does not natively support INT64. Engine build will cast to INT32, which can overflow for large indices.", "remediation": "Before exporting, cast indices to torch.int32: e.g. `idx = idx.to(torch.int32)`. For embedding lookups, ensure the indices tensor is int32.", - "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#layers-precision" + "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#layers-precision", + "confidence": "high", + "verify_required": false }, "int64_input": { + "rule_id": "TRT-DTYPE-INT64-INPUT", "category": "precision", "severity": "warning", "summary": "INT64 graph input detected.", "explanation": "TensorRT does not natively support INT64. Engine build will cast INT64 inputs to INT32, which can overflow for large index values.", "remediation": "Cast the input to int32 before exporting, or feed int32 tensors at inference time. For embedding/gather indices, ensure they are int32.", - "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#layers-precision" + "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#layers-precision", + "confidence": "high", + "verify_required": false }, "uint8_input": { + "rule_id": "TRT-DTYPE-UINT8-INPUT", "category": "precision", "severity": "critical", "summary": "UINT8 graph input detected.", "explanation": "TensorRT inputs must be FLOAT32, FLOAT16, INT32, or INT8. UINT8 image tensors are common in preprocessing but must be cast before reaching the model.", "remediation": "Apply the UINT8 -> FLOAT32 conversion (and normalization) in your preprocessing pipeline, not inside the exported model.", - "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#network_data_types" + "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#network_data_types", + "confidence": "high", + "verify_required": false }, "bf16_unsupported": { + "rule_id": "TRT-DTYPE-BF16", "category": "precision", "severity": "warning", "summary": "BFLOAT16 tensors present.", "explanation": "BF16 is supported only on Ampere+ GPUs and TRT 8.6+. On older targets the build fails or silently promotes to FP32.", "remediation": "If targeting older GPUs or TRT 8.0, export the model in FP16 instead of BF16.", - "docs_link": null + "docs_link": null, + "confidence": "medium", + "verify_required": true }, "float64_tensors": { + "rule_id": "TRT-DTYPE-FP64", "category": "precision", "severity": "critical", "summary": "FLOAT64 (double) tensors detected.", "explanation": "TensorRT does not support double precision (FLOAT64). Models trained or exported with float64 must be downcast before conversion.", "remediation": "Call `model = model.float()` before torch.onnx.export, and verify with `onnx.checker` that no FLOAT64 remains.", - "docs_link": null + "docs_link": null, + "confidence": "high", + "verify_required": false }, "string_tensors": { + "rule_id": "TRT-DTYPE-STRING", "category": "precision", "severity": "critical", "summary": "String tensors detected.", "explanation": "TensorRT has no string type. Models that process raw strings (tokenizers, label encoders) cannot be deployed end-to-end with TRT.", "remediation": "Move string preprocessing out of the model and pass integer token IDs instead.", - "docs_link": null + "docs_link": null, + "confidence": "high", + "verify_required": false }, "fully_dynamic_input_shape": { + "rule_id": "TRT-SHAPE-PROFILE-MISSING", "category": "dynamic_shapes", "severity": "warning", "summary": "Input tensor has every dimension dynamic.", "explanation": "TRT can build dynamic engines, but optimizer cannot estimate memory or fuse layers well when every dimension is symbolic.", "remediation": "Pin spatial dims at export: pass dynamic_axes={'input': {0: 'batch'}} (or whatever truly varies) and leave H,W concrete.", - "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#work_dynamic_shapes" + "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#work_dynamic_shapes", + "confidence": "medium", + "verify_required": true }, "missing_output": { + "rule_id": "TRT-GRAPH-NO-OUTPUT", "category": "graph_structure", "severity": "critical", "summary": "Graph has no declared outputs.", "explanation": "TensorRT refuses to build engines for graphs with zero outputs. Typically caused by aggressive constant-folding or shape inference failure.", - "remediation": "Inspect with `onnx.checker.check_model`. Re-export with do_constant_folding=False to localize the issue." + "remediation": "Inspect with `onnx.checker.check_model`. Re-export with do_constant_folding=False to localize the issue.", + "confidence": "high", + "verify_required": false }, "isolated_node": { + "rule_id": "TRT-GRAPH-ISOLATED-NODE", "category": "graph_structure", "severity": "warning", "summary": "Node has no consumers downstream.", "explanation": "Disconnected subgraphs are silently dropped by TRT, which sometimes hides bugs in the exported graph (e.g. an inplace side effect that ONNX could not express).", - "remediation": "Confirm the orphan node represents intentional dead code. If not, audit the forward() method for unsupported in-place ops." + "remediation": "Confirm the orphan node represents intentional dead code. If not, audit the forward() method for unsupported in-place ops.", + "confidence": "medium", + "verify_required": false }, "duplicate_node_name": { + "rule_id": "TRT-GRAPH-DUP-NODE-NAME", "category": "graph_structure", "severity": "warning", "summary": "Multiple nodes share the same name.", "explanation": "ONNX permits non-unique node names but several TRT tooling paths (engine introspection, layer-precision overrides) assume uniqueness.", - "remediation": "Set unique node names during export, or run an ONNX simplifier pass before TRT conversion." + "remediation": "Set unique node names during export, or run an ONNX simplifier pass before TRT conversion.", + "confidence": "high", + "verify_required": false }, "loop_runtime_trip_count": { + "rule_id": "TRT-CONTROL-LOOP-RUNTIME-TRIP", "category": "control_flow", "severity": "critical", "summary": "Loop trip count fed from a graph input.", "explanation": "TensorRT requires a statically known or shape-inferable trip count for Loop. A trip count read from a graph input is runtime-dynamic by construction and fails at engine build time.", "remediation": "Refactor the loop to use a fixed iteration count baked into the graph. If the count depends on input size, consider unrolling or a fully-vectorized formulation.", - "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#loops" + "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#loops", + "confidence": "high", + "verify_required": false }, "loop_dynamic_trip_count": { + "rule_id": "TRT-CONTROL-LOOP-DYNAMIC-TRIP", "category": "control_flow", "severity": "warning", "summary": "Loop op with non-constant trip count.", "explanation": "TensorRT requires a statically known or shape-inferable trip count for Loop. This trip count is computed inside the graph; TRT may still infer it, but if it depends on runtime tensor values the engine build fails.", "remediation": "Refactor the loop to use a fixed iteration count. If the count depends on input size, consider unrolling or a fully-vectorized formulation.", - "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#loops" + "docs_link": "https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#loops", + "confidence": "low", + "verify_required": true }, "nested_loop": { + "rule_id": "TRT-CONTROL-LOOP-NESTED", "category": "control_flow", "severity": "critical", "summary": "Nested Loop nodes detected.", "explanation": "TensorRT does not support Loop-within-Loop. Inner loops must be unrolled or flattened.", - "remediation": "Restructure the model to use a single Loop with combined trip count, or fully unroll inner iterations." + "remediation": "Restructure the model to use a single Loop with combined trip count, or fully unroll inner iterations.", + "confidence": "high", + "verify_required": false }, "if_branch_shape_mismatch": { + "rule_id": "TRT-CONTROL-IF-SHAPE-MISMATCH", "category": "control_flow", "severity": "critical", "summary": "If branches produce outputs with incompatible shapes or dtypes.", "explanation": "TRT requires both then/else branches to emit identically shaped, identically typed tensors.", - "remediation": "Make the two branches produce the same output shape; pad or broadcast the smaller branch to match." + "remediation": "Make the two branches produce the same output shape; pad or broadcast the smaller branch to match.", + "confidence": "high", + "verify_required": false }, "if_detected_unverified": { + "rule_id": "TRT-CONTROL-IF-UNVERIFIED", "category": "control_flow", "severity": "warning", "summary": "If node detected; branch compatibility not verified.", "explanation": "TRT requires both then/else branches to emit identically shaped, identically typed tensors. trtcheck cannot confirm this without full shape inference, so every If is flagged for manual review.", - "remediation": "Verify both branches return tensors of matching shape and dtype. If they differ, pad or broadcast the smaller branch to match before exporting." + "remediation": "Verify both branches return tensors of matching shape and dtype. If they differ, pad or broadcast the smaller branch to match before exporting.", + "confidence": "low", + "verify_required": true }, "scan_dynamic_length": { + "rule_id": "TRT-CONTROL-SCAN", "category": "control_flow", "severity": "warning", "summary": "Scan op with dynamic sequence length.", "explanation": "TRT requires Scan sequence length to be known at engine build time.", - "remediation": "Fix the scan length at export, or replace with a Loop over a constant range." + "remediation": "Fix the scan length at export, or replace with a Loop over a constant range.", + "confidence": "low", + "verify_required": true }, "unsupported_operator": { + "rule_id": "TRT-OP-UNSUPPORTED", "category": "operator_support", "severity": "critical", "summary": "Operator with no TRT implementation.", "explanation": "This op type has no native TensorRT layer. Engine build will fail.", "remediation": "Replace with an equivalent supported op, write a TRT plugin, or remove from the graph if it is dead.", - "docs_link": "https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md" + "docs_link": "https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md", + "confidence": "high", + "verify_required": false }, "partial_operator": { + "rule_id": "TRT-OP-PARTIAL", "category": "operator_support", "severity": "warning", "summary": "Operator supported with restrictions.", "explanation": "TRT supports this op only for certain attribute or dtype combinations.", - "remediation": "Check the operator-specific notes in the report for the exact restriction." + "remediation": "Check the operator-specific notes in the report for the exact restriction.", + "confidence": "medium", + "verify_required": true }, "large_constant": { + "rule_id": "TRT-GRAPH-LARGE-CONSTANT", "category": "graph_structure", "severity": "info", "summary": "Large Constant node (>10MB).", "explanation": "TRT will lift large constants into engine weights, which can balloon engine file size if the constant is per-batch data instead of a learned weight.", - "remediation": "Verify the constant is genuinely a learned weight rather than e.g. a baked-in image." + "remediation": "Verify the constant is genuinely a learned weight rather than e.g. a baked-in image.", + "confidence": "medium", + "verify_required": false }, "opset_too_old": { + "rule_id": "TRT-OPSET-OLD", "category": "graph_structure", "severity": "warning", "summary": "ONNX opset version below recommended floor.", "explanation": "TRT 10.x has best coverage for opset 17+; older opsets may force conservative decompositions.", - "remediation": "Re-export with opset_version=17 (or the highest your training framework supports)." + "remediation": "Re-export with opset_version=17 (or the highest your training framework supports).", + "confidence": "medium", + "verify_required": false }, "external_data_missing": { + "rule_id": "TRT-GRAPH-EXTERNAL-DATA", "category": "graph_structure", "severity": "critical", "summary": "Model references external weight files that were not found.", "explanation": "ONNX supports storing weights in separate files. If they are missing at analysis time TRT will also fail to load them.", - "remediation": "Ensure all sibling .data files are present, or re-export with save_as_external_data=False." + "remediation": "Ensure all sibling .data files are present, or re-export with save_as_external_data=False.", + "confidence": "high", + "verify_required": false }, "input_with_no_type": { + "rule_id": "TRT-GRAPH-INPUT-UNTYPED", "category": "graph_structure", "severity": "critical", "summary": "Graph input declares no tensor element type.", "explanation": "TRT requires every input to declare its element type.", - "remediation": "Re-export the model; this typically indicates a corrupted ONNX file." + "remediation": "Re-export the model; this typically indicates a corrupted ONNX file.", + "confidence": "high", + "verify_required": false }, "in_place_aliasing": { + "rule_id": "TRT-GRAPH-ALIASING", "category": "graph_structure", "severity": "warning", "summary": "Heuristic detection of in-place aliasing artifacts.", "explanation": "PyTorch in-place operations (e.g. relu_) sometimes survive ONNX export as redundant Identity chains, which can confuse TRT shape inference.", - "remediation": "Replace in-place activations with their out-of-place counterparts." + "remediation": "Replace in-place activations with their out-of-place counterparts.", + "confidence": "medium", + "verify_required": false } } } diff --git a/trtcheck/fixers/__init__.py b/trtcheck/fixers/__init__.py index 923562c..43a7246 100644 --- a/trtcheck/fixers/__init__.py +++ b/trtcheck/fixers/__init__.py @@ -11,6 +11,8 @@ from __future__ import annotations import copy +import os +import traceback from dataclasses import dataclass from typing import Any @@ -57,20 +59,163 @@ def sync_value_info_dtype(graph: onnx.GraphProto, name: str, elem_type: int) -> value_info.type.tensor_type.elem_type = elem_type +@dataclass +class FixFailure: + """A fixer that was rejected by the transactional pipeline. + + ``reason`` is a short human-readable explanation (no traceback -- set + ``TRTCHECK_DEBUG=1`` for those). The failing fixer's changes were + discarded; the model the pipeline returns never contains them. + """ + + fixer: str + reason: str + + def to_dict(self) -> dict[str, Any]: + return {"fixer": self.fixer, "reason": self.reason} + + +@dataclass +class FixOutcome: + """Result of a transactional :func:`run_fixers` pass.""" + + model: onnx.ModelProto + applied: list[FixApplied] + failures: list[FixFailure] + validation: str = "full" # "full" | "basic" (external-data models) + + +def uses_external_data(model: onnx.ModelProto) -> bool: + """True when any initializer stores its payload outside the protobuf.""" + from trtcheck._graph import iter_initializers + + return any( + init.data_location == onnx.TensorProto.EXTERNAL + for init, _ in iter_initializers(model.graph) + ) + + +def validate_model(model: onnx.ModelProto, *, level: str = "full") -> None: + """Validate a (candidate) model, raising on any problem. + + ``level="full"`` runs ``onnx.checker.check_model(..., full_check=True)``, + which includes strict shape/type inference -- this is what catches a + dtype rewrite that basic checking waves through (e.g. an INT32 tensor + feeding an input whose schema demands INT64). ``level="basic"`` is the + fallback for external-data models, where full inference cannot read the + tensor payloads safely from an in-memory proto. ``level="none"`` skips + validation entirely (used only when the input model itself cannot pass + the basic check -- a candidate is never held to a bar the input missed). + """ + if level == "none": + return + onnx.checker.check_model(model, full_check=(level == "full")) + + +def validation_level_for(model: onnx.ModelProto) -> str: + """Choose the strongest validation level the *input* model already passes. + + A candidate is only ever held to a bar the pre-fix model could meet -- + otherwise a pre-existing quirk (custom-domain ops that defeat strict + shape inference, external-data initializers) would make every fix look + like the fixer's failure. Returns ``"full"``, ``"basic"``, or ``"none"`` + (input fails even the basic check; the CLI refuses to --fix such models, + but library callers may still run fixers on them at their own risk). + """ + if uses_external_data(model): + level_ceiling = "basic" + else: + try: + validate_model(model, level="full") + return "full" + except Exception: + level_ceiling = "basic" + try: + validate_model(model, level="basic") + except Exception: + return "none" + return level_ceiling + + +def run_fixers( + model: onnx.ModelProto, + fixers: list[Fixer], + *, + validate: bool = True, +) -> FixOutcome: + """Run every fixer transactionally against a deep-copied candidate. + + Invariants: + - the input ``model`` is never mutated; + - each fixer runs against a fresh deep copy of the last *valid* model, + so a fixer that mutates and then crashes (or produces an invalid + model, or returns malformed records) cannot leak partial edits; + - one failed fixer does not stop later fixers -- they run against the + last valid state. + """ + current = copy.deepcopy(model) + level = validation_level_for(model) + applied: list[FixApplied] = [] + failures: list[FixFailure] = [] + for fixer in fixers: + name = getattr(fixer, "name", fixer.__class__.__name__) + candidate = copy.deepcopy(current) + try: + fixes = fixer.fix(candidate) + except Exception as exc: + reason = f"raised {exc.__class__.__name__}: {exc}; changes discarded" + # Tracebacks (possibly from third-party plugins) are opt-in only. + if os.environ.get("TRTCHECK_DEBUG", "") not in ("", "0"): + reason += "\n" + traceback.format_exc() + failures.append(FixFailure(name, reason)) + continue + if not isinstance(fixes, list) or not all(isinstance(f, FixApplied) for f in fixes): + failures.append(FixFailure(name, "returned malformed fix records; changes discarded")) + continue + if not fixes: + # Fixer claims it changed nothing: discard the candidate anyway + # (an undeclared mutation must not survive). + continue + if candidate.SerializeToString() == current.SerializeToString(): + # Fixer claims fixes but changed nothing: refusing keeps the + # applied-fixes list truthful (a --fix report must never list a + # change that is not in the output model). + failures.append( + FixFailure( + name, + f"reported {len(fixes)} fix(es) but did not modify the model; " + "records discarded", + ) + ) + continue + if validate: + try: + validate_model(candidate, level=level) + except Exception as exc: + failures.append( + FixFailure( + name, + f"produced an invalid model ({exc.__class__.__name__}); " + "changes discarded", + ) + ) + continue + current = candidate + applied.extend(fixes) + return FixOutcome(model=current, applied=applied, failures=failures, validation=level) + + def apply_all( model: onnx.ModelProto, fixers: list[Fixer], ) -> tuple[onnx.ModelProto, list[FixApplied]]: - """Deep-copy `model`, apply every fixer in order, return the new model. + """Deep-copy `model`, apply every fixer transactionally, return the new model. - The original `model` is never mutated. Useful when the caller wants to - keep both the before and after for diffing. + The original `model` is never mutated. Kept for API compatibility; + :func:`run_fixers` additionally reports per-fixer failures. """ - new_model = copy.deepcopy(model) - applied: list[FixApplied] = [] - for fixer in fixers: - applied.extend(fixer.fix(new_model)) - return new_model, applied + outcome = run_fixers(model, fixers) + return outcome.model, outcome.applied def default_fixers() -> list[Fixer]: @@ -93,7 +238,13 @@ def default_fixers() -> list[Fixer]: __all__ = [ "Fixer", "FixApplied", + "FixFailure", + "FixOutcome", "apply_all", + "run_fixers", + "validate_model", + "validation_level_for", + "uses_external_data", "default_fixers", "sync_value_info_dtype", ] diff --git a/trtcheck/fixers/drop_dropout.py b/trtcheck/fixers/drop_dropout.py index 2d7e2ce..d640af2 100644 --- a/trtcheck/fixers/drop_dropout.py +++ b/trtcheck/fixers/drop_dropout.py @@ -7,15 +7,27 @@ The fixer refuses when the Dropout emits a mask output that is referenced elsewhere -- the mask is a real value some models use, not just training noise. + +Removal is inference-semantics-preserving ONLY when the node is provably in +inference mode. Opset >= 12 Dropout takes an optional third ``training_mode`` +input: absent or statically-false means inference (identity); true, dynamic, +or unresolvable means the node's behavior is not the identity and it must be +left alone. Opset <= 6 Dropout carries an ``is_test`` attribute with the same +role (default 0 = training). """ from __future__ import annotations +import logging + import onnx +from onnx import numpy_helper from trtcheck._graph import iter_nodes, iter_subgraphs from trtcheck.fixers import FixApplied +_logger = logging.getLogger("trtcheck.fixers") + class DropDropoutFixer: name = "drop_dropout" @@ -32,12 +44,26 @@ def fix(self, model: onnx.ModelProto) -> list[FixApplied]: def _fix_graph(self, model: onnx.ModelProto, graph: onnx.GraphProto) -> list[FixApplied]: applied: list[FixApplied] = [] + default_opset = max( + (o.version for o in model.opset_import if o.domain in ("", "ai.onnx")), + default=0, + ) # Walk a copy of the node list because we mutate graph.node mid-loop. for node in list(graph.node): if node.op_type != "Dropout": continue + # Only remove a Dropout that is provably in inference mode. + mode = _resolve_inference_mode(model, node, default_opset) + if mode != "inference": + _logger.info( + "drop_dropout: skipping '%s': training mode is %s", + node.name or "", + mode, + ) + continue + # The data output is always output[0]. Outputs[1:] are the mask # (opset 12+); if any of them is referenced anywhere in the model # (including other scopes), refuse. @@ -97,6 +123,77 @@ def _fix_graph(self, model: onnx.ModelProto, graph: onnx.GraphProto) -> list[Fix return applied +def _resolve_inference_mode( + model: onnx.ModelProto, node: onnx.NodeProto, default_opset: int +) -> str: + """Classify a Dropout node's training mode. + + Returns ``"inference"`` when removal preserves inference semantics, + otherwise a short reason string ("training", "dynamic", "ambiguous", ...) + used for the skip log. + """ + if default_opset != 0 and default_opset <= 6: + # Opset <= 6: is_test attribute, default 0 (training behavior). + is_test = next((a.i for a in node.attribute if a.name == "is_test"), 0) + return "inference" if is_test == 1 else "training (is_test != 1)" + + # Opset 7-11 Dropout has no training switch: inference semantics are the + # identity. Opset >= 12 adds the optional training_mode input. + if len(node.input) < 3 or not node.input[2]: + return "inference" + + name = node.input[2] + value, reason = _resolve_static_bool(model, name) + if value is None: + return reason + return "inference" if value is False else "training (training_mode=true)" + + +def _resolve_static_bool(model: onnx.ModelProto, name: str) -> tuple[bool | None, str]: + """Resolve ``name`` to a static scalar bool if it is an initializer or a + Constant node output. Returns (value, reason-if-unresolvable). + + A name produced in more than one scope is ambiguous (shadowing) and is + refused rather than guessed at. + """ + initializers: list[onnx.TensorProto] = [] + producer_nodes: list[onnx.NodeProto] = [] + for graph in iter_subgraphs(model.graph): + for init in graph.initializer: + if init.name == name: + initializers.append(init) + for n in graph.node: + if name in n.output: + producer_nodes.append(n) + + total_defs = len(initializers) + len(producer_nodes) + if total_defs > 1: + return None, "ambiguous (name defined in multiple scopes)" + if total_defs == 0: + return None, "dynamic (fed from a graph input or unresolved name)" + + tensor: onnx.TensorProto | None + if initializers: + tensor = initializers[0] + else: + producer = producer_nodes[0] + if producer.op_type != "Constant": + return None, "dynamic (produced by a non-Constant node)" + tensor = next( + (a.t for a in producer.attribute if a.name == "value" and a.HasField("t")), + None, + ) + if tensor is None: + return None, "Constant producer carries no tensor value" + try: + arr = numpy_helper.to_array(tensor) + except Exception: + return None, "unreadable training_mode tensor" + if arr.size != 1: + return None, "training_mode is not a scalar" + return bool(arr.reshape(())), "" + + def _referenced_in_model(model: onnx.ModelProto, name: str) -> bool: """True if `name` is a graph output of any scope or consumed as an input by any node anywhere in the model. (A Dropout never feeds itself, so there is diff --git a/trtcheck/fixers/int64_to_int32.py b/trtcheck/fixers/int64_to_int32.py index b397f6a..af1b3b9 100644 --- a/trtcheck/fixers/int64_to_int32.py +++ b/trtcheck/fixers/int64_to_int32.py @@ -1,66 +1,170 @@ -"""Cast INT64 initializers to INT32 where values fit. +"""Cast INT64 initializers to INT32 -- only where every use provably allows it. -TensorRT does not natively support INT64 -- it casts to INT32 at engine -build time. Doing the cast at ONNX-rewrite time has two advantages: +TensorRT casts INT64 to INT32 at engine build time. Doing it at ONNX-rewrite +time surfaces overflow early -- but it is only legal where the consuming +operator's schema accepts INT32 at that input. Many ONNX inputs *require* +INT64: ``Reshape``'s ``shape``, ``Slice``'s ``starts``/``ends``, ``Squeeze``'s +``axes``, ``Pad``'s ``pads``, ``Tile``'s ``repeats``, ... Blindly converting +those produces a model that passes the shallow checker but fails full type +inference (and would fail in any conforming runtime). - 1. It surfaces overflow at fix time rather than during engine build. - 2. It shaves a few bytes per element off the engine binary. +This fixer is therefore use-aware and conservative: -The fixer refuses to act if any value falls outside INT32 range. + - Every use of the initializer, across the whole model including nested + If/Loop/Scan bodies (subgraphs may capture outer-scope names), must be at + a consumer input position on the explicit allowlist below -- positions + whose ONNX type constraint admits ``tensor(int32)`` *independently* of + the operator's other inputs and outputs. + - Positions like elementwise ``Add``/``Mul`` operands are deliberately NOT + allowlisted even though int32 is an allowed dtype there: their type + variable binds several inputs/outputs at once, so retyping one operand + breaks the binding. + - A name defined in more than one scope (shadowing), consumed by a + custom-domain or unknown node, exposed as a graph input/output, or not + consumed at all is skipped. + - Values outside INT32 range and empty tensors are skipped. + +No speculative Cast nodes are ever inserted to force a conversion through. +Skipped conversions are logged at INFO level on the ``trtcheck.fixers`` +logger. """ from __future__ import annotations +import logging + import numpy as np import onnx from onnx import TensorProto, numpy_helper -from trtcheck._graph import iter_subgraphs -from trtcheck.fixers import FixApplied, sync_value_info_dtype +from trtcheck._graph import iter_nodes, iter_subgraphs +from trtcheck.fixers import FixApplied + +_logger = logging.getLogger("trtcheck.fixers") _INT32_MIN = -(2**31) _INT32_MAX = 2**31 - 1 +# (op_type, input_index) positions whose ONNX schema accepts tensor(int32) +# through a type variable that binds ONLY that input (so retyping it cannot +# break a binding with another input or output): +# - Gather / GatherElements / ScatterElements ``indices`` use the dedicated +# Tind constraint = {tensor(int32), tensor(int64)}. +# - Cast input 0 accepts any tensor type; its output type comes from the +# ``to`` attribute, unchanged by this rewrite. +# - Shape / Size input 0 accept any tensor type and always emit INT64; +# the input dtype cannot leak anywhere. +_INT32_SAFE_POSITIONS: frozenset[tuple[str, int]] = frozenset( + { + ("Gather", 1), + ("GatherElements", 1), + ("ScatterElements", 1), + ("Cast", 0), + ("Shape", 0), + ("Size", 0), + } +) + +_DEFAULT_DOMAINS = ("", "ai.onnx") + class Int64ToInt32Fixer: name = "int64_to_int32" def fix(self, model: onnx.ModelProto) -> list[FixApplied]: applied: list[FixApplied] = [] - # Descend into If/Loop/Scan subgraphs: an INT64 weight buried in a - # branch body blocks conversion just the same. + uses = _collect_uses(model) + definition_counts = _definition_counts(model) for graph in iter_subgraphs(model.graph): - applied.extend(self._fix_graph(graph)) + boundary_names = {vi.name for vi in graph.input} | {vi.name for vi in graph.output} + for init in graph.initializer: + if init.data_type != TensorProto.INT64: + continue + reason = self._skip_reason(init, uses, definition_counts, boundary_names) + if reason is not None: + _logger.info("int64_to_int32: skipping '%s': %s", init.name, reason) + continue + arr = numpy_helper.to_array(init) + new_init = numpy_helper.from_array(arr.astype(np.int32), name=init.name) + init.CopyFrom(new_init) + applied.append( + FixApplied( + fixer=self.name, + target=init.name, + description=( + f"cast initializer '{init.name}' from INT64 to INT32 " + f"({arr.size} elements, range " + f"[{int(arr.min())}, {int(arr.max())}]); all uses are " + "at INT32-compatible input positions" + ), + ) + ) return applied - def _fix_graph(self, graph: onnx.GraphProto) -> list[FixApplied]: - applied: list[FixApplied] = [] - for init in graph.initializer: - if init.data_type != TensorProto.INT64: - continue - arr = numpy_helper.to_array(init) - if arr.size == 0: - # Empty initializer is trivially in range; casting is a no-op. - # Skip it -- arr.min()/arr.max() on a zero-size array raises. - continue - if arr.min() < _INT32_MIN or arr.max() > _INT32_MAX: - # Out-of-range; the user must handle this manually. - continue - new_arr = arr.astype(np.int32) - new_init = numpy_helper.from_array(new_arr, name=init.name) - init.CopyFrom(new_init) - # If this initializer also shadows a graph input/output, retype that - # ValueInfo to INT32 too -- otherwise full type inference rejects the - # fixed model (legal ONNX: an initializer may also be a graph input). - sync_value_info_dtype(graph, init.name, TensorProto.INT32) - applied.append( - FixApplied( - fixer=self.name, - target=init.name, - description=( - f"cast initializer '{init.name}' from INT64 to INT32 " - f"({arr.size} elements, range [{int(arr.min())}, {int(arr.max())}])" - ), + def _skip_reason( + self, + init: onnx.TensorProto, + uses: dict[str, list[tuple[str, str, int]]], + definition_counts: dict[str, int], + boundary_names: set[str], + ) -> str | None: + """Return why this initializer must not be converted, or None if safe.""" + name = init.name + if definition_counts.get(name, 0) > 1: + return "name is defined in more than one scope (shadowing is ambiguous)" + if name in boundary_names: + return "initializer is also a graph input/output; converting changes the signature" + consumer_positions = uses.get(name, []) + if not consumer_positions: + return "initializer has no consumers; nothing to gain from converting" + for domain, op_type, idx in consumer_positions: + if domain not in _DEFAULT_DOMAINS: + return f"consumed by custom-domain op '{domain}::{op_type}'" + if (op_type, idx) not in _INT32_SAFE_POSITIONS: + return ( + f"consumed by '{op_type}' input {idx}, which is not a " + "known INT32-compatible position (e.g. Reshape's shape " + "input requires INT64)" ) - ) - return applied + arr = numpy_helper.to_array(init) + if arr.size == 0: + return "empty tensor; casting is a no-op" + if arr.min() < _INT32_MIN or arr.max() > _INT32_MAX: + return "values exceed INT32 range" + return None + + +def _collect_uses(model: onnx.ModelProto) -> dict[str, list[tuple[str, str, int]]]: + """name -> [(domain, op_type, input_index)] for every node input in the + model, including nodes inside nested subgraphs (which may capture + outer-scope initializers by name).""" + uses: dict[str, list[tuple[str, str, int]]] = {} + for node, _owner in iter_nodes(model.graph): + for idx, inp in enumerate(node.input): + if inp: + uses.setdefault(inp, []).append((node.domain, node.op_type, idx)) + return uses + + +def _definition_counts(model: onnx.ModelProto) -> dict[str, int]: + """How many scopes define each name (initializers, graph inputs, node + outputs, subgraph inputs). >1 means uses of the name are scope-dependent + and a rename-free rewrite cannot be proven safe.""" + counts: dict[str, int] = {} + + def bump(name: str) -> None: + if name: + counts[name] = counts.get(name, 0) + 1 + + for graph in iter_subgraphs(model.graph): + for init in graph.initializer: + bump(init.name) + for vi in graph.input: + # opset<9 models list initializers in graph.input too; that pair is + # one definition, not two. + if all(init.name != vi.name for init in graph.initializer): + bump(vi.name) + for node in graph.node: + for out in node.output: + bump(out) + return counts diff --git a/trtcheck/remediation.py b/trtcheck/remediation.py index 3de35b7..82aaa6e 100644 --- a/trtcheck/remediation.py +++ b/trtcheck/remediation.py @@ -21,7 +21,7 @@ from importlib import resources from typing import Any -from trtcheck.types import CheckCategory, Issue, Severity +from trtcheck.types import CheckCategory, Confidence, Issue, Severity @dataclass(frozen=True) @@ -34,6 +34,9 @@ class RemediationEntry: explanation: str remediation: str docs_link: str | None = None + rule_id: str = "" + confidence: Confidence = Confidence.HIGH + verify_required: bool = False def _to_entry(key: str, raw: dict[str, Any]) -> RemediationEntry: @@ -51,6 +54,9 @@ def _to_entry(key: str, raw: dict[str, Any]) -> RemediationEntry: explanation=raw["explanation"], remediation=raw["remediation"], docs_link=raw.get("docs_link"), + rule_id=raw["rule_id"], + confidence=Confidence(raw.get("confidence", "high")), + verify_required=bool(raw.get("verify_required", False)), ) except (ValueError, KeyError) as exc: raise ValueError(f"remediation_db.json entry {key!r} is invalid: {exc}") from exc @@ -83,7 +89,20 @@ def known_ids() -> frozenset[str]: return frozenset(_DB) -def make_issue(issue_id: str, *, node_name: str, operator: str, prefix: str) -> Issue: +def rule_ids() -> frozenset[str]: + """Every stable rule id defined in remediation_db.json.""" + return frozenset(e.rule_id for e in _DB.values()) + + +def make_issue( + issue_id: str, + *, + node_name: str, + operator: str, + prefix: str, + graph_scope: str = "", + target_trt: str | None = None, +) -> Issue: """Build an :class:`Issue` for ``issue_id``. The per-node ``prefix`` (built by the checker from node context, e.g. @@ -100,4 +119,9 @@ def make_issue(issue_id: str, *, node_name: str, operator: str, prefix: str) -> message=f"{prefix}. {entry.explanation}", remediation=entry.remediation, docs_link=entry.docs_link, + rule_id=entry.rule_id, + confidence=entry.confidence, + verify_required=entry.verify_required, + graph_scope=graph_scope, + target_trt=target_trt, ) diff --git a/trtcheck/reporters/console.py b/trtcheck/reporters/console.py index 9ec8e31..9d01f41 100644 --- a/trtcheck/reporters/console.py +++ b/trtcheck/reporters/console.py @@ -17,7 +17,7 @@ from rich.table import Table from trtcheck._text import strip_unsafe -from trtcheck.types import AnalysisReport, Severity +from trtcheck.types import AnalysisReport, Severity, Verdict _SEV_COLOR = { Severity.CRITICAL: "red", @@ -25,6 +25,16 @@ Severity.INFO: "blue", } +# Four-state verdict -> (headline, border color). The wording is deliberately +# conservative: LIKELY means "static analysis found no known blocker", never +# "guaranteed to convert". +_VERDICT_STYLE = { + Verdict.BLOCKED: ("CONVERSION BLOCKED -- known critical incompatibilities", "red"), + Verdict.UNVERIFIED: ("UNVERIFIED -- no known blocker, unresolved conditions remain", "yellow"), + Verdict.LIKELY: ("LIKELY -- static analysis found no known blocker", "green"), + Verdict.VERIFIED: ("VERIFIED -- TensorRT runtime build succeeded", "green"), +} + def _sanitize(text: str) -> str: """Make model-derived text safe to print: drop control / bidi-override chars @@ -57,15 +67,12 @@ def render(self, report: AnalysisReport) -> str: return buf.getvalue() def _header(self, report: AnalysisReport) -> Panel: - if report.conversion_likely: - title = "[bold green]LIKELY TO CONVERT[/bold green]" - border = "green" - else: - title = "[bold red]CONVERSION WILL FAIL[/bold red]" - border = "red" + headline, border = _VERDICT_STYLE[report.verdict] + title = f"[bold {border}]{headline}[/bold {border}]" + target = f" target: TensorRT {report.target_trt}" if report.target_trt else "" body = ( f"{title}\n" - f"file: {_sanitize(report.filename)}\n" + f"file: {_sanitize(report.filename)}{target}\n" f"opset: {report.opset_version} producer: {_sanitize(report.producer)} " f"nodes: {report.total_nodes}\n" f"{report.critical_count} critical " @@ -77,6 +84,7 @@ def _header(self, report: AnalysisReport) -> Panel: def _issues_table(self, report: AnalysisReport) -> Table: table = Table(title="Detected issues", show_lines=True) table.add_column("Severity", style="bold") + table.add_column("Rule", overflow="fold") # overflow="fold" hard-wraps long unbroken tokens (export commands, URLs, # paths) instead of clipping them with an ellipsis -- the remediation a # user must apply is never silently truncated. @@ -88,6 +96,7 @@ def _issues_table(self, report: AnalysisReport) -> Table: color = _SEV_COLOR[issue.severity] table.add_row( f"[{color}]{issue.severity.value.upper()}[/{color}]", + _sanitize(issue.rule_id), _sanitize(issue.node_name), _sanitize(issue.operator), _sanitize(issue.message), @@ -96,9 +105,16 @@ def _issues_table(self, report: AnalysisReport) -> Table: return table def _summary(self, report: AnalysisReport) -> str: - if report.conversion_likely: - return f"\nEstimated fix time: {report.estimated_fix_time}" - return ( - f"\nEstimated fix time: {report.estimated_fix_time}.\n" - "Address critical issues first; warnings can often wait." - ) + if report.verdict is Verdict.BLOCKED: + return ( + f"\nEstimated fix time: {report.estimated_fix_time}.\n" + "Address critical issues first; warnings can often wait." + ) + if report.verdict is Verdict.UNVERIFIED: + unresolved = sum(1 for i in report.issues if i.verify_required) + return ( + f"\nEstimated fix time: {report.estimated_fix_time}.\n" + f"{unresolved} finding(s) need runtime verification " + "(trtexec) or manual review before this model can be called safe." + ) + return f"\nEstimated fix time: {report.estimated_fix_time}" diff --git a/trtcheck/reporters/html.py b/trtcheck/reporters/html.py index 3a84697..da8d027 100644 --- a/trtcheck/reporters/html.py +++ b/trtcheck/reporters/html.py @@ -5,7 +5,15 @@ import html from trtcheck._text import strip_unsafe -from trtcheck.types import AnalysisReport, Severity +from trtcheck.types import AnalysisReport, Severity, Verdict + +# Verdict -> (css class, headline). Conservative wording on purpose. +_VERDICT_HTML = { + Verdict.BLOCKED: ("fail", "Conversion blocked"), + Verdict.UNVERIFIED: ("warn", "Unverified — unresolved conditions remain"), + Verdict.LIKELY: ("pass", "Likely — no known blocker (static analysis)"), + Verdict.VERIFIED: ("pass", "Verified — TensorRT runtime build succeeded"), +} def _safe(text: str) -> str: @@ -50,9 +58,11 @@ def _safe(text: str) -> str: border: 1px solid var(--border); } .verdict.pass { background: rgba(78, 201, 176, 0.08); border-color: var(--ok); } +.verdict.warn { background: rgba(255, 180, 84, 0.08); border-color: var(--warn); } .verdict.fail { background: rgba(255, 92, 92, 0.08); border-color: var(--crit); } .verdict h2 { margin: 0 0 0.5rem 0; font-size: 1.25rem; } .verdict.pass h2 { color: var(--ok); } +.verdict.warn h2 { color: var(--warn); } .verdict.fail h2 { color: var(--crit); } .meta { color: var(--muted); font-size: 0.9rem; } .meta span + span::before { content: " · "; } @@ -133,8 +143,7 @@ def render_fragment(self, report: AnalysisReport) -> str: --diff mode. The CSS is shared at the document level; render() injects it once. """ - verdict_class = "pass" if report.conversion_likely else "fail" - verdict_title = "Likely to convert" if report.conversion_likely else "Conversion will fail" + verdict_class, verdict_title = _VERDICT_HTML[report.verdict] parts: list[str] = [] parts.append('
') parts.append("

trtcheck report

") @@ -142,6 +151,8 @@ def render_fragment(self, report: AnalysisReport) -> str: parts.append(f"

{verdict_title}

") parts.append('
') parts.append(f"{_safe(report.filename)}") + if report.target_trt: + parts.append(f"target TensorRT {_safe(report.target_trt)}") parts.append(f"opset {report.opset_version}") parts.append(f"{report.total_nodes} nodes") parts.append(f"{report.critical_count} critical") @@ -155,7 +166,7 @@ def render_fragment(self, report: AnalysisReport) -> str: parts.append("") parts.append( "" - "" + "" "" "" ) @@ -174,6 +185,7 @@ def render_fragment(self, report: AnalysisReport) -> str: parts.append( "" f'' + f"" f"" f"" f"" diff --git a/trtcheck/runtime_verify.py b/trtcheck/runtime_verify.py new file mode 100644 index 0000000..dff291a --- /dev/null +++ b/trtcheck/runtime_verify.py @@ -0,0 +1,179 @@ +"""Optional runtime verification via NVIDIA's ``trtexec``. + +Static analysis predicts; only a real TensorRT parse/build verifies. This +module shells out to ``trtexec --onnx=`` when the user asks for it +(``trtcheck --verify-runtime``). It is deliberately isolated: nothing else +in trtcheck imports TensorRT, and every result state is explicit -- +verification that could not run is never conflated with verification that +passed. + +Security/robustness notes: + - the subprocess is invoked as an argument list (no shell), so a crafted + model filename cannot inject commands; + - a timeout bounds runaway engine builds; + - stdout/stderr are captured and truncated to tails, never echoed raw. +""" + +from __future__ import annotations + +import shutil +import subprocess +import time +from dataclasses import dataclass, field +from enum import Enum +from pathlib import Path +from typing import Any + +DEFAULT_TIMEOUT_S = 600 +_TAIL_CHARS = 2000 + +# Markers that indicate the ONNX parser (not the engine builder) rejected the +# model. Heuristic: trtexec does not exit with distinct codes for the two +# phases, but parser failures consistently mention the ONNX importer. +_PARSER_MARKERS = ( + "failed to parse onnx", + "modelimporter", + "onnx2trt", + "could not parse the model", + "parsing model failed", + "assertion failed", + "in function importmodel", +) + + +class RuntimeStatus(str, Enum): + SUCCESS = "success" + PARSER_FAILURE = "parser_failure" + BUILD_FAILURE = "build_failure" + MISSING_TRTEXEC = "missing_trtexec" + TIMEOUT = "timeout" + ERROR = "error" + + +@dataclass +class RuntimeVerification: + """Outcome of one trtexec run, with enough metadata to reproduce it.""" + + status: RuntimeStatus + trtexec_path: str | None = None + trtexec_version: str | None = None + command: list[str] = field(default_factory=list) + returncode: int | None = None + duration_s: float | None = None + stdout_tail: str = "" + stderr_tail: str = "" + detail: str = "" + + @property + def verified(self) -> bool: + return self.status is RuntimeStatus.SUCCESS + + def to_dict(self) -> dict[str, Any]: + return { + "status": self.status.value, + "trtexec_path": self.trtexec_path, + "trtexec_version": self.trtexec_version, + "command": self.command, + "returncode": self.returncode, + "duration_s": self.duration_s, + "stdout_tail": self.stdout_tail, + "stderr_tail": self.stderr_tail, + "detail": self.detail, + } + + +def find_trtexec(explicit_path: str | None = None) -> str | None: + """Resolve the trtexec executable, or None when unavailable.""" + if explicit_path: + p = Path(explicit_path) + return str(p) if p.is_file() else None + return shutil.which("trtexec") + + +def _tail(text: str) -> str: + return text[-_TAIL_CHARS:] + + +def _extract_version(output: str) -> str | None: + for line in output.splitlines(): + lowered = line.lower() + if "tensorrt" in lowered and ("version" in lowered or " v" in lowered): + return line.strip()[:200] + return None + + +def _looks_like_parser_failure(output: str) -> bool: + lowered = output.lower() + return any(marker in lowered for marker in _PARSER_MARKERS) + + +def verify_model( + model_path: Path | str, + *, + trtexec_path: str | None = None, + timeout_s: int = DEFAULT_TIMEOUT_S, +) -> RuntimeVerification: + """Run ``trtexec --onnx=`` and classify the outcome. + + Never raises for expected failure modes; every outcome is a + :class:`RuntimeVerification` with an explicit status. + """ + exe = find_trtexec(trtexec_path) + if exe is None: + return RuntimeVerification( + status=RuntimeStatus.MISSING_TRTEXEC, + detail=( + "trtexec not found on PATH (or at the given --trtexec path). " + "Install TensorRT or point --trtexec at the executable." + ), + ) + + command = [exe, f"--onnx={Path(model_path)}"] + start = time.monotonic() + try: + proc = subprocess.run( + command, + capture_output=True, + text=True, + timeout=timeout_s, + check=False, + ) + except subprocess.TimeoutExpired: + return RuntimeVerification( + status=RuntimeStatus.TIMEOUT, + trtexec_path=exe, + command=command, + duration_s=time.monotonic() - start, + detail=f"trtexec did not finish within {timeout_s}s", + ) + except OSError as exc: + return RuntimeVerification( + status=RuntimeStatus.ERROR, + trtexec_path=exe, + command=command, + detail=f"could not execute trtexec: {exc}", + ) + + duration = time.monotonic() - start + combined = proc.stdout + "\n" + proc.stderr + version = _extract_version(combined) + if proc.returncode == 0: + status = RuntimeStatus.SUCCESS + detail = "trtexec parsed the model and built an engine" + elif _looks_like_parser_failure(combined): + status = RuntimeStatus.PARSER_FAILURE + detail = "the TensorRT ONNX parser rejected the model" + else: + status = RuntimeStatus.BUILD_FAILURE + detail = "the model parsed but the engine build failed" + return RuntimeVerification( + status=status, + trtexec_path=exe, + trtexec_version=version, + command=command, + returncode=proc.returncode, + duration_s=duration, + stdout_tail=_tail(proc.stdout), + stderr_tail=_tail(proc.stderr), + detail=detail, + ) diff --git a/trtcheck/types.py b/trtcheck/types.py index 9d8f2fa..c6008dc 100644 --- a/trtcheck/types.py +++ b/trtcheck/types.py @@ -32,6 +32,48 @@ class CheckCategory(str, Enum): GRAPH_STRUCTURE = "graph_structure" +class Confidence(str, Enum): + """How much evidence backs a finding. + + HIGH -- documented or empirically verified behavior; acting on the + finding is safe. + MEDIUM -- a static heuristic with known gaps (e.g. partial-support + limitations that depend on exported attributes). + LOW -- an uncertainty marker: trtcheck cannot classify the construct + statically and is saying so rather than guessing. + """ + + HIGH = "high" + MEDIUM = "medium" + LOW = "low" + + +class Verdict(str, Enum): + """Four-state conversion verdict. Replaces the old boolean + ``conversion_likely``. + + BLOCKED -- at least one known-critical incompatibility. + UNVERIFIED -- no known blocker, but unresolved conditions remain + (unclassified operators, custom domains, conditional + support that static analysis cannot settle). + LIKELY -- every static check passed with nothing unresolved. This + is still a static prediction, not a guarantee. + VERIFIED -- an optional real TensorRT (trtexec) parse/build succeeded + for the declared environment. + """ + + BLOCKED = "blocked" + UNVERIFIED = "unverified" + LIKELY = "likely" + VERIFIED = "verified" + + +# JSON report schema version. Bump the major when a field is removed or +# changes meaning; bump the minor when fields are added. Consumers of the +# 1.x schema keep working: every 1.x key is still present in 2.0. +REPORT_SCHEMA_VERSION = "2.0" + + @dataclass class Issue: """A single finding from a checker.""" @@ -43,6 +85,13 @@ class Issue: message: str remediation: str docs_link: str | None = None + # Stable machine-readable identity/metadata (schema 2.0). Defaults keep + # the constructor backward compatible for third-party checkers. + rule_id: str = "" + confidence: Confidence = Confidence.HIGH + verify_required: bool = False + target_trt: str | None = None + graph_scope: str = "" def to_dict(self) -> dict[str, Any]: return { @@ -53,8 +102,22 @@ def to_dict(self) -> dict[str, Any]: "message": self.message, "remediation": self.remediation, "docs_link": self.docs_link, + "rule_id": self.rule_id, + "confidence": self.confidence.value, + "verify_required": self.verify_required, + "target_trt": self.target_trt, + "graph_scope": self.graph_scope, } + def identity(self) -> tuple[str, str, str, str]: + """Stable identity for diffing reports. + + Includes ``graph_scope`` so two same-named nodes in different + subgraphs (legal ONNX: uniqueness is per-graph) never alias in a + before/after comparison. + """ + return (self.rule_id, self.node_name, self.operator, self.graph_scope) + @dataclass class AnalysisReport: @@ -72,6 +135,14 @@ class AnalysisReport: total_nodes: int issues: list[Issue] = field(default_factory=list) + # Which TensorRT version the operator-support checks targeted. + target_trt: str | None = None + # Set only by the optional runtime-verification path (trtexec parse/build + # succeeded). Static analysis never sets this. + runtime_verified: bool = False + # Metadata from the runtime verification run (command, versions, status), + # populated by the CLI when --verify-runtime is used. + runtime_verification: dict[str, Any] | None = None @property def critical_count(self) -> int: @@ -85,9 +156,38 @@ def warning_count(self) -> int: def info_count(self) -> int: return sum(1 for i in self.issues if i.severity is Severity.INFO) + @property + def verdict(self) -> Verdict: + """Four-state verdict derived from the findings (see :class:`Verdict`). + + Precedence: BLOCKED > VERIFIED > UNVERIFIED > LIKELY. A recorded + runtime *failure* (parser or engine build) demotes an otherwise-LIKELY + report to UNVERIFIED -- contradictory runtime evidence must never be + hidden behind a clean static prediction. Verification that could not + run (missing trtexec, timeout, spawn error) leaves the static verdict + untouched; its metadata is still in ``runtime_verification``. + """ + if self.critical_count > 0: + return Verdict.BLOCKED + if self.runtime_verified: + return Verdict.VERIFIED + if any(i.verify_required for i in self.issues): + return Verdict.UNVERIFIED + if self.runtime_verification is not None and self.runtime_verification.get("status") in ( + "parser_failure", + "build_failure", + ): + return Verdict.UNVERIFIED + return Verdict.LIKELY + @property def conversion_likely(self) -> bool: - return self.critical_count == 0 + """Deprecated boolean view of :attr:`verdict`. + + Kept for 1.x JSON consumers: True for every verdict except BLOCKED. + Prefer ``verdict`` -- this property cannot express UNVERIFIED. + """ + return self.verdict is not Verdict.BLOCKED @property def estimated_fix_time(self) -> str: @@ -103,15 +203,20 @@ def estimated_fix_time(self) -> str: def to_dict(self) -> dict[str, Any]: return { + "schema_version": REPORT_SCHEMA_VERSION, "filename": self.filename, "onnx_ir_version": self.onnx_ir_version, "opset_version": self.opset_version, "producer": self.producer, "total_nodes": self.total_nodes, + "target_trt": self.target_trt, "issues": [i.to_dict() for i in self.issues], "critical_count": self.critical_count, "warning_count": self.warning_count, "info_count": self.info_count, + "verdict": self.verdict.value, + "runtime_verified": self.runtime_verified, + "runtime_verification": self.runtime_verification, "conversion_likely": self.conversion_likely, "estimated_fix_time": self.estimated_fix_time, }
SeverityNodeOperatorSeverityRuleNodeOperatorIssueFixDocs
{sev.upper()}{_safe(issue.rule_id)}{_safe(issue.node_name)}{_safe(issue.operator)}{_safe(issue.message)}