Release v1.1.0: safe fixers, honest verdicts, and real TensorRT verification - #23
Merged
Merged
Conversation
… transactional fix pipeline
- Issue gains rule_id/confidence/verify_required/target_trt/graph_scope;
JSON report schema 2.0 (all 1.x keys preserved)
- AnalysisReport.verdict: blocked/unverified/likely/verified;
conversion_likely kept as deprecated compat property
- Unknown default-domain and custom-domain operators now produce honest
unverified findings (aggregated per op type); --plugin-domain declares
plugin-backed domains
- Conditional-support rules in operator_matrix.json (schema 2.0) with
evidence metadata; TopK and Resize converted from upstream onnx-tensorrt
docs (retrieved 2026-07-22)
- Int64ToInt32Fixer is use-aware: converts only initializers whose every
use is at an INT32-compatible schema position (Gather/Scatter indices,
Cast/Shape/Size input); refuses Reshape shape inputs, shadowed names,
signature tensors, overflow
- DropDropoutFixer resolves opset + training_mode; removes only provably
inference-mode Dropouts
- run_fixers(): transactional per-fixer deep-copy candidates, validated
with full_check where the input supports it; failures recorded, never
leaked into the output model
- --fix pipeline: analyze -> fix -> validate -> re-analyze with same
--target-trt; reports resolved/remaining/introduced by stable rule
identity; JSON summary via --format json
- --fail-on {blocked,unverified}; exit codes documented
- trtcheck.runtime_verify: optional trtexec verification (--verify-runtime)
… dropout training-mode, verdicts, conditions, --fix CLI, runtime verify
…trics - predict.py maps schema-2.0 verdicts (blocked/unverified/likely/verified) into fail/unverified/convert; 1.x fallback kept; full report used (no --severity filter, which stripped the INFO findings verdicts need) - score.py: unverified predictions are excluded from the blocker confusion matrix and reported as coverage, split by ground truth -- never counted as success - new deterministic corpus fixtures + manifest rows: topk_unsorted (conditional-support blocker), custom_domain (honest unverified), reshape_int64_shape (fixer-safety regression guard)
…recard
- docs/usage.md: four verdicts, exit codes, new flags, examples
- docs/fixers.md: transactional pipeline + per-fixer refusal contracts
- docs/rules.md: stable rule-id registry (guarded by test)
- docs/design/analysis-verdicts-and-fix-safety.md: invariants + trust model
- docs/design/plugin-sdk.md: schema-2.0 plugin compatibility notes
- README: conservative language ('no known blocker', never 'guaranteed'),
updated fixer/accuracy sections
- SCORECARD.md: 12-model corpus rerun (2026-07-22), blocker
precision/recall 1.000, unverified coverage 0.250, Clip gap story;
states plainly that ground truth is documented behavior, not live trtexec
- CHANGELOG Unreleased + RELEASE_NOTES_DRAFT.md (nothing published)
- matrix: add Clip (TRT 10.x supported per onnx-tensorrt docs; 8.x unknown)
- verdict: a recorded trtexec parser/build failure now demotes an otherwise-likely report to unverified (runtime evidence against the model can no longer hide behind a clean static prediction); statuses that merely could not run (missing/timeout/error) leave the static verdict untouched - fixer contract: a fixer that reports FixApplied records without actually changing the model is rejected (the applied list must never lie about the output) - plugin checker findings with an empty rule_id get a namespaced PLUGIN-<name> fallback so CI rule_id filters always match something - Issue.identity() now includes graph_scope so same-named nodes in different subgraphs cannot alias in --fix before/after diffs - bench/score.py --json writes a machine-readable summary (bench/summary.json committed for the current corpus run)
- REAL_TENSORRT_VALIDATION_REPORT.md: environment record, exhaustive blocker evidence (PATH/disk/ldconfig/apt/pip/containers), the 7-model corpus static verdicts, live validation of the missing-verifier path and fix pipeline, and the exact procedure for a TensorRT machine - bench/real_tensorrt_smoke_results.json: machine-readable corpus results; runtime fields explicitly marked not_run (nothing fabricated) - RELEASE_READINESS_REPORT.md / PR_BODY_DRAFT.md: the external real-smoke check is now recorded as attempted-and-blocked - .gitignore: cover generated TensorRT engines (*.engine, *.plan, *.trt)
- 7-model corpus executed against genuine trtexec (nvcr.io/nvidia/ tensorrt:24.08-py3, digest 9507e5f2..., TensorRT 10.3.0, RTX 4050 Laptop GPU, driver 580.126.20) using the installed wheel: 5 genuine engine builds, 2 genuine parser failures, 0 disagreements between --verify-runtime and independent direct trtexec runs, 0 unexpected outcomes - GPU access via no-root manual passthrough (--device /dev/nvidia* + read-only driver-lib mounts); zero host packages or configuration changed - new repo-owned runner: scripts/real-smoke-container.sh + scripts/real_tensorrt_smoke.py (timeouts, bounded output, no engines saved, deterministic JSON) - recorded TRT-10.3-specific trtexec behavior: dynamic models without shape flags are auto-overridden to 1x1x1x1 with a warning, not rejected; explicit-profile leg passes - evidence: REAL_TENSORRT_VALIDATION_REPORT.md rewritten from BLOCKED to COMPLETE; bench/real_tensorrt_smoke_results.json now carries real outcomes; README/CHANGELOG/RELEASE_READINESS_REPORT/RELEASE_NOTES_DRAFT/ PR_BODY_DRAFT updated conservatively - no trtcheck product defects found; three smoke-runner defects fixed before the final run (truncated-JSON parsing, version extraction, wrong no-profile expectation)
- version 1.1.0 in pyproject.toml, trtcheck/__init__.py, action.yml default - CHANGELOG: Unreleased rolled into [1.1.0] - 2026-07-22 - release notes finalized (draft language removed, install command added) - README action example version updated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
blocked/unverified/likely/verified). Unknown and custom-domain operators no longer passsilently; a failed real trtexec run can no longer hide behind a clean
static prediction.
conversion_likelystays as a deprecated alias.rule_id,confidence,verify_required,target_trt,graph_scope; theregistry (docs/rules.md) is pinned by a stability test; every 1.x JSON
key is preserved.
--fix— per-fixer isolated candidatesvalidated with strict type/shape inference; use-aware INT64
conversion (the Reshape shape-input corruption is a pinned regression);
training-mode-aware Dropout removal; before/after findings diff with
the same
--target-trt.(TopK, Resize) sourced from the upstream onnx-tensorrt table.
--verify-runtime(trtexec,list-args, timeout); only a real successful build yields
verified.unverified coverage (never counted as success), machine-readable
summary, refreshed honest SCORECARD; package smoke test from a fresh
venv; SECURITY_REVIEW.md.
Compatibility
--severityis now display-only.custom ops report
unverified;--fixrefuses invalid inputs anddeclines previously-unsound INT64 conversions.
Test plan
./scripts/run-tests.sh)mypy trtcheck/ --strictclean; black + isort cleanpython -m build+twine checkpassscripts/package-smoke.sh— wheel install in fresh venv, CLIanalyze / JSON / fix / missing-trtexec paths
trtexecsmoke — TensorRT 10.3.0 (NGC24.08-py3container,RTX 4050): 7-model corpus, 5 genuine builds, 2 genuine failures,
0 wrapper/direct disagreements, installed wheel used throughout.
Evidence:
REAL_TENSORRT_VALIDATION_REPORT.mdFinal test summary
./scripts/run-tests.sh: 456 passed, 1 skipped (opt-in benchmark)mypy trtcheck/ --strict: clean · Black + isort: cleanpython -m build+twine check: PASSED (wheel + sdist, v1.1.0)scripts/package-smoke.sh): PASSscripts/real-smoke-container.sh, officialNGC container): 7 fixtures — 5 genuine engine builds, 2 genuine parser
failures, 0 wrapper/direct-trtexec disagreements
Limitations
Runtime evidence covers TensorRT 10.3.0 and 7 bounded generated/public
fixtures only — it validates the verification integration and those cases,
not universal model compatibility. TensorRT 10.16 / 11.0 are not modeled
targets. On TRT 10.3, trtexec auto-builds a degenerate 1x1x1x1 engine for
dynamic models without shape flags; trtcheck's static missing-profile
findings remain the actionable signal.