[None][perf] GVR top-K decode: enable R0 histogram-ladder admission by default

[longcheng-nv]

Summary

Adds an R0 histogram-ladder admission fast path plus a fused rank-and-scatter Phase-4 writeback to the production Blackwell cuTe-DSL GVR top-K decode kernel (GvrTopKKernel) and makes R0 the default (enable_r0=True).

R0 replaces the Phase-2 secant threshold search with a single-pass multi-threshold "rung ladder" admission, seeded by a 256-bin histogram over the prev-topK gathered values (P1b), and a fused rank-and-scatter Phase-4 writeback (barriers ~14 → ~7). Two hardening pieces are included:

• Virtual seed rung (r0_vseed, default on with R0). P1's mean probe (the secant init) is folded into the R0 count pass as one extra virtual rung column, reusing the existing per-thread count buffer (zero SMEM growth, no extra traffic or synchronization). Admission picks the tightest admissible column, and on a full miss the measured count donates an interior bracket point to the fallback refine. This adapts the ladder to each row's value distribution and fixes the cold-hint fat-admission regime (details under Known limitation). With the seed rung, K512/K1024 drop to a single static rung (qfracs=(0.85,), zero column tax); K2048 keeps (0.85, 0.35).
• Exact boundary-tie resolution in Phase-4 (p4_exact_tail, default on for fp32). The P4 fine recursion resolves values to the fine-bin width; distinct fp32 values spaced below that resolution which straddle the top-K boundary landed in one fine sub-bin and were kept in arbitrary arrival order — observed on real DSv4-Pro 512k-ISL captures as a 1-element miss with |dv| ≈ 3e-6. Now the only ambiguous case (straddling-bin tie set overfilling the remaining slots) is gated and re-ranked bit-exactly with an MSB-first 8-bit-digit radix select over order-preserving integer keys; all scratch is reused (zero SMEM growth). Unambiguous rows pay two scalar compares. 16-bit kernels are byte-identical (their upconverted keys are already fully resolved; 16-bit tie plateaus are bitwise-equal, so arrival order is value-exact).

The classic secant path is retained verbatim and stays reachable via enable_r0=False.

This PR lands the kernel capability + flips the default only. It does not touch the call site / dispatch (the custom op does not plumb enable_r0); the activation policy / dispatch guard is a follow-up PR (see Known limitation).

Performance

Methodology. nsys pure-kernel, cold-L2 (512 MB L2 evict outside the timed NVTX range, per timed call), single-GPU same-run A/B (R0 vs the retained secant baseline on the identical inputs), 20 cold reps. B200 (SM100). All numbers below re-measured 2026-07-16 at the current head (seed rung + exact-tail fix included) with both arms driven through the kernel's own launch-shape contract (GvrTopKKernel.launch → pick_config): cluster_size 8/4/2/1 by (BS, N) — incl. the cs=8 tiny-grid pick at N≥128K — T=512/1024, mbpm tiers, fp32-only 256-bit loads; i.e. the shapes the production runner picks. The two data sources are (a) a calibrated synthetic envelope (empirical-CDF + temporally-coherent preIdx) and (b) per-layer indexer top-K captured from production DeepSeek-V4 Flash / Pro and V3.2 decode runs.

Data sources

Synthetic (calibrated synthetic envelope). Generated by the indexer-topk temporal-synth method rather than i.i.d. noise: each row's logit marginal is a per-layer empirical inverse-CDF with a GPD-calibrated tail fit to the real 64K production captures (this reproduces the top-K selection-boundary tail that a single Beta/Gaussian collapses); the previous-step preIdx hint is built by a rank-conditional retention model with per-row hit-rate drawn from the real per-decode-step distribution + AR(1) temporal coherence, plus a marginal-preserving positional model (winner-position density + clustering) so the gather footprint (cache-line segments) matches production. best = warm-hint / higher-hit regime (hit-rate ≈ 0.55); worst = cold-hint tail (hit-rate ≈ 0.05).

Real (production decode capture). Per-layer (indexer logits, indexer top-K) dumped from real BS=1 greedy end-to-end decode runs of DeepSeek-V4 Flash (K=512), DeepSeek-V4 Pro (K=1024) and DeepSeek-V3.2 (K=2048), across 9 input-sequence-length rungs (ISL 4k–1024k) × 3 GVR-active layers each. N = post-compress indexer length (V4 compress_ratio=4, V3.2 cr=1); hit = |preIdx ∩ topK| / K (0.27–1.00 measured across the sweep). For V3.2 (no warm-start) preIdx is reconstructed from the previous decode step's top-K. Correctness is validated against each capture's own recorded top-K reference.

Synthetic (calibrated synthetic envelope, BS=1, fp32) — µs and speedup per seq-len N

best = warm hint (hit-rate ≈ 0.55). Launch shapes per pick_config (BS=1: cs=1 below 64K, cs=4 at 64K, cs=8 from 128K). All cells value-set-exact vs torch.topk.

K512 (V4 Flash)

N	base best (µs)	R0 best (µs)	best ↑
4,096	12.55	9.51	1.32×
8,192	12.53	9.66	1.30×
16,384	13.25	10.45	1.27×
32,768	13.50	11.99	1.13×
65,536	19.52	13.06	1.49×
131,072	17.08	14.41	1.18×
262,144	19.65	15.46	1.27×

K512 (V4 Flash) geomean speedup (N=4K–256K): best 1.276×

K1024 (V4 Pro)

N	base best (µs)	R0 best (µs)	best ↑
4,096	10.17	9.23	1.10×
8,192	18.39	9.83	1.87×
16,384	16.45	10.86	1.52×
32,768	21.91	12.13	1.81×
65,536	18.91	13.99	1.35×
131,072	19.14	14.70	1.30×
262,144	23.45	16.91	1.39×

K1024 (V4 Pro) geomean speedup (N=4K–256K): best 1.454×

K2048 (V3.2)

N	base best (µs)	R0 best (µs)	best ↑
8,192	14.68	12.44	1.18×
16,384	14.63	13.39	1.09×
32,768	17.41	16.32	1.07×
65,536	20.58	18.91	1.09×
131,072	20.41	20.33	1.00×
262,144	24.61	22.16	1.11×
524,288	21.68	22.88	0.948×
1,048,576	38.02	34.29	1.11×

K2048 (V3.2) geomean speedup: best 1.073×

Synthetic overall (22 cells shown; best axis, V4 tables show the deployment-envelope range N=4K–256K): geomean best 1.249×; exactness 52/52 over the full measured envelope (incl. the worst-axis and V4 N≥512K cells not tabulated).

Real decode-capture (production DSv4 / V3.2 indexer top-K, BS=1, fp32) — µs and speedup per seq-len

Per-layer top-K captured from real decode runs; N = post-compress indexer length, hit = preIdx∩topK overlap. Launch shapes per pick_config (cs=8 on the large-N rungs). Time = median over 3 captured layers.

V4 Flash (K512)

ISL	N	hit	base (µs)	R0 (µs)	speedup	exact
4k	1,027	0.64	8.42	7.41	1.14×	✓
8k	2,051	0.33	8.06	7.78	1.04×	✓
16k	4,099	0.34	10.41	8.15	1.28×	✓
32k	8,195	0.69	10.98	8.79	1.25×	✓
64k	16,387	0.08	13.98	11.44	1.22×	✓
128k	32,771	0.70	14.86	10.99	1.35×	✓
256k	65,538	0.28	16.08	13.37	1.20×	✓
512k	131,075	0.06	43.31	20.15	2.15×	✓
1024k	262,127	0.42	19.03	17.02	1.12×	✓

V4 Flash (K512) geomean speedup: 1.276×

V4 Pro (K1024)

ISL	N	hit	base (µs)	R0 (µs)	speedup	exact
4k	1,027	1.00	18.85	9.47	1.99×	✓
8k	2,051	0.46	8.87	8.75	1.01×	✓
16k	4,099	0.74	14.57	9.48	1.54×	✓
32k	8,195	0.53	17.88	11.25	1.59×	✓
64k	16,387	0.31	11.84	11.38	1.04×	✓
128k	32,771	0.33	15.92	12.82	1.24×	✓
256k	65,539	0.36	16.44	14.49	1.13×	✓
512k	131,075	0.23	20.67	21.56	0.959×	✓
1024k	262,127	0.27	23.75	18.98	1.25×	✓

V4 Pro (K1024) geomean speedup: 1.271×

V3.2 (K2048)

ISL	N	hit	base (µs)	R0 (µs)	speedup	exact
4k	4,111	0.73	15.71	11.94	1.32×	✓
8k	8,207	0.84	15.03	14.12	1.06×	✓
16k	16,399	0.53	27.22	13.87	1.96×	✓
32k	32,783	0.43	19.48	16.67	1.17×	✓
64k	65,551	0.41	31.11	20.47	1.52×	✓
128k	131,087	0.62	21.23	17.93	1.18×	✓
256k	163,775	0.87	26.39	18.83	1.40×	✓

V3.2 (K2048) geomean speedup: 1.348×

Real overall (25 cells): geomean 1.294×, 24/25 ≥ 1.0, exactness 25/25.

The single real cell < 1.0 (Pro 512k, 0.959×) is a repair-active row: the exact boundary-tie repair triggers there and pays its re-rank cost — the pre-fix kernel returned a wrong top-K set on exactly these rows (see Correctness), so the previous, faster number was for an incorrect result.

Batch-size scaling + launch-shape policy (pick_config / launch)

The BS=1 tables above raise the natural question of batch scaling. Two findings (same nsys cold-L2 protocol, BS ∈ {1…1024}, synthetic + real inputs replicated to BS independent rows):

• R0 itself is BS-invariant. The R0/base gain stays flat from BS 1 → 1024 (synthetic geomean 1.096× → 1.149× over the FULL seq-len grid, 9 rungs 4K–1M × 3 dtypes; real 1.204× → 1.209× over ALL captured ISL rungs × 3 dtypes) at production launch shapes. R0 changes Phase-2/Phase-4 arithmetic only — grid shape, smem budget, and cluster semantics are identical to base — so it introduces no batch-size coupling.
• Launch-shape choice dominates at large BS — and it is a launch-time decision, not a kernel property. cluster_size / num_threads / min_blocks_per_mp are compile-time constructor knobs: a compiled kernel cannot change its own grid or cluster shape, so batch adaptation must pick a different compiled variant per call. Driving the kernel with a config frozen at the BS=1 optimum (cluster_size = N≥64K ? 4 : 1, T=1024, mbpm=1) is geomean 2.38× slower (max 5.8×) than per-(BS, N) picks across BS ∈ {64, 256, 1024} — e.g. K512 bf16 N=65536 BS=1024: 419 µs frozen vs 72 µs picked. Multi-CTA splitting only pays while the grid is a single wave (num_rows × cluster_size ≤ num_SMs); past that, row parallelism already saturates the SMs and per-row splitting is pure overhead. The production custom op already makes these picks; the risk was only ever to direct-drive users.

To make that policy part of the kernel's contract (and un-mismeasurable), this PR adds two host-side helpers to GvrTopKKernel — no device-code change:

• pick_config(dtype, num_rows, num_candidates, max_seq_len=None) — the (dtype, BS, N) → ctor-kwargs launch-shape policy as a pure classmethod colocated with the kernel: cluster_size 8/4/2/1 by (BS, N) (8 = tiny-grid large-N: BS ≤ 4, N ≥ 128K), threads/occupancy/vec-width tiers, and the CUDA-graph contract (max_seq_len = peak runtime N so variants are picked for the replay shape, not the capture shape). Mirrors the production runner's inline policy; the follow-up dispatch-guard PR unifies the runner onto it.
• launch(logits, pre_idx, seq_lens, out_indices, top_k, ...) — a thin compiled-variant cache (same sym-int + tvm-ffi compile contract as the runner) so tests and benchmarks get production-equivalent shapes by default; **kernel_overrides (e.g. enable_r0=False, cluster_size=8) force any knob and participate in the cache key.

cluster_size = 8 validated (previously untested anywhere): 78/78 standalone exactness cells green (R0 and secant, dtype ∈ {fp32, bf16, fp16} × K ∈ {512, 1024, 2048} × N ∈ {131072, 262144} × warm/mid/cold hint × BS ∈ {1, 4}, fp32 R0 = secant identical index sets), and the cs=8 pick beats a forced cs=4 on 8/8 nsys cells (geomean 0.943×, up to −12 % at N=262144).

Known limitation (disclosed) + follow-up

R0 is not a uniform win on the cold-hint tail, but the seed rung tightened the envelope materially:

• Previously disclosed fat-admission regression: fixed. An earlier revision of this PR regressed to 0.68–0.79× of base on the real Flash 1M-ISL capture at BS ≥ 128 (hit ≈ 0.42): the coarse static rung admitted ~4400 candidates where the row needed ~630 (7× Phase-3/4 work). The virtual seed rung adapts the admission to the row's values and returns those cells to base parity (fp32 1.02–1.03×, up to 1.43× vs the previous revision).
• fp32 (the dtype the production indexer runs): worst-axis synth geomean 1.022× (12/26 worst cells < 1.0, min 0.891×; one best cell 0.948×). The only fp32 full-grid cluster < 0.90× of base is synthetic-worst K2048 N=32768 (~0.89× across BS).
• 16-bit inputs carry the remaining tail. Over the full 2772-cell dtype × BS × N audit grid, R0/base geomean is 1.132× with min 0.790× and 106 cells < 0.90× — 100 of the 106 are bf16/fp16, concentrated at 16-bit big-BS / large-N (chiefly K1024/K512, BS ≥ 128, N ≥ 262k).

The real decode captures do not exhibit the cold-hint distribution (measured hit-rates 0.27–1.00; 24/25 real cells ≥ 1.0), which is why the default is on. A follow-up PR adds a dispatch guard routing the residual regimes to the retained secant path. Because the hint hit-rate is not observable at inference time (it is the overlap of the current top-K with the hint), the guard is designed as an in-kernel admission escape (R0 already counts seed admissions — bail out to the secant path when the count signals a cold seed) and/or trailing-step feedback (the kernel emits its measured admission counter; the host uses the previous step's value as this step's predictor). The enable_r0=False fallback in this PR is exactly what that guard dispatches to.

Correctness of the results

Every timing cell above is value-set-exact vs torch.topk and the exactness is asserted on the same run that produces the time:

• Tie-aware value-set check. top-K is order-independent, so correctness is by set, not position. The check gathers the kernel's selected values and asserts the sorted value-multiset equals torch.topk's (rtol/atol 1e-5), plus in-range + no-duplicate + n_below=0 guards. This is robust to value ties (a different index with an equal value is still correct). Synthetic 52/52, real 25/25 (real vs the captured reference); over the full dtype × BS audit grid, R0 is exact on all 2772 cells.
• Boundary-tie defect found by a bit-exact audit and fixed in this PR. On real Pro 512k fp32 captures, the pre-fix Phase-4 kept an arbitrary member of a sub-resolution tie set straddling the top-K boundary (1-element miss, |dv| ≈ 3e-6 — inside the 1e-5 tolerance above, but not the true set; data-dependent, does not reproduce on tie-free synthetic rows). The p4_exact_tail radix re-rank makes all 12 affected audit cells bit-exact (validated at BS 1/128/1024), with paired cold-L2 A/B on unaffected cells at 0.998 geomean (noise) and 16-bit arms byte-identical.
• Base-secant undershoot repaired. On real Flash 512k (hit-rate ≈ 0.06) the upstream base secant is itself inexact (returns < K unique) and ~2.1× slower; the full dtype × BS grid exposes this same rung as 36 base-inexact cells (base 2736/2772) — R0 is exact on all of them.
• New unit tests. test_cute_dsl_gvr_topk_decode_r0_equivalence (SM100-gated) drives GvrTopKKernel directly (the custom op does not expose enable_r0), runs both enable_r0=True and enable_r0=False, and asserts (1) both arms are a valid top-K vs torch.topk (tie-aware value-multiset) and (2) identical index sets on tie-free fp32. Grid: dtype ∈ {bf16, fp16, fp32} × K ∈ {512, 1024, 2048} × N ∈ {8192, 65536} × BS ∈ {1, 16} × warm/cold hint × cluster_size ∈ {1, 4, 8} (cs=8 gated to N ≥ 65536, its production regime). Additional tests: ..._r0_equivalence_bigbs (multi-wave grids: BS=256 single-CTA, BS=64 cluster_size=4), ..._pick_config_policy (locks the (BS, N) → shape map and the big-BS occupancy knobs), ..._launch_autoconfig (drives launch() across all four cluster regimes incl. a forced-secant override arm), and ..._p4_exact_tail_ties (plants adversarial 5e-8-spaced and 1-ulp bitwise tie bands across the top-K boundary and asserts value-multiset exactness across K × N × cluster_size). This is also the sole coverage of the secant fallback now that op-level tests inherit the new default. A standalone exactness harness independently confirms 186/186 across dtype/K/N/BS + adversarial quantized tie plateaus + cluster.

Scope / risk / rollback

• Touches only tensorrt_llm/_torch/cute_dsl_kernels/blackwell/top_k/gvr_topk_decode.py and its unit-test file. No call-site, dispatch, or config-schema changes.
• enable_r0=False restores the pre-R0 upstream kernel byte-for-byte (all R0 fields are const_expr-folded). Full rollback = flip one default back.
• r0_vseed=False retains the previous static (0.85, 0.35) ladder for all K; p4_exact_tail defaults off for 16-bit (byte-identical there).

Test plan

pytest tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py

on SM100 (B200 / B300).

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes

GVR Top-K execution paths

Layer / File(s)	Summary
R0 configuration and histogram primitives tensorrt_llm/_torch/cute_dsl_kernels/blackwell/top_k/gvr_topk_decode.py	Adds R0 and Phase-4 configuration, cached gathers, histogram rung extraction, multi-threshold counting, and supporting shared-memory buffers.
R0 admission orchestration tensorrt_llm/_torch/cute_dsl_kernels/blackwell/top_k/gvr_topk_decode.py	Uses rung thresholds and GE counts for Phase-2 admission, with bounded refinement or the existing secant fallback.
Rank-scatter dispatch and equivalence validation tensorrt_llm/_torch/cute_dsl_kernels/blackwell/top_k/gvr_topk_decode.py, tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py	Adds optional fused Phase-4 rank-scatter dispatch and direct tests comparing R0 against the baseline path.

Estimated code review effort: 5 (Critical) | ~90+ minutes

Sequence Diagram(s)

sequenceDiagram
  participant GvrTopKKernel
  participant phase1b_hspace_rungs
  participant block_count_ge_multi
  participant phase4_rank_scatter
  GvrTopKKernel->>phase1b_hspace_rungs: compute rung thresholds
  phase1b_hspace_rungs->>block_count_ge_multi: pass thresholds for GE counting
  block_count_ge_multi-->>GvrTopKKernel: return admitted rung counts
  GvrTopKKernel->>phase4_rank_scatter: emit selected top-K entries

Loading

Possibly related PRs

• NVIDIA/TensorRT-LLM#16424: Both changes modify GVR Top-K Phase-2/Phase-4 counting and dispatch logic.

Suggested reviewers: limin2021

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately highlights the main change: enabling R0 histogram-ladder admission by default for GVR top-K decode.
Description check	✅ Passed	The description is thorough and covers summary, performance, correctness, scope, and test plan, which satisfies the template overall.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py (1)

727-733: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add direct R0/secant equivalence coverage for compress_ratio=4.

The only baseline comparison hardcodes compress_ratio=1, although R0 is enabled for the production DSv4 path and GvrParams varies by compression ratio. Parameterize the helper and add a focused DSv4 case with offset-zero pre_idx encoding.

Coverage in tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py is insufficient for this production path. As per path instructions, test feedback must identify the concrete test file and whether coverage is sufficient.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py`
around lines 727 - 733, The helper _run_gvr_direct currently hardcodes
compress_ratio=1, so it cannot compare R0/secant behavior for the production
compression path. Parameterize _run_gvr_direct with compress_ratio, pass it
through to GvrTopKKernel, and add a focused DSv4 test using compress_ratio=4
with offset-zero pre_idx encoding that compares direct R0 results against the
secant path. Ensure coverage is added in
tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/_torch/cute_dsl_kernels/blackwell/top_k/gvr_topk_decode.py`:
- Around line 2008-2139: The final fine-histogram path in the exact rank-scatter
logic still groups distinct fp32 values into one sub-bin and can scatter them in
the wrong order at the K boundary. In the fine-bin handling around sb_star,
refine the selected sub-bin to the actual value boundary using the existing
exact snap logic before writing outputs, while preserving rank_above_fine
ordering. In
tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py lines
803-819, add deterministic fp32 values clustered within one fine-bin width
around K along with broad minimum and maximum outliers to verify exact ordering.

In `@tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py`:
- Around line 759-774: Update _make_r0_pre_idx so encoding preserves pre - 1
without clamping, allowing true index 0 to remain stored as -1 and keeping index
1 distinct. Ensure the test data remains valid for the kernel’s +1 decoding
convention and add or adjust coverage in test_cute_dsl_gvr_topk_decode.py to
exercise the index-zero boundary.

---

Nitpick comments:
In `@tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py`:
- Around line 727-733: The helper _run_gvr_direct currently hardcodes
compress_ratio=1, so it cannot compare R0/secant behavior for the production
compression path. Parameterize _run_gvr_direct with compress_ratio, pass it
through to GvrTopKKernel, and add a focused DSv4 test using compress_ratio=4
with offset-zero pre_idx encoding that compares direct R0 results against the
secant path. Ensure coverage is added in
tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: da5958ef-d00d-44a3-960e-9632c2d86656

📥 Commits

Reviewing files that changed from the base of the PR and between 573bd5f and 39ace82.

📒 Files selected for processing (2)

• tensorrt_llm/_torch/cute_dsl_kernels/blackwell/top_k/gvr_topk_decode.py
• tests/unittest/_torch/attention/sparse/test_cute_dsl_gvr_topk_decode.py

[longcheng-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59547 [ run ] triggered by Bot. Commit: 95a8388 Link to invocation

[tensorrt-cicd]

PR_Github #59547 [ run ] completed with state SUCCESS. Commit: 95a8388
/LLM/main/L0_MergeRequest_PR pipeline #47995 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[longcheng-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59635 [ run ] triggered by Bot. Commit: 0182519 Link to invocation

[longcheng-nv]

/bot run

[tensorrt-cicd]

PR_Github #59736 [ run ] triggered by Bot. Commit: eae3745 Link to invocation

[tensorrt-cicd]

PR_Github #59635 [ run ] completed with state ABORTED. Commit: 0182519

Link to invocation

[tensorrt-cicd]

PR_Github #59736 [ run ] completed with state FAILURE. Commit: eae3745
/LLM/main/L0_MergeRequest_PR pipeline #48163 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[longcheng-nv]

/bot run

[tensorrt-cicd]

PR_Github #59901 [ run ] triggered by Bot. Commit: af7351c Link to invocation

[longcheng-nv]

/bot run

[tensorrt-cicd]

PR_Github #59906 [ run ] triggered by Bot. Commit: 7408f3b Link to invocation

[tensorrt-cicd]

PR_Github #59901 [ run ] completed with state ABORTED. Commit: af7351c

Link to invocation

[tensorrt-cicd]

PR_Github #59906 [ run ] completed with state SUCCESS. Commit: 7408f3b
/LLM/main/L0_MergeRequest_PR pipeline #48308 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation