[NVBUG-6327718][test] Unwaive test_disaggregated_videomme[nemotron_na… - #17031
[NVBUG-6327718][test] Unwaive test_disaggregated_videomme[nemotron_na…#17031aswinvisva wants to merge 1 commit into
Conversation
|
/bot run --stage-list "DGX_B200-PyTorch-4" |
WalkthroughThe VideoMME E/PD test now runs repetitions 20 through 39 and records periodic thread stacks during execution. B200 and H100 test lists select the repetitions, QA entries use ChangesVideoMME repetition coverage
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
|
PR_Github #62622 [ run ] triggered by Bot. Commit: |
|
PR_Github #62622 [ run ] completed with state |
|
The two lines removed are Two other gaps against the description: nvbugs/6327718 is still P0 Could you attach a passing run of the fp8 variant on B200 and H20 (host + commit + pytest command), the way repair-bot did for nvfp4? For what it's worth the fp8 variant already runs unwaived pre-merge via |
brnguyen2
left a comment
brnguyen2
left a comment
There was a problem hiding this comment.
This looks like a no-op. [tests/integration/test_lists/waives.txt:16](https://github.com/NVIDIA/TensorRT-LLM/pull/17031/files#diff-621bd2af82a3b97c7a5948368d36c14582ffdf73361deb7f42d2ff395b22167eR16) still waives the same node ID unconditionally (accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_fp8], nvbugs/6478692), so the test stays skipped on every platform and CI can't confirm the fix. Also, the fp8 param only appears in test-db/l0_h100.yml:164 and qa/llm_function_core.txt:86 — l0_b200.yml lists only the qwen3vl/nvfp4 variants — so the B200 entry was already vestigial. Worth confirming the 6327718 signature matches what you believe is fixed, and updating that bug alongside the removal.
08015b1 to
eefdfc4
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #65073 [ run ] triggered by Bot. Commit: |
|
PR_Github #65073 [ run ] completed with state
|
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #65111 [ run ] triggered by Bot. Commit: |
|
PR_Github #65111 [ run ] completed with state |
eefdfc4 to
9264ea2
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
1 similar comment
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
9264ea2 to
6dc8867
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
PR_Github #65340 [ run ] triggered by Bot. Commit: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/test_lists/test-db/l0_b200.yml`:
- Around line 62-82: Add the omitted TestVideoMMEEPD.test_disaggregated_videomme
FP8 entries for rep20–rep39 to tests/integration/test_lists/test-db/l0_b200.yml
to provide B200 waiver coverage. The corresponding site in
tests/integration/test_lists/test-db/l0_h100.yml requires no direct change
because its *h100* targeting does not cover H20; alternatively, create and use
an H20-specific test-db list for H20 waiver coverage.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a2add11e-57b7-4626-9e05-9ceb25051585
📒 Files selected for processing (5)
tests/integration/defs/accuracy/test_epd_disagg_multimodal.pytests/integration/test_lists/qa/llm_function_core.txttests/integration/test_lists/test-db/l0_b200.ymltests/integration/test_lists/test-db/l0_h100.ymltests/integration/test_lists/waives.txt
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/integration/test_lists/waives.txt
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[qwen3vl_2b_instruct-rep20] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep20] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep21] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep22] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep23] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep24] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep25] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep26] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep27] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep28] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep29] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep30] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep31] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep32] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep33] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep34] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep35] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep36] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep37] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep38] | ||
| - accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_nvfp4-rep39] |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- changed files ---'
git status --short
git diff --stat
printf '%s\n' '--- relevant diff: B200 ---'
git diff -- tests/integration/test_lists/test-db/l0_b200.yml
printf '%s\n' '--- relevant diff: H100 ---'
git diff -- tests/integration/test_lists/test-db/l0_h100.yml
printf '%s\n' '--- matching test files and list files ---'
git ls-files | rg 'test_epd_disagg_multimodal\.py|tests/integration/test_lists/(test-db|qa)/.*(b200|h100|h20|h20|video|multimodal)'
printf '%s\n' '--- GPU/model/variant references ---'
rg -n -i 'H20|H100|B200|fp8|nemotron_nano_v3_omni_fp8|qwen3vl_2b_instruct|rep20|rep39' \
tests/integration/test_lists tests 2>/dev/null | head -n 500Repository: NVIDIA/TensorRT-LLM
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- l0_b200 metadata and relevant entries ---'
sed -n '1,35p' tests/integration/test_lists/test-db/l0_b200.yml
sed -n '50,95p' tests/integration/test_lists/test-db/l0_b200.yml
printf '%s\n' '--- l0_h100 metadata and relevant entries ---'
sed -n '1,35p' tests/integration/test_lists/test-db/l0_h100.yml
sed -n '140,185p' tests/integration/test_lists/test-db/l0_h100.yml
printf '%s\n' '--- test source outline and relevant symbols ---'
ast-grep outline tests/integration/defs/accuracy/test_epd_disagg_multimodal.py
rg -n -C 8 'TestVideoMMEEPD|test_disaggregated_videomme|qwen3vl_2b_instruct|nemotron_nano_v3_omni_(fp8|nvfp4)' \
tests/integration/defs/accuracy/test_epd_disagg_multimodal.py
printf '%s\n' '--- exact matching entries by test-db file ---'
python3 - <<'PY'
from pathlib import Path
needle = "accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme"
for p in sorted(Path("tests/integration/test_lists/test-db").glob("*.yml")):
hits = [line.strip() for line in p.read_text().splitlines()
if needle in line]
if hits:
print(f"{p}: {len(hits)} entries")
for line in hits:
print(" " + line)
PYRepository: NVIDIA/TensorRT-LLM
Length of output: 22475
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- marker definitions ---'
rg -n -C 5 'skip_pre_hopper|skip_pre_blackwell' tests/integration | head -n 160
printf '%s\n' '--- exact VideoMME waiver records ---'
rg -n -C 2 'test_epd_disagg_multimodal\.py::TestVideoMMEEPD::test_disaggregated_videomme' \
tests/integration/test_lists/waives.txt || true
printf '%s\n' '--- test-db GPU selectors ---'
python3 - <<'PY'
from pathlib import Path
import re
for p in sorted(Path("tests/integration/test_lists/test-db").glob("*.yml")):
text = p.read_text()
if re.search(r'(?i)\bh20\b|\*h20\*|\*b200\*|\bb200\b', text):
selectors = []
for m in re.finditer(r'(?im)^\s*-\s*[\'"]?(\*?[a-z0-9_-]*h20[a-z0-9_*-]*|\*?[a-z0-9_-]*b200[a-z0-9_*-]*)[\'"]?\s*$', text):
selectors.append(m.group(1))
exact_fp8 = [
line.strip() for line in text.splitlines()
if "test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_fp8-" in line
]
print(f"{p}: selectors={selectors or 'inspect'} exact_fp8_entries={len(exact_fp8)}")
PY
printf '%s\n' '--- all exact FP8 entries and their files ---'
rg -l 'accuracy/test_epd_disagg_multimodal\.py::TestVideoMMEEPD::test_disaggregated_videomme\[nemotron_nano_v3_omni_fp8-' \
tests/integration/test_lists || trueRepository: NVIDIA/TensorRT-LLM
Length of output: 17381
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- marker definitions and hardware predicates ---'
rg -n -C 8 'skip_pre_hopper|skip_pre_blackwell|def is_hopper|def is_blackwell|sm_90|sm_100|sm_120' \
tests/integration --glob '*.py' | head -n 240
printf '%s\n' '--- QA coverage for the exact test ---'
python3 - <<'PY'
from pathlib import Path
needle = "accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme"
for p in sorted(Path("tests/integration/test_lists/qa").glob("*")):
if not p.is_file():
continue
hits = [line.strip() for line in p.read_text(errors="replace").splitlines()
if needle in line]
if hits:
print(f"{p}: {len(hits)} entries")
for line in hits:
print(" " + line)
PY
printf '%s\n' '--- QA file context for the exact test ---'
rg -n -C 4 'test_epd_disagg_multimodal\.py|VideoMMEEPD|nemotron_nano_v3_omni_fp8' \
tests/integration/test_lists/qa/llm_function_core.txt || true
printf '%s\n' '--- candidate H20 test-db filenames and selectors ---'
find tests/integration/test_lists/test-db -maxdepth 1 -type f -printf '%f\n' | sort | rg -i 'h20|h100|b200'
rg -n -i -C 2 'gpu:|h20|h100|b200' tests/integration/test_lists/test-db/l0_dgx_h100.yml \
tests/integration/test_lists/test-db/l0_h100.yml | head -n 180Repository: NVIDIA/TensorRT-LLM
Length of output: 37798
Add FP8 coverage for the intended GPU tier.
TestVideoMMEEPD.test_disaggregated_videomme runs the FP8 variant for rep20–rep39 on Hopper and newer GPUs. l0_b200.yml omits these entries, while l0_h100.yml targets only *h100* and cannot cover H20. Add the entries to the B200 list for a B200 waiver, or create/use an H20 test-db list for an H20 waiver.
Coverage summary: no test functions changed; coverage is insufficient for B200 FP8 or H20.
📍 Affects 2 files
tests/integration/test_lists/test-db/l0_b200.yml#L62-L82(this comment)tests/integration/test_lists/test-db/l0_h100.yml#L153-L172
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/integration/test_lists/test-db/l0_b200.yml` around lines 62 - 82, Add
the omitted TestVideoMMEEPD.test_disaggregated_videomme FP8 entries for
rep20–rep39 to tests/integration/test_lists/test-db/l0_b200.yml to provide B200
waiver coverage. The corresponding site in
tests/integration/test_lists/test-db/l0_h100.yml requires no direct change
because its *h100* targeting does not cover H20; alternatively, create and use
an H20-specific test-db list for H20 waiver coverage.
Source: Path instructions
|
PR_Github #65343 [ run ] triggered by Bot. Commit: |
|
PR_Github #65340 [ run ] completed with state |
|
PR_Github #65343 [ run ] completed with state |
6dc8867 to
caf2375
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #65380 [ run ] triggered by Bot. Commit: |
Throwaway diagnostic PR — not for merge. Iteration 4. * Expand to 100 reps (range(60,160)) so each CI run exercises 100 unique test IDs. With 9 SLURM groups, ~11 reps per group run in parallel — all 100 complete in same wallclock time as the 7 we were getting before. * This gives ~63% probability of catching the 1% SIGABRT flake in a single run vs ~7% previously. * faulthandler.dump_traceback_later(60) still active. * max_workers=128 (workaround removed). Previous results: 21/21 videomme reps PASSED across 3 CI runs (rep20-59). Cumulative 0 failures observed. Continuing to accumulate. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
caf2375 to
5f8be05
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
1 similar comment
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #65403 [ run ] triggered by Bot. Commit: |
|
PR_Github #65404 [ run ] triggered by Bot. Commit: |
|
PR_Github #65403 [ run ] completed with state |
|
PR_Github #65380 [ run ] completed with state |
|
PR_Github #65404 [ run ] completed with state |
5 participants
…no_v3_omni_fp8] on B200 and H20
The root cause of the pre-fix failure —
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a unionattensorrt_llm/executor/proxy.pywhen the test infrastructure's session-reuse cache monkey-patchesMpiPoolSessionin the proxy module namespace with a factory function — is already fixed on main. The proxy code was refactored to identify pool-backed sessions by excluding(MpiCommSession, RemoteMpiCommSessionClient)instead ofisinstance(x, MpiPoolSession).Remove the two waivers linked to NVBug 6327718 so CI can verify the fix on both platforms.
Dev Engineer Review
nvbug: 6327718waivers for the targeted test on B200 and H20.rep20throughrep39and periodic thread-stack dumps.QA Engineer Review
tests/integration/test_lists/qa/llm_function_core.txt,tests/integration/test_lists/test-db/l0_b200.yml, andtests/integration/test_lists/test-db/l0_h100.yml.test_disaggregated_videommeto accept the repetition parameter.rep20throughrep39coverage in the B200 and H100 test lists.test-db/for CI andqa/for manual QA.Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.