[trtllm-agent]

Description

Temporarily waive 7 failed case(s) on DGX_H100. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_delays_commit_until_last_context_chunk broken on main for DGX_H100: failure signature failed on setup with tensorrt_llm.bindings.intern...hostoomerror: mmap failed: cannot allocate memory, onset 2026-11-05T07:53:46Z, latest failure 2026-11-05T19:23:27Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7053208 tracks the fix. Scope DGX_H100 is the stage prefix that ran the failing test.

• Bug: https://nvbugs/7053208 (details, logs and 7-day test history are in the bug)
• Restore coverage when: Remove this line when https://nvbugs/7053208 is fixed or closed and the test passes on DGX_H100 post-merge; un-waive follows the bug, not the PR.
• Scope reviewed by: auto-waive at 2026-11-05T21:05:50.165789+00:00

Tests in this break event:

• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_delays_commit_until_last_context_chunk on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_drops_previous_divergent_blocks on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_ignores_overlapping_request on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_releases_cancelled_request on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_retains_configured_number_of_turns[2] on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_without_params_uses_per_request_commit on DGX_H100
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_preemption_waits_for_a_connector_still_reading_the_victims_pages on DGX_H100

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

• [TRTLLMINF-451][infra] Capture setup timeouts and defer unfinished cleanup until JUnit save #19333
• [https://nvbugs/6858496][fix] Fail disagg gen requests whose grammar matcher cannot be created #20903
• [TRTLLM-15944][feat] Enable SWA scratch reuse for Gemma3/4 with TRTLLM attention #20971
• [https://nvbugs/6891165][fix] Pass chat_template_kwargs in the request body of the JSON-schema serve example #21026
• [None][feat] Rubin merge back: MLA, DSpark, DWDP, Helix #21067
• [TRTLLM-17516][fix] Bound context logits memory with CPU fallback #21315
• [None][feat] Keep hybrid speculative replay state in private KV cache buffers #21324
• [None][feat] GEN slot preemption: pause decode requests in place to admit at-risk waiters #21334
• [TRTLLM-17272][feat] Add public LLM health queries and a private-access lint for the serving frontends #21393
• [None][fix] Fix draft KV over-allocation during chunked prefill #21546
• [TRTLLM-16624][fix] Fix Helix MLA decode kernel selection #21582
• [TRTLLM-16731][fix] Autotuner: skip redundant per-profile work for exclude_from_cache ops in tuning mode #21593
• [None][feat] Responses: name the engine request behind each reply #21602
• [None][perf] Build MSA fmha_sm100 JIT kernels for the visible GPUs' target only #21681

full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_delays_commit_until_last_context_chunk SKIP (https://nvbugs/7053208)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_drops_previous_divergent_blocks SKIP (https://nvbugs/7053208)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_ignores_overlapping_request SKIP (https://nvbugs/7053208)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_releases_cancelled_request SKIP (https://nvbugs/7053208)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_retains_configured_number_of_turns[2] SKIP (https://nvbugs/7053208)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_without_params_uses_per_request_commit SKIP (https://nvbugs/7053208)
full:DGX_H100/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_preemption_waits_for_a_connector_still_reading_the_victims_pages SKIP (https://nvbugs/7053208)

What to check before merging:

• The bug's test-history links show the cases failing on main, not only on the trigger PR.
• The scope DGX_H100 matches the failing stage and no narrower waiver already exists.
• Nothing else is needed from you after merging: the bug owner removes these lines with the fix.

Provenance

• Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
• Run 2e53fc1c-881a-4bae-860a-8a58bfecaccc; snapshot SHA-256 b3efa90d2937fc83ec49ddd8ca2285fda6706230bb41e303acbc0e6b26a20290; config SHA-256 622a43f13fb241c37a015e4ec2c812d7c8cecc1a04413b2618ca4ac119a77c81
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_delays_commit_until_last_context_chunk: incident f01f310fdee8eb3117fa24e5e7620e3750870726af4006d80b4d266af7cf1bb5, observation f478c34f31b5ce6043bdd1cc4398f65f9a8c324a0f8cda06575ab36bc2ef61fc
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_drops_previous_divergent_blocks: incident eda6f7935cca4632afc4382466da9ac07453fbde0e7613d521fb2858b08484c7, observation bce437c3a4c022bfd69f9f5d56b7b33252d26a9486016de6b6ecc94c374aa3c5
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_ignores_overlapping_request: incident d2323e091c80c592a001353c31baa83ed3431a0d1f5fa15aad0dc8b1b04802f5, observation 17ab51a9e7efc66b096d23a66a9597235368e0957f2e882516762dde8a7bf301
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_releases_cancelled_request: incident a318d2886a3a5597228c28c808e700bfdbb7f80764e5f7242ec106b6ca85ab57, observation da2cea4ebe4396cd6ed1891fdf18f1e16096a26ef512b0f5a5d29694e063c965
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_retains_configured_number_of_turns[2]: incident aaebeaff5f180cbbbd8c1a6757d3c1294a0e4900676eb23329550b59f6e20683, observation f99e6d010fb5e63b8f5d665a9dfdcb4e2217b5243aee1884ee49d5e1c929c481
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_per_conversation_policy_without_params_uses_per_request_commit: incident dd85099b05e87d28fb8150eea9fd8f90095ffc5c4aff8b840f522e3d0ab4b43e, observation 4d8778c527a3ef7a1e4de38741595a8760a500baafce0d0b4dd71d067ea02e89
• unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py::test_preemption_waits_for_a_connector_still_reading_the_victims_pages: incident a2f89a168b3c80e0e3c5747748e99b2f2f98d5163205e3e43c660a6f8a26f56f, observation b6f2becc81877bfa26fc1f40e9384701dea641ef28d887e61cb0a8803d377311

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

• Please check this after reviewing the above items as appropriate for this PR.

[karljang]

Closing: these failures were a DGX_H100 host-memory condition (host OOM / CUDA invalid device context across 16 nodes, 37 of 50 builds passing), not a code break, and the stage recovered on its own. No waiver needed. The linked NVBug is being closed as not a bug.