[https://nvbugs/6448152][perf] diagnose 2x admission with PP consensus

[chienchunhung]

Caution

DIAGNOSTIC ONLY — DO NOT MERGE.

Question

Does the conservative generation-side KV-transfer admission budget cause the GB300 DeepSeek-R1 throughput regression, and does PP context-transfer consensus become material after that budget is relaxed?

This is the 2× admission / PP consensus enabled arm. The matched no-consensus arm is #16517.

Controlled change

• Start from the exact tested [https://nvbugs/6448152][perf] diagnose PP consensus traffic #16487 head (1522ea7230).

• Keep max_tokens_in_buffer: 131104; physical C++/NIXL transfer-buffer allocation is unchanged.

• Apply a diagnostic-only multiplier to PyExecutor's logical admission budget:

  2 * ceil(max_tokens_in_buffer / tokens_per_block)

• Remove only TRTLLM_DIAGNOSTIC_DISABLE_PP_CONTEXT_CONSENSUS_TRAFFIC=1; the [https://nvbugs/6448152][perf] publish PP transfer status from workers #16386 worker-published PP consensus path remains active.

• Leave the diagnostic multiplier default at 1; only the exact GB300 perf-sanity YAML sets it to 2.

For this FlashMLA workload, the expected base budget is 2,049 blocks. A 131,072-token request costs 2,048 blocks, so the effective 4,098-block budget admits two full transfers (4,096 blocks) without resizing the transport buffer.

The run must contain both markers:

DIAGNOSTIC ONLY: disaggregated admission base window=...
DIAGNOSTIC OBSERVED: disaggregated admission used ... exceeding the base window ... multiplier=2

It should also log activation of worker-published context-transfer consensus and must not log the no-consensus completion marker.

Experiment matrix

PP context consensus	Admission budget	Result
enabled	1×	~796.69 tok/s (preliminary #16386 point)
disabled	1×	798.58 tok/s (#16487)
enabled	2×	pending (this PR)
disabled	2×	pending (#16517)

The two 2× arms share the same code and differ only by the no-consensus environment flag. If their results are unexpectedly close to a threshold, rerun a same-harness 1× control to remove the negligible fast-return diagnostic method-call delta from the historical controls.

flowchart LR
    A["#16487 tested anchor<br/>1× / consensus off"] --> H["common logical 2× admission harness"]
    H --> OFF["#16517<br/>2× / consensus off"]
    H --> ON["#16518<br/>2× / consensus on"]

Loading

Interpretation

• Both 2× arms recover similarly: admission control dominates; PP consensus is overlapped or immaterial for this workload.
• Only 2×/consensus-off recovers: admission masks a PP-consensus bottleneck; both factors interact.
• Both improve, but consensus-off is materially higher: both admission and consensus contribute.
• Neither improves: investigate retained transfer lifecycle/state serialization or another bottleneck.

Validation

• Full changed-file pre-commit suite under Python 3.13: passed.
• Python bytecode compilation: passed.
• Focused pytest collection is unavailable on this Mac checkout because the local environment lacks transformers and the compiled TRT-LLM runtime; CI owns runtime validation.
• Independent diff review found no correctness blocker and confirmed the exact workload uses PyExecutor admission with the C++ NIXL transceiver.
• The arm-specific delta from [https://nvbugs/6448152][perf] diagnose 2x admission without PP consensus #16517 is exactly one YAML environment-variable removal.

Target CI

Run only:

GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1

Compare output-token throughput with 798.58 (#16487) and 1562.83 baseline.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #59814 [ run ] triggered by Bot. Commit: 87d3d2a Link to invocation

[tensorrt-cicd]

PR_Github #59814 [ run ] completed with state FAILURE. Commit: 87d3d2a
/LLM/main/L0_MergeRequest_PR pipeline #48223 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation