Add an all-reduce bandwidth bench at prefill chunk sizes

[lvyufeng]

Why

tp_all_reduce is 25.7% of 65K prefill time, and the existing bench_qwen_tp_allreduce only covers decode/draft shapes (1 and 8 rows, ~80 KB). Long prefill reduces whole chunks: 4096 rows x 5120 hidden is 41.9 MB per call, two calls per layer, 85.9 GB over a 64-layer 65K prefill. Without a measurement in that regime there was no way to tell whether that 25% was a hardware floor or our own overhead.

Measured

4x RTX 2080 Ti, TP4 (GPU0-1 PHB, GPU2-3 NV2, cross-pair SYS over QPI):

rows	payload	ms/call	algbw	busbw
512	5.2 MB	0.895	5.86 GB/s	8.79 GB/s
1024	10.5 MB	1.746	6.01	9.01
2048	21.0 MB	3.496	6.00	9.00
4096	41.9 MB	6.864	6.11	9.17
8192	83.9 MB	13.725	6.11	9.17

Conclusion

Bandwidth is flat from 5 MB upward, so the collective is bandwidth-bound rather than launch-bound. 2048 calls x 6.864 ms = 14.06 s, against a measured tp_all_reduce of 14.64 s — the phase is at its floor.

This also explains the chunk sweep: doubling the chunk halves the call count and doubles per-call time for the same total, and measured prefill at 4096/8192/16384 was 1157/1140/1125 tok/s. Chunk size is not a lever here.

Test plan

New standalone bench, no engine code touched. Run with:

for r in 0 1 2 3; do CUDA_VISIBLE_DEVICES=$r build/tests/bench_qwen_prefill_allreduce \
  --nccl-id-path /tmp/ar.id --tp-world 4 --tp-rank $r --device 0 --iters 30 & done; wait

Requires an NCCL-enabled build (-DNCCL_ROOT=...); prints a clear message and exits non-zero otherwise.

[lvyufeng]

Closing: this cannot build against master, and its subject is now measured on the hardware that actually runs this engine.

It cannot build, three ways. The target it links is dsv4_cpp_core, which no longer exists — master's CMake tree defines pocket_cpp_core, and there are zero occurrences of dsv4_cpp_core anywhere in it. The source calls dsv4::nccl_available() and dsv4::nccl_all_reduce_sum_f16_inplace, while cpp_engine/include/tp_comm.hpp on master declares namespace pocket with tp_comm_available() and tp_all_reduce_sum_f16_inplace. And it is a CUDA program — #include <cuda_fp16.h>, cudaMalloc, cudaDeviceSynchronize — so even renamed to the pocket symbols it would only ever build on the GPU host, where the collective in question does not run.

Its subject is covered. The question this bench exists to answer — how the TP all-reduce behaves as the payload grows with the prefill chunk, and whether that is a hardware floor — is answered on the Ascend side in docs/performance/ascend_decode_collective_ab.md §2, using bench_qwen_ascend_allreduce, which is in master and covers 10 KB / 80 KB / 640 KB plus the prefill-scale sweep. The finding there is sharper than a bandwidth curve: HcclAllReduce blocks the calling host thread for ~0.35 ms per call regardless of payload, 95% of the collective is host time inside the call, and the device has room during it — which is why the payload sweep in §7 shows the chunk growing while prefill throughput falls (1157 → 1140 → 1125 tok/s at 4096 → 8192 → 16384) rather than a bandwidth ceiling being reached.

The branch is deleted. Thanks — the payload-sweep question was the right one to ask, it just turned out to have a latency answer rather than a bandwidth one, on a different backend than this was written for.