【No.21】Hybrid-async 多模态可归因性能优化

[Pinoeer-kingxi]

Summary

Closes #151.

This PR now separates Task 21 into attributable, default-off optimization paths instead of presenting one bundled throughput claim:

1. Schedule-matched chunk overlap: fetch and run old-policy actor forward on prompt-group-aligned chunks while later rollout groups are still being produced.
2. ProcessorPool + train-forward log-prob reuse: expose the existing multimodal processor pool in the reference launcher and, under strict true-on-policy guards, reuse detached log-probs from the training forward instead of running a redundant actor log-prob forward.

The rollout batch, sample/token/image work, full-batch advantage domain, and one-update-per-mini optimizer semantics remain unchanged.

Important benchmark scope correction

The existing published GPU evidence is:

• fixed Qwen3-VL-8B-Instruct;
• actor TP=2 x CP=2 x DP=1 plus one rollout GPU;
• four paired seeds in ABBAABBA order;
• a fresh process for every run;
• exactly one rollout and one optimizer step per run.

It is therefore a fresh-process first-step benchmark, not steady-state training throughput. The analyzer now records:

measurement_scope=fresh_process_first_step

for --steady-windows 0-0 and emits an explicit claim limit. A steady-state claim requires multi-step runs, excluded warmup, preregistered later windows, and at least two independent runs per condition.

Additional optimizations in this update

• Parallel multimodal preprocessing: expose the existing ProcessorPool through MM_PROCESSOR_POOL_SIZE in the launcher, making processor concurrency explicit and independently switchable.
• Train-forward log-prob reuse: under strict true-on-policy guards, reuse detached log-probs from the training forward and skip the redundant actor log-prob forward.
• Attributable controls: keep ProcessorPool, log-prob reuse, and chunk overlap behind independent environment controls and ablation conditions. Reuse and overlap cannot be enabled together, preventing their performance effects from being conflated.
• Benchmark scope classification: mark fresh-process, single-step runs as fresh_process_first_step and emit the corresponding claim limit.

Controls and ablations

Condition	Environment controls	Purpose
B	MM_PROCESSOR_POOL_SIZE=0 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=0	compatibility baseline
P	MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=0	ProcessorPool only
P+R	MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=1 HYBRID_PIPELINE_FORWARD=0	ProcessorPool + train-forward reuse
P+S	MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=1	ProcessorPool + chunk overlap

The reference launcher rejects HYBRID_REUSE_TRAIN_LOGPROBS=1 together with HYBRID_PIPELINE_FORWARD=1, so their effects cannot be accidentally conflated.

Correctness guards for train-forward reuse

The fast path requires:

• Hybrid mode and exactly one optimizer mini per rollout partition;
• deterministic forward behavior (zero attention/hidden/LoRA dropout);
• no reward KL, KL loss, OPD, old-actor snapshot, rollout-logprob old-policy source, custom before-log-prob hook, or routing replay;
• TIS when max_staleness > 0;
• one CP-aligned rollout_log_probs entry per response.

When actor log-probs are intentionally absent, rollout log-probs are used only to construct zero-KL tensors for advantage shape/device compatibility. Reward KL still requires real actor and reference log-probs and fails explicitly if either is missing.

Existing first-step evidence

The prior schedule-matched overlap experiment remains valid within its narrower scope:

Seed	Throughput baseline -> overlap	Speedup	Phase-1 reduction
20260811	597.43 -> 658.46 token/s	+10.22%	-19.11%
20260812	576.83 -> 647.27 token/s	+12.21%	-20.33%
20260813	559.25 -> 630.86 token/s	+12.80%	-23.53%
20260814	569.19 -> 642.15 token/s	+12.82%	-21.81%

Aggregate paired geometric-mean first-step throughput speedup: 12.01%. This result must not be extrapolated to sustained training throughput.

The frozen replay parity remains unchanged: 1,024 rank-local samples match for tokens, masks, rewards, advantages, returns, multimodal tensors, and dynamic microbatch schedules; loss is identical, maximum log-prob delta is 2.3842e-7, and gradient-norm symmetric relative difference is 0.207%.

Steady-state benchmark protocol

For each B/P/P+R/P+S condition:

• run at least 40 optimizer steps per fresh process;
• exclude warmup and preregister later measurement windows;
• run at least two independent repetitions;
• keep candidate commit, model/data, seed policy, topology, batch/token limits, image, dependencies, and hardware fixed;
• report mean, range, per-run values, finite loss/reward/PPO-KL/grad-norm coverage, sample conservation, image count, and optimizer update count.

No new multi-step GPU result is claimed yet. An 8 x RTX A6000 fixed-hardware campaign is queued with the actor on four GPUs at TP2 x CP2 x DP1 and two TP2 rollout engines on four GPUs. Sustained-throughput results will be reported only after all eight B/P/P+R/P+S runs and the preregistered guardrails pass.

Verification

Current head: e2be8cd158609cc2dfae72b7ba92df72cacb3091.

128 passed, 8 skipped

The skips are existing dependency/runtime gates for unavailable local Ray/Megatron components. The following also pass:

• pre-commit run --all-files --show-diff-on-failure;
• Ruff check and format;
• Python compileall;
• Bash syntax and ShellCheck (excluding the script's existing dynamic-source SC1090/SC1091 diagnostics);
• git diff --check.

Scope

Changed:

• guarded Hybrid true-on-policy train-forward log-prob reuse;
• advantage handling when the dedicated actor forward is intentionally absent;
• reference launcher controls for ProcessorPool and reuse;
• benchmark measurement-scope classification;
• targeted tests and English/Chinese documentation.

Not changed:

• rollout generation, reward calculation, GRPO grouping, optimizer count, checkpoint/data formats, or dependencies;
• owner/ref multimodal payload transport;
• the existing schedule-replay and TransferQueue correctness contracts.

[xiaoliang0601]

Hi @Pinoeer-kingxi ，是否还要进一步验收和整理代码？我看当前还是 Draft 状态。

[Pinoeer-kingxi]

已完成进一步验收和代码整理，PR 现已转为 Ready：

• frozen-input parity 已通过；
• 物理 GPU 1,2,3,4,6 上完成 baseline 4 次 + experiment 4 次；
• paired token-throughput geometric mean 提升 12.01%，phase-1 geometric mean 降低 21.21%，experiment overlap 100%；
• 五张对比曲线、精简 CSV、SHA256、完整命令与日志说明已回填 PR 和 [Task 21][Hybrid-Async] 固定样本窗口流水化 actor forward，消除整批多模态等待 #151；
• 当前 head 56300da，最新 CI 正在运行。

[Pinoeer-kingxi]

补充：当前 head 3a1c9f88d0b4b187545bd95e3dd68643c0ef1e55 的 CI 已全部通过：Lint、Pre-commit Checks、Python 3.10/3.11/3.12 Tests 均为绿色。

[Pinoeer-kingxi]

您好@xiaoliang0601 ，我已更新，麻烦您验收一下

[xiaoliang0601]

收益部分：现有 benchmark 是固定 Qwen3-VL、TP2×CP2×DP1 拓扑下的四组 fresh-process、单 optimizer-step 运行。它更接近 first-step benchmark，而不是稳态训练吞吐。

效果部分：感觉没有完全对齐，甚至差得有点大。

[Pinoeer-kingxi]

好的明白，我接下来进行优化