diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/README.md b/docs/assets/task21-hybrid-mm-forward-pipeline/README.md new file mode 100644 index 000000000..e03b30381 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/README.md @@ -0,0 +1,97 @@ +# Task 21: Hybrid-async multimodal pipeline benchmark results + +## Steady-state result + +The formal steady-state campaign completed on commit `e2be8cd158609cc2dfae72b7ba92df72cacb3091` using physical GPUs `0,1,2,6,7`: actor TP2 x CP2 x DP1 on four GPUs and one rollout engine on one GPU. Each B/P/P+R/P+S condition used two paired seeds (`20260816`, `20260817`), 40 optimizer updates, 10 warmup updates, and measured windows `10-19`, `20-29`, and `30-39`. + +| Condition | Steady token throughput (mean, range) | Samples/s (mean) | Geomean vs reference | +|---|---:|---:|---:| +| B | 1,209.50 (1,144.11-1,274.89) token/s | 1.562 | reference | +| P | 1,197.49 (1,184.17-1,210.81) token/s | 1.516 | -0.85% token/s vs B | +| P+R | 1,565.68 (1,559.71-1,571.65) token/s | 1.988 | **+29.64% token/s vs B**; +30.75% vs P | +| P+S | 1,188.79 (1,142.59-1,234.99) token/s | 1.498 | -1.64% token/s vs B | + +P+R is positive in both paired seeds (`+36.33%`, `+23.28%` token/s vs B), with a 21.50% geometric-mean step-time reduction. The ProcessorPool-only and chunk-overlap paths do not show a sustained throughput gain in this campaign; their values remain reported as attributable ablations rather than being combined into the P+R claim. + +The three warmup-excluded window means (token/s) were: + +| Condition | 10-19 | 20-29 | 30-39 | +|---|---:|---:|---:| +| B | 1,221.46 | 1,188.81 | 1,219.31 | +| P | 1,202.85 | 1,204.60 | 1,185.82 | +| P+R | 1,556.00 | 1,565.52 | 1,575.02 | +| P+S | 1,198.25 | 1,178.24 | 1,190.35 | + +## 40-step training curves + +The gradient norm, training loss, and raw reward curves below come from the same eight formal real-training runs used for the steady-state throughput result. They show every optimizer step from `0` through `39`; the shaded `0-9` interval is warmup, thick lines are the two-seed means, thin lines are the individual seeds, and bands span the two seed values at each step. The values are plotted directly from the TensorBoard scalar exports without smoothing or interpolation. The underlying per-step data are available in [`task21_steady_training_curves.csv`](steady-state/task21_steady_training_curves.csv). + +| Condition | Steady grad norm mean (max) | Steady loss mean | Steady raw reward mean | +| --- | ---: | ---: | ---: | +| B | 0.67076 (2.20789) | 0.01818 | 0.42181 | +| P | 0.75002 (3.24743) | 0.01381 | 0.41393 | +| P+R | 1.43510 (43.00600) | 0.01920 | 0.41706 | +| P+S | 0.62810 (2.39713) | 0.01823 | 0.41888 | + +![40-step gradient norm](steady-state/task21_steady_grad_norm.png) + +P+R seed `20260817` has one finite gradient-norm spike at step `21` (`43.00600`). It returns to the surrounding range on the next step, and the corresponding loss (`0.02013`) and raw reward (`0.39453`) remain within the observed trajectories rather than diverging. + +![40-step training loss](steady-state/task21_steady_loss.png) + +![40-step raw reward](steady-state/task21_steady_reward.png) + +Every formal run conserved 256 actor samples per update and exactly 40 optimizer updates; measured image count was 1 per sample, metrics were finite, and all five selected GPUs have steady NVML samples. The analyzer reports `measurement_scope=steady_state`, `validation=passed`, two independent seeds, and 30 measured optimizer steps per run. The reviewable campaign report and paired CSVs are checked in under [`steady-state/`](steady-state/); the report records the benchmark commit separately from later documentation and merge-conflict fixes. + +## Protocol + +- Performance-code commit: `780c742793fea54acdb28cb40ffb591fb909ba51`. +- Base commit: `9a5674afde12f608698ab4f60cdb9849a0eb6cb3`. +- Hardware: physical RTX A6000 GPUs `1,2,3,4,6`; four actor GPUs (`TP=2`, `CP=2`) plus one rollout GPU. +- Workload: Qwen3-VL-8B-Instruct, OpenR1-Multimodal, 256 generated samples per run, response cap 1024, one real optimizer step. +- Design: paired seeds `20260811`–`20260814`, ABBAABBA order; baseline and experiment differ only in `--hybrid-pipeline-overlap`. +- Statistics: four fresh-process paired runs, each containing one optimizer step. The analyzer validated every run and all preregistered first-step targets. + +These results measure first-step latency and throughput under the fixed Qwen3-VL, +TP2 x CP2 x DP1 profile. They do not establish steady-state training +throughput. A steady-state claim requires multi-step runs with warmup excluded +and later measurement windows declared before analysis. + +## Result summary + +| Metric | Baseline | Experiment | Result | +|---|---:|---:|---:| +| step token throughput, arithmetic mean | 575.68 token/s | 644.69 token/s | paired geometric-mean speedup **12.01%** | +| hybrid phase-1 time, arithmetic mean | 165.38 s | 130.38 s | paired geometric-mean reduction **21.21%** | +| end-to-end step time, arithmetic mean | 342.24 s | 305.61 s | **10.70% lower** | +| producer/actor overlap | 0/4 runs | 4/4 runs | **100%** experiment overlap | +| first-step-window GPU utilization, arithmetic mean | 40.01% | 44.37% | +4.36 percentage points | +| sampled peak VRAM, maximum | 47,380 MiB | 47,140 MiB | no regression | + +Every paired run exceeded the preregistered 5% throughput target: `10.22%`, `12.21%`, `12.80%`, and `12.82%`. Every pair also reduced phase-1 time by `19.11%`–`23.53%`. + +The throughput plot shows both total-token and response-token throughput for each matched seed. The experiment point is above its baseline point in every pair. + +![Step throughput](task21_step_throughput.png) + +The phase-1 plot shows the producer/actor overlap interval, phase-1 latency, and producer lead. The experiment overlaps producer work with actor forward in all four runs, while the baseline deliberately disables that overlap. + +![Phase-1 overlap](task21_phase1_overlap.png) + +The GPU plot combines the 500 ms NVML time series with first-step-window utilization, idle ratio, and sampled peak VRAM. It describes these fresh-process runs only. + +![GPU utilization and VRAM](task21_gpu_util_vram.png) + +The quality plot pairs reward, response length, truncation, loss, gradient norm, and PPO KL by seed. Workload, reward, response length, and truncation are exact within each pair; loss differs by at most `6.21e-5`, PPO KL remains within `2.37e-11`, and every value is finite. Seed `20260814` shows a disclosed gradient-norm difference (`1.0033` vs `0.4775`) caused by different asynchronous microbatch grouping; the deterministic replay parity run controls grouping and bounds the patched-path gradient-norm difference to `0.207%`. + +![Correctness and quality](task21_correctness_quality.png) + +The window summary shows each paired throughput speedup and the geometric mean against the preregistered 5% threshold. + +![Window and paired summary](task21_window_summary.png) + +## Correctness evidence + +The final replay parity dataset contains 1,024 samples across four actor ranks. Baseline/control/experiment agree exactly for tokens, response lengths, loss masks, rewards, raw rewards, advantages, returns, truncation flags, multimodal tensors, and dynamic-microbatch schedules. Loss is exactly `0.03446025028824806`; experiment log-probability maximum absolute difference is `2.3842e-7`; gradient-norm symmetric relative difference is `0.207%` (below the BF16 0.5% guardrail). + +Full raw logs, manifests, TensorBoard events, timeline JSONL, NVML CSV, summaries, and generated figures are stored under `/data01/LWX/relax-task21/`. Compact paired data and checksums are included in this directory. diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/SHA256SUMS b/docs/assets/task21-hybrid-mm-forward-pipeline/SHA256SUMS new file mode 100644 index 000000000..172e31328 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/SHA256SUMS @@ -0,0 +1,10 @@ +5016e5fc77093f2df3d85942e8abf95999738c75191c17a37176d53bbfc8a4b3 /data01/LWX/relax-task21/comparisons/FORMAL-FINAL8-Q3VL8B-R1024-C4-DET-GPU12346-FIRSTSTEP/comparison_summary.json +6e72053891db350c2863bbf3508188a3740d13d176cd86eaf0b56261d3875938 /data01/LWX/relax-task21/comparisons/FORMAL-FINAL8-Q3VL8B-R1024-C4-DET-GPU12346-FIRSTSTEP/console-final.log +025300b86959bc7c0dfb7dd941989423be873a3b8b06452c1697a8886da73018 docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv +da9b541bc2fbd26d5a4730eb9096d3b152d9470d2dd81c03f5104b4a62d03306 docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv +7408c899219b63619335ebf42f6374421df46a664b7d22c4a45ed03567b38f5e docs/assets/task21-hybrid-mm-forward-pipeline/task21_correctness_quality.png +38c6f272370cb0e12abb1d9a403596681ae4f150f30624e35d0b85943adf8c91 docs/assets/task21-hybrid-mm-forward-pipeline/task21_gpu_util_vram.png +e1256d0fdd7b5b22f81411ba8e20224f26f58e0955142352a37b492aa9b6a708 docs/assets/task21-hybrid-mm-forward-pipeline/task21_phase1_overlap.png +bbad23dc4059ffc0610786b4d042b9af37558a8753cb298e5bf5429e44011134 docs/assets/task21-hybrid-mm-forward-pipeline/task21_step_throughput.png +c989433666edfcecbe5d0761c0e03b41d104772a232c15fb6abdfe5c6699bd0a docs/assets/task21-hybrid-mm-forward-pipeline/task21_window_summary.png +ea8cfbd0958c16f54ef799aecf9a9bc45601b7861e8dd01ba7dd8a8ed033eb64 docs/assets/task21-hybrid-mm-forward-pipeline/window_speedup_summary.csv diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv new file mode 100644 index 000000000..7f00a2b90 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv @@ -0,0 +1,5 @@ +seed,baseline_step_token_per_s,experiment_step_token_per_s,throughput_speedup_percent,baseline_phase1_s,experiment_phase1_s,phase1_reduction_percent,baseline_step_time_s,experiment_step_time_s,baseline_peak_vram_mib,experiment_peak_vram_mib,raw_reward,truncated_ratio,baseline_loss,experiment_loss,baseline_grad_norm,experiment_grad_norm +20260811,597.4306030273438,658.458984375,10.215141480601897,169.09609985351562,136.78187561035156,19.10997608528948,348.0404357910156,315.78277587890625,47380.0,47140.0,0.296875,0.1875,0.013692490756511688,0.013677351176738739,0.4145539104938507,0.4117335081100464 +20260812,576.8270874023438,647.2715454101562,12.21240464365998,171.17022705078125,136.36746215820312,20.33225374074731,353.73858642578125,315.24017333984375,46818.0,46898.0,0.41796875,0.10546875,0.012462313286960125,0.012400214560329914,0.44756120443344116,0.4443718492984772 +20260813,559.2531127929688,630.8582763671875,12.803713012273654,162.420166015625,124.19634246826172,23.53391483646563,342.5908508300781,303.7052917480469,46912.0,47104.0,0.35546875,0.11328125,-0.021680761128664017,-0.021680761128664017,0.6372919082641602,0.6365648508071899 +20260814,569.1944580078125,642.1531372070312,12.817882917302992,158.83006286621094,124.18817901611328,21.81065928260567,324.609619140625,287.7288818359375,46916.0,46824.0,0.38671875,0.1640625,0.004201771225780249,0.004198123700916767,1.0032663345336914,0.4775027334690094 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv new file mode 100644 index 000000000..44cc84d57 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv @@ -0,0 +1,5 @@ +actor_fetch_samples:baseline,actor_fetch_samples:experiment,actor_fetch_samples:relative_delta,actor_multimodal_tensor_bytes:baseline,actor_multimodal_tensor_bytes:experiment,actor_multimodal_tensor_bytes:relative_delta,actor_response_tokens:baseline,actor_response_tokens:experiment,actor_response_tokens:relative_delta,actor_total_tokens:baseline,actor_total_tokens:experiment,actor_total_tokens:relative_delta,nvml_peak_memory_mib:baseline,nvml_peak_memory_mib:delta,nvml_peak_memory_mib:experiment,perf/hybrid_phase1_time:baseline,perf/hybrid_phase1_time:experiment,perf/hybrid_phase1_time:improvement,perf/step_resp_token_per_s:baseline,perf/step_resp_token_per_s:experiment,perf/step_resp_token_per_s:improvement,perf/step_time:baseline,perf/step_time:experiment,perf/step_time:improvement,perf/step_time:p95_baseline,perf/step_time:p95_experiment,perf/step_time:p95_regression,perf/step_token_per_s:baseline,perf/step_token_per_s:experiment,perf/step_token_per_s:improvement,perf/wall_clock_samples_per_s:baseline,perf/wall_clock_samples_per_s:experiment,perf/wall_clock_samples_per_s:improvement,producer_lead_at_first_forward:baseline,producer_lead_at_first_forward:delta,producer_lead_at_first_forward:experiment,rollout/raw_reward:baseline,rollout/raw_reward:delta,rollout/raw_reward:experiment,rollout/truncated_ratio:baseline,rollout/truncated_ratio:delta,rollout/truncated_ratio:experiment,seed,train/grad_norm:baseline,train/grad_norm:delta,train/grad_norm:experiment,train/loss:baseline,train/loss:delta,train/loss:experiment,train/pg_clipfrac:baseline,train/pg_clipfrac:delta,train/pg_clipfrac:experiment,train/ppo_kl:baseline,train/ppo_kl:delta,train/ppo_kl:experiment +256,256,0.0,1327110144,1327110144,0.0,119338,119338,0.0,207930,207930,0.0,47380.0,-240.0,47140.0,169.09609985351562,136.78187561035156,0.19109976085289482,342.8854675292969,377.91168212890625,0.10215135348836757,348.0404357910156,315.78277587890625,0.09268365567579862,348.0404357910156,315.78277587890625,-0.09268365567579862,597.4306030273438,658.458984375,0.10215141480601897,0.7355467172030487,0.8106838610417711,0.10215142299109847,0.0,0.0,0.0,0.296875,0.0,0.296875,0.1875,0.0,0.1875,20260811,0.4145539104938507,-0.0028204023838043213,0.4117335081100464,0.013692490756511688,-1.5139579772949219e-05,0.013677351176738739,0.0,0.0,0.0,-2.8720626137762606e-12,2.8720626137762606e-12,0.0 +256,256,0.0,1468078080,1468078080,0.0,109958,109958,0.0,204046,204046,0.0,46818.0,80.0,46898.0,171.17022705078125,136.36746215820312,0.20332253740747308,310.8453674316406,348.80706787109375,0.12212406687322264,353.73858642578125,315.24017333984375,0.10883294772823704,353.73858642578125,315.24017333984375,-0.10883294772823704,576.8270874023438,647.2715454101562,0.1221240464365998,0.7236982614383574,0.8120792387841379,0.12212407028603667,0.0,0.0,0.0,0.41796875,0.0,0.41796875,0.10546875,0.0,0.10546875,20260812,0.44756120443344116,-0.0031893551349639893,0.4443718492984772,0.012462313286960125,-6.209872663021088e-05,0.012400214560329914,0.0,0.0,0.0,0.0,0.0,0.0 +256,256,0.0,1581717504,1581717504,0.0,94331,94331,0.0,191595,191595,0.0,46912.0,192.0,47104.0,162.420166015625,124.19634246826172,0.2353391483646563,275.345947265625,310.6004638671875,0.12803717269734372,342.5908508300781,303.7052917480469,0.1135043711407171,342.5908508300781,303.7052917480469,-0.1135043711407171,559.2531127929688,630.8582763671875,0.12803713012273654,0.7472470422946981,0.842922421688908,0.1280371469927848,0.0,0.0,0.0,0.35546875,0.0,0.35546875,0.11328125,0.0,0.11328125,20260813,0.6372919082641602,-0.0007270574569702148,0.6365648508071899,-0.021680761128664017,0.0,-0.021680761128664017,0.0,0.0,0.0,-2.2905178188176167e-12,2.2905178188176167e-12,0.0 +256,256,0.0,958666752,958666752,0.0,109918,109918,0.0,184766,184766,0.0,46916.0,-92.0,46824.0,158.83006286621094,124.18817901611328,0.2181065928260567,338.615966796875,382.0193786621094,0.12817886963750502,324.609619140625,287.7288818359375,0.11361566364646236,324.609619140625,287.7288818359375,-0.11361566364646236,569.1944580078125,642.1531372070312,0.12817882917302992,0.7886395993986166,0.8897264618223858,0.12817878090429868,0.0,0.0,0.0,0.38671875,0.0,0.38671875,0.1640625,0.0,0.1640625,20260814,1.0032663345336914,-0.525763601064682,0.4775027334690094,0.004201771225780249,-3.6475248634815216e-06,0.004198123700916767,0.0,0.0,0.0,-2.20278361523496e-11,2.3654630120981124e-11,1.6267939686315236e-12 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_R_vs_B_paired_runs.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_R_vs_B_paired_runs.csv new file mode 100644 index 000000000..a02d653a4 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_R_vs_B_paired_runs.csv @@ -0,0 +1,3 @@ +candidate,nvml/peak_memory_used_mib:candidate,nvml/peak_memory_used_mib:reference,nvml/steady_mean_gpu_util_percent:candidate,nvml/steady_mean_gpu_util_percent:reference,perf/step_resp_token_per_s:candidate,perf/step_resp_token_per_s:improvement,perf/step_resp_token_per_s:reference,perf/step_time:candidate,perf/step_time:reduction,perf/step_time:reference,perf/step_token_per_s:candidate,perf/step_token_per_s:improvement,perf/step_token_per_s:reference,perf/wall_clock_samples_per_s:candidate,perf/wall_clock_samples_per_s:improvement,perf/wall_clock_samples_per_s:reference,reference,rollout/image_count/mean:candidate,rollout/image_count/mean:delta,rollout/image_count/mean:reference,rollout/raw_reward:candidate,rollout/raw_reward:delta,rollout/raw_reward:reference,seed,train/grad_norm:candidate,train/grad_norm:delta,train/grad_norm:reference,train/loss:candidate,train/loss:delta,train/loss:reference,train/ppo_kl:candidate,train/ppo_kl:delta,train/ppo_kl:reference +P+R,48393.0,46741.0,83.92935579272213,77.59728409534975,799.7521633699627,0.41034894101985686,567.0597822349148,129.96772181193035,0.24067978916955646,171.16325887044272,1559.7070045237251,0.36325051708314526,1144.108866990144,1.9697198383645176,0.3169674476415332,1.4956480829438525,B,1.0,0.0,1.0,0.41822916666666665,-0.00911458333333337,0.42734375,20260816,0.725117082397143,0.0517819702625274,0.6733351121346156,0.016472606247407384,-0.0018613979414415852,0.01833400418884897,0.0,1.3656563582533274e-13,-1.3656563582533274e-13 +P+R,48335.0,47354.0,85.42939171473519,74.53677597955706,804.4803081955772,0.2334115741806173,652.2399538288842,127.61269302368164,0.1883688675515388,157.22991383870442,1571.6453641874045,0.23276455720934708,1274.894995152354,2.006070038444318,0.2320867941367446,1.62818889707349,B,1.0,0.0,1.0,0.4158854166666667,-0.0003906249999999778,0.41627604166666665,20260817,2.1450912207365036,1.4769096086422602,0.6681816120942433,0.021927844686433672,0.003910016796241204,0.01801782789019247,0.0,3.738612998103903e-12,-3.738612998103903e-12 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_R_vs_P_paired_runs.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_R_vs_P_paired_runs.csv new file mode 100644 index 000000000..62e6331ac --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_R_vs_P_paired_runs.csv @@ -0,0 +1,3 @@ +candidate,nvml/peak_memory_used_mib:candidate,nvml/peak_memory_used_mib:reference,nvml/steady_mean_gpu_util_percent:candidate,nvml/steady_mean_gpu_util_percent:reference,perf/step_resp_token_per_s:candidate,perf/step_resp_token_per_s:improvement,perf/step_resp_token_per_s:reference,perf/step_time:candidate,perf/step_time:reduction,perf/step_time:reference,perf/step_token_per_s:candidate,perf/step_token_per_s:improvement,perf/step_token_per_s:reference,perf/wall_clock_samples_per_s:candidate,perf/wall_clock_samples_per_s:improvement,perf/wall_clock_samples_per_s:reference,reference,rollout/image_count/mean:candidate,rollout/image_count/mean:delta,rollout/image_count/mean:reference,rollout/raw_reward:candidate,rollout/raw_reward:delta,rollout/raw_reward:reference,seed,train/grad_norm:candidate,train/grad_norm:delta,train/grad_norm:reference,train/loss:candidate,train/loss:delta,train/loss:reference,train/ppo_kl:candidate,train/ppo_kl:delta,train/ppo_kl:reference +P+R,48393.0,46725.0,83.92935579272213,78.87460953143773,799.7521633699627,0.29281391681209845,618.6135165856209,129.96772181193035,0.2207435029812811,166.78426462809244,1559.7070045237251,0.28814826316148245,1210.8132651561089,1.9697198383645176,0.28327451080074684,1.534916981352211,P,1.0,0.0,1.0,0.41822916666666665,-0.0013020833333333703,0.41953125,20260816,0.725117082397143,0.057410457730293185,0.6677066246668498,0.016472606247407384,0.006813187155057675,0.009659419092349708,0.0,0.0,0.0 +P+R,48335.0,47779.0,85.42939171473519,79.27988663017983,804.4803081955772,0.3154239474112164,611.574929724225,127.61269302368164,0.2536193243751721,170.9753443400065,1571.6453641874045,0.32721025428832173,1184.1721077043321,2.006070038444318,0.3397988890358883,1.4972919106447946,P,1.0,0.0,1.0,0.4158854166666667,0.007552083333333348,0.4083333333333333,20260817,2.1450912207365036,1.312765329082807,0.8323258916536967,0.021927844686433672,0.00397575895379608,0.017952085732637593,0.0,6.887873663538197e-12,-6.887873663538197e-12 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_S_vs_B_paired_runs.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_S_vs_B_paired_runs.csv new file mode 100644 index 000000000..40a1f75ad --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_S_vs_B_paired_runs.csv @@ -0,0 +1,3 @@ +candidate,nvml/peak_memory_used_mib:candidate,nvml/peak_memory_used_mib:reference,nvml/steady_mean_gpu_util_percent:candidate,nvml/steady_mean_gpu_util_percent:reference,perf/step_resp_token_per_s:candidate,perf/step_resp_token_per_s:improvement,perf/step_resp_token_per_s:reference,perf/step_time:candidate,perf/step_time:reduction,perf/step_time:reference,perf/step_token_per_s:candidate,perf/step_token_per_s:improvement,perf/step_token_per_s:reference,perf/wall_clock_samples_per_s:candidate,perf/wall_clock_samples_per_s:improvement,perf/wall_clock_samples_per_s:reference,reference,rollout/image_count/mean:candidate,rollout/image_count/mean:delta,rollout/image_count/mean:reference,rollout/raw_reward:candidate,rollout/raw_reward:delta,rollout/raw_reward:reference,seed,train/grad_norm:candidate,train/grad_norm:delta,train/grad_norm:reference,train/loss:candidate,train/loss:delta,train/loss:reference,train/ppo_kl:candidate,train/ppo_kl:delta,train/ppo_kl:reference +P+S,48521.0,46741.0,80.10122448979592,77.59728409534975,631.3004921111158,0.11328736737952627,567.0597822349148,163.6089864095052,0.04413489501654977,171.16325887044272,1234.9935087294768,0.0794370574003389,1144.108866990144,1.5647062280506074,0.04617272331257838,1.4956480829438525,B,1.0,0.0,1.0,0.42291666666666666,-0.004427083333333359,0.42734375,20260816,0.6221419324477514,-0.05119317968686421,0.6733351121346156,0.022442292957566678,0.004108288768717709,0.01833400418884897,0.0,1.3656563582533274e-13,-1.3656563582533274e-13 +P+S,48261.0,47354.0,75.54465007045562,74.53677597955706,595.4728904942285,-0.08703401716103476,652.2399538288842,178.93946228027343,-0.13807517864469432,157.22991383870442,1142.585300313896,-0.10378085673059434,1274.894995152354,1.4306514434419524,-0.12132342505626448,1.62818889707349,B,1.0,0.0,1.0,0.41484375,-0.0014322916666666408,0.41627604166666665,20260817,0.6340679069360097,-0.03411370515823364,0.6681816120942433,0.014008935376962958,-0.004008892513229511,0.01801782789019247,2.8543548456540686e-14,3.7671565465604436e-12,-3.738612998103903e-12 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_S_vs_P_paired_runs.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_S_vs_P_paired_runs.csv new file mode 100644 index 000000000..f1f2f8e0a --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_plus_S_vs_P_paired_runs.csv @@ -0,0 +1,3 @@ +candidate,nvml/peak_memory_used_mib:candidate,nvml/peak_memory_used_mib:reference,nvml/steady_mean_gpu_util_percent:candidate,nvml/steady_mean_gpu_util_percent:reference,perf/step_resp_token_per_s:candidate,perf/step_resp_token_per_s:improvement,perf/step_resp_token_per_s:reference,perf/step_time:candidate,perf/step_time:reduction,perf/step_time:reference,perf/step_token_per_s:candidate,perf/step_token_per_s:improvement,perf/step_token_per_s:reference,perf/wall_clock_samples_per_s:candidate,perf/wall_clock_samples_per_s:improvement,perf/wall_clock_samples_per_s:reference,reference,rollout/image_count/mean:candidate,rollout/image_count/mean:delta,rollout/image_count/mean:reference,rollout/raw_reward:candidate,rollout/raw_reward:delta,rollout/raw_reward:reference,seed,train/grad_norm:candidate,train/grad_norm:delta,train/grad_norm:reference,train/loss:candidate,train/loss:delta,train/loss:reference,train/ppo_kl:candidate,train/ppo_kl:delta,train/ppo_kl:reference +P+S,48521.0,46725.0,80.10122448979592,78.87460953143773,631.3004921111158,0.020508726669147936,618.6135165856209,163.6089864095052,0.019038236164950595,166.78426462809244,1234.9935087294768,0.019970249970998166,1210.8132651561089,1.5647062280506074,0.01940772501725352,1.534916981352211,P,1.0,0.0,1.0,0.42291666666666666,0.0033854166666666408,0.41953125,20260816,0.6221419324477514,-0.045564692219098424,0.6677066246668498,0.022442292957566678,0.01278287386521697,0.009659419092349708,0.0,0.0,0.0 +P+S,48261.0,47779.0,75.54465007045562,79.27988663017983,595.4728904942285,-0.026328808535786896,611.574929724225,178.93946228027343,-0.04658050534133884,170.9753443400065,1142.585300313896,-0.035118887803443966,1184.1721077043321,1.4306514434419524,-0.04450733135540963,1.4972919106447946,P,1.0,0.0,1.0,0.41484375,0.006510416666666685,0.4083333333333333,20260817,0.6340679069360097,-0.19825798471768707,0.8323258916536967,0.014008935376962958,-0.003943150355674635,0.017952085732637593,2.8543548456540686e-14,6.916417211994738e-12,-6.887873663538197e-12 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_vs_B_paired_runs.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_vs_B_paired_runs.csv new file mode 100644 index 000000000..144beb591 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/P_vs_B_paired_runs.csv @@ -0,0 +1,3 @@ +candidate,nvml/peak_memory_used_mib:candidate,nvml/peak_memory_used_mib:reference,nvml/steady_mean_gpu_util_percent:candidate,nvml/steady_mean_gpu_util_percent:reference,perf/step_resp_token_per_s:candidate,perf/step_resp_token_per_s:improvement,perf/step_resp_token_per_s:reference,perf/step_time:candidate,perf/step_time:reduction,perf/step_time:reference,perf/step_token_per_s:candidate,perf/step_token_per_s:improvement,perf/step_token_per_s:reference,perf/wall_clock_samples_per_s:candidate,perf/wall_clock_samples_per_s:improvement,perf/wall_clock_samples_per_s:reference,reference,rollout/image_count/mean:candidate,rollout/image_count/mean:delta,rollout/image_count/mean:reference,rollout/raw_reward:candidate,rollout/raw_reward:delta,rollout/raw_reward:reference,seed,train/grad_norm:candidate,train/grad_norm:delta,train/grad_norm:reference,train/loss:candidate,train/loss:delta,train/loss:reference,train/ppo_kl:candidate,train/ppo_kl:delta,train/ppo_kl:reference +P,46725.0,46741.0,78.87460953143773,77.59728409534975,618.6135165856209,0.09091410811664491,567.0597822349148,166.78426462809244,0.025583727905442832,171.16325887044272,1210.8132651561089,0.05830249208840321,1144.108866990144,1.534916981352211,0.02625543993682422,1.4956480829438525,B,1.0,0.0,1.0,0.41953125,-0.0078125,0.42734375,20260816,0.6677066246668498,-0.005628487467765786,0.6733351121346156,0.009659419092349708,-0.00867458509649926,0.01833400418884897,0.0,1.3656563582533274e-13,-1.3656563582533274e-13 +P,47779.0,47354.0,79.27988663017983,74.53677597955706,611.574929724225,-0.062346723573036056,652.2399538288842,170.9753443400065,-0.08742248956138798,157.22991383870442,1184.1721077043321,-0.07116106643526376,1274.894995152354,1.4972919106447946,-0.08039422616378833,1.62818889707349,B,1.0,0.0,1.0,0.4083333333333333,-0.007942708333333326,0.41627604166666665,20260817,0.8323258916536967,0.16414427955945343,0.6681816120942433,0.017952085732637593,-6.574215755487597e-05,0.01801782789019247,-6.887873663538197e-12,-3.1492606654342942e-12,-3.738612998103903e-12 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/steady_campaign_report.json b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/steady_campaign_report.json new file mode 100644 index 000000000..a0e457932 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/steady_campaign_report.json @@ -0,0 +1,1757 @@ +{ + "schema_version": 1, + "benchmark_commit": "e2be8cd158609cc2dfae72b7ba92df72cacb3091", + "campaign_id": "STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267", + "physical_gpu_indices": [ + 0, + 1, + 2, + 6, + 7 + ], + "topology": { + "actor": "TP2 x CP2 x DP1", + "actor_gpu_count": 4, + "rollout_engine_count": 1, + "rollout_gpu_count": 1 + }, + "excluded_attempts": [ + "B1-attempt1", + "B1-attempt2", + "R2-attempt1", + "R2-attempt2", + "SMOKE-B" + ], + "validation": "passed", + "steady_campaign": { + "claim_limit": null, + "comparisons": { + "P+R_vs_B": { + "paired_runs": [ + { + "candidate": "P+R", + "nvml/peak_memory_used_mib:candidate": 48393, + "nvml/peak_memory_used_mib:reference": 46741, + "nvml/steady_mean_gpu_util_percent:candidate": 83.92935579272213, + "nvml/steady_mean_gpu_util_percent:reference": 77.59728409534975, + "perf/step_resp_token_per_s:candidate": 799.7521633699627, + "perf/step_resp_token_per_s:improvement": 0.41034894101985686, + "perf/step_resp_token_per_s:reference": 567.0597822349148, + "perf/step_time:candidate": 129.96772181193035, + "perf/step_time:reduction": 0.24067978916955646, + "perf/step_time:reference": 171.16325887044272, + "perf/step_token_per_s:candidate": 1559.7070045237251, + "perf/step_token_per_s:improvement": 0.36325051708314526, + "perf/step_token_per_s:reference": 1144.108866990144, + "perf/wall_clock_samples_per_s:candidate": 1.9697198383645176, + "perf/wall_clock_samples_per_s:improvement": 0.3169674476415332, + "perf/wall_clock_samples_per_s:reference": 1.4956480829438525, + "reference": "B", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.41822916666666665, + "rollout/raw_reward:delta": -0.00911458333333337, + "rollout/raw_reward:reference": 0.42734375, + "seed": 20260816, + "train/grad_norm:candidate": 0.725117082397143, + "train/grad_norm:delta": 0.0517819702625274, + "train/grad_norm:reference": 0.6733351121346156, + "train/loss:candidate": 0.016472606247407384, + "train/loss:delta": -0.0018613979414415852, + "train/loss:reference": 0.01833400418884897, + "train/ppo_kl:candidate": 0, + "train/ppo_kl:delta": 1.3656563582533274e-13, + "train/ppo_kl:reference": -1.3656563582533274e-13 + }, + { + "candidate": "P+R", + "nvml/peak_memory_used_mib:candidate": 48335, + "nvml/peak_memory_used_mib:reference": 47354, + "nvml/steady_mean_gpu_util_percent:candidate": 85.42939171473519, + "nvml/steady_mean_gpu_util_percent:reference": 74.53677597955706, + "perf/step_resp_token_per_s:candidate": 804.4803081955772, + "perf/step_resp_token_per_s:improvement": 0.2334115741806173, + "perf/step_resp_token_per_s:reference": 652.2399538288842, + "perf/step_time:candidate": 127.61269302368164, + "perf/step_time:reduction": 0.1883688675515388, + "perf/step_time:reference": 157.22991383870442, + "perf/step_token_per_s:candidate": 1571.6453641874045, + "perf/step_token_per_s:improvement": 0.23276455720934708, + "perf/step_token_per_s:reference": 1274.894995152354, + "perf/wall_clock_samples_per_s:candidate": 2.006070038444318, + "perf/wall_clock_samples_per_s:improvement": 0.2320867941367446, + "perf/wall_clock_samples_per_s:reference": 1.62818889707349, + "reference": "B", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.4158854166666667, + "rollout/raw_reward:delta": -0.0003906249999999778, + "rollout/raw_reward:reference": 0.41627604166666665, + "seed": 20260817, + "train/grad_norm:candidate": 2.1450912207365036, + "train/grad_norm:delta": 1.4769096086422602, + "train/grad_norm:reference": 0.6681816120942433, + "train/loss:candidate": 0.021927844686433672, + "train/loss:delta": 0.003910016796241204, + "train/loss:reference": 0.01801782789019247, + "train/ppo_kl:candidate": 0, + "train/ppo_kl:delta": 3.738612998103903e-12, + "train/ppo_kl:reference": -3.738612998103903e-12 + } + ], + "paired_seed_count": 2, + "perf/step_resp_token_per_s:geomean_improvement": 0.31891648995956845, + "perf/step_time:geomean_reduction": 0.21495992292920663, + "perf/step_token_per_s:geomean_improvement": 0.2963668153950165, + "perf/wall_clock_samples_per_s:geomean_improvement": 0.27382031721397326 + }, + "P+R_vs_P": { + "paired_runs": [ + { + "candidate": "P+R", + "nvml/peak_memory_used_mib:candidate": 48393, + "nvml/peak_memory_used_mib:reference": 46725, + "nvml/steady_mean_gpu_util_percent:candidate": 83.92935579272213, + "nvml/steady_mean_gpu_util_percent:reference": 78.87460953143773, + "perf/step_resp_token_per_s:candidate": 799.7521633699627, + "perf/step_resp_token_per_s:improvement": 0.29281391681209845, + "perf/step_resp_token_per_s:reference": 618.6135165856209, + "perf/step_time:candidate": 129.96772181193035, + "perf/step_time:reduction": 0.2207435029812811, + "perf/step_time:reference": 166.78426462809244, + "perf/step_token_per_s:candidate": 1559.7070045237251, + "perf/step_token_per_s:improvement": 0.28814826316148245, + "perf/step_token_per_s:reference": 1210.8132651561089, + "perf/wall_clock_samples_per_s:candidate": 1.9697198383645176, + "perf/wall_clock_samples_per_s:improvement": 0.28327451080074684, + "perf/wall_clock_samples_per_s:reference": 1.534916981352211, + "reference": "P", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.41822916666666665, + "rollout/raw_reward:delta": -0.0013020833333333703, + "rollout/raw_reward:reference": 0.41953125, + "seed": 20260816, + "train/grad_norm:candidate": 0.725117082397143, + "train/grad_norm:delta": 0.057410457730293185, + "train/grad_norm:reference": 0.6677066246668498, + "train/loss:candidate": 0.016472606247407384, + "train/loss:delta": 0.006813187155057675, + "train/loss:reference": 0.009659419092349708, + "train/ppo_kl:candidate": 0, + "train/ppo_kl:delta": 0, + "train/ppo_kl:reference": 0 + }, + { + "candidate": "P+R", + "nvml/peak_memory_used_mib:candidate": 48335, + "nvml/peak_memory_used_mib:reference": 47779, + "nvml/steady_mean_gpu_util_percent:candidate": 85.42939171473519, + "nvml/steady_mean_gpu_util_percent:reference": 79.27988663017983, + "perf/step_resp_token_per_s:candidate": 804.4803081955772, + "perf/step_resp_token_per_s:improvement": 0.3154239474112164, + "perf/step_resp_token_per_s:reference": 611.574929724225, + "perf/step_time:candidate": 127.61269302368164, + "perf/step_time:reduction": 0.2536193243751721, + "perf/step_time:reference": 170.9753443400065, + "perf/step_token_per_s:candidate": 1571.6453641874045, + "perf/step_token_per_s:improvement": 0.32721025428832173, + "perf/step_token_per_s:reference": 1184.1721077043321, + "perf/wall_clock_samples_per_s:candidate": 2.006070038444318, + "perf/wall_clock_samples_per_s:improvement": 0.3397988890358883, + "perf/wall_clock_samples_per_s:reference": 1.4972919106447946, + "reference": "P", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.4158854166666667, + "rollout/raw_reward:delta": 0.007552083333333348, + "rollout/raw_reward:reference": 0.4083333333333333, + "seed": 20260817, + "train/grad_norm:candidate": 2.1450912207365036, + "train/grad_norm:delta": 1.312765329082807, + "train/grad_norm:reference": 0.8323258916536967, + "train/loss:candidate": 0.021927844686433672, + "train/loss:delta": 0.00397575895379608, + "train/loss:reference": 0.017952085732637593, + "train/ppo_kl:candidate": 0, + "train/ppo_kl:delta": 6.887873663538197e-12, + "train/ppo_kl:reference": -6.887873663538197e-12 + } + ], + "paired_seed_count": 2, + "perf/step_resp_token_per_s:geomean_improvement": 0.30406993130013027, + "perf/step_time:geomean_reduction": 0.2373585437901582, + "perf/step_token_per_s:geomean_improvement": 0.3075333968628149, + "perf/wall_clock_samples_per_s:geomean_improvement": 0.3112321548447907 + }, + "P+S_vs_B": { + "paired_runs": [ + { + "candidate": "P+S", + "nvml/peak_memory_used_mib:candidate": 48521, + "nvml/peak_memory_used_mib:reference": 46741, + "nvml/steady_mean_gpu_util_percent:candidate": 80.10122448979592, + "nvml/steady_mean_gpu_util_percent:reference": 77.59728409534975, + "perf/step_resp_token_per_s:candidate": 631.3004921111158, + "perf/step_resp_token_per_s:improvement": 0.11328736737952627, + "perf/step_resp_token_per_s:reference": 567.0597822349148, + "perf/step_time:candidate": 163.6089864095052, + "perf/step_time:reduction": 0.04413489501654977, + "perf/step_time:reference": 171.16325887044272, + "perf/step_token_per_s:candidate": 1234.9935087294768, + "perf/step_token_per_s:improvement": 0.0794370574003389, + "perf/step_token_per_s:reference": 1144.108866990144, + "perf/wall_clock_samples_per_s:candidate": 1.5647062280506074, + "perf/wall_clock_samples_per_s:improvement": 0.04617272331257838, + "perf/wall_clock_samples_per_s:reference": 1.4956480829438525, + "reference": "B", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.42291666666666666, + "rollout/raw_reward:delta": -0.004427083333333359, + "rollout/raw_reward:reference": 0.42734375, + "seed": 20260816, + "train/grad_norm:candidate": 0.6221419324477514, + "train/grad_norm:delta": -0.05119317968686421, + "train/grad_norm:reference": 0.6733351121346156, + "train/loss:candidate": 0.022442292957566678, + "train/loss:delta": 0.004108288768717709, + "train/loss:reference": 0.01833400418884897, + "train/ppo_kl:candidate": 0, + "train/ppo_kl:delta": 1.3656563582533274e-13, + "train/ppo_kl:reference": -1.3656563582533274e-13 + }, + { + "candidate": "P+S", + "nvml/peak_memory_used_mib:candidate": 48261, + "nvml/peak_memory_used_mib:reference": 47354, + "nvml/steady_mean_gpu_util_percent:candidate": 75.54465007045562, + "nvml/steady_mean_gpu_util_percent:reference": 74.53677597955706, + "perf/step_resp_token_per_s:candidate": 595.4728904942285, + "perf/step_resp_token_per_s:improvement": -0.08703401716103476, + "perf/step_resp_token_per_s:reference": 652.2399538288842, + "perf/step_time:candidate": 178.93946228027343, + "perf/step_time:reduction": -0.13807517864469432, + "perf/step_time:reference": 157.22991383870442, + "perf/step_token_per_s:candidate": 1142.585300313896, + "perf/step_token_per_s:improvement": -0.10378085673059434, + "perf/step_token_per_s:reference": 1274.894995152354, + "perf/wall_clock_samples_per_s:candidate": 1.4306514434419524, + "perf/wall_clock_samples_per_s:improvement": -0.12132342505626448, + "perf/wall_clock_samples_per_s:reference": 1.62818889707349, + "reference": "B", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.41484375, + "rollout/raw_reward:delta": -0.0014322916666666408, + "rollout/raw_reward:reference": 0.41627604166666665, + "seed": 20260817, + "train/grad_norm:candidate": 0.6340679069360097, + "train/grad_norm:delta": -0.03411370515823364, + "train/grad_norm:reference": 0.6681816120942433, + "train/loss:candidate": 0.014008935376962958, + "train/loss:delta": -0.004008892513229511, + "train/loss:reference": 0.01801782789019247, + "train/ppo_kl:candidate": 2.8543548456540686e-14, + "train/ppo_kl:delta": 3.7671565465604436e-12, + "train/ppo_kl:reference": -3.738612998103903e-12 + } + ], + "paired_seed_count": 2, + "perf/step_resp_token_per_s:geomean_improvement": 0.008163427000728474, + "perf/step_time:geomean_reduction": -0.042998729680084224, + "perf/step_token_per_s:geomean_improvement": -0.01642887659479353, + "perf/wall_clock_samples_per_s:geomean_improvement": -0.041226061409752046 + }, + "P+S_vs_P": { + "paired_runs": [ + { + "candidate": "P+S", + "nvml/peak_memory_used_mib:candidate": 48521, + "nvml/peak_memory_used_mib:reference": 46725, + "nvml/steady_mean_gpu_util_percent:candidate": 80.10122448979592, + "nvml/steady_mean_gpu_util_percent:reference": 78.87460953143773, + "perf/step_resp_token_per_s:candidate": 631.3004921111158, + "perf/step_resp_token_per_s:improvement": 0.020508726669147936, + "perf/step_resp_token_per_s:reference": 618.6135165856209, + "perf/step_time:candidate": 163.6089864095052, + "perf/step_time:reduction": 0.019038236164950595, + "perf/step_time:reference": 166.78426462809244, + "perf/step_token_per_s:candidate": 1234.9935087294768, + "perf/step_token_per_s:improvement": 0.019970249970998166, + "perf/step_token_per_s:reference": 1210.8132651561089, + "perf/wall_clock_samples_per_s:candidate": 1.5647062280506074, + "perf/wall_clock_samples_per_s:improvement": 0.01940772501725352, + "perf/wall_clock_samples_per_s:reference": 1.534916981352211, + "reference": "P", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.42291666666666666, + "rollout/raw_reward:delta": 0.0033854166666666408, + "rollout/raw_reward:reference": 0.41953125, + "seed": 20260816, + "train/grad_norm:candidate": 0.6221419324477514, + "train/grad_norm:delta": -0.045564692219098424, + "train/grad_norm:reference": 0.6677066246668498, + "train/loss:candidate": 0.022442292957566678, + "train/loss:delta": 0.01278287386521697, + "train/loss:reference": 0.009659419092349708, + "train/ppo_kl:candidate": 0, + "train/ppo_kl:delta": 0, + "train/ppo_kl:reference": 0 + }, + { + "candidate": "P+S", + "nvml/peak_memory_used_mib:candidate": 48261, + "nvml/peak_memory_used_mib:reference": 47779, + "nvml/steady_mean_gpu_util_percent:candidate": 75.54465007045562, + "nvml/steady_mean_gpu_util_percent:reference": 79.27988663017983, + "perf/step_resp_token_per_s:candidate": 595.4728904942285, + "perf/step_resp_token_per_s:improvement": -0.026328808535786896, + "perf/step_resp_token_per_s:reference": 611.574929724225, + "perf/step_time:candidate": 178.93946228027343, + "perf/step_time:reduction": -0.04658050534133884, + "perf/step_time:reference": 170.9753443400065, + "perf/step_token_per_s:candidate": 1142.585300313896, + "perf/step_token_per_s:improvement": -0.035118887803443966, + "perf/step_token_per_s:reference": 1184.1721077043321, + "perf/wall_clock_samples_per_s:candidate": 1.4306514434419524, + "perf/wall_clock_samples_per_s:improvement": -0.04450733135540963, + "perf/wall_clock_samples_per_s:reference": 1.4972919106447946, + "reference": "P", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.41484375, + "rollout/raw_reward:delta": 0.006510416666666685, + "rollout/raw_reward:reference": 0.4083333333333333, + "seed": 20260817, + "train/grad_norm:candidate": 0.6340679069360097, + "train/grad_norm:delta": -0.19825798471768707, + "train/grad_norm:reference": 0.8323258916536967, + "train/loss:candidate": 0.014008935376962958, + "train/loss:delta": -0.003943150355674635, + "train/loss:reference": 0.017952085732637593, + "train/ppo_kl:candidate": 2.8543548456540686e-14, + "train/ppo_kl:delta": 6.916417211994738e-12, + "train/ppo_kl:reference": -6.887873663538197e-12 + } + ], + "paired_seed_count": 2, + "perf/step_resp_token_per_s:geomean_improvement": -0.003185098528530106, + "perf/step_time:geomean_reduction": -0.013240079406167782, + "perf/step_token_per_s:geomean_improvement": -0.00795663945600844, + "perf/wall_clock_samples_per_s:geomean_improvement": -0.013067070357034694 + }, + "P_vs_B": { + "paired_runs": [ + { + "candidate": "P", + "nvml/peak_memory_used_mib:candidate": 46725, + "nvml/peak_memory_used_mib:reference": 46741, + "nvml/steady_mean_gpu_util_percent:candidate": 78.87460953143773, + "nvml/steady_mean_gpu_util_percent:reference": 77.59728409534975, + "perf/step_resp_token_per_s:candidate": 618.6135165856209, + "perf/step_resp_token_per_s:improvement": 0.09091410811664491, + "perf/step_resp_token_per_s:reference": 567.0597822349148, + "perf/step_time:candidate": 166.78426462809244, + "perf/step_time:reduction": 0.025583727905442832, + "perf/step_time:reference": 171.16325887044272, + "perf/step_token_per_s:candidate": 1210.8132651561089, + "perf/step_token_per_s:improvement": 0.05830249208840321, + "perf/step_token_per_s:reference": 1144.108866990144, + "perf/wall_clock_samples_per_s:candidate": 1.534916981352211, + "perf/wall_clock_samples_per_s:improvement": 0.02625543993682422, + "perf/wall_clock_samples_per_s:reference": 1.4956480829438525, + "reference": "B", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.41953125, + "rollout/raw_reward:delta": -0.0078125, + "rollout/raw_reward:reference": 0.42734375, + "seed": 20260816, + "train/grad_norm:candidate": 0.6677066246668498, + "train/grad_norm:delta": -0.005628487467765786, + "train/grad_norm:reference": 0.6733351121346156, + "train/loss:candidate": 0.009659419092349708, + "train/loss:delta": -0.00867458509649926, + "train/loss:reference": 0.01833400418884897, + "train/ppo_kl:candidate": 0, + "train/ppo_kl:delta": 1.3656563582533274e-13, + "train/ppo_kl:reference": -1.3656563582533274e-13 + }, + { + "candidate": "P", + "nvml/peak_memory_used_mib:candidate": 47779, + "nvml/peak_memory_used_mib:reference": 47354, + "nvml/steady_mean_gpu_util_percent:candidate": 79.27988663017983, + "nvml/steady_mean_gpu_util_percent:reference": 74.53677597955706, + "perf/step_resp_token_per_s:candidate": 611.574929724225, + "perf/step_resp_token_per_s:improvement": -0.062346723573036056, + "perf/step_resp_token_per_s:reference": 652.2399538288842, + "perf/step_time:candidate": 170.9753443400065, + "perf/step_time:reduction": -0.08742248956138798, + "perf/step_time:reference": 157.22991383870442, + "perf/step_token_per_s:candidate": 1184.1721077043321, + "perf/step_token_per_s:improvement": -0.07116106643526376, + "perf/step_token_per_s:reference": 1274.894995152354, + "perf/wall_clock_samples_per_s:candidate": 1.4972919106447946, + "perf/wall_clock_samples_per_s:improvement": -0.08039422616378833, + "perf/wall_clock_samples_per_s:reference": 1.62818889707349, + "reference": "B", + "rollout/image_count/mean:candidate": 1, + "rollout/image_count/mean:delta": 0, + "rollout/image_count/mean:reference": 1, + "rollout/raw_reward:candidate": 0.4083333333333333, + "rollout/raw_reward:delta": -0.007942708333333326, + "rollout/raw_reward:reference": 0.41627604166666665, + "seed": 20260817, + "train/grad_norm:candidate": 0.8323258916536967, + "train/grad_norm:delta": 0.16414427955945343, + "train/grad_norm:reference": 0.6681816120942433, + "train/loss:candidate": 0.017952085732637593, + "train/loss:delta": -6.574215755487597e-05, + "train/loss:reference": 0.01801782789019247, + "train/ppo_kl:candidate": -6.887873663538197e-12, + "train/ppo_kl:delta": -3.1492606654342942e-12, + "train/ppo_kl:reference": -3.738612998103903e-12 + } + ], + "paired_seed_count": 2, + "perf/step_resp_token_per_s:geomean_improvement": 0.011384787198211344, + "perf/step_time:geomean_reduction": -0.029369791897056796, + "perf/step_token_per_s:geomean_improvement": -0.008540188338279076, + "perf/wall_clock_samples_per_s:geomean_improvement": -0.028531818330252756 + } + }, + "conditions": { + "B": { + "perf/step_token_per_s": { + "coefficient_of_variation": 0.054066109694580484, + "count": 2, + "max": 1274.894995152354, + "mean": 1209.5019310712491, + "median": 1209.5019310712491, + "min": 1144.108866990144, + "population_stddev": 65.39306408110508 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.04242885111402373, + "count": 2, + "max": 1.62818889707349, + "mean": 1.5619184900086713, + "median": 1.5619184900086713, + "min": 1.4956480829438525, + "population_stddev": 0.06627040706481868 + }, + "runs": [ + { + "measured_optimizer_step_count": 30, + "nvml": { + "gpu_count": 5, + "peak_memory_used_mib": 46741, + "steady_mean_gpu_util_percent": 77.59728409534975, + "steady_idle_ratio_below_10_percent": 0.1677413051973427, + "per_gpu": { + "0": { + "steady_sample_count": 10236, + "steady_mean_gpu_util_percent": 84.3358733880422, + "steady_idle_ratio_below_10_percent": 0.08987885892926925, + "peak_memory_used_mib": 42574 + }, + "1": { + "steady_sample_count": 10236, + "steady_mean_gpu_util_percent": 84.92008597108246, + "steady_idle_ratio_below_10_percent": 0.08137944509574052, + "peak_memory_used_mib": 42332 + }, + "2": { + "steady_sample_count": 10236, + "steady_mean_gpu_util_percent": 84.13862837045721, + "steady_idle_ratio_below_10_percent": 0.1011137162954279, + "peak_memory_used_mib": 42395 + }, + "6": { + "steady_sample_count": 10236, + "steady_mean_gpu_util_percent": 82.82874169597498, + "steady_idle_ratio_below_10_percent": 0.10922235248143806, + "peak_memory_used_mib": 42047 + }, + "7": { + "steady_sample_count": 10236, + "steady_mean_gpu_util_percent": 51.76309105119187, + "steady_idle_ratio_below_10_percent": 0.45711215318483783, + "peak_memory_used_mib": 46741 + } + } + }, + "optimizer_update_count": 40, + "perf/step_time": { + "count": 30, + "max": 229.48023986816406, + "mean": 171.16325887044272, + "min": 143.80343627929688, + "p50": 166.24190521240234, + "p95": 212.10879516601562 + }, + "perf/step_token_per_s": 1144.108866990144, + "perf/wall_clock_samples_per_s": 1.4956480829438525, + "rollout/image_count/mean": { + "count": 30, + "max": 1, + "mean": 1, + "min": 1, + "p50": 1, + "p95": 1 + }, + "rollout/raw_reward": { + "count": 30, + "max": 0.62109375, + "mean": 0.42734375, + "min": 0.234375, + "p50": 0.41796875, + "p95": 0.6015625 + }, + "seed": 20260816, + "steady_actor_fetch_samples": 7680, + "steady_actor_multimodal_tensor_bytes": 47368359936, + "steady_actor_response_tokens": 2911794, + "steady_actor_total_tokens": 5874882, + "train/grad_norm": { + "count": 30, + "max": 2.207888603210449, + "mean": 0.6733351121346156, + "min": 0.2713528871536255, + "p50": 0.5761384069919586, + "p95": 1.401735782623291 + }, + "train/loss": { + "count": 30, + "max": 0.07299309968948364, + "mean": 0.01833400418884897, + "min": -0.013865238055586815, + "p50": 0.015135759487748146, + "p95": 0.060980722308158875 + }, + "train/ppo_kl": { + "count": 30, + "max": 2.681594312922697e-12, + "mean": -1.3656563582533274e-13, + "min": -6.778563387682679e-12, + "p50": 0, + "p95": 0 + }, + "windows": [ + { + "end": 19, + "perf/step_token_per_s": 1132.0432167205465, + "perf/wall_clock_samples_per_s": 1.4385605737458247, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": 1142.4302550609464, + "perf/wall_clock_samples_per_s": 1.6307091678869532, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": 1157.6104558930774, + "perf/wall_clock_samples_per_s": 1.4337945371447864, + "start": 30 + } + ] + }, + { + "measured_optimizer_step_count": 30, + "nvml": { + "gpu_count": 5, + "peak_memory_used_mib": 47354, + "steady_mean_gpu_util_percent": 74.53677597955706, + "steady_idle_ratio_below_10_percent": 0.1799403747870528, + "per_gpu": { + "0": { + "steady_sample_count": 9392, + "steady_mean_gpu_util_percent": 58.17717206132879, + "steady_idle_ratio_below_10_percent": 0.40129897785349233, + "peak_memory_used_mib": 47354 + }, + "1": { + "steady_sample_count": 9392, + "steady_mean_gpu_util_percent": 79.04631601362863, + "steady_idle_ratio_below_10_percent": 0.11829216354344123, + "peak_memory_used_mib": 42310 + }, + "2": { + "steady_sample_count": 9392, + "steady_mean_gpu_util_percent": 82.07815161839864, + "steady_idle_ratio_below_10_percent": 0.092206132879046, + "peak_memory_used_mib": 42067 + }, + "6": { + "steady_sample_count": 9392, + "steady_mean_gpu_util_percent": 76.32218909710392, + "steady_idle_ratio_below_10_percent": 0.14948892674616696, + "peak_memory_used_mib": 42381 + }, + "7": { + "steady_sample_count": 9392, + "steady_mean_gpu_util_percent": 77.06005110732538, + "steady_idle_ratio_below_10_percent": 0.13841567291311754, + "peak_memory_used_mib": 42361 + } + } + }, + "optimizer_update_count": 40, + "perf/step_time": { + "count": 30, + "max": 193.93869018554688, + "mean": 157.22991383870442, + "min": 125.9332275390625, + "p50": 155.28236389160156, + "p95": 190.11888122558594 + }, + "perf/step_token_per_s": 1274.894995152354, + "perf/wall_clock_samples_per_s": 1.62818889707349, + "rollout/image_count/mean": { + "count": 30, + "max": 1, + "mean": 1, + "min": 1, + "p50": 1, + "p95": 1 + }, + "rollout/raw_reward": { + "count": 30, + "max": 0.53125, + "mean": 0.41627604166666665, + "min": 0.28515625, + "p50": 0.41015625, + "p95": 0.5078125 + }, + "seed": 20260817, + "steady_actor_fetch_samples": 7680, + "steady_actor_multimodal_tensor_bytes": 46333218816, + "steady_actor_response_tokens": 3076549, + "steady_actor_total_tokens": 6013549, + "train/grad_norm": { + "count": 30, + "max": 1.791407585144043, + "mean": 0.6681816120942433, + "min": 0.35177525877952576, + "p50": 0.5901691615581512, + "p95": 1.270756483078003 + }, + "train/loss": { + "count": 30, + "max": 0.08514145761728287, + "mean": 0.01801782789019247, + "min": -0.05296846851706505, + "p50": 0.01855332124978304, + "p95": 0.05735928565263748 + }, + "train/ppo_kl": { + "count": 30, + "max": 1.94881905751787e-11, + "mean": -3.738612998103903e-12, + "min": -1.8820169381261742e-11, + "p50": -6.535148724257089e-12, + "p95": 1.487116159382129e-11 + }, + "windows": [ + { + "end": 19, + "perf/step_token_per_s": 1310.873688323878, + "perf/wall_clock_samples_per_s": 1.675954122023373, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": 1235.1915050047178, + "perf/wall_clock_samples_per_s": 1.5754389811660137, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": 1281.0050178190913, + "perf/wall_clock_samples_per_s": 1.636341740964244, + "start": 30 + } + ] + } + ], + "windows": [ + { + "end": 19, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.07320366535352109, + "count": 2, + "max": 1310.873688323878, + "mean": 1221.4584525222122, + "median": 1221.4584525222122, + "min": 1132.0432167205465, + "population_stddev": 89.41523580166574 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.07622168185625423, + "count": 2, + "max": 1.675954122023373, + "mean": 1.5572573478845988, + "median": 1.5572573478845988, + "min": 1.4385605737458247, + "population_stddev": 0.11869677413877411 + }, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.039014300550988215, + "count": 2, + "max": 1235.1915050047178, + "mean": 1188.810880032832, + "median": 1188.810880032832, + "min": 1142.4302550609464, + "population_stddev": 46.38062497188571 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.017238812478851062, + "count": 2, + "max": 1.6307091678869532, + "mean": 1.6030740745264835, + "median": 1.6030740745264835, + "min": 1.5754389811660137, + "population_stddev": 0.027635093360469765 + }, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.050600253814586535, + "count": 2, + "max": 1281.0050178190913, + "mean": 1219.3077368560844, + "median": 1219.3077368560844, + "min": 1157.6104558930774, + "population_stddev": 61.69728096300696 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.06597335931426834, + "count": 2, + "max": 1.636341740964244, + "mean": 1.5350681390545153, + "median": 1.5350681390545153, + "min": 1.4337945371447864, + "population_stddev": 0.10127360190972878 + }, + "start": 30 + } + ] + }, + "P": { + "perf/step_token_per_s": { + "coefficient_of_variation": 0.011123724492712862, + "count": 2, + "max": 1210.8132651561089, + "mean": 1197.4926864302206, + "median": 1197.4926864302206, + "min": 1184.1721077043321, + "population_stddev": 13.320578725888367 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.012408469220811697, + "count": 2, + "max": 1.534916981352211, + "mean": 1.516104445998503, + "median": 1.516104445998503, + "min": 1.4972919106447946, + "population_stddev": 0.018812535353708193 + }, + "runs": [ + { + "measured_optimizer_step_count": 30, + "nvml": { + "gpu_count": 5, + "peak_memory_used_mib": 46725, + "steady_mean_gpu_util_percent": 78.87460953143773, + "steady_idle_ratio_below_10_percent": 0.15440528634361234, + "per_gpu": { + "0": { + "steady_sample_count": 9988, + "steady_mean_gpu_util_percent": 84.81067280736885, + "steady_idle_ratio_below_10_percent": 0.08089707649179015, + "peak_memory_used_mib": 42460 + }, + "1": { + "steady_sample_count": 9988, + "steady_mean_gpu_util_percent": 86.51732078494193, + "steady_idle_ratio_below_10_percent": 0.06978374048858631, + "peak_memory_used_mib": 42068 + }, + "2": { + "steady_sample_count": 9988, + "steady_mean_gpu_util_percent": 85.19012815378454, + "steady_idle_ratio_below_10_percent": 0.09261113336003204, + "peak_memory_used_mib": 42435 + }, + "6": { + "steady_sample_count": 9988, + "steady_mean_gpu_util_percent": 83.46846215458551, + "steady_idle_ratio_below_10_percent": 0.09921906287545054, + "peak_memory_used_mib": 42289 + }, + "7": { + "steady_sample_count": 9988, + "steady_mean_gpu_util_percent": 54.38646375650781, + "steady_idle_ratio_below_10_percent": 0.42951541850220265, + "peak_memory_used_mib": 46725 + } + } + }, + "optimizer_update_count": 40, + "perf/step_time": { + "count": 30, + "max": 225.04490661621094, + "mean": 166.78426462809244, + "min": 141.3410186767578, + "p50": 160.32127380371094, + "p95": 209.2008514404297 + }, + "perf/step_token_per_s": 1210.8132651561089, + "perf/wall_clock_samples_per_s": 1.534916981352211, + "rollout/image_count/mean": { + "count": 30, + "max": 1, + "mean": 1, + "min": 1, + "p50": 1, + "p95": 1 + }, + "rollout/raw_reward": { + "count": 30, + "max": 0.609375, + "mean": 0.41953125, + "min": 0.234375, + "p50": 0.41796875, + "p95": 0.6015625 + }, + "seed": 20260816, + "steady_actor_fetch_samples": 7680, + "steady_actor_multimodal_tensor_bytes": 47368359936, + "steady_actor_response_tokens": 3095250, + "steady_actor_total_tokens": 6058338, + "train/grad_norm": { + "count": 30, + "max": 1.6172339916229248, + "mean": 0.6677066246668498, + "min": 0.31992343068122864, + "p50": 0.5627448260784149, + "p95": 1.561697006225586 + }, + "train/loss": { + "count": 30, + "max": 0.04433699697256088, + "mean": 0.009659419092349708, + "min": -0.034701038151979446, + "p50": 0.010162549559026957, + "p95": 0.033727746456861496 + }, + "train/ppo_kl": { + "count": 30, + "max": 0, + "mean": 0, + "min": 0, + "p50": 0, + "p95": 0 + }, + "windows": [ + { + "end": 19, + "perf/step_token_per_s": 1224.642519017699, + "perf/wall_clock_samples_per_s": 1.5145088841407948, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": 1216.1593197420632, + "perf/wall_clock_samples_per_s": 1.6732257428225803, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": 1193.1177721173674, + "perf/wall_clock_samples_per_s": 1.4355954619201463, + "start": 30 + } + ] + }, + { + "measured_optimizer_step_count": 30, + "nvml": { + "gpu_count": 5, + "peak_memory_used_mib": 47779, + "steady_mean_gpu_util_percent": 79.27988663017983, + "steady_idle_ratio_below_10_percent": 0.15093823299452697, + "per_gpu": { + "0": { + "steady_sample_count": 10232, + "steady_mean_gpu_util_percent": 85.57183346364347, + "steady_idle_ratio_below_10_percent": 0.07984753713838937, + "peak_memory_used_mib": 41964 + }, + "1": { + "steady_sample_count": 10232, + "steady_mean_gpu_util_percent": 87.18852619233776, + "steady_idle_ratio_below_10_percent": 0.06665363565285379, + "peak_memory_used_mib": 41770 + }, + "2": { + "steady_sample_count": 10232, + "steady_mean_gpu_util_percent": 85.49824081313527, + "steady_idle_ratio_below_10_percent": 0.08668881939014855, + "peak_memory_used_mib": 42499 + }, + "6": { + "steady_sample_count": 10232, + "steady_mean_gpu_util_percent": 84.11884284597342, + "steady_idle_ratio_below_10_percent": 0.0985144644253323, + "peak_memory_used_mib": 42109 + }, + "7": { + "steady_sample_count": 10232, + "steady_mean_gpu_util_percent": 54.02198983580923, + "steady_idle_ratio_below_10_percent": 0.42298670836591085, + "peak_memory_used_mib": 47779 + } + } + }, + "optimizer_update_count": 40, + "perf/step_time": { + "count": 30, + "max": 198.459228515625, + "mean": 170.9753443400065, + "min": 129.9396514892578, + "p50": 171.22992706298828, + "p95": 196.6987762451172 + }, + "perf/step_token_per_s": 1184.1721077043321, + "perf/wall_clock_samples_per_s": 1.4972919106447946, + "rollout/image_count/mean": { + "count": 30, + "max": 1, + "mean": 1, + "min": 1, + "p50": 1, + "p95": 1 + }, + "rollout/raw_reward": { + "count": 30, + "max": 0.51953125, + "mean": 0.4083333333333333, + "min": 0.2890625, + "p50": 0.404296875, + "p95": 0.51171875 + }, + "seed": 20260817, + "steady_actor_fetch_samples": 7680, + "steady_actor_multimodal_tensor_bytes": 46333218816, + "steady_actor_response_tokens": 3136927, + "steady_actor_total_tokens": 6073927, + "train/grad_norm": { + "count": 30, + "max": 3.2474284172058105, + "mean": 0.8323258916536967, + "min": 0.35890820622444153, + "p50": 0.5539738833904266, + "p95": 2.7423908710479736 + }, + "train/loss": { + "count": 30, + "max": 0.0633721798658371, + "mean": 0.017952085732637593, + "min": -0.037975482642650604, + "p50": 0.018570133484899998, + "p95": 0.06103434041142464 + }, + "train/ppo_kl": { + "count": 30, + "max": 1.2377899052951058e-11, + "mean": -6.887873663538197e-12, + "min": -2.404459444049767e-11, + "p50": -8.556255530478563e-12, + "p95": 7.826240523700623e-12 + }, + "windows": [ + { + "end": 19, + "perf/step_token_per_s": 1181.059742148176, + "perf/wall_clock_samples_per_s": 1.4990705828435051, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": 1193.0424243624066, + "perf/wall_clock_samples_per_s": 1.5078004775254163, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": 1178.5183966942907, + "perf/wall_clock_samples_per_s": 1.4851788153251944, + "start": 30 + } + ] + } + ], + "windows": [ + { + "end": 19, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.018116446732856104, + "count": 2, + "max": 1224.642519017699, + "mean": 1202.8511305829375, + "median": 1202.8511305829375, + "min": 1181.059742148176, + "population_stddev": 21.79138843476153 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.0051229116293186015, + "count": 2, + "max": 1.5145088841407948, + "mean": 1.50678973349215, + "median": 1.50678973349215, + "min": 1.4990705828435051, + "population_stddev": 0.0077191506486448125 + }, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.009595250973159768, + "count": 2, + "max": 1216.1593197420632, + "mean": 1204.600872052235, + "median": 1204.600872052235, + "min": 1193.0424243624066, + "population_stddev": 11.558447689828313 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.05200374150926263, + "count": 2, + "max": 1.6732257428225803, + "mean": 1.5905131101739984, + "median": 1.5905131101739984, + "min": 1.5078004775254163, + "population_stddev": 0.08271263264858197 + }, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.0061558242428019355, + "count": 2, + "max": 1193.1177721173674, + "mean": 1185.818084405829, + "median": 1185.818084405829, + "min": 1178.5183966942907, + "population_stddev": 7.299687711538354 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.016976099040358374, + "count": 2, + "max": 1.4851788153251944, + "mean": 1.4603871386226703, + "median": 1.4603871386226703, + "min": 1.4355954619201463, + "population_stddev": 0.024791676702524024 + }, + "start": 30 + } + ] + }, + "P+R": { + "perf/step_token_per_s": { + "coefficient_of_variation": 0.0038125251514230804, + "count": 2, + "max": 1571.6453641874045, + "mean": 1565.6761843555648, + "median": 1565.6761843555648, + "min": 1559.7070045237251, + "population_stddev": 5.9691798318397105 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.009142887628904842, + "count": 2, + "max": 2.006070038444318, + "mean": 1.987894938404418, + "median": 1.987894938404418, + "min": 1.9697198383645176, + "population_stddev": 0.018175100039900305 + }, + "runs": [ + { + "measured_optimizer_step_count": 30, + "nvml": { + "gpu_count": 5, + "peak_memory_used_mib": 48393, + "steady_mean_gpu_util_percent": 83.92935579272213, + "steady_idle_ratio_below_10_percent": 0.1046933264755047, + "per_gpu": { + "0": { + "steady_sample_count": 7777, + "steady_mean_gpu_util_percent": 87.26038318117526, + "steady_idle_ratio_below_10_percent": 0.05644850199305645, + "peak_memory_used_mib": 47586 + }, + "1": { + "steady_sample_count": 7777, + "steady_mean_gpu_util_percent": 89.97492606403497, + "steady_idle_ratio_below_10_percent": 0.03896103896103896, + "peak_memory_used_mib": 47386 + }, + "2": { + "steady_sample_count": 7777, + "steady_mean_gpu_util_percent": 87.08075093223609, + "steady_idle_ratio_below_10_percent": 0.07175003214607174, + "peak_memory_used_mib": 47189 + }, + "6": { + "steady_sample_count": 7777, + "steady_mean_gpu_util_percent": 85.92002057348591, + "steady_idle_ratio_below_10_percent": 0.07805066220907805, + "peak_memory_used_mib": 46979 + }, + "7": { + "steady_sample_count": 7777, + "steady_mean_gpu_util_percent": 69.41069821267841, + "steady_idle_ratio_below_10_percent": 0.27825639706827826, + "peak_memory_used_mib": 48393 + } + } + }, + "optimizer_update_count": 40, + "perf/step_time": { + "count": 30, + "max": 175.11614990234375, + "mean": 129.96772181193035, + "min": 110.51399993896484, + "p50": 128.03932571411133, + "p95": 152.3202667236328 + }, + "perf/step_token_per_s": 1559.7070045237251, + "perf/wall_clock_samples_per_s": 1.9697198383645176, + "rollout/image_count/mean": { + "count": 30, + "max": 1, + "mean": 1, + "min": 1, + "p50": 1, + "p95": 1 + }, + "rollout/raw_reward": { + "count": 30, + "max": 0.59375, + "mean": 0.41822916666666665, + "min": 0.234375, + "p50": 0.42578125, + "p95": 0.58203125 + }, + "seed": 20260816, + "steady_actor_fetch_samples": 7680, + "steady_actor_multimodal_tensor_bytes": 47368359936, + "steady_actor_response_tokens": 3118259, + "steady_actor_total_tokens": 6081347, + "train/grad_norm": { + "count": 30, + "max": 2.4391937255859375, + "mean": 0.725117082397143, + "min": 0.3615645170211792, + "p50": 0.5901145040988922, + "p95": 1.7316131591796875 + }, + "train/loss": { + "count": 30, + "max": 0.04787826165556908, + "mean": 0.016472606247407384, + "min": -0.016523413360118866, + "p50": 0.017509877681732178, + "p95": 0.042700476944446564 + }, + "train/ppo_kl": { + "count": 30, + "max": 0, + "mean": 0, + "min": 0, + "p50": 0, + "p95": 0 + }, + "windows": [ + { + "end": 19, + "perf/step_token_per_s": 1558.661227134313, + "perf/wall_clock_samples_per_s": 1.9331173847287595, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": 1551.7418301677114, + "perf/wall_clock_samples_per_s": 2.127978454486029, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": 1567.7021646499002, + "perf/wall_clock_samples_per_s": 1.8662617786165707, + "start": 30 + } + ] + }, + { + "measured_optimizer_step_count": 30, + "nvml": { + "gpu_count": 5, + "peak_memory_used_mib": 48335, + "steady_mean_gpu_util_percent": 85.42939171473519, + "steady_idle_ratio_below_10_percent": 0.09399580492920818, + "per_gpu": { + "0": { + "steady_sample_count": 7628, + "steady_mean_gpu_util_percent": 88.06620346093341, + "steady_idle_ratio_below_10_percent": 0.05256948085998951, + "peak_memory_used_mib": 47056 + }, + "1": { + "steady_sample_count": 7628, + "steady_mean_gpu_util_percent": 90.54417933927635, + "steady_idle_ratio_below_10_percent": 0.033560566334556896, + "peak_memory_used_mib": 46670 + }, + "2": { + "steady_sample_count": 7628, + "steady_mean_gpu_util_percent": 89.24777136864185, + "steady_idle_ratio_below_10_percent": 0.05060304142632407, + "peak_memory_used_mib": 47185 + }, + "6": { + "steady_sample_count": 7628, + "steady_mean_gpu_util_percent": 87.66203460933403, + "steady_idle_ratio_below_10_percent": 0.06856318825380178, + "peak_memory_used_mib": 47089 + }, + "7": { + "steady_sample_count": 7628, + "steady_mean_gpu_util_percent": 71.6267697954903, + "steady_idle_ratio_below_10_percent": 0.26468274777136863, + "peak_memory_used_mib": 48335 + } + } + }, + "optimizer_update_count": 40, + "perf/step_time": { + "count": 30, + "max": 153.7805938720703, + "mean": 127.61269302368164, + "min": 100.02981567382812, + "p50": 128.27655029296875, + "p95": 150.72384643554688 + }, + "perf/step_token_per_s": 1571.6453641874045, + "perf/wall_clock_samples_per_s": 2.006070038444318, + "rollout/image_count/mean": { + "count": 30, + "max": 1, + "mean": 1, + "min": 1, + "p50": 1, + "p95": 1 + }, + "rollout/raw_reward": { + "count": 30, + "max": 0.515625, + "mean": 0.4158854166666667, + "min": 0.28125, + "p50": 0.41015625, + "p95": 0.51171875 + }, + "seed": 20260817, + "steady_actor_fetch_samples": 7680, + "steady_actor_multimodal_tensor_bytes": 46333218816, + "steady_actor_response_tokens": 3079857, + "steady_actor_total_tokens": 6016857, + "train/grad_norm": { + "count": 30, + "max": 43.00599670410156, + "mean": 2.1450912207365036, + "min": 0.3485816717147827, + "p50": 0.6160759031772614, + "p95": 2.993971347808838 + }, + "train/loss": { + "count": 30, + "max": 0.07517870515584946, + "mean": 0.021927844686433672, + "min": -0.022954167798161507, + "p50": 0.022239123471081257, + "p95": 0.05392253398895264 + }, + "train/ppo_kl": { + "count": 30, + "max": 0, + "mean": 0, + "min": 0, + "p50": 0, + "p95": 0 + }, + "windows": [ + { + "end": 19, + "perf/step_token_per_s": 1553.338985344529, + "perf/wall_clock_samples_per_s": 2.001135218408179, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": 1579.3025587046557, + "perf/wall_clock_samples_per_s": 2.031472697455935, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": 1582.3282019020912, + "perf/wall_clock_samples_per_s": 1.9861321648992198, + "start": 30 + } + ] + } + ], + "windows": [ + { + "end": 19, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.0017102318208213538, + "count": 2, + "max": 1558.661227134313, + "mean": 1556.000106239421, + "median": 1556.000106239421, + "min": 1553.338985344529, + "population_stddev": 2.661120894892065 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.017288629008006845, + "count": 2, + "max": 2.001135218408179, + "mean": 1.9671263015684692, + "median": 1.9671263015684692, + "min": 1.9331173847287595, + "population_stddev": 0.03400891683970986 + }, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.0088024074762064, + "count": 2, + "max": 1579.3025587046557, + "mean": 1565.5221944361836, + "median": 1565.5221944361836, + "min": 1551.7418301677114, + "population_stddev": 13.780364268472113 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.023201560375348367, + "count": 2, + "max": 2.127978454486029, + "mean": 2.079725575970982, + "median": 2.079725575970982, + "min": 2.031472697455935, + "population_stddev": 0.04825287851504689 + }, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.004643141668567673, + "count": 2, + "max": 1582.3282019020912, + "mean": 1575.0151832759957, + "median": 1575.0151832759957, + "min": 1567.7021646499002, + "population_stddev": 7.313018626095527 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.03111581734376117, + "count": 2, + "max": 1.9861321648992198, + "mean": 1.9261969717578953, + "median": 1.9261969717578953, + "min": 1.8662617786165707, + "population_stddev": 0.05993519314132456 + }, + "start": 30 + } + ] + }, + "P+S": { + "perf/step_token_per_s": { + "coefficient_of_variation": 0.03886651751104802, + "count": 2, + "max": 1234.9935087294768, + "mean": 1188.7894045216863, + "median": 1188.7894045216863, + "min": 1142.585300313896, + "population_stddev": 46.204104207790465 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.044754182742342344, + "count": 2, + "max": 1.5647062280506074, + "mean": 1.4976788357462798, + "median": 1.4976788357462798, + "min": 1.4306514434419524, + "population_stddev": 0.06702739230432753 + }, + "runs": [ + { + "measured_optimizer_step_count": 30, + "nvml": { + "gpu_count": 5, + "peak_memory_used_mib": 48521, + "steady_mean_gpu_util_percent": 80.10122448979592, + "steady_idle_ratio_below_10_percent": 0.14806122448979592, + "per_gpu": { + "0": { + "steady_sample_count": 9800, + "steady_mean_gpu_util_percent": 87.28051020408164, + "steady_idle_ratio_below_10_percent": 0.061632653061224486, + "peak_memory_used_mib": 44110 + }, + "1": { + "steady_sample_count": 9800, + "steady_mean_gpu_util_percent": 87.53357142857143, + "steady_idle_ratio_below_10_percent": 0.06581632653061224, + "peak_memory_used_mib": 44536 + }, + "2": { + "steady_sample_count": 9800, + "steady_mean_gpu_util_percent": 85.85367346938776, + "steady_idle_ratio_below_10_percent": 0.08561224489795918, + "peak_memory_used_mib": 43839 + }, + "6": { + "steady_sample_count": 9800, + "steady_mean_gpu_util_percent": 84.40714285714286, + "steady_idle_ratio_below_10_percent": 0.09897959183673469, + "peak_memory_used_mib": 44725 + }, + "7": { + "steady_sample_count": 9800, + "steady_mean_gpu_util_percent": 55.431224489795916, + "steady_idle_ratio_below_10_percent": 0.42826530612244895, + "peak_memory_used_mib": 48521 + } + } + }, + "optimizer_update_count": 40, + "perf/step_time": { + "count": 30, + "max": 220.90484619140625, + "mean": 163.6089864095052, + "min": 140.037353515625, + "p50": 156.3287811279297, + "p95": 204.94578552246094 + }, + "perf/step_token_per_s": 1234.9935087294768, + "perf/wall_clock_samples_per_s": 1.5647062280506074, + "rollout/image_count/mean": { + "count": 30, + "max": 1, + "mean": 1, + "min": 1, + "p50": 1, + "p95": 1 + }, + "rollout/raw_reward": { + "count": 30, + "max": 0.6015625, + "mean": 0.42291666666666666, + "min": 0.24609375, + "p50": 0.42578125, + "p95": 0.58203125 + }, + "seed": 20260816, + "steady_actor_fetch_samples": 7680, + "steady_actor_multimodal_tensor_bytes": 47368359936, + "steady_actor_response_tokens": 3098593, + "steady_actor_total_tokens": 6061681, + "train/grad_norm": { + "count": 30, + "max": 2.397125005722046, + "mean": 0.6221419324477514, + "min": 0.39037948846817017, + "p50": 0.5365633964538574, + "p95": 0.8694257736206055 + }, + "train/loss": { + "count": 30, + "max": 0.08942538499832153, + "mean": 0.022442292957566678, + "min": -0.0131751149892807, + "p50": 0.022053017280995846, + "p95": 0.06375551968812943 + }, + "train/ppo_kl": { + "count": 30, + "max": 0, + "mean": 0, + "min": 0, + "p50": 0, + "p95": 0 + }, + "windows": [ + { + "end": 19, + "perf/step_token_per_s": 1232.7710861277762, + "perf/wall_clock_samples_per_s": 1.531808898302748, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": 1234.0340106953922, + "perf/wall_clock_samples_per_s": 1.6989084322219754, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": 1237.9758302190091, + "perf/wall_clock_samples_per_s": 1.4796037655020937, + "start": 30 + } + ] + }, + { + "measured_optimizer_step_count": 30, + "nvml": { + "gpu_count": 5, + "peak_memory_used_mib": 48261, + "steady_mean_gpu_util_percent": 75.54465007045562, + "steady_idle_ratio_below_10_percent": 0.1775293565054016, + "per_gpu": { + "0": { + "steady_sample_count": 10645, + "steady_mean_gpu_util_percent": 81.29760450915923, + "steady_idle_ratio_below_10_percent": 0.10474401127289808, + "peak_memory_used_mib": 43794 + }, + "1": { + "steady_sample_count": 10645, + "steady_mean_gpu_util_percent": 83.71996242367308, + "steady_idle_ratio_below_10_percent": 0.08670737435415689, + "peak_memory_used_mib": 43000 + }, + "2": { + "steady_sample_count": 10645, + "steady_mean_gpu_util_percent": 80.64396430248944, + "steady_idle_ratio_below_10_percent": 0.12550493189290746, + "peak_memory_used_mib": 44585 + }, + "6": { + "steady_sample_count": 10645, + "steady_mean_gpu_util_percent": 79.93865664631282, + "steady_idle_ratio_below_10_percent": 0.12794739314232034, + "peak_memory_used_mib": 43089 + }, + "7": { + "steady_sample_count": 10645, + "steady_mean_gpu_util_percent": 52.1230624706435, + "steady_idle_ratio_below_10_percent": 0.4427430718647252, + "peak_memory_used_mib": 48261 + } + } + }, + "optimizer_update_count": 40, + "perf/step_time": { + "count": 30, + "max": 209.2067108154297, + "mean": 178.93946228027343, + "min": 144.9739532470703, + "p50": 177.79007720947266, + "p95": 206.41053771972656 + }, + "perf/step_token_per_s": 1142.585300313896, + "perf/wall_clock_samples_per_s": 1.4306514434419524, + "rollout/image_count/mean": { + "count": 30, + "max": 1, + "mean": 1, + "min": 1, + "p50": 1, + "p95": 1 + }, + "rollout/raw_reward": { + "count": 30, + "max": 0.51953125, + "mean": 0.41484375, + "min": 0.28515625, + "p50": 0.41015625, + "p95": 0.51171875 + }, + "seed": 20260817, + "steady_actor_fetch_samples": 7680, + "steady_actor_multimodal_tensor_bytes": 46333218816, + "steady_actor_response_tokens": 3196608, + "steady_actor_total_tokens": 6133608, + "train/grad_norm": { + "count": 30, + "max": 1.8864530324935913, + "mean": 0.6340679069360097, + "min": 0.36473023891448975, + "p50": 0.5427196621894836, + "p95": 1.3270831108093262 + }, + "train/loss": { + "count": 30, + "max": 0.05062403157353401, + "mean": 0.014008935376962958, + "min": -0.03006720542907715, + "p50": 0.014289181213825941, + "p95": 0.04760771989822388 + }, + "train/ppo_kl": { + "count": 30, + "max": 2.313238793225092e-12, + "mean": 2.8543548456540686e-14, + "min": -1.6613609359755754e-12, + "p50": 0, + "p95": 1.829344401654831e-12 + }, + "windows": [ + { + "end": 19, + "perf/step_token_per_s": 1163.7225262538445, + "perf/wall_clock_samples_per_s": 1.4780449282695804, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": 1122.4379451954799, + "perf/wall_clock_samples_per_s": 1.3994885763846747, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": 1142.7204973644643, + "perf/wall_clock_samples_per_s": 1.4167704122156008, + "start": 30 + } + ] + } + ], + "windows": [ + { + "end": 19, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.028812327943286975, + "count": 2, + "max": 1232.7710861277762, + "mean": 1198.2468061908103, + "median": 1198.2468061908103, + "min": 1163.7225262538445, + "population_stddev": 34.524279936965854 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.017862651520985946, + "count": 2, + "max": 1.531808898302748, + "mean": 1.5049269132861642, + "median": 1.5049269132861642, + "min": 1.4780449282695804, + "population_stddev": 0.026881985016583787 + }, + "start": 10 + }, + { + "end": 29, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.04735726441425996, + "count": 2, + "max": 1234.0340106953922, + "mean": 1178.235977945436, + "median": 1178.235977945436, + "min": 1122.4379451954799, + "population_stddev": 55.79803274995618 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.09663702069346816, + "count": 2, + "max": 1.6989084322219754, + "mean": 1.549198504303325, + "median": 1.549198504303325, + "min": 1.3994885763846747, + "population_stddev": 0.14970992791865034 + }, + "start": 20 + }, + { + "end": 39, + "perf/step_token_per_s": { + "coefficient_of_variation": 0.04001154273684026, + "count": 2, + "max": 1237.9758302190091, + "mean": 1190.3481637917366, + "median": 1190.3481637917366, + "min": 1142.7204973644643, + "population_stddev": 47.6276664272724 + }, + "perf/wall_clock_samples_per_s": { + "coefficient_of_variation": 0.0216937969444282, + "count": 2, + "max": 1.4796037655020937, + "mean": 1.4481870888588473, + "median": 1.4481870888588473, + "min": 1.4167704122156008, + "population_stddev": 0.03141667664324643 + }, + "start": 30 + } + ] + } + }, + "independent_seed_count": 2, + "measured_optimizer_step_count": 30, + "measurement_scope": "steady_state", + "seeds": [ + 20260816, + 20260817 + ], + "steady_windows": [ + [ + 10, + 19 + ], + [ + 20, + 29 + ], + [ + 30, + 39 + ] + ], + "validation": "passed", + "warmup_excluded_step_count": 10 + } +} diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_grad_norm.png b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_grad_norm.png new file mode 100644 index 000000000..8a524ca52 Binary files /dev/null and b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_grad_norm.png differ diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_loss.png b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_loss.png new file mode 100644 index 000000000..2c9430351 Binary files /dev/null and b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_loss.png differ diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_reward.png b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_reward.png new file mode 100644 index 000000000..cad4d523a Binary files /dev/null and b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_reward.png differ diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_training_curves.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_training_curves.csv new file mode 100644 index 000000000..4cf5530ad --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/steady-state/task21_steady_training_curves.csv @@ -0,0 +1,321 @@ +condition,seed,step,grad_norm,loss,raw_reward,run +B,20260816,0,0.4293786585330963,0.02194426953792572,0.31640625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,1,0.6030386686325073,0.03660392016172409,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,2,0.5094283819198608,0.01914382167160511,0.45703125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,3,0.8793339729309082,-0.008980737067759037,0.25,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,4,0.6675768494606018,0.01620444655418396,0.26953125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,5,0.691041886806488,0.023224251344799995,0.36328125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,6,0.504104495048523,0.008557893335819244,0.48046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,7,0.48701781034469604,0.06600134074687958,0.35546875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,8,0.6853346824645996,0.022806523367762566,0.328125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,9,0.48825782537460327,0.03132158890366554,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,10,0.589114248752594,0.0011430374579504132,0.546875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,11,0.707213282585144,0.02271563932299614,0.46875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,12,0.5321775078773499,0.05539065599441528,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,13,0.44210878014564514,0.037019599229097366,0.36328125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,14,0.2713528871536255,0.00037191444425843656,0.41796875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,15,0.3973892629146576,0.037509653717279434,0.42578125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,16,0.5631625652313232,0.01085006631910801,0.45703125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,17,0.5541276931762695,0.003918529488146305,0.33984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,18,0.48848646879196167,0.07299309968948364,0.40625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,19,0.7457377314567566,0.024170609191060066,0.4765625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,20,0.6154473423957825,0.016352348029613495,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,21,0.4154365658760071,0.013116193935275078,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,22,0.42639413475990295,-0.002228914760053158,0.35546875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,23,0.5218038558959961,0.016374053433537483,0.234375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,24,1.401735782623291,0.060980722308158875,0.42578125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,25,1.264049768447876,-0.008451191708445549,0.3828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,26,0.676998496055603,0.034920889884233475,0.54296875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,27,0.6578500270843506,0.002403048099949956,0.46875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,28,0.6799176335334778,0.048895567655563354,0.41796875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,29,0.6584834456443787,0.015481147915124893,0.48046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,30,0.5294660329818726,0.015398098155856133,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,31,0.37154221534729004,0.01487342081964016,0.33203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,32,0.7131593823432922,0.03901878371834755,0.46875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,33,0.431210458278656,-0.01000908762216568,0.4375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,34,0.6419727802276611,-0.013865238055586815,0.6015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,35,0.49169793725013733,0.014147582463920116,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,36,1.090413212776184,0.004179161507636309,0.4140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,37,2.207888603210449,0.020711280405521393,0.43359375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,38,0.6232101917266846,-0.0069035980850458145,0.62109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260816,39,0.49050506949424744,0.008543052710592747,0.40625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B1-GPU01267-seed20260816-attempt3 +B,20260817,0,2.7602903842926025,-0.007886980660259724,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,1,0.5793234705924988,0.00012385645823087543,0.3203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,2,0.6315371990203857,0.010395298711955547,0.48828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,3,0.3681282699108124,0.03301849588751793,0.39453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,4,0.6795704364776611,0.02745025046169758,0.4296875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,5,0.7129818201065063,0.0028278380632400513,0.3515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,6,0.6459104418754578,0.037297897040843964,0.515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,7,0.7951655387878418,-0.0030044622253626585,0.2890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,8,0.6429036259651184,0.02712106890976429,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,9,0.3982057571411133,0.041706327348947525,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,10,0.6968140602111816,0.029543645679950714,0.44921875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,11,0.8779623508453369,0.0014343725051730871,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,12,0.47050410509109497,0.011719683185219765,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,13,0.47031351923942566,0.08514145761728287,0.5078125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,14,0.35177525877952576,0.03660881519317627,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,15,0.4154990613460541,0.02410261519253254,0.4921875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,16,0.5306638479232788,0.0003302302211523056,0.49609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,17,0.9289870858192444,0.04197186604142189,0.42578125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,18,1.270756483078003,0.01603897102177143,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,19,0.6841323375701904,0.026416698470711708,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,20,0.9330576062202454,-0.016183488070964813,0.48046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,21,0.621405303478241,0.0018361995462328196,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,22,0.35751873254776,-0.016900058835744858,0.31640625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,23,0.7542665004730225,-0.014334156177937984,0.28515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,24,0.40014591813087463,0.01798744685947895,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,25,0.4755542278289795,0.0009055822156369686,0.43359375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,26,0.4592970609664917,0.05735928565263748,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,27,0.4665921628475189,0.011776498518884182,0.5,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,28,0.5185763835906982,0.030008960515260696,0.390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,29,0.6370179057121277,0.014245674014091492,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,30,0.6065971255302429,0.018642403185367584,0.3984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,31,0.7751550078392029,0.036152880638837814,0.53125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,32,0.45251333713531494,0.04169318452477455,0.3984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,33,0.9118742942810059,0.016989264637231827,0.42578125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,34,1.0193886756896973,0.03357290104031563,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,35,0.5737411975860596,0.026598958298563957,0.3125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,36,0.41577407717704773,0.02144905924797058,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,37,0.5253522992134094,0.018464239314198494,0.41796875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,38,0.6528048515319824,0.019930114969611168,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +B,20260817,39,1.791407585144043,-0.05296846851706505,0.5078125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-B2-GPU01267-seed20260817-attempt1 +P,20260816,0,0.4729405343532562,0.021885205060243607,0.31640625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,1,0.5295196175575256,0.03654491901397705,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,2,1.575691819190979,0.01941690966486931,0.5390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,3,0.35747092962265015,-0.007692968472838402,0.1484375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,4,0.5637974739074707,-0.003757946426048875,0.26171875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,5,0.6574077010154724,0.019289309158921242,0.32421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,6,0.44566258788108826,-0.00298148300498724,0.4765625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,7,0.5316495299339294,0.03223445266485214,0.38671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,8,0.8140038251876831,0.042023442685604095,0.3125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,9,0.5527770519256592,0.04901901260018349,0.48046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,10,0.5284427404403687,-0.004423350561410189,0.5546875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,11,1.561697006225586,0.017107928171753883,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,12,0.526813268661499,0.029104452580213547,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,13,0.9332507848739624,-0.013619291596114635,0.33984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,14,0.31992343068122864,-0.0063894446939229965,0.43359375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,15,0.4586181938648224,0.0021749415900558233,0.4296875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,16,0.393179714679718,0.004191843327134848,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,17,0.4663388133049011,0.017029572278261185,0.3359375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,18,1.6172339916229248,0.024319294840097427,0.4140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,19,1.1146173477172852,0.018690943717956543,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,20,0.4738696217536926,-0.012374713085591793,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,21,0.7066768407821655,0.01751827821135521,0.3984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,22,0.4950488805770874,0.00659859599545598,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,23,0.8662661910057068,-0.034701038151979446,0.234375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,24,0.8038854002952576,0.04433699697256088,0.3984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,25,0.36424684524536133,-0.004955354146659374,0.3515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,26,0.496289998292923,0.028196178376674652,0.49609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,27,0.6089060306549072,-0.010121394880115986,0.4609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,28,0.6345522999763489,0.029895776882767677,0.48828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,29,0.627535343170166,0.010277565568685532,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,30,0.5086640119552612,-0.0020217131823301315,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,31,0.4774720370769501,0.004586010240018368,0.30859375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,32,0.6346595883369446,0.026090625673532486,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,33,0.867675244808197,0.010047533549368382,0.44921875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,34,0.5568360090255737,0.007228540722280741,0.609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,35,0.5686536431312561,0.00447122985497117,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,36,0.5462238192558289,0.01280716061592102,0.35546875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,37,0.7333060503005981,0.01197870634496212,0.4609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,38,0.5862743854522705,0.033727746456861496,0.6015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260816,39,0.5540412068367004,0.018008951097726822,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P1-GPU01267-seed20260816-attempt1 +P,20260817,0,2.7640109062194824,-0.007886978797614574,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,1,0.4308488368988037,-2.2280830307863653e-05,0.3203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,2,0.7398115396499634,0.00816586334258318,0.484375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,3,0.3527854382991791,0.033995695412158966,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,4,0.5030105113983154,0.032886143773794174,0.390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,5,0.6556617021560669,0.0377010777592659,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,6,0.7234382629394531,0.005457190331071615,0.49609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,7,0.7995889782905579,0.012613939121365547,0.31640625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,8,0.42740944027900696,0.020417576655745506,0.3828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,9,0.48466211557388306,0.03388126939535141,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,10,0.5708556175231934,0.003057429101318121,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,11,0.6612260937690735,0.0075317807495594025,0.5,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,12,1.0779743194580078,-0.037975482642650604,0.34375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,13,0.687943160533905,0.0633721798658371,0.51953125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,14,0.41971448063850403,0.024807997047901154,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,15,0.4858563542366028,0.04377193748950958,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,16,0.35890820622444153,0.026464229449629784,0.4921875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,17,0.7046846151351929,0.022650450468063354,0.4140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,18,2.7423908710479736,0.018046656623482704,0.35546875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,19,2.1464362144470215,0.004594968166202307,0.390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,20,0.5370921492576599,-0.0011166563490405679,0.5078125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,21,0.48108991980552673,0.0272072646766901,0.3828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,22,0.4602321684360504,0.06103434041142464,0.2890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,23,1.0165218114852905,-0.03124105930328369,0.296875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,24,1.0365668535232544,0.01909361034631729,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,25,3.2474284172058105,-0.0033748229034245014,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,26,0.4717111885547638,0.05393484979867935,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,27,0.4742216169834137,0.011475887149572372,0.51171875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,28,0.704470157623291,0.019410425797104836,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,29,0.5298402309417725,0.012227463535964489,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,30,0.4461480975151062,0.042198970913887024,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,31,0.776177704334259,0.046349991112947464,0.48046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,32,0.6546857953071594,0.03304167091846466,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,33,0.5368762016296387,-0.011446693912148476,0.42578125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,34,0.9210311770439148,0.041022077202796936,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,35,0.5244991779327393,0.01295415312051773,0.3046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,36,0.5175149440765381,0.00398045452311635,0.39453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,37,0.49947023391723633,0.0135958855971694,0.3984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,38,0.5015979409217834,0.03426378592848778,0.32421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P,20260817,39,0.7766110301017761,-0.02237117290496826,0.47265625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-P2-GPU01267-seed20260817-attempt1 +P+R,20260816,0,0.47338244318962097,0.021885205060243607,0.31640625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,1,0.4133748412132263,0.036569032818078995,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,2,1.2601121664047241,0.0031040278263390064,0.5390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,3,0.38716182112693787,-0.010105925612151623,0.21875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,4,0.7740347981452942,-0.01162716094404459,0.3125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,5,0.6072826385498047,0.02193485200405121,0.28515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,6,0.6372195482254028,0.040430497378110886,0.4140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,7,0.8195988535881042,0.046747103333473206,0.34765625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,8,0.6811588406562805,0.02610916644334793,0.3515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,9,0.5834916234016418,0.04901912063360214,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,10,0.4997939169406891,0.03105984441936016,0.5703125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,11,0.9879876971244812,0.021020395681262016,0.47265625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,12,0.6146997213363647,0.0251451525837183,0.38671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,13,0.6214061379432678,0.022901711985468864,0.36328125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,14,0.38057181239128113,0.015889422968029976,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,15,0.5835083723068237,0.008634221740067005,0.45703125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,16,0.4432726502418518,-0.0087343854829669,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,17,1.7316131591796875,0.0003724276612047106,0.3515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,18,0.547851026058197,0.016921337693929672,0.33203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,19,0.5062233209609985,0.027165746316313744,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,20,0.7204654216766357,0.02426760457456112,0.36328125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,21,0.3980172872543335,0.02142534777522087,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,22,0.7968775629997253,-0.016523413360118866,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,23,0.5967206358909607,-0.0067781973630189896,0.234375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,24,0.8269916772842407,0.028393074870109558,0.39453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,25,0.6944261789321899,0.011044951155781746,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,26,0.4281648099422455,0.015420375391840935,0.44921875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,27,0.983905553817749,0.042700476944446564,0.4609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,28,0.7144696116447449,0.04787826165556908,0.4296875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,29,0.9917157888412476,0.006035372149199247,0.45703125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,30,0.533187985420227,0.026935089379549026,0.33203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,31,0.4661215543746948,0.009520302526652813,0.30078125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,32,2.4391937255859375,0.0028886613436043262,0.48046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,33,0.42798563838005066,0.018098417669534683,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,34,0.9939094185829163,-0.012546969577670097,0.59375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,35,0.3615645170211792,0.013003774918615818,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,36,0.557252824306488,0.030962297692894936,0.38671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,37,0.8705654144287109,0.0009490616503171623,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,38,0.5605778694152832,0.03320450708270073,0.58203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260816,39,0.47447118163108826,0.03692331537604332,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R1-GPU01267-seed20260816-attempt1 +P+R,20260817,0,2.769972324371338,-0.007886979728937149,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,1,0.4073338210582733,2.5601180823286995e-05,0.3203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,2,0.8779855966567993,-0.007682532537728548,0.5390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,3,0.5521566867828369,0.060523830354213715,0.38671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,4,0.5532578825950623,0.02635067328810692,0.39453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,5,0.4826686978340149,0.056096307933330536,0.23046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,6,0.46833711862564087,0.01779118925333023,0.48828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,7,1.2048625946044922,0.024395130574703217,0.3046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,8,0.5749351978302002,0.018735729157924652,0.390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,9,0.4800719916820526,0.04862082377076149,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,10,0.5682895183563232,0.05392253398895264,0.48046875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,11,0.7575269341468811,0.008287351578474045,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,12,0.42612501978874207,0.016615981236100197,0.3359375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,13,0.3485816717147827,0.0517825111746788,0.51171875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,14,0.48275449872016907,0.047310762107372284,0.3828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,15,0.5142583847045898,0.036744046956300735,0.515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,16,0.477422297000885,-0.006742516998201609,0.5,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,17,0.712772011756897,0.006858513690531254,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,18,0.7620577216148376,0.028426725417375565,0.38671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,19,2.993971347808838,0.027448929846286774,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,20,0.5199701189994812,0.009012168273329735,0.49609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,21,43.00599670410156,0.020127348601818085,0.39453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,22,1.1236159801483154,0.040389832109212875,0.28515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,23,1.2452892065048218,-0.022954167798161507,0.28125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,24,0.4607888460159302,0.07517870515584946,0.41796875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,25,1.018632411956787,-0.008603663183748722,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,26,0.6951339840888977,0.050391752272844315,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,27,0.5321242213249207,0.0023959081154316664,0.4921875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,28,0.45838674902915955,0.02200007252395153,0.43359375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,29,0.5991933345794678,0.034824881702661514,0.390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,30,0.7337131500244141,0.021401014178991318,0.36328125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,31,0.8726151585578918,0.022478174418210983,0.49609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,32,0.6386184096336365,0.05023587495088577,0.39453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,33,0.6329584717750549,0.022834869101643562,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,34,0.5662320852279663,0.002279069507494569,0.4140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,35,0.5631394982337952,0.0031576985493302345,0.328125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,36,0.6972376108169556,0.027901742607355118,0.38671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,37,0.5924823880195618,0.028340574353933334,0.38671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,38,0.5909588932991028,0.0004916982725262642,0.3359375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+R,20260817,39,0.7618899941444397,-0.014703052118420601,0.49609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-R2-GPU01267-seed20260817-attempt3 +P+S,20260816,0,0.4426826536655426,0.021955233067274094,0.31640625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,1,0.6483739018440247,0.03639811649918556,0.375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,2,0.8764793276786804,-0.0030085884500294924,0.59765625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,3,0.3284049332141876,0.00045440380927175283,0.19140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,4,0.4164806604385376,0.020659249275922775,0.3203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,5,2.2859389781951904,0.0005944211734458804,0.21875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,6,0.4988037347793579,0.04318106546998024,0.46484375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,7,0.5222917795181274,0.03241543844342232,0.35546875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,8,0.7516077756881714,0.04426337033510208,0.2890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,9,0.4641008675098419,0.04562118276953697,0.46875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,10,0.5124375224113464,0.05547500401735306,0.50390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,11,0.8694257736206055,0.02416735514998436,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,12,0.6893731951713562,0.06297720968723297,0.3828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,13,0.45012399554252625,0.04446928948163986,0.32421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,14,0.5614160299301147,0.002928223228082061,0.4375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,15,0.49210411310195923,-0.002379387617111206,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,16,0.44307735562324524,-0.0131751149892807,0.4296875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,17,0.5211713910102844,0.017818881198763847,0.32421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,18,0.5807622075080872,0.024183599278330803,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,19,0.4659591615200043,0.028578417375683784,0.484375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,20,0.5391765236854553,0.037605639547109604,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,21,0.512180507183075,0.005223340354859829,0.421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,22,0.4232514798641205,0.0022321806754916906,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,23,0.6198500990867615,0.02118271216750145,0.24609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,24,0.7422933578491211,0.06375551968812943,0.4140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,25,0.5339502692222595,0.004959962330758572,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,26,0.522504448890686,0.018330475315451622,0.5390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,27,0.5526503920555115,0.010945763438940048,0.5078125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,28,0.6060343980789185,0.025453349575400352,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,29,0.48795178532600403,0.001662274356931448,0.46484375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,30,0.6469882726669312,0.030046548694372177,0.37890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,31,0.39037948846817017,0.003324196906760335,0.32421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,32,2.397125005722046,0.02602141909301281,0.4453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,33,0.4895153045654297,0.02513805963099003,0.453125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,34,0.5872070789337158,0.016267960891127586,0.58203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,35,0.46160319447517395,-0.005560477264225483,0.38671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,36,0.6630641222000122,0.022923322394490242,0.3828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,37,0.47393468022346497,-0.009997839108109474,0.4375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,38,0.7641799449920654,0.08942538499832153,0.6015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260816,39,0.6645668745040894,0.03928551822900772,0.390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S1-GPU01267-seed20260816-attempt1 +P+S,20260817,0,3.230241537094116,-0.007915235124528408,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,1,0.5655097365379333,0.00012966521899215877,0.3203125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,2,0.7203531861305237,0.014087941497564316,0.5546875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,3,0.4997330605983734,0.043957460671663284,0.33984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,4,0.6875415444374084,0.02294766530394554,0.4140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,5,0.6712652444839478,0.011057074181735516,0.33984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,6,0.3869166672229767,0.003252739552408457,0.52734375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,7,1.1427068710327148,0.012456369586288929,0.32421875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,8,2.6959166526794434,0.016448810696601868,0.40234375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,9,0.6206415891647339,0.04283830150961876,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,10,0.7651659846305847,0.041223153471946716,0.4296875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,11,0.8064144253730774,-0.005293759051710367,0.4609375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,12,0.5529792904853821,0.007359628099948168,0.359375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,13,0.36473023891448975,0.030213119462132454,0.5,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,14,0.45431602001190186,0.02219964563846588,0.3984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,15,0.4689640402793884,-0.0002905672008637339,0.50390625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,16,0.510126531124115,-0.014062651433050632,0.5078125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,17,0.5868541598320007,0.02608966827392578,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,18,0.9269288182258606,0.026228947564959526,0.37109375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,19,0.52507084608078,-0.005479198414832354,0.41796875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,20,1.8864530324935913,0.04760771989822388,0.51953125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,21,0.6607013940811157,0.01321273110806942,0.36328125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,22,0.4104609191417694,0.02149779722094536,0.2890625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,23,0.4958585798740387,0.015453586354851723,0.30078125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,24,0.49870139360427856,0.0023238183930516243,0.44140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,25,0.4156319797039032,0.017407333478331566,0.3828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,26,0.53021240234375,0.05062403157353401,0.40625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,27,0.42380091547966003,0.029940683394670486,0.48828125,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,28,0.5976197123527527,0.02401004731655121,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,29,0.5965951681137085,0.026034558191895485,0.3984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,30,0.5253026485443115,0.01341143250465393,0.3984375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,31,0.68580162525177,0.020512228831648827,0.51171875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,32,0.5869836807250977,0.01028822734951973,0.40234375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,33,0.4716886579990387,-0.03006720542907715,0.4375,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,34,1.3270831108093262,0.011893467046320438,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,35,0.5324600338935852,-0.0026947318110615015,0.28515625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,36,0.5863320827484131,0.0053273895755410194,0.41015625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,37,0.4263865649700165,-0.001380586763843894,0.4140625,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,38,0.5714513063430786,0.015166929922997952,0.3671875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 +P+S,20260817,39,0.8309616446495056,0.001510616741143167,0.4921875,STEADY40-ALL-IDLE-e2be8cd-20260817-210540-GPU01267-S2-GPU01267-seed20260817-attempt1 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/task21_correctness_quality.png b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_correctness_quality.png new file mode 100644 index 000000000..da9f12c36 Binary files /dev/null and b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_correctness_quality.png differ diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/task21_gpu_util_vram.png b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_gpu_util_vram.png new file mode 100644 index 000000000..bff7977ee Binary files /dev/null and b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_gpu_util_vram.png differ diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/task21_phase1_overlap.png b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_phase1_overlap.png new file mode 100644 index 000000000..7bad4e38b Binary files /dev/null and b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_phase1_overlap.png differ diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/task21_step_throughput.png b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_step_throughput.png new file mode 100644 index 000000000..c1e0b7190 Binary files /dev/null and b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_step_throughput.png differ diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/task21_window_summary.png b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_window_summary.png new file mode 100644 index 000000000..839d6ffe4 Binary files /dev/null and b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_window_summary.png differ diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/window_speedup_summary.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/window_speedup_summary.csv new file mode 100644 index 000000000..e595f3d6d --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/window_speedup_summary.csv @@ -0,0 +1,5 @@ +baseline,experiment,seed,speedup,window_end,window_start +597.4306030273438,658.458984375,20260811,0.10215141480601897,0,0 +576.8270874023438,647.2715454101562,20260812,0.1221240464365998,0,0 +559.2531127929688,630.8582763671875,20260813,0.12803713012273654,0,0 +569.1944580078125,642.1531372070312,20260814,0.12817882917302992,0,0 diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index 18d4119af..4effd0f99 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -14,7 +14,7 @@ Concretely, Actor and Rollout still run on **separate GPU placement groups** (li | Dimension | Colocate (Sync) | Fully Async | Hybrid | | ------------------- | ---------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- | | **GPU layout** | Actor and Rollout time-share same GPUs | Actor / Rollout / ActorFwd / Reference each have own GPUs | Actor and Rollout on separate GPUs; ref / actor_fwd / adv share actor's GPUs | -| **Data pipeline** | TransferQueue, batch-synchronous | TransferQueue + StreamingDataLoader, fully streaming | TransferQueue + sub-batch streaming (`num-iters-per-train-update`) | +| **Data pipeline** | TransferQueue, batch-synchronous | TransferQueue + StreamingDataLoader, fully streaming | TransferQueue optimizer minis; optional producer-chunk actor-forward pipeline | | **Weight sync** | In-process tensor copy | NCCL broadcast via DCS (Checkpoint Engine) | Sync `UpdateWeightFromTensor` to rollout; TensorBackuper for ref/actor_fwd | | **Staleness** | `max_staleness = 0` (strict on-policy) | Configurable `max_staleness` | Configurable `max_staleness` | | **Roles deployed** | `actor`, `critic`, `rollout` | `actor`, `critic`, `rollout`, `advantages`, `reference`, `actor_fwd` | `actor`, `critic`, `rollout` (same as Colocate; ref/actor_fwd live inside actor) | @@ -113,15 +113,21 @@ ______________________________________________________________________ `relax/backends/megatron/actor.py:708` implements the hybrid training step in three phases: -1. **Collect sub-batches and compute forward log-probs (small memory footprint)** +1. **Collect optimizer minis and compute forward log-probs** - The global batch is split into `num_iters_per_train_update` sub-batches. For each sub-batch the actor: + The actor requests each optimizer mini derived from + `rollout_batch_size * n_samples_per_prompt / global_batch_size`. For each mini it: - pulls data from TransferQueue (`_get_data_from_transfer_queue("train", rollout_id, fields, batch_size, batch_index)`) - runs `_switch_model("ref")` (if ref weights are backed up) and computes ref log-probs - runs `_switch_model("teacher")` (if OPD teacher weights are backed up) and computes teacher log-probs - runs `_switch_model("old_actor" or "actor")` and computes current actor log-probs - - appends the enriched sub-batch to an in-memory list + - appends the enriched mini to an in-memory list + + With the optional incremental actor-forward pipeline enabled, one optimizer + mini is consumed in fixed sample windows aligned to the rollout producer's + nominal transfer threshold. Physical producer put grouping may vary; this + actor split does not change the advantage or optimizer boundary. 2. **Merge sub-batches and compute advantages globally** @@ -131,7 +137,8 @@ ______________________________________________________________________ A single `train(...)` call runs the optimizer step on the merged batch. Afterwards the actor backs up the new weights to the `actor` tag and (on the ref-update interval) refreshes the `ref` tag, then calls `self.update_weights()` to push the updated weights to rollout via `UpdateWeightFromTensor`. -The sub-batched forward keeps peak activation memory bounded — matching Fully Async behavior — while the merged training step preserves Colocate-style global statistics. +The forward phase remains bounded to one fetched unit at a time, while the +merged training step preserves Colocate-style global statistics. ______________________________________________________________________ @@ -143,7 +150,7 @@ ______________________________________________________________________ | ------------------------------- | ------------------------------------------------------------------------------------ | | `--hybrid` | Enable hybrid mode (resolves to `fully_async=True, colocate=True` internally) | | `--resource '{...}'` | Declare `actor` and `rollout` placement groups separately, e.g. `{"actor":[1,4],"rollout":[1,4]}` | -| `--num-iters-per-train-update` | Number of sub-batches per global batch (larger → smaller peak memory, more TQ polls) | +| `--num-iters-per-train-update` | Producer transfer threshold and the fixed actor fetch/forward chunk count | | `--max-staleness` | Off-policy budget (0 = strict on-policy, >0 allows staleness) | ### Optional but Common @@ -167,6 +174,278 @@ When `--hybrid` is set, `relax/utils/arguments.py` defaults the following (unles Hybrid and pure fully async both use `StreamingTokenBudgetSampler` when `--use-dynamic-batch-size` is enabled, so DP token balancing is automatic on that path. Keep `--balance-data` for static/seqlen-balanced batches; it has no additional effect on dynamic batching. ::: +### Incremental actor-forward pipeline + +Hybrid can optionally request fixed sample-count actor chunks from TransferQueue +instead of waiting for a complete optimizer mini: + +| Option | Default | Purpose | +| --- | --- | --- | +| `--hybrid-pipeline-forward` | off | Fetch and forward each fixed actor chunk as soon as enough samples are ready | +| `--hybrid-pipeline-overlap` / `--no-hybrid-pipeline-overlap` | on | Forward immediately, or fetch all identical chunks first for a schedule-matched performance control | +| `--hybrid-pipeline-trace-dir PATH` | unset | Write content-free producer, fetch, restore, forward, advantage, and optimizer events | +| `--hybrid-pipeline-fetch-timeout-s SECONDS` | `600` | Fail an incomplete chunk wait with rollout/mini/chunk context | + +The switch is deliberately off by default. With the reference Qwen3.5-9B +recipe (`global_batch_size=256`, `num_iters_per_train_update=2`, DP=1), the +producer targets 128 samples per transfer, while the actor always requests two +complete 128-sample chunks. Producer `async_put` grouping is intentionally not +part of the contract: `FIRST_COMPLETED` coalescing, tail flush, or backfill can +legitimately produce one, two, or more puts as long as all 256 samples and their +global-index fingerprint are conserved. The optional path: + +1. restores the actor exactly once for the optimizer mini, before waiting for + the first chunk; +2. fetches and forwards chunk 0 while rollout can continue producing chunk 1; +3. fetches and forwards chunk 1; +4. orders every per-sample field by `BatchMeta.global_indexes`; +5. records each chunk's dynamically balanced microbatch schedule, translates + it to the canonical merged-batch indexes, and replays the same sample + grouping and order during training; +6. computes advantages once over all 256 samples and performs one optimizer + step. + +It does not change producer transfer policy, multimodal preprocessing, pixel +tensor values, GRPO group boundaries, reward normalization, or optimizer +semantics. The additional actor fetch is intended to expose rollout/actor +overlap, not to reduce work. + +The schedule replay is a correctness requirement, not a performance tuning +heuristic. Packed multimodal kernels can produce batch-shape-dependent BF16 +rounding differences. Comparing chunked old-policy log-probs with a differently +packed full-batch training forward would therefore create a non-zero PPO ratio +before any weight update. Replaying the exact chunk microbatches keeps the +old-policy and training forward shapes aligned while retaining one optimizer +update over the complete mini. For this reason, the switch requires +`--use-dynamic-batch-size`; an incompatible batch mode fails during actor +startup. + +`--no-hybrid-pipeline-overlap` changes only the ordering of the same chunk +operations: the actor fetches every chunk before starting any chunk forward. +It preserves the chunk-local dynamic schedules and their single merged +optimizer update. This is the registered baseline for a causal performance +comparison. Omitting `--hybrid-pipeline-forward` remains the compatibility +rollback, but its full-batch packing is not a schedule-matched performance +control. + +The first implementation is intentionally limited and fails fast instead of +silently falling back: + +| Dimension | Supported with the switch enabled | +| --- | --- | +| Mode | Hybrid | +| Workload | Multimodal, dynamic-batch GRPO | +| Forward roles | Actor only; no ref/KL, teacher/OPD, old actor, critic, or routing replay | +| Parallel topology | TP=2, DP=1, PP=1, VPP=1, CP=2, EP=1, ETP=1 | +| Offload | `offload_train=False`, `offload_rollout=False` | +| Dropout | Attention and hidden dropout both zero | +| Batch policy | Dynamic microbatching; exactly one fixed optimizer mini per rollout (`rollout_batch_size * n_samples_per_prompt == global_batch_size`); no partial or dynamic-global batch | +| Log-prob source | Actor-computed log-prob; no true-on-policy or rollout-log-prob shortcut | +| TensorBackuper | Normal enabled backuper with only the `actor` tag | + +Chunk sizes must reconstruct the optimizer mini exactly and remain a multiple +of `n_samples_per_prompt`. Duplicate, missing, underfilled, or overfilled +`BatchMeta.global_indexes` terminate the step with an actionable error. +Startup also requires TransferQueue `>=0.1.10.dev0` with +`BatchMeta.global_indexes` and the `async_put(custom_meta=..., is_last=...)` +contract; an incompatible installation fails before Ray workers or rollout +producers can write data. + +The reference launcher exposes the options as environment variables: + +```bash +HYBRID_PIPELINE_FORWARD=1 \ +HYBRID_PIPELINE_TRACE_DIR=/data01/LWX/relax-task21/runs/smoke/timeline \ +HYBRID_PIPELINE_FETCH_TIMEOUT_S=600 \ +NUM_ITERS_PER_TRAIN_UPDATE=4 \ +bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ + hybrid-async +``` + +The same launcher provides default-preserving experiment overrides so a smoke +run or paired benchmark does not require editing the script: + +| Environment variable | Default | Purpose | +| --- | --- | --- | +| `MODEL_CONFIG_FILE` | `qwen35-9B.sh` | Select the model-parallel configuration script | +| `MODEL_NAME` / `MODEL_RUN_NAME` | `Qwen3.5-9B` / `qwen35-9b` | Select the checkpoint subdirectory and log prefix | +| `MODEL_CHECKPOINT_DIR` / `REFERENCE_CHECKPOINT_DIR` | `${MODEL_DIR}/${MODEL_NAME}` | Pin actor and reference inputs explicitly | +| `CHECKPOINT_SAVE` | `1` | Set to `0` to omit all `--save*` arguments and use separate rollout/TensorBoard outputs | +| `ROLLOUT_RESULT_DIR` / `TENSORBOARD_DIR` | No override while saving; `${EXP_DIR}/rollout_result` and `${EXP_DIR}/tensorboard_log` without saving | Preserve raw results and scalars in no-save runs; `TENSORBOARD_DIR` is exported for MetricsService | +| `NUM_ITERS_PER_TRAIN_UPDATE` | `2` | Set the prompt-group-aligned producer and actor chunk count for each optimizer mini | +| `HYBRID_PIPELINE_OVERLAP` | `1` | Set to `0` only with `HYBRID_PIPELINE_FORWARD=1` to run the schedule-matched no-overlap control | +| `ROLLOUT_MAX_RESPONSE_LEN` / `ROLLOUT_MAX_PROMPT_LEN` / `ROLLOUT_MAX_CONTEXT_LEN` | `10240` / `2048` / `12288` | Pin generation and context limits | +| `ACTOR_MAX_TOKENS_PER_GPU` | `12288` | Bound dynamic-batch actor microbatch tokens | +| `HYBRID_ACTOR_GPUS` / `HYBRID_ROLLOUT_GPUS` | `4` / `4` | Build the Hybrid placement resource | +| `ROLLOUT_NUM_GPUS_PER_ENGINE` | `2` | Set each SGLang engine's tensor-parallel width | +| `SGLANG_MEM_FRACTION_STATIC` | `0.8` | Set SGLang's static memory fraction | + +Before submitting a Ray Job, the launcher validates positive integers, context +capacity, rollout-GPU divisibility, and that actor GPU counts are multiples of +the current TP=2, CP=2 topology. Invalid combinations fail before Ray workers +start instead of silently degrading. + +With the reference `global_batch_size=256` and +`n_samples_per_prompt=8`, setting `NUM_ITERS_PER_TRAIN_UPDATE=4` creates four +64-sample stages. Each stage therefore contains eight complete prompt groups. +Baseline and experiment runs in a strict pair both enable chunk forwarding and +use the same value. The baseline sets `HYBRID_PIPELINE_OVERLAP=0`, so it fetches +all four stages before their forwards; the experiment sets it to `1`, so each +ready stage is forwarded while later stages are still produced. + +For example, this is a no-save Qwen3-VL-8B configuration for a constrained +machine. It may be used for a resource-qualified smoke or paired performance +benchmark, but its results must be labeled separately from the default 8-GPU +Qwen3.5 recipe: + +```bash +MODEL_CONFIG_FILE="${MODEL_CONFIG_DIR}/qwen3-vl-8B.sh" \ +MODEL_NAME=Qwen3-VL-8B-Instruct \ +MODEL_RUN_NAME=qwen3-vl-8b \ +CHECKPOINT_SAVE=0 \ +ROLLOUT_MAX_RESPONSE_LEN=512 \ +ROLLOUT_MAX_PROMPT_LEN=2048 \ +ROLLOUT_MAX_CONTEXT_LEN=2560 \ +ACTOR_MAX_TOKENS_PER_GPU=6144 \ +NUM_ITERS_PER_TRAIN_UPDATE=4 \ +HYBRID_ACTOR_GPUS=4 \ +HYBRID_ROLLOUT_GPUS=1 \ +ROLLOUT_NUM_GPUS_PER_ENGINE=1 \ +HYBRID_PIPELINE_FORWARD=1 \ +bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ + hybrid-async +``` + +After changing the model, resource topology, or length limits, compare only +against a baseline with the identical configuration. Do not pool such runs +with the default 8-GPU recipe. + +Trace files are separated by hostname, role, PID, and global rank. They contain +timestamps, counts, token totals, multimodal tensor byte counts, CUDA peaks, +and an irreversible global-index fingerprint; prompts, responses, images, and +sample tensors are never serialized. + +Validate one run: + +```bash +python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ + --run-dir /data01/LWX/relax-task21/runs/smoke \ + --validate-only +``` + +Compare two baseline and two experiment runs and generate the registered CSV, +JSON, and curve artifacts: + +```bash +python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ + --run-dir /data01/LWX/relax-task21/runs/A1-baseline-seed20260728 \ + --run-dir /data01/LWX/relax-task21/runs/A2-baseline-seed20260729 \ + --run-dir /data01/LWX/relax-task21/runs/B1-experiment-seed20260728 \ + --run-dir /data01/LWX/relax-task21/runs/B2-experiment-seed20260729 \ + --output-dir /data01/LWX/relax-task21/comparison \ + --enforce-targets +``` + +For every paired seed, launch the baseline with +`HYBRID_PIPELINE_FORWARD=1 HYBRID_PIPELINE_OVERLAP=0` and the experiment with +`HYBRID_PIPELINE_FORWARD=1 HYBRID_PIPELINE_OVERLAP=1`. All other manifest +fields, including candidate commit, model, data, limits, topology, seed, image, +and hardware fingerprint, must match. + +The analyzer validates event closure, one restore and optimizer step, sample +conservation, same-host monotonic timing, finite metrics, producer/fetch +fingerprints, steady-step coverage, and the registered performance thresholds. +Producer put count is diagnostic only; actor fetch/forward count remains +strictly fixed. Strict producer overlap means the first actor forward starts +before the producer starts its final put. The final put completion is retained +only as a transfer-stage diagnostic because its trace write can lose a +scheduling race with the consumer. With `--enforce-targets`, the analyzer also +checks that the baseline fetched all chunks before its first forward, that the +experiment forwarded before its final fetch completed, per-run strict producer +overlap, step-time p95, +the configured expected-GPU NVML coverage, peak VRAM, +token/multimodal-byte workload, and the +raw-reward, truncation-rate, staleness, same-weight PPO KL, and policy +clip-fraction guardrails. Same-weight `abs(train/ppo_kl)` and +`abs(train/pg_clipfrac)` must each remain at or below `1e-7`. It uses `sum(step_tokens) / +sum(step_time)` for aggregate throughput rather than averaging per-step rates. +The truncation guardrail reads the rollout-side +`rollout/truncated_ratio`; the training-side `rollout/truncated` scalar is not +used because chunk aggregation can sum that value more than once per step. +GPU utilization and the below-10% idle ratio use only 500 ms NVML samples +whose wall time falls inside the registered steady step intervals reconstructed +from TensorBoard `perf/step_time`; sampled peak VRAM remains a full-run safety +metric. +The strict comparison additionally requires a clean run manifest, identical +candidate/image/TransferQueue identities, verified static-input hashes, +dependency freeze, wheel hash, launcher log, and zero training/validation/final +exit-status artifacts for every run. +The manifest also records and cross-checks `max_staleness`, global/rollout +batch sizes, samples per prompt, and actor chunk count, so CLI expectations +cannot silently disagree with the measured workload. +The registered four-stage protocol requires exactly four actor fetch/forward +chunks of 64 samples. Producer puts may be regrouped by asynchronous completion; +their events must close and their aggregate sample count and fingerprint must +match the actor-consumed workload exactly. +Before any paired statistics are calculated, the analyzer also requires +identical model/config/data paths, prompt/response/context limits, actor token +budget, actor/rollout resource topology, SGLang determinism and memory +settings, physical-to-container GPU mapping, checkpoint mode, and debug +capture/replay settings. Missing workload fields or any mismatch fail closed. +The hostname and a SHA-256 fingerprint over GPU UUID, model, PCI address, and +driver version must also match across the comparison. +Paired global-index fingerprints must match exactly. For the steady-state +campaign, the ordered schedule may differ under asynchronous chunk forwarding, +so the analyzer compares the full warmup-excluded input fingerprint/sample +multiset. Generated response lengths and packed multimodal-byte totals remain +reported per run and are included in the token-throughput denominator; they are +outputs of sampling and dynamic microbatching rather than fixed prompt/image +inputs. The first-step comparison retains its stricter exact token and byte +workload guard. The comparison JSON reports the mean, median, range, population +standard deviation, and coefficient of variation across paired repeats. +The staleness curve is the trace-derived producer lead at the first actor +forward: the largest completed producer rollout ID minus the actor rollout ID +at that timestamp. The current rollout is considered ready once its actor fetch +completes even if the producer trace write is scheduled slightly later. The +lead must remain at most the manifest's configured `max_staleness` value +(`2` in the reference recipe), and its paired steady mean may increase by at +most `0.25`. + +The higher-confidence Task 21 mechanisms should be benchmarked as separate +cumulative ablations, following the accepted PR #201 protocol. In particular, +compare ProcessorPool and true-on-policy train-forward log-prob reuse separately +from chunk overlap. The labels below intentionally do not reuse PR #201's +`A1/A2/A3` names because this branch does not bundle its owner/ref group payload +dedup, whose paired mean regressed samples/s in that PR: + +```text +B : MM_PROCESSOR_POOL_SIZE=0 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=0 +P : MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=0 +P+R : MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=1 HYBRID_PIPELINE_FORWARD=0 +P+S : MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=1 +``` + +Run at least 40 optimizer steps per fresh process, exclude warmup, use at least +two independent runs per condition, and report the range as well as the mean. +`HYBRID_REUSE_TRAIN_LOGPROBS=1` requires deterministic forwards, one optimizer +mini per rollout partition, and TIS when `max_staleness > 0`. It intentionally +cannot be combined with `HYBRID_PIPELINE_FORWARD=1` in the reference launcher, +so the two effects remain attributable. + +To roll back, omit `--hybrid-pipeline-forward` (or set +`HYBRID_PIPELINE_FORWARD=0`). No checkpoint or dataset conversion is needed. +Before widening the support matrix, add collective-order and restore-count +tests for the new DP/PP/VPP or role graph, then rerun frozen-input parity, +multimodal smoke, and paired performance measurements. + +`--steady-windows 0-0` deliberately measures a fresh-process first optimizer +step. It is useful when only one training step fits the fixed resource window, +but it is not a steady-state throughput claim. Label it as a paired first-step +benchmark, balance launch order across at least two seeds, and use later +multi-step windows whenever resources permit. The analyzer records this as +`measurement_scope=fresh_process_first_step`; later windows with at least three +selected steps are recorded as `measurement_scope=steady_state`. + ______________________________________________________________________ ## Quick Start @@ -183,7 +462,7 @@ ray job submit --address="http://127.0.0.1:8265" \ --resource '{"actor": [1, 4], "rollout": [1, 4]}' \ --max-staleness 2 \ --num-data-storage-units 1 \ - --num-iters-per-train-update 8 \ + --num-iters-per-train-update 2 \ --balance-data \ --hybrid \ "${MODEL_ARGS[@]}" \ @@ -200,9 +479,11 @@ Key points in this configuration: - 8 total GPUs split 4 + 4 between actor and rollout - `max-staleness 2` — actor may consume rollout output up to 2 steps behind the freshest weights -- `num-iters-per-train-update 8` — each global batch is split into 8 sub-batches for forward passes -- `balance-data` — accepted by this script; DP token balancing is handled by the dynamic batch path -- GRPO algorithm with `--use-kl-loss` and `--use-tis` (these are algorithm flags, orthogonal to hybrid) +- `num-iters-per-train-update 2` — rollout targets half-batch transfers and + the optional incremental path performs two fixed 128-sample actor fetches; + the physical producer put count may vary +- `balance-data` — DP load balancing enabled +- GRPO algorithm with `--use-tis`; KL/ref forward is disabled in this recipe ______________________________________________________________________ diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index cfdf44190..d3bfbabef 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -14,7 +14,7 @@ | 维度 | Colocate（同步） | Fully Async（全异步） | Hybrid | | ------------------- | ----------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------- | | **GPU 布局** | Actor 与 Rollout 分时复用同一组 GPU | Actor / Rollout / ActorFwd / Reference 各自独立 GPU | Actor 与 Rollout 独立 GPU；ref / actor_fwd / adv 复用 actor 的 GPU | -| **数据流水线** | TransferQueue，批同步 | TransferQueue + StreamingDataLoader，完全流式 | TransferQueue + 子批次流式（`num-iters-per-train-update`） | +| **数据流水线** | TransferQueue，批同步 | TransferQueue + StreamingDataLoader，完全流式 | TransferQueue optimizer mini；可选 producer-chunk actor-forward 流水线 | | **权重同步** | 进程内 tensor 拷贝 | 通过 DCS（Checkpoint Engine）做 NCCL broadcast | 同步的 `UpdateWeightFromTensor` 推给 rollout；ref/actor_fwd 走 TensorBackuper | | **Staleness** | `max_staleness = 0`（严格 on-policy） | 可配置 `max_staleness` | 可配置 `max_staleness` | | **部署的角色** | `actor`, `critic`, `rollout` | `actor`, `critic`, `rollout`, `advantages`, `reference`, `actor_fwd` | `actor`, `critic`, `rollout`（与 Colocate 相同；ref/actor_fwd 在 actor 内部） | @@ -115,15 +115,21 @@ ______________________________________________________________________ `relax/backends/megatron/actor.py:708` 实现的 Hybrid 训练步骤分为三个阶段： -1. **采集子批次并完成小批次 forward 计算（峰值显存小）** +1. **采集 optimizer mini 并计算 forward log-probs** - 全局 batch 被切分为 `num_iters_per_train_update` 份子批次。对每个子批次，actor 会： + actor 按 + `rollout_batch_size * n_samples_per_prompt / global_batch_size` + 推导 optimizer mini。对每个 mini，actor 会： - 从 TransferQueue 拉取数据（`_get_data_from_transfer_queue("train", rollout_id, fields, batch_size, batch_index)`） - 若已备份 ref 权重，执行 `_switch_model("ref")` 并计算 ref log-probs - 若已备份 teacher 权重（OPD 场景），执行 `_switch_model("teacher")` 并计算 teacher log-probs - 执行 `_switch_model("old_actor" 或 "actor")` 并计算当前 actor 的 log-probs - - 把扩充后的子批次追加到内存列表 + - 把扩充后的 mini 追加到内存列表 + + 打开可选增量 actor-forward 流水线后，一个 optimizer mini 会按与 rollout + producer 名义传输阈值对齐的固定 sample 窗口消费；物理 producer put 分组 + 可以变化，该额外 actor 切分不会改变 advantage 或 optimizer 边界。 2. **合并子批次并做全局 Advantages 归一化** @@ -133,7 +139,8 @@ ______________________________________________________________________ 一次 `train(...)` 调用基于合并后的 batch 完成优化器步进。随后 actor 把新权重备份到 `actor` tag（如果到达 ref 更新间隔，也刷新 `ref` tag），然后调用 `self.update_weights()` 通过 `UpdateWeightFromTensor` 把最新权重同步给 rollout。 -子批次 forward 控制了激活峰值显存（与 Fully Async 行为一致），而合并后的训练步则保留了 Colocate 风格的全局统计量。 +forward 阶段每次只处理一个已拉取单元，而合并后的训练步保留 Colocate 风格的 +全局统计量。 ______________________________________________________________________ @@ -145,7 +152,7 @@ ______________________________________________________________________ | ------------------------------- | ----------------------------------------------------------------------------------------------- | | `--hybrid` | 启用 Hybrid 模式（内部展开为 `fully_async=True, colocate=True`） | | `--resource '{...}'` | 分别声明 `actor` 与 `rollout` 的 placement group，例如 `{"actor":[1,4],"rollout":[1,4]}` | -| `--num-iters-per-train-update` | 每个全局 batch 切分的子批次数量（越大 → 峰值显存越小，TransferQueue 轮询次数越多） | +| `--num-iters-per-train-update` | producer 传输阈值，同时作为固定的 actor fetch/forward chunk 数量 | | `--max-staleness` | Off-policy 容忍度（0 = 严格 on-policy，>0 允许一定程度滞后） | ### 常用可选参数 @@ -169,6 +176,243 @@ ______________________________________________________________________ Hybrid 和纯 fully async 在启用 `--use-dynamic-batch-size` 时都会使用 `StreamingTokenBudgetSampler`，因此动态 batch 路径会自动做 DP token 均衡。`--balance-data` 主要用于静态/序列长度均衡 batch；在动态 batch 路径上没有额外效果。 ::: +### 增量 actor-forward 流水线 + +Hybrid 可选地从 TransferQueue 按固定 sample count 增量请求 actor chunk， +而不是等待完整 optimizer mini： + +| 参数 | 默认值 | 作用 | +| --- | --- | --- | +| `--hybrid-pipeline-forward` | 关闭 | 足量 sample ready 后立即 fetch 固定 actor chunk 并执行 forward | +| `--hybrid-pipeline-overlap` / `--no-hybrid-pipeline-overlap` | 开启 | chunk ready 后立即 forward；或先 fetch 全部相同 chunk，作为调度匹配的性能对照 | +| `--hybrid-pipeline-trace-dir PATH` | 未设置 | 记录不含样本内容的 producer、fetch、restore、forward、advantage 和 optimizer 事件 | +| `--hybrid-pipeline-fetch-timeout-s SECONDS` | `600` | chunk 未完整到达时，带 rollout/mini/chunk 上下文终止等待 | + +该开关默认关闭。参考 Qwen3.5-9B 配方中， +`global_batch_size=256`、`num_iters_per_train_update=2`、DP=1，producer +以 128 samples 为传输目标，而 actor 固定请求两个完整的 128-sample chunk。 +producer 的 `async_put` 分组不是契约：`FIRST_COMPLETED` 合并、tail flush 或 +backfill 都可能合法地产生 1 次、2 次或更多次 put；只要 256 samples 及 +global-index fingerprint 完整守恒即可。打开开关后： + +1. 每个 optimizer mini 只 restore 一次 actor，并在等待首块前执行； +2. chunk 0 ready 后立即 fetch 和 forward，此时 rollout 可继续产生 chunk 1； +3. 再 fetch 和 forward chunk 1； +4. 按 `BatchMeta.global_indexes` 对所有 per-sample 字段恢复确定顺序； +5. 记录每个 chunk 的动态均衡 microbatch 调度，将其映射为合并 batch 中的 + canonical index，并在训练 forward/backward 中回放相同的样本分组与顺序； +6. 在完整 256 samples 上只计算一次 advantage，并只执行一次 optimizer step。 + +该路径不修改 producer 传输策略、多模态预处理、pixel tensor 数值、GRPO +group 边界、reward normalization 或 optimizer 语义。多出的一次 actor fetch +用于暴露 rollout/actor 重叠窗口，而不是减少工作量。 + +microbatch 调度回放是正确性约束，而不是性能调参。多模态 packed kernel +在 BF16 下可能因 batch shape 不同产生数值舍入差异；如果 old-policy +log-prob 按 chunk 计算、训练 forward 却按另一种完整 batch 分组，就会在 +权重尚未更新时产生非零 PPO ratio。回放完全相同的 chunk microbatch，可在 +保持整个 optimizer mini 只更新一次的同时，对齐 old-policy 与训练 forward +的计算形状。因此该开关强制要求 `--use-dynamic-batch-size`；batch 模式不兼容 +时会在 actor 启动阶段直接失败。 + +`--no-hybrid-pipeline-overlap` 只改变同一组 chunk 操作的执行顺序：actor +先 fetch 完所有 chunk，再开始任何 chunk forward。chunk 内动态 microbatch +调度及合并后的单次 optimizer update 均保持不变，因此它是因果性能比较的 +注册 baseline。完全去掉 `--hybrid-pipeline-forward` 仍是兼容性回滚方式， +但 full-batch packing 不属于调度匹配的性能对照。 + +首版支持范围有意收窄；不支持的组合会 fail fast，不会静默回退： + +| 维度 | 打开开关时支持的范围 | +| --- | --- | +| 模式 | Hybrid | +| 负载 | 多模态、dynamic-batch GRPO | +| Forward role | 仅 actor；不支持 ref/KL、teacher/OPD、old actor、critic、routing replay | +| 并行拓扑 | TP=2、DP=1、PP=1、VPP=1、CP=2、EP=1、ETP=1 | +| Offload | `offload_train=False`、`offload_rollout=False` | +| Dropout | attention/hidden dropout 均为 0 | +| Batch 策略 | 使用 dynamic microbatch；每次 rollout 恰好一个固定 optimizer mini（`rollout_batch_size * n_samples_per_prompt == global_batch_size`）；不支持 partial/dynamic-global batch | +| Log-prob 来源 | actor 计算；不支持 true-on-policy 或 rollout-log-prob 快捷路径 | +| TensorBackuper | 启用普通 backuper，且只有 `actor` tag | + +chunk 大小必须精确重建 optimizer mini，并且是 `n_samples_per_prompt` 的 +整数倍。`BatchMeta.global_indexes` 发生重复、缺失、少取或多取时，当前 step +会带可定位信息直接失败。 +启动时还要求 TransferQueue `>=0.1.10.dev0`，并具备 +`BatchMeta.global_indexes` 与 `async_put(custom_meta=..., is_last=...)` +契约；版本或 API 不兼容会在 Ray worker 和 rollout producer 写入数据前失败。 + +参考脚本通过环境变量暴露这些参数： + +```bash +HYBRID_PIPELINE_FORWARD=1 \ +HYBRID_PIPELINE_TRACE_DIR=/data01/LWX/relax-task21/runs/smoke/timeline \ +HYBRID_PIPELINE_FETCH_TIMEOUT_S=600 \ +NUM_ITERS_PER_TRAIN_UPDATE=4 \ +bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ + hybrid-async +``` + +同一脚本还提供默认行为不变的实验开关，避免为了 smoke 或配对基准临时修改 +脚本正文： + +| 环境变量 | 默认值 | 作用 | +| --- | --- | --- | +| `MODEL_CONFIG_FILE` | `qwen35-9B.sh` | 选择模型并行配置脚本 | +| `MODEL_NAME` / `MODEL_RUN_NAME` | `Qwen3.5-9B` / `qwen35-9b` | 选择 checkpoint 子目录和日志前缀 | +| `MODEL_CHECKPOINT_DIR` / `REFERENCE_CHECKPOINT_DIR` | `${MODEL_DIR}/${MODEL_NAME}` | 显式固定 actor 与 reference 输入 | +| `CHECKPOINT_SAVE` | `1` | 设为 `0` 时不传 `--save*`；改用独立 rollout 与 TensorBoard 输出目录 | +| `ROLLOUT_RESULT_DIR` / `TENSORBOARD_DIR` | 保存开启时不覆盖；关闭时为 `${EXP_DIR}/rollout_result` 和 `${EXP_DIR}/tensorboard_log` | 在 no-save 运行中保留原始结果和标量；`TENSORBOARD_DIR` 会导出给 MetricsService | +| `NUM_ITERS_PER_TRAIN_UPDATE` | `2` | 设置每个 optimizer mini 中按完整 prompt group 对齐的 producer/actor 分块数 | +| `HYBRID_PIPELINE_OVERLAP` | `1` | 仅在 `HYBRID_PIPELINE_FORWARD=1` 时设为 `0`，运行调度匹配的 no-overlap 对照 | +| `ROLLOUT_MAX_RESPONSE_LEN` / `ROLLOUT_MAX_PROMPT_LEN` / `ROLLOUT_MAX_CONTEXT_LEN` | `10240` / `2048` / `12288` | 固定生成和上下文上限 | +| `ACTOR_MAX_TOKENS_PER_GPU` | `12288` | 控制 dynamic-batch actor microbatch 的 token 上限 | +| `HYBRID_ACTOR_GPUS` / `HYBRID_ROLLOUT_GPUS` | `4` / `4` | 构造 Hybrid placement resource | +| `ROLLOUT_NUM_GPUS_PER_ENGINE` | `2` | 设置每个 SGLang engine 的 TP 数 | +| `SGLANG_MEM_FRACTION_STATIC` | `0.8` | 设置 SGLang 静态显存比例 | + +脚本会在提交 Ray Job 前校验正整数、上下文容量、rollout GPU 可整除关系， +以及 actor GPU 数是当前 TP=2、CP=2 拓扑的整数倍。不满足约束会直接退出， +不会在 Ray worker 内静默降级。 + +参考配置的 `global_batch_size=256`、`n_samples_per_prompt=8`。将 +`NUM_ITERS_PER_TRAIN_UPDATE` 设为 `4` 时，每个 optimizer mini 会形成四个 +64-sample stage，每个 stage 包含八组完整 prompt。严格配对中的 baseline 与 +experiment 都启用 chunk forward 并使用相同分块数。baseline 设置 +`HYBRID_PIPELINE_OVERLAP=0`，先 fetch 完四个 stage 再依次 forward；experiment +设置为 `1`，每个 stage ready 后立即 forward，并与后续 stage 的生成重叠。 + +例如，以下命令是资源受限环境中的 no-save Qwen3-VL-8B 配置，可用于明确 +标注资源条件的 smoke 或配对性能实验，但结果不能与默认 8 卡 Qwen3.5 配方 +合并： + +```bash +MODEL_CONFIG_FILE="${MODEL_CONFIG_DIR}/qwen3-vl-8B.sh" \ +MODEL_NAME=Qwen3-VL-8B-Instruct \ +MODEL_RUN_NAME=qwen3-vl-8b \ +CHECKPOINT_SAVE=0 \ +ROLLOUT_MAX_RESPONSE_LEN=512 \ +ROLLOUT_MAX_PROMPT_LEN=2048 \ +ROLLOUT_MAX_CONTEXT_LEN=2560 \ +ACTOR_MAX_TOKENS_PER_GPU=6144 \ +NUM_ITERS_PER_TRAIN_UPDATE=4 \ +HYBRID_ACTOR_GPUS=4 \ +HYBRID_ROLLOUT_GPUS=1 \ +ROLLOUT_NUM_GPUS_PER_ENGINE=1 \ +HYBRID_PIPELINE_FORWARD=1 \ +bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ + hybrid-async +``` + +资源拓扑、模型或长度上限变化后，只能与完全相同配置的 baseline 做配对 +比较；不得与默认 8 卡配方合并统计。 + +Trace 按 hostname、role、PID 和 global rank 分文件。内容仅包括时间戳、 +样本与 token 计数、多模态 tensor 字节数、CUDA 峰值以及不可逆的 global-index +fingerprint；不会写 prompt、response、图片或样本 tensor。 + +校验单次运行： + +```bash +python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ + --run-dir /data01/LWX/relax-task21/runs/smoke \ + --validate-only +``` + +每个配对 seed 中，baseline 使用 +`HYBRID_PIPELINE_FORWARD=1 HYBRID_PIPELINE_OVERLAP=0`，experiment 使用 +`HYBRID_PIPELINE_FORWARD=1 HYBRID_PIPELINE_OVERLAP=1`。candidate commit、 +模型、数据、长度上限、拓扑、seed、镜像及硬件指纹等其余 manifest 字段必须一致。 + +比较两次 baseline 和两次 experiment，并生成注册的 CSV、JSON 与曲线： + +```bash +python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ + --run-dir /data01/LWX/relax-task21/runs/A1-baseline-seed20260728 \ + --run-dir /data01/LWX/relax-task21/runs/A2-baseline-seed20260729 \ + --run-dir /data01/LWX/relax-task21/runs/B1-experiment-seed20260728 \ + --run-dir /data01/LWX/relax-task21/runs/B2-experiment-seed20260729 \ + --output-dir /data01/LWX/relax-task21/comparison \ + --enforce-targets +``` + +分析器会校验事件闭合、每 mini 一次 restore、每 step 一次 optimizer、样本 +守恒、同机 monotonic 计时、有限数值、producer/fetch fingerprint、稳态 step +覆盖率以及预注册性能门槛。producer put 次数仅作为诊断值，actor +fetch/forward 次数仍严格固定。严格 producer 重叠定义为首个 actor forward +早于 producer 最后一次 put 的开始时刻；最后一次 put 的完成时刻仅作为传输 +阶段诊断，因为其 trace 写入可能在调度上晚于 consumer。`--enforce-targets` +还会检查 baseline 在首次 forward 前已 fetch 完全部 chunk、experiment 在 +最后一次 fetch 完成前已开始 forward、每次实验的严格 producer 重叠比例、 +step-time p95、命令指定数量的 GPU NVML 覆盖、峰值显存、token/多模态字节工作量和 +raw-reward、截断率、staleness、同权重 PPO KL 与 policy clip fraction 护栏。 +同权重 `abs(train/ppo_kl)` 与 `abs(train/pg_clipfrac)` 都必须不超过 `1e-7`。 +aggregate throughput 使用 +`sum(step_tokens) / sum(step_time)`，不会对 per-step rate 做简单平均。 +截断率护栏读取 rollout 侧的 `rollout/truncated_ratio`；训练侧 +`rollout/truncated` 在分块聚合时可能被一个 step 累加多次，因此不作为截断率 +统计口径。 +GPU utilization 与低于 10% 的 idle ratio 只使用墙钟时间落入预注册稳态 +step 区间的 500 ms NVML 样本；区间由 TensorBoard `perf/step_time` 的 +step end wall time 和 duration 重建。sampled peak VRAM 仍使用 full-run +安全口径。 +严格比较还要求每次 run 的 manifest 为 clean tree，candidate/image/TransferQueue +身份完全一致，并存在已校验的静态输入 hash、依赖 freeze、wheel hash、 +launcher log，以及值为 0 的训练、验证和最终退出状态文件。 +manifest 还会记录并交叉校验 `max_staleness`、global/rollout batch size、 +每 prompt 样本数和 actor chunk 数，避免分析器 CLI 与实际工作负载静默不一致。 +注册的四阶段协议要求 actor 恰好执行四次、每次 64 sample 的 fetch/forward。 +producer put 可因异步完成顺序重新分组，但事件必须闭合，且聚合后的 sample 总数与 +fingerprint 必须和 actor 实际消费的工作量精确一致。 +在计算任何配对统计前，分析器还要求模型、模型配置、数据路径、prompt/response/ +context 上限、actor token 预算、actor/rollout 资源拓扑、SGLang 确定性与显存 +配置、物理卡到容器卡映射、checkpoint 模式以及 debug 捕获/回放配置完全一致； +字段缺失或取值不同都会 fail closed。 +此外还要求 hostname，以及由 GPU UUID、型号、PCI 地址和驱动版本生成的 +SHA-256 硬件指纹完全一致。 +成对 run 的 global-index fingerprint 必须精确一致。对于稳态 campaign，异步 +chunk forwarding 允许 step 间调度顺序不同，因此 analyzer 比较 warmup 之后完整 +窗口的输入 fingerprint/sample 多重集合。生成的 response 长度和动态 +microbatching 产生的多模态 tensor 字节数仍按 run 报告，并计入 token 吞吐率的 +分子；它们是采样与动态打包的输出，不是固定 prompt/image 输入。first-step +comparison 仍保留严格的 token 与字节工作量精确匹配护栏。comparison JSON 同时 +报告重复实验的均值、中位数、范围、总体标准差和变异系数。 +staleness 曲线来自 trace：在 actor 首次 forward 时，用已完成 put 的最大 +producer rollout ID 减去当前 actor rollout ID；若 producer 的 trace 写入稍晚， +已完成的 actor fetch 本身可证明当前 rollout 已 ready。该值不得超过 manifest +记录的 `max_staleness`（参考配方为 `2`），paired 稳态均值最多增加 `0.25`。 + +Task 21 中置信度更高的机制应按 PR #201 已验收的方法做独立累计消融，不要把 +ProcessorPool、true-on-policy train-forward log-prob 复用与 chunk overlap 捆成 +一个 experiment。以下标签有意不沿用 PR #201 的 `A1/A2/A3`：本分支没有捆绑 +其 owner/ref group payload dedup，因为该机制在 #201 的配对均值中使 samples/s +回退： + +```text +B : MM_PROCESSOR_POOL_SIZE=0 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=0 +P : MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=0 +P+R : MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=1 HYBRID_PIPELINE_FORWARD=0 +P+S : MM_PROCESSOR_POOL_SIZE=8 HYBRID_REUSE_TRAIN_LOGPROBS=0 HYBRID_PIPELINE_FORWARD=1 +``` + +每个 fresh process 至少运行 40 个 optimizer step，排除 warmup；每个条件至少 +两次独立运行，同时报告均值和范围。`HYBRID_REUSE_TRAIN_LOGPROBS=1` 要求前向 +确定性、每个 rollout partition 恰好一个 optimizer mini，且 `max_staleness > 0` +时必须启用 TIS。参考 launcher 有意禁止它与 `HYBRID_PIPELINE_FORWARD=1` 同时 +启用，以保持两项收益可归因。 + +回滚时去掉 `--hybrid-pipeline-forward`，或设置 +`HYBRID_PIPELINE_FORWARD=0`；无需转换 checkpoint 或数据集。若要扩展 +DP/PP/VPP 或 role graph，必须先新增 collective-order 与 restore-count 测试， +再重跑 frozen-input parity、多模态 smoke 和成对性能实验。 + +`--steady-windows 0-0` 明确统计“每次启动新进程后的第一个 optimizer step”。 +它适用于固定资源窗口只能容纳单步训练的场景，但不能表述为稳态吞吐。报告中 +应称为配对 first-step benchmark，至少使用两个 seed 并平衡启动顺序；资源允许 +时仍应补充后续多 step 的稳态窗口。分析器会将前者记录为 +`measurement_scope=fresh_process_first_step`；至少包含三个后续 step 的窗口记录为 +`measurement_scope=steady_state`。 + ______________________________________________________________________ ## 快速开始 @@ -185,7 +429,7 @@ ray job submit --address="http://127.0.0.1:8265" \ --resource '{"actor": [1, 4], "rollout": [1, 4]}' \ --max-staleness 2 \ --num-data-storage-units 1 \ - --num-iters-per-train-update 8 \ + --num-iters-per-train-update 2 \ --balance-data \ --hybrid \ "${MODEL_ARGS[@]}" \ @@ -202,9 +446,10 @@ ray job submit --address="http://127.0.0.1:8265" \ - 8 GPU 总量，actor 与 rollout 各占 4 张 - `max-staleness 2` —— actor 可以消费比最新权重落后最多 2 个 step 的 rollout 输出 -- `num-iters-per-train-update 8` —— 每个全局 batch 在 forward 阶段被切分为 8 个子批次 -- `balance-data` —— 该脚本可传入；DP token 均衡由动态 batch 路径处理 -- 算法采用 GRPO，附带 `--use-kl-loss` 与 `--use-tis`（这些是算法参数，与 Hybrid 正交） +- `num-iters-per-train-update 2` —— rollout 以半批为传输目标，可选增量路径 + 固定执行两次 128-sample actor fetch；物理 producer put 次数可以变化 +- `balance-data` —— 启用 DP 间负载均衡 +- 算法采用 GRPO 与 `--use-tis`；该参考配方关闭 KL/ref forward ______________________________________________________________________ diff --git a/relax/backends/megatron/actor.py b/relax/backends/megatron/actor.py index 75143efa6..ae91c5ddc 100644 --- a/relax/backends/megatron/actor.py +++ b/relax/backends/megatron/actor.py @@ -6,6 +6,7 @@ import socket import time from argparse import Namespace +from contextlib import nullcontext from functools import partial from typing import Any, List @@ -64,6 +65,12 @@ from relax.utils.tracking_utils import init_tracking from relax.utils.training import train_dump_utils from relax.utils.training.data_fields import build_data_fields +from relax.utils.training.hybrid_forward_pipeline import ( + canonicalize_hybrid_microbatch_schedule, + execute_hybrid_forward_mini, + fetch_exact_chunk_with_timeout, +) +from relax.utils.training.hybrid_pipeline_trace import emit_hybrid_pipeline_event from relax.utils.training.routing_replay import RoutingReplay from relax.utils.types import RolloutBatch from relax.utils.utils import ( @@ -79,11 +86,19 @@ from ...utils.training.tensor_backper import TensorBackuper from .checkpoint import load_checkpoint from .collective_utils import _agree_drained -from .cp_utils import all_gather_with_cp, maybe_padded_total_lengths, slice_with_cp +from .cp_utils import ( + all_gather_with_cp, + get_logits_and_tokens_offset_with_cp, + maybe_padded_total_lengths, + slice_with_cp, +) from .data import ( ROLLOUT_MINI_LOCAL_SAMPLE_COUNTS_KEY, DataIterator, + RolloutMiniBatchPlan, + build_hybrid_forward_chunk_plan, build_rollout_minibatch_plan, + canonicalize_rollout_chunks, concat_rollout_batches, get_data_iterator, log_perf_data, @@ -131,14 +146,12 @@ def _slice(value): return value if isinstance(value, (list, tuple)) and len(value) == num_samples: return value[start:end] - ndim = getattr(value, "ndim", None) - size = getattr(value, "size", None) - if ndim and ndim > 0 and callable(size): - try: - if int(size(0)) == num_samples: - return value[start:end] - except (TypeError, ValueError, IndexError): - pass + shape = getattr(value, "shape", None) + try: + if shape is not None and len(shape) > 0 and int(shape[0]) == num_samples: + return value[start:end] + except (TypeError, ValueError, IndexError): + pass return value return {k: _slice(v) for k, v in rollout_data.items()} @@ -248,6 +261,7 @@ def _init( # internally via _switch_model and pushes weights to rollout via # UpdateWeightFromTensor instead of DCS. use_tensor_backuper = not self.args.fully_async or self.args.hybrid + self._hybrid_pipeline_chunk_plan = None if use_tensor_backuper: self.weights_backuper = TensorBackuper.create( source_getter=lambda: named_params_and_buffers( @@ -275,6 +289,14 @@ def _init( if args.update_weights_interval == 1: self.weights_backuper.backup("rollout_actor") + if getattr(self.args, "hybrid_pipeline_forward", False): + dp_size = mpu.get_data_parallel_world_size(with_context_parallel=False) + rollout_plan = build_rollout_minibatch_plan(self.args, dp_size) + self._hybrid_pipeline_chunk_plan = self._validate_hybrid_pipeline_runtime( + rollout_plan, + dp_size, + ) + update_weight_cls = UpdateWeightFromTensor if self.args.colocate else UpdateWeightFromDistributed # Push-side repack is decided by the HF config: an FP8 release auto-routes # through quantize_params_fp8, a compressed-tensors release through @@ -1078,13 +1100,31 @@ def compute_actor_log_prob(self, rollout_id: int) -> None: self.recv_weight_fully_async(rollout_id) log_perf_data_fwd(self.args, rollout_id) - def _hybrid_forward_subbatch(self, sub_batch: RolloutBatch) -> None: + def _hybrid_forward_subbatch( + self, + sub_batch: RolloutBatch, + *, + rollout_id: int, + chunk_index: int, + global_indexes: list[int] | None = None, + reuse_train_forward_log_probs: bool = False, + ) -> None: """Run the ref/teacher/actor forward passes for a single hybrid sub- batch in place. Shared by the streaming and debug_train_only paths so both compute identical log-probs before advantages are merged. """ + needs_ref_forward = "ref" in self.weights_backuper.backup_tags + needs_teacher_forward = "teacher" in self.weights_backuper.backup_tags + needs_actor_forward = not reuse_train_forward_log_probs and ( + not self.args.use_rollout_logprobs or self.args.get_mismatch_metrics + ) + if not self.args.compute_advantages_and_returns or not ( + needs_ref_forward or needs_teacher_forward or needs_actor_forward + ): + return + data_iterator, num_microbatches = get_data_iterator(self.args, self.model, sub_batch) # Separate iterator with larger token budget for ref/teacher log-probs (fallthrough mode). if self.args.use_dynamic_batch_size and self.args.log_probs_max_tokens_per_gpu != self.args.max_tokens_per_gpu: @@ -1102,7 +1142,7 @@ def _hybrid_forward_subbatch(self, sub_batch: RolloutBatch) -> None: if self.args.compute_advantages_and_returns: # Ref forward - if "ref" in self.weights_backuper.backup_tags: + if needs_ref_forward: if self.args.use_routing_replay: os.environ["ROUTING_REPLAY_STAGE"] = "fallthrough" self._switch_model("ref") @@ -1111,7 +1151,7 @@ def _hybrid_forward_subbatch(self, sub_batch: RolloutBatch) -> None: ) # Teacher forward for Megatron-based OPD - if "teacher" in self.weights_backuper.backup_tags: + if needs_teacher_forward: if self.args.use_routing_replay: os.environ["ROUTING_REPLAY_STAGE"] = "fallthrough" self._switch_model("teacher") @@ -1120,40 +1160,366 @@ def _hybrid_forward_subbatch(self, sub_batch: RolloutBatch) -> None: ) # Actor forward - self._switch_model("old_actor" if self.args.keep_old_actor else "actor") - if not self.args.use_rollout_logprobs or self.args.get_mismatch_metrics: + if needs_actor_forward: + target_tag = "old_actor" if self.args.keep_old_actor else "actor" + emit_hybrid_pipeline_event( + self.args, + "actor_restore_start", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + details={"target_tag": target_tag}, + ) + self._switch_model(target_tag) + emit_hybrid_pipeline_event( + self.args, + "actor_restore_end", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + details={"target_tag": target_tag}, + ) if self.args.use_routing_replay: if self.args.use_rollout_routing_replay: os.environ["ROUTING_REPLAY_STAGE"] = "replay_forward" else: os.environ["ROUTING_REPLAY_STAGE"] = "record" + emit_hybrid_pipeline_event( + self.args, + "actor_forward_start", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + ) sub_batch.update(self.compute_log_prob(data_iterator, num_microbatches, store_prefix="")) + emit_hybrid_pipeline_event( + self.args, + "actor_forward_end", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + ) if self.args.use_rollout_routing_replay: RoutingReplay.clear_all_forward() + def _should_reuse_hybrid_train_forward_log_probs(self, plan: RolloutMiniBatchPlan) -> bool: + """Return whether Hybrid can skip its dedicated old-policy forward.""" + if not getattr(self.args, "true_on_policy_mode", False): + return False + if plan.num_rollout_minis != 1: + raise ValueError( + "--true-on-policy-mode in hybrid requires exactly one optimizer mini per rollout partition, " + f"got {plan.num_rollout_minis}." + ) + if getattr(self.args, "keep_old_actor", False): + raise ValueError("--true-on-policy-mode in hybrid does not support --keep-old-actor.") + if getattr(self.args, "use_rollout_logprobs", False): + raise ValueError("--true-on-policy-mode and --use-rollout-logprobs select different old-policy sources.") + if getattr(self.args, "max_staleness", 0) > 0 and not getattr(self.args, "use_tis", False): + raise ValueError( + "--true-on-policy-mode in stale hybrid training requires --use-tis and aligned rollout log-probs." + ) + if getattr(self.args, "custom_megatron_before_log_prob_hook_path", None): + raise ValueError( + "--true-on-policy-mode in hybrid does not support a custom before-log-prob hook because " + "the train forward does not execute that hook." + ) + + dropout_fields = ("attention_dropout", "hidden_dropout", "lora_dropout") + nonzero_dropout = { + name: float(getattr(self.args, name, 0.0) or 0.0) + for name in dropout_fields + if float(getattr(self.args, name, 0.0) or 0.0) != 0.0 + } + if nonzero_dropout: + raise ValueError( + "--true-on-policy-mode in hybrid requires deterministic forwards; " + f"non-zero dropout: {nonzero_dropout}." + ) + + incompatible_features = [] + if self.args.kl_coef != 0: + incompatible_features.append("reward KL") + if getattr(self.args, "use_kl_loss", False): + incompatible_features.append("KL loss") + if getattr(self.args, "use_opd", False): + incompatible_features.append("OPD") + if getattr(self.args, "use_routing_replay", False) or getattr(self.args, "use_rollout_routing_replay", False): + incompatible_features.append("routing replay") + if incompatible_features: + raise ValueError( + "--true-on-policy-mode Hybrid fast path does not support: " + ", ".join(incompatible_features) + ) + return True + + def _validate_hybrid_rollout_log_probs(self, rollout_data: RolloutBatch) -> None: + """Validate rollout log-probs in their current context-parallel + layout.""" + rollout_log_probs = rollout_data.get("rollout_log_probs") + total_lengths = rollout_data.get("total_lengths") + response_lengths = rollout_data.get("response_lengths") + if ( + rollout_log_probs is None + or total_lengths is None + or response_lengths is None + or len(rollout_log_probs) != len(response_lengths) + or len(total_lengths) != len(response_lengths) + ): + raise RuntimeError( + "Hybrid train-forward log-prob reuse requires one rollout_log_probs entry per response." + ) + + cp_size = mpu.get_context_parallel_world_size() + if cp_size == 1 or getattr(self.args, "dynamic_context_parallel", False): + expected_lengths = response_lengths + else: + max_seq_lens = rollout_data.get("max_seq_lens") + padded_total_lengths = maybe_padded_total_lengths( + total_lengths, + self.args.qkv_format, + getattr(self.args, "is_vl_model", False) + or rollout_data.get("multimodal_train_inputs") is not None + or getattr(self.args, "uses_unsplit_forward", False), + ) + expected_lengths = [] + for index, (total_length, response_length) in enumerate(zip(total_lengths, response_lengths, strict=True)): + _, _, logits_offsets, _ = get_logits_and_tokens_offset_with_cp( + total_length, + response_length, + self.args.qkv_format, + max_seq_lens[index] if max_seq_lens is not None else None, + padded_total_length=(padded_total_lengths[index] if padded_total_lengths is not None else None), + ) + expected_lengths.append(sum(end - start for start, end in logits_offsets)) + + mismatches = [ + (index, None if log_probs is None else len(log_probs), expected_length) + for index, (log_probs, expected_length) in enumerate(zip(rollout_log_probs, expected_lengths, strict=True)) + if log_probs is None or len(log_probs) != expected_length + ] + if mismatches: + raise RuntimeError( + "Hybrid train-forward log-prob reuse requires CP-aligned rollout_log_probs; " + f"first mismatches (index, actual, expected): {mismatches[:3]}." + ) + + def _validate_hybrid_pipeline_runtime(self, rollout_plan, dp_size: int): + if dp_size != 1: + raise RuntimeError(f"--hybrid-pipeline-forward currently requires DP=1, detected DP={dp_size}") + if rollout_plan.num_rollout_minis != 1: + raise RuntimeError( + "--hybrid-pipeline-forward currently requires exactly one optimizer mini per rollout, " + f"detected {rollout_plan.num_rollout_minis}" + ) + pp_size = mpu.get_pipeline_model_parallel_world_size() + vpp_size = mpu.get_virtual_pipeline_model_parallel_world_size() or 1 + if pp_size != 1 or vpp_size != 1: + raise RuntimeError( + f"--hybrid-pipeline-forward currently requires PP=1 and VPP=1, detected PP={pp_size}, VPP={vpp_size}" + ) + tp_size = mpu.get_tensor_model_parallel_world_size() + cp_size = mpu.get_context_parallel_world_size() + ep_size = mpu.get_expert_model_parallel_world_size() + etp_size = int(getattr(self.args, "expert_tensor_parallel_size", 1) or 1) + if (tp_size, cp_size, ep_size, etp_size) != (2, 2, 1, 1): + raise RuntimeError( + "--hybrid-pipeline-forward currently requires TP=2, CP=2, EP=1, and ETP=1, " + f"detected TP={tp_size}, CP={cp_size}, EP={ep_size}, ETP={etp_size}" + ) + if self.args.offload_train or self.args.offload_rollout: + raise RuntimeError( + "--hybrid-pipeline-forward requires offload_train=False and offload_rollout=False, " + f"detected offload_train={self.args.offload_train}, " + f"offload_rollout={self.args.offload_rollout}" + ) + backup_tags = set(self.weights_backuper.backup_tags) + if backup_tags != {"actor"}: + raise RuntimeError( + f"--hybrid-pipeline-forward requires actor-only TensorBackuper tags, detected {sorted(backup_tags)}" + ) + if not self.args.compute_advantages_and_returns: + raise RuntimeError("--hybrid-pipeline-forward requires compute_advantages_and_returns=True") + if not self.args.use_dynamic_batch_size: + raise RuntimeError("--hybrid-pipeline-forward requires use_dynamic_batch_size=True") + return build_hybrid_forward_chunk_plan(self.args, rollout_plan, dp_size) + + def _restore_hybrid_pipeline_actor( + self, + *, + rollout_id: int, + chunk_index: int, + sample_count: int, + ) -> None: + emit_hybrid_pipeline_event( + self.args, + "actor_restore_start", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + sample_count=sample_count, + details={"target_tag": "actor"}, + ) + self._switch_model("actor") + emit_hybrid_pipeline_event( + self.args, + "actor_restore_end", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + sample_count=sample_count, + details={"target_tag": "actor"}, + ) + if self._active_model_tag != "actor": + raise RuntimeError( + "--hybrid-pipeline-forward actor restore completed with unexpected active tag " + f"{self._active_model_tag!r}" + ) + + def _hybrid_actor_forward_without_switch( + self, + sub_batch: RolloutBatch, + *, + rollout_id: int, + chunk_index: int, + global_indexes: list[int], + ) -> list[list[int]]: + if self._active_model_tag != "actor": + raise RuntimeError( + "--hybrid-pipeline-forward requires the actor model to remain active before " + f"chunk forward, detected {self._active_model_tag!r}" + ) + + data_iterator, num_microbatches = get_data_iterator(self.args, self.model, sub_batch) + if len(data_iterator) != 1 or len(num_microbatches) != 1: + raise RuntimeError( + "--hybrid-pipeline-forward expected one PP/VPP iterator and one optimizer step " + f"per chunk, got iterators={len(data_iterator)}, steps={num_microbatches}" + ) + microbatch_indices = data_iterator[0].micro_batch_indices + if microbatch_indices is None: + raise RuntimeError("--hybrid-pipeline-forward requires an explicit dynamic microbatch schedule") + forward_schedule = [list(indices) for indices in microbatch_indices] + flattened_schedule = [index for indices in forward_schedule for index in indices] + if len(forward_schedule) != num_microbatches[0] or sorted(flattened_schedule) != list( + range(len(sub_batch["total_lengths"])) + ): + raise RuntimeError( + "--hybrid-pipeline-forward produced an invalid chunk microbatch schedule: " + f"chunk_index={chunk_index}, microbatches={len(forward_schedule)}, " + f"expected_microbatches={num_microbatches[0]}, samples={len(flattened_schedule)}" + ) + emit_hybrid_pipeline_event( + self.args, + "actor_forward_start", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + ) + sub_batch.update(self.compute_log_prob(data_iterator, num_microbatches, store_prefix="")) + emit_hybrid_pipeline_event( + self.args, + "actor_forward_end", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + ) + if self._active_model_tag != "actor": + raise RuntimeError( + f"--hybrid-pipeline-forward actor tag changed during chunk forward: {self._active_model_tag!r}" + ) + return forward_schedule + + def _fetch_hybrid_pipeline_chunk( + self, + *, + rollout_id: int, + data_fields: list[str], + expected_samples: int, + batch_index: int, + mini_index: int, + ) -> tuple[RolloutBatch, list[int]]: + emit_hybrid_pipeline_event( + self.args, + "chunk_fetch_start", + rollout_id=rollout_id, + role="actor", + chunk_index=batch_index, + sample_count=expected_samples, + details={ + "batch_index": batch_index, + "mini_index": mini_index, + "expected_samples": expected_samples, + }, + ) + + def fetch_once(): + with timer("train_get_data"): + return self._get_data_from_transfer_queue( + "train", + rollout_id, + data_fields, + expected_samples, + batch_index, + ) + + error_context = ( + "--hybrid-pipeline-forward TransferQueue chunk failure: " + f"rollout_id={rollout_id}, mini_index={mini_index}, chunk_index={batch_index}" + ) + sub_batch, batch_meta, elapsed = fetch_exact_chunk_with_timeout( + fetch_once=fetch_once, + expected_samples=expected_samples, + timeout_s=self.args.hybrid_pipeline_fetch_timeout_s, + error_context=error_context, + ) + actual_samples = len(sub_batch["total_lengths"]) + global_indexes = list(getattr(batch_meta, "global_indexes", [])) + if len(global_indexes) != actual_samples: + raise RuntimeError( + "--hybrid-pipeline-forward received invalid BatchMeta.global_indexes: " + f"rollout_id={rollout_id}, mini_index={mini_index}, " + f"chunk_index={batch_index}, samples={actual_samples}, " + f"indexes={len(global_indexes)}" + ) + emit_hybrid_pipeline_event( + self.args, + "chunk_fetch_end", + rollout_id=rollout_id, + role="actor", + chunk_index=batch_index, + batch=sub_batch, + global_indexes=global_indexes, + details={ + "batch_index": batch_index, + "mini_index": mini_index, + "expected_samples": expected_samples, + "elapsed_s": elapsed, + }, + ) + return sub_batch, global_indexes + @staticmethod def _split_rollout_batch(rollout_data: RolloutBatch, num_chunks: int) -> List[RolloutBatch]: - """Split a merged rollout batch (dict of per-sample lists) into at most - ``num_chunks`` roughly equal sub-batches along the sample dimension. - - Keys whose value is not a per-sample list are copied into every chunk - unchanged. Used by the debug_train_only path to feed the collected- - sub-batch forward loop (one global batch per chunk). - """ + """Split a debug rollout into roughly equal per-sample chunks.""" num_samples = len(rollout_data["tokens"]) num_chunks = max(1, min(num_chunks, num_samples)) chunk_size = (num_samples + num_chunks - 1) // num_chunks - chunks: List[RolloutBatch] = [] - for start in range(0, num_samples, chunk_size): - end = min(start + chunk_size, num_samples) - chunk: RolloutBatch = {} - for key, value in rollout_data.items(): - if isinstance(value, list) and len(value) == num_samples: - chunk[key] = value[start:end] - else: - chunk[key] = value - chunks.append(chunk) - return chunks + return [ + _slice_rollout_batch(rollout_data, start, min(start + chunk_size, num_samples)) + for start in range(0, num_samples, chunk_size) + ] def _use_streaming_fwd(self) -> bool: """Whether ref / actor_fwd forward should stream via token-budget @@ -1243,9 +1609,61 @@ def train_hybrid(self, rollout_id) -> None: plan = build_rollout_minibatch_plan(self.args, dp_size) batch_size = plan.mini_local_sample_request + pipeline_enabled = bool(getattr(self.args, "hybrid_pipeline_forward", False)) + reuse_train_forward_log_probs = self._should_reuse_hybrid_train_forward_log_probs(plan) + if reuse_train_forward_log_probs: + logger.info( + "train_hybrid(%s) reuses detached log-probs from the deterministic train forward; " + "skipping the redundant actor log-prob forward", + rollout_id, + ) + trace_enabled = bool(getattr(self.args, "hybrid_pipeline_trace_dir", None)) + chunk_plan = self._hybrid_pipeline_chunk_plan if pipeline_enabled else None + if pipeline_enabled and chunk_plan is None: + raise RuntimeError( + "--hybrid-pipeline-forward was enabled without an initialized chunk plan; " + "the actor startup compatibility checks did not complete." + ) + + def phase1_timer(): + return timer("hybrid_phase1") if trace_enabled else nullcontext() + + data_fields = [ + "tokens", + "total_lengths", + "response_lengths", + "loss_masks", + "rollout_log_probs", + "rewards", + "raw_reward", + ] + if trace_enabled: + data_fields.append("truncated") + data_fields += ["rollout_routed_experts"] if self.args.use_rollout_routing_replay else [] + if self.args.multimodal_keys is not None: + data_fields.append("multimodal_train_inputs") + if self.args.use_opd and self.args.opd_type == "sglang": + data_fields.append("teacher_log_probs") + # ── Phase 1: Collect sub-batches and compute ref/actor forward in small chunks ── collected_batches: list[RolloutBatch] = [] + collected_global_indexes: list[int] = [] rollout_mini_local_sample_counts: list[int] = [] + pipeline_chunk_schedules: list[tuple[list[int], list[list[int]]]] = [] + + def forward_pipeline_chunk( + sub_batch: RolloutBatch, + tq_batch_index: int, + global_indexes: list[int], + ) -> None: + forward_schedule = self._hybrid_actor_forward_without_switch( + sub_batch, + rollout_id=rollout_id, + chunk_index=tq_batch_index, + global_indexes=global_indexes, + ) + pipeline_chunk_schedules.append((list(global_indexes), forward_schedule)) + if self.args.debug_train_only: # Bypass the transfer queue and load the offline debug rollout dump # directly (mirrors `train`'s debug_train_only path). The dump holds @@ -1256,72 +1674,183 @@ def train_hybrid(self, rollout_id) -> None: full_batch_size = plan.mini_local_sample_request * plan.num_rollout_minis debug_data = get_debug_data(self.args, rollout_id, full_batch_size, dp_rank=mpu.get_data_parallel_rank()) post_process_rollout_data(self.args, debug_data) - for sub_batch in self._split_rollout_batch(debug_data, plan.num_rollout_minis): - if len(sub_batch["total_lengths"]) != batch_size: - raise RuntimeError( - f"debug rollout mini batch local size mismatch for train_hybrid({rollout_id}): " - f"expected {batch_size}, got {len(sub_batch['total_lengths'])}." + if pipeline_enabled: + for mini_index in range(plan.num_rollout_minis): + + def fetch_debug_chunk(tq_batch_index): + start = tq_batch_index * chunk_plan.chunk_local_samples + end = start + chunk_plan.chunk_local_samples + return _slice_rollout_batch(debug_data, start, end), list(range(start, end)) + + with phase1_timer(): + mini_chunks = execute_hybrid_forward_mini( + chunks_per_mini=chunk_plan.chunks_per_mini, + batch_index_for_chunk=lambda chunk_index: chunk_plan.transfer_queue_batch_index( + mini_index, chunk_index + ), + restore_actor=lambda tq_batch_index: self._restore_hybrid_pipeline_actor( + rollout_id=rollout_id, + chunk_index=tq_batch_index, + sample_count=batch_size, + ), + fetch_chunk=fetch_debug_chunk, + forward_chunk=forward_pipeline_chunk, + overlap_producer=bool(getattr(self.args, "hybrid_pipeline_overlap", True)), + ) + mini_batch, canonical_indexes = canonicalize_rollout_chunks( + mini_chunks, + expected_sample_count=batch_size, ) - self._hybrid_forward_subbatch(sub_batch) - collected_batches.append(sub_batch) - rollout_mini_local_sample_counts.append(len(sub_batch["total_lengths"])) + collected_batches.append(mini_batch) + collected_global_indexes.extend(canonical_indexes) + rollout_mini_local_sample_counts.append(batch_size) + else: + debug_batches = self._split_rollout_batch(debug_data, plan.num_rollout_minis) + for mini_index, sub_batch in enumerate(debug_batches): + if len(sub_batch["total_lengths"]) != batch_size: + raise RuntimeError( + f"debug rollout mini batch local size mismatch for train_hybrid({rollout_id}): " + f"expected {batch_size}, got {len(sub_batch['total_lengths'])}." + ) + start = mini_index * batch_size + global_indexes = list(range(start, start + batch_size)) + with phase1_timer(): + self._hybrid_forward_subbatch( + sub_batch, + rollout_id=rollout_id, + chunk_index=mini_index, + global_indexes=global_indexes, + reuse_train_forward_log_probs=reuse_train_forward_log_probs, + ) + collected_batches.append(sub_batch) + collected_global_indexes.extend(global_indexes) + rollout_mini_local_sample_counts.append(len(sub_batch["total_lengths"])) else: - batch_index = 0 - # Surface stuck-loop conditions: when the partition can never reach the - # requested batch_size (e.g. rollout dropped samples without refilling), - # `get_meta` keeps returning size=0 while `all_consumed` stays False, - # producing a silent infinite spin. Warn periodically so the failure mode - # is visible in logs instead of presenting as a totally silent hang. - loop_start = time.monotonic() - last_progress = loop_start - last_warn = loop_start - while batch_index < plan.num_rollout_minis and not self.all_consumed("train", rollout_id): - data_fields = [ - "tokens", - "total_lengths", - "response_lengths", - "loss_masks", - "rollout_log_probs", - "rewards", - "raw_reward", - ] - data_fields += ["rollout_routed_experts"] if self.args.use_rollout_routing_replay else [] - if self.args.multimodal_keys is not None: - data_fields.append("multimodal_train_inputs") - if self.args.use_opd and self.args.opd_type == "sglang": - data_fields.append("teacher_log_probs") - with timer("train_get_data"): - sub_batch, batch_meta = self._get_data_from_transfer_queue( - "train", rollout_id, data_fields, batch_size, batch_index - ) - if sub_batch is None: - now = time.monotonic() - stalled = now - last_progress - if now - last_warn >= 60.0 and stalled >= 60.0: - logger.warning( - f"train_hybrid({rollout_id}) batch_index={batch_index} stalled for {stalled:.0f}s: " - f"partition train_{rollout_id} has no data of size={batch_size} available but " - f"all_consumed=False. Likely the rollout under-filled this partition." + if pipeline_enabled: + seen_global_indexes: set[int] = set() + for mini_index in range(plan.num_rollout_minis): + + def fetch_pipeline_chunk(tq_batch_index): + sub_batch, global_indexes = self._fetch_hybrid_pipeline_chunk( + rollout_id=rollout_id, + data_fields=data_fields, + expected_samples=chunk_plan.chunk_local_samples, + batch_index=tq_batch_index, + mini_index=mini_index, + ) + overlap = seen_global_indexes.intersection(global_indexes) + if overlap: + raise RuntimeError( + "--hybrid-pipeline-forward received duplicate global indexes " + f"across optimizer minis: {sorted(overlap)}" + ) + seen_global_indexes.update(global_indexes) + return sub_batch, global_indexes + + with phase1_timer(): + mini_chunks = execute_hybrid_forward_mini( + chunks_per_mini=chunk_plan.chunks_per_mini, + batch_index_for_chunk=lambda chunk_index: chunk_plan.transfer_queue_batch_index( + mini_index, chunk_index + ), + restore_actor=lambda tq_batch_index: self._restore_hybrid_pipeline_actor( + rollout_id=rollout_id, + chunk_index=tq_batch_index, + sample_count=batch_size, + ), + fetch_chunk=fetch_pipeline_chunk, + forward_chunk=forward_pipeline_chunk, + overlap_producer=bool(getattr(self.args, "hybrid_pipeline_overlap", True)), ) - last_warn = now - # Throttle the spin so the controller is not hammered with metadata - # polls while we wait for upstream data. - time.sleep(0.1) - continue - last_progress = time.monotonic() - last_warn = last_progress - batch_index += 1 - # Forward passes on this sub-batch (small memory footprint) - if len(sub_batch["total_lengths"]) != batch_size: - raise RuntimeError( - f"rollout mini batch local size mismatch for train_hybrid({rollout_id}), " - f"batch_index={batch_index - 1}: expected {batch_size}, " - f"got {len(sub_batch['total_lengths'])}." + mini_batch, canonical_indexes = canonicalize_rollout_chunks( + mini_chunks, + expected_sample_count=batch_size, ) - self._hybrid_forward_subbatch(sub_batch) - collected_batches.append(sub_batch) - rollout_mini_local_sample_counts.append(len(sub_batch["total_lengths"])) + collected_batches.append(mini_batch) + collected_global_indexes.extend(canonical_indexes) + rollout_mini_local_sample_counts.append(batch_size) + else: + # Preserve the baseline sample-count request and unbounded retry + # behavior when the optimization switch is disabled. + for mini_index in range(plan.num_rollout_minis): + loop_start = time.monotonic() + last_progress = loop_start + last_warn = loop_start + emit_hybrid_pipeline_event( + self.args, + "chunk_fetch_start", + rollout_id=rollout_id, + role="actor", + chunk_index=mini_index, + sample_count=batch_size, + details={ + "batch_index": mini_index, + "mini_index": mini_index, + "expected_samples": batch_size, + }, + ) + with phase1_timer(): + while True: + if self.all_consumed("train", rollout_id): + raise RuntimeError( + f"TransferQueue was consumed before train_hybrid({rollout_id}) " + f"received rollout mini {mini_index}." + ) + with timer("train_get_data"): + sub_batch, batch_meta = self._get_data_from_transfer_queue( + "train", + rollout_id, + data_fields, + batch_size, + mini_index, + ) + if sub_batch is not None: + break + now = time.monotonic() + stalled = now - last_progress + if now - last_warn >= 60.0 and stalled >= 60.0: + logger.warning( + f"train_hybrid({rollout_id}) batch_index={mini_index} " + f"stalled for {stalled:.0f}s: partition train_{rollout_id} " + f"has no data of size={batch_size} available but " + "all_consumed=False. Likely the rollout under-filled this partition." + ) + last_warn = now + time.sleep(0.1) + + actual_samples = len(sub_batch["total_lengths"]) + if actual_samples != batch_size: + raise RuntimeError( + f"rollout mini batch local size mismatch for train_hybrid({rollout_id}), " + f"batch_index={mini_index}: expected {batch_size}, got {actual_samples}." + ) + global_indexes = list(getattr(batch_meta, "global_indexes", [])) + emit_hybrid_pipeline_event( + self.args, + "chunk_fetch_end", + rollout_id=rollout_id, + role="actor", + chunk_index=mini_index, + batch=sub_batch, + global_indexes=global_indexes, + details={ + "batch_index": mini_index, + "mini_index": mini_index, + "expected_samples": batch_size, + "elapsed_s": time.monotonic() - loop_start, + }, + ) + self._hybrid_forward_subbatch( + sub_batch, + rollout_id=rollout_id, + chunk_index=mini_index, + global_indexes=global_indexes, + reuse_train_forward_log_probs=reuse_train_forward_log_probs, + ) + collected_batches.append(sub_batch) + collected_global_indexes.extend(global_indexes) + rollout_mini_local_sample_counts.append(actual_samples) if len(collected_batches) != plan.num_rollout_minis: raise RuntimeError( @@ -1333,21 +1862,47 @@ def train_hybrid(self, rollout_id) -> None: self._switch_model("actor") # ── Phase 2: Merge sub-batches and compute advantages with correct global normalization ── - # Merge all sub-batch dicts: each value is a list, so we concatenate them. - rollout_data: RolloutBatch = {} - for sb in collected_batches: - for key, value in sb.items(): - if key not in rollout_data: - rollout_data[key] = [] - if isinstance(value, (list, tuple)) and not isinstance(value, (str, bytes)): - rollout_data[key].extend(value) - else: - rollout_data[key].append(value) + if pipeline_enabled: + rollout_data = concat_rollout_batches(collected_batches) + pipeline_train_microbatch_indices = canonicalize_hybrid_microbatch_schedule( + pipeline_chunk_schedules, + collected_global_indexes, + ) + else: + # Keep the flag-off merge path byte-for-byte compatible with the + # pre-optimization implementation. + rollout_data: RolloutBatch = {} + for sb in collected_batches: + for key, value in sb.items(): + if key not in rollout_data: + rollout_data[key] = [] + if isinstance(value, (list, tuple)) and not isinstance(value, (str, bytes)): + rollout_data[key].extend(value) + else: + rollout_data[key].append(value) rollout_data[ROLLOUT_MINI_LOCAL_SAMPLE_COUNTS_KEY] = rollout_mini_local_sample_counts + if reuse_train_forward_log_probs: + self._validate_hybrid_rollout_log_probs(rollout_data) with inverse_timer("train_wait"), timer("train"): if self.args.compute_advantages_and_returns: + emit_hybrid_pipeline_event( + self.args, + "advantages_start", + rollout_id=rollout_id, + role="actor", + batch=rollout_data, + global_indexes=collected_global_indexes, + ) compute_advantages_and_returns(self.args, rollout_data) + emit_hybrid_pipeline_event( + self.args, + "advantages_end", + rollout_id=rollout_id, + role="actor", + batch=rollout_data, + global_indexes=collected_global_indexes, + ) if self.rollout_data_postprocess is not None: self.rollout_data_postprocess(self.args) @@ -1355,10 +1910,29 @@ def train_hybrid(self, rollout_id) -> None: log_rollout_data(rollout_id, self.args, rollout_data) # ── Phase 3: Train on the full merged batch ── - data_iterator, num_microbatches = get_data_iterator(self.args, self.model, rollout_data) + if pipeline_enabled: + data_iterator = [ + DataIterator( + rollout_data, + micro_batch_indices=pipeline_train_microbatch_indices, + max_tokens_per_gpu=self.args.max_tokens_per_gpu, + ) + ] + num_microbatches = [len(pipeline_train_microbatch_indices)] + else: + data_iterator, num_microbatches = get_data_iterator(self.args, self.model, rollout_data) if self.args.use_routing_replay: os.environ["ROUTING_REPLAY_STAGE"] = "replay_backward" with timer("actor_train"): + emit_hybrid_pipeline_event( + self.args, + "optimizer_start", + rollout_id=rollout_id, + role="actor", + batch=rollout_data, + global_indexes=collected_global_indexes, + details={"scope": "actor_train"}, + ) train( rollout_id, self.model, @@ -1367,6 +1941,15 @@ def train_hybrid(self, rollout_id) -> None: data_iterator, num_microbatches, ) + emit_hybrid_pipeline_event( + self.args, + "optimizer_end", + rollout_id=rollout_id, + role="actor", + batch=rollout_data, + global_indexes=collected_global_indexes, + details={"scope": "actor_train"}, + ) self.prof.step(rollout_id=rollout_id) @@ -1398,6 +1981,18 @@ def train_hybrid(self, rollout_id) -> None: ) all_total_lengths = sum(all_total_lengths, []) # flatten Timer().seq_lens = all_total_lengths + if trace_enabled: + response_token_counts = [ + int(mask.sum().item()) if isinstance(mask, torch.Tensor) else int(sum(mask)) + for mask in rollout_data["loss_masks"] + ] + all_response_token_counts = [None] * mpu.get_data_parallel_world_size(with_context_parallel=False) + dist.all_gather_object( + all_response_token_counts, + response_token_counts, + group=mpu.get_data_parallel_group(with_context_parallel=False), + ) + Timer().response_lens = sum(all_response_token_counts, []) mm_inputs = rollout_data.get("multimodal_train_inputs") if mm_inputs is not None: images_seqlens = _extract_images_seqlens(mm_inputs) diff --git a/relax/backends/megatron/data.py b/relax/backends/megatron/data.py index 9aff3da6f..b300fdd76 100644 --- a/relax/backends/megatron/data.py +++ b/relax/backends/megatron/data.py @@ -1,6 +1,7 @@ # Copyright (c) 2026 Relax Authors. All Rights Reserved. from argparse import Namespace +from collections import Counter from collections.abc import Sequence from copy import deepcopy from dataclasses import dataclass @@ -52,6 +53,20 @@ class RolloutMiniBatchPlan: mini_local_sample_request: int | None +@dataclass(frozen=True) +class HybridForwardChunkPlan: + chunks_per_mini: int + chunk_global_samples: int + chunk_local_samples: int + + def transfer_queue_batch_index(self, mini_index: int, chunk_index: int) -> int: + if mini_index < 0: + raise ValueError(f"mini_index must be non-negative, got {mini_index}") + if not 0 <= chunk_index < self.chunks_per_mini: + raise ValueError(f"chunk_index must be in [0, {self.chunks_per_mini}), got {chunk_index}") + return mini_index * self.chunks_per_mini + chunk_index + + def build_rollout_minibatch_plan(args: Namespace, dp_size: int) -> RolloutMiniBatchPlan: """Build a prompt-group based mini plan for one rollout partition.""" if dp_size <= 0: @@ -119,9 +134,63 @@ def build_rollout_minibatch_plan(args: Namespace, dp_size: int) -> RolloutMiniBa ) +def build_hybrid_forward_chunk_plan( + args: Namespace, + rollout_plan: RolloutMiniBatchPlan, + dp_size: int, +) -> HybridForwardChunkPlan: + """Split one optimizer mini into fixed actor sample-count chunks.""" + if dp_size <= 0: + raise ValueError(f"dp_size must be positive, got {dp_size}") + + chunks_per_mini = int(args.num_iters_per_train_update) + if chunks_per_mini < 2: + raise ValueError(f"hybrid pipeline forward requires num_iters_per_train_update >= 2, got {chunks_per_mini}") + + mini_global_samples = rollout_plan.mini_global_samples + mini_local_samples = rollout_plan.mini_local_sample_request + n_samples_per_prompt = rollout_plan.fixed_n_samples_per_prompt + if mini_global_samples is None or mini_local_samples is None or n_samples_per_prompt is None: + raise ValueError("hybrid pipeline forward requires a fixed-size rollout mini plan") + if mini_global_samples % chunks_per_mini != 0: + raise ValueError( + "mini_global_samples must be divisible by num_iters_per_train_update, " + f"got mini_global_samples={mini_global_samples}, chunks={chunks_per_mini}" + ) + + chunk_global_samples = mini_global_samples // chunks_per_mini + if chunk_global_samples % dp_size != 0: + raise ValueError( + "chunk_global_samples must be divisible by data parallel size, " + f"got chunk_global_samples={chunk_global_samples}, dp_size={dp_size}" + ) + if chunk_global_samples % n_samples_per_prompt != 0: + raise ValueError( + "hybrid pipeline forward chunks must preserve complete prompt groups, " + f"got chunk_global_samples={chunk_global_samples}, " + f"n_samples_per_prompt={n_samples_per_prompt}" + ) + + chunk_local_samples = chunk_global_samples // dp_size + if chunk_local_samples * chunks_per_mini != mini_local_samples: + raise ValueError( + "hybrid pipeline chunk plan does not reconstruct the optimizer mini, " + f"chunk_local_samples={chunk_local_samples}, chunks={chunks_per_mini}, " + f"mini_local_samples={mini_local_samples}" + ) + + return HybridForwardChunkPlan( + chunks_per_mini=chunks_per_mini, + chunk_global_samples=chunk_global_samples, + chunk_local_samples=chunk_local_samples, + ) + + def _same_scalar_value(lhs: Any, rhs: Any) -> bool: if isinstance(lhs, torch.Tensor) and isinstance(rhs, torch.Tensor): return torch.equal(lhs, rhs) + if isinstance(lhs, np.ndarray) and isinstance(rhs, np.ndarray): + return np.array_equal(lhs, rhs) return lhs == rhs @@ -130,24 +199,37 @@ def concat_rollout_batches(rollout_batches: Sequence[RolloutBatch]) -> RolloutBa if not rollout_batches: raise ValueError("rollout_batches must not be empty") + expected_keys = set(rollout_batches[0]) merged: RolloutBatch = {} tensor_batches: dict[str, list[torch.Tensor]] = {} + array_batches: dict[str, list[np.ndarray]] = {} scalar_values: dict[str, Any] = {} - for batch in rollout_batches: + for batch_index, batch in enumerate(rollout_batches): if batch is None: raise ValueError("rollout_batches must not contain None") + if set(batch) != expected_keys: + missing = sorted(expected_keys - set(batch)) + extra = sorted(set(batch) - expected_keys) + raise ValueError(f"rollout batch {batch_index} schema mismatch: missing={missing}, extra={extra}") batch_size = len(batch.get("total_lengths", [])) if batch_size <= 0: raise ValueError("rollout mini batch must contain at least one sample") for key, value in batch.items(): if isinstance(value, (list, tuple)) and not isinstance(value, (str, bytes)): + if len(value) != batch_size: + raise ValueError( + f"Per-sample rollout field {key!r} in batch {batch_index} has " + f"length={len(value)}, expected={batch_size}" + ) if key not in merged: merged[key] = [] merged[key].extend(value) elif isinstance(value, torch.Tensor) and value.ndim > 0 and value.size(0) == batch_size: tensor_batches.setdefault(key, []).append(value) + elif isinstance(value, np.ndarray) and value.ndim > 0 and value.shape[0] == batch_size: + array_batches.setdefault(key, []).append(value) else: if key in scalar_values: if not _same_scalar_value(scalar_values[key], value): @@ -160,6 +242,11 @@ def concat_rollout_batches(rollout_batches: Sequence[RolloutBatch]) -> RolloutBa raise ValueError(f"Rollout field {key!r} appears as both list-like and tensor-like") merged[key] = torch.cat(tensors, dim=0) + for key, arrays in array_batches.items(): + if key in merged or key in tensor_batches: + raise ValueError(f"Rollout field {key!r} has inconsistent batched types") + merged[key] = np.concatenate(arrays, axis=0) + for key, value in scalar_values.items(): if key in merged: raise ValueError(f"Rollout field {key!r} appears as both scalar and batched data") @@ -168,6 +255,55 @@ def concat_rollout_batches(rollout_batches: Sequence[RolloutBatch]) -> RolloutBa return merged +def canonicalize_rollout_chunks( + chunks_with_global_indexes: Sequence[tuple[RolloutBatch, Sequence[int]]], + expected_sample_count: int, +) -> tuple[RolloutBatch, list[int]]: + """Merge chunks and restore deterministic TransferQueue global-index + order.""" + if expected_sample_count <= 0: + raise ValueError(f"expected_sample_count must be positive, got {expected_sample_count}") + if not chunks_with_global_indexes: + raise ValueError("chunks_with_global_indexes must not be empty") + + chunks: list[RolloutBatch] = [] + global_indexes: list[int] = [] + for chunk_index, (chunk, indexes) in enumerate(chunks_with_global_indexes): + batch_size = len(chunk.get("total_lengths", [])) + normalized_indexes = list(indexes) + if len(normalized_indexes) != batch_size: + raise ValueError( + f"chunk {chunk_index} BatchMeta.global_indexes length mismatch: " + f"indexes={len(normalized_indexes)}, samples={batch_size}" + ) + if not all(type(index) is int for index in normalized_indexes): + raise TypeError(f"chunk {chunk_index} BatchMeta.global_indexes must contain only int values") + chunks.append(chunk) + global_indexes.extend(normalized_indexes) + + if len(global_indexes) != expected_sample_count: + raise ValueError( + f"hybrid pipeline sample count mismatch: expected={expected_sample_count}, actual={len(global_indexes)}" + ) + if len(set(global_indexes)) != len(global_indexes): + duplicates = sorted(index for index, count in Counter(global_indexes).items() if count > 1) + raise ValueError(f"hybrid pipeline BatchMeta.global_indexes contain duplicates: {duplicates}") + + merged = concat_rollout_batches(chunks) + permutation = sorted(range(len(global_indexes)), key=global_indexes.__getitem__) + canonical_indexes = [global_indexes[index] for index in permutation] + + for key, value in list(merged.items()): + if isinstance(value, list) and len(value) == expected_sample_count: + merged[key] = [value[index] for index in permutation] + elif isinstance(value, torch.Tensor) and value.ndim > 0 and value.size(0) == expected_sample_count: + merged[key] = value[permutation] + elif isinstance(value, np.ndarray) and value.ndim > 0 and value.shape[0] == expected_sample_count: + merged[key] = value[permutation] + + return merged, canonical_indexes + + PAD_RULES = { # shape like [1, 128, 1036] "input_features": dict( diff --git a/relax/backends/megatron/loss.py b/relax/backends/megatron/loss.py index 170265a24..2fe77caf9 100644 --- a/relax/backends/megatron/loss.py +++ b/relax/backends/megatron/loss.py @@ -542,13 +542,18 @@ def compute_advantages_and_returns(args: Namespace, rollout_data: RolloutBatch) "total_lengths"). Modified in-place to add "advantages" and "returns" keys, each mapping to lists of tensors per sample. """ - log_probs: list[torch.Tensor] = rollout_data.get("rollout_log_probs" if args.use_rollout_logprobs else "log_probs") - ref_log_probs: list[torch.Tensor] = rollout_data.get("ref_log_probs") + log_probs: list[torch.Tensor] | None = rollout_data.get( + "rollout_log_probs" if args.use_rollout_logprobs else "log_probs" + ) + ref_log_probs: list[torch.Tensor] | None = rollout_data.get("ref_log_probs") rewards: list[float] = rollout_data.get("rewards") values: None | list[torch.Tensor] = rollout_data.get("values") response_lengths: list[int] = rollout_data.get("response_lengths") loss_masks: list[torch.Tensor] = rollout_data.get("loss_masks") total_lengths: list[int] = rollout_data.get("total_lengths") + rollout_log_probs: list[torch.Tensor] | None = ( + rollout_data.get("rollout_log_probs") if getattr(args, "true_on_policy_mode", False) else None + ) max_seq_lens: list[int] | None = rollout_data.get("max_seq_lens", None) padded_total_lengths: list[int] | None = maybe_padded_total_lengths( total_lengths, @@ -562,11 +567,25 @@ def compute_advantages_and_returns(args: Namespace, rollout_data: RolloutBatch) if not mpu.is_pipeline_last_stage(): return - if args.kl_coef == 0 or not log_probs: + if args.kl_coef == 0: # when kl_coef is 0, we won't compute ref_log_prob - xs = log_probs if log_probs is not None else values + xs = log_probs or values or rollout_log_probs + if not xs: + raise RuntimeError( + "Advantage computation on the pipeline last stage requires log_probs, values, or " + "true-on-policy rollout_log_probs." + ) kl = [torch.zeros_like(x, dtype=torch.float32, device=x.device) for x in xs] else: + if not log_probs: + raise RuntimeError("Reward KL requires actor log_probs; rollout_log_probs cannot replace them.") + if not ref_log_probs: + raise RuntimeError("Reward KL requires ref_log_probs.") + if len(log_probs) != len(ref_log_probs): + raise RuntimeError( + "Reward KL requires one ref_log_probs entry per actor log_probs entry, " + f"got {len(ref_log_probs)} and {len(log_probs)}." + ) kl = [ compute_approx_kl( log_probs[i], diff --git a/relax/utils/arguments.py b/relax/utils/arguments.py index f7d1e710e..e14054ef5 100644 --- a/relax/utils/arguments.py +++ b/relax/utils/arguments.py @@ -2,6 +2,7 @@ import argparse import json +import math import os import sys import warnings @@ -75,6 +76,54 @@ def _positive_int(value: str) -> int: return parsed +def check_hybrid_pipeline_transfer_queue_contract() -> None: + """Validate the TransferQueue API needed by chunked Hybrid forwarding + before any rollout can write data.""" + import inspect + + check_transfer_queue_version() + try: + from transfer_queue import BatchMeta, TransferQueueClient + except (ImportError, AttributeError) as exc: + raise RuntimeError( + "transferqueue does not expose BatchMeta and TransferQueueClient required by " + "Hybrid pipeline forwarding/tracing. Upgrade with:\n" + f" {_TQ_UPGRADE_CMD}\nor use the latest image." + ) from exc + + slots = set(getattr(BatchMeta, "__slots__", ())) + annotations = set(getattr(BatchMeta, "__annotations__", {})) + if "global_indexes" not in slots | annotations and not hasattr(BatchMeta, "global_indexes"): + raise RuntimeError( + "transferqueue BatchMeta does not expose global_indexes required by " + "Hybrid pipeline forwarding/tracing. Upgrade with:\n" + f" {_TQ_UPGRADE_CMD}\nor use the latest image." + ) + + try: + async_put_parameters = inspect.signature(TransferQueueClient.async_put).parameters + except (TypeError, ValueError, AttributeError) as exc: + raise RuntimeError( + "transferqueue TransferQueueClient.async_put cannot be inspected for " + "Hybrid pipeline forwarding/tracing compatibility." + ) from exc + missing = sorted({"custom_meta", "is_last"} - set(async_put_parameters)) + if missing: + raise RuntimeError( + "transferqueue TransferQueueClient.async_put is missing parameters " + f"{missing} required by Hybrid pipeline forwarding/tracing. Upgrade with:\n" + f" {_TQ_UPGRADE_CMD}\nor use the latest image." + ) + + +def check_transfer_queue_runtime(args) -> None: + """Select the startup contract required by the resolved execution mode.""" + if getattr(args, "hybrid_pipeline_forward", False) or getattr(args, "hybrid_pipeline_trace_dir", None): + check_hybrid_pipeline_transfer_queue_contract() + elif getattr(args, "fully_async", False): + check_transfer_queue_version() + + def reset_arg(parser, name, **kwargs): """Reset the default value of a Megatron argument. @@ -178,6 +227,45 @@ def add_serve_arguments(parser): "Mutually exclusive with passing --fully-async and --colocate together." ), ) + parser.add_argument( + "--hybrid-pipeline-forward", + action="store_true", + default=False, + help=( + "In the supported Hybrid actor-only topology, split one optimizer mini " + "into num-iters-per-train-update fixed sample-count actor chunks and " + "overlap actor log-prob forward with later rollout production. " + "Default: disabled." + ), + ) + parser.add_argument( + "--hybrid-pipeline-overlap", + action=argparse.BooleanOptionalAction, + default=True, + help=( + "When Hybrid pipeline forwarding is enabled, forward each chunk immediately " + "instead of waiting until all chunks have been fetched. Disable only to run a " + "schedule-matched no-overlap performance control. Default: enabled." + ), + ) + parser.add_argument( + "--hybrid-pipeline-trace-dir", + type=str, + default=None, + help=( + "Write per-process JSONL events for Hybrid producer puts, actor fetch, " + "restore, forward, advantages, and optimizer phases. No sample content is written." + ), + ) + parser.add_argument( + "--hybrid-pipeline-fetch-timeout-s", + type=float, + default=600.0, + help=( + "Maximum seconds the Hybrid pipeline path waits for one exact " + "TransferQueue chunk before failing with rollout/mini/chunk context." + ), + ) parser.add_argument( "--checkpoint-engine-backend", type=str, @@ -2735,10 +2823,10 @@ def _parse_args_impl(add_custom_arguments=None, *, provider_source=None): if not args.debug_train_only: sglang_validate_args(args) - # Only fully-async mode relies on the newer TransferQueue (e.g. - # StreamingTokenBudgetSampler), so gate the version requirement on it. - if getattr(args, "fully_async", False): - check_transfer_queue_version() + # Chunked Hybrid forwarding additionally depends on BatchMeta global + # indexes and the atomic custom-meta async_put contract. Validate that + # API before Ray workers or rollout producers can create side effects. + check_transfer_queue_runtime(args) return args @@ -2963,6 +3051,95 @@ def _normalize_sync_ppo_kl_args(args) -> bool: return True +def _validate_hybrid_pipeline_args(args) -> None: + enabled = bool(getattr(args, "hybrid_pipeline_forward", False)) + overlap_enabled = bool(getattr(args, "hybrid_pipeline_overlap", True)) + trace_enabled = bool(getattr(args, "hybrid_pipeline_trace_dir", None)) + if (enabled or trace_enabled or not overlap_enabled) and not getattr(args, "hybrid", False): + raise ValueError( + "--hybrid-pipeline-forward, its overlap control, and --hybrid-pipeline-trace-dir " + "are only supported with --hybrid." + ) + if not enabled: + if not overlap_enabled: + raise ValueError("--no-hybrid-pipeline-overlap requires --hybrid-pipeline-forward.") + return + + timeout = getattr(args, "hybrid_pipeline_fetch_timeout_s", 600.0) + if not isinstance(timeout, (int, float)) or not math.isfinite(timeout) or timeout <= 0: + raise ValueError("--hybrid-pipeline-fetch-timeout-s must be a finite value greater than 0.") + + conflicts = [] + + def reject(condition: bool, option: str) -> None: + if condition: + conflicts.append(option) + + reject(not getattr(args, "use_dynamic_batch_size", False), "missing --use-dynamic-batch-size") + reject(getattr(args, "multimodal_keys", None) is None, "missing --multimodal-keys") + reject(getattr(args, "advantage_estimator", None) != "grpo", "--advantage-estimator must be grpo") + reject(not getattr(args, "compute_advantages_and_returns", False), "advantages must be computed in actor") + reject(not getattr(args, "enable_weights_backuper", True), "--disable-weights-backuper") + reject(getattr(args, "kl_coef", 0.0) != 0, "--kl-coef") + reject(getattr(args, "use_kl_loss", False), "--use-kl-loss") + reject(getattr(args, "use_opd", False), "--use-opd") + reject(getattr(args, "keep_old_actor", False), "--keep-old-actor") + reject(getattr(args, "true_on_policy_mode", False), "--true-on-policy-mode") + reject(getattr(args, "use_rollout_logprobs", False), "--use-rollout-logprobs") + reject(getattr(args, "get_mismatch_metrics", False), "--get-mismatch-metrics") + reject(getattr(args, "use_routing_replay", False), "--use-routing-replay") + reject(getattr(args, "use_rollout_routing_replay", False), "--use-rollout-routing-replay") + reject(getattr(args, "use_agentic_rollout", False), "--use-agentic-rollout") + reject(getattr(args, "partial_rollout", False), "--partial-rollout") + reject(getattr(args, "use_dynamic_global_batch_size", False), "--use-dynamic-global-batch-size") + reject(getattr(args, "use_critic", False), "--advantage-estimator ppo / critic") + reject(getattr(args, "per_rank_fetch", False), "--per-rank-fetch") + reject(float(getattr(args, "attention_dropout", 0.0) or 0.0) != 0.0, "--attention-dropout") + reject(float(getattr(args, "hidden_dropout", 0.0) or 0.0) != 0.0, "--hidden-dropout") + reject( + int(getattr(args, "pipeline_model_parallel_size", 1) or 1) != 1, + "--pipeline-model-parallel-size", + ) + reject( + int(getattr(args, "tensor_model_parallel_size", 1) or 1) != 2, + "--tensor-model-parallel-size must be 2", + ) + reject( + int(getattr(args, "context_parallel_size", 1) or 1) != 2, + "--context-parallel-size must be 2", + ) + reject( + int(getattr(args, "expert_model_parallel_size", 1) or 1) != 1, + "--expert-model-parallel-size must be 1", + ) + reject( + int(getattr(args, "expert_tensor_parallel_size", 1) or 1) != 1, + "--expert-tensor-parallel-size must be 1", + ) + reject(bool(getattr(args, "offload_train", False)), "--offload-train") + reject(bool(getattr(args, "offload_rollout", False)), "--offload-rollout") + reject( + int(getattr(args, "num_iters_per_train_update", 0) or 0) < 2, + "--num-iters-per-train-update must be >= 2", + ) + rollout_batch_size = int(getattr(args, "rollout_batch_size", 0) or 0) + n_samples_per_prompt = int(getattr(args, "n_samples_per_prompt", 0) or 0) + global_batch_size = int(getattr(args, "global_batch_size", 0) or 0) + reject( + rollout_batch_size * n_samples_per_prompt != global_batch_size, + "rollout_batch_size * n_samples_per_prompt must equal global_batch_size " + "(exactly one optimizer mini per rollout)", + ) + + if conflicts: + raise ValueError( + "--hybrid-pipeline-forward currently supports only actor-only multimodal " + "Hybrid GRPO with dynamic batching, zero dropout, TP2/PP1/CP2/EP1/ETP1, " + "offload disabled, and the normal TensorBackuper. " + f"Conflicting configuration: {', '.join(conflicts)}." + ) + + def slime_validate_args(args): # Backward compatibility: old scripts may pass --enable-gloo-process-groups if not hasattr(args, "use_gloo_process_groups"): @@ -3500,6 +3677,8 @@ def slime_validate_args(args): ) args.global_batch_size = global_batch_size + _validate_hybrid_pipeline_args(args) + if args.advantage_estimator == "rloo": if args.n_samples_per_prompt < 2: raise ValueError( diff --git a/relax/utils/training/hybrid_forward_pipeline.py b/relax/utils/training/hybrid_forward_pipeline.py new file mode 100644 index 000000000..e20af3f6e --- /dev/null +++ b/relax/utils/training/hybrid_forward_pipeline.py @@ -0,0 +1,140 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import math +import time +from collections.abc import Callable, Sequence +from typing import Any + + +def execute_hybrid_forward_mini( + *, + chunks_per_mini: int, + batch_index_for_chunk: Callable[[int], int], + restore_actor: Callable[[int], None], + fetch_chunk: Callable[[int], tuple[Any, list[int]]], + forward_chunk: Callable[[Any, int, list[int]], None], + overlap_producer: bool = True, +) -> list[tuple[Any, list[int]]]: + """Restore once, then execute a matched chunk schedule with optional + overlap.""" + if chunks_per_mini <= 0: + raise ValueError(f"chunks_per_mini must be positive, got {chunks_per_mini}") + if type(overlap_producer) is not bool: + raise TypeError(f"overlap_producer must be bool, got {overlap_producer!r}") + + batch_indexes = [batch_index_for_chunk(chunk_index) for chunk_index in range(chunks_per_mini)] + restore_actor(batch_indexes[0]) + + chunks: list[tuple[Any, list[int]]] = [] + for batch_index in batch_indexes: + batch, global_indexes = fetch_chunk(batch_index) + chunks.append((batch, global_indexes)) + if overlap_producer: + forward_chunk(batch, batch_index, global_indexes) + + if not overlap_producer: + for (batch, global_indexes), batch_index in zip(chunks, batch_indexes, strict=True): + forward_chunk(batch, batch_index, global_indexes) + return chunks + + +def canonicalize_hybrid_microbatch_schedule( + chunk_schedules: Sequence[tuple[Sequence[int], Sequence[Sequence[int]]]], + canonical_global_indexes: Sequence[int], +) -> list[list[int]]: + """Translate chunk-local forward schedules into merged-batch indexes. + + The actor old-logprob forward chooses its dynamic microbatches + independently for each producer chunk. Training must replay those exact + sample groups and their order; otherwise batch-shape-dependent numerics + create an artificial PPO ratio even though the weights did not change. + """ + canonical_indexes = list(canonical_global_indexes) + if not canonical_indexes: + raise ValueError("canonical_global_indexes must not be empty") + if not all(type(index) is int for index in canonical_indexes): + raise TypeError("canonical_global_indexes must contain only int values") + if len(set(canonical_indexes)) != len(canonical_indexes): + raise ValueError("canonical_global_indexes must not contain duplicates") + + canonical_positions = {index: position for position, index in enumerate(canonical_indexes)} + observed_global_indexes: list[int] = [] + merged_schedule: list[list[int]] = [] + + for chunk_index, (global_indexes, local_schedule) in enumerate(chunk_schedules): + chunk_global_indexes = list(global_indexes) + if not chunk_global_indexes: + raise ValueError(f"chunk {chunk_index} global_indexes must not be empty") + if not all(type(index) is int for index in chunk_global_indexes): + raise TypeError(f"chunk {chunk_index} global_indexes must contain only int values") + + flattened_local_indexes: list[int] = [] + for microbatch_index, local_indexes in enumerate(local_schedule): + normalized_local_indexes = list(local_indexes) + if not normalized_local_indexes: + raise ValueError(f"chunk {chunk_index} microbatch {microbatch_index} must not be empty") + if not all(type(index) is int for index in normalized_local_indexes): + raise TypeError(f"chunk {chunk_index} microbatch {microbatch_index} must contain only int indexes") + try: + merged_schedule.append( + [ + canonical_positions[chunk_global_indexes[local_index]] + for local_index in normalized_local_indexes + ] + ) + except IndexError as exc: + raise ValueError( + f"chunk {chunk_index} microbatch {microbatch_index} contains an out-of-range local index" + ) from exc + except KeyError as exc: + raise ValueError( + f"chunk {chunk_index} references global index {exc.args[0]} outside canonical_global_indexes" + ) from exc + flattened_local_indexes.extend(normalized_local_indexes) + + expected_local_indexes = list(range(len(chunk_global_indexes))) + if sorted(flattened_local_indexes) != expected_local_indexes: + raise ValueError(f"chunk {chunk_index} microbatch schedule must cover each local sample exactly once") + observed_global_indexes.extend(chunk_global_indexes) + + if sorted(observed_global_indexes) != sorted(canonical_indexes): + raise ValueError("chunk schedules must cover each canonical global index exactly once") + if sorted(index for microbatch in merged_schedule for index in microbatch) != list(range(len(canonical_indexes))): + raise ValueError("merged microbatch schedule must cover the merged batch exactly once") + return merged_schedule + + +def fetch_exact_chunk_with_timeout( + *, + fetch_once: Callable[[], tuple[Any | None, Any]], + expected_samples: int, + timeout_s: float, + error_context: str, + poll_interval_s: float = 0.1, + clock: Callable[[], float] = time.monotonic, + sleep: Callable[[float], None] = time.sleep, +) -> tuple[Any, Any, float]: + """Poll an exact sample-count fetch and fail instead of spinning + forever.""" + if expected_samples <= 0: + raise ValueError(f"expected_samples must be positive, got {expected_samples}") + if not math.isfinite(timeout_s) or timeout_s <= 0: + raise ValueError(f"timeout_s must be positive and finite, got {timeout_s}") + if not math.isfinite(poll_interval_s) or poll_interval_s <= 0: + raise ValueError(f"poll_interval_s must be positive and finite, got {poll_interval_s}") + + start = clock() + while True: + batch, metadata = fetch_once() + if batch is not None: + actual_samples = len(batch.get("total_lengths", [])) + if actual_samples != expected_samples: + raise RuntimeError(f"{error_context}, expected={expected_samples}, last_returned={actual_samples}") + return batch, metadata, clock() - start + + elapsed = clock() - start + if elapsed >= timeout_s: + raise TimeoutError( + f"{error_context}, expected={expected_samples}, last_returned=0, elapsed={elapsed:.3f}s" + ) + sleep(poll_interval_s) diff --git a/relax/utils/training/hybrid_pipeline_trace.py b/relax/utils/training/hybrid_pipeline_trace.py new file mode 100644 index 000000000..62c5bf3a9 --- /dev/null +++ b/relax/utils/training/hybrid_pipeline_trace.py @@ -0,0 +1,191 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +from __future__ import annotations + +import atexit +import hashlib +import json +import os +import socket +import threading +import time +from pathlib import Path +from typing import Any, TextIO + + +_WRITERS: dict[Path, TextIO] = {} +_WRITER_LOCK = threading.Lock() +_SCALAR_TYPES = (str, int, float, bool, type(None)) +_FINGERPRINT_BITS = 128 +_FINGERPRINT_MODULUS = 1 << _FINGERPRINT_BITS +_RECORD_FIELDS = { + "event", + "monotonic_ns", + "rollout_id", + "chunk_index", + "sample_count", + "total_tokens", + "response_tokens", + "multimodal_tensor_bytes", + "role", + "hostname", + "pid", + "global_rank", + "cuda_visible_devices", + "cuda_max_allocated_bytes", + "cuda_max_reserved_bytes", + "global_indexes_fingerprint", +} + + +def _close_writers() -> None: + with _WRITER_LOCK: + for writer in _WRITERS.values(): + writer.close() + _WRITERS.clear() + + +atexit.register(_close_writers) + + +def _global_rank() -> int: + try: + import torch.distributed as dist + + if dist.is_available() and dist.is_initialized(): + return dist.get_rank() + except (ImportError, RuntimeError): + pass + return -1 + + +def _cuda_memory_peaks() -> tuple[int | None, int | None]: + try: + import torch + + if torch.cuda.is_available(): + return ( + int(torch.cuda.max_memory_allocated()), + int(torch.cuda.max_memory_reserved()), + ) + except (ImportError, RuntimeError): + pass + return None, None + + +def _tensor_bytes(value: Any) -> int: + if value is None or isinstance(value, (str, bytes)): + return 0 + if isinstance(value, dict): + return sum(_tensor_bytes(item) for item in value.values()) + if isinstance(value, (list, tuple)): + return sum(_tensor_bytes(item) for item in value) + + nelement = getattr(value, "nelement", None) + element_size = getattr(value, "element_size", None) + if callable(nelement) and callable(element_size): + return int(nelement()) * int(element_size()) + nbytes = getattr(value, "nbytes", None) + if nbytes is not None: + return int(nbytes) + return 0 + + +def _sum_ints(values: Any) -> int | None: + if values is None: + return None + try: + return sum(int(value) for value in values) + except (TypeError, ValueError): + return None + + +def fingerprint_global_indexes(global_indexes: Any) -> str | None: + """Return an order-independent, multiplicity-sensitive digest. + + Digests are added modulo 2**128 so an analyzer can combine chunk digests + without recording sample indexes or depending on producer/fetch grouping. + """ + if global_indexes is None: + return None + normalized = [int(index) for index in global_indexes] + accumulator = 0 + for index in normalized: + digest = hashlib.blake2b(str(index).encode("ascii"), digest_size=16).digest() + accumulator = (accumulator + int.from_bytes(digest, "big")) % _FINGERPRINT_MODULUS + return f"{accumulator:032x}" + + +def _trace_path(trace_dir: str, role: str, hostname: str, pid: int, rank: int) -> Path: + safe_hostname = hostname.replace("/", "_") + safe_role = role.replace("/", "_") + return Path(trace_dir) / f"hybrid-pipeline-{safe_role}-{safe_hostname}-pid{pid}-rank{rank}.jsonl" + + +def emit_hybrid_pipeline_event( + args: Any, + event: str, + *, + rollout_id: int, + role: str, + chunk_index: int | None = None, + sample_count: int | None = None, + batch: dict[str, Any] | None = None, + global_indexes: Any = None, + monotonic_ns: int | None = None, + details: dict[str, Any] | None = None, +) -> dict[str, Any] | None: + """Append one content-free Hybrid pipeline event to this process's + JSONL.""" + trace_dir = getattr(args, "hybrid_pipeline_trace_dir", None) + if not trace_dir: + return None + if details is not None: + invalid = {key: value for key, value in details.items() if not isinstance(value, _SCALAR_TYPES)} + if invalid: + raise TypeError(f"Hybrid pipeline trace details must be scalar, got {invalid}") + reserved = sorted(_RECORD_FIELDS.intersection(details)) + if reserved: + raise ValueError(f"Hybrid pipeline trace details cannot replace record fields: {reserved}") + + hostname = socket.gethostname() + pid = os.getpid() + rank = _global_rank() + cuda_allocated, cuda_reserved = _cuda_memory_peaks() + total_lengths = batch.get("total_lengths") if batch is not None else None + response_lengths = batch.get("response_lengths") if batch is not None else None + if sample_count is None and total_lengths is not None: + sample_count = len(total_lengths) + + record = { + "event": event, + "monotonic_ns": time.monotonic_ns() if monotonic_ns is None else int(monotonic_ns), + "rollout_id": int(rollout_id), + "chunk_index": chunk_index, + "sample_count": sample_count, + "total_tokens": _sum_ints(total_lengths), + "response_tokens": _sum_ints(response_lengths), + "multimodal_tensor_bytes": ( + _tensor_bytes(batch.get("multimodal_train_inputs")) if batch is not None else None + ), + "role": role, + "hostname": hostname, + "pid": pid, + "global_rank": rank, + "cuda_visible_devices": os.environ.get("CUDA_VISIBLE_DEVICES"), + "cuda_max_allocated_bytes": cuda_allocated, + "cuda_max_reserved_bytes": cuda_reserved, + "global_indexes_fingerprint": fingerprint_global_indexes(global_indexes), + } + if details: + record.update(details) + + path = _trace_path(os.fspath(trace_dir), role, hostname, pid, rank) + with _WRITER_LOCK: + writer = _WRITERS.get(path) + if writer is None: + path.parent.mkdir(parents=True, exist_ok=True) + writer = path.open("a", encoding="utf-8", buffering=1) + _WRITERS[path] = writer + writer.write(json.dumps(record, sort_keys=True, separators=(",", ":")) + "\n") + return record diff --git a/relax/utils/utils.py b/relax/utils/utils.py index d7f5e93f3..ec2f30e2f 100644 --- a/relax/utils/utils.py +++ b/relax/utils/utils.py @@ -2,6 +2,7 @@ import os import socket +import time from argparse import Namespace from pathlib import Path from typing import Any, Dict, List, Optional, Union @@ -15,6 +16,7 @@ from relax.utils.env import Envs, validate_env from relax.utils.logging_utils import get_logger from relax.utils.misc import load_function +from relax.utils.training.hybrid_pipeline_trace import emit_hybrid_pipeline_event from relax.utils.training.ppo_utils import compute_rloo_leave_one_out_rewards from relax.utils.types import Sample @@ -522,9 +524,48 @@ async def transfer_batch_to_data_system( # fallback and defeating dynamic batching). total_lengths = rollout_batch.get("total_lengths", None) custom_meta = [{"total_lengths": int(tl)} for tl in total_lengths] if total_lengths is not None else None - await data_system_client.async_put( + trace_enabled = bool(getattr(args, "hybrid_pipeline_trace_dir", None)) + if trace_enabled: + put_start_ns = time.monotonic_ns() + put_event_id = f"{os.getpid()}-{put_start_ns}" + emit_hybrid_pipeline_event( + args, + "tq_put_start", + rollout_id=rollout_id, + role="rollout", + chunk_index=batch_count, + batch=rollout_batch, + monotonic_ns=put_start_ns, + details={ + "batch_count": batch_count, + "is_last": is_last, + "event_id": put_event_id, + }, + ) + batch_meta = await data_system_client.async_put( data=rollout_batch, partition_id=f"train_{rollout_id}", custom_meta=custom_meta, is_last=is_last ) + if trace_enabled: + global_indexes = getattr(batch_meta, "global_indexes", None) + if global_indexes is None: + raise RuntimeError( + "Hybrid pipeline trace requires async_put() to return " + "BatchMeta.global_indexes; install the pinned TransferQueue version." + ) + emit_hybrid_pipeline_event( + args, + "tq_put_done", + rollout_id=rollout_id, + role="rollout", + chunk_index=batch_count, + batch=rollout_batch, + global_indexes=global_indexes, + details={ + "batch_count": batch_count, + "is_last": is_last, + "event_id": put_event_id, + }, + ) logger.info(f"Batch {batch_count} transferred successfully for rollout_id: {rollout_id}") except Exception as e: diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py new file mode 100644 index 000000000..531643757 --- /dev/null +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -0,0 +1,2302 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +"""Validate and summarize Task 21 Hybrid pipeline benchmark artifacts.""" + +from __future__ import annotations + +import argparse +import csv +import json +import math +import os +import statistics +import sys +from collections import defaultdict +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path +from typing import Any, Iterable, Sequence + + +TRACE_REQUIRED_FIELDS = { + "event", + "monotonic_ns", + "rollout_id", + "chunk_index", + "sample_count", + "total_tokens", + "response_tokens", + "multimodal_tensor_bytes", + "role", + "hostname", + "pid", + "global_rank", + "cuda_visible_devices", + "cuda_max_allocated_bytes", + "cuda_max_reserved_bytes", + "global_indexes_fingerprint", +} +NVML_COLUMNS = ( + "timestamp", + "gpu_index", + "gpu_name", + "pstate", + "temperature_c", + "sm_clock_mhz", + "memory_clock_mhz", + "gpu_util_percent", + "memory_util_percent", + "memory_used_mib", + "power_w", +) +DEFAULT_WINDOWS = ((4, 8), (9, 13), (14, 18)) +PERFORMANCE_TAGS = ( + "perf/step_token_per_s", + "perf/step_resp_token_per_s", + "perf/step_time", + "perf/hybrid_phase1_time", +) +COMPARISON_PERFORMANCE_TAGS = PERFORMANCE_TAGS + ("perf/wall_clock_samples_per_s",) +CORRECTNESS_GUARDRAIL_TAGS = ( + "rollout/raw_reward", + "rollout/truncated_ratio", + "train/loss", + "train/grad_norm", + "train/ppo_kl", + "train/pg_clipfrac", +) +STEADY_CAMPAIGN_CONDITIONS = { + "B": (0, 0, 0), + "P": (8, 0, 0), + "P+R": (8, 1, 0), + "P+S": (8, 0, 1), +} +STEADY_CAMPAIGN_REQUIRED_TAGS = ( + "perf/step_token_per_s", + "perf/step_resp_token_per_s", + "perf/step_time", + "rollout/raw_reward", + "rollout/image_count/mean", + "train/loss", + "train/grad_norm", + "train/ppo_kl", +) +RUN_MANIFEST_REQUIRED_FIELDS = { + "hostname", + "condition", + "order", + "seed", + "rollout_seed", + "num_rollout", + "max_staleness", + "global_batch_size", + "rollout_batch_size", + "n_samples_per_prompt", + "num_iters_per_train_update", + "hybrid_pipeline_forward", + "hybrid_pipeline_trace_dir", + "hybrid_pipeline_fetch_timeout_s", + "git_commit", + "git_branch", + "git_status_porcelain", + "image_archive_sha256", + "image_manifest_digest", + "image_id", + "transferqueue_commit", + "python", + "entrypoint", +} +COMPARISON_FIXED_MANIFEST_FIELDS = ( + "hostname", + "num_rollout", + "max_staleness", + "global_batch_size", + "rollout_batch_size", + "n_samples_per_prompt", + "num_iters_per_train_update", + "hybrid_pipeline_fetch_timeout_s", + "git_commit", + "git_branch", + "image_archive_sha256", + "image_manifest_digest", + "image_id", + "transferqueue_commit", + "python", + "entrypoint", +) +COMPARISON_WORKLOAD_MANIFEST_FIELDS = ( + "schema_version", + "baseline_commit", + "model_variant", + "model_name", + "model_dir", + "model_config_file", + "data_file", + "rollout_max_response_len", + "rollout_max_prompt_len", + "rollout_max_context_len", + "actor_max_tokens_per_gpu", + "resource", + "rollout_num_gpus_per_engine", + "physical_gpu_indices", + "container_cuda_visible_devices", + "gpu_hardware_fingerprint", + "checkpoint_save", + "sglang_deterministic_inference", + "sglang_mem_fraction_static", + "load_debug_rollout_data", + "save_debug_rollout_data", + "save_debug_train_data", +) +REPRODUCIBILITY_ARTIFACTS = ( + "pip-freeze.txt", + "inputs.sha256", + "transferqueue-wheel.sha256", + "logs/launcher.log", + "manifests/static-input-verification.log", +) +EXIT_STATUS_ARTIFACTS = ( + "training_exit_status.txt", + "validation_exit_status.txt", + "exit_status.txt", +) + + +class BenchmarkValidationError(RuntimeError): + """Raised when benchmark artifacts violate a registered invariant.""" + + +@dataclass(frozen=True) +class RunAnalysis: + run_dir: Path + manifest: dict[str, Any] + trace_rows: list[dict[str, Any]] + actor_rank_rows: list[dict[str, Any]] + scalar_rows: list[dict[str, Any]] + nvml_rows: list[dict[str, Any]] + summary: dict[str, Any] + + +def _fail(message: str) -> None: + raise BenchmarkValidationError(message) + + +def _validate_finite(value: Any, context: str) -> None: + if isinstance(value, bool) or value is None or isinstance(value, str): + return + if isinstance(value, (int, float)): + if not math.isfinite(value): + _fail(f"{context} contains non-finite numeric value {value!r}") + return + if isinstance(value, dict): + for key, item in value.items(): + _validate_finite(item, f"{context}.{key}") + return + if isinstance(value, (list, tuple)): + for index, item in enumerate(value): + _validate_finite(item, f"{context}[{index}]") + + +def _load_manifest(run_dir: Path) -> dict[str, Any]: + path = run_dir / "run_manifest.json" + if not path.is_file(): + _fail(f"missing run manifest: {path}") + try: + manifest = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + _fail(f"cannot parse {path}: {exc}") + if not isinstance(manifest, dict): + _fail(f"{path} must contain a JSON object") + missing = sorted(RUN_MANIFEST_REQUIRED_FIELDS - set(manifest)) + if missing: + _fail(f"{path} is missing required fields {missing}") + _validate_finite(manifest, str(path)) + return manifest + + +def _validate_hex_digest(value: Any, *, length: int, context: str, prefix: str = "") -> None: + if not isinstance(value, str) or not value.startswith(prefix): + _fail(f"{context} must be a {prefix!r}-prefixed hexadecimal string") + payload = value[len(prefix) :] + if len(payload) != length: + _fail(f"{context} must contain {length} hexadecimal characters, got {len(payload)}") + try: + int(payload, 16) + except ValueError: + _fail(f"{context} contains non-hexadecimal characters") + + +def _validate_run_manifest( + manifest: dict[str, Any], + *, + run_dir: Path, + windows: Sequence[tuple[int, int]], +) -> None: + for key in ( + "seed", + "rollout_seed", + "num_rollout", + "max_staleness", + "global_batch_size", + "rollout_batch_size", + "n_samples_per_prompt", + "num_iters_per_train_update", + "hybrid_pipeline_forward", + ): + if type(manifest[key]) is not int: + _fail(f"{run_dir} manifest field {key!r} must be an integer, got {manifest[key]!r}") + if "hybrid_pipeline_overlap" in manifest and ( + type(manifest["hybrid_pipeline_overlap"]) is not int or manifest["hybrid_pipeline_overlap"] not in (0, 1) + ): + _fail( + f"{run_dir} manifest field 'hybrid_pipeline_overlap' must be integer 0 or 1, " + f"got {manifest['hybrid_pipeline_overlap']!r}" + ) + if manifest["seed"] != manifest["rollout_seed"]: + _fail( + f"{run_dir} must use the same paired Megatron/rollout seed, got " + f"{manifest['seed']} and {manifest['rollout_seed']}" + ) + if manifest["num_rollout"] <= max(end for _, end in windows): + _fail( + f"{run_dir} num_rollout={manifest['num_rollout']} does not cover " + f"steady window ending at step {max(end for _, end in windows)}" + ) + for key in ( + "global_batch_size", + "rollout_batch_size", + "n_samples_per_prompt", + "num_iters_per_train_update", + ): + if manifest[key] <= 0: + _fail(f"{run_dir} manifest field {key!r} must be positive, got {manifest[key]!r}") + if manifest["max_staleness"] < 0: + _fail(f"{run_dir} max_staleness must be non-negative, got {manifest['max_staleness']!r}") + produced_samples = manifest["rollout_batch_size"] * manifest["n_samples_per_prompt"] + if produced_samples != manifest["global_batch_size"]: + _fail( + f"{run_dir} must benchmark exactly one optimizer mini per rollout: " + f"rollout_batch_size * n_samples_per_prompt={produced_samples}, " + f"global_batch_size={manifest['global_batch_size']}" + ) + if manifest["git_status_porcelain"] != "": + _fail(f"{run_dir} was captured from a dirty working tree: {manifest['git_status_porcelain']!r}") + + timeout = manifest["hybrid_pipeline_fetch_timeout_s"] + if ( + isinstance(timeout, bool) + or not isinstance(timeout, (int, float)) + or not math.isfinite(timeout) + or timeout <= 0 + ): + _fail(f"{run_dir} has invalid hybrid_pipeline_fetch_timeout_s={timeout!r}") + + trace_dir = Path(str(manifest["hybrid_pipeline_trace_dir"])).resolve() + expected_trace_dir = (run_dir / "timeline").resolve() + if trace_dir != expected_trace_dir: + _fail(f"{run_dir} manifest trace directory is {trace_dir}, expected {expected_trace_dir}") + + for key in ("hostname", "condition", "order", "git_branch", "python", "entrypoint"): + if not isinstance(manifest[key], str) or not manifest[key].strip(): + _fail(f"{run_dir} manifest field {key!r} must be a non-empty string") + + _validate_hex_digest(manifest["git_commit"], length=40, context=f"{run_dir} git_commit") + _validate_hex_digest( + manifest["image_archive_sha256"], + length=64, + context=f"{run_dir} image_archive_sha256", + ) + _validate_hex_digest( + manifest["image_manifest_digest"], + length=64, + prefix="sha256:", + context=f"{run_dir} image_manifest_digest", + ) + _validate_hex_digest( + manifest["image_id"], + length=64, + prefix="sha256:", + context=f"{run_dir} image_id", + ) + _validate_hex_digest( + manifest["transferqueue_commit"], + length=40, + context=f"{run_dir} transferqueue_commit", + ) + if "gpu_hardware_fingerprint" in manifest: + _validate_hex_digest( + manifest["gpu_hardware_fingerprint"], + length=64, + context=f"{run_dir} gpu_hardware_fingerprint", + ) + + +def _require_reproducibility_artifacts(run_dir: Path) -> None: + missing = [] + for relative_path in REPRODUCIBILITY_ARTIFACTS: + path = run_dir / relative_path + if not path.is_file() or path.stat().st_size == 0: + missing.append(relative_path) + if missing: + _fail(f"{run_dir} is missing non-empty reproducibility artifacts {missing}") + for relative_path in EXIT_STATUS_ARTIFACTS: + path = run_dir / relative_path + if not path.is_file(): + _fail(f"{run_dir} is missing exit-status artifact {relative_path}") + try: + status = path.read_text(encoding="utf-8").strip() + except OSError as exc: + _fail(f"cannot read {path}: {exc}") + if status != "0": + _fail(f"{run_dir} has non-zero or invalid {relative_path}: {status!r}") + + +def _load_trace_rows(run_dir: Path) -> list[dict[str, Any]]: + trace_dir = run_dir / "timeline" + paths = sorted(trace_dir.glob("*.jsonl")) + if not paths: + _fail(f"no Hybrid pipeline JSONL files found under {trace_dir}") + + rows: list[dict[str, Any]] = [] + for path in paths: + previous_ns = -1 + with path.open(encoding="utf-8") as reader: + for line_number, line in enumerate(reader, start=1): + if not line.strip(): + continue + try: + row = json.loads(line) + except json.JSONDecodeError as exc: + _fail(f"{path}:{line_number} is invalid JSON: {exc}") + if not isinstance(row, dict): + _fail(f"{path}:{line_number} must contain a JSON object") + missing = sorted(TRACE_REQUIRED_FIELDS - set(row)) + if missing: + _fail(f"{path}:{line_number} is missing trace fields {missing}") + _validate_finite(row, f"{path}:{line_number}") + monotonic_ns = row["monotonic_ns"] + if type(monotonic_ns) is not int or monotonic_ns < 0: + _fail(f"{path}:{line_number} has invalid monotonic_ns={monotonic_ns!r}") + if monotonic_ns < previous_ns: + _fail(f"{path}:{line_number} is not monotonic: previous={previous_ns}, current={monotonic_ns}") + previous_ns = monotonic_ns + row["_source_file"] = path.name + row["_source_line"] = line_number + rows.append(row) + + if not rows: + _fail(f"trace files under {trace_dir} contain no events") + hostnames = {row["hostname"] for row in rows} + if len(hostnames) != 1: + _fail(f"trace events span multiple hostnames and cannot share one monotonic clock: {sorted(hostnames)}") + return sorted(rows, key=lambda row: (row["monotonic_ns"], row["_source_file"], row["_source_line"])) + + +def _pair_events( + rows: Sequence[dict[str, Any]], + start_event: str, + end_event: str, + *, + key: str, + context: str, +) -> list[tuple[dict[str, Any], dict[str, Any]]]: + starts: dict[Any, list[dict[str, Any]]] = defaultdict(list) + ends: dict[Any, list[dict[str, Any]]] = defaultdict(list) + for row in rows: + if row["event"] == start_event: + starts[row.get(key)].append(row) + elif row["event"] == end_event: + ends[row.get(key)].append(row) + if set(starts) != set(ends): + _fail( + f"{context} has unmatched {start_event}/{end_event} keys: " + f"starts={sorted(starts, key=str)}, ends={sorted(ends, key=str)}" + ) + + pairs = [] + for event_key in sorted(starts, key=str): + if len(starts[event_key]) != 1 or len(ends[event_key]) != 1: + _fail( + f"{context} requires one {start_event}/{end_event} pair for {key}={event_key!r}, " + f"got starts={len(starts[event_key])}, ends={len(ends[event_key])}" + ) + start, end = starts[event_key][0], ends[event_key][0] + if start["monotonic_ns"] > end["monotonic_ns"]: + _fail( + f"{context} has {start_event} after {end_event} for {key}={event_key!r}: " + f"{start['monotonic_ns']} > {end['monotonic_ns']}" + ) + pairs.append((start, end)) + return pairs + + +def _require_count(rows: Sequence[dict[str, Any]], event: str, count: int, context: str) -> list[dict[str, Any]]: + matches = [row for row in rows if row["event"] == event] + if len(matches) != count: + _fail(f"{context} expected {count} {event!r} events, got {len(matches)}") + return matches + + +def _stream_key(row: dict[str, Any]) -> tuple[str, int, int]: + return row["hostname"], int(row["pid"]), int(row["global_rank"]) + + +def _combine_global_index_fingerprints( + rows: Sequence[dict[str, Any]], + *, + context: str, +) -> str: + """Combine additive 128-bit chunk digests without exposing sample + indexes.""" + accumulator = 0 + modulus = 1 << 128 + for row in rows: + fingerprint = row.get("global_indexes_fingerprint") + if not isinstance(fingerprint, str) or len(fingerprint) != 32: + _fail(f"{context} has invalid global index fingerprint {fingerprint!r}") + try: + value = int(fingerprint, 16) + except ValueError: + _fail(f"{context} has non-hex global index fingerprint {fingerprint!r}") + accumulator = (accumulator + value) % modulus + return f"{accumulator:032x}" + + +def _analyze_trace( + rows: Sequence[dict[str, Any]], + *, + pipeline_enabled: bool, + pipeline_overlap_enabled: bool, + reuse_train_forward_log_probs: bool, + expected_samples: int, + expected_actor_chunks: int, +) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]: + producer_by_rollout: dict[int, list[dict[str, Any]]] = defaultdict(list) + actor_by_rollout_stream: dict[tuple[int, tuple[str, int, int]], list[dict[str, Any]]] = defaultdict(list) + for row in rows: + rollout_id = int(row["rollout_id"]) + if row["role"] == "rollout": + producer_by_rollout[rollout_id].append(row) + elif row["role"] == "actor": + actor_by_rollout_stream[(rollout_id, _stream_key(row))].append(row) + + producer_rollouts = set(producer_by_rollout) + actor_rollouts = {rollout_id for rollout_id, _ in actor_by_rollout_stream} + if not producer_rollouts or producer_rollouts != actor_rollouts: + _fail( + "producer and actor rollout IDs differ: " + f"producer={sorted(producer_rollouts)}, actor={sorted(actor_rollouts)}" + ) + + expected_stream_chunks = expected_actor_chunks if pipeline_enabled else 1 + rollout_rows: list[dict[str, Any]] = [] + actor_rank_rows: list[dict[str, Any]] = [] + + for rollout_id in sorted(producer_rollouts): + producer_rows = producer_by_rollout[rollout_id] + put_pairs = _pair_events( + producer_rows, + "tq_put_start", + "tq_put_done", + key="event_id", + context=f"rollout_id={rollout_id} producer", + ) + if not put_pairs: + _fail(f"rollout_id={rollout_id} producer has no completed puts") + put_start = [start for start, _ in put_pairs] + put_done = [end for _, end in put_pairs] + if any(row["sample_count"] is None for row in put_done): + _fail(f"rollout_id={rollout_id} producer put is missing sample_count") + producer_samples = sum(int(row["sample_count"]) for row in put_done) + if producer_samples != expected_samples: + _fail( + f"rollout_id={rollout_id} producer sample conservation failed: " + f"expected={expected_samples}, actual={producer_samples}" + ) + producer_fingerprint = _combine_global_index_fingerprints( + put_done, + context=f"rollout_id={rollout_id} producer", + ) + + streams = { + stream: stream_rows + for (stream_rollout_id, stream), stream_rows in actor_by_rollout_stream.items() + if stream_rollout_id == rollout_id + } + if not streams: + _fail(f"rollout_id={rollout_id} has no actor trace stream") + + for stream, stream_rows in streams.items(): + context = f"rollout_id={rollout_id} actor_stream={stream}" + fetch_pairs = _pair_events( + stream_rows, + "chunk_fetch_start", + "chunk_fetch_end", + key="chunk_index", + context=context, + ) + forward_pairs = _pair_events( + stream_rows, + "actor_forward_start", + "actor_forward_end", + key="chunk_index", + context=context, + ) + restore_pairs = _pair_events( + stream_rows, + "actor_restore_start", + "actor_restore_end", + key="chunk_index", + context=context, + ) + expected_forward_chunks = 0 if reuse_train_forward_log_probs else expected_stream_chunks + if len(fetch_pairs) != expected_stream_chunks or len(forward_pairs) != expected_forward_chunks: + _fail( + f"{context} expected fetch={expected_stream_chunks}, forward={expected_forward_chunks} chunks, " + f"got fetch={len(fetch_pairs)}, forward={len(forward_pairs)}" + ) + expected_restore_count = 0 if reuse_train_forward_log_probs else 1 + if len(restore_pairs) != expected_restore_count: + _fail(f"{context} expected {expected_restore_count} actor restores, got {len(restore_pairs)}") + _pair_events( + stream_rows, + "advantages_start", + "advantages_end", + key="chunk_index", + context=context, + ) + _pair_events( + stream_rows, + "optimizer_start", + "optimizer_end", + key="chunk_index", + context=context, + ) + _require_count(stream_rows, "advantages_start", 1, context) + _require_count(stream_rows, "optimizer_start", 1, context) + + fetch_end = [end for _, end in fetch_pairs] + forward_start = [start for start, _ in forward_pairs] + forward_end = [end for _, end in forward_pairs] + sample_event_groups = [(fetch_end, "fetch")] + if not reuse_train_forward_log_probs: + sample_event_groups.append((forward_start, "forward")) + for event_rows, event_name in sample_event_groups: + if any(row["sample_count"] is None for row in event_rows): + _fail(f"{context} {event_name} event is missing sample_count") + actual_samples = sum(int(row["sample_count"]) for row in event_rows) + if actual_samples != expected_samples: + _fail( + f"{context} {event_name} sample conservation failed: " + f"expected={expected_samples}, actual={actual_samples}" + ) + if any(row["global_indexes_fingerprint"] is None for row in fetch_end): + _fail(f"{context} fetch event is missing global index fingerprint") + fetch_fingerprint = _combine_global_index_fingerprints( + fetch_end, + context=f"{context} fetch", + ) + forward_fingerprint = ( + fetch_fingerprint + if reuse_train_forward_log_probs + else _combine_global_index_fingerprints( + forward_start, + context=f"{context} forward", + ) + ) + if fetch_fingerprint != producer_fingerprint: + _fail( + f"{context} producer/fetch global index fingerprints differ: " + f"producer={producer_fingerprint}, actor={fetch_fingerprint}" + ) + if forward_fingerprint != fetch_fingerprint: + _fail( + f"{context} fetch/forward global index fingerprints differ: " + f"fetch={fetch_fingerprint}, forward={forward_fingerprint}" + ) + + if not reuse_train_forward_log_probs: + fetch_by_chunk = {end["chunk_index"]: (start, end) for start, end in fetch_pairs} + for forward_start_row, forward_end_row in forward_pairs: + chunk_index = forward_start_row["chunk_index"] + if chunk_index not in fetch_by_chunk: + _fail(f"{context} forward chunk {chunk_index!r} has no matching fetch") + _, fetch_end_row = fetch_by_chunk[chunk_index] + if fetch_end_row["monotonic_ns"] > forward_start_row["monotonic_ns"]: + _fail(f"{context} forward chunk {chunk_index!r} starts before fetch completes") + if forward_start_row["monotonic_ns"] > forward_end_row["monotonic_ns"]: + _fail(f"{context} forward chunk {chunk_index!r} ends before it starts") + + first_fetch_ns = min(start["monotonic_ns"] for start, _ in fetch_pairs) + first_phase_ns = ( + first_fetch_ns + if reuse_train_forward_log_probs + else min(first_fetch_ns, restore_pairs[0][0]["monotonic_ns"]) + ) + last_forward_ns = max( + row["monotonic_ns"] for row in (fetch_end if reuse_train_forward_log_probs else forward_end) + ) + first_forward_ns = min( + row["monotonic_ns"] for row in (fetch_end if reuse_train_forward_log_probs else forward_start) + ) + last_fetch_end_ns = max(row["monotonic_ns"] for row in fetch_end) + last_put_start_ns = max(row["monotonic_ns"] for row in put_start) + last_put_done_ns = max(row["monotonic_ns"] for row in put_done) + chunk_schedule_overlapped = not reuse_train_forward_log_probs and first_forward_ns < last_fetch_end_ns + if pipeline_enabled and chunk_schedule_overlapped != pipeline_overlap_enabled: + expected_order = ( + "the first forward before the final fetch completed" + if pipeline_overlap_enabled + else "all fetches before the first forward" + ) + _fail(f"{context} does not implement the requested overlap mode: expected {expected_order}") + ready_rollout_ids = [ + int(row["rollout_id"]) + for row in rows + if row["role"] == "rollout" + and row["event"] == "tq_put_done" + and row["monotonic_ns"] <= first_forward_ns + ] + # A completed actor fetch proves the current rollout was ready even + # if the producer's post-async_put trace write loses the scheduling + # race with the consumer process. + ready_rollout_ids.append(rollout_id) + producer_lead = max(ready_rollout_ids) - rollout_id + actor_rank_rows.append( + { + "rollout_id": rollout_id, + "hostname": stream[0], + "pid": stream[1], + "global_rank": stream[2], + "pipeline_enabled": pipeline_enabled, + "pipeline_overlap_enabled": pipeline_overlap_enabled, + "reuse_train_forward_log_probs": reuse_train_forward_log_probs, + "producer_samples": producer_samples, + "actor_fetch_samples": sum(int(row["sample_count"]) for row in fetch_end), + "actor_forward_samples": ( + sum(int(row["sample_count"]) for row in fetch_end) + if reuse_train_forward_log_probs + else sum(int(row["sample_count"]) for row in forward_start) + ), + "producer_put_count": len(put_pairs), + "actor_fetch_count": len(fetch_pairs), + "actor_forward_count": len(forward_pairs), + "actor_restore_count": len(restore_pairs), + "phase1_s": (last_forward_ns - first_phase_ns) / 1e9, + # Strict evidence of producer/actor overlap: the actor + # starts forwarding before the producer even begins its + # final put. A delayed post-put trace write alone cannot + # make this condition true. + "first_forward_before_last_put_start": first_forward_ns < last_put_start_ns, + # Diagnostic only: async_put may have made data visible + # before the producer coroutine records tq_put_done. + "first_forward_before_last_put_done": first_forward_ns < last_put_done_ns, + "first_forward_before_last_fetch_end": chunk_schedule_overlapped, + "all_chunks_fetched_before_first_forward": not chunk_schedule_overlapped, + "producer_overlap_s": max(0, last_put_start_ns - first_forward_ns) / 1e9, + "transfer_overlap_s": max(0, last_put_done_ns - first_forward_ns) / 1e9, + "producer_lead_at_first_forward": producer_lead, + "actor_multimodal_tensor_bytes": sum( + int(row["multimodal_tensor_bytes"] or 0) for row in fetch_end + ), + "actor_total_tokens": sum(int(row["total_tokens"] or 0) for row in fetch_end), + "actor_response_tokens": sum(int(row["response_tokens"] or 0) for row in fetch_end), + "cuda_max_allocated_bytes": max( + (int(row["cuda_max_allocated_bytes"] or 0) for row in stream_rows), + default=0, + ), + "cuda_max_reserved_bytes": max( + (int(row["cuda_max_reserved_bytes"] or 0) for row in stream_rows), + default=0, + ), + "fetch_global_indexes_fingerprint": fetch_fingerprint, + "forward_global_indexes_fingerprint": forward_fingerprint, + } + ) + + primary = min( + (row for row in actor_rank_rows if row["rollout_id"] == rollout_id), + key=lambda row: ( + row["global_rank"] < 0, + row["global_rank"] if row["global_rank"] >= 0 else row["pid"], + ), + ) + put_start_times = [row["monotonic_ns"] for row in put_start] + put_done_times = [row["monotonic_ns"] for row in put_done] + rollout_rows.append( + { + **primary, + "producer_global_indexes_fingerprint": producer_fingerprint, + "producer_first_put_start_ns": min(put_start_times), + "producer_last_put_start_ns": max(put_start_times), + "producer_first_put_done_ns": min(put_done_times), + "producer_last_put_done_ns": max(put_done_times), + "producer_ready_window_s": (max(put_done_times) - min(put_done_times)) / 1e9, + "producer_multimodal_tensor_bytes": sum(int(row["multimodal_tensor_bytes"] or 0) for row in put_done), + "producer_total_tokens": sum(int(row["total_tokens"] or 0) for row in put_done), + "producer_response_tokens": sum(int(row["response_tokens"] or 0) for row in put_done), + } + ) + + producer_overlap_count = sum(bool(row["first_forward_before_last_put_start"]) for row in rollout_rows) + transfer_overlap_count = sum(bool(row["first_forward_before_last_put_done"]) for row in rollout_rows) + chunk_schedule_overlap_count = sum(bool(row["first_forward_before_last_fetch_end"]) for row in rollout_rows) + summary = { + "hostname": next(iter({row["hostname"] for row in rows})), + "pipeline_enabled": pipeline_enabled, + "pipeline_overlap_enabled": pipeline_overlap_enabled, + "reuse_train_forward_log_probs": reuse_train_forward_log_probs, + "rollout_count": len(rollout_rows), + "actor_stream_count": len({_stream_key(row) for row in rows if row["role"] == "actor"}), + "producer_overlap_rollout_count": producer_overlap_count, + "producer_overlap_rollout_ratio": producer_overlap_count / len(rollout_rows), + "transfer_overlap_rollout_count": transfer_overlap_count, + "transfer_overlap_rollout_ratio": transfer_overlap_count / len(rollout_rows), + "chunk_schedule_overlap_rollout_count": chunk_schedule_overlap_count, + "chunk_schedule_overlap_rollout_ratio": chunk_schedule_overlap_count / len(rollout_rows), + "mean_phase1_s": statistics.fmean(row["phase1_s"] for row in rollout_rows), + "mean_producer_overlap_s": statistics.fmean(row["producer_overlap_s"] for row in rollout_rows), + "mean_transfer_overlap_s": statistics.fmean(row["transfer_overlap_s"] for row in rollout_rows), + "mean_producer_lead_at_first_forward": statistics.fmean( + row["producer_lead_at_first_forward"] for row in rollout_rows + ), + "max_producer_lead_at_first_forward": max(row["producer_lead_at_first_forward"] for row in rollout_rows), + "mean_producer_ready_window_s": statistics.fmean(row["producer_ready_window_s"] for row in rollout_rows), + "max_cuda_allocated_bytes": max(row["cuda_max_allocated_bytes"] for row in actor_rank_rows), + "max_cuda_reserved_bytes": max(row["cuda_max_reserved_bytes"] for row in actor_rank_rows), + } + return rollout_rows, actor_rank_rows, summary + + +def _load_tensorboard_scalars(run_dir: Path) -> list[dict[str, Any]]: + event_paths = sorted(path for path in run_dir.rglob("events.out.tfevents.*") if path.is_file()) + if not event_paths: + return [] + # TensorBoard 2.10's generated protobuf bindings need the pure-Python + # compatibility path when the host has a newer protobuf runtime. + os.environ.setdefault("PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION", "python") + try: + from tensorboard.backend.event_processing.event_accumulator import EventAccumulator + except ImportError: + return [] + + by_key: dict[tuple[str, int], dict[str, Any]] = {} + for path in event_paths: + accumulator = EventAccumulator(str(path), size_guidance={"scalars": 0}) + try: + accumulator.Reload() + except Exception as exc: + _fail(f"cannot load TensorBoard event file {path}: {exc}") + for tag in accumulator.Tags().get("scalars", []): + for event in accumulator.Scalars(tag): + _validate_finite(event.value, f"{path}:{tag}:step={event.step}") + row = { + "tag": tag, + "step": int(event.step), + "value": float(event.value), + "wall_time": float(event.wall_time), + "source_file": str(path.relative_to(run_dir)), + } + key = tag, int(event.step) + if key not in by_key or row["wall_time"] >= by_key[key]["wall_time"]: + by_key[key] = row + return sorted(by_key.values(), key=lambda row: (row["tag"], row["step"])) + + +def _parse_nvml_rows(run_dir: Path) -> list[dict[str, Any]]: + path = run_dir / "telemetry" / "nvidia-smi.csv" + if not path.is_file(): + return [] + + rows = [] + with path.open(encoding="utf-8", newline="") as reader: + for line_number, values in enumerate(csv.reader(reader), start=1): + if not values: + continue + if len(values) != len(NVML_COLUMNS): + _fail(f"{path}:{line_number} expected {len(NVML_COLUMNS)} columns, got {len(values)}") + row = {key: value.strip() for key, value in zip(NVML_COLUMNS, values, strict=True)} + for key in ( + "gpu_index", + "temperature_c", + "sm_clock_mhz", + "memory_clock_mhz", + "gpu_util_percent", + "memory_util_percent", + "memory_used_mib", + "power_w", + ): + try: + row[key] = float(row[key]) + except ValueError as exc: + _fail(f"{path}:{line_number} has non-numeric {key}={row[key]!r}: {exc}") + try: + row["wall_time"] = datetime.strptime( + row["timestamp"], + "%Y/%m/%d %H:%M:%S.%f", + ).timestamp() + except ValueError: + row["wall_time"] = None + _validate_finite(row, f"{path}:{line_number}") + rows.append(row) + return rows + + +def _parse_windows(value: str) -> tuple[tuple[int, int], ...]: + windows = [] + for item in value.split(","): + try: + start_text, end_text = item.strip().split("-", maxsplit=1) + start, end = int(start_text), int(end_text) + except ValueError: + _fail(f"invalid steady window {item!r}; expected START-END") + if start < 0 or end < start: + _fail(f"invalid steady window {item!r}; require 0 <= START <= END") + windows.append((start, end)) + if not windows: + _fail("at least one steady window is required") + flattened = [step for start, end in windows for step in range(start, end + 1)] + if len(flattened) != len(set(flattened)): + _fail(f"steady windows overlap: {windows}") + return tuple(windows) + + +def _stable_steps(windows: Sequence[tuple[int, int]]) -> set[int]: + return {step for start, end in windows for step in range(start, end + 1)} + + +def _measurement_scope(windows: Sequence[tuple[int, int]]) -> str: + steps = sorted(_stable_steps(windows)) + if steps == [0]: + return "fresh_process_first_step" + if len(steps) >= 3 and steps[0] > 0: + return "steady_state" + return "selected_steps" + + +def _scalar_map(rows: Sequence[dict[str, Any]], tag: str) -> dict[int, float]: + return {int(row["step"]): float(row["value"]) for row in rows if row["tag"] == tag} + + +def _aggregate_throughput( + scalar_rows: Sequence[dict[str, Any]], + throughput_tag: str, + windows: Sequence[tuple[int, int]], +) -> float | None: + step_time = _scalar_map(scalar_rows, "perf/step_time") + throughput = _scalar_map(scalar_rows, throughput_tag) + steps = sorted(_stable_steps(windows).intersection(step_time, throughput)) + if not steps: + return None + total_time = sum(step_time[step] for step in steps) + if total_time <= 0: + _fail(f"{throughput_tag} has non-positive aggregate steady step time") + return sum(throughput[step] * step_time[step] for step in steps) / total_time + + +def _aggregate_samples_per_second( + scalar_rows: Sequence[dict[str, Any]], + windows: Sequence[tuple[int, int]], + samples_per_step: int, +) -> float | None: + step_time = _scalar_map(scalar_rows, "perf/step_time") + steps = sorted(_stable_steps(windows).intersection(step_time)) + if not steps: + return None + total_time = sum(step_time[step] for step in steps) + if total_time <= 0: + _fail("perf/step_time has non-positive aggregate steady time") + return samples_per_step * len(steps) / total_time + + +def _metric_summary( + scalar_rows: Sequence[dict[str, Any]], + windows: Sequence[tuple[int, int]], +) -> dict[str, Any]: + steps = _stable_steps(windows) + tags = sorted({row["tag"] for row in scalar_rows}) + summary: dict[str, Any] = {} + for tag in tags: + values = [float(row["value"]) for row in scalar_rows if row["tag"] == tag and row["step"] in steps] + if not values: + continue + ordered = sorted(values) + summary[tag] = { + "count": len(values), + "mean": statistics.fmean(values), + "min": ordered[0], + "max": ordered[-1], + "p50": statistics.median(ordered), + "p95": ordered[math.ceil(0.95 * len(ordered)) - 1], + } + for tag in ("perf/step_token_per_s", "perf/step_resp_token_per_s"): + aggregate = _aggregate_throughput(scalar_rows, tag, windows) + if aggregate is not None: + summary.setdefault(tag, {})["aggregate"] = aggregate + return summary + + +def _steady_wall_time_intervals( + scalar_rows: Sequence[dict[str, Any]], + windows: Sequence[tuple[int, int]], +) -> list[tuple[float, float]]: + """Map registered steady steps to wall-clock intervals using the + TensorBoard step end time and perf/step_time duration.""" + steady_steps = _stable_steps(windows) + intervals = [] + for row in scalar_rows: + if row["tag"] != "perf/step_time" or int(row["step"]) not in steady_steps: + continue + duration_s = float(row["value"]) + wall_time = row.get("wall_time") + if duration_s <= 0: + _fail(f"perf/step_time step={row['step']} must be positive, got {duration_s}") + if not isinstance(wall_time, (int, float)) or not math.isfinite(wall_time): + _fail(f"perf/step_time step={row['step']} is missing a finite TensorBoard wall_time") + intervals.append((float(wall_time) - duration_s, float(wall_time))) + return sorted(intervals) + + +def _nvml_summary( + rows: Sequence[dict[str, Any]], + *, + steady_intervals: Sequence[tuple[float, float]], +) -> dict[str, Any]: + if not rows: + return {} + + def is_steady(row: dict[str, Any]) -> bool: + wall_time = row.get("wall_time") + return isinstance(wall_time, (int, float)) and any( + start <= float(wall_time) <= end for start, end in steady_intervals + ) + + by_gpu: dict[int, list[dict[str, Any]]] = defaultdict(list) + for row in rows: + by_gpu[int(row["gpu_index"])].append(row) + per_gpu = {} + for gpu_index, gpu_rows in sorted(by_gpu.items()): + full_utilization = [float(row["gpu_util_percent"]) for row in gpu_rows] + steady_rows = [row for row in gpu_rows if is_steady(row)] + steady_utilization = [float(row["gpu_util_percent"]) for row in steady_rows] + per_gpu[str(gpu_index)] = { + "full_run_sample_count": len(gpu_rows), + "full_run_mean_gpu_util_percent": statistics.fmean(full_utilization), + "steady_sample_count": len(steady_rows), + "steady_mean_gpu_util_percent": (statistics.fmean(steady_utilization) if steady_utilization else None), + "steady_idle_ratio_below_10_percent": ( + sum(value < 10 for value in steady_utilization) / len(steady_utilization) + if steady_utilization + else None + ), + "peak_memory_used_mib": max(float(row["memory_used_mib"]) for row in gpu_rows), + "mean_power_w": statistics.fmean(float(row["power_w"]) for row in gpu_rows), + } + steady_gpu_utilization = [ + item["steady_mean_gpu_util_percent"] + for item in per_gpu.values() + if item["steady_mean_gpu_util_percent"] is not None + ] + steady_idle_ratios = [ + item["steady_idle_ratio_below_10_percent"] + for item in per_gpu.values() + if item["steady_idle_ratio_below_10_percent"] is not None + ] + return { + "gpu_count": len(per_gpu), + "per_gpu": per_gpu, + "peak_memory_used_mib": max(item["peak_memory_used_mib"] for item in per_gpu.values()), + "full_run_mean_gpu_util_percent": statistics.fmean( + item["full_run_mean_gpu_util_percent"] for item in per_gpu.values() + ), + "steady_mean_gpu_util_percent": (statistics.fmean(steady_gpu_utilization) if steady_gpu_utilization else None), + "steady_idle_ratio_below_10_percent": (statistics.fmean(steady_idle_ratios) if steady_idle_ratios else None), + "steady_wall_time_intervals": [list(interval) for interval in steady_intervals], + } + + +def _write_csv(path: Path, rows: Sequence[dict[str, Any]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + if not rows: + path.write_text("", encoding="utf-8") + return + fieldnames = sorted({key for row in rows for key in row}) + with path.open("w", encoding="utf-8", newline="") as writer_file: + writer = csv.DictWriter(writer_file, fieldnames=fieldnames) + writer.writeheader() + for row in rows: + writer.writerow( + { + key: ( + json.dumps(value, sort_keys=True, separators=(",", ":")) + if isinstance(value, (dict, list, tuple)) + else value + ) + for key, value in row.items() + } + ) + + +def analyze_run( + run_dir: Path, + *, + windows: Sequence[tuple[int, int]] = DEFAULT_WINDOWS, + expected_samples: int = 256, + expected_actor_chunks: int = 2, + write_outputs: bool = True, + require_reproducibility_artifacts: bool = False, +) -> RunAnalysis: + run_dir = run_dir.resolve() + manifest = _load_manifest(run_dir) + _validate_run_manifest(manifest, run_dir=run_dir, windows=windows) + if require_reproducibility_artifacts: + _require_reproducibility_artifacts(run_dir) + trace_rows = _load_trace_rows(run_dir) + pipeline_flag = manifest["hybrid_pipeline_forward"] + if type(pipeline_flag) is not int or pipeline_flag not in (0, 1): + _fail(f"{run_dir} hybrid_pipeline_forward must be integer 0 or 1, got {pipeline_flag!r}") + pipeline_enabled = bool(pipeline_flag) + overlap_flag = manifest.get("hybrid_pipeline_overlap", pipeline_flag) + if type(overlap_flag) is not int or overlap_flag not in (0, 1): + _fail(f"{run_dir} hybrid_pipeline_overlap must be integer 0 or 1, got {overlap_flag!r}") + pipeline_overlap_enabled = bool(overlap_flag) + reuse_flag = manifest.get("hybrid_reuse_train_logprobs", 0) + if type(reuse_flag) is not int or reuse_flag not in (0, 1): + _fail(f"{run_dir} hybrid_reuse_train_logprobs must be integer 0 or 1, got {reuse_flag!r}") + reuse_train_forward_log_probs = bool(reuse_flag) + if reuse_train_forward_log_probs and pipeline_enabled: + _fail(f"{run_dir} cannot combine train-forward log-prob reuse with pipeline forwarding") + if manifest["global_batch_size"] != expected_samples: + _fail( + f"{run_dir} expected_samples={expected_samples} disagrees with " + f"manifest global_batch_size={manifest['global_batch_size']}" + ) + if manifest["num_iters_per_train_update"] != expected_actor_chunks: + _fail( + f"{run_dir} expected_actor_chunks={expected_actor_chunks} disagrees with " + f"manifest num_iters_per_train_update={manifest['num_iters_per_train_update']}" + ) + condition = str(manifest["condition"]) + if condition not in {"baseline", "experiment"}: + _fail(f"{run_dir} has unsupported condition {condition!r}; expected 'baseline' or 'experiment'") + if condition == "baseline" and pipeline_enabled and pipeline_overlap_enabled: + _fail(f"{run_dir} is labeled baseline but both chunk forwarding and producer overlap are enabled") + if condition == "experiment" and (not pipeline_enabled or not pipeline_overlap_enabled): + _fail(f"{run_dir} is labeled experiment but chunk forwarding with producer overlap is not enabled") + rollout_rows, actor_rank_rows, trace_summary = _analyze_trace( + trace_rows, + pipeline_enabled=pipeline_enabled, + pipeline_overlap_enabled=pipeline_overlap_enabled, + reuse_train_forward_log_probs=reuse_train_forward_log_probs, + expected_samples=expected_samples, + expected_actor_chunks=expected_actor_chunks, + ) + scalar_rows = _load_tensorboard_scalars(run_dir) + nvml_rows = _parse_nvml_rows(run_dir) + steady_wall_time_intervals = _steady_wall_time_intervals(scalar_rows, windows) + metrics = _metric_summary(scalar_rows, windows) + samples_per_second = _aggregate_samples_per_second(scalar_rows, windows, expected_samples) + if samples_per_second is not None: + metrics["perf/wall_clock_samples_per_s"] = {"aggregate": samples_per_second} + summary = { + "run_dir": str(run_dir), + "condition": manifest["condition"], + "seed": manifest["seed"], + "measurement_scope": _measurement_scope(windows), + "measurement_windows": [list(window) for window in windows], + "hybrid_pipeline_forward": pipeline_enabled, + "hybrid_pipeline_overlap": pipeline_overlap_enabled, + "hybrid_reuse_train_logprobs": reuse_train_forward_log_probs, + "steady_windows": [list(window) for window in windows], + "trace": trace_summary, + "metrics": metrics, + "nvml": _nvml_summary(nvml_rows, steady_intervals=steady_wall_time_intervals), + "validation": "passed", + } + + if write_outputs: + output_dir = run_dir / "analysis" + _write_csv(output_dir / "trace_events.csv", trace_rows) + _write_csv(output_dir / "rollout_summary.csv", rollout_rows) + _write_csv(output_dir / "actor_rank_summary.csv", actor_rank_rows) + _write_csv(output_dir / "tensorboard_scalars.csv", scalar_rows) + _write_csv(output_dir / "nvml_samples.csv", nvml_rows) + (output_dir / "summary.json").write_text( + json.dumps(summary, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + return RunAnalysis( + run_dir=run_dir, + manifest=manifest, + trace_rows=rollout_rows, + actor_rank_rows=actor_rank_rows, + scalar_rows=scalar_rows, + nvml_rows=nvml_rows, + summary=summary, + ) + + +def _geometric_mean(values: Iterable[float]) -> float: + values = list(values) + if not values or any(value <= 0 for value in values): + _fail(f"geometric mean requires positive values, got {values}") + return math.exp(statistics.fmean(math.log(value) for value in values)) + + +def _distribution_summary(values: Iterable[float]) -> dict[str, float | int]: + samples = list(values) + if not samples: + _fail("distribution summary requires at least one value") + mean = statistics.fmean(samples) + stddev = statistics.pstdev(samples) + return { + "count": len(samples), + "mean": mean, + "median": statistics.median(samples), + "min": min(samples), + "max": max(samples), + "population_stddev": stddev, + "coefficient_of_variation": stddev / abs(mean) if mean else 0.0, + } + + +def _steady_condition(analysis: RunAnalysis) -> str: + manifest = analysis.manifest + missing = sorted( + {"mm_processor_pool_size", "hybrid_reuse_train_logprobs", "hybrid_pipeline_forward"} - set(manifest) + ) + if missing: + _fail(f"{analysis.run_dir} is missing steady attribution controls {missing}") + controls = tuple( + manifest[key] for key in ("mm_processor_pool_size", "hybrid_reuse_train_logprobs", "hybrid_pipeline_forward") + ) + if any(type(value) is not int for value in controls): + _fail(f"{analysis.run_dir} steady attribution controls must be integers, got {controls}") + matches = [label for label, expected in STEADY_CAMPAIGN_CONDITIONS.items() if controls == expected] + if not matches: + _fail(f"{analysis.run_dir} has unsupported steady attribution controls {controls}") + label = matches[0] + expected_overlap = int(label == "P+S") + overlap = manifest.get("hybrid_pipeline_overlap") + if type(overlap) is not int or overlap != expected_overlap: + _fail( + f"{analysis.run_dir} condition {label} requires " + f"hybrid_pipeline_overlap={expected_overlap}, got {overlap!r}" + ) + expected_manifest_condition = "experiment" if label == "P+S" else "baseline" + if manifest["condition"] != expected_manifest_condition: + _fail( + f"{analysis.run_dir} controls identify {label}, but manifest condition is " + f"{manifest['condition']!r}; expected {expected_manifest_condition!r}" + ) + return label + + +def _steady_metric_value(analysis: RunAnalysis, tag: str) -> float: + metric = analysis.summary["metrics"].get(tag) + if not metric: + _fail(f"{analysis.run_dir} is missing steady metric {tag}") + key = ( + "aggregate" + if tag in {"perf/step_token_per_s", "perf/step_resp_token_per_s", "perf/wall_clock_samples_per_s"} + else "mean" + ) + value = metric.get(key) + if not isinstance(value, (int, float)) or isinstance(value, bool) or not math.isfinite(value): + _fail(f"{analysis.run_dir} has invalid steady metric {tag}.{key}={value!r}") + return float(value) + + +def _steady_input_workload( + analysis: RunAnalysis, + steady_steps: set[int], +) -> list[tuple[Any, ...]]: + # Prompt/image identity is the controlled workload. Generated response + # lengths and packed tensor bytes remain observable outputs and are + # reported per run, but can vary with sampling and dynamic microbatching. + return sorted( + ( + row["producer_global_indexes_fingerprint"], + int(row["actor_fetch_samples"]), + ) + for row in analysis.trace_rows + if int(row["rollout_id"]) in steady_steps + ) + + +def _steady_pair_summary( + by_seed: dict[Any, dict[str, RunAnalysis]], + *, + candidate: str, + reference: str, + steady_steps: set[int], +) -> dict[str, Any]: + paired_runs = [] + ratio_tags = ( + "perf/step_token_per_s", + "perf/step_resp_token_per_s", + "perf/wall_clock_samples_per_s", + ) + delta_tags = ( + "rollout/raw_reward", + "rollout/image_count/mean", + "train/loss", + "train/grad_norm", + "train/ppo_kl", + ) + for seed, conditions in sorted(by_seed.items(), key=lambda item: str(item[0])): + reference_run = conditions[reference] + candidate_run = conditions[candidate] + reference_workload = _steady_input_workload(reference_run, steady_steps) + candidate_workload = _steady_input_workload(candidate_run, steady_steps) + if reference_workload != candidate_workload: + _fail( + f"seed {seed} steady workload differs for {candidate} versus {reference}: " + f"{candidate_workload=} {reference_workload=}" + ) + row: dict[str, Any] = { + "seed": seed, + "reference": reference, + "candidate": candidate, + } + for tag in ratio_tags: + reference_value = _steady_metric_value(reference_run, tag) + candidate_value = _steady_metric_value(candidate_run, tag) + if reference_value <= 0 or candidate_value <= 0: + _fail(f"{tag} must be positive for seed {seed}, got {reference_value}, {candidate_value}") + row[f"{tag}:reference"] = reference_value + row[f"{tag}:candidate"] = candidate_value + row[f"{tag}:improvement"] = candidate_value / reference_value - 1 + reference_step_time = _steady_metric_value(reference_run, "perf/step_time") + candidate_step_time = _steady_metric_value(candidate_run, "perf/step_time") + if reference_step_time <= 0 or candidate_step_time <= 0: + _fail(f"perf/step_time must be positive for seed {seed}") + row["perf/step_time:reference"] = reference_step_time + row["perf/step_time:candidate"] = candidate_step_time + row["perf/step_time:reduction"] = 1 - candidate_step_time / reference_step_time + for tag in delta_tags: + reference_value = _steady_metric_value(reference_run, tag) + candidate_value = _steady_metric_value(candidate_run, tag) + row[f"{tag}:reference"] = reference_value + row[f"{tag}:candidate"] = candidate_value + row[f"{tag}:delta"] = candidate_value - reference_value + row["nvml/steady_mean_gpu_util_percent:reference"] = reference_run.summary["nvml"].get( + "steady_mean_gpu_util_percent" + ) + row["nvml/steady_mean_gpu_util_percent:candidate"] = candidate_run.summary["nvml"].get( + "steady_mean_gpu_util_percent" + ) + row["nvml/peak_memory_used_mib:reference"] = reference_run.summary["nvml"].get("peak_memory_used_mib") + row["nvml/peak_memory_used_mib:candidate"] = candidate_run.summary["nvml"].get("peak_memory_used_mib") + paired_runs.append(row) + + result: dict[str, Any] = {"paired_runs": paired_runs, "paired_seed_count": len(paired_runs)} + for tag in ratio_tags: + ratios = [1 + row[f"{tag}:improvement"] for row in paired_runs] + result[f"{tag}:geomean_improvement"] = _geometric_mean(ratios) - 1 + step_time_ratios = [1 - row["perf/step_time:reduction"] for row in paired_runs] + result["perf/step_time:geomean_reduction"] = 1 - _geometric_mean(step_time_ratios) + return result + + +def _build_steady_campaign( + analyses: Sequence[RunAnalysis], + *, + windows: Sequence[tuple[int, int]], + expected_gpu_count: int, + enforce_targets: bool, +) -> dict[str, Any]: + if _measurement_scope(windows) != "steady_state": + _fail("steady campaign requires warmup-excluded steady-state measurement windows") + steady_steps = _stable_steps(windows) + if len(steady_steps) < 30: + _fail(f"steady campaign requires at least 30 measured optimizer steps, got {len(steady_steps)}") + if expected_gpu_count <= 0: + _fail(f"expected_gpu_count must be positive, got {expected_gpu_count}") + + by_condition: dict[str, list[RunAnalysis]] = defaultdict(list) + for analysis in analyses: + by_condition[_steady_condition(analysis)].append(analysis) + if set(by_condition) != set(STEADY_CAMPAIGN_CONDITIONS): + _fail(f"steady campaign requires B/P/P+R/P+S, got {sorted(by_condition)}") + + comparison_fields = COMPARISON_FIXED_MANIFEST_FIELDS + COMPARISON_WORKLOAD_MANIFEST_FIELDS + missing_fields = { + str(analysis.run_dir): sorted(set(comparison_fields) - set(analysis.manifest)) + for analysis in analyses + if set(comparison_fields) - set(analysis.manifest) + } + if missing_fields: + _fail(f"steady campaign manifests are missing fixed fields: {missing_fields}") + for field in comparison_fields: + values = {json.dumps(analysis.manifest[field], sort_keys=True) for analysis in analyses} + if len(values) != 1: + _fail(f"steady campaign requires identical manifest field {field!r}, got {sorted(values)}") + + by_seed: dict[Any, dict[str, RunAnalysis]] = defaultdict(dict) + for condition, condition_runs in by_condition.items(): + for analysis in condition_runs: + seed = analysis.manifest["seed"] + if condition in by_seed[seed]: + _fail(f"duplicate steady condition {condition} for seed {seed}") + by_seed[seed][condition] = analysis + incomplete = { + seed: sorted(conditions) + for seed, conditions in by_seed.items() + if set(conditions) != set(STEADY_CAMPAIGN_CONDITIONS) + } + if incomplete: + _fail(f"steady campaign has unpaired condition seeds: {incomplete}") + if enforce_targets and len(by_seed) < 2: + _fail(f"steady campaign requires at least two independent seeds, got {len(by_seed)}") + + condition_summaries: dict[str, Any] = {} + for condition, condition_runs in sorted(by_condition.items()): + run_summaries = [] + for analysis in sorted(condition_runs, key=lambda item: str(item.manifest["seed"])): + context = f"{condition} seed={analysis.manifest['seed']}" + if int(analysis.manifest["num_rollout"]) < 40: + _fail(f"{context} requires at least 40 optimizer steps") + rollout_count = int(analysis.summary["trace"]["rollout_count"]) + if rollout_count != int(analysis.manifest["num_rollout"]): + _fail( + f"{context} optimizer update conservation failed: " + f"trace={rollout_count}, manifest={analysis.manifest['num_rollout']}" + ) + trace_steps = {int(row["rollout_id"]) for row in analysis.trace_rows} + missing_trace_steps = sorted(steady_steps - trace_steps) + if missing_trace_steps: + _fail(f"{context} is missing steady trace steps {missing_trace_steps}") + for tag in STEADY_CAMPAIGN_REQUIRED_TAGS: + metric_steps = { + int(row["step"]) + for row in analysis.scalar_rows + if row["tag"] == tag and int(row["step"]) in steady_steps + } + missing_metric_steps = sorted(steady_steps - metric_steps) + if missing_metric_steps: + _fail(f"{context} {tag} is missing steady steps {missing_metric_steps}") + gpu_count = analysis.summary["nvml"].get("gpu_count") + if gpu_count != expected_gpu_count: + _fail(f"{context} expected NVML data for {expected_gpu_count} GPUs, got {gpu_count!r}") + per_gpu = analysis.summary["nvml"].get("per_gpu", {}) + missing_nvml = sorted( + gpu for gpu, summary in per_gpu.items() if int(summary.get("steady_sample_count", 0)) <= 0 + ) + if len(per_gpu) != expected_gpu_count or missing_nvml: + _fail(f"{context} has incomplete steady per-GPU NVML data: {missing_nvml}") + + steady_rows = [row for row in analysis.trace_rows if int(row["rollout_id"]) in steady_steps] + expected_samples = int(analysis.manifest["global_batch_size"]) + if any(int(row["actor_fetch_samples"]) != expected_samples for row in steady_rows): + _fail(f"{context} failed steady sample conservation") + run_summaries.append( + { + "run_dir": str(analysis.run_dir), + "seed": analysis.manifest["seed"], + "optimizer_update_count": rollout_count, + "measured_optimizer_step_count": len(steady_steps), + "perf/step_token_per_s": _steady_metric_value(analysis, "perf/step_token_per_s"), + "perf/wall_clock_samples_per_s": _steady_metric_value(analysis, "perf/wall_clock_samples_per_s"), + "windows": [ + { + "start": start, + "end": end, + "perf/step_token_per_s": _aggregate_throughput( + analysis.scalar_rows, + "perf/step_token_per_s", + ((start, end),), + ), + "perf/wall_clock_samples_per_s": _aggregate_samples_per_second( + analysis.scalar_rows, + ((start, end),), + expected_samples, + ), + } + for start, end in windows + ], + "perf/step_time": analysis.summary["metrics"]["perf/step_time"], + "rollout/raw_reward": analysis.summary["metrics"]["rollout/raw_reward"], + "rollout/image_count/mean": analysis.summary["metrics"]["rollout/image_count/mean"], + "train/loss": analysis.summary["metrics"]["train/loss"], + "train/grad_norm": analysis.summary["metrics"]["train/grad_norm"], + "train/ppo_kl": analysis.summary["metrics"]["train/ppo_kl"], + "steady_actor_fetch_samples": sum(int(row["actor_fetch_samples"]) for row in steady_rows), + "steady_actor_total_tokens": sum(int(row["actor_total_tokens"]) for row in steady_rows), + "steady_actor_response_tokens": sum(int(row["actor_response_tokens"]) for row in steady_rows), + "steady_actor_multimodal_tensor_bytes": sum( + int(row["actor_multimodal_tensor_bytes"]) for row in steady_rows + ), + "nvml": analysis.summary["nvml"], + } + ) + condition_summaries[condition] = { + "runs": run_summaries, + "perf/step_token_per_s": _distribution_summary(row["perf/step_token_per_s"] for row in run_summaries), + "perf/wall_clock_samples_per_s": _distribution_summary( + row["perf/wall_clock_samples_per_s"] for row in run_summaries + ), + "windows": [ + { + "start": start, + "end": end, + "perf/step_token_per_s": _distribution_summary( + row["windows"][index]["perf/step_token_per_s"] for row in run_summaries + ), + "perf/wall_clock_samples_per_s": _distribution_summary( + row["windows"][index]["perf/wall_clock_samples_per_s"] for row in run_summaries + ), + } + for index, (start, end) in enumerate(windows) + ], + } + + comparisons = {} + for candidate, reference in ( + ("P", "B"), + ("P+R", "B"), + ("P+S", "B"), + ("P+R", "P"), + ("P+S", "P"), + ): + comparisons[f"{candidate}_vs_{reference}"] = _steady_pair_summary( + by_seed, + candidate=candidate, + reference=reference, + steady_steps=steady_steps, + ) + return { + "measurement_scope": "steady_state", + "claim_limit": None, + "steady_windows": [list(window) for window in windows], + "warmup_excluded_step_count": min(steady_steps), + "measured_optimizer_step_count": len(steady_steps), + "independent_seed_count": len(by_seed), + "seeds": sorted(by_seed, key=str), + "conditions": condition_summaries, + "comparisons": comparisons, + "validation": "passed", + } + + +def _build_comparison( + analyses: Sequence[RunAnalysis], + *, + windows: Sequence[tuple[int, int]], + enforce_targets: bool, + expected_gpu_count: int = 8, +) -> dict[str, Any]: + if expected_gpu_count <= 0: + _fail(f"expected_gpu_count must be positive, got {expected_gpu_count}") + + by_condition: dict[str, list[RunAnalysis]] = defaultdict(list) + for analysis in analyses: + by_condition[str(analysis.manifest["condition"])].append(analysis) + if set(by_condition) != {"baseline", "experiment"}: + _fail(f"comparison requires conditions named 'baseline' and 'experiment', got {sorted(by_condition)}") + + comparison_fields = COMPARISON_FIXED_MANIFEST_FIELDS + COMPARISON_WORKLOAD_MANIFEST_FIELDS + missing_workload_fields = { + str(analysis.run_dir): sorted(set(COMPARISON_WORKLOAD_MANIFEST_FIELDS) - set(analysis.manifest)) + for analysis in analyses + if set(COMPARISON_WORKLOAD_MANIFEST_FIELDS) - set(analysis.manifest) + } + if missing_workload_fields: + _fail(f"comparison manifests are missing workload fields: {missing_workload_fields}") + + for field in comparison_fields: + values = {json.dumps(analysis.manifest[field], sort_keys=True) for analysis in analyses} + if len(values) != 1: + _fail(f"comparison requires identical manifest field {field!r}, got {sorted(values)}") + + by_seed: dict[Any, dict[str, RunAnalysis]] = defaultdict(dict) + for condition, condition_runs in by_condition.items(): + for analysis in condition_runs: + seed = analysis.manifest["seed"] + if condition in by_seed[seed]: + _fail(f"duplicate {condition} run for seed {seed}") + by_seed[seed][condition] = analysis + incomplete = {seed: sorted(pair) for seed, pair in by_seed.items() if set(pair) != {"baseline", "experiment"}} + if incomplete: + _fail(f"comparison has unpaired seeds: {incomplete}") + + steady_steps = _stable_steps(windows) + if enforce_targets: + missing_overlap_mode = [ + str(analysis.run_dir) for analysis in analyses if "hybrid_pipeline_overlap" not in analysis.manifest + ] + if missing_overlap_mode: + _fail( + f"performance targets require an explicit hybrid_pipeline_overlap manifest field: {missing_overlap_mode}" + ) + invalid_modes = { + str(analysis.run_dir): { + "condition": analysis.manifest["condition"], + "hybrid_pipeline_forward": analysis.manifest["hybrid_pipeline_forward"], + "hybrid_pipeline_overlap": analysis.manifest["hybrid_pipeline_overlap"], + } + for analysis in analyses + if analysis.manifest["hybrid_pipeline_forward"] not in (1, True) + or bool(analysis.manifest["hybrid_pipeline_overlap"]) != (analysis.manifest["condition"] == "experiment") + } + if invalid_modes: + _fail( + "performance targets require schedule-matched chunk forwarding with overlap disabled " + f"for baseline and enabled for experiment: {invalid_modes}" + ) + required_tags = PERFORMANCE_TAGS + CORRECTNESS_GUARDRAIL_TAGS + for analysis in analyses: + context = f"{analysis.manifest['condition']} seed={analysis.manifest['seed']}" + trace_steps = {int(row["rollout_id"]) for row in analysis.trace_rows} + missing_trace_steps = sorted(steady_steps - trace_steps) + if missing_trace_steps: + _fail(f"{context} is missing steady trace steps {missing_trace_steps}") + for tag in required_tags: + metric_steps = { + int(row["step"]) + for row in analysis.scalar_rows + if row["tag"] == tag and int(row["step"]) in steady_steps + } + missing_metric_steps = sorted(steady_steps - metric_steps) + if missing_metric_steps: + _fail(f"{context} {tag} is missing steady steps {missing_metric_steps}") + gpu_count = analysis.summary["nvml"].get("gpu_count") + if gpu_count != expected_gpu_count: + _fail(f"{context} expected NVML data for {expected_gpu_count} GPUs, got {gpu_count!r}") + per_gpu_nvml = analysis.summary["nvml"].get("per_gpu", {}) + if len(per_gpu_nvml) != expected_gpu_count: + _fail( + f"{context} expected per-GPU NVML summaries for {expected_gpu_count} GPUs, got {len(per_gpu_nvml)}" + ) + missing_steady_nvml = sorted( + gpu_index + for gpu_index, gpu_summary in per_gpu_nvml.items() + if int(gpu_summary.get("steady_sample_count", 0)) <= 0 + ) + if missing_steady_nvml: + _fail( + f"{context} has no NVML samples aligned to registered steady steps for GPUs {missing_steady_nvml}" + ) + steady_trace_rows = [row for row in analysis.trace_rows if int(row["rollout_id"]) in steady_steps] + observed_lead = max(row["producer_lead_at_first_forward"] for row in steady_trace_rows) + max_staleness = int(analysis.manifest["max_staleness"]) + if observed_lead > max_staleness: + _fail( + f"{context} observed producer lead exceeds configured " + f"max_staleness={max_staleness}: {observed_lead}" + ) + + paired = [] + window_speedups = [] + for seed, pair in sorted(by_seed.items(), key=lambda item: str(item[0])): + row: dict[str, Any] = {"seed": seed} + baseline_fingerprints = { + int(trace_row["rollout_id"]): trace_row["producer_global_indexes_fingerprint"] + for trace_row in pair["baseline"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + } + experiment_fingerprints = { + int(trace_row["rollout_id"]): trace_row["producer_global_indexes_fingerprint"] + for trace_row in pair["experiment"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + } + if baseline_fingerprints != experiment_fingerprints: + _fail( + f"seed {seed} producer global-index fingerprints differ between " + f"baseline and experiment: baseline={baseline_fingerprints}, " + f"experiment={experiment_fingerprints}" + ) + for tag in COMPARISON_PERFORMANCE_TAGS: + baseline_metrics = pair["baseline"].summary["metrics"].get(tag, {}) + experiment_metrics = pair["experiment"].summary["metrics"].get(tag, {}) + key = "aggregate" if "per_s" in tag else "mean" + baseline_value = baseline_metrics.get(key) + experiment_value = experiment_metrics.get(key) + if baseline_value is None or experiment_value is None: + continue + if baseline_value <= 0 or experiment_value <= 0: + _fail(f"{tag} must be positive for seed {seed}, got {baseline_value=} and {experiment_value=}") + row[f"{tag}:baseline"] = baseline_value + row[f"{tag}:experiment"] = experiment_value + if "time" in tag: + row[f"{tag}:improvement"] = 1 - experiment_value / baseline_value + else: + row[f"{tag}:improvement"] = experiment_value / baseline_value - 1 + + for window_start, window_end in windows: + window = ((window_start, window_end),) + baseline_throughput = _aggregate_throughput( + pair["baseline"].scalar_rows, + "perf/step_token_per_s", + window, + ) + experiment_throughput = _aggregate_throughput( + pair["experiment"].scalar_rows, + "perf/step_token_per_s", + window, + ) + if baseline_throughput is not None and experiment_throughput is not None: + if baseline_throughput <= 0 or experiment_throughput <= 0: + _fail(f"window {window_start}-{window_end} throughput must be positive for seed {seed}") + window_speedups.append( + { + "seed": seed, + "window_start": window_start, + "window_end": window_end, + "baseline": baseline_throughput, + "experiment": experiment_throughput, + "speedup": experiment_throughput / baseline_throughput - 1, + } + ) + + baseline_step_p95 = pair["baseline"].summary["metrics"].get("perf/step_time", {}).get("p95") + experiment_step_p95 = pair["experiment"].summary["metrics"].get("perf/step_time", {}).get("p95") + if baseline_step_p95 is not None and experiment_step_p95 is not None: + if baseline_step_p95 <= 0 or experiment_step_p95 <= 0: + _fail( + f"perf/step_time p95 must be positive for seed {seed}, " + f"got {baseline_step_p95=} and {experiment_step_p95=}" + ) + row["perf/step_time:p95_baseline"] = baseline_step_p95 + row["perf/step_time:p95_experiment"] = experiment_step_p95 + row["perf/step_time:p95_regression"] = experiment_step_p95 / baseline_step_p95 - 1 + + for tag in CORRECTNESS_GUARDRAIL_TAGS: + baseline_value = pair["baseline"].summary["metrics"].get(tag, {}).get("mean") + experiment_value = pair["experiment"].summary["metrics"].get(tag, {}).get("mean") + if baseline_value is not None and experiment_value is not None: + row[f"{tag}:baseline"] = baseline_value + row[f"{tag}:experiment"] = experiment_value + row[f"{tag}:delta"] = experiment_value - baseline_value + + for field in ( + "actor_fetch_samples", + "actor_total_tokens", + "actor_response_tokens", + "actor_multimodal_tensor_bytes", + ): + baseline_total = sum( + int(trace_row[field]) + for trace_row in pair["baseline"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + ) + experiment_total = sum( + int(trace_row[field]) + for trace_row in pair["experiment"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + ) + row[f"{field}:baseline"] = baseline_total + row[f"{field}:experiment"] = experiment_total + row[f"{field}:relative_delta"] = ( + experiment_total / baseline_total - 1 + if baseline_total + else (0.0 if experiment_total == 0 else math.inf) + ) + + baseline_vram = pair["baseline"].summary["nvml"].get("peak_memory_used_mib") + experiment_vram = pair["experiment"].summary["nvml"].get("peak_memory_used_mib") + if baseline_vram is not None and experiment_vram is not None: + row["nvml_peak_memory_mib:baseline"] = baseline_vram + row["nvml_peak_memory_mib:experiment"] = experiment_vram + row["nvml_peak_memory_mib:delta"] = experiment_vram - baseline_vram + + baseline_lead = statistics.fmean( + trace_row["producer_lead_at_first_forward"] + for trace_row in pair["baseline"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + ) + experiment_lead = statistics.fmean( + trace_row["producer_lead_at_first_forward"] + for trace_row in pair["experiment"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + ) + row["producer_lead_at_first_forward:baseline"] = baseline_lead + row["producer_lead_at_first_forward:experiment"] = experiment_lead + row["producer_lead_at_first_forward:delta"] = experiment_lead - baseline_lead + paired.append(row) + + token_ratios = [ + 1 + row["perf/step_token_per_s:improvement"] for row in paired if "perf/step_token_per_s:improvement" in row + ] + phase_ratios = [ + 1 - row["perf/hybrid_phase1_time:improvement"] + for row in paired + if "perf/hybrid_phase1_time:improvement" in row + ] + experiment_overlap_rows = [ + row + for analysis in by_condition["experiment"] + for row in analysis.trace_rows + if int(row["rollout_id"]) in _stable_steps(windows) + ] + experiment_producer_overlap_by_run = {} + for analysis in by_condition["experiment"]: + rows = [row for row in analysis.trace_rows if int(row["rollout_id"]) in steady_steps] + experiment_producer_overlap_by_run[str(analysis.run_dir)] = ( + sum(bool(row["first_forward_before_last_put_start"]) for row in rows) / len(rows) if rows else None + ) + comparison = { + "measurement_scope": _measurement_scope(windows), + "claim_limit": ( + "Fresh-process first optimizer step only; do not report as steady-state training throughput." + if _measurement_scope(windows) == "fresh_process_first_step" + else None + ), + "paired_runs": paired, + "window_speedups": window_speedups, + "paired_seed_count": len(paired), + "token_throughput_geomean_speedup": (_geometric_mean(token_ratios) - 1 if token_ratios else None), + "hybrid_phase1_geomean_reduction": (1 - _geometric_mean(phase_ratios) if phase_ratios else None), + "experiment_steady_producer_overlap_ratio": ( + sum(bool(row["first_forward_before_last_put_start"]) for row in experiment_overlap_rows) + / len(experiment_overlap_rows) + if experiment_overlap_rows + else None + ), + "experiment_steady_producer_overlap_ratio_by_run": experiment_producer_overlap_by_run, + "distributions": { + "baseline_step_token_per_s": _distribution_summary( + row["perf/step_token_per_s:baseline"] for row in paired if "perf/step_token_per_s:baseline" in row + ), + "experiment_step_token_per_s": _distribution_summary( + row["perf/step_token_per_s:experiment"] for row in paired if "perf/step_token_per_s:experiment" in row + ), + "paired_step_token_per_s_speedup": _distribution_summary( + row["perf/step_token_per_s:improvement"] + for row in paired + if "perf/step_token_per_s:improvement" in row + ), + "paired_hybrid_phase1_reduction": _distribution_summary( + row["perf/hybrid_phase1_time:improvement"] + for row in paired + if "perf/hybrid_phase1_time:improvement" in row + ), + }, + } + + if enforce_targets: + if len(paired) < 2: + _fail(f"performance targets require at least two paired seeds, got {len(paired)}") + improvements = [row.get("perf/step_token_per_s:improvement") for row in paired] + if any(value is None for value in improvements): + _fail("performance targets require perf/step_token_per_s for every paired run") + if any("perf/wall_clock_samples_per_s:baseline" not in row for row in paired): + _fail("performance targets require wall-clock samples/s for every paired run") + if any(value <= 0 for value in improvements): + _fail(f"every paired token-throughput speedup must be positive, got {improvements}") + if comparison["token_throughput_geomean_speedup"] < 0.05: + _fail( + "token-throughput geometric-mean speedup is below 5%: " + f"{comparison['token_throughput_geomean_speedup']:.4%}" + ) + phase_improvements = [row.get("perf/hybrid_phase1_time:improvement") for row in paired] + if any(value is None for value in phase_improvements): + _fail("performance targets require perf/hybrid_phase1_time for every paired run") + if any(value < 0 for value in phase_improvements): + _fail(f"every paired Hybrid phase-1 result must be non-regressive, got {phase_improvements}") + if comparison["hybrid_phase1_geomean_reduction"] < 0.15: + _fail( + "Hybrid phase-1 geometric-mean reduction is below 15%: " + f"{comparison['hybrid_phase1_geomean_reduction']:.4%}" + ) + failing_overlap = { + run_dir: ratio + for run_dir, ratio in experiment_producer_overlap_by_run.items() + if ratio is None or ratio < 0.8 + } + if failing_overlap: + _fail(f"each experiment run requires at least 80% steady producer overlap, got {failing_overlap}") + + step_p95_regressions = [row.get("perf/step_time:p95_regression") for row in paired] + if any(value is None for value in step_p95_regressions): + _fail("performance targets require perf/step_time p95 for every paired run") + if any(value > 0.05 for value in step_p95_regressions): + _fail(f"paired step-time p95 regression exceeds 5%: {step_p95_regressions}") + + for row in paired: + seed = row["seed"] + if row["actor_fetch_samples:baseline"] != row["actor_fetch_samples:experiment"]: + _fail(f"seed {seed} actor fetch sample count changed") + deterministic = int(by_seed[seed]["baseline"].manifest["sglang_deterministic_inference"]) == 1 + for field in ( + "actor_total_tokens", + "actor_response_tokens", + "actor_multimodal_tensor_bytes", + ): + baseline_total = row[f"{field}:baseline"] + experiment_total = row[f"{field}:experiment"] + if baseline_total <= 0 or experiment_total <= 0: + _fail(f"seed {seed} {field} must be positive, got {baseline_total}, {experiment_total}") + relative_delta = row[f"{field}:relative_delta"] + if deterministic and baseline_total != experiment_total: + _fail( + f"seed {seed} {field} must match exactly under deterministic " + f"inference, got {baseline_total} and {experiment_total}" + ) + if not deterministic and abs(relative_delta) > 0.01: + _fail(f"seed {seed} {field} changed by more than 1%: {relative_delta:.4%}") + + vram_baseline = row.get("nvml_peak_memory_mib:baseline") + vram_delta = row.get("nvml_peak_memory_mib:delta") + if vram_baseline is None or vram_delta is None: + _fail(f"seed {seed} is missing paired NVML peak memory") + allowed_vram_delta = max(1024.0, 0.03 * vram_baseline) + if vram_delta > allowed_vram_delta: + _fail(f"seed {seed} peak VRAM increased by {vram_delta:.1f} MiB; allowed={allowed_vram_delta:.1f} MiB") + + accuracy_deltas = [row.get("rollout/raw_reward:delta") for row in paired] + if any(value is None for value in accuracy_deltas): + _fail("correctness targets require rollout/raw_reward for every paired run") + if any(value < -0.03 for value in accuracy_deltas): + _fail(f"a paired raw-reward/accuracy drop exceeds 3 percentage points: {accuracy_deltas}") + if statistics.fmean(accuracy_deltas) < -0.02: + _fail(f"mean paired raw-reward/accuracy drop exceeds 2 percentage points: {accuracy_deltas}") + + truncation_deltas = [row.get("rollout/truncated_ratio:delta") for row in paired] + if any(value is None for value in truncation_deltas): + _fail("correctness targets require rollout/truncated_ratio for every paired run") + if any(value > 0.02 for value in truncation_deltas): + _fail(f"a paired truncation-rate increase exceeds 2 percentage points: {truncation_deltas}") + + for analysis in analyses: + ppo_kl = analysis.summary["metrics"]["train/ppo_kl"]["mean"] + pg_clipfrac = analysis.summary["metrics"]["train/pg_clipfrac"]["mean"] + if abs(ppo_kl) > 1e-7: + _fail(f"{analysis.run_dir} same-weight train/ppo_kl exceeds 1e-7: {ppo_kl}") + if abs(pg_clipfrac) > 1e-7: + _fail(f"{analysis.run_dir} same-weight train/pg_clipfrac exceeds 1e-7: {pg_clipfrac}") + + staleness_deltas = [row["producer_lead_at_first_forward:delta"] for row in paired] + if any(value > 0.25 for value in staleness_deltas): + _fail(f"a paired average producer-lead increase exceeds 0.25: {staleness_deltas}") + + baseline_throughputs = [row["perf/step_token_per_s:baseline"] for row in paired] + if len(paired) == 2: + baseline_cv = statistics.pstdev(baseline_throughputs) / statistics.fmean(baseline_throughputs) + if baseline_cv > 0.05: + _fail( + f"two baseline runs have CV={baseline_cv:.2%} > 5%; " + "the preregistered protocol requires a third paired seed" + ) + return comparison + + +def _plot_comparison( + analyses: Sequence[RunAnalysis], + comparison: dict[str, Any], + output_dir: Path, + windows: Sequence[tuple[int, int]], +) -> list[str]: + try: + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + except ImportError as exc: + raise BenchmarkValidationError( + "comparison plot generation requires the optional dependency " + "'matplotlib'; install it in the benchmark environment" + ) from exc + + output_dir.mkdir(parents=True, exist_ok=True) + generated = [] + + def finish(filename: str) -> None: + path = output_dir / filename + plt.tight_layout() + plt.savefig(path, dpi=160) + plt.close() + generated.append(str(path)) + + figure, axes = plt.subplots(2, 1, figsize=(11, 9), sharex=True) + for axis, tag in zip( + axes, + ("perf/step_token_per_s", "perf/step_resp_token_per_s"), + strict=True, + ): + for analysis in analyses: + label = f"{analysis.manifest['condition']}-seed{analysis.manifest['seed']}" + values = [row for row in analysis.scalar_rows if row["tag"] == tag] + if values: + axis.plot( + [row["step"] for row in values], + [row["value"] for row in values], + marker="o", + label=label, + ) + for start, end in windows: + axis.axvspan(start, end, color="grey", alpha=0.08) + axis.set_ylabel(tag) + axis.legend() + axes[-1].set_xlabel("rollout / optimizer step") + figure.suptitle("Task 21 step token throughput") + finish("task21_step_throughput.png") + + figure, axes = plt.subplots(3, 1, figsize=(11, 12), sharex=True) + for analysis in analyses: + label = f"{analysis.manifest['condition']}-seed{analysis.manifest['seed']}" + axes[0].plot( + [row["rollout_id"] for row in analysis.trace_rows], + [row["producer_overlap_s"] for row in analysis.trace_rows], + marker="o", + label=label, + ) + phase1 = [row for row in analysis.scalar_rows if row["tag"] == "perf/hybrid_phase1_time"] + if phase1: + axes[1].plot( + [row["step"] for row in phase1], + [row["value"] for row in phase1], + marker="o", + label=label, + ) + axes[2].plot( + [row["rollout_id"] for row in analysis.trace_rows], + [row["producer_lead_at_first_forward"] for row in analysis.trace_rows], + marker="o", + label=label, + ) + axes[0].axhline(0, color="black", linewidth=0.8) + axes[0].set_ylabel("last put start - first actor forward start (s)") + axes[1].set_ylabel("perf/hybrid_phase1_time (s)") + max_staleness = int(analyses[0].manifest["max_staleness"]) + axes[2].axhline( + max_staleness, + color="red", + linestyle="--", + label=f"max_staleness={max_staleness}", + ) + axes[2].set_xlabel("rollout / optimizer step") + axes[2].set_ylabel("producer lead at actor forward (steps)") + for axis in axes: + axis.legend() + figure.suptitle("Task 21 producer / actor overlap and phase-1 time") + finish("task21_phase1_overlap.png") + + def numeric_or_nan(value: Any) -> float: + return float(value) if value is not None else float("nan") + + labels = [f"{analysis.manifest['condition']}-s{analysis.manifest['seed']}" for analysis in analyses] + utilization = [ + numeric_or_nan(analysis.summary["nvml"].get("steady_mean_gpu_util_percent")) for analysis in analyses + ] + idle_ratio = [ + numeric_or_nan(analysis.summary["nvml"].get("steady_idle_ratio_below_10_percent")) * 100 + for analysis in analyses + ] + vram = [numeric_or_nan(analysis.summary["nvml"].get("peak_memory_used_mib")) / 1024 for analysis in analyses] + figure, axes = plt.subplots(2, 2, figsize=(14, 9)) + for analysis in analyses: + label = f"{analysis.manifest['condition']}-seed{analysis.manifest['seed']}" + by_timestamp: dict[float, list[dict[str, Any]]] = defaultdict(list) + for row in analysis.nvml_rows: + if row["wall_time"] is not None: + by_timestamp[float(row["wall_time"])].append(row) + if by_timestamp: + start_time = min(by_timestamp) + timestamps = sorted(by_timestamp) + elapsed = [timestamp - start_time for timestamp in timestamps] + mean_utilization = [ + statistics.fmean(float(row["gpu_util_percent"]) for row in by_timestamp[timestamp]) + for timestamp in timestamps + ] + peak_vram = [ + max(float(row["memory_used_mib"]) for row in by_timestamp[timestamp]) / 1024 + for timestamp in timestamps + ] + axes[0, 0].plot(elapsed, mean_utilization, label=label) + axes[1, 0].plot(elapsed, peak_vram, label=label) + for start, end in analysis.summary["nvml"].get("steady_wall_time_intervals", []): + axes[0, 0].axvspan(start - start_time, end - start_time, color="grey", alpha=0.025) + axes[1, 0].axvspan(start - start_time, end - start_time, color="grey", alpha=0.025) + axes[0, 0].set_ylabel("mean GPU utilization (%)") + axes[0, 0].set_xlabel("seconds since first NVML sample") + axes[1, 0].set_ylabel("max per-GPU VRAM (GiB)") + axes[1, 0].set_xlabel("seconds since first NVML sample") + for axis in (axes[0, 0], axes[1, 0]): + handles, _ = axis.get_legend_handles_labels() + if handles: + axis.legend() + positions = list(range(len(labels))) + axes[0, 1].bar([position - 0.2 for position in positions], utilization, width=0.4, label="mean util") + axes[0, 1].bar([position + 0.2 for position in positions], idle_ratio, width=0.4, label="idle <10%") + axes[0, 1].set_xticks(positions, labels) + scope_label = "first-step" if _measurement_scope(windows) == "fresh_process_first_step" else "selected-window" + axes[0, 1].set_ylabel(f"{scope_label} percent") + axes[0, 1].legend() + axes[1, 1].bar(labels, vram) + axes[1, 1].set_ylabel("full-run sampled peak VRAM (GiB)") + for axis in (axes[0, 1], axes[1, 1]): + axis.tick_params(axis="x", rotation=25) + figure.suptitle("Task 21 GPU utilization and VRAM") + finish("task21_gpu_util_vram.png") + + quality_tags = ( + "rollout/raw_reward", + "rollout/response_lengths", + "rollout/truncated_ratio", + "train/loss", + "train/grad_norm", + "train/ppo_kl", + ) + figure, axes = plt.subplots(3, 2, figsize=(14, 12)) + seeds = sorted({int(analysis.manifest["seed"]) for analysis in analyses}) + for axis, tag in zip((axis for row in axes for axis in row), quality_tags, strict=True): + for condition in ("baseline", "experiment"): + condition_runs = { + int(analysis.manifest["seed"]): analysis + for analysis in analyses + if analysis.manifest["condition"] == condition + } + values = [] + for seed in seeds: + analysis = condition_runs.get(seed) + metric = analysis.summary["metrics"].get(tag) if analysis is not None else None + values.append(numeric_or_nan(metric.get("mean") if metric is not None else None)) + axis.plot(seeds, values, marker="o", label=condition) + axis.set_title(tag) + axis.set_xlabel("seed") + axis.set_ylabel("mean metric value") + axis.tick_params(axis="x", rotation=20) + axis.legend() + figure.suptitle("Task 21 correctness and quality guardrails") + finish("task21_correctness_quality.png") + + paired = comparison["paired_runs"] + figure, axes = plt.subplots(2, 1, figsize=(12, 9)) + window_speedups = comparison["window_speedups"] + window_labels = [f"s{row['seed']}:{row['window_start']}-{row['window_end']}" for row in window_speedups] + axes[0].scatter( + window_labels, + [100 * row["speedup"] for row in window_speedups], + marker="o", + ) + axes[0].axhline(5, color="red", linestyle="--", label="5% run-level target") + axes[0].set_ylabel("window token-throughput speedup (%)") + axes[0].tick_params(axis="x", rotation=25) + axes[0].legend() + + pair_labels = [f"seed {row['seed']}" for row in paired] + speedups = [100 * row.get("perf/step_token_per_s:improvement", float("nan")) for row in paired] + pair_labels.append("geometric mean") + speedups.append(100 * (comparison["token_throughput_geomean_speedup"] or 0)) + axes[1].bar(pair_labels, speedups) + axes[1].axhline(5, color="red", linestyle="--", label="5% target") + axes[1].set_ylabel("paired run token-throughput speedup (%)") + axes[1].legend() + figure.suptitle("Task 21 preregistered windows and paired run summary") + finish("task21_window_summary.png") + return generated + + +def _load_steady_campaign_descriptor(path: Path) -> dict[str, Any]: + path = path.resolve() + try: + descriptor = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + _fail(f"cannot parse steady campaign descriptor {path}: {exc}") + if not isinstance(descriptor, dict): + _fail(f"steady campaign descriptor {path} must contain a JSON object") + if descriptor.get("schema_version") != 1: + _fail(f"steady campaign descriptor {path} requires schema_version=1") + + run_dirs = descriptor.get("run_dirs") + completed_runs_file = descriptor.get("completed_runs_file") + if (run_dirs is None) == (completed_runs_file is None): + _fail(f"{path} must define exactly one of run_dirs or completed_runs_file") + if completed_runs_file is not None: + completed_path = Path(str(completed_runs_file)) + if not completed_path.is_absolute(): + completed_path = path.parent / completed_path + try: + run_dirs = [ + line.strip() for line in completed_path.read_text(encoding="utf-8").splitlines() if line.strip() + ] + except OSError as exc: + _fail(f"cannot read completed steady runs {completed_path}: {exc}") + if not isinstance(run_dirs, list) or not run_dirs or any(not isinstance(item, str) for item in run_dirs): + _fail(f"{path} run_dirs must be a non-empty string list") + + resolved_run_dirs = [] + for item in run_dirs: + run_dir = Path(item) + if not run_dir.is_absolute(): + run_dir = path.parent / run_dir + resolved_run_dirs.append(run_dir.resolve()) + descriptor["run_dirs"] = resolved_run_dirs + return descriptor + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--run-dir", + action="append", + type=Path, + default=[], + help="Benchmark run directory. Repeat for paired baseline/experiment comparison.", + ) + parser.add_argument( + "--steady-campaign", + type=Path, + default=None, + help=( + "JSON descriptor for a B/P/P+R/P+S steady-state campaign. The descriptor supplies " + "run_dirs (or completed_runs_file), steady_windows, and expected topology values." + ), + ) + parser.add_argument( + "--output-dir", + type=Path, + default=None, + help="Comparison output directory; required only when more than one --run-dir is supplied.", + ) + parser.add_argument( + "--steady-windows", + default="4-8,9-13,14-18", + help=( + "Inclusive, non-overlapping measurement windows as START-END comma-separated ranges. " + "Using 0-0 is classified as a fresh-process first-step benchmark, not steady state." + ), + ) + parser.add_argument("--expected-samples", type=int, default=256) + parser.add_argument( + "--expected-actor-chunks", + type=int, + default=2, + help="Expected actor fetch/forward chunks when the pipeline is enabled.", + ) + parser.add_argument( + "--expected-gpu-count", + type=int, + default=8, + help="Required distinct GPU indexes in NVML telemetry when enforcing targets.", + ) + parser.add_argument( + "--validate-only", + action="store_true", + help="Validate each run without requiring paired performance target checks.", + ) + parser.add_argument( + "--enforce-targets", + action="store_true", + help="Require the preregistered 5%% throughput, 15%% phase-1, and 80%% overlap targets.", + ) + parser.add_argument( + "--enforce-steady-targets", + action="store_true", + help=( + "Require the four-condition steady protocol: at least 40 optimizer steps, 30 " + "warmup-excluded measured steps, two paired seeds, fixed workload/hardware, complete " + "correctness metrics, and steady NVML coverage." + ), + ) + parser.add_argument( + "--no-plots", + action="store_true", + help="Do not generate comparison PNG files.", + ) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + args = build_parser().parse_args(argv) + try: + descriptor = None + run_dirs = list(args.run_dir) + steady_windows = args.steady_windows + expected_samples = args.expected_samples + expected_actor_chunks = args.expected_actor_chunks + expected_gpu_count = args.expected_gpu_count + output_dir = args.output_dir + if args.steady_campaign is not None: + if run_dirs: + _fail("--steady-campaign cannot be combined with --run-dir") + descriptor = _load_steady_campaign_descriptor(args.steady_campaign) + run_dirs = descriptor["run_dirs"] + steady_windows = descriptor.get("steady_windows", steady_windows) + expected_samples = descriptor.get("expected_samples", expected_samples) + expected_actor_chunks = descriptor.get("expected_actor_chunks", expected_actor_chunks) + expected_gpu_count = descriptor.get("expected_gpu_count", expected_gpu_count) + if output_dir is None and descriptor.get("output_dir") is not None: + output_dir = Path(str(descriptor["output_dir"])) + if not run_dirs: + _fail("at least one --run-dir or --steady-campaign is required") + windows = _parse_windows(str(steady_windows)) + if expected_samples <= 0 or expected_actor_chunks <= 0 or expected_gpu_count <= 0: + _fail("expected sample, actor chunk, and GPU counts must be positive") + if len(run_dirs) > 1 and output_dir is None and not args.validate_only: + _fail("--output-dir is required when comparing multiple runs") + if args.validate_only and args.enforce_targets: + _fail("--validate-only and --enforce-targets are mutually exclusive") + if args.enforce_targets and args.enforce_steady_targets: + _fail("--enforce-targets and --enforce-steady-targets are mutually exclusive") + if args.enforce_steady_targets and args.steady_campaign is None: + _fail("--enforce-steady-targets requires --steady-campaign") + + analyses = [ + analyze_run( + run_dir, + windows=windows, + expected_samples=expected_samples, + expected_actor_chunks=expected_actor_chunks, + require_reproducibility_artifacts=(args.enforce_targets or args.enforce_steady_targets), + ) + for run_dir in run_dirs + ] + result: dict[str, Any] = { + "measurement_scope": _measurement_scope(windows), + "claim_limit": ( + "Fresh-process first optimizer step only; do not report as steady-state training throughput." + if _measurement_scope(windows) == "fresh_process_first_step" + else None + ), + "runs": [analysis.summary for analysis in analyses], + "validation": "passed", + } + if descriptor is not None: + campaign = _build_steady_campaign( + analyses, + windows=windows, + enforce_targets=args.enforce_steady_targets, + expected_gpu_count=expected_gpu_count, + ) + result["steady_campaign"] = campaign + if output_dir is not None: + resolved_output_dir = output_dir.resolve() + resolved_output_dir.mkdir(parents=True, exist_ok=True) + (resolved_output_dir / "steady_campaign_summary.json").write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + for name, comparison in campaign["comparisons"].items(): + _write_csv( + resolved_output_dir / f"{name.replace('+', '_plus_')}_paired_runs.csv", + comparison["paired_runs"], + ) + elif len(analyses) > 1: + comparison = _build_comparison( + analyses, + windows=windows, + enforce_targets=args.enforce_targets, + expected_gpu_count=args.expected_gpu_count, + ) + result["comparison"] = comparison + resolved_output_dir = output_dir.resolve() + resolved_output_dir.mkdir(parents=True, exist_ok=True) + if not args.no_plots: + result["plots"] = _plot_comparison(analyses, comparison, resolved_output_dir, windows) + (resolved_output_dir / "comparison_summary.json").write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + _write_csv(resolved_output_dir / "paired_run_summary.csv", comparison["paired_runs"]) + _write_csv(resolved_output_dir / "window_speedup_summary.csv", comparison["window_speedups"]) + print(json.dumps(result, indent=2, sort_keys=True)) + return 0 + except BenchmarkValidationError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/tools/plot_task21_steady_training_curves.py b/scripts/tools/plot_task21_steady_training_curves.py new file mode 100644 index 000000000..9c15e411e --- /dev/null +++ b/scripts/tools/plot_task21_steady_training_curves.py @@ -0,0 +1,221 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +"""Plot Task 21 training curves from validated TensorBoard scalar exports.""" + +from __future__ import annotations + +import argparse +import csv +from collections import defaultdict +from pathlib import Path +from typing import Sequence + + +METRICS = { + "train/grad_norm": ("Gradient norm", "task21_steady_grad_norm.png"), + "train/loss": ("Training loss", "task21_steady_loss.png"), + "rollout/raw_reward": ("Raw reward", "task21_steady_reward.png"), +} +CONDITION_ORDER = ("B", "P", "P+R", "P+S") +COLORS = { + "B": "#343A40", + "P": "#168C83", + "P+R": "#C43D3D", + "P+S": "#2867B2", +} +SEED_STYLES = ("--", ":") + + +def _parse_run(value: str) -> tuple[str, int, Path]: + try: + condition, seed_text, path_text = value.split("=", 2) + seed = int(seed_text) + except (ValueError, TypeError): + raise argparse.ArgumentTypeError("--run must be CONDITION=SEED=RUN_DIR") from None + if condition not in CONDITION_ORDER: + raise argparse.ArgumentTypeError(f"unsupported condition {condition!r}") + run_dir = Path(path_text).resolve() + scalar_path = run_dir / "analysis" / "tensorboard_scalars.csv" + if not scalar_path.is_file(): + raise argparse.ArgumentTypeError(f"missing scalar export: {scalar_path}") + return condition, seed, run_dir + + +def _load_series(run_dir: Path, *, expected_steps: int) -> dict[str, dict[int, float]]: + scalar_path = run_dir / "analysis" / "tensorboard_scalars.csv" + series = {tag: {} for tag in METRICS} + with scalar_path.open(newline="", encoding="utf-8") as handle: + for row in csv.DictReader(handle): + tag = row["tag"] + if tag not in series: + continue + step = int(row["step"]) + if step in series[tag]: + raise ValueError(f"{scalar_path}: duplicate {tag} step {step}") + series[tag][step] = float(row["value"]) + + expected = set(range(expected_steps)) + for tag, values in series.items(): + actual = set(values) + if actual != expected: + missing = sorted(expected - actual) + extra = sorted(actual - expected) + raise ValueError(f"{scalar_path}: {tag} step mismatch: missing={missing}, extra={extra}") + return series + + +def _write_csv( + output_path: Path, + rows: Sequence[tuple[str, int, Path, dict[str, dict[int, float]]]], +) -> None: + with output_path.open("w", newline="", encoding="utf-8") as handle: + writer = csv.writer(handle) + writer.writerow(("condition", "seed", "step", "grad_norm", "loss", "raw_reward", "run")) + for condition, seed, run_dir, series in rows: + for step in sorted(series["train/grad_norm"]): + writer.writerow( + ( + condition, + seed, + step, + series["train/grad_norm"][step], + series["train/loss"][step], + series["rollout/raw_reward"][step], + run_dir.name, + ) + ) + + +def _plot_metric( + output_path: Path, + tag: str, + grouped: dict[str, list[tuple[int, dict[int, float]]]], + *, + expected_steps: int, +) -> None: + try: + import matplotlib.pyplot as plt + from matplotlib.lines import Line2D + from matplotlib.patches import Patch + except ImportError as exc: + raise RuntimeError("plotting requires matplotlib") from exc + + title, _ = METRICS[tag] + steps = list(range(expected_steps)) + figure, axis = plt.subplots(figsize=(12, 6.75), dpi=160) + axis.axvspan(-0.5, 9.5, color="#E9ECEF", alpha=0.7, label="Warmup (0-9)", zorder=0) + axis.axvline(9.5, color="#868E96", linewidth=1, zorder=1) + + for condition in CONDITION_ORDER: + seed_series = sorted(grouped[condition], key=lambda item: item[0]) + values_by_seed = [[values[step] for step in steps] for _, values in seed_series] + lower = [min(values[index] for values in values_by_seed) for index in steps] + upper = [max(values[index] for values in values_by_seed) for index in steps] + mean = [sum(values[index] for values in values_by_seed) / len(values_by_seed) for index in steps] + color = COLORS[condition] + axis.fill_between(steps, lower, upper, color=color, alpha=0.1, linewidth=0) + for index, (seed, values) in enumerate(seed_series): + axis.plot( + steps, + [values[step] for step in steps], + color=color, + linestyle=SEED_STYLES[index], + linewidth=1, + alpha=0.45, + ) + axis.plot(steps, mean, color=color, linewidth=2.4, label=f"{condition} mean", zorder=3) + + if tag == "train/grad_norm": + axis.set_yscale("log") + spike_seed, spike_values = max( + grouped["P+R"], + key=lambda item: max(item[1].values()), + ) + spike_step, spike_value = max(spike_values.items(), key=lambda item: item[1]) + axis.annotate( + f"P+R seed {spike_seed}: {spike_value:.2f}", + xy=(spike_step, spike_value), + xytext=(spike_step + 2, spike_value * 0.72), + arrowprops={"arrowstyle": "->", "color": COLORS["P+R"]}, + color=COLORS["P+R"], + fontsize=9, + ) + elif tag == "train/loss": + axis.axhline(0, color="#ADB5BD", linewidth=1) + elif tag == "rollout/raw_reward": + axis.set_ylim(0, 0.7) + + axis.set_title(f"Task 21 steady training: {title}", fontsize=16, pad=14) + axis.set_xlabel("Optimizer step") + axis.set_ylabel(title) + axis.set_xlim(-0.5, expected_steps - 0.5) + axis.set_xticks(range(0, expected_steps, 5)) + axis.grid(axis="y", color="#DEE2E6", linewidth=0.8) + axis.spines[["top", "right"]].set_visible(False) + handles, labels = axis.get_legend_handles_labels() + handles.extend( + [ + Line2D([0], [0], color="#6C757D", linestyle="--", linewidth=1, alpha=0.6), + Patch(facecolor="#6C757D", alpha=0.12, edgecolor="none"), + ] + ) + labels.extend(("Individual seed", "Two-seed range")) + axis.legend(handles, labels, ncol=4, frameon=False, fontsize=9, loc="best") + figure.tight_layout() + figure.savefig(output_path, bbox_inches="tight") + plt.close(figure) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--run", + action="append", + required=True, + type=_parse_run, + help="Validated formal run as CONDITION=SEED=RUN_DIR; repeat twice per condition.", + ) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--expected-steps", type=int, default=40) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + args = build_parser().parse_args(argv) + if args.expected_steps <= 0: + raise SystemExit("--expected-steps must be positive") + + runs_by_condition: dict[str, list[tuple[int, Path]]] = defaultdict(list) + for condition, seed, run_dir in args.run: + runs_by_condition[condition].append((seed, run_dir)) + invalid = { + condition: len(runs_by_condition.get(condition, [])) + for condition in CONDITION_ORDER + if len(runs_by_condition.get(condition, [])) != 2 + } + if invalid: + raise SystemExit(f"expected exactly two runs per condition, got {invalid}") + + loaded = [] + grouped: dict[str, dict[str, list[tuple[int, dict[int, float]]]]] = {tag: defaultdict(list) for tag in METRICS} + for condition in CONDITION_ORDER: + for seed, run_dir in sorted(runs_by_condition[condition]): + series = _load_series(run_dir, expected_steps=args.expected_steps) + loaded.append((condition, seed, run_dir, series)) + for tag in METRICS: + grouped[tag][condition].append((seed, series[tag])) + + args.output_dir.mkdir(parents=True, exist_ok=True) + _write_csv(args.output_dir / "task21_steady_training_curves.csv", loaded) + for tag, (_, filename) in METRICS.items(): + _plot_metric( + args.output_dir / filename, + tag, + grouped[tag], + expected_steps=args.expected_steps, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh index 00228498d..f2f2cb0f8 100644 --- a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh +++ b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh @@ -6,7 +6,7 @@ # The Ray cluster is managed externally — do NOT kill ray or start a new cluster. # # Usage: -# bash scripts/training/multimodal/run-qwen35-9B-8xgpu-async.sh [async|sync] +# bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh [hybrid-async|sync] set -ex set -o pipefail @@ -21,46 +21,324 @@ SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" if [ -z "${RELAX_ENTRYPOINT_MODE:-}" ]; then source "${SCRIPT_DIR}/../../entrypoint/local.sh" fi -source "${MODEL_CONFIG_DIR}/qwen35-9B.sh" -# source "${MODEL_CONFIG_DIR}/qwen3-vl-4B.sh" +MODEL_CONFIG_FILE="${MODEL_CONFIG_FILE:-${MODEL_CONFIG_DIR}/qwen35-9B.sh}" +if [ ! -f "${MODEL_CONFIG_FILE}" ]; then + echo "MODEL_CONFIG_FILE does not exist: ${MODEL_CONFIG_FILE}" >&2 + exit 2 +fi +source "${MODEL_CONFIG_FILE}" PROJECT_NAME="${PROJECT_NAME:=Relax/dev/fully_async_openr1mm}" EXP_DIR="${EXP_DIR:-${SCRIPT_DIR}/../../../../exps}" MODEL_DIR="${MODEL_DIR:-${EXP_DIR}}" DATA_DIR="${DATA_DIR:-${EXP_DIR}}" NUM_ROLLOUT="${NUM_ROLLOUT:=200}" +MM_PROCESSOR_POOL_SIZE="${MM_PROCESSOR_POOL_SIZE:-0}" +HYBRID_REUSE_TRAIN_LOGPROBS="${HYBRID_REUSE_TRAIN_LOGPROBS:-0}" +MODEL_NAME="${MODEL_NAME:-Qwen3.5-9B}" +MODEL_RUN_NAME="${MODEL_RUN_NAME:-qwen35-9b}" +MODEL_CHECKPOINT_DIR="${MODEL_CHECKPOINT_DIR:-${MODEL_DIR}/${MODEL_NAME}}" +REFERENCE_CHECKPOINT_DIR="${REFERENCE_CHECKPOINT_DIR:-${MODEL_CHECKPOINT_DIR}}" + +CHECKPOINT_SAVE="${CHECKPOINT_SAVE:-1}" +CHECKPOINT_DIR="${CHECKPOINT_DIR:-${EXP_DIR}/${MODEL_NAME}_mcore_8xgpu/}" +CHECKPOINT_SAVE_INTERVAL="${CHECKPOINT_SAVE_INTERVAL:-100}" +MAX_ACTOR_CKPT_TO_KEEP="${MAX_ACTOR_CKPT_TO_KEEP:-1}" +ROLLOUT_RESULT_DIR="${ROLLOUT_RESULT_DIR:-}" +TENSORBOARD_DIR="${TENSORBOARD_DIR:-}" + +HYBRID_PIPELINE_FORWARD="${HYBRID_PIPELINE_FORWARD:-0}" +HYBRID_PIPELINE_OVERLAP="${HYBRID_PIPELINE_OVERLAP:-1}" +HYBRID_PIPELINE_TRACE_DIR="${HYBRID_PIPELINE_TRACE_DIR:-}" +HYBRID_PIPELINE_FETCH_TIMEOUT_S="${HYBRID_PIPELINE_FETCH_TIMEOUT_S:-600}" +NUM_ITERS_PER_TRAIN_UPDATE="${NUM_ITERS_PER_TRAIN_UPDATE:-2}" +SGLANG_DETERMINISTIC_INFERENCE="${SGLANG_DETERMINISTIC_INFERENCE:-0}" +SEED="${SEED:-}" +ROLLOUT_SEED="${ROLLOUT_SEED:-}" + +ROLLOUT_MAX_RESPONSE_LEN="${ROLLOUT_MAX_RESPONSE_LEN:-10240}" +ROLLOUT_MAX_PROMPT_LEN="${ROLLOUT_MAX_PROMPT_LEN:-2048}" +ROLLOUT_MAX_CONTEXT_LEN="${ROLLOUT_MAX_CONTEXT_LEN:-12288}" +ACTOR_MAX_TOKENS_PER_GPU="${ACTOR_MAX_TOKENS_PER_GPU:-12288}" +HYBRID_ACTOR_GPUS="${HYBRID_ACTOR_GPUS:-4}" +HYBRID_ROLLOUT_GPUS="${HYBRID_ROLLOUT_GPUS:-4}" +SYNC_GPUS="${SYNC_GPUS:-8}" +ROLLOUT_NUM_GPUS_PER_ENGINE="${ROLLOUT_NUM_GPUS_PER_ENGINE:-2}" +SGLANG_MEM_FRACTION_STATIC="${SGLANG_MEM_FRACTION_STATIC:-0.8}" + +require_positive_integer() { + local name="$1" + local value="$2" + if ! [[ "${value}" =~ ^[1-9][0-9]*$ ]]; then + echo "${name} must be a positive integer, got ${value}" >&2 + exit 2 + fi +} + +require_nonnegative_integer() { + local name="$1" + local value="$2" + if ! [[ "${value}" =~ ^[0-9]+$ ]]; then + echo "${name} must be a non-negative integer, got ${value}" >&2 + exit 2 + fi +} + +require_fraction() { + local name="$1" + local value="$2" + if ! [[ "${value}" =~ ^(0\.[0-9]*[1-9][0-9]*|1(\.0+)?)$ ]]; then + echo "${name} must be greater than 0 and at most 1, got ${value}" >&2 + exit 2 + fi +} + +case "${MODE}" in + hybrid-async|sync) ;; + *) + echo "MODE must be hybrid-async or sync, got ${MODE}" >&2 + exit 2 + ;; +esac +case "${HYBRID_PIPELINE_FORWARD}" in + 0|1) ;; + *) + echo "HYBRID_PIPELINE_FORWARD must be 0 or 1, got ${HYBRID_PIPELINE_FORWARD}" >&2 + exit 2 + ;; +esac +case "${HYBRID_PIPELINE_OVERLAP}" in + 0|1) ;; + *) + echo "HYBRID_PIPELINE_OVERLAP must be 0 or 1, got ${HYBRID_PIPELINE_OVERLAP}" >&2 + exit 2 + ;; +esac +case "${HYBRID_REUSE_TRAIN_LOGPROBS}" in + 0|1) ;; + *) + echo "HYBRID_REUSE_TRAIN_LOGPROBS must be 0 or 1, got ${HYBRID_REUSE_TRAIN_LOGPROBS}" >&2 + exit 2 + ;; +esac +case "${SGLANG_DETERMINISTIC_INFERENCE}" in + 0|1) ;; + *) + echo "SGLANG_DETERMINISTIC_INFERENCE must be 0 or 1, got ${SGLANG_DETERMINISTIC_INFERENCE}" >&2 + exit 2 + ;; +esac +case "${CHECKPOINT_SAVE}" in + 0|1) ;; + *) + echo "CHECKPOINT_SAVE must be 0 or 1, got ${CHECKPOINT_SAVE}" >&2 + exit 2 + ;; +esac +if [ "${MODE}" != "hybrid-async" ] && { + [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] || [ -n "${HYBRID_PIPELINE_TRACE_DIR}" ] || + [ "${HYBRID_REUSE_TRAIN_LOGPROBS}" = "1" ]; +}; then + echo "Hybrid pipeline forward/trace and train-logprob reuse options require MODE=hybrid-async" >&2 + exit 2 +fi + +for item in \ + "NUM_ROLLOUT:${NUM_ROLLOUT}" \ + "HYBRID_PIPELINE_FETCH_TIMEOUT_S:${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" \ + "NUM_ITERS_PER_TRAIN_UPDATE:${NUM_ITERS_PER_TRAIN_UPDATE}" \ + "ROLLOUT_MAX_RESPONSE_LEN:${ROLLOUT_MAX_RESPONSE_LEN}" \ + "ROLLOUT_MAX_PROMPT_LEN:${ROLLOUT_MAX_PROMPT_LEN}" \ + "ROLLOUT_MAX_CONTEXT_LEN:${ROLLOUT_MAX_CONTEXT_LEN}" \ + "ACTOR_MAX_TOKENS_PER_GPU:${ACTOR_MAX_TOKENS_PER_GPU}" \ + "HYBRID_ACTOR_GPUS:${HYBRID_ACTOR_GPUS}" \ + "HYBRID_ROLLOUT_GPUS:${HYBRID_ROLLOUT_GPUS}" \ + "SYNC_GPUS:${SYNC_GPUS}" \ + "ROLLOUT_NUM_GPUS_PER_ENGINE:${ROLLOUT_NUM_GPUS_PER_ENGINE}"; do + require_positive_integer "${item%%:*}" "${item#*:}" +done +require_nonnegative_integer "MM_PROCESSOR_POOL_SIZE" "${MM_PROCESSOR_POOL_SIZE}" +if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] && (( NUM_ITERS_PER_TRAIN_UPDATE < 2 )); then + echo "HYBRID_PIPELINE_FORWARD requires NUM_ITERS_PER_TRAIN_UPDATE >= 2" >&2 + exit 2 +fi +if [ "${HYBRID_PIPELINE_FORWARD}" = "0" ] && [ "${HYBRID_PIPELINE_OVERLAP}" = "0" ]; then + echo "HYBRID_PIPELINE_OVERLAP=0 requires HYBRID_PIPELINE_FORWARD=1" >&2 + exit 2 +fi +if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] && [ "${HYBRID_REUSE_TRAIN_LOGPROBS}" = "1" ]; then + echo "HYBRID_PIPELINE_FORWARD and HYBRID_REUSE_TRAIN_LOGPROBS are separate ablations and cannot be enabled together" >&2 + exit 2 +fi +if [ "${CHECKPOINT_SAVE}" = "1" ]; then + require_positive_integer "CHECKPOINT_SAVE_INTERVAL" "${CHECKPOINT_SAVE_INTERVAL}" + require_positive_integer "MAX_ACTOR_CKPT_TO_KEEP" "${MAX_ACTOR_CKPT_TO_KEEP}" +fi +if (( ROLLOUT_MAX_CONTEXT_LEN < ROLLOUT_MAX_PROMPT_LEN + ROLLOUT_MAX_RESPONSE_LEN )); then + echo "ROLLOUT_MAX_CONTEXT_LEN must cover prompt + response limits" >&2 + exit 2 +fi +if (( HYBRID_ROLLOUT_GPUS % ROLLOUT_NUM_GPUS_PER_ENGINE != 0 )); then + echo "HYBRID_ROLLOUT_GPUS must be divisible by ROLLOUT_NUM_GPUS_PER_ENGINE" >&2 + exit 2 +fi +if (( SYNC_GPUS % ROLLOUT_NUM_GPUS_PER_ENGINE != 0 )); then + echo "SYNC_GPUS must be divisible by ROLLOUT_NUM_GPUS_PER_ENGINE" >&2 + exit 2 +fi +if (( HYBRID_ACTOR_GPUS % 4 != 0 || SYNC_GPUS % 4 != 0 )); then + echo "actor GPU counts must be multiples of TP(2) * CP(2)" >&2 + exit 2 +fi +if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] && [ "${HYBRID_ACTOR_GPUS}" != "4" ]; then + echo "HYBRID_PIPELINE_FORWARD currently requires TP=2, CP=2, DP=1 (4 actor GPUs)" >&2 + exit 2 +fi +require_fraction "SGLANG_MEM_FRACTION_STATIC" "${SGLANG_MEM_FRACTION_STATIC}" + +if [ "${CHECKPOINT_SAVE}" = "0" ]; then + ROLLOUT_RESULT_DIR="${ROLLOUT_RESULT_DIR:-${EXP_DIR}/rollout_result}" + TENSORBOARD_DIR="${TENSORBOARD_DIR:-${EXP_DIR}/tensorboard_log}" +fi +if [ -n "${TENSORBOARD_DIR}" ]; then + export TENSORBOARD_DIR + RUNTIME_ENV_JSON_INPUT="${RUNTIME_ENV_JSON:-}" + if [ -z "${RUNTIME_ENV_JSON_INPUT}" ]; then + RUNTIME_ENV_JSON_INPUT='{}' + fi + RUNTIME_ENV_JSON="$( + python3 - "${RUNTIME_ENV_JSON_INPUT}" "${TENSORBOARD_DIR}" <<'PY' +import json +import sys + +runtime_env = json.loads(sys.argv[1]) +if not isinstance(runtime_env, dict): + raise SystemExit("RUNTIME_ENV_JSON must decode to an object") +env_vars = runtime_env.setdefault("env_vars", {}) +if not isinstance(env_vars, dict): + raise SystemExit("RUNTIME_ENV_JSON env_vars must be an object") +env_vars["TENSORBOARD_DIR"] = sys.argv[2] +print(json.dumps(runtime_env, separators=(",", ":"), sort_keys=True)) +PY + )" +fi + +HYBRID_RESOURCE="{\"actor\": [1, ${HYBRID_ACTOR_GPUS}], \"rollout\": [1, ${HYBRID_ROLLOUT_GPUS}]}" +SYNC_RESOURCE="{\"actor\": [1, ${SYNC_GPUS}], \"rollout\": [1, ${SYNC_GPUS}]}" +if [ "${MODE}" = "hybrid-async" ]; then + RUN_GPU_COUNT=$((HYBRID_ACTOR_GPUS + HYBRID_ROLLOUT_GPUS)) +else + RUN_GPU_COUNT="${SYNC_GPUS}" +fi + +HYBRID_PIPELINE_ARGS=() +if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ]; then + HYBRID_PIPELINE_ARGS+=( + --hybrid-pipeline-forward + --hybrid-pipeline-fetch-timeout-s "${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" + ) + if [ "${HYBRID_PIPELINE_OVERLAP}" = "0" ]; then + HYBRID_PIPELINE_ARGS+=(--no-hybrid-pipeline-overlap) + fi +fi + +TASK21_OPT_ARGS=(--mm-processor-pool-size "${MM_PROCESSOR_POOL_SIZE}") +if [ "${HYBRID_REUSE_TRAIN_LOGPROBS}" = "1" ]; then + TASK21_OPT_ARGS+=(--true-on-policy-mode) +fi +if [ -n "${HYBRID_PIPELINE_TRACE_DIR}" ]; then + HYBRID_PIPELINE_ARGS+=( + --hybrid-pipeline-trace-dir "${HYBRID_PIPELINE_TRACE_DIR}" + ) +fi + +REPRO_ARGS=() +if [ -n "${SEED}" ]; then + REPRO_ARGS+=(--seed "${SEED}") +fi +if [ -n "${ROLLOUT_SEED}" ]; then + REPRO_ARGS+=(--rollout-seed "${ROLLOUT_SEED}") +fi + +DEBUG_ARGS=() +if [ -n "${SAVE_DEBUG_ROLLOUT_DATA:-}" ]; then + DEBUG_ARGS+=(--save-debug-rollout-data "${SAVE_DEBUG_ROLLOUT_DATA}") +fi +if [ -n "${LOAD_DEBUG_ROLLOUT_DATA:-}" ]; then + DEBUG_ARGS+=(--load-debug-rollout-data "${LOAD_DEBUG_ROLLOUT_DATA}") +fi +if [ -n "${SAVE_DEBUG_TRAIN_DATA:-}" ]; then + DEBUG_ARGS+=(--save-debug-train-data "${SAVE_DEBUG_TRAIN_DATA}") +fi +if [ "${SGLANG_DETERMINISTIC_INFERENCE}" = "1" ]; then + DEBUG_ARGS+=(--sglang-enable-deterministic-inference) +fi + +printf '%s\n' \ + "MODE=${MODE}" \ + "NUM_ROLLOUT=${NUM_ROLLOUT}" \ + "MM_PROCESSOR_POOL_SIZE=${MM_PROCESSOR_POOL_SIZE}" \ + "HYBRID_REUSE_TRAIN_LOGPROBS=${HYBRID_REUSE_TRAIN_LOGPROBS}" \ + "HYBRID_PIPELINE_FORWARD=${HYBRID_PIPELINE_FORWARD}" \ + "HYBRID_PIPELINE_OVERLAP=${HYBRID_PIPELINE_OVERLAP}" \ + "HYBRID_PIPELINE_TRACE_DIR=${HYBRID_PIPELINE_TRACE_DIR}" \ + "HYBRID_PIPELINE_FETCH_TIMEOUT_S=${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" \ + "NUM_ITERS_PER_TRAIN_UPDATE=${NUM_ITERS_PER_TRAIN_UPDATE}" \ + "SEED=${SEED}" \ + "ROLLOUT_SEED=${ROLLOUT_SEED}" \ + "SGLANG_DETERMINISTIC_INFERENCE=${SGLANG_DETERMINISTIC_INFERENCE}" \ + "MODEL_CONFIG_FILE=${MODEL_CONFIG_FILE}" \ + "MODEL_CHECKPOINT_DIR=${MODEL_CHECKPOINT_DIR}" \ + "REFERENCE_CHECKPOINT_DIR=${REFERENCE_CHECKPOINT_DIR}" \ + "CHECKPOINT_SAVE=${CHECKPOINT_SAVE}" \ + "CHECKPOINT_DIR=${CHECKPOINT_DIR}" \ + "ROLLOUT_RESULT_DIR=${ROLLOUT_RESULT_DIR}" \ + "TENSORBOARD_DIR=${TENSORBOARD_DIR}" \ + "ACTOR_MAX_TOKENS_PER_GPU=${ACTOR_MAX_TOKENS_PER_GPU}" \ + "HYBRID_RESOURCE=${HYBRID_RESOURCE}" \ + "SYNC_RESOURCE=${SYNC_RESOURCE}" \ + "RUN_GPU_COUNT=${RUN_GPU_COUNT}" CKPT_ARGS=( - --hf-checkpoint ${MODEL_DIR}/Qwen3.5-9B - --ref-load ${MODEL_DIR}/Qwen3.5-9B - # --hf-checkpoint ${MODEL_DIR}/Qwen3-VL-4B-Instruct + --hf-checkpoint "${MODEL_CHECKPOINT_DIR}" + --ref-load "${REFERENCE_CHECKPOINT_DIR}" --megatron-to-hf-mode bridge --warm-hf-checkpoint-page-cache - # --ref-load ${MODEL_DIR}/Qwen3-VL-4B-Instruct - # --load ${EXP_DIR}/Qwen3.5-9B_mcore_8xgpu/ - --save ${EXP_DIR}/Qwen3.5-9B_mcore_8xgpu/ - --save-interval 100 - --max-actor-ckpt-to-keep 1 ) +if [ "${CHECKPOINT_SAVE}" = "1" ]; then + CKPT_ARGS+=( + --save "${CHECKPOINT_DIR}" + --save-interval "${CHECKPOINT_SAVE_INTERVAL}" + --max-actor-ckpt-to-keep "${MAX_ACTOR_CKPT_TO_KEEP}" + ) +fi + +OUTPUT_ARGS=() +if [ -n "${ROLLOUT_RESULT_DIR}" ]; then + OUTPUT_ARGS+=(--rollout-result-dir "${ROLLOUT_RESULT_DIR}") +fi +if [ -n "${TENSORBOARD_DIR}" ]; then + OUTPUT_ARGS+=(--tensorboard-dir "${TENSORBOARD_DIR}") +fi -PROMPT_SET=${DATA_DIR}/multimodal-open-r1-8k-verified/data/train-00000-of-00001_converted_noextract.parquet +PROMPT_SET="${PROMPT_SET:-${DATA_DIR}/multimodal-open-r1-8k-verified/data/train-00000-of-00001_converted_noextract.parquet}" SYSTEM_PROMPT="A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within and tags, respectively, i.e., reasoning process here answer here " ROLLOUT_ARGS=( - --prompt-data ${PROMPT_SET} + --prompt-data "${PROMPT_SET}" --input-key prompt --label-key label --apply-chat-template --rollout-shuffle --rm-type openr1mm - --num-rollout ${NUM_ROLLOUT} + --num-rollout "${NUM_ROLLOUT}" --rollout-batch-size 32 --n-samples-per-prompt 8 - --rollout-max-response-len 10240 - --rollout-max-prompt-len 2048 - --rollout-max-context-len 12288 + --rollout-max-response-len "${ROLLOUT_MAX_RESPONSE_LEN}" + --rollout-max-prompt-len "${ROLLOUT_MAX_PROMPT_LEN}" + --rollout-max-context-len "${ROLLOUT_MAX_CONTEXT_LEN}" --rollout-temperature 0.8 --global-batch-size 256 --multimodal-keys '{"image":"image"}' @@ -84,7 +362,7 @@ PERF_ARGS=( # --micro-batch-size 16 # --qkv-format bshd --use-dynamic-batch-size - --max-tokens-per-gpu 12288 + --max-tokens-per-gpu "${ACTOR_MAX_TOKENS_PER_GPU}" --no-rope-fusion ) @@ -118,13 +396,13 @@ WANDB_ARGS=( --use-tensorboard --use-clearml --use-metrics-service - --tb-project-name ${PROJECT_NAME} - --tb-experiment-name qwen35-9b-GRPO-gpu8-${MODE}-${now} + --tb-project-name "${PROJECT_NAME}" + --tb-experiment-name "${MODEL_RUN_NAME}-GRPO-gpu${RUN_GPU_COUNT}-${MODE}-${now}" ) SGLANG_ARGS=( - --rollout-num-gpus-per-engine 2 - --sglang-mem-fraction-static 0.8 + --rollout-num-gpus-per-engine "${ROLLOUT_NUM_GPUS_PER_ENGINE}" + --sglang-mem-fraction-static "${SGLANG_MEM_FRACTION_STATIC}" ) MISC_ARGS=( @@ -139,43 +417,53 @@ MISC_ARGS=( ) -mkdir -p log -if [ ${MODE} = "hybrid-async" ]; then +LOG_DIR="${EXP_DIR}/logs" +mkdir -p "${LOG_DIR}" +if [ "${MODE}" = "hybrid-async" ]; then ray job submit ${RAY_NO_WAIT:+--no-wait} --address="http://127.0.0.1:8265" \ --runtime-env-json="${RUNTIME_ENV_JSON}" \ -- python3 -m relax.entrypoints.train \ - --resource '{"actor": [1, 4], "rollout": [1, 4]}'\ + --resource "${HYBRID_RESOURCE}" \ --max-staleness 2 \ --num-data-storage-units 1 \ - --num-iters-per-train-update 2 \ + --num-iters-per-train-update "${NUM_ITERS_PER_TRAIN_UPDATE}" \ --balance-data \ --hybrid \ + "${HYBRID_PIPELINE_ARGS[@]}" \ + "${TASK21_OPT_ARGS[@]}" \ + "${REPRO_ARGS[@]}" \ + "${DEBUG_ARGS[@]}" \ "${MODEL_ARGS[@]}" \ "${CKPT_ARGS[@]}" \ + "${OUTPUT_ARGS[@]}" \ "${ROLLOUT_ARGS[@]}" \ "${OPTIMIZER_ARGS[@]}" \ "${GRPO_ARGS[@]}" \ "${WANDB_ARGS[@]}" \ "${PERF_ARGS[@]}" \ "${SGLANG_ARGS[@]}" \ - "${MISC_ARGS[@]}" 2>&1 | tee log/qwen35-9b-GRPO-gpu8-hybrid-async-${now}.log + "${MISC_ARGS[@]}" 2>&1 | tee "${LOG_DIR}/${MODEL_RUN_NAME}-GRPO-gpu${RUN_GPU_COUNT}-hybrid-async-${now}.log" else ray job submit ${RAY_NO_WAIT:+--no-wait} --address="http://127.0.0.1:8265" \ --runtime-env-json="${RUNTIME_ENV_JSON}" \ -- python3 -m relax.entrypoints.train \ - --resource '{"actor": [1, 8], "rollout": [1, 8]}'\ + --resource "${SYNC_RESOURCE}" \ --max-staleness 0 \ --num-data-storage-units 1 \ --colocate \ --use-health-check \ --balance-data \ + "${TASK21_OPT_ARGS[@]}" \ + "${REPRO_ARGS[@]}" \ + "${DEBUG_ARGS[@]}" \ "${MODEL_ARGS[@]}" \ "${CKPT_ARGS[@]}" \ + "${OUTPUT_ARGS[@]}" \ "${ROLLOUT_ARGS[@]}" \ "${OPTIMIZER_ARGS[@]}" \ "${GRPO_ARGS[@]}" \ "${WANDB_ARGS[@]}" \ "${PERF_ARGS[@]}" \ "${SGLANG_ARGS[@]}" \ - "${MISC_ARGS[@]}" 2>&1 | tee log/qwen35-9b-GRPO-gpu8-fully-sync-${now}.log + "${MISC_ARGS[@]}" 2>&1 | tee "${LOG_DIR}/${MODEL_RUN_NAME}-GRPO-gpu${RUN_GPU_COUNT}-fully-sync-${now}.log" fi diff --git a/tests/backends/megatron/test_data_vpp.py b/tests/backends/megatron/test_data_vpp.py index c2f8980aa..ae419ecbe 100644 --- a/tests/backends/megatron/test_data_vpp.py +++ b/tests/backends/megatron/test_data_vpp.py @@ -15,6 +15,7 @@ def _load_data_module(monkeypatch): training = types.ModuleType("megatron.training") global_vars = types.ModuleType("megatron.training.global_vars") tracking_utils = types.ModuleType("relax.utils.tracking_utils") + ray = types.ModuleType("ray") class _PackedSeqParams: pass @@ -31,6 +32,7 @@ class _PackedSeqParams: "megatron.training": training, "megatron.training.global_vars": global_vars, "relax.utils.tracking_utils": tracking_utils, + "ray": ray, } for name, module in modules.items(): monkeypatch.setitem(sys.modules, name, module) @@ -94,6 +96,81 @@ def test_rollout_minibatch_plan_rejects_non_divisible_prompt_groups(monkeypatch) data_module.build_rollout_minibatch_plan(args, dp_size=2) +def test_hybrid_forward_chunk_plan_matches_producer_granularity(monkeypatch): + data_module = _load_data_module(monkeypatch) + args = Namespace( + rollout_batch_size=32, + n_samples_per_prompt=8, + global_batch_size=256, + num_steps_per_rollout=None, + num_iters_per_train_update=2, + ) + rollout_plan = data_module.build_rollout_minibatch_plan(args, dp_size=1) + + chunk_plan = data_module.build_hybrid_forward_chunk_plan(args, rollout_plan, dp_size=1) + + assert chunk_plan.chunks_per_mini == 2 + assert chunk_plan.chunk_global_samples == 128 + assert chunk_plan.chunk_local_samples == 128 + assert [ + chunk_plan.transfer_queue_batch_index(mini_index, chunk_index) + for mini_index in range(3) + for chunk_index in range(chunk_plan.chunks_per_mini) + ] == [0, 1, 2, 3, 4, 5] + + +def test_hybrid_forward_chunk_plan_supports_four_prompt_aligned_stages(monkeypatch): + data_module = _load_data_module(monkeypatch) + args = Namespace( + rollout_batch_size=32, + n_samples_per_prompt=8, + global_batch_size=256, + num_steps_per_rollout=None, + num_iters_per_train_update=4, + ) + rollout_plan = data_module.build_rollout_minibatch_plan(args, dp_size=1) + + chunk_plan = data_module.build_hybrid_forward_chunk_plan(args, rollout_plan, dp_size=1) + + assert chunk_plan.chunks_per_mini == 4 + assert chunk_plan.chunk_global_samples == 64 + assert chunk_plan.chunk_local_samples == 64 + assert [ + chunk_plan.transfer_queue_batch_index(mini_index, chunk_index) + for mini_index in range(2) + for chunk_index in range(chunk_plan.chunks_per_mini) + ] == list(range(8)) + + +@pytest.mark.parametrize( + ("num_iters", "global_batch_size", "n_samples_per_prompt", "error"), + [ + (1, 256, 8, "num_iters_per_train_update >= 2"), + (3, 256, 8, "mini_global_samples must be divisible"), + (4, 24, 8, "must preserve complete prompt groups"), + ], +) +def test_hybrid_forward_chunk_plan_rejects_unsafe_boundaries( + monkeypatch, + num_iters, + global_batch_size, + n_samples_per_prompt, + error, +): + data_module = _load_data_module(monkeypatch) + args = Namespace( + rollout_batch_size=global_batch_size // n_samples_per_prompt, + n_samples_per_prompt=n_samples_per_prompt, + global_batch_size=global_batch_size, + num_steps_per_rollout=None, + num_iters_per_train_update=num_iters, + ) + rollout_plan = data_module.build_rollout_minibatch_plan(args, dp_size=1) + + with pytest.raises(ValueError, match=error): + data_module.build_hybrid_forward_chunk_plan(args, rollout_plan, dp_size=1) + + def test_concat_rollout_batches_preserves_order_and_scalar_metadata(monkeypatch): data_module = _load_data_module(monkeypatch) @@ -120,6 +197,117 @@ def test_concat_rollout_batches_preserves_order_and_scalar_metadata(monkeypatch) assert merged["weight_version"] == 7 +@pytest.mark.parametrize( + ("second_batch", "error"), + [ + ( + {"tokens": ["c"], "total_lengths": [3]}, + "schema mismatch", + ), + ( + { + "tokens": ["c", "unexpected"], + "total_lengths": [3], + "weight_version": 7, + }, + "Per-sample rollout field", + ), + ], +) +def test_concat_rollout_batches_rejects_incomplete_chunks(monkeypatch, second_batch, error): + data_module = _load_data_module(monkeypatch) + + with pytest.raises(ValueError, match=error): + data_module.concat_rollout_batches( + [ + { + "tokens": ["a", "b"], + "total_lengths": [1, 2], + "weight_version": 7, + }, + second_batch, + ] + ) + + +def test_canonicalize_rollout_chunks_reorders_every_sample_field(monkeypatch): + data_module = _load_data_module(monkeypatch) + chunks = [ + ( + { + "tokens": ["token-12", "token-10"], + "total_lengths": [12, 10], + "multimodal_train_inputs": [{"image": "12"}, {"image": "10"}], + "scores": torch.tensor([[12], [10]]), + "array": data_module.np.array([[12], [10]]), + "weight_version": 7, + }, + [12, 10], + ), + ( + { + "tokens": ["token-13", "token-11"], + "total_lengths": [13, 11], + "multimodal_train_inputs": [{"image": "13"}, {"image": "11"}], + "scores": torch.tensor([[13], [11]]), + "array": data_module.np.array([[13], [11]]), + "weight_version": 7, + }, + [13, 11], + ), + ] + + merged, global_indexes = data_module.canonicalize_rollout_chunks(chunks, expected_sample_count=4) + + assert global_indexes == [10, 11, 12, 13] + assert merged["tokens"] == ["token-10", "token-11", "token-12", "token-13"] + assert merged["total_lengths"] == [10, 11, 12, 13] + assert merged["multimodal_train_inputs"] == [ + {"image": "10"}, + {"image": "11"}, + {"image": "12"}, + {"image": "13"}, + ] + assert torch.equal(merged["scores"], torch.tensor([[10], [11], [12], [13]])) + assert merged["array"].tolist() == [[10], [11], [12], [13]] + assert merged["weight_version"] == 7 + + +@pytest.mark.parametrize( + ("chunks", "expected_count", "error"), + [ + ( + [ + ({"total_lengths": [1, 1]}, [10, 11]), + ({"total_lengths": [1, 1]}, [11, 12]), + ], + 4, + "contain duplicates", + ), + ( + [({"total_lengths": [1, 1]}, [10])], + 2, + "length mismatch", + ), + ( + [({"total_lengths": [1]}, [10])], + 2, + "sample count mismatch", + ), + ], +) +def test_canonicalize_rollout_chunks_rejects_invalid_metadata( + monkeypatch, + chunks, + expected_count, + error, +): + data_module = _load_data_module(monkeypatch) + + with pytest.raises((ValueError, TypeError), match=error): + data_module.canonicalize_rollout_chunks(chunks, expected_count) + + def test_get_data_iterator_uses_rollout_mini_boundaries_with_balance_data(monkeypatch): data_module = _load_data_module(monkeypatch) monkeypatch.setattr( diff --git a/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py new file mode 100644 index 000000000..74ccfdd03 --- /dev/null +++ b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py @@ -0,0 +1,293 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import importlib +from contextlib import nullcontext +from types import SimpleNamespace + +import numpy as np +import pytest +import torch + + +try: + actor_module = importlib.import_module("relax.backends.megatron.actor") +except ImportError: + actor_module = None + +pytestmark = pytest.mark.skipif( + actor_module is None, + reason="Megatron/Ray/TransferQueue runtime dependencies are required for the actor wiring test", +) + + +def test_hybrid_pipeline_runtime_rechecks_supported_parallel_topology(monkeypatch): + assert actor_module is not None + actor = object.__new__(actor_module.MegatronTrainRayActor) + actor.args = SimpleNamespace( + num_iters_per_train_update=2, + n_samples_per_prompt=2, + expert_tensor_parallel_size=1, + offload_train=False, + offload_rollout=False, + compute_advantages_and_returns=True, + use_dynamic_batch_size=True, + ) + actor.weights_backuper = SimpleNamespace(backup_tags={"actor"}) + rollout_plan = SimpleNamespace( + num_rollout_minis=1, + mini_global_samples=4, + mini_local_sample_request=4, + fixed_n_samples_per_prompt=2, + ) + monkeypatch.setattr(actor_module.mpu, "get_pipeline_model_parallel_world_size", lambda: 1) + monkeypatch.setattr(actor_module.mpu, "get_virtual_pipeline_model_parallel_world_size", lambda: None) + monkeypatch.setattr(actor_module.mpu, "get_tensor_model_parallel_world_size", lambda: 2) + monkeypatch.setattr(actor_module.mpu, "get_context_parallel_world_size", lambda: 2) + monkeypatch.setattr(actor_module.mpu, "get_expert_model_parallel_world_size", lambda: 1) + + chunk_plan = actor._validate_hybrid_pipeline_runtime(rollout_plan, dp_size=1) + + assert chunk_plan.chunks_per_mini == 2 + assert chunk_plan.chunk_local_samples == 2 + + actor.args.use_dynamic_batch_size = False + with pytest.raises(RuntimeError, match="requires use_dynamic_batch_size=True"): + actor._validate_hybrid_pipeline_runtime(rollout_plan, dp_size=1) + actor.args.use_dynamic_batch_size = True + + monkeypatch.setattr(actor_module.mpu, "get_context_parallel_world_size", lambda: 1) + with pytest.raises(RuntimeError, match="requires TP=2, CP=2, EP=1, and ETP=1"): + actor._validate_hybrid_pipeline_runtime(rollout_plan, dp_size=1) + + +def test_hybrid_true_on_policy_reuse_requires_deterministic_single_mini(): + assert actor_module is not None + actor = object.__new__(actor_module.MegatronTrainRayActor) + actor.args = SimpleNamespace( + true_on_policy_mode=True, + keep_old_actor=False, + use_rollout_logprobs=False, + max_staleness=2, + use_tis=True, + custom_megatron_before_log_prob_hook_path=None, + attention_dropout=0.0, + hidden_dropout=0.0, + lora_dropout=0.0, + kl_coef=0.0, + use_kl_loss=False, + use_opd=False, + use_routing_replay=False, + use_rollout_routing_replay=False, + ) + + assert actor._should_reuse_hybrid_train_forward_log_probs(SimpleNamespace(num_rollout_minis=1)) is True + + actor.args.use_tis = False + with pytest.raises(ValueError, match="requires --use-tis"): + actor._should_reuse_hybrid_train_forward_log_probs(SimpleNamespace(num_rollout_minis=1)) + + +def test_hybrid_true_on_policy_skips_dedicated_actor_forward(monkeypatch): + assert actor_module is not None + actor = object.__new__(actor_module.MegatronTrainRayActor) + actor.args = SimpleNamespace( + compute_advantages_and_returns=True, + use_rollout_logprobs=False, + get_mismatch_metrics=False, + ) + actor.weights_backuper = SimpleNamespace(backup_tags={"actor"}) + monkeypatch.setattr( + actor_module, + "get_data_iterator", + lambda *_args, **_kwargs: pytest.fail("no forward iterator should be built"), + ) + + actor._hybrid_forward_subbatch( + {"total_lengths": [1]}, + rollout_id=0, + chunk_index=0, + reuse_train_forward_log_probs=True, + ) + + +def _rollout_batch(start: int, count: int) -> dict: + values = list(range(start, start + count)) + return { + "tokens": [[value] for value in values], + "total_lengths": [1] * count, + "response_lengths": [1] * count, + "loss_masks": [[1] for _ in values], + "rollout_log_probs": [[0.0] for _ in values], + "rewards": [1.0] * count, + "raw_reward": [1.0] * count, + } + + +def test_debug_rollout_chunking_slices_every_per_sample_container(): + assert actor_module is not None + shared = {"source": "frozen-rollout"} + rollout_data = { + "tokens": [[0], [1], [2], [3]], + "total_lengths": [1, 1, 1, 1], + "tensor_field": torch.arange(8).reshape(4, 2), + "array_field": np.arange(12).reshape(4, 3), + "tuple_field": ("a", "b", "c", "d"), + "shared": shared, + } + + chunks = actor_module.MegatronTrainRayActor._split_rollout_batch(rollout_data, 2) + + assert len(chunks) == 2 + assert chunks[0]["tokens"] == [[0], [1]] + assert chunks[1]["tokens"] == [[2], [3]] + assert torch.equal(chunks[0]["tensor_field"], torch.tensor([[0, 1], [2, 3]])) + assert torch.equal(chunks[1]["tensor_field"], torch.tensor([[4, 5], [6, 7]])) + np.testing.assert_array_equal(chunks[0]["array_field"], np.arange(6).reshape(2, 3)) + np.testing.assert_array_equal(chunks[1]["array_field"], np.arange(6, 12).reshape(2, 3)) + assert chunks[0]["tuple_field"] == ("a", "b") + assert chunks[1]["tuple_field"] == ("c", "d") + assert chunks[0]["shared"] is shared + assert chunks[1]["shared"] is shared + + +@pytest.mark.parametrize( + ("pipeline_enabled", "pipeline_overlap", "expected_fetch_sizes", "expected_forward_count"), + [ + (False, True, [4], 1), + (True, True, [2, 2], 2), + (True, False, [2, 2], 2), + ], +) +def test_train_hybrid_wires_one_update_around_flagged_actor_chunks( + monkeypatch, + pipeline_enabled, + pipeline_overlap, + expected_fetch_sizes, + expected_forward_count, +): + assert actor_module is not None + events = [] + actor = object.__new__(actor_module.MegatronTrainRayActor) + actor.args = SimpleNamespace( + rollout_batch_size=2, + n_samples_per_prompt=2, + global_batch_size=4, + num_steps_per_rollout=None, + num_iters_per_train_update=2, + hybrid_pipeline_forward=pipeline_enabled, + hybrid_pipeline_overlap=pipeline_overlap, + hybrid_pipeline_trace_dir=None, + hybrid_pipeline_fetch_timeout_s=1.0, + use_rollout_routing_replay=False, + multimodal_keys={"image": "image"}, + use_opd=False, + debug_train_only=False, + compute_advantages_and_returns=True, + use_routing_replay=False, + ref_update_interval=None, + num_rollout=1, + save=None, + ) + actor._hybrid_pipeline_chunk_plan = SimpleNamespace( + chunks_per_mini=2, + chunk_local_samples=2, + transfer_queue_batch_index=lambda mini_index, chunk_index: mini_index * 2 + chunk_index, + ) + actor._active_model_tag = "actor" + actor.rollout_data_postprocess = None + actor.model = actor.optimizer = actor.opt_param_scheduler = None + actor.tokenizer = actor.flops_counter = None + actor.prof = SimpleNamespace(step=lambda **_kwargs: events.append("profile")) + actor.weights_backuper = SimpleNamespace( + backup_tags={"actor"}, + backup=lambda tag: events.append(("backup", tag)), + ) + + def get_data(_task, _rollout_id, _fields, expected_samples, batch_index): + events.append(("fetch", expected_samples, batch_index)) + start = batch_index * expected_samples + return ( + _rollout_batch(start, expected_samples), + SimpleNamespace(global_indexes=list(range(start, start + expected_samples))), + ) + + actor._get_data_from_transfer_queue = get_data + actor.all_consumed = lambda *_args, **_kwargs: False + actor._restore_hybrid_pipeline_actor = lambda **kwargs: events.append(("restore", kwargs["chunk_index"])) + + def forward_chunk(batch, **kwargs): + events.append(("forward", kwargs["chunk_index"])) + return [list(range(len(batch["total_lengths"])))] + + actor._hybrid_actor_forward_without_switch = forward_chunk + actor._hybrid_forward_subbatch = lambda _batch, **kwargs: events.append(("forward", kwargs["chunk_index"])) + actor._switch_model = lambda tag: events.append(("switch", tag)) + actor._wait_for_previous_eval = lambda: None + actor._check_services_health = lambda: None + actor.update_weights = lambda: events.append("update_weights") + actor._run_step_evaluation = lambda *_args, **_kwargs: None + + monkeypatch.setattr( + actor_module.mpu, + "get_data_parallel_world_size", + lambda **_kwargs: 1, + ) + monkeypatch.setattr(actor_module.mpu, "get_data_parallel_rank", lambda: 0) + monkeypatch.setattr(actor_module.mpu, "get_data_parallel_group", lambda **_kwargs: None) + monkeypatch.setattr(actor_module, "timer", lambda *_args, **_kwargs: nullcontext()) + monkeypatch.setattr(actor_module, "inverse_timer", lambda *_args, **_kwargs: nullcontext()) + monkeypatch.setattr(actor_module, "emit_hybrid_pipeline_event", lambda *_args, **_kwargs: None) + monkeypatch.setattr( + actor_module, + "compute_advantages_and_returns", + lambda _args, _batch: events.append("advantages"), + ) + monkeypatch.setattr(actor_module, "log_rollout_data", lambda *_args, **_kwargs: None) + monkeypatch.setattr(actor_module, "get_data_iterator", lambda *_args, **_kwargs: ([], [])) + + def train(_rollout_id, _model, _optimizer, _scheduler, data_iterator, num_microbatches): + events.append( + ( + "optimizer_schedule", + data_iterator[0].micro_batch_indices if pipeline_enabled else None, + num_microbatches, + ) + ) + events.append("optimizer") + + monkeypatch.setattr(actor_module, "train", train) + monkeypatch.setattr(actor_module.train_dump_utils, "save_debug_train_data", lambda *_args, **_kwargs: None) + monkeypatch.setattr(actor_module, "Timer", lambda: SimpleNamespace(seq_lens=None)) + monkeypatch.setattr(actor_module, "log_perf_data", lambda *_args, **_kwargs: None) + monkeypatch.setattr(actor_module.tracking_utils, "flush_metrics", lambda *_args, **_kwargs: None) + monkeypatch.setattr(actor_module, "compute_rollout_step", lambda *_args, **_kwargs: 0) + monkeypatch.setattr(actor_module, "get_gloo_group", lambda: None) + + def all_gather_object(output, value, **_kwargs): + output[:] = [value] + + monkeypatch.setattr(actor_module.dist, "all_gather_object", all_gather_object) + monkeypatch.setattr(actor_module.dist, "barrier", lambda **_kwargs: None) + + actor.train_hybrid(rollout_id=0) + + assert [event[1] for event in events if isinstance(event, tuple) and event[0] == "fetch"] == (expected_fetch_sizes) + assert sum(isinstance(event, tuple) and event[0] == "forward" for event in events) == expected_forward_count + assert sum(isinstance(event, tuple) and event[0] == "restore" for event in events) == int(pipeline_enabled) + fetch_forward_events = [ + event[0] for event in events if isinstance(event, tuple) and event[0] in {"fetch", "forward"} + ] + if pipeline_enabled: + assert fetch_forward_events == ( + ["fetch", "forward", "fetch", "forward"] if pipeline_overlap else ["fetch", "fetch", "forward", "forward"] + ) + assert events.count("advantages") == 1 + assert events.count("optimizer") == 1 + assert events.count("update_weights") == 1 + optimizer_schedule = next( + event for event in events if isinstance(event, tuple) and event[0] == "optimizer_schedule" + ) + if pipeline_enabled: + assert optimizer_schedule[1:] == ([[0, 1], [2, 3]], [2]) + else: + assert optimizer_schedule[1:] == (None, []) diff --git a/tests/backends/megatron/test_reinforce_plus_plus_loss.py b/tests/backends/megatron/test_reinforce_plus_plus_loss.py index b8d43448e..c7c0408c3 100644 --- a/tests/backends/megatron/test_reinforce_plus_plus_loss.py +++ b/tests/backends/megatron/test_reinforce_plus_plus_loss.py @@ -15,6 +15,21 @@ compute_policy_loss = torch.compiler.disable(_compiled_compute_policy_loss) +def _load_loss_module(monkeypatch): + try: + import megatron # noqa: F401 + except ModuleNotFoundError: + megatron = ModuleType("megatron") + megatron_core = ModuleType("megatron.core") + megatron_core.mpu = SimpleNamespace() + megatron.core = megatron_core + monkeypatch.setitem(sys.modules, "megatron", megatron) + monkeypatch.setitem(sys.modules, "megatron.core", megatron_core) + if "ray" not in sys.modules: + monkeypatch.setitem(sys.modules, "ray", ModuleType("ray")) + return importlib.import_module("relax.backends.megatron.loss") + + def _response_mean(values: torch.Tensor, response_lengths: list[int], masks: list[torch.Tensor]) -> torch.Tensor: chunks = values.split(response_lengths) per_response = [(chunk * mask).sum() / mask.sum() for chunk, mask in zip(chunks, masks, strict=True)] @@ -81,18 +96,39 @@ def test_nonfinite_masked_tokens_are_scoped_to_reinforce_plus_plus(monkeypatch): assert torch.isnan(response_mean(values)) assert torch.isnan(token_sum(values)) - try: - import megatron # noqa: F401 - except ModuleNotFoundError: - megatron = ModuleType("megatron") - megatron_core = ModuleType("megatron.core") - megatron_core.mpu = SimpleNamespace() - megatron.core = megatron_core - monkeypatch.setitem(sys.modules, "megatron", megatron) - monkeypatch.setitem(sys.modules, "megatron.core", megatron_core) - loss_module = importlib.import_module("relax.backends.megatron.loss") + loss_module = _load_loss_module(monkeypatch) safe_response_mean = loss_module._get_reinforce_plus_plus_mask_safe_reducer(response_mean, masks) safe_token_sum = loss_module._get_reinforce_plus_plus_mask_safe_reducer(token_sum, masks) torch.testing.assert_close(safe_response_mean(values), torch.tensor(4.5), atol=0, rtol=0) torch.testing.assert_close(safe_token_sum(values), torch.tensor(8.0), atol=0, rtol=0) + + +def test_true_on_policy_advantages_use_rollout_log_probs_as_zero_kl_shape(monkeypatch): + loss_module = _load_loss_module(monkeypatch) + args = SimpleNamespace( + advantage_estimator="grpo", + is_vl_model=False, + kl_coef=0.0, + normalize_advantages=False, + qkv_format="thd", + true_on_policy_mode=True, + use_opd=False, + use_rollout_logprobs=False, + uses_unsplit_forward=False, + ) + rollout_data = { + "loss_masks": [torch.ones(2), torch.ones(1)], + "response_lengths": [2, 1], + "rewards": [1.5, -0.5], + "rollout_log_probs": [torch.tensor([-0.2, -0.4]), torch.tensor([-0.1])], + "total_lengths": [2, 1], + } + monkeypatch.setattr(cp_utils, "mpu", SimpleNamespace(get_context_parallel_world_size=lambda: 1)) + monkeypatch.setattr(loss_module.mpu, "is_pipeline_last_stage", lambda: True, raising=False) + + loss_module.compute_advantages_and_returns(args, rollout_data) + + assert "log_probs" not in rollout_data + assert torch.equal(rollout_data["advantages"][0], torch.tensor([1.5, 1.5])) + assert torch.equal(rollout_data["advantages"][1], torch.tensor([-0.5])) diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py new file mode 100644 index 000000000..97be3b4f4 --- /dev/null +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -0,0 +1,1181 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import importlib.util +import json +import sys +import types +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + + +SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "tools" / "analyze_hybrid_pipeline_benchmark.py" +SPEC = importlib.util.spec_from_file_location("analyze_hybrid_pipeline_benchmark", SCRIPT_PATH) +analyzer = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +sys.modules[SPEC.name] = analyzer +SPEC.loader.exec_module(analyzer) + +FINGERPRINT_0 = "00000000000000000000000000000001" +FINGERPRINT_1 = "00000000000000000000000000000002" +FULL_FINGERPRINT = "00000000000000000000000000000003" + + +def test_parser_help_renders_percent_targets(capsys): + with pytest.raises(SystemExit) as exc_info: + analyzer.build_parser().parse_args(["--help"]) + + assert exc_info.value.code == 0 + output = capsys.readouterr().out + assert "--enforce-targets" in output + assert "--enforce-steady-targets" in output + assert "--steady-campaign" in output + assert "5% throughput, 15% phase-1, and 80% overlap" in " ".join(output.split()) + + +def _install_fake_matplotlib(monkeypatch): + """Install the plotting API subset used by the analyzer for minimal CI.""" + matplotlib = types.ModuleType("matplotlib") + matplotlib.__path__ = [] + matplotlib.use = MagicMock() + pyplot = types.ModuleType("matplotlib.pyplot") + pyplot.tight_layout = MagicMock() + pyplot.close = MagicMock() + pyplot.figure = MagicMock() + pyplot.plot = MagicMock() + pyplot.xlabel = MagicMock() + pyplot.ylabel = MagicMock() + pyplot.legend = MagicMock() + pyplot.title = MagicMock() + pyplot.savefig = MagicMock(side_effect=lambda path, **_kwargs: Path(path).write_bytes(b"deterministic plot")) + + def axes(count): + return [MagicMock() for _ in range(count)] + + class AxesGrid: + def __init__(self, rows, columns): + self._axes = {(row, column): MagicMock() for row in range(rows) for column in range(columns)} + self._rows = [[self._axes[row, column] for column in range(columns)] for row in range(rows)] + for axis in self._axes.values(): + axis.get_legend_handles_labels.return_value = ([], []) + + def __getitem__(self, key): + return self._axes[key] + + def __iter__(self): + return iter(self._rows) + + pyplot.subplots = MagicMock( + side_effect=[ + (MagicMock(), axes(2)), + (MagicMock(), axes(3)), + (MagicMock(), AxesGrid(2, 2)), + (MagicMock(), AxesGrid(3, 2)), + (MagicMock(), axes(2)), + ] + ) + matplotlib.pyplot = pyplot + monkeypatch.setitem(sys.modules, "matplotlib", matplotlib) + monkeypatch.setitem(sys.modules, "matplotlib.pyplot", pyplot) + + +def _event( + event, + monotonic_ns, + *, + role, + rollout_id=4, + chunk_index=None, + sample_count=None, + fingerprint=None, + hostname="test-host", + pid=None, + global_rank=None, + **details, +): + row = { + "event": event, + "monotonic_ns": monotonic_ns, + "rollout_id": rollout_id, + "chunk_index": chunk_index, + "sample_count": sample_count, + "total_tokens": sample_count * 10 if sample_count is not None else None, + "response_tokens": sample_count * 4 if sample_count is not None else None, + "multimodal_tensor_bytes": sample_count * 100 if sample_count is not None else None, + "role": role, + "hostname": hostname, + "pid": (10 if role == "rollout" else 20) if pid is None else pid, + "global_rank": (-1 if role == "rollout" else 0) if global_rank is None else global_rank, + "cuda_visible_devices": "0,1,2,3", + "cuda_max_allocated_bytes": 1024, + "cuda_max_reserved_bytes": 2048, + "global_indexes_fingerprint": fingerprint, + } + row.update(details) + return row + + +def _write_run( + tmp_path, + *, + name="run", + pipeline_enabled=True, + pipeline_overlap=None, + hostname="test-host", + missing_event=None, + nonfinite=False, + producer_put_count=2, + producer_last_start_ns=700, + reuse_train_forward_log_probs=False, +): + if pipeline_overlap is None: + pipeline_overlap = pipeline_enabled + run_dir = tmp_path / name + timeline = run_dir / "timeline" + timeline.mkdir(parents=True) + (run_dir / "run_manifest.json").write_text( + json.dumps( + { + "condition": "experiment" if pipeline_enabled and pipeline_overlap else "baseline", + "hostname": hostname, + "order": "B1" if pipeline_enabled else "A1", + "seed": 7, + "rollout_seed": 7, + "num_rollout": 20, + "max_staleness": 2, + "global_batch_size": 4, + "rollout_batch_size": 2, + "n_samples_per_prompt": 2, + "num_iters_per_train_update": 2, + "hybrid_pipeline_forward": int(pipeline_enabled), + "hybrid_pipeline_overlap": int(pipeline_overlap), + "hybrid_reuse_train_logprobs": int(reuse_train_forward_log_probs), + "hybrid_pipeline_trace_dir": str(timeline), + "hybrid_pipeline_fetch_timeout_s": 600, + "git_commit": "a" * 40, + "git_branch": "perf/task21", + "git_status_porcelain": "", + "image_archive_sha256": "b" * 64, + "image_manifest_digest": "sha256:" + "c" * 64, + "image_id": "sha256:" + "d" * 64, + "transferqueue_commit": "e" * 40, + "python": "3.12.3", + "entrypoint": "bash target.sh hybrid-async", + } + ), + encoding="utf-8", + ) + + producer_shapes = { + 1: [(4, FULL_FINGERPRINT)], + 2: [(2, FINGERPRINT_0), (2, FINGERPRINT_1)], + # Fingerprints are additive, so three value-1 digests preserve + # the same full digest as the actor's value-1 + value-2 chunks. + 3: [(1, FINGERPRINT_0), (1, FINGERPRINT_0), (2, FINGERPRINT_0)], + } + if producer_put_count not in producer_shapes: + raise ValueError(f"unsupported producer_put_count={producer_put_count}") + producer = [] + for chunk_index, (sample_count, _) in enumerate(producer_shapes[producer_put_count]): + start_ns = producer_last_start_ns if chunk_index == producer_put_count - 1 else 100 + chunk_index * 100 + producer.append( + _event( + "tq_put_start", + start_ns, + role="rollout", + chunk_index=chunk_index, + sample_count=sample_count, + hostname=hostname, + event_id=f"put-{chunk_index}", + is_last=chunk_index == producer_put_count - 1, + ) + ) + for chunk_index, (sample_count, fingerprint) in enumerate(producer_shapes[producer_put_count]): + done_ns = 800 if chunk_index == producer_put_count - 1 else 150 + chunk_index * 100 + producer.append( + _event( + "tq_put_done", + done_ns, + role="rollout", + chunk_index=chunk_index, + sample_count=sample_count, + fingerprint=fingerprint, + hostname=hostname, + event_id=f"put-{chunk_index}", + is_last=chunk_index == producer_put_count - 1, + ) + ) + producer.sort(key=lambda row: row["monotonic_ns"]) + if nonfinite: + producer[0]["total_tokens"] = float("nan") + + if pipeline_enabled and pipeline_overlap: + actor = [ + _event("actor_restore_start", 210, role="actor", chunk_index=0, sample_count=4), + _event("actor_restore_end", 300, role="actor", chunk_index=0, sample_count=4), + _event("chunk_fetch_start", 310, role="actor", chunk_index=0, sample_count=2), + _event( + "chunk_fetch_end", + 350, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event( + "actor_forward_start", + 400, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event( + "actor_forward_end", + 600, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event("chunk_fetch_start", 810, role="actor", chunk_index=1, sample_count=2), + _event( + "chunk_fetch_end", + 850, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + _event( + "actor_forward_start", + 860, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + _event( + "actor_forward_end", + 1000, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + ] + elif pipeline_enabled: + actor = [ + _event("actor_restore_start", 210, role="actor", chunk_index=0, sample_count=4), + _event("actor_restore_end", 300, role="actor", chunk_index=0, sample_count=4), + _event("chunk_fetch_start", 310, role="actor", chunk_index=0, sample_count=2), + _event( + "chunk_fetch_end", + 350, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event("chunk_fetch_start", 810, role="actor", chunk_index=1, sample_count=2), + _event( + "chunk_fetch_end", + 850, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + _event( + "actor_forward_start", + 860, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event( + "actor_forward_end", + 900, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event( + "actor_forward_start", + 910, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + _event( + "actor_forward_end", + 1000, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + ] + else: + actor = [ + _event("chunk_fetch_start", 810, role="actor", chunk_index=0, sample_count=4), + _event( + "chunk_fetch_end", + 850, + role="actor", + chunk_index=0, + sample_count=4, + fingerprint=FULL_FINGERPRINT, + ), + _event("actor_restore_start", 860, role="actor", chunk_index=0, sample_count=4), + _event("actor_restore_end", 900, role="actor", chunk_index=0, sample_count=4), + _event( + "actor_forward_start", + 910, + role="actor", + chunk_index=0, + sample_count=4, + fingerprint=FULL_FINGERPRINT, + ), + _event( + "actor_forward_end", + 1000, + role="actor", + chunk_index=0, + sample_count=4, + fingerprint=FULL_FINGERPRINT, + ), + ] + if reuse_train_forward_log_probs: + actor = [ + row + for row in actor + if row["event"] + not in { + "actor_restore_start", + "actor_restore_end", + "actor_forward_start", + "actor_forward_end", + } + ] + actor += [ + _event("advantages_start", 1010, role="actor", sample_count=4), + _event("advantages_end", 1020, role="actor", sample_count=4), + _event("optimizer_start", 1030, role="actor", sample_count=4), + _event("optimizer_end", 1040, role="actor", sample_count=4), + ] + if missing_event is not None: + actor = [row for row in actor if row["event"] != missing_event] + + for filename, rows in (("rollout.jsonl", producer), ("actor.jsonl", actor)): + (timeline / filename).write_text( + "".join(json.dumps(row) + "\n" for row in rows), + encoding="utf-8", + ) + return run_dir + + +def test_complete_pipeline_trace_is_validated_and_summarized(tmp_path): + run_dir = _write_run(tmp_path) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.summary["validation"] == "passed" + assert analysis.summary["trace"]["producer_overlap_rollout_ratio"] == 1.0 + assert analysis.trace_rows[0]["actor_fetch_count"] == 2 + assert analysis.trace_rows[0]["actor_restore_count"] == 1 + assert analysis.trace_rows[0]["actor_forward_samples"] == 4 + assert (run_dir / "analysis" / "summary.json").is_file() + assert (run_dir / "analysis" / "trace_events.csv").is_file() + + +def test_train_forward_reuse_trace_skips_dedicated_actor_forward(tmp_path): + run_dir = _write_run( + tmp_path, + pipeline_enabled=False, + reuse_train_forward_log_probs=True, + ) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.summary["trace"]["reuse_train_forward_log_probs"] is True + assert analysis.trace_rows[0]["actor_forward_count"] == 0 + assert analysis.trace_rows[0]["actor_forward_samples"] == 4 + assert analysis.trace_rows[0]["actor_restore_count"] == 0 + + +@pytest.mark.parametrize("producer_put_count", [1, 2, 3]) +def test_producer_put_grouping_is_independent_from_actor_chunks(tmp_path, producer_put_count): + run_dir = _write_run(tmp_path, producer_put_count=producer_put_count) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.trace_rows[0]["producer_put_count"] == producer_put_count + assert analysis.trace_rows[0]["actor_fetch_count"] == 2 + assert analysis.trace_rows[0]["actor_forward_count"] == 2 + + +def test_delayed_put_done_is_not_strict_producer_overlap(tmp_path): + run_dir = _write_run(tmp_path, producer_last_start_ns=150) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + row = analysis.trace_rows[0] + assert row["first_forward_before_last_put_start"] is False + assert row["producer_overlap_s"] == 0 + assert row["first_forward_before_last_put_done"] is True + assert row["transfer_overlap_s"] > 0 + + +def test_baseline_trace_preserves_one_full_fetch_and_no_overlap(tmp_path): + run_dir = _write_run(tmp_path, pipeline_enabled=False) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.trace_rows[0]["actor_fetch_count"] == 1 + assert analysis.trace_rows[0]["actor_restore_count"] == 1 + assert analysis.trace_rows[0]["first_forward_before_last_put_start"] is False + + +def test_schedule_matched_baseline_fetches_all_chunks_before_forward(tmp_path): + run_dir = _write_run(tmp_path, pipeline_enabled=True, pipeline_overlap=False) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.trace_rows[0]["actor_fetch_count"] == 2 + assert analysis.trace_rows[0]["all_chunks_fetched_before_first_forward"] is True + assert analysis.summary["hybrid_pipeline_overlap"] is False + + +def test_measurement_scope_distinguishes_first_step_from_steady_state(): + assert analyzer._measurement_scope(((0, 0),)) == "fresh_process_first_step" + assert analyzer._measurement_scope(((4, 8), (9, 13))) == "steady_state" + assert analyzer._measurement_scope(((0, 2),)) == "selected_steps" + + +@pytest.mark.parametrize( + ("kwargs", "error"), + [ + ({"missing_event": "optimizer_end"}, "optimizer_end"), + ({"nonfinite": True}, "non-finite"), + ], +) +def test_invalid_trace_returns_nonzero_validation(tmp_path, kwargs, error, capsys): + run_dir = _write_run(tmp_path, **kwargs) + + status = analyzer.main( + [ + "--run-dir", + str(run_dir), + "--steady-windows", + "4-4", + "--expected-samples", + "4", + "--expected-actor-chunks", + "2", + "--validate-only", + ] + ) + + assert status == 2 + assert error in capsys.readouterr().err + + +def test_cross_hostname_trace_is_rejected(tmp_path): + run_dir = _write_run(tmp_path) + actor_path = run_dir / "timeline" / "actor.jsonl" + rows = [json.loads(line) for line in actor_path.read_text(encoding="utf-8").splitlines()] + rows[0]["hostname"] = "another-host" + actor_path.write_text( + "".join(json.dumps(row) + "\n" for row in rows), + encoding="utf-8", + ) + + with pytest.raises(analyzer.BenchmarkValidationError, match="multiple hostnames"): + analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + +def test_dirty_manifest_and_missing_reproducibility_artifacts_fail_closed(tmp_path): + dirty_run = _write_run(tmp_path, name="dirty") + manifest_path = dirty_run / "run_manifest.json" + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["git_status_porcelain"] = " M relax/file.py" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(analyzer.BenchmarkValidationError, match="dirty working tree"): + analyzer.analyze_run( + dirty_run, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + incomplete_run = _write_run(tmp_path, name="missing-artifacts") + with pytest.raises(analyzer.BenchmarkValidationError, match="reproducibility artifacts"): + analyzer.analyze_run( + incomplete_run, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + require_reproducibility_artifacts=True, + ) + + +def test_invalid_or_overlapping_windows_are_rejected(): + with pytest.raises(analyzer.BenchmarkValidationError, match="overlap"): + analyzer._parse_windows("4-8,8-10") + with pytest.raises(analyzer.BenchmarkValidationError, match="0 <= START <= END"): + analyzer._parse_windows("5-4") + + +def test_global_index_fingerprint_combination_ignores_chunk_grouping(): + producer = [ + {"global_indexes_fingerprint": FINGERPRINT_0}, + {"global_indexes_fingerprint": FINGERPRINT_1}, + ] + regrouped_fetch = [{"global_indexes_fingerprint": FULL_FINGERPRINT}] + + assert analyzer._combine_global_index_fingerprints( + producer, + context="producer", + ) == analyzer._combine_global_index_fingerprints( + regrouped_fetch, + context="fetch", + ) + + +def test_aggregate_throughput_uses_token_sum_over_time_sum(): + rows = [ + {"tag": "perf/step_time", "step": 4, "value": 3.0}, + {"tag": "perf/step_time", "step": 5, "value": 1.0}, + {"tag": "perf/step_token_per_s", "step": 4, "value": 100.0}, + {"tag": "perf/step_token_per_s", "step": 5, "value": 200.0}, + ] + + result = analyzer._aggregate_throughput( + rows, + "perf/step_token_per_s", + ((4, 5),), + ) + + assert result == 125.0 + assert analyzer._aggregate_samples_per_second(rows, ((4, 5),), samples_per_step=256) == 128.0 + + +def test_nvml_utilization_uses_registered_steady_step_wall_time(): + scalar_rows = [ + { + "tag": "perf/step_time", + "step": 4, + "value": 2.0, + "wall_time": 100.0, + } + ] + intervals = analyzer._steady_wall_time_intervals(scalar_rows, ((4, 4),)) + nvml_rows = [ + { + "gpu_index": 0.0, + "wall_time": wall_time, + "gpu_util_percent": utilization, + "memory_used_mib": memory, + "power_w": 100.0, + } + for wall_time, utilization, memory in ( + (97.0, 0.0, 1000.0), + (99.0, 80.0, 2000.0), + (101.0, 0.0, 3000.0), + ) + ] + + summary = analyzer._nvml_summary(nvml_rows, steady_intervals=intervals) + + assert intervals == [(98.0, 100.0)] + assert summary["per_gpu"]["0"]["steady_sample_count"] == 1 + assert summary["steady_mean_gpu_util_percent"] == 80.0 + assert summary["steady_idle_ratio_below_10_percent"] == 0.0 + # Peak memory deliberately remains a full-run safety metric. + assert summary["peak_memory_used_mib"] == 3000.0 + + +def _comparison_analysis( + condition, + seed, + throughput, + phase1, + *, + overlap=None, + accuracy=0.5, + peak_vram_mib=10_000, +): + if overlap is None: + overlap = condition == "experiment" + metrics = { + "perf/step_token_per_s": {"aggregate": throughput}, + "perf/step_resp_token_per_s": {"aggregate": throughput / 2}, + "perf/wall_clock_samples_per_s": {"aggregate": 12.8}, + "perf/step_time": {"mean": 20.0, "p95": 21.0}, + "perf/hybrid_phase1_time": {"mean": phase1}, + "rollout/raw_reward": {"mean": accuracy}, + "rollout/truncated_ratio": {"mean": 0.0}, + "train/loss": {"mean": 1.0}, + "train/grad_norm": {"mean": 0.5}, + "train/ppo_kl": {"mean": 0.0}, + "train/pg_clipfrac": {"mean": 0.0}, + } + trace_rows = [ + { + "rollout_id": rollout_id, + "first_forward_before_last_put_start": overlap, + "first_forward_before_last_put_done": overlap, + "producer_overlap_s": 1.0 if overlap else 0.0, + "transfer_overlap_s": 1.0 if overlap else 0.0, + "actor_fetch_samples": 256, + "actor_total_tokens": 10_000 + seed, + "actor_response_tokens": 5_000 + seed, + "actor_multimodal_tensor_bytes": 1_000_000 + seed, + "producer_lead_at_first_forward": 1.0, + "producer_global_indexes_fingerprint": f"{seed:032x}", + } + for rollout_id in range(4, 19) + ] + scalar_values = { + "perf/step_token_per_s": throughput, + "perf/step_resp_token_per_s": throughput / 2, + "perf/step_time": 20.0, + "perf/hybrid_phase1_time": phase1, + "rollout/raw_reward": accuracy, + "rollout/truncated_ratio": 0.0, + "train/loss": 1.0, + "train/grad_norm": 0.5, + "train/ppo_kl": 0.0, + "train/pg_clipfrac": 0.0, + } + scalar_rows = [ + {"tag": tag, "step": step, "value": value} for step in range(4, 19) for tag, value in scalar_values.items() + ] + return analyzer.RunAnalysis( + run_dir=Path(f"/{condition}-{seed}"), + manifest={ + "hostname": "test-host", + "condition": condition, + "order": ("A" if condition == "baseline" else "B") + str(seed), + "seed": seed, + "rollout_seed": seed, + "num_rollout": 20, + "max_staleness": 2, + "global_batch_size": 256, + "rollout_batch_size": 32, + "n_samples_per_prompt": 8, + "num_iters_per_train_update": 2, + "hybrid_pipeline_forward": True, + "hybrid_pipeline_overlap": condition == "experiment", + "hybrid_pipeline_fetch_timeout_s": 600, + "git_commit": "a" * 40, + "git_branch": "perf/task21", + "image_archive_sha256": "b" * 64, + "image_manifest_digest": "sha256:" + "c" * 64, + "image_id": "sha256:" + "d" * 64, + "transferqueue_commit": "e" * 40, + "python": "3.12.3", + "entrypoint": "bash target.sh hybrid-async", + "schema_version": 1, + "baseline_commit": "f" * 40, + "model_variant": "qwen3vl8b", + "model_name": "Qwen3-VL-8B-Instruct", + "model_dir": "/data01/LWX/Qwen3-VL-8B-Instruct", + "model_config_file": "/workspace/Relax/scripts/models/qwen3-vl-8B.sh", + "data_file": "/data01/LWX/openr1mm/train.parquet", + "rollout_max_response_len": 1024, + "rollout_max_prompt_len": 2048, + "rollout_max_context_len": 3072, + "actor_max_tokens_per_gpu": 6144, + "resource": {"actor": [1, 4], "rollout": [1, 1]}, + "rollout_num_gpus_per_engine": 1, + "physical_gpu_indices": [0, 1, 2, 3, 5], + "container_cuda_visible_devices": [0, 1, 2, 3, 4], + "gpu_hardware_fingerprint": "9" * 64, + "checkpoint_save": False, + "sglang_deterministic_inference": 1, + "sglang_mem_fraction_static": 0.8, + "load_debug_rollout_data": None, + "save_debug_rollout_data": None, + "save_debug_train_data": None, + }, + trace_rows=trace_rows, + actor_rank_rows=[], + scalar_rows=scalar_rows, + nvml_rows=[], + summary={ + "metrics": metrics, + "nvml": { + "gpu_count": 8, + "per_gpu": { + str(gpu_index): { + "steady_sample_count": 10, + "steady_mean_gpu_util_percent": 80.0, + "steady_idle_ratio_below_10_percent": 0.05, + } + for gpu_index in range(8) + }, + "steady_mean_gpu_util_percent": 80.0, + "steady_idle_ratio_below_10_percent": 0.05, + "peak_memory_used_mib": peak_vram_mib, + }, + }, + ) + + +def _steady_campaign_analysis(label, seed, throughput): + condition = "experiment" if label == "P+S" else "baseline" + analysis = _comparison_analysis( + condition, + seed, + throughput, + phase1=8.0 if label == "P+S" else 10.0, + overlap=label == "P+S", + ) + pool_size, reuse, pipeline = analyzer.STEADY_CAMPAIGN_CONDITIONS[label] + analysis.manifest.update( + { + "order": f"{label}-{seed}", + "num_rollout": 40, + "num_iters_per_train_update": 4, + "hybrid_pipeline_forward": pipeline, + "hybrid_pipeline_overlap": pipeline, + "mm_processor_pool_size": pool_size, + "hybrid_reuse_train_logprobs": reuse, + "physical_gpu_indices": [0, 1, 2, 3, 4], + } + ) + trace_template = analysis.trace_rows[0] + analysis.trace_rows[:] = [ + { + **trace_template, + "rollout_id": step, + "first_forward_before_last_put_start": label == "P+S", + "first_forward_before_last_put_done": label == "P+S", + } + for step in range(40) + ] + scalar_values = { + "perf/step_token_per_s": throughput, + "perf/step_resp_token_per_s": throughput / 2, + "perf/step_time": 20.0 * 100.0 / throughput, + "perf/hybrid_phase1_time": 8.0 if label == "P+S" else 10.0, + "rollout/raw_reward": 0.5, + "rollout/truncated_ratio": 0.0, + "rollout/image_count/mean": 1.0, + "train/loss": 1.0, + "train/grad_norm": 0.5, + "train/ppo_kl": 0.0, + "train/pg_clipfrac": 0.0, + } + analysis.scalar_rows[:] = [ + {"tag": tag, "step": step, "value": value} for step in range(40) for tag, value in scalar_values.items() + ] + analysis.summary["trace"] = {"rollout_count": 40} + analysis.summary["metrics"]["perf/step_time"] = { + "count": 30, + "mean": scalar_values["perf/step_time"], + "min": scalar_values["perf/step_time"], + "max": scalar_values["perf/step_time"], + "p50": scalar_values["perf/step_time"], + "p95": scalar_values["perf/step_time"], + } + analysis.summary["metrics"]["rollout/image_count/mean"] = { + "count": 30, + "mean": 1.0, + "min": 1.0, + "max": 1.0, + "p50": 1.0, + "p95": 1.0, + } + analysis.summary["metrics"]["perf/wall_clock_samples_per_s"] = {"aggregate": 12.8 * throughput / 100.0} + analysis.summary["nvml"] = { + "gpu_count": 5, + "per_gpu": { + str(gpu_index): { + "steady_sample_count": 30, + "steady_mean_gpu_util_percent": 80.0, + "steady_idle_ratio_below_10_percent": 0.05, + "peak_memory_used_mib": 10_000, + } + for gpu_index in range(5) + }, + "steady_mean_gpu_util_percent": 80.0, + "steady_idle_ratio_below_10_percent": 0.05, + "peak_memory_used_mib": 10_000, + } + return analysis + + +def _steady_campaign_analyses(): + throughputs = { + "B": (100.0, 102.0), + "P": (104.0, 106.08), + "P+R": (110.0, 112.2), + "P+S": (112.0, 114.24), + } + return [ + _steady_campaign_analysis(label, seed, throughput) + for label, values in throughputs.items() + for seed, throughput in zip((1, 2), values, strict=True) + ] + + +def test_four_condition_steady_campaign_reports_attributable_throughput(): + campaign = analyzer._build_steady_campaign( + _steady_campaign_analyses(), + windows=((10, 19), (20, 29), (30, 39)), + expected_gpu_count=5, + enforce_targets=True, + ) + + assert campaign["measurement_scope"] == "steady_state" + assert campaign["independent_seed_count"] == 2 + assert campaign["measured_optimizer_step_count"] == 30 + assert set(campaign["conditions"]) == {"B", "P", "P+R", "P+S"} + assert campaign["comparisons"]["P_vs_B"]["perf/step_token_per_s:geomean_improvement"] == pytest.approx(0.04) + assert campaign["comparisons"]["P+R_vs_P"]["paired_seed_count"] == 2 + assert campaign["comparisons"]["P+S_vs_B"]["paired_runs"][0]["rollout/image_count/mean:delta"] == 0 + assert campaign["conditions"]["B"]["runs"][0]["optimizer_update_count"] == 40 + assert campaign["conditions"]["B"]["runs"][0]["steady_actor_fetch_samples"] == 30 * 256 + assert campaign["conditions"]["B"]["windows"][0]["perf/step_token_per_s"]["mean"] == pytest.approx(101.0) + assert campaign["conditions"]["B"]["runs"][0]["windows"][0] == { + "start": 10, + "end": 19, + "perf/step_token_per_s": pytest.approx(100.0), + "perf/wall_clock_samples_per_s": pytest.approx(12.8), + } + + +def test_steady_input_workload_ignores_schedule_and_generated_output_variation(): + reference = _steady_campaign_analysis("B", 1, 100.0) + candidate = _steady_campaign_analysis("P+S", 1, 112.0) + fingerprints = [f"{index:032x}" for index in range(3)] + + for index, row in enumerate(reference.trace_rows): + row["producer_global_indexes_fingerprint"] = fingerprints[index % 3] + for index, row in enumerate(candidate.trace_rows): + row["producer_global_indexes_fingerprint"] = fingerprints[(-index) % 3] + row["actor_total_tokens"] += index + row["actor_response_tokens"] += index + row["actor_multimodal_tensor_bytes"] += index + + steady_steps = set(range(10, 40)) + assert analyzer._steady_input_workload(reference, steady_steps) == analyzer._steady_input_workload( + candidate, steady_steps + ) + + candidate.trace_rows[10]["producer_global_indexes_fingerprint"] = "f" * 32 + assert analyzer._steady_input_workload(reference, steady_steps) != analyzer._steady_input_workload( + candidate, steady_steps + ) + + +def test_four_condition_steady_campaign_fails_closed_on_protocol_gaps(): + analyses = _steady_campaign_analyses() + analyses.pop() + with pytest.raises(analyzer.BenchmarkValidationError, match="unpaired condition seeds"): + analyzer._build_steady_campaign( + analyses, + windows=((10, 19), (20, 29), (30, 39)), + expected_gpu_count=5, + enforce_targets=True, + ) + + analyses = _steady_campaign_analyses() + analyses[0].manifest["num_rollout"] = 39 + with pytest.raises(analyzer.BenchmarkValidationError, match="identical manifest field 'num_rollout'"): + analyzer._build_steady_campaign( + analyses, + windows=((10, 19), (20, 29), (30, 39)), + expected_gpu_count=5, + enforce_targets=True, + ) + + with pytest.raises(analyzer.BenchmarkValidationError, match="warmup-excluded"): + analyzer._build_steady_campaign( + _steady_campaign_analyses(), + windows=((0, 0),), + expected_gpu_count=5, + enforce_targets=True, + ) + + +def test_steady_campaign_descriptor_loads_completed_run_list(tmp_path): + completed = tmp_path / "completed-runs.txt" + completed.write_text("run-a\nrun-b\n", encoding="utf-8") + descriptor = tmp_path / "campaign.json" + descriptor.write_text( + json.dumps( + { + "schema_version": 1, + "completed_runs_file": "completed-runs.txt", + "steady_windows": "10-19,20-29,30-39", + } + ), + encoding="utf-8", + ) + + loaded = analyzer._load_steady_campaign_descriptor(descriptor) + + assert loaded["run_dirs"] == [(tmp_path / "run-a").resolve(), (tmp_path / "run-b").resolve()] + + +def test_preregistered_targets_pass_and_fail_deterministically(): + passing = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + + comparison = analyzer._build_comparison( + passing, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + assert comparison["token_throughput_geomean_speedup"] == pytest.approx(0.06) + assert comparison["hybrid_phase1_geomean_reduction"] == pytest.approx(0.2) + assert comparison["experiment_steady_producer_overlap_ratio"] == 1.0 + assert len(comparison["window_speedups"]) == 6 + assert comparison["distributions"]["paired_step_token_per_s_speedup"]["count"] == 2 + assert comparison["distributions"]["baseline_step_token_per_s"]["coefficient_of_variation"] > 0 + + failing = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 104, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 106.08, 8.16), + ] + with pytest.raises(analyzer.BenchmarkValidationError, match="below 5%"): + analyzer._build_comparison( + failing, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + +def test_performance_targets_require_schedule_matched_overlap_modes(): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + analyses[0].manifest["hybrid_pipeline_forward"] = False + + with pytest.raises(analyzer.BenchmarkValidationError, match="schedule-matched chunk forwarding"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + +def test_comparison_rejects_mixed_candidate_commits(): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + analyses[1].manifest["git_commit"] = "f" * 40 + + with pytest.raises(analyzer.BenchmarkValidationError, match="git_commit"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + +def test_comparison_rejects_mixed_or_incomplete_workload_manifests(): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + ] + analyses[1].manifest["rollout_max_response_len"] = 2048 + + with pytest.raises(analyzer.BenchmarkValidationError, match="rollout_max_response_len"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=False, + ) + + analyses[1].manifest["rollout_max_response_len"] = 1024 + del analyses[1].manifest["physical_gpu_indices"] + with pytest.raises(analyzer.BenchmarkValidationError, match="missing workload fields"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=False, + ) + + +def test_comparison_rejects_changed_global_index_fingerprint(): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + ] + analyses[1].trace_rows[0]["producer_global_indexes_fingerprint"] = "f" * 32 + + with pytest.raises(analyzer.BenchmarkValidationError, match="global-index fingerprints differ"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=False, + ) + + +def test_comparison_plot_bundle_is_generated(tmp_path, monkeypatch): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + comparison = analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + if importlib.util.find_spec("matplotlib") is None: + _install_fake_matplotlib(monkeypatch) + + generated = analyzer._plot_comparison( + analyses, + comparison, + tmp_path, + ((4, 8), (9, 13), (14, 18)), + ) + + assert {Path(path).name for path in generated} == { + "task21_correctness_quality.png", + "task21_gpu_util_vram.png", + "task21_phase1_overlap.png", + "task21_step_throughput.png", + "task21_window_summary.png", + } + assert all(Path(path).stat().st_size > 0 for path in generated) + + +def test_comparison_plot_bundle_fails_fast_without_matplotlib(tmp_path, monkeypatch): + monkeypatch.setitem(sys.modules, "matplotlib", None) + + with pytest.raises( + analyzer.BenchmarkValidationError, + match="plot generation requires the optional dependency 'matplotlib'", + ): + analyzer._plot_comparison([], {}, tmp_path, ()) + + +@pytest.mark.parametrize( + ("guardrail", "error"), + [ + ("overlap", "at least 80% steady producer overlap"), + ("accuracy", "accuracy drop exceeds 3 percentage points"), + ("truncation", "truncation-rate increase exceeds 2 percentage points"), + ("staleness", "average producer-lead increase exceeds 0.25"), + ("staleness_max", "observed producer lead exceeds configured max_staleness=2"), + ("vram", "peak VRAM increased"), + ("workload", "actor_total_tokens must match exactly"), + ("ppo_kl", "same-weight train/ppo_kl exceeds"), + ("pg_clipfrac", "same-weight train/pg_clipfrac exceeds"), + ], +) +def test_preregistered_guardrails_fail_closed(guardrail, error): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + experiment = analyses[1] + if guardrail == "overlap": + for row in experiment.trace_rows: + row["first_forward_before_last_put_start"] = False + elif guardrail == "accuracy": + experiment.summary["metrics"]["rollout/raw_reward"]["mean"] = 0.46 + elif guardrail == "truncation": + experiment.summary["metrics"]["rollout/truncated_ratio"]["mean"] = 0.03 + elif guardrail == "staleness": + for row in experiment.trace_rows: + row["producer_lead_at_first_forward"] = 1.3 + elif guardrail == "staleness_max": + experiment.trace_rows[0]["producer_lead_at_first_forward"] = 3.0 + elif guardrail == "vram": + experiment.summary["nvml"]["peak_memory_used_mib"] = 12_000 + elif guardrail == "workload": + for row in experiment.trace_rows: + row["actor_total_tokens"] *= 2 + elif guardrail == "ppo_kl": + experiment.summary["metrics"]["train/ppo_kl"]["mean"] = 1e-4 + elif guardrail == "pg_clipfrac": + experiment.summary["metrics"]["train/pg_clipfrac"]["mean"] = 1e-4 + + with pytest.raises(analyzer.BenchmarkValidationError, match=error): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + +def test_reproducibility_artifacts_require_all_zero_exit_statuses(tmp_path): + run_dir = tmp_path / "run" + for relative_path in analyzer.REPRODUCIBILITY_ARTIFACTS: + path = run_dir / relative_path + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("captured\n", encoding="utf-8") + for relative_path in analyzer.EXIT_STATUS_ARTIFACTS: + (run_dir / relative_path).write_text("0\n", encoding="utf-8") + + analyzer._require_reproducibility_artifacts(run_dir) + + (run_dir / "validation_exit_status.txt").write_text("2\n", encoding="utf-8") + with pytest.raises(analyzer.BenchmarkValidationError, match="non-zero or invalid"): + analyzer._require_reproducibility_artifacts(run_dir) diff --git a/tests/scripts/test_multimodal_hybrid_async_launcher.py b/tests/scripts/test_multimodal_hybrid_async_launcher.py new file mode 100644 index 000000000..f51206269 --- /dev/null +++ b/tests/scripts/test_multimodal_hybrid_async_launcher.py @@ -0,0 +1,234 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import json +import os +import subprocess +from pathlib import Path + + +REPO_ROOT = Path(__file__).resolve().parents[2] +LAUNCHER = REPO_ROOT / "scripts" / "training" / "multimodal" / "run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh" + + +def _argument_value(arguments: list[str], flag: str) -> str: + if flag in arguments: + index = arguments.index(flag) + return arguments[index + 1] + prefix = f"{flag}=" + for argument in arguments: + if argument.startswith(prefix): + return argument.removeprefix(prefix) + raise ValueError(f"{flag!r} is not present") + + +def _run_launcher( + tmp_path: Path, + *, + overrides: dict[str, str] | None = None, + mode: str = "hybrid-async", +) -> tuple[subprocess.CompletedProcess[str], list[str], Path, Path]: + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + capture_path = tmp_path / "ray-arguments.txt" + tensorboard_capture_path = tmp_path / "ray-tensorboard-dir.txt" + fake_ray = bin_dir / "ray" + fake_ray.write_text( + """#!/usr/bin/env bash +set -e +printf '%s\\n' "$@" > "${RAY_CAPTURE}" +printf '%s' "${TENSORBOARD_DIR:-}" > "${RAY_TENSORBOARD_CAPTURE}" +""", + encoding="utf-8", + ) + fake_ray.chmod(0o755) + + model_config = tmp_path / "test-model.sh" + model_config.write_text( + "MODEL_ARGS=(--test-model-arg test-model-value)\n", + encoding="utf-8", + ) + + env = { + "PATH": f"{bin_dir}:{os.environ['PATH']}", + "HOME": str(tmp_path), + "RELAX_ENTRYPOINT_MODE": "ray-job", + "MODEL_CONFIG_DIR": str(tmp_path), + "MODEL_CONFIG_FILE": str(model_config), + "RUNTIME_ENV_JSON": "{}", + "RAY_CAPTURE": str(capture_path), + "RAY_TENSORBOARD_CAPTURE": str(tensorboard_capture_path), + "EXP_DIR": str(tmp_path / "exp"), + "MODEL_DIR": str(tmp_path / "models"), + "DATA_DIR": str(tmp_path / "data"), + "PROJECT_NAME": "Relax/test-launcher", + "NUM_ROLLOUT": "2", + } + if overrides: + env.update(overrides) + + result = subprocess.run( + ["bash", str(LAUNCHER), mode], + cwd=REPO_ROOT, + env=env, + check=False, + capture_output=True, + text=True, + ) + arguments = capture_path.read_text(encoding="utf-8").splitlines() if capture_path.exists() else [] + return result, arguments, capture_path, tensorboard_capture_path + + +def test_launcher_preserves_default_qwen35_recipe(tmp_path): + result, arguments, _, tensorboard_capture = _run_launcher(tmp_path) + + assert result.returncode == 0, result.stderr + assert _argument_value(arguments, "--resource") == '{"actor": [1, 4], "rollout": [1, 4]}' + assert _argument_value(arguments, "--hf-checkpoint") == str(tmp_path / "models" / "Qwen3.5-9B") + assert _argument_value(arguments, "--ref-load") == str(tmp_path / "models" / "Qwen3.5-9B") + assert _argument_value(arguments, "--rollout-max-response-len") == "10240" + assert _argument_value(arguments, "--rollout-max-context-len") == "12288" + assert _argument_value(arguments, "--max-tokens-per-gpu") == "12288" + assert _argument_value(arguments, "--rollout-num-gpus-per-engine") == "2" + assert _argument_value(arguments, "--num-iters-per-train-update") == "2" + assert _argument_value(arguments, "--mm-processor-pool-size") == "0" + assert "--true-on-policy-mode" not in arguments + assert _argument_value(arguments, "--save") == f"{tmp_path / 'exp' / 'Qwen3.5-9B_mcore_8xgpu'}/" + assert _argument_value(arguments, "--save-interval") == "100" + assert "--rollout-result-dir" not in arguments + assert "--tensorboard-dir" not in arguments + assert tensorboard_capture.read_text(encoding="utf-8") == "" + assert _argument_value(arguments, "--test-model-arg") == "test-model-value" + + +def test_launcher_supports_no_save_memory_safe_smoke_configuration(tmp_path): + result, arguments, _, tensorboard_capture = _run_launcher( + tmp_path, + overrides={ + "MODEL_NAME": "Qwen3-VL-8B-Instruct", + "MODEL_RUN_NAME": "qwen3-vl-8b", + "CHECKPOINT_SAVE": "0", + "ROLLOUT_MAX_RESPONSE_LEN": "512", + "ROLLOUT_MAX_PROMPT_LEN": "2048", + "ROLLOUT_MAX_CONTEXT_LEN": "2560", + "ACTOR_MAX_TOKENS_PER_GPU": "6144", + "HYBRID_ROLLOUT_GPUS": "1", + "ROLLOUT_NUM_GPUS_PER_ENGINE": "1", + "NUM_ITERS_PER_TRAIN_UPDATE": "4", + }, + ) + + assert result.returncode == 0, result.stderr + assert _argument_value(arguments, "--resource") == '{"actor": [1, 4], "rollout": [1, 1]}' + assert _argument_value(arguments, "--hf-checkpoint") == str(tmp_path / "models" / "Qwen3-VL-8B-Instruct") + assert _argument_value(arguments, "--rollout-max-response-len") == "512" + assert _argument_value(arguments, "--rollout-max-context-len") == "2560" + assert _argument_value(arguments, "--max-tokens-per-gpu") == "6144" + assert _argument_value(arguments, "--rollout-num-gpus-per-engine") == "1" + assert _argument_value(arguments, "--num-iters-per-train-update") == "4" + assert "--save" not in arguments + assert "--save-interval" not in arguments + assert _argument_value(arguments, "--rollout-result-dir") == str(tmp_path / "exp" / "rollout_result") + expected_tensorboard_dir = tmp_path / "exp" / "tensorboard_log" + assert _argument_value(arguments, "--tensorboard-dir") == str(expected_tensorboard_dir) + assert tensorboard_capture.read_text(encoding="utf-8") == str(expected_tensorboard_dir) + runtime_env = json.loads(_argument_value(arguments, "--runtime-env-json")) + assert runtime_env["env_vars"]["TENSORBOARD_DIR"] == str(expected_tensorboard_dir) + log_names = [path.name for path in (tmp_path / "exp" / "logs").iterdir()] + assert len(log_names) == 1 + assert log_names[0].startswith("qwen3-vl-8b-GRPO-gpu5-hybrid-async-") + + +def test_launcher_rejects_invalid_checkpoint_switch_before_ray(tmp_path): + result, arguments, capture_path, _ = _run_launcher( + tmp_path, + overrides={"CHECKPOINT_SAVE": "2"}, + ) + + assert result.returncode == 2 + assert "CHECKPOINT_SAVE must be 0 or 1" in result.stderr + assert arguments == [] + assert not capture_path.exists() + + +def test_launcher_rejects_invalid_actor_chunk_count_before_ray(tmp_path): + result, arguments, capture_path, _ = _run_launcher( + tmp_path, + overrides={ + "HYBRID_PIPELINE_FORWARD": "1", + "NUM_ITERS_PER_TRAIN_UPDATE": "1", + }, + ) + + assert result.returncode == 2 + assert "requires NUM_ITERS_PER_TRAIN_UPDATE >= 2" in result.stderr + assert arguments == [] + assert not capture_path.exists() + + +def test_launcher_exposes_schedule_matched_no_overlap_control(tmp_path): + result, arguments, _, _ = _run_launcher( + tmp_path, + overrides={ + "HYBRID_PIPELINE_FORWARD": "1", + "HYBRID_PIPELINE_OVERLAP": "0", + }, + ) + + assert result.returncode == 0, result.stderr + assert "--hybrid-pipeline-forward" in arguments + assert "--no-hybrid-pipeline-overlap" in arguments + + +def test_launcher_exposes_processor_pool_and_train_logprob_reuse(tmp_path): + result, arguments, _, _ = _run_launcher( + tmp_path, + overrides={ + "MM_PROCESSOR_POOL_SIZE": "8", + "HYBRID_REUSE_TRAIN_LOGPROBS": "1", + }, + ) + + assert result.returncode == 0, result.stderr + assert _argument_value(arguments, "--mm-processor-pool-size") == "8" + assert "--true-on-policy-mode" in arguments + assert "--hybrid-pipeline-forward" not in arguments + + +def test_launcher_keeps_pipeline_and_train_logprob_reuse_as_separate_ablations(tmp_path): + result, arguments, capture_path, _ = _run_launcher( + tmp_path, + overrides={ + "HYBRID_PIPELINE_FORWARD": "1", + "HYBRID_REUSE_TRAIN_LOGPROBS": "1", + }, + ) + + assert result.returncode == 2 + assert "are separate ablations" in result.stderr + assert arguments == [] + assert not capture_path.exists() + + +def test_launcher_rejects_train_logprob_reuse_in_sync_mode(tmp_path): + result, arguments, capture_path, _ = _run_launcher( + tmp_path, + overrides={"HYBRID_REUSE_TRAIN_LOGPROBS": "1"}, + mode="sync", + ) + + assert result.returncode == 2 + assert "require MODE=hybrid-async" in result.stderr + assert arguments == [] + assert not capture_path.exists() + + +def test_launcher_rejects_overlap_control_without_pipeline_forward(tmp_path): + result, arguments, capture_path, _ = _run_launcher( + tmp_path, + overrides={"HYBRID_PIPELINE_OVERLAP": "0"}, + ) + + assert result.returncode == 2 + assert "requires HYBRID_PIPELINE_FORWARD=1" in result.stderr + assert arguments == [] + assert not capture_path.exists() diff --git a/tests/utils/test_arguments_opd_teacher_colocate.py b/tests/utils/test_arguments_opd_teacher_colocate.py index b87721536..5cdf6ce52 100644 --- a/tests/utils/test_arguments_opd_teacher_colocate.py +++ b/tests/utils/test_arguments_opd_teacher_colocate.py @@ -73,6 +73,226 @@ def test_sft_invalid_multimodal_strategy_option(arguments_module, argv, expected assert args.sft_invalid_multimodal_strategy == expected +def test_hybrid_pipeline_options_default_off(arguments_module): + arguments_module.RouterArgs = SimpleNamespace(add_cli_args=lambda parser, **_kwargs: parser) + parser = argparse.ArgumentParser() + arguments_module.get_slime_extra_args_provider()(parser) + + args = parser.parse_args([]) + + assert args.hybrid_pipeline_forward is False + assert args.hybrid_pipeline_overlap is True + assert args.hybrid_pipeline_trace_dir is None + assert args.hybrid_pipeline_fetch_timeout_s == 600.0 + + args = parser.parse_args(["--hybrid-pipeline-forward", "--no-hybrid-pipeline-overlap"]) + assert args.hybrid_pipeline_forward is True + assert args.hybrid_pipeline_overlap is False + + +def _hybrid_pipeline_args() -> SimpleNamespace: + return SimpleNamespace( + hybrid=True, + hybrid_pipeline_forward=True, + hybrid_pipeline_overlap=True, + hybrid_pipeline_trace_dir="/tmp/trace", + hybrid_pipeline_fetch_timeout_s=600.0, + use_dynamic_batch_size=True, + multimodal_keys={"image": "image"}, + advantage_estimator="grpo", + compute_advantages_and_returns=True, + enable_weights_backuper=True, + kl_coef=0.0, + use_kl_loss=False, + use_opd=False, + keep_old_actor=False, + true_on_policy_mode=False, + use_rollout_logprobs=False, + get_mismatch_metrics=False, + use_routing_replay=False, + use_rollout_routing_replay=False, + use_agentic_rollout=False, + partial_rollout=False, + use_dynamic_global_batch_size=False, + use_critic=False, + per_rank_fetch=False, + attention_dropout=0.0, + hidden_dropout=0.0, + tensor_model_parallel_size=2, + pipeline_model_parallel_size=1, + context_parallel_size=2, + expert_model_parallel_size=1, + expert_tensor_parallel_size=1, + offload_train=False, + offload_rollout=False, + num_iters_per_train_update=2, + rollout_batch_size=32, + n_samples_per_prompt=8, + global_batch_size=256, + ) + + +def test_hybrid_pipeline_supported_configuration_is_accepted(arguments_module): + arguments_module._validate_hybrid_pipeline_args(_hybrid_pipeline_args()) + + +def test_hybrid_pipeline_overlap_control_requires_chunk_forward(arguments_module): + args = _hybrid_pipeline_args() + args.hybrid_pipeline_forward = False + args.hybrid_pipeline_overlap = False + + with pytest.raises(ValueError, match="requires --hybrid-pipeline-forward"): + arguments_module._validate_hybrid_pipeline_args(args) + + +@pytest.mark.parametrize( + ("field", "value", "error"), + [ + ("hybrid", False, "only supported with --hybrid"), + ("use_dynamic_batch_size", False, "missing --use-dynamic-batch-size"), + ("multimodal_keys", None, "missing --multimodal-keys"), + ("enable_weights_backuper", False, "--disable-weights-backuper"), + ("kl_coef", 0.1, "--kl-coef"), + ("use_opd", True, "--use-opd"), + ("true_on_policy_mode", True, "--true-on-policy-mode"), + ("use_rollout_routing_replay", True, "--use-rollout-routing-replay"), + ("use_agentic_rollout", True, "--use-agentic-rollout"), + ("partial_rollout", True, "--partial-rollout"), + ("attention_dropout", 0.1, "--attention-dropout"), + ("tensor_model_parallel_size", 1, "--tensor-model-parallel-size"), + ("pipeline_model_parallel_size", 2, "--pipeline-model-parallel-size"), + ("context_parallel_size", 1, "--context-parallel-size"), + ("expert_model_parallel_size", 2, "--expert-model-parallel-size"), + ("expert_tensor_parallel_size", 2, "--expert-tensor-parallel-size"), + ("offload_train", True, "--offload-train"), + ("offload_rollout", True, "--offload-rollout"), + ("num_iters_per_train_update", 1, "must be >= 2"), + ("global_batch_size", 128, "exactly one optimizer mini per rollout"), + ], +) +def test_hybrid_pipeline_rejects_unsupported_configuration( + arguments_module, + field, + value, + error, +): + args = _hybrid_pipeline_args() + setattr(args, field, value) + + with pytest.raises(ValueError, match=error): + arguments_module._validate_hybrid_pipeline_args(args) + + +@pytest.mark.parametrize("timeout", [0, -1, float("nan"), float("inf")]) +def test_hybrid_pipeline_timeout_must_be_positive_and_finite(arguments_module, timeout): + args = _hybrid_pipeline_args() + args.hybrid_pipeline_fetch_timeout_s = timeout + + with pytest.raises(ValueError, match="finite value greater than 0"): + arguments_module._validate_hybrid_pipeline_args(args) + + +def _install_transfer_queue_contract(monkeypatch, *, global_indexes=True, custom_meta=True): + transfer_queue = ModuleType("transfer_queue") + + class BatchMeta: + __slots__ = ("global_indexes",) if global_indexes else ("partition_ids",) + + if custom_meta: + + class TransferQueueClient: + async def async_put(self, data, partition_id=None, custom_meta=None, is_last=False): + return None + + else: + + class TransferQueueClient: + async def async_put(self, data, partition_id=None, is_last=False): + return None + + transfer_queue.BatchMeta = BatchMeta + transfer_queue.TransferQueueClient = TransferQueueClient + monkeypatch.setitem(sys.modules, "transfer_queue", transfer_queue) + + +def test_hybrid_pipeline_transfer_queue_contract_is_checked_before_runtime( + arguments_module, + monkeypatch, +): + _install_transfer_queue_contract(monkeypatch) + monkeypatch.setattr("importlib.metadata.version", lambda _package: "0.1.10.dev0") + + arguments_module.check_hybrid_pipeline_transfer_queue_contract() + + +@pytest.mark.parametrize( + ("kwargs", "error"), + [ + ({"global_indexes": False}, "global_indexes"), + ({"custom_meta": False}, "custom_meta"), + ], +) +def test_hybrid_pipeline_transfer_queue_contract_fails_fast( + arguments_module, + monkeypatch, + kwargs, + error, +): + _install_transfer_queue_contract(monkeypatch, **kwargs) + monkeypatch.setattr("importlib.metadata.version", lambda _package: "0.1.10.dev0") + + with pytest.raises(RuntimeError, match=error): + arguments_module.check_hybrid_pipeline_transfer_queue_contract() + + +@pytest.mark.parametrize( + ("args", "expected"), + [ + ( + SimpleNamespace( + hybrid_pipeline_forward=False, + hybrid_pipeline_trace_dir="/tmp/timeline", + fully_async=True, + ), + "hybrid", + ), + ( + SimpleNamespace( + hybrid_pipeline_forward=False, + hybrid_pipeline_trace_dir=None, + fully_async=True, + ), + "version", + ), + ( + SimpleNamespace( + hybrid_pipeline_forward=False, + hybrid_pipeline_trace_dir=None, + fully_async=False, + ), + None, + ), + ], +) +def test_transfer_queue_runtime_selects_trace_contract_before_side_effects( + arguments_module, + monkeypatch, + args, + expected, +): + calls = [] + monkeypatch.setattr( + arguments_module, + "check_hybrid_pipeline_transfer_queue_contract", + lambda: calls.append("hybrid"), + ) + monkeypatch.setattr(arguments_module, "check_transfer_queue_version", lambda: calls.append("version")) + + arguments_module.check_transfer_queue_runtime(args) + + assert calls == ([] if expected is None else [expected]) + + def _opd_args() -> SimpleNamespace: return SimpleNamespace( loss_type="grpo", diff --git a/tests/utils/test_hybrid_forward_pipeline.py b/tests/utils/test_hybrid_forward_pipeline.py new file mode 100644 index 000000000..e679a106c --- /dev/null +++ b/tests/utils/test_hybrid_forward_pipeline.py @@ -0,0 +1,201 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import pytest + +from relax.utils.training.hybrid_forward_pipeline import ( + canonicalize_hybrid_microbatch_schedule, + execute_hybrid_forward_mini, + fetch_exact_chunk_with_timeout, +) + + +def test_first_chunk_forward_runs_before_later_chunk_is_ready(): + events = [] + chunk_one_ready = False + + def restore_actor(batch_index): + events.append(("restore", batch_index)) + + def fetch_chunk(batch_index): + nonlocal chunk_one_ready + if batch_index == 1: + assert chunk_one_ready, "chunk 1 was fetched before chunk 0 forward completed" + events.append(("fetch", batch_index)) + return {"total_lengths": [batch_index + 1]}, [100 + batch_index] + + def forward_chunk(batch, batch_index, global_indexes): + nonlocal chunk_one_ready + events.append(("forward", batch_index, global_indexes)) + if batch_index == 0: + chunk_one_ready = True + + chunks = execute_hybrid_forward_mini( + chunks_per_mini=2, + batch_index_for_chunk=lambda chunk_index: chunk_index, + restore_actor=restore_actor, + fetch_chunk=fetch_chunk, + forward_chunk=forward_chunk, + ) + + assert events == [ + ("restore", 0), + ("fetch", 0), + ("forward", 0, [100]), + ("fetch", 1), + ("forward", 1, [101]), + ] + assert chunks == [ + ({"total_lengths": [1]}, [100]), + ({"total_lengths": [2]}, [101]), + ] + + +def test_restore_occurs_once_per_optimizer_mini(): + restore_calls = [] + + for mini_index in range(3): + execute_hybrid_forward_mini( + chunks_per_mini=2, + batch_index_for_chunk=lambda chunk_index, mini=mini_index: mini * 2 + chunk_index, + restore_actor=restore_calls.append, + fetch_chunk=lambda batch_index: ({"total_lengths": [1]}, [batch_index]), + forward_chunk=lambda batch, batch_index, global_indexes: None, + ) + + assert restore_calls == [0, 2, 4] + + +def test_no_overlap_control_fetches_all_chunks_before_matched_forwards(): + events = [] + + chunks = execute_hybrid_forward_mini( + chunks_per_mini=3, + batch_index_for_chunk=lambda chunk_index: 10 + chunk_index, + restore_actor=lambda batch_index: events.append(("restore", batch_index)), + fetch_chunk=lambda batch_index: ( + events.append(("fetch", batch_index)) or {"total_lengths": [batch_index]}, + [100 + batch_index], + ), + forward_chunk=lambda _batch, batch_index, global_indexes: events.append( + ("forward", batch_index, global_indexes) + ), + overlap_producer=False, + ) + + assert events == [ + ("restore", 10), + ("fetch", 10), + ("fetch", 11), + ("fetch", 12), + ("forward", 10, [110]), + ("forward", 11, [111]), + ("forward", 12, [112]), + ] + assert [indexes for _, indexes in chunks] == [[110], [111], [112]] + + +def test_chunk_microbatch_schedule_is_replayed_on_canonical_batch(): + schedule = canonicalize_hybrid_microbatch_schedule( + [ + ([12, 10], [[1], [0]]), + ([13, 11], [[0, 1]]), + ], + [10, 11, 12, 13], + ) + + assert schedule == [[0], [2], [3, 1]] + + +@pytest.mark.parametrize( + ("chunk_schedules", "canonical_indexes", "error"), + [ + ([([10, 11], [[0]])], [10, 11], "cover each local sample exactly once"), + ([([10, 11], [[0, 0], [1]])], [10, 11], "cover each local sample exactly once"), + ([([10, 11], [[0, 2]])], [10, 11], "out-of-range local index"), + ([([10], [[0]])], [10, 11], "cover each canonical global index exactly once"), + ([([10, 12], [[0, 1]])], [10, 11], "outside canonical_global_indexes"), + ], +) +def test_chunk_microbatch_schedule_rejects_incomplete_or_invalid_coverage( + chunk_schedules, + canonical_indexes, + error, +): + with pytest.raises(ValueError, match=error): + canonicalize_hybrid_microbatch_schedule(chunk_schedules, canonical_indexes) + + +def test_invalid_chunk_count_fails_before_restore(): + restore_calls = [] + + with pytest.raises(ValueError, match="chunks_per_mini must be positive"): + execute_hybrid_forward_mini( + chunks_per_mini=0, + batch_index_for_chunk=lambda chunk_index: chunk_index, + restore_actor=restore_calls.append, + fetch_chunk=lambda batch_index: ({}, []), + forward_chunk=lambda batch, batch_index, global_indexes: None, + ) + + assert restore_calls == [] + + +def test_exact_fetch_retries_then_returns_complete_chunk(): + attempts = iter( + [ + (None, None), + ({"total_lengths": [1, 2]}, "metadata"), + ] + ) + now = [0.0] + + batch, metadata, elapsed = fetch_exact_chunk_with_timeout( + fetch_once=lambda: next(attempts), + expected_samples=2, + timeout_s=5, + error_context="rollout_id=7, mini_index=0, chunk_index=1", + clock=lambda: now[0], + sleep=lambda seconds: now.__setitem__(0, now[0] + seconds), + ) + + assert batch == {"total_lengths": [1, 2]} + assert metadata == "metadata" + assert elapsed == 0.1 + + +def test_exact_fetch_rejects_underfilled_chunk(): + with pytest.raises(RuntimeError, match="expected=2, last_returned=1"): + fetch_exact_chunk_with_timeout( + fetch_once=lambda: ({"total_lengths": [1]}, "metadata"), + expected_samples=2, + timeout_s=5, + error_context="rollout_id=7, mini_index=0, chunk_index=1", + ) + + +def test_exact_fetch_timeout_contains_actionable_context(): + now = [0.0] + + with pytest.raises( + TimeoutError, + match="rollout_id=7, mini_index=0, chunk_index=1, expected=2, last_returned=0", + ): + fetch_exact_chunk_with_timeout( + fetch_once=lambda: (None, None), + expected_samples=2, + timeout_s=0.2, + error_context="rollout_id=7, mini_index=0, chunk_index=1", + clock=lambda: now[0], + sleep=lambda seconds: now.__setitem__(0, now[0] + seconds), + ) + + +@pytest.mark.parametrize("timeout_s", [float("nan"), float("inf"), 0, -1]) +def test_exact_fetch_rejects_invalid_timeout(timeout_s): + with pytest.raises(ValueError, match="positive and finite"): + fetch_exact_chunk_with_timeout( + fetch_once=lambda: (None, None), + expected_samples=2, + timeout_s=timeout_s, + error_context="rollout_id=7", + ) diff --git a/tests/utils/test_hybrid_pipeline_trace.py b/tests/utils/test_hybrid_pipeline_trace.py new file mode 100644 index 000000000..9766ef60c --- /dev/null +++ b/tests/utils/test_hybrid_pipeline_trace.py @@ -0,0 +1,105 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import json +from types import SimpleNamespace + +import pytest +import torch + +from relax.utils.training import hybrid_pipeline_trace + + +def test_global_index_fingerprint_is_order_independent_and_composable(): + combined = hybrid_pipeline_trace.fingerprint_global_indexes([11, 12, 11]) + reordered = hybrid_pipeline_trace.fingerprint_global_indexes([12, 11, 11]) + first = hybrid_pipeline_trace.fingerprint_global_indexes([11]) + second = hybrid_pipeline_trace.fingerprint_global_indexes([12, 11]) + + assert combined == reordered + assert combined != hybrid_pipeline_trace.fingerprint_global_indexes([11, 12]) + assert int(combined, 16) == (int(first, 16) + int(second, 16)) % (1 << 128) + + +def test_trace_disabled_creates_no_files(tmp_path): + args = SimpleNamespace(hybrid_pipeline_trace_dir=None) + + record = hybrid_pipeline_trace.emit_hybrid_pipeline_event( + args, + "actor_forward_start", + rollout_id=1, + role="actor", + ) + + assert record is None + assert list(tmp_path.iterdir()) == [] + + +def test_trace_records_metrics_without_sample_content(tmp_path, monkeypatch): + hybrid_pipeline_trace._close_writers() + monkeypatch.setattr(hybrid_pipeline_trace.socket, "gethostname", lambda: "test-host") + monkeypatch.setattr(hybrid_pipeline_trace.os, "getpid", lambda: 123) + monkeypatch.setattr(hybrid_pipeline_trace, "_global_rank", lambda: 4) + monkeypatch.setattr(hybrid_pipeline_trace, "_cuda_memory_peaks", lambda: (1024, 2048)) + args = SimpleNamespace(hybrid_pipeline_trace_dir=str(tmp_path)) + batch = { + "total_lengths": [10, 20], + "response_lengths": [4, 5], + "multimodal_train_inputs": [ + {"pixel_values": torch.zeros((2, 3), dtype=torch.float32)}, + None, + ], + "prompt": "SECRET_SAMPLE_CONTENT", + } + + record = hybrid_pipeline_trace.emit_hybrid_pipeline_event( + args, + "chunk_fetch_end", + rollout_id=7, + role="actor", + chunk_index=1, + batch=batch, + global_indexes=[11, 12], + monotonic_ns=99, + details={"mini_index": 0, "is_last": False}, + ) + + assert record["sample_count"] == 2 + assert record["total_tokens"] == 30 + assert record["response_tokens"] == 9 + assert record["multimodal_tensor_bytes"] == 24 + assert record["global_indexes_fingerprint"] == hybrid_pipeline_trace.fingerprint_global_indexes([11, 12]) + assert record["cuda_max_allocated_bytes"] == 1024 + assert record["cuda_max_reserved_bytes"] == 2048 + + trace_files = list(tmp_path.glob("*.jsonl")) + assert [path.name for path in trace_files] == ["hybrid-pipeline-actor-test-host-pid123-rank4.jsonl"] + payload = trace_files[0].read_text(encoding="utf-8") + assert "SECRET_SAMPLE_CONTENT" not in payload + assert json.loads(payload)["event"] == "chunk_fetch_end" + hybrid_pipeline_trace._close_writers() + + +def test_trace_rejects_non_scalar_details(tmp_path): + args = SimpleNamespace(hybrid_pipeline_trace_dir=str(tmp_path)) + + with pytest.raises(TypeError, match="trace details must be scalar"): + hybrid_pipeline_trace.emit_hybrid_pipeline_event( + args, + "event", + rollout_id=1, + role="actor", + details={"sample": {"must": "not be serialized"}}, + ) + + +def test_trace_rejects_reserved_detail_fields(tmp_path): + args = SimpleNamespace(hybrid_pipeline_trace_dir=str(tmp_path)) + + with pytest.raises(ValueError, match="cannot replace record fields"): + hybrid_pipeline_trace.emit_hybrid_pipeline_event( + args, + "event", + rollout_id=1, + role="actor", + details={"event": "overwritten"}, + )