From c1c2c4097105809c68f667644462a4e7922ec0ea Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Wed, 29 Jul 2026 15:53:46 +0800 Subject: [PATCH 01/21] perf(hybrid): expose actor work while rollout keeps producing Gate a fixed sample-window actor-forward path behind an explicit switch so multimodal fetch and log-prob work can overlap later rollout production without changing full-batch GRPO or optimizer semantics. Constraint: The first release is limited to actor-only multimodal Hybrid GRPO on TP2/DP1/PP1/VPP1/CP2/EP1/ETP1 with offload and dropout disabled. Rejected: bind actor chunks to physical producer puts | FIRST_COMPLETED, tail flush, and backfill make put grouping dynamic. Rejected: token-budget drain or actor prefetch threads | they widen correctness and collective-order risk before the fixed-window path is measured. Confidence: medium Scope-risk: moderate Directive: Do not widen topology or claim performance until frozen-input parity, 8-GPU smoke, and paired A/B satisfy Issue #151. Tested: 94 host tests with 3 dependency-gated skips; 103 fixed-image tests including actor wiring and TransferQueue regressions; pre-commit all files. Not-tested: 8-GPU multimodal smoke, frozen-input parity, and 2x baseline/2x experiment measurements because the pinned dataset and eight simultaneously idle GPUs are unavailable. --- docs/en/guide/hybrid-training.md | 154 +- docs/zh/guide/hybrid-training.md | 137 +- relax/backends/megatron/actor.py | 587 +++++- relax/backends/megatron/data.py | 138 +- relax/utils/arguments.py | 171 +- .../utils/training/hybrid_forward_pipeline.py | 66 + relax/utils/training/hybrid_pipeline_trace.py | 191 ++ relax/utils/utils.py | 43 +- .../analyze_hybrid_pipeline_benchmark.py | 1629 +++++++++++++++++ ...n-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh | 96 +- tests/backends/megatron/test_data_vpp.py | 165 ++ .../test_hybrid_pipeline_actor_wiring.py | 177 ++ .../test_analyze_hybrid_pipeline_benchmark.py | 684 +++++++ .../test_arguments_opd_teacher_colocate.py | 205 +++ tests/utils/test_hybrid_forward_pipeline.py | 140 ++ tests/utils/test_hybrid_pipeline_trace.py | 105 ++ 16 files changed, 4586 insertions(+), 102 deletions(-) create mode 100644 relax/utils/training/hybrid_forward_pipeline.py create mode 100644 relax/utils/training/hybrid_pipeline_trace.py create mode 100644 scripts/tools/analyze_hybrid_pipeline_benchmark.py create mode 100644 tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py create mode 100644 tests/scripts/test_analyze_hybrid_pipeline_benchmark.py create mode 100644 tests/utils/test_hybrid_forward_pipeline.py create mode 100644 tests/utils/test_hybrid_pipeline_trace.py diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index adaed00f3..deebc116e 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -14,7 +14,7 @@ Concretely, Actor and Rollout still run on **separate GPU placement groups** (li | Dimension | Colocate (Sync) | Fully Async | Hybrid | | ------------------- | ---------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- | | **GPU layout** | Actor and Rollout time-share same GPUs | Actor / Rollout / ActorFwd / Reference each have own GPUs | Actor and Rollout on separate GPUs; ref / actor_fwd / adv share actor's GPUs | -| **Data pipeline** | TransferQueue, batch-synchronous | TransferQueue + StreamingDataLoader, fully streaming | TransferQueue + sub-batch streaming (`num-iters-per-train-update`) | +| **Data pipeline** | TransferQueue, batch-synchronous | TransferQueue + StreamingDataLoader, fully streaming | TransferQueue optimizer minis; optional producer-chunk actor-forward pipeline | | **Weight sync** | In-process tensor copy | NCCL broadcast via DCS (Checkpoint Engine) | Sync `UpdateWeightFromTensor` to rollout; TensorBackuper for ref/actor_fwd | | **Staleness** | `max_staleness = 0` (strict on-policy) | Configurable `max_staleness` | Configurable `max_staleness` | | **Roles deployed** | `actor`, `critic`, `rollout` | `actor`, `critic`, `rollout`, `advantages`, `reference`, `actor_fwd` | `actor`, `critic`, `rollout` (same as Colocate; ref/actor_fwd live inside actor) | @@ -113,15 +113,21 @@ ______________________________________________________________________ `relax/backends/megatron/actor.py:708` implements the hybrid training step in three phases: -1. **Collect sub-batches and compute forward log-probs (small memory footprint)** +1. **Collect optimizer minis and compute forward log-probs** - The global batch is split into `num_iters_per_train_update` sub-batches. For each sub-batch the actor: + The actor requests each optimizer mini derived from + `rollout_batch_size * n_samples_per_prompt / global_batch_size`. For each mini it: - pulls data from TransferQueue (`_get_data_from_transfer_queue("train", rollout_id, fields, batch_size, batch_index)`) - runs `_switch_model("ref")` (if ref weights are backed up) and computes ref log-probs - runs `_switch_model("teacher")` (if OPD teacher weights are backed up) and computes teacher log-probs - runs `_switch_model("old_actor" or "actor")` and computes current actor log-probs - - appends the enriched sub-batch to an in-memory list + - appends the enriched mini to an in-memory list + + With the optional incremental actor-forward pipeline enabled, one optimizer + mini is consumed in fixed sample windows aligned to the rollout producer's + nominal transfer threshold. Physical producer put grouping may vary; this + actor split does not change the advantage or optimizer boundary. 2. **Merge sub-batches and compute advantages globally** @@ -131,7 +137,8 @@ ______________________________________________________________________ A single `train(...)` call runs the optimizer step on the merged batch. Afterwards the actor backs up the new weights to the `actor` tag and (on the ref-update interval) refreshes the `ref` tag, then calls `self.update_weights()` to push the updated weights to rollout via `UpdateWeightFromTensor`. -The sub-batched forward keeps peak activation memory bounded — matching Fully Async behavior — while the merged training step preserves Colocate-style global statistics. +The forward phase remains bounded to one fetched unit at a time, while the +merged training step preserves Colocate-style global statistics. ______________________________________________________________________ @@ -143,7 +150,7 @@ ______________________________________________________________________ | ------------------------------- | ------------------------------------------------------------------------------------ | | `--hybrid` | Enable hybrid mode (resolves to `fully_async=True, colocate=True` internally) | | `--resource '{...}'` | Declare `actor` and `rollout` placement groups separately, e.g. `{"actor":[1,4],"rollout":[1,4]}` | -| `--num-iters-per-train-update` | Number of sub-batches per global batch (larger → smaller peak memory, more TQ polls) | +| `--num-iters-per-train-update` | Producer transfer threshold and the fixed actor fetch/forward chunk count | | `--max-staleness` | Off-policy budget (0 = strict on-policy, >0 allows staleness) | ### Optional but Common @@ -167,6 +174,133 @@ When `--hybrid` is set, `relax/utils/arguments.py` defaults the following (unles `--balance-data` requires `--hybrid` if you also want a streaming pipeline. The combination `--fully-async --balance-data` (without `--hybrid`) is rejected at argument parse time. ::: +### Incremental actor-forward pipeline + +Hybrid can optionally request fixed sample-count actor chunks from TransferQueue +instead of waiting for a complete optimizer mini: + +| Option | Default | Purpose | +| --- | --- | --- | +| `--hybrid-pipeline-forward` | off | Fetch and forward each fixed actor chunk as soon as enough samples are ready | +| `--hybrid-pipeline-trace-dir PATH` | unset | Write content-free producer, fetch, restore, forward, advantage, and optimizer events | +| `--hybrid-pipeline-fetch-timeout-s SECONDS` | `600` | Fail an incomplete chunk wait with rollout/mini/chunk context | + +The switch is deliberately off by default. With the reference Qwen3.5-9B +recipe (`global_batch_size=256`, `num_iters_per_train_update=2`, DP=1), the +producer targets 128 samples per transfer, while the actor always requests two +complete 128-sample chunks. Producer `async_put` grouping is intentionally not +part of the contract: `FIRST_COMPLETED` coalescing, tail flush, or backfill can +legitimately produce one, two, or more puts as long as all 256 samples and their +global-index fingerprint are conserved. The optional path: + +1. restores the actor exactly once for the optimizer mini, before waiting for + the first chunk; +2. fetches and forwards chunk 0 while rollout can continue producing chunk 1; +3. fetches and forwards chunk 1; +4. orders every per-sample field by `BatchMeta.global_indexes`; +5. computes advantages once over all 256 samples and performs one optimizer + step. + +It does not change producer transfer policy, multimodal preprocessing, pixel +tensor values, GRPO group boundaries, reward normalization, or optimizer +semantics. The additional actor fetch is intended to expose rollout/actor +overlap, not to reduce work. + +The first implementation is intentionally limited and fails fast instead of +silently falling back: + +| Dimension | Supported with the switch enabled | +| --- | --- | +| Mode | Hybrid | +| Workload | Multimodal, dynamic-batch GRPO | +| Forward roles | Actor only; no ref/KL, teacher/OPD, old actor, critic, or routing replay | +| Parallel topology | TP=2, DP=1, PP=1, VPP=1, CP=2, EP=1, ETP=1 | +| Offload | `offload_train=False`, `offload_rollout=False` | +| Dropout | Attention and hidden dropout both zero | +| Batch policy | Exactly one fixed optimizer mini per rollout (`rollout_batch_size * n_samples_per_prompt == global_batch_size`); no partial or dynamic-global batch | +| Log-prob source | Actor-computed log-prob; no true-on-policy or rollout-log-prob shortcut | +| TensorBackuper | Normal enabled backuper with only the `actor` tag | + +Chunk sizes must reconstruct the optimizer mini exactly and remain a multiple +of `n_samples_per_prompt`. Duplicate, missing, underfilled, or overfilled +`BatchMeta.global_indexes` terminate the step with an actionable error. +Startup also requires TransferQueue `>=0.1.10.dev0` with +`BatchMeta.global_indexes` and the `async_put(custom_meta=..., is_last=...)` +contract; an incompatible installation fails before Ray workers or rollout +producers can write data. + +The reference launcher exposes the options as environment variables: + +```bash +HYBRID_PIPELINE_FORWARD=1 \ +HYBRID_PIPELINE_TRACE_DIR=/data01/LWX/relax-task21/runs/smoke/timeline \ +HYBRID_PIPELINE_FETCH_TIMEOUT_S=600 \ +bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ + hybrid-async +``` + +Trace files are separated by hostname, role, PID, and global rank. They contain +timestamps, counts, token totals, multimodal tensor byte counts, CUDA peaks, +and an irreversible global-index fingerprint; prompts, responses, images, and +sample tensors are never serialized. + +Validate one run: + +```bash +python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ + --run-dir /data01/LWX/relax-task21/runs/smoke \ + --validate-only +``` + +Compare two baseline and two experiment runs and generate the registered CSV, +JSON, and curve artifacts: + +```bash +python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ + --run-dir /data01/LWX/relax-task21/runs/A1-baseline-seed20260728 \ + --run-dir /data01/LWX/relax-task21/runs/A2-baseline-seed20260729 \ + --run-dir /data01/LWX/relax-task21/runs/B1-experiment-seed20260728 \ + --run-dir /data01/LWX/relax-task21/runs/B2-experiment-seed20260729 \ + --output-dir /data01/LWX/relax-task21/comparison \ + --enforce-targets +``` + +The analyzer validates event closure, one restore and optimizer step, sample +conservation, same-host monotonic timing, finite metrics, producer/fetch +fingerprints, steady-step coverage, and the registered performance thresholds. +Producer put count is diagnostic only; actor fetch/forward count remains +strictly fixed. Strict producer overlap means the first actor forward starts +before the producer starts its final put. The final put completion is retained +only as a transfer-stage diagnostic because its trace write can lose a +scheduling race with the consumer. With `--enforce-targets`, the analyzer also +checks per-run strict producer overlap, step-time p95, +eight-GPU NVML coverage, peak VRAM, token/multimodal-byte workload, and the +raw-reward, truncation-rate, and staleness guardrails. It uses `sum(step_tokens) / +sum(step_time)` for aggregate throughput rather than averaging per-step rates. +GPU utilization and the below-10% idle ratio use only 500 ms NVML samples +whose wall time falls inside the registered steady step intervals reconstructed +from TensorBoard `perf/step_time`; sampled peak VRAM remains a full-run safety +metric. +The strict comparison additionally requires a clean run manifest, identical +candidate/image/TransferQueue identities, and non-empty input hashes, +dependency freeze, wheel hash, and launcher log for every run. +The manifest also records and cross-checks `max_staleness`, global/rollout +batch sizes, samples per prompt, and actor chunk count, so CLI expectations +cannot silently disagree with the measured workload. +The staleness curve is the trace-derived producer lead at the first actor +forward: the largest completed producer rollout ID minus the actor rollout ID +at that timestamp. The current rollout is considered ready once its actor fetch +completes even if the producer trace write is scheduled slightly later. The +lead must remain at most the manifest's configured `max_staleness` value +(`2` in the reference recipe), and its paired steady mean may increase by at +most `0.25`. + +To roll back, omit `--hybrid-pipeline-forward` (or set +`HYBRID_PIPELINE_FORWARD=0`). No checkpoint or dataset conversion is needed. +Before widening the support matrix, add collective-order and restore-count +tests for the new DP/PP/VPP or role graph, then rerun frozen-input parity, +multimodal smoke, and paired performance measurements. + ______________________________________________________________________ ## Quick Start @@ -183,7 +317,7 @@ ray job submit --address="http://127.0.0.1:8265" \ --resource '{"actor": [1, 4], "rollout": [1, 4]}' \ --max-staleness 2 \ --num-data-storage-units 1 \ - --num-iters-per-train-update 8 \ + --num-iters-per-train-update 2 \ --balance-data \ --hybrid \ "${MODEL_ARGS[@]}" \ @@ -200,9 +334,11 @@ Key points in this configuration: - 8 total GPUs split 4 + 4 between actor and rollout - `max-staleness 2` — actor may consume rollout output up to 2 steps behind the freshest weights -- `num-iters-per-train-update 8` — each global batch is split into 8 sub-batches for forward passes +- `num-iters-per-train-update 2` — rollout targets half-batch transfers and + the optional incremental path performs two fixed 128-sample actor fetches; + the physical producer put count may vary - `balance-data` — DP load balancing enabled -- GRPO algorithm with `--use-kl-loss` and `--use-tis` (these are algorithm flags, orthogonal to hybrid) +- GRPO algorithm with `--use-tis`; KL/ref forward is disabled in this recipe ______________________________________________________________________ diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index 13d96733f..08267aa1f 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -14,7 +14,7 @@ | 维度 | Colocate（同步） | Fully Async（全异步） | Hybrid | | ------------------- | ----------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------- | | **GPU 布局** | Actor 与 Rollout 分时复用同一组 GPU | Actor / Rollout / ActorFwd / Reference 各自独立 GPU | Actor 与 Rollout 独立 GPU；ref / actor_fwd / adv 复用 actor 的 GPU | -| **数据流水线** | TransferQueue，批同步 | TransferQueue + StreamingDataLoader，完全流式 | TransferQueue + 子批次流式（`num-iters-per-train-update`） | +| **数据流水线** | TransferQueue，批同步 | TransferQueue + StreamingDataLoader，完全流式 | TransferQueue optimizer mini；可选 producer-chunk actor-forward 流水线 | | **权重同步** | 进程内 tensor 拷贝 | 通过 DCS（Checkpoint Engine）做 NCCL broadcast | 同步的 `UpdateWeightFromTensor` 推给 rollout；ref/actor_fwd 走 TensorBackuper | | **Staleness** | `max_staleness = 0`（严格 on-policy） | 可配置 `max_staleness` | 可配置 `max_staleness` | | **部署的角色** | `actor`, `critic`, `rollout` | `actor`, `critic`, `rollout`, `advantages`, `reference`, `actor_fwd` | `actor`, `critic`, `rollout`（与 Colocate 相同；ref/actor_fwd 在 actor 内部） | @@ -115,15 +115,21 @@ ______________________________________________________________________ `relax/backends/megatron/actor.py:708` 实现的 Hybrid 训练步骤分为三个阶段： -1. **采集子批次并完成小批次 forward 计算（峰值显存小）** +1. **采集 optimizer mini 并计算 forward log-probs** - 全局 batch 被切分为 `num_iters_per_train_update` 份子批次。对每个子批次，actor 会： + actor 按 + `rollout_batch_size * n_samples_per_prompt / global_batch_size` + 推导 optimizer mini。对每个 mini，actor 会： - 从 TransferQueue 拉取数据（`_get_data_from_transfer_queue("train", rollout_id, fields, batch_size, batch_index)`） - 若已备份 ref 权重，执行 `_switch_model("ref")` 并计算 ref log-probs - 若已备份 teacher 权重（OPD 场景），执行 `_switch_model("teacher")` 并计算 teacher log-probs - 执行 `_switch_model("old_actor" 或 "actor")` 并计算当前 actor 的 log-probs - - 把扩充后的子批次追加到内存列表 + - 把扩充后的 mini 追加到内存列表 + + 打开可选增量 actor-forward 流水线后，一个 optimizer mini 会按与 rollout + producer 名义传输阈值对齐的固定 sample 窗口消费；物理 producer put 分组 + 可以变化，该额外 actor 切分不会改变 advantage 或 optimizer 边界。 2. **合并子批次并做全局 Advantages 归一化** @@ -133,7 +139,8 @@ ______________________________________________________________________ 一次 `train(...)` 调用基于合并后的 batch 完成优化器步进。随后 actor 把新权重备份到 `actor` tag（如果到达 ref 更新间隔，也刷新 `ref` tag），然后调用 `self.update_weights()` 通过 `UpdateWeightFromTensor` 把最新权重同步给 rollout。 -子批次 forward 控制了激活峰值显存（与 Fully Async 行为一致），而合并后的训练步则保留了 Colocate 风格的全局统计量。 +forward 阶段每次只处理一个已拉取单元，而合并后的训练步保留 Colocate 风格的 +全局统计量。 ______________________________________________________________________ @@ -145,7 +152,7 @@ ______________________________________________________________________ | ------------------------------- | ----------------------------------------------------------------------------------------------- | | `--hybrid` | 启用 Hybrid 模式（内部展开为 `fully_async=True, colocate=True`） | | `--resource '{...}'` | 分别声明 `actor` 与 `rollout` 的 placement group，例如 `{"actor":[1,4],"rollout":[1,4]}` | -| `--num-iters-per-train-update` | 每个全局 batch 切分的子批次数量（越大 → 峰值显存越小，TransferQueue 轮询次数越多） | +| `--num-iters-per-train-update` | producer 传输阈值，同时作为固定的 actor fetch/forward chunk 数量 | | `--max-staleness` | Off-policy 容忍度（0 = 严格 on-policy，>0 允许一定程度滞后） | ### 常用可选参数 @@ -169,6 +176,117 @@ ______________________________________________________________________ 如果你既想做流式数据流水线，又需要 `--balance-data`，必须使用 `--hybrid`。`--fully-async --balance-data`（不带 `--hybrid`）会在参数解析阶段被拒绝。 ::: +### 增量 actor-forward 流水线 + +Hybrid 可选地从 TransferQueue 按固定 sample count 增量请求 actor chunk， +而不是等待完整 optimizer mini： + +| 参数 | 默认值 | 作用 | +| --- | --- | --- | +| `--hybrid-pipeline-forward` | 关闭 | 足量 sample ready 后立即 fetch 固定 actor chunk 并执行 forward | +| `--hybrid-pipeline-trace-dir PATH` | 未设置 | 记录不含样本内容的 producer、fetch、restore、forward、advantage 和 optimizer 事件 | +| `--hybrid-pipeline-fetch-timeout-s SECONDS` | `600` | chunk 未完整到达时，带 rollout/mini/chunk 上下文终止等待 | + +该开关默认关闭。参考 Qwen3.5-9B 配方中， +`global_batch_size=256`、`num_iters_per_train_update=2`、DP=1，producer +以 128 samples 为传输目标，而 actor 固定请求两个完整的 128-sample chunk。 +producer 的 `async_put` 分组不是契约：`FIRST_COMPLETED` 合并、tail flush 或 +backfill 都可能合法地产生 1 次、2 次或更多次 put；只要 256 samples 及 +global-index fingerprint 完整守恒即可。打开开关后： + +1. 每个 optimizer mini 只 restore 一次 actor，并在等待首块前执行； +2. chunk 0 ready 后立即 fetch 和 forward，此时 rollout 可继续产生 chunk 1； +3. 再 fetch 和 forward chunk 1； +4. 按 `BatchMeta.global_indexes` 对所有 per-sample 字段恢复确定顺序； +5. 在完整 256 samples 上只计算一次 advantage，并只执行一次 optimizer step。 + +该路径不修改 producer 传输策略、多模态预处理、pixel tensor 数值、GRPO +group 边界、reward normalization 或 optimizer 语义。多出的一次 actor fetch +用于暴露 rollout/actor 重叠窗口，而不是减少工作量。 + +首版支持范围有意收窄；不支持的组合会 fail fast，不会静默回退： + +| 维度 | 打开开关时支持的范围 | +| --- | --- | +| 模式 | Hybrid | +| 负载 | 多模态、dynamic-batch GRPO | +| Forward role | 仅 actor；不支持 ref/KL、teacher/OPD、old actor、critic、routing replay | +| 并行拓扑 | TP=2、DP=1、PP=1、VPP=1、CP=2、EP=1、ETP=1 | +| Offload | `offload_train=False`、`offload_rollout=False` | +| Dropout | attention/hidden dropout 均为 0 | +| Batch 策略 | 每次 rollout 恰好一个固定 optimizer mini（`rollout_batch_size * n_samples_per_prompt == global_batch_size`）；不支持 partial/dynamic-global batch | +| Log-prob 来源 | actor 计算；不支持 true-on-policy 或 rollout-log-prob 快捷路径 | +| TensorBackuper | 启用普通 backuper，且只有 `actor` tag | + +chunk 大小必须精确重建 optimizer mini，并且是 `n_samples_per_prompt` 的 +整数倍。`BatchMeta.global_indexes` 发生重复、缺失、少取或多取时，当前 step +会带可定位信息直接失败。 +启动时还要求 TransferQueue `>=0.1.10.dev0`，并具备 +`BatchMeta.global_indexes` 与 `async_put(custom_meta=..., is_last=...)` +契约；版本或 API 不兼容会在 Ray worker 和 rollout producer 写入数据前失败。 + +参考脚本通过环境变量暴露这些参数： + +```bash +HYBRID_PIPELINE_FORWARD=1 \ +HYBRID_PIPELINE_TRACE_DIR=/data01/LWX/relax-task21/runs/smoke/timeline \ +HYBRID_PIPELINE_FETCH_TIMEOUT_S=600 \ +bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ + hybrid-async +``` + +Trace 按 hostname、role、PID 和 global rank 分文件。内容仅包括时间戳、 +样本与 token 计数、多模态 tensor 字节数、CUDA 峰值以及不可逆的 global-index +fingerprint；不会写 prompt、response、图片或样本 tensor。 + +校验单次运行： + +```bash +python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ + --run-dir /data01/LWX/relax-task21/runs/smoke \ + --validate-only +``` + +比较两次 baseline 和两次 experiment，并生成注册的 CSV、JSON 与曲线： + +```bash +python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ + --run-dir /data01/LWX/relax-task21/runs/A1-baseline-seed20260728 \ + --run-dir /data01/LWX/relax-task21/runs/A2-baseline-seed20260729 \ + --run-dir /data01/LWX/relax-task21/runs/B1-experiment-seed20260728 \ + --run-dir /data01/LWX/relax-task21/runs/B2-experiment-seed20260729 \ + --output-dir /data01/LWX/relax-task21/comparison \ + --enforce-targets +``` + +分析器会校验事件闭合、每 mini 一次 restore、每 step 一次 optimizer、样本 +守恒、同机 monotonic 计时、有限数值、producer/fetch fingerprint、稳态 step +覆盖率以及预注册性能门槛。producer put 次数仅作为诊断值，actor +fetch/forward 次数仍严格固定。严格 producer 重叠定义为首个 actor forward +早于 producer 最后一次 put 的开始时刻；最后一次 put 的完成时刻仅作为传输 +阶段诊断，因为其 trace 写入可能在调度上晚于 consumer。`--enforce-targets` +还会检查每次实验的严格 producer 重叠比例、 +step-time p95、8 卡 NVML 覆盖、峰值显存、token/多模态字节工作量和 +raw-reward、截断率、staleness 非劣护栏。aggregate throughput 使用 +`sum(step_tokens) / sum(step_time)`，不会对 per-step rate 做简单平均。 +GPU utilization 与低于 10% 的 idle ratio 只使用墙钟时间落入预注册稳态 +step 区间的 500 ms NVML 样本；区间由 TensorBoard `perf/step_time` 的 +step end wall time 和 duration 重建。sampled peak VRAM 仍使用 full-run +安全口径。 +严格比较还要求每次 run 的 manifest 为 clean tree，candidate/image/TransferQueue +身份完全一致，并存在非空的输入 hash、依赖 freeze、wheel hash 和 launcher log。 +manifest 还会记录并交叉校验 `max_staleness`、global/rollout batch size、 +每 prompt 样本数和 actor chunk 数，避免分析器 CLI 与实际工作负载静默不一致。 +staleness 曲线来自 trace：在 actor 首次 forward 时，用已完成 put 的最大 +producer rollout ID 减去当前 actor rollout ID；若 producer 的 trace 写入稍晚， +已完成的 actor fetch 本身可证明当前 rollout 已 ready。该值不得超过 manifest +记录的 `max_staleness`（参考配方为 `2`），paired 稳态均值最多增加 `0.25`。 + +回滚时去掉 `--hybrid-pipeline-forward`，或设置 +`HYBRID_PIPELINE_FORWARD=0`；无需转换 checkpoint 或数据集。若要扩展 +DP/PP/VPP 或 role graph，必须先新增 collective-order 与 restore-count 测试， +再重跑 frozen-input parity、多模态 smoke 和成对性能实验。 + ______________________________________________________________________ ## 快速开始 @@ -185,7 +303,7 @@ ray job submit --address="http://127.0.0.1:8265" \ --resource '{"actor": [1, 4], "rollout": [1, 4]}' \ --max-staleness 2 \ --num-data-storage-units 1 \ - --num-iters-per-train-update 8 \ + --num-iters-per-train-update 2 \ --balance-data \ --hybrid \ "${MODEL_ARGS[@]}" \ @@ -202,9 +320,10 @@ ray job submit --address="http://127.0.0.1:8265" \ - 8 GPU 总量，actor 与 rollout 各占 4 张 - `max-staleness 2` —— actor 可以消费比最新权重落后最多 2 个 step 的 rollout 输出 -- `num-iters-per-train-update 8` —— 每个全局 batch 在 forward 阶段被切分为 8 个子批次 +- `num-iters-per-train-update 2` —— rollout 以半批为传输目标，可选增量路径 + 固定执行两次 128-sample actor fetch；物理 producer put 次数可以变化 - `balance-data` —— 启用 DP 间负载均衡 -- 算法采用 GRPO，附带 `--use-kl-loss` 与 `--use-tis`（这些是算法参数，与 Hybrid 正交） +- 算法采用 GRPO 与 `--use-tis`；该参考配方关闭 KL/ref forward ______________________________________________________________________ diff --git a/relax/backends/megatron/actor.py b/relax/backends/megatron/actor.py index 4b948cc86..032e39189 100644 --- a/relax/backends/megatron/actor.py +++ b/relax/backends/megatron/actor.py @@ -63,6 +63,11 @@ from relax.utils.tracking_utils import init_tracking from relax.utils.training import train_dump_utils from relax.utils.training.data_fields import build_data_fields +from relax.utils.training.hybrid_forward_pipeline import ( + execute_hybrid_forward_mini, + fetch_exact_chunk_with_timeout, +) +from relax.utils.training.hybrid_pipeline_trace import emit_hybrid_pipeline_event from relax.utils.training.routing_replay import RoutingReplay from relax.utils.types import RolloutBatch from relax.utils.utils import ( @@ -81,7 +86,9 @@ from .data import ( ROLLOUT_MINI_LOCAL_SAMPLE_COUNTS_KEY, DataIterator, + build_hybrid_forward_chunk_plan, build_rollout_minibatch_plan, + canonicalize_rollout_chunks, concat_rollout_batches, get_data_iterator, log_perf_data, @@ -241,6 +248,7 @@ def _init( # internally via _switch_model and pushes weights to rollout via # UpdateWeightFromTensor instead of DCS. use_tensor_backuper = not self.args.fully_async or self.args.hybrid + self._hybrid_pipeline_chunk_plan = None if use_tensor_backuper: self.weights_backuper = TensorBackuper.create( source_getter=lambda: named_params_and_buffers( @@ -268,6 +276,14 @@ def _init( if args.update_weights_interval == 1: self.weights_backuper.backup("rollout_actor") + if getattr(self.args, "hybrid_pipeline_forward", False): + dp_size = mpu.get_data_parallel_world_size(with_context_parallel=False) + rollout_plan = build_rollout_minibatch_plan(self.args, dp_size) + self._hybrid_pipeline_chunk_plan = self._validate_hybrid_pipeline_runtime( + rollout_plan, + dp_size, + ) + update_weight_cls = UpdateWeightFromTensor if self.args.colocate else UpdateWeightFromDistributed # Push-side repack is decided by the HF config: an FP8 release auto-routes # through quantize_params_fp8, a compressed-tensors release through @@ -1071,7 +1087,14 @@ def compute_actor_log_prob(self, rollout_id: int) -> None: self.recv_weight_fully_async(rollout_id) log_perf_data_fwd(self.args, rollout_id) - def _hybrid_forward_subbatch(self, sub_batch: RolloutBatch) -> None: + def _hybrid_forward_subbatch( + self, + sub_batch: RolloutBatch, + *, + rollout_id: int, + chunk_index: int, + global_indexes: list[int] | None = None, + ) -> None: """Run the ref/teacher/actor forward passes for a single hybrid sub- batch in place. @@ -1113,17 +1136,234 @@ def _hybrid_forward_subbatch(self, sub_batch: RolloutBatch) -> None: ) # Actor forward - self._switch_model("old_actor" if self.args.keep_old_actor else "actor") + target_tag = "old_actor" if self.args.keep_old_actor else "actor" + emit_hybrid_pipeline_event( + self.args, + "actor_restore_start", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + details={"target_tag": target_tag}, + ) + self._switch_model(target_tag) + emit_hybrid_pipeline_event( + self.args, + "actor_restore_end", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + details={"target_tag": target_tag}, + ) if not self.args.use_rollout_logprobs or self.args.get_mismatch_metrics: if self.args.use_routing_replay: if self.args.use_rollout_routing_replay: os.environ["ROUTING_REPLAY_STAGE"] = "replay_forward" else: os.environ["ROUTING_REPLAY_STAGE"] = "record" + emit_hybrid_pipeline_event( + self.args, + "actor_forward_start", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + ) sub_batch.update(self.compute_log_prob(data_iterator, num_microbatches, store_prefix="")) + emit_hybrid_pipeline_event( + self.args, + "actor_forward_end", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + ) if self.args.use_rollout_routing_replay: RoutingReplay.clear_all_forward() + def _validate_hybrid_pipeline_runtime(self, rollout_plan, dp_size: int): + if dp_size != 1: + raise RuntimeError(f"--hybrid-pipeline-forward currently requires DP=1, detected DP={dp_size}") + if rollout_plan.num_rollout_minis != 1: + raise RuntimeError( + "--hybrid-pipeline-forward currently requires exactly one optimizer mini per rollout, " + f"detected {rollout_plan.num_rollout_minis}" + ) + pp_size = mpu.get_pipeline_model_parallel_world_size() + vpp_size = mpu.get_virtual_pipeline_model_parallel_world_size() or 1 + if pp_size != 1 or vpp_size != 1: + raise RuntimeError( + f"--hybrid-pipeline-forward currently requires PP=1 and VPP=1, detected PP={pp_size}, VPP={vpp_size}" + ) + tp_size = mpu.get_tensor_model_parallel_world_size() + cp_size = mpu.get_context_parallel_world_size() + ep_size = mpu.get_expert_model_parallel_world_size() + etp_size = int(getattr(self.args, "expert_tensor_parallel_size", 1) or 1) + if (tp_size, cp_size, ep_size, etp_size) != (2, 2, 1, 1): + raise RuntimeError( + "--hybrid-pipeline-forward currently requires TP=2, CP=2, EP=1, and ETP=1, " + f"detected TP={tp_size}, CP={cp_size}, EP={ep_size}, ETP={etp_size}" + ) + if self.args.offload_train or self.args.offload_rollout: + raise RuntimeError( + "--hybrid-pipeline-forward requires offload_train=False and offload_rollout=False, " + f"detected offload_train={self.args.offload_train}, " + f"offload_rollout={self.args.offload_rollout}" + ) + backup_tags = set(self.weights_backuper.backup_tags) + if backup_tags != {"actor"}: + raise RuntimeError( + f"--hybrid-pipeline-forward requires actor-only TensorBackuper tags, detected {sorted(backup_tags)}" + ) + if not self.args.compute_advantages_and_returns: + raise RuntimeError("--hybrid-pipeline-forward requires compute_advantages_and_returns=True") + return build_hybrid_forward_chunk_plan(self.args, rollout_plan, dp_size) + + def _restore_hybrid_pipeline_actor( + self, + *, + rollout_id: int, + chunk_index: int, + sample_count: int, + ) -> None: + emit_hybrid_pipeline_event( + self.args, + "actor_restore_start", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + sample_count=sample_count, + details={"target_tag": "actor"}, + ) + self._switch_model("actor") + emit_hybrid_pipeline_event( + self.args, + "actor_restore_end", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + sample_count=sample_count, + details={"target_tag": "actor"}, + ) + if self._active_model_tag != "actor": + raise RuntimeError( + "--hybrid-pipeline-forward actor restore completed with unexpected active tag " + f"{self._active_model_tag!r}" + ) + + def _hybrid_actor_forward_without_switch( + self, + sub_batch: RolloutBatch, + *, + rollout_id: int, + chunk_index: int, + global_indexes: list[int], + ) -> None: + if self._active_model_tag != "actor": + raise RuntimeError( + "--hybrid-pipeline-forward requires the actor model to remain active before " + f"chunk forward, detected {self._active_model_tag!r}" + ) + + data_iterator, num_microbatches = get_data_iterator(self.args, self.model, sub_batch) + emit_hybrid_pipeline_event( + self.args, + "actor_forward_start", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + ) + sub_batch.update(self.compute_log_prob(data_iterator, num_microbatches, store_prefix="")) + emit_hybrid_pipeline_event( + self.args, + "actor_forward_end", + rollout_id=rollout_id, + role="actor", + chunk_index=chunk_index, + batch=sub_batch, + global_indexes=global_indexes, + ) + if self._active_model_tag != "actor": + raise RuntimeError( + f"--hybrid-pipeline-forward actor tag changed during chunk forward: {self._active_model_tag!r}" + ) + + def _fetch_hybrid_pipeline_chunk( + self, + *, + rollout_id: int, + data_fields: list[str], + expected_samples: int, + batch_index: int, + mini_index: int, + ) -> tuple[RolloutBatch, list[int]]: + emit_hybrid_pipeline_event( + self.args, + "chunk_fetch_start", + rollout_id=rollout_id, + role="actor", + chunk_index=batch_index, + sample_count=expected_samples, + details={ + "batch_index": batch_index, + "mini_index": mini_index, + "expected_samples": expected_samples, + }, + ) + + def fetch_once(): + with timer("train_get_data"): + return self._get_data_from_transfer_queue( + "train", + rollout_id, + data_fields, + expected_samples, + batch_index, + ) + + error_context = ( + "--hybrid-pipeline-forward TransferQueue chunk failure: " + f"rollout_id={rollout_id}, mini_index={mini_index}, chunk_index={batch_index}" + ) + sub_batch, batch_meta, elapsed = fetch_exact_chunk_with_timeout( + fetch_once=fetch_once, + expected_samples=expected_samples, + timeout_s=self.args.hybrid_pipeline_fetch_timeout_s, + error_context=error_context, + ) + actual_samples = len(sub_batch["total_lengths"]) + global_indexes = list(getattr(batch_meta, "global_indexes", [])) + if len(global_indexes) != actual_samples: + raise RuntimeError( + "--hybrid-pipeline-forward received invalid BatchMeta.global_indexes: " + f"rollout_id={rollout_id}, mini_index={mini_index}, " + f"chunk_index={batch_index}, samples={actual_samples}, " + f"indexes={len(global_indexes)}" + ) + emit_hybrid_pipeline_event( + self.args, + "chunk_fetch_end", + rollout_id=rollout_id, + role="actor", + chunk_index=batch_index, + batch=sub_batch, + global_indexes=global_indexes, + details={ + "batch_index": batch_index, + "mini_index": mini_index, + "expected_samples": expected_samples, + "elapsed_s": elapsed, + }, + ) + return sub_batch, global_indexes + @staticmethod def _split_rollout_batch(rollout_data: RolloutBatch, num_chunks: int) -> List[RolloutBatch]: """Split a merged rollout batch (dict of per-sample lists) into at most @@ -1236,8 +1476,38 @@ def train_hybrid(self, rollout_id) -> None: plan = build_rollout_minibatch_plan(self.args, dp_size) batch_size = plan.mini_local_sample_request + pipeline_enabled = bool(getattr(self.args, "hybrid_pipeline_forward", False)) + trace_enabled = bool(getattr(self.args, "hybrid_pipeline_trace_dir", None)) + chunk_plan = self._hybrid_pipeline_chunk_plan if pipeline_enabled else None + if pipeline_enabled and chunk_plan is None: + raise RuntimeError( + "--hybrid-pipeline-forward was enabled without an initialized chunk plan; " + "the actor startup compatibility checks did not complete." + ) + + def phase1_timer(): + return timer("hybrid_phase1") if trace_enabled else nullcontext() + + data_fields = [ + "tokens", + "total_lengths", + "response_lengths", + "loss_masks", + "rollout_log_probs", + "rewards", + "raw_reward", + ] + if trace_enabled: + data_fields.append("truncated") + data_fields += ["rollout_routed_experts"] if self.args.use_rollout_routing_replay else [] + if self.args.multimodal_keys is not None: + data_fields.append("multimodal_train_inputs") + if self.args.use_opd and self.args.opd_type == "sglang": + data_fields.append("teacher_log_probs") + # ── Phase 1: Collect sub-batches and compute ref/actor forward in small chunks ── collected_batches: list[RolloutBatch] = [] + collected_global_indexes: list[int] = [] rollout_mini_local_sample_counts: list[int] = [] if self.args.debug_train_only: # Bypass the transfer queue and load the offline debug rollout dump @@ -1249,72 +1519,193 @@ def train_hybrid(self, rollout_id) -> None: full_batch_size = plan.mini_local_sample_request * plan.num_rollout_minis debug_data = get_debug_data(self.args, rollout_id, full_batch_size, dp_rank=mpu.get_data_parallel_rank()) post_process_rollout_data(self.args, debug_data) - for sub_batch in self._split_rollout_batch(debug_data, plan.num_rollout_minis): - if len(sub_batch["total_lengths"]) != batch_size: - raise RuntimeError( - f"debug rollout mini batch local size mismatch for train_hybrid({rollout_id}): " - f"expected {batch_size}, got {len(sub_batch['total_lengths'])}." + if pipeline_enabled: + for mini_index in range(plan.num_rollout_minis): + + def fetch_debug_chunk(tq_batch_index): + start = tq_batch_index * chunk_plan.chunk_local_samples + end = start + chunk_plan.chunk_local_samples + return _slice_rollout_batch(debug_data, start, end), list(range(start, end)) + + with phase1_timer(): + mini_chunks = execute_hybrid_forward_mini( + chunks_per_mini=chunk_plan.chunks_per_mini, + batch_index_for_chunk=lambda chunk_index: chunk_plan.transfer_queue_batch_index( + mini_index, chunk_index + ), + restore_actor=lambda tq_batch_index: self._restore_hybrid_pipeline_actor( + rollout_id=rollout_id, + chunk_index=tq_batch_index, + sample_count=batch_size, + ), + fetch_chunk=fetch_debug_chunk, + forward_chunk=lambda sub_batch, tq_batch_index, global_indexes: ( + self._hybrid_actor_forward_without_switch( + sub_batch, + rollout_id=rollout_id, + chunk_index=tq_batch_index, + global_indexes=global_indexes, + ) + ), + ) + mini_batch, canonical_indexes = canonicalize_rollout_chunks( + mini_chunks, + expected_sample_count=batch_size, ) - self._hybrid_forward_subbatch(sub_batch) - collected_batches.append(sub_batch) - rollout_mini_local_sample_counts.append(len(sub_batch["total_lengths"])) + collected_batches.append(mini_batch) + collected_global_indexes.extend(canonical_indexes) + rollout_mini_local_sample_counts.append(batch_size) + else: + debug_batches = self._split_rollout_batch(debug_data, plan.num_rollout_minis) + for mini_index, sub_batch in enumerate(debug_batches): + if len(sub_batch["total_lengths"]) != batch_size: + raise RuntimeError( + f"debug rollout mini batch local size mismatch for train_hybrid({rollout_id}): " + f"expected {batch_size}, got {len(sub_batch['total_lengths'])}." + ) + start = mini_index * batch_size + global_indexes = list(range(start, start + batch_size)) + with phase1_timer(): + self._hybrid_forward_subbatch( + sub_batch, + rollout_id=rollout_id, + chunk_index=mini_index, + global_indexes=global_indexes, + ) + collected_batches.append(sub_batch) + collected_global_indexes.extend(global_indexes) + rollout_mini_local_sample_counts.append(len(sub_batch["total_lengths"])) else: - batch_index = 0 - # Surface stuck-loop conditions: when the partition can never reach the - # requested batch_size (e.g. rollout dropped samples without refilling), - # `get_meta` keeps returning size=0 while `all_consumed` stays False, - # producing a silent infinite spin. Warn periodically so the failure mode - # is visible in logs instead of presenting as a totally silent hang. - loop_start = time.monotonic() - last_progress = loop_start - last_warn = loop_start - while batch_index < plan.num_rollout_minis and not self.all_consumed("train", rollout_id): - data_fields = [ - "tokens", - "total_lengths", - "response_lengths", - "loss_masks", - "rollout_log_probs", - "rewards", - "raw_reward", - ] - data_fields += ["rollout_routed_experts"] if self.args.use_rollout_routing_replay else [] - if self.args.multimodal_keys is not None: - data_fields.append("multimodal_train_inputs") - if self.args.use_opd and self.args.opd_type == "sglang": - data_fields.append("teacher_log_probs") - with timer("train_get_data"): - sub_batch, batch_meta = self._get_data_from_transfer_queue( - "train", rollout_id, data_fields, batch_size, batch_index - ) - if sub_batch is None: - now = time.monotonic() - stalled = now - last_progress - if now - last_warn >= 60.0 and stalled >= 60.0: - logger.warning( - f"train_hybrid({rollout_id}) batch_index={batch_index} stalled for {stalled:.0f}s: " - f"partition train_{rollout_id} has no data of size={batch_size} available but " - f"all_consumed=False. Likely the rollout under-filled this partition." + if pipeline_enabled: + seen_global_indexes: set[int] = set() + for mini_index in range(plan.num_rollout_minis): + + def fetch_pipeline_chunk(tq_batch_index): + sub_batch, global_indexes = self._fetch_hybrid_pipeline_chunk( + rollout_id=rollout_id, + data_fields=data_fields, + expected_samples=chunk_plan.chunk_local_samples, + batch_index=tq_batch_index, + mini_index=mini_index, + ) + overlap = seen_global_indexes.intersection(global_indexes) + if overlap: + raise RuntimeError( + "--hybrid-pipeline-forward received duplicate global indexes " + f"across optimizer minis: {sorted(overlap)}" + ) + seen_global_indexes.update(global_indexes) + return sub_batch, global_indexes + + with phase1_timer(): + mini_chunks = execute_hybrid_forward_mini( + chunks_per_mini=chunk_plan.chunks_per_mini, + batch_index_for_chunk=lambda chunk_index: chunk_plan.transfer_queue_batch_index( + mini_index, chunk_index + ), + restore_actor=lambda tq_batch_index: self._restore_hybrid_pipeline_actor( + rollout_id=rollout_id, + chunk_index=tq_batch_index, + sample_count=batch_size, + ), + fetch_chunk=fetch_pipeline_chunk, + forward_chunk=lambda sub_batch, tq_batch_index, global_indexes: ( + self._hybrid_actor_forward_without_switch( + sub_batch, + rollout_id=rollout_id, + chunk_index=tq_batch_index, + global_indexes=global_indexes, + ) + ), ) - last_warn = now - # Throttle the spin so the controller is not hammered with metadata - # polls while we wait for upstream data. - time.sleep(0.1) - continue - last_progress = time.monotonic() - last_warn = last_progress - batch_index += 1 - # Forward passes on this sub-batch (small memory footprint) - if len(sub_batch["total_lengths"]) != batch_size: - raise RuntimeError( - f"rollout mini batch local size mismatch for train_hybrid({rollout_id}), " - f"batch_index={batch_index - 1}: expected {batch_size}, " - f"got {len(sub_batch['total_lengths'])}." + mini_batch, canonical_indexes = canonicalize_rollout_chunks( + mini_chunks, + expected_sample_count=batch_size, + ) + collected_batches.append(mini_batch) + collected_global_indexes.extend(canonical_indexes) + rollout_mini_local_sample_counts.append(batch_size) + else: + # Preserve the baseline sample-count request and unbounded retry + # behavior when the optimization switch is disabled. + for mini_index in range(plan.num_rollout_minis): + loop_start = time.monotonic() + last_progress = loop_start + last_warn = loop_start + emit_hybrid_pipeline_event( + self.args, + "chunk_fetch_start", + rollout_id=rollout_id, + role="actor", + chunk_index=mini_index, + sample_count=batch_size, + details={ + "batch_index": mini_index, + "mini_index": mini_index, + "expected_samples": batch_size, + }, ) - self._hybrid_forward_subbatch(sub_batch) - collected_batches.append(sub_batch) - rollout_mini_local_sample_counts.append(len(sub_batch["total_lengths"])) + with phase1_timer(): + while True: + if self.all_consumed("train", rollout_id): + raise RuntimeError( + f"TransferQueue was consumed before train_hybrid({rollout_id}) " + f"received rollout mini {mini_index}." + ) + with timer("train_get_data"): + sub_batch, batch_meta = self._get_data_from_transfer_queue( + "train", + rollout_id, + data_fields, + batch_size, + mini_index, + ) + if sub_batch is not None: + break + now = time.monotonic() + stalled = now - last_progress + if now - last_warn >= 60.0 and stalled >= 60.0: + logger.warning( + f"train_hybrid({rollout_id}) batch_index={mini_index} " + f"stalled for {stalled:.0f}s: partition train_{rollout_id} " + f"has no data of size={batch_size} available but " + "all_consumed=False. Likely the rollout under-filled this partition." + ) + last_warn = now + time.sleep(0.1) + + actual_samples = len(sub_batch["total_lengths"]) + if actual_samples != batch_size: + raise RuntimeError( + f"rollout mini batch local size mismatch for train_hybrid({rollout_id}), " + f"batch_index={mini_index}: expected {batch_size}, got {actual_samples}." + ) + global_indexes = list(getattr(batch_meta, "global_indexes", [])) + emit_hybrid_pipeline_event( + self.args, + "chunk_fetch_end", + rollout_id=rollout_id, + role="actor", + chunk_index=mini_index, + batch=sub_batch, + global_indexes=global_indexes, + details={ + "batch_index": mini_index, + "mini_index": mini_index, + "expected_samples": batch_size, + "elapsed_s": time.monotonic() - loop_start, + }, + ) + self._hybrid_forward_subbatch( + sub_batch, + rollout_id=rollout_id, + chunk_index=mini_index, + global_indexes=global_indexes, + ) + collected_batches.append(sub_batch) + collected_global_indexes.extend(global_indexes) + rollout_mini_local_sample_counts.append(actual_samples) if len(collected_batches) != plan.num_rollout_minis: raise RuntimeError( @@ -1326,21 +1717,41 @@ def train_hybrid(self, rollout_id) -> None: self._switch_model("actor") # ── Phase 2: Merge sub-batches and compute advantages with correct global normalization ── - # Merge all sub-batch dicts: each value is a list, so we concatenate them. - rollout_data: RolloutBatch = {} - for sb in collected_batches: - for key, value in sb.items(): - if key not in rollout_data: - rollout_data[key] = [] - if isinstance(value, (list, tuple)) and not isinstance(value, (str, bytes)): - rollout_data[key].extend(value) - else: - rollout_data[key].append(value) + if pipeline_enabled: + rollout_data = concat_rollout_batches(collected_batches) + else: + # Keep the flag-off merge path byte-for-byte compatible with the + # pre-optimization implementation. + rollout_data: RolloutBatch = {} + for sb in collected_batches: + for key, value in sb.items(): + if key not in rollout_data: + rollout_data[key] = [] + if isinstance(value, (list, tuple)) and not isinstance(value, (str, bytes)): + rollout_data[key].extend(value) + else: + rollout_data[key].append(value) rollout_data[ROLLOUT_MINI_LOCAL_SAMPLE_COUNTS_KEY] = rollout_mini_local_sample_counts with inverse_timer("train_wait"), timer("train"): if self.args.compute_advantages_and_returns: + emit_hybrid_pipeline_event( + self.args, + "advantages_start", + rollout_id=rollout_id, + role="actor", + batch=rollout_data, + global_indexes=collected_global_indexes, + ) compute_advantages_and_returns(self.args, rollout_data) + emit_hybrid_pipeline_event( + self.args, + "advantages_end", + rollout_id=rollout_id, + role="actor", + batch=rollout_data, + global_indexes=collected_global_indexes, + ) if self.rollout_data_postprocess is not None: self.rollout_data_postprocess(self.args) @@ -1352,6 +1763,15 @@ def train_hybrid(self, rollout_id) -> None: if self.args.use_routing_replay: os.environ["ROUTING_REPLAY_STAGE"] = "replay_backward" with timer("actor_train"): + emit_hybrid_pipeline_event( + self.args, + "optimizer_start", + rollout_id=rollout_id, + role="actor", + batch=rollout_data, + global_indexes=collected_global_indexes, + details={"scope": "actor_train"}, + ) train( rollout_id, self.model, @@ -1360,6 +1780,15 @@ def train_hybrid(self, rollout_id) -> None: data_iterator, num_microbatches, ) + emit_hybrid_pipeline_event( + self.args, + "optimizer_end", + rollout_id=rollout_id, + role="actor", + batch=rollout_data, + global_indexes=collected_global_indexes, + details={"scope": "actor_train"}, + ) self.prof.step(rollout_id=rollout_id) @@ -1391,6 +1820,18 @@ def train_hybrid(self, rollout_id) -> None: ) all_total_lengths = sum(all_total_lengths, []) # flatten Timer().seq_lens = all_total_lengths + if trace_enabled: + response_token_counts = [ + int(mask.sum().item()) if isinstance(mask, torch.Tensor) else int(sum(mask)) + for mask in rollout_data["loss_masks"] + ] + all_response_token_counts = [None] * mpu.get_data_parallel_world_size(with_context_parallel=False) + dist.all_gather_object( + all_response_token_counts, + response_token_counts, + group=mpu.get_data_parallel_group(with_context_parallel=False), + ) + Timer().response_lens = sum(all_response_token_counts, []) mm_inputs = rollout_data.get("multimodal_train_inputs") if mm_inputs is not None: images_seqlens = _extract_images_seqlens(mm_inputs) diff --git a/relax/backends/megatron/data.py b/relax/backends/megatron/data.py index 9aff3da6f..b300fdd76 100644 --- a/relax/backends/megatron/data.py +++ b/relax/backends/megatron/data.py @@ -1,6 +1,7 @@ # Copyright (c) 2026 Relax Authors. All Rights Reserved. from argparse import Namespace +from collections import Counter from collections.abc import Sequence from copy import deepcopy from dataclasses import dataclass @@ -52,6 +53,20 @@ class RolloutMiniBatchPlan: mini_local_sample_request: int | None +@dataclass(frozen=True) +class HybridForwardChunkPlan: + chunks_per_mini: int + chunk_global_samples: int + chunk_local_samples: int + + def transfer_queue_batch_index(self, mini_index: int, chunk_index: int) -> int: + if mini_index < 0: + raise ValueError(f"mini_index must be non-negative, got {mini_index}") + if not 0 <= chunk_index < self.chunks_per_mini: + raise ValueError(f"chunk_index must be in [0, {self.chunks_per_mini}), got {chunk_index}") + return mini_index * self.chunks_per_mini + chunk_index + + def build_rollout_minibatch_plan(args: Namespace, dp_size: int) -> RolloutMiniBatchPlan: """Build a prompt-group based mini plan for one rollout partition.""" if dp_size <= 0: @@ -119,9 +134,63 @@ def build_rollout_minibatch_plan(args: Namespace, dp_size: int) -> RolloutMiniBa ) +def build_hybrid_forward_chunk_plan( + args: Namespace, + rollout_plan: RolloutMiniBatchPlan, + dp_size: int, +) -> HybridForwardChunkPlan: + """Split one optimizer mini into fixed actor sample-count chunks.""" + if dp_size <= 0: + raise ValueError(f"dp_size must be positive, got {dp_size}") + + chunks_per_mini = int(args.num_iters_per_train_update) + if chunks_per_mini < 2: + raise ValueError(f"hybrid pipeline forward requires num_iters_per_train_update >= 2, got {chunks_per_mini}") + + mini_global_samples = rollout_plan.mini_global_samples + mini_local_samples = rollout_plan.mini_local_sample_request + n_samples_per_prompt = rollout_plan.fixed_n_samples_per_prompt + if mini_global_samples is None or mini_local_samples is None or n_samples_per_prompt is None: + raise ValueError("hybrid pipeline forward requires a fixed-size rollout mini plan") + if mini_global_samples % chunks_per_mini != 0: + raise ValueError( + "mini_global_samples must be divisible by num_iters_per_train_update, " + f"got mini_global_samples={mini_global_samples}, chunks={chunks_per_mini}" + ) + + chunk_global_samples = mini_global_samples // chunks_per_mini + if chunk_global_samples % dp_size != 0: + raise ValueError( + "chunk_global_samples must be divisible by data parallel size, " + f"got chunk_global_samples={chunk_global_samples}, dp_size={dp_size}" + ) + if chunk_global_samples % n_samples_per_prompt != 0: + raise ValueError( + "hybrid pipeline forward chunks must preserve complete prompt groups, " + f"got chunk_global_samples={chunk_global_samples}, " + f"n_samples_per_prompt={n_samples_per_prompt}" + ) + + chunk_local_samples = chunk_global_samples // dp_size + if chunk_local_samples * chunks_per_mini != mini_local_samples: + raise ValueError( + "hybrid pipeline chunk plan does not reconstruct the optimizer mini, " + f"chunk_local_samples={chunk_local_samples}, chunks={chunks_per_mini}, " + f"mini_local_samples={mini_local_samples}" + ) + + return HybridForwardChunkPlan( + chunks_per_mini=chunks_per_mini, + chunk_global_samples=chunk_global_samples, + chunk_local_samples=chunk_local_samples, + ) + + def _same_scalar_value(lhs: Any, rhs: Any) -> bool: if isinstance(lhs, torch.Tensor) and isinstance(rhs, torch.Tensor): return torch.equal(lhs, rhs) + if isinstance(lhs, np.ndarray) and isinstance(rhs, np.ndarray): + return np.array_equal(lhs, rhs) return lhs == rhs @@ -130,24 +199,37 @@ def concat_rollout_batches(rollout_batches: Sequence[RolloutBatch]) -> RolloutBa if not rollout_batches: raise ValueError("rollout_batches must not be empty") + expected_keys = set(rollout_batches[0]) merged: RolloutBatch = {} tensor_batches: dict[str, list[torch.Tensor]] = {} + array_batches: dict[str, list[np.ndarray]] = {} scalar_values: dict[str, Any] = {} - for batch in rollout_batches: + for batch_index, batch in enumerate(rollout_batches): if batch is None: raise ValueError("rollout_batches must not contain None") + if set(batch) != expected_keys: + missing = sorted(expected_keys - set(batch)) + extra = sorted(set(batch) - expected_keys) + raise ValueError(f"rollout batch {batch_index} schema mismatch: missing={missing}, extra={extra}") batch_size = len(batch.get("total_lengths", [])) if batch_size <= 0: raise ValueError("rollout mini batch must contain at least one sample") for key, value in batch.items(): if isinstance(value, (list, tuple)) and not isinstance(value, (str, bytes)): + if len(value) != batch_size: + raise ValueError( + f"Per-sample rollout field {key!r} in batch {batch_index} has " + f"length={len(value)}, expected={batch_size}" + ) if key not in merged: merged[key] = [] merged[key].extend(value) elif isinstance(value, torch.Tensor) and value.ndim > 0 and value.size(0) == batch_size: tensor_batches.setdefault(key, []).append(value) + elif isinstance(value, np.ndarray) and value.ndim > 0 and value.shape[0] == batch_size: + array_batches.setdefault(key, []).append(value) else: if key in scalar_values: if not _same_scalar_value(scalar_values[key], value): @@ -160,6 +242,11 @@ def concat_rollout_batches(rollout_batches: Sequence[RolloutBatch]) -> RolloutBa raise ValueError(f"Rollout field {key!r} appears as both list-like and tensor-like") merged[key] = torch.cat(tensors, dim=0) + for key, arrays in array_batches.items(): + if key in merged or key in tensor_batches: + raise ValueError(f"Rollout field {key!r} has inconsistent batched types") + merged[key] = np.concatenate(arrays, axis=0) + for key, value in scalar_values.items(): if key in merged: raise ValueError(f"Rollout field {key!r} appears as both scalar and batched data") @@ -168,6 +255,55 @@ def concat_rollout_batches(rollout_batches: Sequence[RolloutBatch]) -> RolloutBa return merged +def canonicalize_rollout_chunks( + chunks_with_global_indexes: Sequence[tuple[RolloutBatch, Sequence[int]]], + expected_sample_count: int, +) -> tuple[RolloutBatch, list[int]]: + """Merge chunks and restore deterministic TransferQueue global-index + order.""" + if expected_sample_count <= 0: + raise ValueError(f"expected_sample_count must be positive, got {expected_sample_count}") + if not chunks_with_global_indexes: + raise ValueError("chunks_with_global_indexes must not be empty") + + chunks: list[RolloutBatch] = [] + global_indexes: list[int] = [] + for chunk_index, (chunk, indexes) in enumerate(chunks_with_global_indexes): + batch_size = len(chunk.get("total_lengths", [])) + normalized_indexes = list(indexes) + if len(normalized_indexes) != batch_size: + raise ValueError( + f"chunk {chunk_index} BatchMeta.global_indexes length mismatch: " + f"indexes={len(normalized_indexes)}, samples={batch_size}" + ) + if not all(type(index) is int for index in normalized_indexes): + raise TypeError(f"chunk {chunk_index} BatchMeta.global_indexes must contain only int values") + chunks.append(chunk) + global_indexes.extend(normalized_indexes) + + if len(global_indexes) != expected_sample_count: + raise ValueError( + f"hybrid pipeline sample count mismatch: expected={expected_sample_count}, actual={len(global_indexes)}" + ) + if len(set(global_indexes)) != len(global_indexes): + duplicates = sorted(index for index, count in Counter(global_indexes).items() if count > 1) + raise ValueError(f"hybrid pipeline BatchMeta.global_indexes contain duplicates: {duplicates}") + + merged = concat_rollout_batches(chunks) + permutation = sorted(range(len(global_indexes)), key=global_indexes.__getitem__) + canonical_indexes = [global_indexes[index] for index in permutation] + + for key, value in list(merged.items()): + if isinstance(value, list) and len(value) == expected_sample_count: + merged[key] = [value[index] for index in permutation] + elif isinstance(value, torch.Tensor) and value.ndim > 0 and value.size(0) == expected_sample_count: + merged[key] = value[permutation] + elif isinstance(value, np.ndarray) and value.ndim > 0 and value.shape[0] == expected_sample_count: + merged[key] = value[permutation] + + return merged, canonical_indexes + + PAD_RULES = { # shape like [1, 128, 1036] "input_features": dict( diff --git a/relax/utils/arguments.py b/relax/utils/arguments.py index 22430e883..353d4f813 100644 --- a/relax/utils/arguments.py +++ b/relax/utils/arguments.py @@ -2,6 +2,7 @@ import argparse import json +import math import os import warnings from typing import Any @@ -74,6 +75,54 @@ def _positive_int(value: str) -> int: return parsed +def check_hybrid_pipeline_transfer_queue_contract() -> None: + """Validate the TransferQueue API needed by chunked Hybrid forwarding + before any rollout can write data.""" + import inspect + + check_transfer_queue_version() + try: + from transfer_queue import BatchMeta, TransferQueueClient + except (ImportError, AttributeError) as exc: + raise RuntimeError( + "transferqueue does not expose BatchMeta and TransferQueueClient required by " + "Hybrid pipeline forwarding/tracing. Upgrade with:\n" + f" {_TQ_UPGRADE_CMD}\nor use the latest image." + ) from exc + + slots = set(getattr(BatchMeta, "__slots__", ())) + annotations = set(getattr(BatchMeta, "__annotations__", {})) + if "global_indexes" not in slots | annotations and not hasattr(BatchMeta, "global_indexes"): + raise RuntimeError( + "transferqueue BatchMeta does not expose global_indexes required by " + "Hybrid pipeline forwarding/tracing. Upgrade with:\n" + f" {_TQ_UPGRADE_CMD}\nor use the latest image." + ) + + try: + async_put_parameters = inspect.signature(TransferQueueClient.async_put).parameters + except (TypeError, ValueError, AttributeError) as exc: + raise RuntimeError( + "transferqueue TransferQueueClient.async_put cannot be inspected for " + "Hybrid pipeline forwarding/tracing compatibility." + ) from exc + missing = sorted({"custom_meta", "is_last"} - set(async_put_parameters)) + if missing: + raise RuntimeError( + "transferqueue TransferQueueClient.async_put is missing parameters " + f"{missing} required by Hybrid pipeline forwarding/tracing. Upgrade with:\n" + f" {_TQ_UPGRADE_CMD}\nor use the latest image." + ) + + +def check_transfer_queue_runtime(args) -> None: + """Select the startup contract required by the resolved execution mode.""" + if getattr(args, "hybrid_pipeline_forward", False) or getattr(args, "hybrid_pipeline_trace_dir", None): + check_hybrid_pipeline_transfer_queue_contract() + elif getattr(args, "fully_async", False): + check_transfer_queue_version() + + def reset_arg(parser, name, **kwargs): """Reset the default value of a Megatron argument. @@ -177,6 +226,35 @@ def add_serve_arguments(parser): "Mutually exclusive with passing --fully-async and --colocate together." ), ) + parser.add_argument( + "--hybrid-pipeline-forward", + action="store_true", + default=False, + help=( + "In the supported Hybrid actor-only topology, split one optimizer mini " + "into num-iters-per-train-update fixed sample-count actor chunks and " + "overlap actor log-prob forward with later rollout production. " + "Default: disabled." + ), + ) + parser.add_argument( + "--hybrid-pipeline-trace-dir", + type=str, + default=None, + help=( + "Write per-process JSONL events for Hybrid producer puts, actor fetch, " + "restore, forward, advantages, and optimizer phases. No sample content is written." + ), + ) + parser.add_argument( + "--hybrid-pipeline-fetch-timeout-s", + type=float, + default=600.0, + help=( + "Maximum seconds the Hybrid pipeline path waits for one exact " + "TransferQueue chunk before failing with rollout/mini/chunk context." + ), + ) parser.add_argument( "--checkpoint-engine-backend", type=str, @@ -2569,10 +2647,10 @@ def parse_args(add_custom_arguments=None): if not args.debug_train_only: sglang_validate_args(args) - # Only fully-async mode relies on the newer TransferQueue (e.g. - # StreamingTokenBudgetSampler), so gate the version requirement on it. - if getattr(args, "fully_async", False): - check_transfer_queue_version() + # Chunked Hybrid forwarding additionally depends on BatchMeta global + # indexes and the atomic custom-meta async_put contract. Validate that + # API before Ray workers or rollout producers can create side effects. + check_transfer_queue_runtime(args) return args @@ -2762,6 +2840,89 @@ def _normalize_sync_ppo_kl_args(args) -> bool: return True +def _validate_hybrid_pipeline_args(args) -> None: + enabled = bool(getattr(args, "hybrid_pipeline_forward", False)) + trace_enabled = bool(getattr(args, "hybrid_pipeline_trace_dir", None)) + if (enabled or trace_enabled) and not getattr(args, "hybrid", False): + raise ValueError("--hybrid-pipeline-forward and --hybrid-pipeline-trace-dir are only supported with --hybrid.") + if not enabled: + return + + timeout = getattr(args, "hybrid_pipeline_fetch_timeout_s", 600.0) + if not isinstance(timeout, (int, float)) or not math.isfinite(timeout) or timeout <= 0: + raise ValueError("--hybrid-pipeline-fetch-timeout-s must be a finite value greater than 0.") + + conflicts = [] + + def reject(condition: bool, option: str) -> None: + if condition: + conflicts.append(option) + + reject(not getattr(args, "use_dynamic_batch_size", False), "missing --use-dynamic-batch-size") + reject(getattr(args, "multimodal_keys", None) is None, "missing --multimodal-keys") + reject(getattr(args, "advantage_estimator", None) != "grpo", "--advantage-estimator must be grpo") + reject(not getattr(args, "compute_advantages_and_returns", False), "advantages must be computed in actor") + reject(not getattr(args, "enable_weights_backuper", True), "--disable-weights-backuper") + reject(getattr(args, "kl_coef", 0.0) != 0, "--kl-coef") + reject(getattr(args, "use_kl_loss", False), "--use-kl-loss") + reject(getattr(args, "use_opd", False), "--use-opd") + reject(getattr(args, "keep_old_actor", False), "--keep-old-actor") + reject(getattr(args, "true_on_policy_mode", False), "--true-on-policy-mode") + reject(getattr(args, "use_rollout_logprobs", False), "--use-rollout-logprobs") + reject(getattr(args, "get_mismatch_metrics", False), "--get-mismatch-metrics") + reject(getattr(args, "use_routing_replay", False), "--use-routing-replay") + reject(getattr(args, "use_rollout_routing_replay", False), "--use-rollout-routing-replay") + reject(getattr(args, "use_agentic_rollout", False), "--use-agentic-rollout") + reject(getattr(args, "partial_rollout", False), "--partial-rollout") + reject(getattr(args, "use_dynamic_global_batch_size", False), "--use-dynamic-global-batch-size") + reject(getattr(args, "use_critic", False), "--advantage-estimator ppo / critic") + reject(getattr(args, "per_rank_fetch", False), "--per-rank-fetch") + reject(float(getattr(args, "attention_dropout", 0.0) or 0.0) != 0.0, "--attention-dropout") + reject(float(getattr(args, "hidden_dropout", 0.0) or 0.0) != 0.0, "--hidden-dropout") + reject( + int(getattr(args, "pipeline_model_parallel_size", 1) or 1) != 1, + "--pipeline-model-parallel-size", + ) + reject( + int(getattr(args, "tensor_model_parallel_size", 1) or 1) != 2, + "--tensor-model-parallel-size must be 2", + ) + reject( + int(getattr(args, "context_parallel_size", 1) or 1) != 2, + "--context-parallel-size must be 2", + ) + reject( + int(getattr(args, "expert_model_parallel_size", 1) or 1) != 1, + "--expert-model-parallel-size must be 1", + ) + reject( + int(getattr(args, "expert_tensor_parallel_size", 1) or 1) != 1, + "--expert-tensor-parallel-size must be 1", + ) + reject(bool(getattr(args, "offload_train", False)), "--offload-train") + reject(bool(getattr(args, "offload_rollout", False)), "--offload-rollout") + reject( + int(getattr(args, "num_iters_per_train_update", 0) or 0) < 2, + "--num-iters-per-train-update must be >= 2", + ) + rollout_batch_size = int(getattr(args, "rollout_batch_size", 0) or 0) + n_samples_per_prompt = int(getattr(args, "n_samples_per_prompt", 0) or 0) + global_batch_size = int(getattr(args, "global_batch_size", 0) or 0) + reject( + rollout_batch_size * n_samples_per_prompt != global_batch_size, + "rollout_batch_size * n_samples_per_prompt must equal global_batch_size " + "(exactly one optimizer mini per rollout)", + ) + + if conflicts: + raise ValueError( + "--hybrid-pipeline-forward currently supports only actor-only multimodal " + "Hybrid GRPO with dynamic batching, zero dropout, TP2/PP1/CP2/EP1/ETP1, " + "offload disabled, and the normal TensorBackuper. " + f"Conflicting configuration: {', '.join(conflicts)}." + ) + + def slime_validate_args(args): # Backward compatibility: old scripts may pass --enable-gloo-process-groups if not hasattr(args, "use_gloo_process_groups"): @@ -3271,6 +3432,8 @@ def slime_validate_args(args): ) args.global_batch_size = global_batch_size + _validate_hybrid_pipeline_args(args) + if args.n_samples_per_prompt == 1: args.grpo_std_normalization = False logger.info("n_samples_per_prompt is set to 1, grpo_std_normalization will be set to False.") diff --git a/relax/utils/training/hybrid_forward_pipeline.py b/relax/utils/training/hybrid_forward_pipeline.py new file mode 100644 index 000000000..8996d1921 --- /dev/null +++ b/relax/utils/training/hybrid_forward_pipeline.py @@ -0,0 +1,66 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import math +import time +from collections.abc import Callable +from typing import Any + + +def execute_hybrid_forward_mini( + *, + chunks_per_mini: int, + batch_index_for_chunk: Callable[[int], int], + restore_actor: Callable[[int], None], + fetch_chunk: Callable[[int], tuple[Any, list[int]]], + forward_chunk: Callable[[Any, int, list[int]], None], +) -> list[tuple[Any, list[int]]]: + """Restore once, then fetch and forward each fixed actor chunk in order.""" + if chunks_per_mini <= 0: + raise ValueError(f"chunks_per_mini must be positive, got {chunks_per_mini}") + + first_batch_index = batch_index_for_chunk(0) + restore_actor(first_batch_index) + + chunks = [] + for chunk_index in range(chunks_per_mini): + batch_index = batch_index_for_chunk(chunk_index) + batch, global_indexes = fetch_chunk(batch_index) + forward_chunk(batch, batch_index, global_indexes) + chunks.append((batch, global_indexes)) + return chunks + + +def fetch_exact_chunk_with_timeout( + *, + fetch_once: Callable[[], tuple[Any | None, Any]], + expected_samples: int, + timeout_s: float, + error_context: str, + poll_interval_s: float = 0.1, + clock: Callable[[], float] = time.monotonic, + sleep: Callable[[float], None] = time.sleep, +) -> tuple[Any, Any, float]: + """Poll an exact sample-count fetch and fail instead of spinning + forever.""" + if expected_samples <= 0: + raise ValueError(f"expected_samples must be positive, got {expected_samples}") + if not math.isfinite(timeout_s) or timeout_s <= 0: + raise ValueError(f"timeout_s must be positive and finite, got {timeout_s}") + if not math.isfinite(poll_interval_s) or poll_interval_s <= 0: + raise ValueError(f"poll_interval_s must be positive and finite, got {poll_interval_s}") + + start = clock() + while True: + batch, metadata = fetch_once() + if batch is not None: + actual_samples = len(batch.get("total_lengths", [])) + if actual_samples != expected_samples: + raise RuntimeError(f"{error_context}, expected={expected_samples}, last_returned={actual_samples}") + return batch, metadata, clock() - start + + elapsed = clock() - start + if elapsed >= timeout_s: + raise TimeoutError( + f"{error_context}, expected={expected_samples}, last_returned=0, elapsed={elapsed:.3f}s" + ) + sleep(poll_interval_s) diff --git a/relax/utils/training/hybrid_pipeline_trace.py b/relax/utils/training/hybrid_pipeline_trace.py new file mode 100644 index 000000000..62c5bf3a9 --- /dev/null +++ b/relax/utils/training/hybrid_pipeline_trace.py @@ -0,0 +1,191 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +from __future__ import annotations + +import atexit +import hashlib +import json +import os +import socket +import threading +import time +from pathlib import Path +from typing import Any, TextIO + + +_WRITERS: dict[Path, TextIO] = {} +_WRITER_LOCK = threading.Lock() +_SCALAR_TYPES = (str, int, float, bool, type(None)) +_FINGERPRINT_BITS = 128 +_FINGERPRINT_MODULUS = 1 << _FINGERPRINT_BITS +_RECORD_FIELDS = { + "event", + "monotonic_ns", + "rollout_id", + "chunk_index", + "sample_count", + "total_tokens", + "response_tokens", + "multimodal_tensor_bytes", + "role", + "hostname", + "pid", + "global_rank", + "cuda_visible_devices", + "cuda_max_allocated_bytes", + "cuda_max_reserved_bytes", + "global_indexes_fingerprint", +} + + +def _close_writers() -> None: + with _WRITER_LOCK: + for writer in _WRITERS.values(): + writer.close() + _WRITERS.clear() + + +atexit.register(_close_writers) + + +def _global_rank() -> int: + try: + import torch.distributed as dist + + if dist.is_available() and dist.is_initialized(): + return dist.get_rank() + except (ImportError, RuntimeError): + pass + return -1 + + +def _cuda_memory_peaks() -> tuple[int | None, int | None]: + try: + import torch + + if torch.cuda.is_available(): + return ( + int(torch.cuda.max_memory_allocated()), + int(torch.cuda.max_memory_reserved()), + ) + except (ImportError, RuntimeError): + pass + return None, None + + +def _tensor_bytes(value: Any) -> int: + if value is None or isinstance(value, (str, bytes)): + return 0 + if isinstance(value, dict): + return sum(_tensor_bytes(item) for item in value.values()) + if isinstance(value, (list, tuple)): + return sum(_tensor_bytes(item) for item in value) + + nelement = getattr(value, "nelement", None) + element_size = getattr(value, "element_size", None) + if callable(nelement) and callable(element_size): + return int(nelement()) * int(element_size()) + nbytes = getattr(value, "nbytes", None) + if nbytes is not None: + return int(nbytes) + return 0 + + +def _sum_ints(values: Any) -> int | None: + if values is None: + return None + try: + return sum(int(value) for value in values) + except (TypeError, ValueError): + return None + + +def fingerprint_global_indexes(global_indexes: Any) -> str | None: + """Return an order-independent, multiplicity-sensitive digest. + + Digests are added modulo 2**128 so an analyzer can combine chunk digests + without recording sample indexes or depending on producer/fetch grouping. + """ + if global_indexes is None: + return None + normalized = [int(index) for index in global_indexes] + accumulator = 0 + for index in normalized: + digest = hashlib.blake2b(str(index).encode("ascii"), digest_size=16).digest() + accumulator = (accumulator + int.from_bytes(digest, "big")) % _FINGERPRINT_MODULUS + return f"{accumulator:032x}" + + +def _trace_path(trace_dir: str, role: str, hostname: str, pid: int, rank: int) -> Path: + safe_hostname = hostname.replace("/", "_") + safe_role = role.replace("/", "_") + return Path(trace_dir) / f"hybrid-pipeline-{safe_role}-{safe_hostname}-pid{pid}-rank{rank}.jsonl" + + +def emit_hybrid_pipeline_event( + args: Any, + event: str, + *, + rollout_id: int, + role: str, + chunk_index: int | None = None, + sample_count: int | None = None, + batch: dict[str, Any] | None = None, + global_indexes: Any = None, + monotonic_ns: int | None = None, + details: dict[str, Any] | None = None, +) -> dict[str, Any] | None: + """Append one content-free Hybrid pipeline event to this process's + JSONL.""" + trace_dir = getattr(args, "hybrid_pipeline_trace_dir", None) + if not trace_dir: + return None + if details is not None: + invalid = {key: value for key, value in details.items() if not isinstance(value, _SCALAR_TYPES)} + if invalid: + raise TypeError(f"Hybrid pipeline trace details must be scalar, got {invalid}") + reserved = sorted(_RECORD_FIELDS.intersection(details)) + if reserved: + raise ValueError(f"Hybrid pipeline trace details cannot replace record fields: {reserved}") + + hostname = socket.gethostname() + pid = os.getpid() + rank = _global_rank() + cuda_allocated, cuda_reserved = _cuda_memory_peaks() + total_lengths = batch.get("total_lengths") if batch is not None else None + response_lengths = batch.get("response_lengths") if batch is not None else None + if sample_count is None and total_lengths is not None: + sample_count = len(total_lengths) + + record = { + "event": event, + "monotonic_ns": time.monotonic_ns() if monotonic_ns is None else int(monotonic_ns), + "rollout_id": int(rollout_id), + "chunk_index": chunk_index, + "sample_count": sample_count, + "total_tokens": _sum_ints(total_lengths), + "response_tokens": _sum_ints(response_lengths), + "multimodal_tensor_bytes": ( + _tensor_bytes(batch.get("multimodal_train_inputs")) if batch is not None else None + ), + "role": role, + "hostname": hostname, + "pid": pid, + "global_rank": rank, + "cuda_visible_devices": os.environ.get("CUDA_VISIBLE_DEVICES"), + "cuda_max_allocated_bytes": cuda_allocated, + "cuda_max_reserved_bytes": cuda_reserved, + "global_indexes_fingerprint": fingerprint_global_indexes(global_indexes), + } + if details: + record.update(details) + + path = _trace_path(os.fspath(trace_dir), role, hostname, pid, rank) + with _WRITER_LOCK: + writer = _WRITERS.get(path) + if writer is None: + path.parent.mkdir(parents=True, exist_ok=True) + writer = path.open("a", encoding="utf-8", buffering=1) + _WRITERS[path] = writer + writer.write(json.dumps(record, sort_keys=True, separators=(",", ":")) + "\n") + return record diff --git a/relax/utils/utils.py b/relax/utils/utils.py index 92bca9407..9b6fe5102 100644 --- a/relax/utils/utils.py +++ b/relax/utils/utils.py @@ -2,6 +2,7 @@ import os import socket +import time from argparse import Namespace from pathlib import Path from typing import Any, Dict, List, Optional, Union @@ -15,6 +16,7 @@ from relax.utils.env import Envs, validate_env from relax.utils.logging_utils import get_logger from relax.utils.misc import load_function +from relax.utils.training.hybrid_pipeline_trace import emit_hybrid_pipeline_event from relax.utils.types import Sample @@ -508,9 +510,48 @@ async def transfer_batch_to_data_system( # fallback and defeating dynamic batching). total_lengths = rollout_batch.get("total_lengths", None) custom_meta = [{"total_lengths": int(tl)} for tl in total_lengths] if total_lengths is not None else None - await data_system_client.async_put( + trace_enabled = bool(getattr(args, "hybrid_pipeline_trace_dir", None)) + if trace_enabled: + put_start_ns = time.monotonic_ns() + put_event_id = f"{os.getpid()}-{put_start_ns}" + emit_hybrid_pipeline_event( + args, + "tq_put_start", + rollout_id=rollout_id, + role="rollout", + chunk_index=batch_count, + batch=rollout_batch, + monotonic_ns=put_start_ns, + details={ + "batch_count": batch_count, + "is_last": is_last, + "event_id": put_event_id, + }, + ) + batch_meta = await data_system_client.async_put( data=rollout_batch, partition_id=f"train_{rollout_id}", custom_meta=custom_meta, is_last=is_last ) + if trace_enabled: + global_indexes = getattr(batch_meta, "global_indexes", None) + if global_indexes is None: + raise RuntimeError( + "Hybrid pipeline trace requires async_put() to return " + "BatchMeta.global_indexes; install the pinned TransferQueue version." + ) + emit_hybrid_pipeline_event( + args, + "tq_put_done", + rollout_id=rollout_id, + role="rollout", + chunk_index=batch_count, + batch=rollout_batch, + global_indexes=global_indexes, + details={ + "batch_count": batch_count, + "is_last": is_last, + "event_id": put_event_id, + }, + ) logger.info(f"Batch {batch_count} transferred successfully for rollout_id: {rollout_id}") except Exception as e: diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py new file mode 100644 index 000000000..b4d9609ae --- /dev/null +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -0,0 +1,1629 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +"""Validate and summarize Task 21 Hybrid pipeline benchmark artifacts.""" + +from __future__ import annotations + +import argparse +import csv +import json +import math +import statistics +import sys +from collections import defaultdict +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path +from typing import Any, Iterable, Sequence + + +TRACE_REQUIRED_FIELDS = { + "event", + "monotonic_ns", + "rollout_id", + "chunk_index", + "sample_count", + "total_tokens", + "response_tokens", + "multimodal_tensor_bytes", + "role", + "hostname", + "pid", + "global_rank", + "cuda_visible_devices", + "cuda_max_allocated_bytes", + "cuda_max_reserved_bytes", + "global_indexes_fingerprint", +} +NVML_COLUMNS = ( + "timestamp", + "gpu_index", + "gpu_name", + "pstate", + "temperature_c", + "sm_clock_mhz", + "memory_clock_mhz", + "gpu_util_percent", + "memory_util_percent", + "memory_used_mib", + "power_w", +) +DEFAULT_WINDOWS = ((4, 8), (9, 13), (14, 18)) +PERFORMANCE_TAGS = ( + "perf/step_token_per_s", + "perf/step_resp_token_per_s", + "perf/step_time", + "perf/hybrid_phase1_time", +) +COMPARISON_PERFORMANCE_TAGS = PERFORMANCE_TAGS + ("perf/wall_clock_samples_per_s",) +CORRECTNESS_GUARDRAIL_TAGS = ( + "rollout/raw_reward", + "rollout/truncated", + "train/loss", + "train/grad_norm", +) +QUALITY_TAG_FRAGMENTS = ( + "raw_reward", + "reward", + "loss", + "grad_norm", + "response_length", + "truncated", + "staleness", +) +RUN_MANIFEST_REQUIRED_FIELDS = { + "condition", + "order", + "seed", + "rollout_seed", + "num_rollout", + "max_staleness", + "global_batch_size", + "rollout_batch_size", + "n_samples_per_prompt", + "num_iters_per_train_update", + "hybrid_pipeline_forward", + "hybrid_pipeline_trace_dir", + "hybrid_pipeline_fetch_timeout_s", + "git_commit", + "git_branch", + "git_status_porcelain", + "image_archive_sha256", + "image_manifest_digest", + "image_id", + "transferqueue_commit", + "python", + "entrypoint", +} +COMPARISON_FIXED_MANIFEST_FIELDS = ( + "num_rollout", + "max_staleness", + "global_batch_size", + "rollout_batch_size", + "n_samples_per_prompt", + "num_iters_per_train_update", + "hybrid_pipeline_fetch_timeout_s", + "git_commit", + "git_branch", + "image_archive_sha256", + "image_manifest_digest", + "image_id", + "transferqueue_commit", + "python", + "entrypoint", +) +REPRODUCIBILITY_ARTIFACTS = ( + "pip-freeze.txt", + "inputs.sha256", + "transferqueue-wheel.sha256", + "logs/launcher.log", +) + + +class BenchmarkValidationError(RuntimeError): + """Raised when benchmark artifacts violate a registered invariant.""" + + +@dataclass(frozen=True) +class RunAnalysis: + run_dir: Path + manifest: dict[str, Any] + trace_rows: list[dict[str, Any]] + actor_rank_rows: list[dict[str, Any]] + scalar_rows: list[dict[str, Any]] + nvml_rows: list[dict[str, Any]] + summary: dict[str, Any] + + +def _fail(message: str) -> None: + raise BenchmarkValidationError(message) + + +def _validate_finite(value: Any, context: str) -> None: + if isinstance(value, bool) or value is None or isinstance(value, str): + return + if isinstance(value, (int, float)): + if not math.isfinite(value): + _fail(f"{context} contains non-finite numeric value {value!r}") + return + if isinstance(value, dict): + for key, item in value.items(): + _validate_finite(item, f"{context}.{key}") + return + if isinstance(value, (list, tuple)): + for index, item in enumerate(value): + _validate_finite(item, f"{context}[{index}]") + + +def _load_manifest(run_dir: Path) -> dict[str, Any]: + path = run_dir / "run_manifest.json" + if not path.is_file(): + _fail(f"missing run manifest: {path}") + try: + manifest = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + _fail(f"cannot parse {path}: {exc}") + if not isinstance(manifest, dict): + _fail(f"{path} must contain a JSON object") + missing = sorted(RUN_MANIFEST_REQUIRED_FIELDS - set(manifest)) + if missing: + _fail(f"{path} is missing required fields {missing}") + _validate_finite(manifest, str(path)) + return manifest + + +def _validate_hex_digest(value: Any, *, length: int, context: str, prefix: str = "") -> None: + if not isinstance(value, str) or not value.startswith(prefix): + _fail(f"{context} must be a {prefix!r}-prefixed hexadecimal string") + payload = value[len(prefix) :] + if len(payload) != length: + _fail(f"{context} must contain {length} hexadecimal characters, got {len(payload)}") + try: + int(payload, 16) + except ValueError: + _fail(f"{context} contains non-hexadecimal characters") + + +def _validate_run_manifest( + manifest: dict[str, Any], + *, + run_dir: Path, + windows: Sequence[tuple[int, int]], +) -> None: + for key in ( + "seed", + "rollout_seed", + "num_rollout", + "max_staleness", + "global_batch_size", + "rollout_batch_size", + "n_samples_per_prompt", + "num_iters_per_train_update", + "hybrid_pipeline_forward", + ): + if type(manifest[key]) is not int: + _fail(f"{run_dir} manifest field {key!r} must be an integer, got {manifest[key]!r}") + if manifest["seed"] != manifest["rollout_seed"]: + _fail( + f"{run_dir} must use the same paired Megatron/rollout seed, got " + f"{manifest['seed']} and {manifest['rollout_seed']}" + ) + if manifest["num_rollout"] <= max(end for _, end in windows): + _fail( + f"{run_dir} num_rollout={manifest['num_rollout']} does not cover " + f"steady window ending at step {max(end for _, end in windows)}" + ) + for key in ( + "global_batch_size", + "rollout_batch_size", + "n_samples_per_prompt", + "num_iters_per_train_update", + ): + if manifest[key] <= 0: + _fail(f"{run_dir} manifest field {key!r} must be positive, got {manifest[key]!r}") + if manifest["max_staleness"] < 0: + _fail(f"{run_dir} max_staleness must be non-negative, got {manifest['max_staleness']!r}") + produced_samples = manifest["rollout_batch_size"] * manifest["n_samples_per_prompt"] + if produced_samples != manifest["global_batch_size"]: + _fail( + f"{run_dir} must benchmark exactly one optimizer mini per rollout: " + f"rollout_batch_size * n_samples_per_prompt={produced_samples}, " + f"global_batch_size={manifest['global_batch_size']}" + ) + if manifest["git_status_porcelain"] != "": + _fail(f"{run_dir} was captured from a dirty working tree: {manifest['git_status_porcelain']!r}") + + timeout = manifest["hybrid_pipeline_fetch_timeout_s"] + if ( + isinstance(timeout, bool) + or not isinstance(timeout, (int, float)) + or not math.isfinite(timeout) + or timeout <= 0 + ): + _fail(f"{run_dir} has invalid hybrid_pipeline_fetch_timeout_s={timeout!r}") + + trace_dir = Path(str(manifest["hybrid_pipeline_trace_dir"])).resolve() + expected_trace_dir = (run_dir / "timeline").resolve() + if trace_dir != expected_trace_dir: + _fail(f"{run_dir} manifest trace directory is {trace_dir}, expected {expected_trace_dir}") + + for key in ("condition", "order", "git_branch", "python", "entrypoint"): + if not isinstance(manifest[key], str) or not manifest[key].strip(): + _fail(f"{run_dir} manifest field {key!r} must be a non-empty string") + + _validate_hex_digest(manifest["git_commit"], length=40, context=f"{run_dir} git_commit") + _validate_hex_digest( + manifest["image_archive_sha256"], + length=64, + context=f"{run_dir} image_archive_sha256", + ) + _validate_hex_digest( + manifest["image_manifest_digest"], + length=64, + prefix="sha256:", + context=f"{run_dir} image_manifest_digest", + ) + _validate_hex_digest( + manifest["image_id"], + length=64, + prefix="sha256:", + context=f"{run_dir} image_id", + ) + _validate_hex_digest( + manifest["transferqueue_commit"], + length=40, + context=f"{run_dir} transferqueue_commit", + ) + + +def _require_reproducibility_artifacts(run_dir: Path) -> None: + missing = [] + for relative_path in REPRODUCIBILITY_ARTIFACTS: + path = run_dir / relative_path + if not path.is_file() or path.stat().st_size == 0: + missing.append(relative_path) + if missing: + _fail(f"{run_dir} is missing non-empty reproducibility artifacts {missing}") + + +def _load_trace_rows(run_dir: Path) -> list[dict[str, Any]]: + trace_dir = run_dir / "timeline" + paths = sorted(trace_dir.glob("*.jsonl")) + if not paths: + _fail(f"no Hybrid pipeline JSONL files found under {trace_dir}") + + rows: list[dict[str, Any]] = [] + for path in paths: + previous_ns = -1 + with path.open(encoding="utf-8") as reader: + for line_number, line in enumerate(reader, start=1): + if not line.strip(): + continue + try: + row = json.loads(line) + except json.JSONDecodeError as exc: + _fail(f"{path}:{line_number} is invalid JSON: {exc}") + if not isinstance(row, dict): + _fail(f"{path}:{line_number} must contain a JSON object") + missing = sorted(TRACE_REQUIRED_FIELDS - set(row)) + if missing: + _fail(f"{path}:{line_number} is missing trace fields {missing}") + _validate_finite(row, f"{path}:{line_number}") + monotonic_ns = row["monotonic_ns"] + if type(monotonic_ns) is not int or monotonic_ns < 0: + _fail(f"{path}:{line_number} has invalid monotonic_ns={monotonic_ns!r}") + if monotonic_ns < previous_ns: + _fail(f"{path}:{line_number} is not monotonic: previous={previous_ns}, current={monotonic_ns}") + previous_ns = monotonic_ns + row["_source_file"] = path.name + row["_source_line"] = line_number + rows.append(row) + + if not rows: + _fail(f"trace files under {trace_dir} contain no events") + hostnames = {row["hostname"] for row in rows} + if len(hostnames) != 1: + _fail(f"trace events span multiple hostnames and cannot share one monotonic clock: {sorted(hostnames)}") + return sorted(rows, key=lambda row: (row["monotonic_ns"], row["_source_file"], row["_source_line"])) + + +def _pair_events( + rows: Sequence[dict[str, Any]], + start_event: str, + end_event: str, + *, + key: str, + context: str, +) -> list[tuple[dict[str, Any], dict[str, Any]]]: + starts: dict[Any, list[dict[str, Any]]] = defaultdict(list) + ends: dict[Any, list[dict[str, Any]]] = defaultdict(list) + for row in rows: + if row["event"] == start_event: + starts[row.get(key)].append(row) + elif row["event"] == end_event: + ends[row.get(key)].append(row) + if set(starts) != set(ends): + _fail( + f"{context} has unmatched {start_event}/{end_event} keys: " + f"starts={sorted(starts, key=str)}, ends={sorted(ends, key=str)}" + ) + + pairs = [] + for event_key in sorted(starts, key=str): + if len(starts[event_key]) != 1 or len(ends[event_key]) != 1: + _fail( + f"{context} requires one {start_event}/{end_event} pair for {key}={event_key!r}, " + f"got starts={len(starts[event_key])}, ends={len(ends[event_key])}" + ) + start, end = starts[event_key][0], ends[event_key][0] + if start["monotonic_ns"] > end["monotonic_ns"]: + _fail( + f"{context} has {start_event} after {end_event} for {key}={event_key!r}: " + f"{start['monotonic_ns']} > {end['monotonic_ns']}" + ) + pairs.append((start, end)) + return pairs + + +def _require_count(rows: Sequence[dict[str, Any]], event: str, count: int, context: str) -> list[dict[str, Any]]: + matches = [row for row in rows if row["event"] == event] + if len(matches) != count: + _fail(f"{context} expected {count} {event!r} events, got {len(matches)}") + return matches + + +def _stream_key(row: dict[str, Any]) -> tuple[str, int, int]: + return row["hostname"], int(row["pid"]), int(row["global_rank"]) + + +def _combine_global_index_fingerprints( + rows: Sequence[dict[str, Any]], + *, + context: str, +) -> str: + """Combine additive 128-bit chunk digests without exposing sample + indexes.""" + accumulator = 0 + modulus = 1 << 128 + for row in rows: + fingerprint = row.get("global_indexes_fingerprint") + if not isinstance(fingerprint, str) or len(fingerprint) != 32: + _fail(f"{context} has invalid global index fingerprint {fingerprint!r}") + try: + value = int(fingerprint, 16) + except ValueError: + _fail(f"{context} has non-hex global index fingerprint {fingerprint!r}") + accumulator = (accumulator + value) % modulus + return f"{accumulator:032x}" + + +def _analyze_trace( + rows: Sequence[dict[str, Any]], + *, + pipeline_enabled: bool, + expected_samples: int, + expected_actor_chunks: int, +) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]: + producer_by_rollout: dict[int, list[dict[str, Any]]] = defaultdict(list) + actor_by_rollout_stream: dict[tuple[int, tuple[str, int, int]], list[dict[str, Any]]] = defaultdict(list) + for row in rows: + rollout_id = int(row["rollout_id"]) + if row["role"] == "rollout": + producer_by_rollout[rollout_id].append(row) + elif row["role"] == "actor": + actor_by_rollout_stream[(rollout_id, _stream_key(row))].append(row) + + producer_rollouts = set(producer_by_rollout) + actor_rollouts = {rollout_id for rollout_id, _ in actor_by_rollout_stream} + if not producer_rollouts or producer_rollouts != actor_rollouts: + _fail( + "producer and actor rollout IDs differ: " + f"producer={sorted(producer_rollouts)}, actor={sorted(actor_rollouts)}" + ) + + expected_stream_chunks = expected_actor_chunks if pipeline_enabled else 1 + rollout_rows: list[dict[str, Any]] = [] + actor_rank_rows: list[dict[str, Any]] = [] + + for rollout_id in sorted(producer_rollouts): + producer_rows = producer_by_rollout[rollout_id] + put_pairs = _pair_events( + producer_rows, + "tq_put_start", + "tq_put_done", + key="event_id", + context=f"rollout_id={rollout_id} producer", + ) + if not put_pairs: + _fail(f"rollout_id={rollout_id} producer has no completed puts") + put_start = [start for start, _ in put_pairs] + put_done = [end for _, end in put_pairs] + if any(row["sample_count"] is None for row in put_done): + _fail(f"rollout_id={rollout_id} producer put is missing sample_count") + producer_samples = sum(int(row["sample_count"]) for row in put_done) + if producer_samples != expected_samples: + _fail( + f"rollout_id={rollout_id} producer sample conservation failed: " + f"expected={expected_samples}, actual={producer_samples}" + ) + producer_fingerprint = _combine_global_index_fingerprints( + put_done, + context=f"rollout_id={rollout_id} producer", + ) + + streams = { + stream: stream_rows + for (stream_rollout_id, stream), stream_rows in actor_by_rollout_stream.items() + if stream_rollout_id == rollout_id + } + if not streams: + _fail(f"rollout_id={rollout_id} has no actor trace stream") + + for stream, stream_rows in streams.items(): + context = f"rollout_id={rollout_id} actor_stream={stream}" + fetch_pairs = _pair_events( + stream_rows, + "chunk_fetch_start", + "chunk_fetch_end", + key="chunk_index", + context=context, + ) + forward_pairs = _pair_events( + stream_rows, + "actor_forward_start", + "actor_forward_end", + key="chunk_index", + context=context, + ) + restore_pairs = _pair_events( + stream_rows, + "actor_restore_start", + "actor_restore_end", + key="chunk_index", + context=context, + ) + if len(fetch_pairs) != expected_stream_chunks or len(forward_pairs) != expected_stream_chunks: + _fail( + f"{context} expected {expected_stream_chunks} fetch/forward chunks, " + f"got fetch={len(fetch_pairs)}, forward={len(forward_pairs)}" + ) + if len(restore_pairs) != 1: + _fail(f"{context} expected exactly one actor restore, got {len(restore_pairs)}") + _pair_events( + stream_rows, + "advantages_start", + "advantages_end", + key="chunk_index", + context=context, + ) + _pair_events( + stream_rows, + "optimizer_start", + "optimizer_end", + key="chunk_index", + context=context, + ) + _require_count(stream_rows, "advantages_start", 1, context) + _require_count(stream_rows, "optimizer_start", 1, context) + + fetch_end = [end for _, end in fetch_pairs] + forward_start = [start for start, _ in forward_pairs] + forward_end = [end for _, end in forward_pairs] + for event_rows, event_name in ((fetch_end, "fetch"), (forward_start, "forward")): + if any(row["sample_count"] is None for row in event_rows): + _fail(f"{context} {event_name} event is missing sample_count") + actual_samples = sum(int(row["sample_count"]) for row in event_rows) + if actual_samples != expected_samples: + _fail( + f"{context} {event_name} sample conservation failed: " + f"expected={expected_samples}, actual={actual_samples}" + ) + if any(row["global_indexes_fingerprint"] is None for row in fetch_end): + _fail(f"{context} fetch event is missing global index fingerprint") + fetch_fingerprint = _combine_global_index_fingerprints( + fetch_end, + context=f"{context} fetch", + ) + forward_fingerprint = _combine_global_index_fingerprints( + forward_start, + context=f"{context} forward", + ) + if fetch_fingerprint != producer_fingerprint: + _fail( + f"{context} producer/fetch global index fingerprints differ: " + f"producer={producer_fingerprint}, actor={fetch_fingerprint}" + ) + if forward_fingerprint != fetch_fingerprint: + _fail( + f"{context} fetch/forward global index fingerprints differ: " + f"fetch={fetch_fingerprint}, forward={forward_fingerprint}" + ) + + fetch_by_chunk = {end["chunk_index"]: (start, end) for start, end in fetch_pairs} + for forward_start_row, forward_end_row in forward_pairs: + chunk_index = forward_start_row["chunk_index"] + if chunk_index not in fetch_by_chunk: + _fail(f"{context} forward chunk {chunk_index!r} has no matching fetch") + _, fetch_end_row = fetch_by_chunk[chunk_index] + if fetch_end_row["monotonic_ns"] > forward_start_row["monotonic_ns"]: + _fail(f"{context} forward chunk {chunk_index!r} starts before fetch completes") + if forward_start_row["monotonic_ns"] > forward_end_row["monotonic_ns"]: + _fail(f"{context} forward chunk {chunk_index!r} ends before it starts") + + first_phase_ns = min( + min(start["monotonic_ns"] for start, _ in fetch_pairs), + restore_pairs[0][0]["monotonic_ns"], + ) + last_forward_ns = max(row["monotonic_ns"] for row in forward_end) + first_forward_ns = min(row["monotonic_ns"] for row in forward_start) + last_put_start_ns = max(row["monotonic_ns"] for row in put_start) + last_put_done_ns = max(row["monotonic_ns"] for row in put_done) + ready_rollout_ids = [ + int(row["rollout_id"]) + for row in rows + if row["role"] == "rollout" + and row["event"] == "tq_put_done" + and row["monotonic_ns"] <= first_forward_ns + ] + # A completed actor fetch proves the current rollout was ready even + # if the producer's post-async_put trace write loses the scheduling + # race with the consumer process. + ready_rollout_ids.append(rollout_id) + producer_lead = max(ready_rollout_ids) - rollout_id + actor_rank_rows.append( + { + "rollout_id": rollout_id, + "hostname": stream[0], + "pid": stream[1], + "global_rank": stream[2], + "pipeline_enabled": pipeline_enabled, + "producer_samples": producer_samples, + "actor_fetch_samples": sum(int(row["sample_count"]) for row in fetch_end), + "actor_forward_samples": sum(int(row["sample_count"]) for row in forward_start), + "producer_put_count": len(put_pairs), + "actor_fetch_count": len(fetch_pairs), + "actor_forward_count": len(forward_pairs), + "actor_restore_count": len(restore_pairs), + "phase1_s": (last_forward_ns - first_phase_ns) / 1e9, + # Strict evidence of producer/actor overlap: the actor + # starts forwarding before the producer even begins its + # final put. A delayed post-put trace write alone cannot + # make this condition true. + "first_forward_before_last_put_start": first_forward_ns < last_put_start_ns, + # Diagnostic only: async_put may have made data visible + # before the producer coroutine records tq_put_done. + "first_forward_before_last_put_done": first_forward_ns < last_put_done_ns, + "producer_overlap_s": max(0, last_put_start_ns - first_forward_ns) / 1e9, + "transfer_overlap_s": max(0, last_put_done_ns - first_forward_ns) / 1e9, + "producer_lead_at_first_forward": producer_lead, + "actor_multimodal_tensor_bytes": sum( + int(row["multimodal_tensor_bytes"] or 0) for row in fetch_end + ), + "actor_total_tokens": sum(int(row["total_tokens"] or 0) for row in fetch_end), + "actor_response_tokens": sum(int(row["response_tokens"] or 0) for row in fetch_end), + "cuda_max_allocated_bytes": max( + (int(row["cuda_max_allocated_bytes"] or 0) for row in stream_rows), + default=0, + ), + "cuda_max_reserved_bytes": max( + (int(row["cuda_max_reserved_bytes"] or 0) for row in stream_rows), + default=0, + ), + "fetch_global_indexes_fingerprint": fetch_fingerprint, + "forward_global_indexes_fingerprint": forward_fingerprint, + } + ) + + primary = min( + (row for row in actor_rank_rows if row["rollout_id"] == rollout_id), + key=lambda row: ( + row["global_rank"] < 0, + row["global_rank"] if row["global_rank"] >= 0 else row["pid"], + ), + ) + put_start_times = [row["monotonic_ns"] for row in put_start] + put_done_times = [row["monotonic_ns"] for row in put_done] + rollout_rows.append( + { + **primary, + "producer_global_indexes_fingerprint": producer_fingerprint, + "producer_first_put_start_ns": min(put_start_times), + "producer_last_put_start_ns": max(put_start_times), + "producer_first_put_done_ns": min(put_done_times), + "producer_last_put_done_ns": max(put_done_times), + "producer_ready_window_s": (max(put_done_times) - min(put_done_times)) / 1e9, + "producer_multimodal_tensor_bytes": sum(int(row["multimodal_tensor_bytes"] or 0) for row in put_done), + "producer_total_tokens": sum(int(row["total_tokens"] or 0) for row in put_done), + "producer_response_tokens": sum(int(row["response_tokens"] or 0) for row in put_done), + } + ) + + producer_overlap_count = sum(bool(row["first_forward_before_last_put_start"]) for row in rollout_rows) + transfer_overlap_count = sum(bool(row["first_forward_before_last_put_done"]) for row in rollout_rows) + summary = { + "hostname": next(iter({row["hostname"] for row in rows})), + "pipeline_enabled": pipeline_enabled, + "rollout_count": len(rollout_rows), + "actor_stream_count": len({_stream_key(row) for row in rows if row["role"] == "actor"}), + "producer_overlap_rollout_count": producer_overlap_count, + "producer_overlap_rollout_ratio": producer_overlap_count / len(rollout_rows), + "transfer_overlap_rollout_count": transfer_overlap_count, + "transfer_overlap_rollout_ratio": transfer_overlap_count / len(rollout_rows), + "mean_phase1_s": statistics.fmean(row["phase1_s"] for row in rollout_rows), + "mean_producer_overlap_s": statistics.fmean(row["producer_overlap_s"] for row in rollout_rows), + "mean_transfer_overlap_s": statistics.fmean(row["transfer_overlap_s"] for row in rollout_rows), + "mean_producer_lead_at_first_forward": statistics.fmean( + row["producer_lead_at_first_forward"] for row in rollout_rows + ), + "max_producer_lead_at_first_forward": max(row["producer_lead_at_first_forward"] for row in rollout_rows), + "mean_producer_ready_window_s": statistics.fmean(row["producer_ready_window_s"] for row in rollout_rows), + "max_cuda_allocated_bytes": max(row["cuda_max_allocated_bytes"] for row in actor_rank_rows), + "max_cuda_reserved_bytes": max(row["cuda_max_reserved_bytes"] for row in actor_rank_rows), + } + return rollout_rows, actor_rank_rows, summary + + +def _load_tensorboard_scalars(run_dir: Path) -> list[dict[str, Any]]: + event_paths = sorted(path for path in run_dir.rglob("events.out.tfevents.*") if path.is_file()) + if not event_paths: + return [] + try: + from tensorboard.backend.event_processing.event_accumulator import EventAccumulator + except ImportError: + return [] + + by_key: dict[tuple[str, int], dict[str, Any]] = {} + for path in event_paths: + accumulator = EventAccumulator(str(path), size_guidance={"scalars": 0}) + try: + accumulator.Reload() + except Exception as exc: + _fail(f"cannot load TensorBoard event file {path}: {exc}") + for tag in accumulator.Tags().get("scalars", []): + for event in accumulator.Scalars(tag): + _validate_finite(event.value, f"{path}:{tag}:step={event.step}") + row = { + "tag": tag, + "step": int(event.step), + "value": float(event.value), + "wall_time": float(event.wall_time), + "source_file": str(path.relative_to(run_dir)), + } + key = tag, int(event.step) + if key not in by_key or row["wall_time"] >= by_key[key]["wall_time"]: + by_key[key] = row + return sorted(by_key.values(), key=lambda row: (row["tag"], row["step"])) + + +def _parse_nvml_rows(run_dir: Path) -> list[dict[str, Any]]: + path = run_dir / "telemetry" / "nvidia-smi.csv" + if not path.is_file(): + return [] + + rows = [] + with path.open(encoding="utf-8", newline="") as reader: + for line_number, values in enumerate(csv.reader(reader), start=1): + if not values: + continue + if len(values) != len(NVML_COLUMNS): + _fail(f"{path}:{line_number} expected {len(NVML_COLUMNS)} columns, got {len(values)}") + row = {key: value.strip() for key, value in zip(NVML_COLUMNS, values, strict=True)} + for key in ( + "gpu_index", + "temperature_c", + "sm_clock_mhz", + "memory_clock_mhz", + "gpu_util_percent", + "memory_util_percent", + "memory_used_mib", + "power_w", + ): + try: + row[key] = float(row[key]) + except ValueError as exc: + _fail(f"{path}:{line_number} has non-numeric {key}={row[key]!r}: {exc}") + try: + row["wall_time"] = datetime.strptime( + row["timestamp"], + "%Y/%m/%d %H:%M:%S.%f", + ).timestamp() + except ValueError: + row["wall_time"] = None + _validate_finite(row, f"{path}:{line_number}") + rows.append(row) + return rows + + +def _parse_windows(value: str) -> tuple[tuple[int, int], ...]: + windows = [] + for item in value.split(","): + try: + start_text, end_text = item.strip().split("-", maxsplit=1) + start, end = int(start_text), int(end_text) + except ValueError: + _fail(f"invalid steady window {item!r}; expected START-END") + if start < 0 or end < start: + _fail(f"invalid steady window {item!r}; require 0 <= START <= END") + windows.append((start, end)) + if not windows: + _fail("at least one steady window is required") + flattened = [step for start, end in windows for step in range(start, end + 1)] + if len(flattened) != len(set(flattened)): + _fail(f"steady windows overlap: {windows}") + return tuple(windows) + + +def _stable_steps(windows: Sequence[tuple[int, int]]) -> set[int]: + return {step for start, end in windows for step in range(start, end + 1)} + + +def _scalar_map(rows: Sequence[dict[str, Any]], tag: str) -> dict[int, float]: + return {int(row["step"]): float(row["value"]) for row in rows if row["tag"] == tag} + + +def _aggregate_throughput( + scalar_rows: Sequence[dict[str, Any]], + throughput_tag: str, + windows: Sequence[tuple[int, int]], +) -> float | None: + step_time = _scalar_map(scalar_rows, "perf/step_time") + throughput = _scalar_map(scalar_rows, throughput_tag) + steps = sorted(_stable_steps(windows).intersection(step_time, throughput)) + if not steps: + return None + total_time = sum(step_time[step] for step in steps) + if total_time <= 0: + _fail(f"{throughput_tag} has non-positive aggregate steady step time") + return sum(throughput[step] * step_time[step] for step in steps) / total_time + + +def _aggregate_samples_per_second( + scalar_rows: Sequence[dict[str, Any]], + windows: Sequence[tuple[int, int]], + samples_per_step: int, +) -> float | None: + step_time = _scalar_map(scalar_rows, "perf/step_time") + steps = sorted(_stable_steps(windows).intersection(step_time)) + if not steps: + return None + total_time = sum(step_time[step] for step in steps) + if total_time <= 0: + _fail("perf/step_time has non-positive aggregate steady time") + return samples_per_step * len(steps) / total_time + + +def _metric_summary( + scalar_rows: Sequence[dict[str, Any]], + windows: Sequence[tuple[int, int]], +) -> dict[str, Any]: + steps = _stable_steps(windows) + tags = sorted({row["tag"] for row in scalar_rows}) + summary: dict[str, Any] = {} + for tag in tags: + values = [float(row["value"]) for row in scalar_rows if row["tag"] == tag and row["step"] in steps] + if not values: + continue + ordered = sorted(values) + summary[tag] = { + "count": len(values), + "mean": statistics.fmean(values), + "min": ordered[0], + "max": ordered[-1], + "p50": statistics.median(ordered), + "p95": ordered[math.ceil(0.95 * len(ordered)) - 1], + } + for tag in ("perf/step_token_per_s", "perf/step_resp_token_per_s"): + aggregate = _aggregate_throughput(scalar_rows, tag, windows) + if aggregate is not None: + summary.setdefault(tag, {})["aggregate"] = aggregate + return summary + + +def _steady_wall_time_intervals( + scalar_rows: Sequence[dict[str, Any]], + windows: Sequence[tuple[int, int]], +) -> list[tuple[float, float]]: + """Map registered steady steps to wall-clock intervals using the + TensorBoard step end time and perf/step_time duration.""" + steady_steps = _stable_steps(windows) + intervals = [] + for row in scalar_rows: + if row["tag"] != "perf/step_time" or int(row["step"]) not in steady_steps: + continue + duration_s = float(row["value"]) + wall_time = row.get("wall_time") + if duration_s <= 0: + _fail(f"perf/step_time step={row['step']} must be positive, got {duration_s}") + if not isinstance(wall_time, (int, float)) or not math.isfinite(wall_time): + _fail(f"perf/step_time step={row['step']} is missing a finite TensorBoard wall_time") + intervals.append((float(wall_time) - duration_s, float(wall_time))) + return sorted(intervals) + + +def _nvml_summary( + rows: Sequence[dict[str, Any]], + *, + steady_intervals: Sequence[tuple[float, float]], +) -> dict[str, Any]: + if not rows: + return {} + + def is_steady(row: dict[str, Any]) -> bool: + wall_time = row.get("wall_time") + return isinstance(wall_time, (int, float)) and any( + start <= float(wall_time) <= end for start, end in steady_intervals + ) + + by_gpu: dict[int, list[dict[str, Any]]] = defaultdict(list) + for row in rows: + by_gpu[int(row["gpu_index"])].append(row) + per_gpu = {} + for gpu_index, gpu_rows in sorted(by_gpu.items()): + full_utilization = [float(row["gpu_util_percent"]) for row in gpu_rows] + steady_rows = [row for row in gpu_rows if is_steady(row)] + steady_utilization = [float(row["gpu_util_percent"]) for row in steady_rows] + per_gpu[str(gpu_index)] = { + "full_run_sample_count": len(gpu_rows), + "full_run_mean_gpu_util_percent": statistics.fmean(full_utilization), + "steady_sample_count": len(steady_rows), + "steady_mean_gpu_util_percent": (statistics.fmean(steady_utilization) if steady_utilization else None), + "steady_idle_ratio_below_10_percent": ( + sum(value < 10 for value in steady_utilization) / len(steady_utilization) + if steady_utilization + else None + ), + "peak_memory_used_mib": max(float(row["memory_used_mib"]) for row in gpu_rows), + "mean_power_w": statistics.fmean(float(row["power_w"]) for row in gpu_rows), + } + steady_gpu_utilization = [ + item["steady_mean_gpu_util_percent"] + for item in per_gpu.values() + if item["steady_mean_gpu_util_percent"] is not None + ] + steady_idle_ratios = [ + item["steady_idle_ratio_below_10_percent"] + for item in per_gpu.values() + if item["steady_idle_ratio_below_10_percent"] is not None + ] + return { + "gpu_count": len(per_gpu), + "per_gpu": per_gpu, + "peak_memory_used_mib": max(item["peak_memory_used_mib"] for item in per_gpu.values()), + "full_run_mean_gpu_util_percent": statistics.fmean( + item["full_run_mean_gpu_util_percent"] for item in per_gpu.values() + ), + "steady_mean_gpu_util_percent": (statistics.fmean(steady_gpu_utilization) if steady_gpu_utilization else None), + "steady_idle_ratio_below_10_percent": (statistics.fmean(steady_idle_ratios) if steady_idle_ratios else None), + "steady_wall_time_intervals": [list(interval) for interval in steady_intervals], + } + + +def _write_csv(path: Path, rows: Sequence[dict[str, Any]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + if not rows: + path.write_text("", encoding="utf-8") + return + fieldnames = sorted({key for row in rows for key in row}) + with path.open("w", encoding="utf-8", newline="") as writer_file: + writer = csv.DictWriter(writer_file, fieldnames=fieldnames) + writer.writeheader() + for row in rows: + writer.writerow( + { + key: ( + json.dumps(value, sort_keys=True, separators=(",", ":")) + if isinstance(value, (dict, list, tuple)) + else value + ) + for key, value in row.items() + } + ) + + +def analyze_run( + run_dir: Path, + *, + windows: Sequence[tuple[int, int]] = DEFAULT_WINDOWS, + expected_samples: int = 256, + expected_actor_chunks: int = 2, + write_outputs: bool = True, + require_reproducibility_artifacts: bool = False, +) -> RunAnalysis: + run_dir = run_dir.resolve() + manifest = _load_manifest(run_dir) + _validate_run_manifest(manifest, run_dir=run_dir, windows=windows) + if require_reproducibility_artifacts: + _require_reproducibility_artifacts(run_dir) + trace_rows = _load_trace_rows(run_dir) + pipeline_flag = manifest["hybrid_pipeline_forward"] + if type(pipeline_flag) is not int or pipeline_flag not in (0, 1): + _fail(f"{run_dir} hybrid_pipeline_forward must be integer 0 or 1, got {pipeline_flag!r}") + pipeline_enabled = bool(pipeline_flag) + if manifest["global_batch_size"] != expected_samples: + _fail( + f"{run_dir} expected_samples={expected_samples} disagrees with " + f"manifest global_batch_size={manifest['global_batch_size']}" + ) + if manifest["num_iters_per_train_update"] != expected_actor_chunks: + _fail( + f"{run_dir} expected_actor_chunks={expected_actor_chunks} disagrees with " + f"manifest num_iters_per_train_update={manifest['num_iters_per_train_update']}" + ) + condition = str(manifest["condition"]) + if condition not in {"baseline", "experiment"}: + _fail(f"{run_dir} has unsupported condition {condition!r}; expected 'baseline' or 'experiment'") + if condition == "baseline" and pipeline_enabled: + _fail(f"{run_dir} is labeled baseline but hybrid_pipeline_forward is enabled") + if condition == "experiment" and not pipeline_enabled: + _fail(f"{run_dir} is labeled experiment but hybrid_pipeline_forward is disabled") + rollout_rows, actor_rank_rows, trace_summary = _analyze_trace( + trace_rows, + pipeline_enabled=pipeline_enabled, + expected_samples=expected_samples, + expected_actor_chunks=expected_actor_chunks, + ) + scalar_rows = _load_tensorboard_scalars(run_dir) + nvml_rows = _parse_nvml_rows(run_dir) + steady_wall_time_intervals = _steady_wall_time_intervals(scalar_rows, windows) + metrics = _metric_summary(scalar_rows, windows) + samples_per_second = _aggregate_samples_per_second(scalar_rows, windows, expected_samples) + if samples_per_second is not None: + metrics["perf/wall_clock_samples_per_s"] = {"aggregate": samples_per_second} + summary = { + "run_dir": str(run_dir), + "condition": manifest["condition"], + "seed": manifest["seed"], + "hybrid_pipeline_forward": pipeline_enabled, + "steady_windows": [list(window) for window in windows], + "trace": trace_summary, + "metrics": metrics, + "nvml": _nvml_summary(nvml_rows, steady_intervals=steady_wall_time_intervals), + "validation": "passed", + } + + if write_outputs: + output_dir = run_dir / "analysis" + _write_csv(output_dir / "trace_events.csv", trace_rows) + _write_csv(output_dir / "rollout_summary.csv", rollout_rows) + _write_csv(output_dir / "actor_rank_summary.csv", actor_rank_rows) + _write_csv(output_dir / "tensorboard_scalars.csv", scalar_rows) + _write_csv(output_dir / "nvml_samples.csv", nvml_rows) + (output_dir / "summary.json").write_text( + json.dumps(summary, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + return RunAnalysis( + run_dir=run_dir, + manifest=manifest, + trace_rows=rollout_rows, + actor_rank_rows=actor_rank_rows, + scalar_rows=scalar_rows, + nvml_rows=nvml_rows, + summary=summary, + ) + + +def _geometric_mean(values: Iterable[float]) -> float: + values = list(values) + if not values or any(value <= 0 for value in values): + _fail(f"geometric mean requires positive values, got {values}") + return math.exp(statistics.fmean(math.log(value) for value in values)) + + +def _build_comparison( + analyses: Sequence[RunAnalysis], + *, + windows: Sequence[tuple[int, int]], + enforce_targets: bool, + expected_gpu_count: int = 8, +) -> dict[str, Any]: + if expected_gpu_count <= 0: + _fail(f"expected_gpu_count must be positive, got {expected_gpu_count}") + + by_condition: dict[str, list[RunAnalysis]] = defaultdict(list) + for analysis in analyses: + by_condition[str(analysis.manifest["condition"])].append(analysis) + if set(by_condition) != {"baseline", "experiment"}: + _fail(f"comparison requires conditions named 'baseline' and 'experiment', got {sorted(by_condition)}") + + for field in COMPARISON_FIXED_MANIFEST_FIELDS: + values = {json.dumps(analysis.manifest[field], sort_keys=True) for analysis in analyses} + if len(values) != 1: + _fail(f"comparison requires identical manifest field {field!r}, got {sorted(values)}") + + by_seed: dict[Any, dict[str, RunAnalysis]] = defaultdict(dict) + for condition, condition_runs in by_condition.items(): + for analysis in condition_runs: + seed = analysis.manifest["seed"] + if condition in by_seed[seed]: + _fail(f"duplicate {condition} run for seed {seed}") + by_seed[seed][condition] = analysis + incomplete = {seed: sorted(pair) for seed, pair in by_seed.items() if set(pair) != {"baseline", "experiment"}} + if incomplete: + _fail(f"comparison has unpaired seeds: {incomplete}") + + steady_steps = _stable_steps(windows) + if enforce_targets: + required_tags = PERFORMANCE_TAGS + CORRECTNESS_GUARDRAIL_TAGS + for analysis in analyses: + context = f"{analysis.manifest['condition']} seed={analysis.manifest['seed']}" + trace_steps = {int(row["rollout_id"]) for row in analysis.trace_rows} + missing_trace_steps = sorted(steady_steps - trace_steps) + if missing_trace_steps: + _fail(f"{context} is missing steady trace steps {missing_trace_steps}") + for tag in required_tags: + metric_steps = { + int(row["step"]) + for row in analysis.scalar_rows + if row["tag"] == tag and int(row["step"]) in steady_steps + } + missing_metric_steps = sorted(steady_steps - metric_steps) + if missing_metric_steps: + _fail(f"{context} {tag} is missing steady steps {missing_metric_steps}") + gpu_count = analysis.summary["nvml"].get("gpu_count") + if gpu_count != expected_gpu_count: + _fail(f"{context} expected NVML data for {expected_gpu_count} GPUs, got {gpu_count!r}") + per_gpu_nvml = analysis.summary["nvml"].get("per_gpu", {}) + if len(per_gpu_nvml) != expected_gpu_count: + _fail( + f"{context} expected per-GPU NVML summaries for {expected_gpu_count} GPUs, got {len(per_gpu_nvml)}" + ) + missing_steady_nvml = sorted( + gpu_index + for gpu_index, gpu_summary in per_gpu_nvml.items() + if int(gpu_summary.get("steady_sample_count", 0)) <= 0 + ) + if missing_steady_nvml: + _fail( + f"{context} has no NVML samples aligned to registered steady steps for GPUs {missing_steady_nvml}" + ) + steady_trace_rows = [row for row in analysis.trace_rows if int(row["rollout_id"]) in steady_steps] + observed_lead = max(row["producer_lead_at_first_forward"] for row in steady_trace_rows) + max_staleness = int(analysis.manifest["max_staleness"]) + if observed_lead > max_staleness: + _fail( + f"{context} observed producer lead exceeds configured " + f"max_staleness={max_staleness}: {observed_lead}" + ) + + paired = [] + window_speedups = [] + for seed, pair in sorted(by_seed.items(), key=lambda item: str(item[0])): + row: dict[str, Any] = {"seed": seed} + for tag in COMPARISON_PERFORMANCE_TAGS: + baseline_metrics = pair["baseline"].summary["metrics"].get(tag, {}) + experiment_metrics = pair["experiment"].summary["metrics"].get(tag, {}) + key = "aggregate" if "per_s" in tag else "mean" + baseline_value = baseline_metrics.get(key) + experiment_value = experiment_metrics.get(key) + if baseline_value is None or experiment_value is None: + continue + if baseline_value <= 0 or experiment_value <= 0: + _fail(f"{tag} must be positive for seed {seed}, got {baseline_value=} and {experiment_value=}") + row[f"{tag}:baseline"] = baseline_value + row[f"{tag}:experiment"] = experiment_value + if "time" in tag: + row[f"{tag}:improvement"] = 1 - experiment_value / baseline_value + else: + row[f"{tag}:improvement"] = experiment_value / baseline_value - 1 + + for window_start, window_end in windows: + window = ((window_start, window_end),) + baseline_throughput = _aggregate_throughput( + pair["baseline"].scalar_rows, + "perf/step_token_per_s", + window, + ) + experiment_throughput = _aggregate_throughput( + pair["experiment"].scalar_rows, + "perf/step_token_per_s", + window, + ) + if baseline_throughput is not None and experiment_throughput is not None: + if baseline_throughput <= 0 or experiment_throughput <= 0: + _fail(f"window {window_start}-{window_end} throughput must be positive for seed {seed}") + window_speedups.append( + { + "seed": seed, + "window_start": window_start, + "window_end": window_end, + "baseline": baseline_throughput, + "experiment": experiment_throughput, + "speedup": experiment_throughput / baseline_throughput - 1, + } + ) + + baseline_step_p95 = pair["baseline"].summary["metrics"].get("perf/step_time", {}).get("p95") + experiment_step_p95 = pair["experiment"].summary["metrics"].get("perf/step_time", {}).get("p95") + if baseline_step_p95 is not None and experiment_step_p95 is not None: + if baseline_step_p95 <= 0 or experiment_step_p95 <= 0: + _fail( + f"perf/step_time p95 must be positive for seed {seed}, " + f"got {baseline_step_p95=} and {experiment_step_p95=}" + ) + row["perf/step_time:p95_baseline"] = baseline_step_p95 + row["perf/step_time:p95_experiment"] = experiment_step_p95 + row["perf/step_time:p95_regression"] = experiment_step_p95 / baseline_step_p95 - 1 + + for tag in CORRECTNESS_GUARDRAIL_TAGS: + baseline_value = pair["baseline"].summary["metrics"].get(tag, {}).get("mean") + experiment_value = pair["experiment"].summary["metrics"].get(tag, {}).get("mean") + if baseline_value is not None and experiment_value is not None: + row[f"{tag}:baseline"] = baseline_value + row[f"{tag}:experiment"] = experiment_value + row[f"{tag}:delta"] = experiment_value - baseline_value + + for field in ( + "actor_fetch_samples", + "actor_total_tokens", + "actor_response_tokens", + "actor_multimodal_tensor_bytes", + ): + baseline_total = sum( + int(trace_row[field]) + for trace_row in pair["baseline"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + ) + experiment_total = sum( + int(trace_row[field]) + for trace_row in pair["experiment"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + ) + row[f"{field}:baseline"] = baseline_total + row[f"{field}:experiment"] = experiment_total + row[f"{field}:relative_delta"] = ( + experiment_total / baseline_total - 1 + if baseline_total + else (0.0 if experiment_total == 0 else math.inf) + ) + + baseline_vram = pair["baseline"].summary["nvml"].get("peak_memory_used_mib") + experiment_vram = pair["experiment"].summary["nvml"].get("peak_memory_used_mib") + if baseline_vram is not None and experiment_vram is not None: + row["nvml_peak_memory_mib:baseline"] = baseline_vram + row["nvml_peak_memory_mib:experiment"] = experiment_vram + row["nvml_peak_memory_mib:delta"] = experiment_vram - baseline_vram + + baseline_lead = statistics.fmean( + trace_row["producer_lead_at_first_forward"] + for trace_row in pair["baseline"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + ) + experiment_lead = statistics.fmean( + trace_row["producer_lead_at_first_forward"] + for trace_row in pair["experiment"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + ) + row["producer_lead_at_first_forward:baseline"] = baseline_lead + row["producer_lead_at_first_forward:experiment"] = experiment_lead + row["producer_lead_at_first_forward:delta"] = experiment_lead - baseline_lead + paired.append(row) + + token_ratios = [ + 1 + row["perf/step_token_per_s:improvement"] for row in paired if "perf/step_token_per_s:improvement" in row + ] + phase_ratios = [ + 1 - row["perf/hybrid_phase1_time:improvement"] + for row in paired + if "perf/hybrid_phase1_time:improvement" in row + ] + experiment_overlap_rows = [ + row + for analysis in by_condition["experiment"] + for row in analysis.trace_rows + if int(row["rollout_id"]) in _stable_steps(windows) + ] + experiment_producer_overlap_by_run = {} + for analysis in by_condition["experiment"]: + rows = [row for row in analysis.trace_rows if int(row["rollout_id"]) in steady_steps] + experiment_producer_overlap_by_run[str(analysis.run_dir)] = ( + sum(bool(row["first_forward_before_last_put_start"]) for row in rows) / len(rows) if rows else None + ) + comparison = { + "paired_runs": paired, + "window_speedups": window_speedups, + "paired_seed_count": len(paired), + "token_throughput_geomean_speedup": (_geometric_mean(token_ratios) - 1 if token_ratios else None), + "hybrid_phase1_geomean_reduction": (1 - _geometric_mean(phase_ratios) if phase_ratios else None), + "experiment_steady_producer_overlap_ratio": ( + sum(bool(row["first_forward_before_last_put_start"]) for row in experiment_overlap_rows) + / len(experiment_overlap_rows) + if experiment_overlap_rows + else None + ), + "experiment_steady_producer_overlap_ratio_by_run": experiment_producer_overlap_by_run, + } + + if enforce_targets: + if len(paired) < 2: + _fail(f"performance targets require at least two paired seeds, got {len(paired)}") + improvements = [row.get("perf/step_token_per_s:improvement") for row in paired] + if any(value is None for value in improvements): + _fail("performance targets require perf/step_token_per_s for every paired run") + if any("perf/wall_clock_samples_per_s:baseline" not in row for row in paired): + _fail("performance targets require wall-clock samples/s for every paired run") + if any(value <= 0 for value in improvements): + _fail(f"every paired token-throughput speedup must be positive, got {improvements}") + if comparison["token_throughput_geomean_speedup"] < 0.05: + _fail( + "token-throughput geometric-mean speedup is below 5%: " + f"{comparison['token_throughput_geomean_speedup']:.4%}" + ) + phase_improvements = [row.get("perf/hybrid_phase1_time:improvement") for row in paired] + if any(value is None for value in phase_improvements): + _fail("performance targets require perf/hybrid_phase1_time for every paired run") + if any(value < 0 for value in phase_improvements): + _fail(f"every paired Hybrid phase-1 result must be non-regressive, got {phase_improvements}") + if comparison["hybrid_phase1_geomean_reduction"] < 0.15: + _fail( + "Hybrid phase-1 geometric-mean reduction is below 15%: " + f"{comparison['hybrid_phase1_geomean_reduction']:.4%}" + ) + failing_overlap = { + run_dir: ratio + for run_dir, ratio in experiment_producer_overlap_by_run.items() + if ratio is None or ratio < 0.8 + } + if failing_overlap: + _fail(f"each experiment run requires at least 80% steady producer overlap, got {failing_overlap}") + + step_p95_regressions = [row.get("perf/step_time:p95_regression") for row in paired] + if any(value is None for value in step_p95_regressions): + _fail("performance targets require perf/step_time p95 for every paired run") + if any(value > 0.05 for value in step_p95_regressions): + _fail(f"paired step-time p95 regression exceeds 5%: {step_p95_regressions}") + + for row in paired: + seed = row["seed"] + if row["actor_fetch_samples:baseline"] != row["actor_fetch_samples:experiment"]: + _fail(f"seed {seed} actor fetch sample count changed") + for field in ( + "actor_total_tokens", + "actor_response_tokens", + "actor_multimodal_tensor_bytes", + ): + baseline_total = row[f"{field}:baseline"] + experiment_total = row[f"{field}:experiment"] + if baseline_total <= 0 or experiment_total <= 0: + _fail(f"seed {seed} {field} must be positive, got {baseline_total}, {experiment_total}") + relative_delta = row[f"{field}:relative_delta"] + if abs(relative_delta) > 0.01: + _fail(f"seed {seed} {field} changed by more than 1%: {relative_delta:.4%}") + + vram_baseline = row.get("nvml_peak_memory_mib:baseline") + vram_delta = row.get("nvml_peak_memory_mib:delta") + if vram_baseline is None or vram_delta is None: + _fail(f"seed {seed} is missing paired NVML peak memory") + allowed_vram_delta = max(1024.0, 0.03 * vram_baseline) + if vram_delta > allowed_vram_delta: + _fail(f"seed {seed} peak VRAM increased by {vram_delta:.1f} MiB; allowed={allowed_vram_delta:.1f} MiB") + + accuracy_deltas = [row.get("rollout/raw_reward:delta") for row in paired] + if any(value is None for value in accuracy_deltas): + _fail("correctness targets require rollout/raw_reward for every paired run") + if any(value < -0.03 for value in accuracy_deltas): + _fail(f"a paired raw-reward/accuracy drop exceeds 3 percentage points: {accuracy_deltas}") + if statistics.fmean(accuracy_deltas) < -0.02: + _fail(f"mean paired raw-reward/accuracy drop exceeds 2 percentage points: {accuracy_deltas}") + + truncation_deltas = [row.get("rollout/truncated:delta") for row in paired] + if any(value is None for value in truncation_deltas): + _fail("correctness targets require rollout/truncated for every paired run") + if any(value > 0.02 for value in truncation_deltas): + _fail(f"a paired truncation-rate increase exceeds 2 percentage points: {truncation_deltas}") + + staleness_deltas = [row["producer_lead_at_first_forward:delta"] for row in paired] + if any(value > 0.25 for value in staleness_deltas): + _fail(f"a paired average producer-lead increase exceeds 0.25: {staleness_deltas}") + + baseline_throughputs = [row["perf/step_token_per_s:baseline"] for row in paired] + if len(paired) == 2: + baseline_cv = statistics.pstdev(baseline_throughputs) / statistics.fmean(baseline_throughputs) + if baseline_cv > 0.05: + _fail( + f"two baseline runs have CV={baseline_cv:.2%} > 5%; " + "the preregistered protocol requires a third paired seed" + ) + return comparison + + +def _plot_comparison( + analyses: Sequence[RunAnalysis], + comparison: dict[str, Any], + output_dir: Path, + windows: Sequence[tuple[int, int]], +) -> list[str]: + try: + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + except ImportError: + return [] + + output_dir.mkdir(parents=True, exist_ok=True) + generated = [] + + def finish(filename: str) -> None: + path = output_dir / filename + plt.tight_layout() + plt.savefig(path, dpi=160) + plt.close() + generated.append(str(path)) + + figure, axes = plt.subplots(2, 1, figsize=(11, 9), sharex=True) + for axis, tag in zip( + axes, + ("perf/step_token_per_s", "perf/step_resp_token_per_s"), + strict=True, + ): + for analysis in analyses: + label = f"{analysis.manifest['condition']}-seed{analysis.manifest['seed']}" + values = [row for row in analysis.scalar_rows if row["tag"] == tag] + if values: + axis.plot( + [row["step"] for row in values], + [row["value"] for row in values], + marker="o", + label=label, + ) + for start, end in windows: + axis.axvspan(start, end, color="grey", alpha=0.08) + axis.set_ylabel(tag) + axis.legend() + axes[-1].set_xlabel("rollout / optimizer step") + figure.suptitle("Task 21 step token throughput") + finish("task21_step_throughput.png") + + figure, axes = plt.subplots(3, 1, figsize=(11, 12), sharex=True) + for analysis in analyses: + label = f"{analysis.manifest['condition']}-seed{analysis.manifest['seed']}" + axes[0].plot( + [row["rollout_id"] for row in analysis.trace_rows], + [row["producer_overlap_s"] for row in analysis.trace_rows], + marker="o", + label=label, + ) + phase1 = [row for row in analysis.scalar_rows if row["tag"] == "perf/hybrid_phase1_time"] + if phase1: + axes[1].plot( + [row["step"] for row in phase1], + [row["value"] for row in phase1], + marker="o", + label=label, + ) + axes[2].plot( + [row["rollout_id"] for row in analysis.trace_rows], + [row["producer_lead_at_first_forward"] for row in analysis.trace_rows], + marker="o", + label=label, + ) + axes[0].axhline(0, color="black", linewidth=0.8) + axes[0].set_ylabel("last put start - first actor forward start (s)") + axes[1].set_ylabel("perf/hybrid_phase1_time (s)") + max_staleness = int(analyses[0].manifest["max_staleness"]) + axes[2].axhline( + max_staleness, + color="red", + linestyle="--", + label=f"max_staleness={max_staleness}", + ) + axes[2].set_xlabel("rollout / optimizer step") + axes[2].set_ylabel("producer lead at actor forward (steps)") + for axis in axes: + axis.legend() + figure.suptitle("Task 21 producer / actor overlap and phase-1 time") + finish("task21_phase1_overlap.png") + + def numeric_or_nan(value: Any) -> float: + return float(value) if value is not None else float("nan") + + labels = [f"{analysis.manifest['condition']}-s{analysis.manifest['seed']}" for analysis in analyses] + utilization = [ + numeric_or_nan(analysis.summary["nvml"].get("steady_mean_gpu_util_percent")) for analysis in analyses + ] + idle_ratio = [ + numeric_or_nan(analysis.summary["nvml"].get("steady_idle_ratio_below_10_percent")) * 100 + for analysis in analyses + ] + vram = [numeric_or_nan(analysis.summary["nvml"].get("peak_memory_used_mib")) / 1024 for analysis in analyses] + figure, axes = plt.subplots(2, 2, figsize=(14, 9)) + for analysis in analyses: + label = f"{analysis.manifest['condition']}-seed{analysis.manifest['seed']}" + by_timestamp: dict[float, list[dict[str, Any]]] = defaultdict(list) + for row in analysis.nvml_rows: + if row["wall_time"] is not None: + by_timestamp[float(row["wall_time"])].append(row) + if by_timestamp: + start_time = min(by_timestamp) + timestamps = sorted(by_timestamp) + elapsed = [timestamp - start_time for timestamp in timestamps] + mean_utilization = [ + statistics.fmean(float(row["gpu_util_percent"]) for row in by_timestamp[timestamp]) + for timestamp in timestamps + ] + peak_vram = [ + max(float(row["memory_used_mib"]) for row in by_timestamp[timestamp]) / 1024 + for timestamp in timestamps + ] + axes[0, 0].plot(elapsed, mean_utilization, label=label) + axes[1, 0].plot(elapsed, peak_vram, label=label) + for start, end in analysis.summary["nvml"].get("steady_wall_time_intervals", []): + axes[0, 0].axvspan(start - start_time, end - start_time, color="grey", alpha=0.025) + axes[1, 0].axvspan(start - start_time, end - start_time, color="grey", alpha=0.025) + axes[0, 0].set_ylabel("mean GPU utilization (%)") + axes[0, 0].set_xlabel("seconds since first NVML sample") + axes[1, 0].set_ylabel("max per-GPU VRAM (GiB)") + axes[1, 0].set_xlabel("seconds since first NVML sample") + for axis in (axes[0, 0], axes[1, 0]): + handles, _ = axis.get_legend_handles_labels() + if handles: + axis.legend() + positions = list(range(len(labels))) + axes[0, 1].bar([position - 0.2 for position in positions], utilization, width=0.4, label="mean util") + axes[0, 1].bar([position + 0.2 for position in positions], idle_ratio, width=0.4, label="idle <10%") + axes[0, 1].set_xticks(positions, labels) + axes[0, 1].set_ylabel("steady-window percent") + axes[0, 1].legend() + axes[1, 1].bar(labels, vram) + axes[1, 1].set_ylabel("full-run sampled peak VRAM (GiB)") + for axis in (axes[0, 1], axes[1, 1]): + axis.tick_params(axis="x", rotation=25) + figure.suptitle("Task 21 GPU utilization and VRAM") + finish("task21_gpu_util_vram.png") + + quality_tags = sorted( + { + row["tag"] + for analysis in analyses + for row in analysis.scalar_rows + if any(fragment in row["tag"].lower() for fragment in QUALITY_TAG_FRAGMENTS) + } + ) + plt.figure(figsize=(12, 7)) + for analysis in analyses: + for tag in quality_tags: + values = [row for row in analysis.scalar_rows if row["tag"] == tag] + if values: + plt.plot( + [row["step"] for row in values], + [row["value"] for row in values], + label=f"{analysis.manifest['condition']}-s{analysis.manifest['seed']}:{tag}", + ) + plt.xlabel("rollout / optimizer step") + plt.ylabel("raw metric value") + if quality_tags: + plt.legend(fontsize=7, ncol=2) + plt.title("Task 21 correctness and quality guardrails") + finish("task21_correctness_quality.png") + + paired = comparison["paired_runs"] + figure, axes = plt.subplots(2, 1, figsize=(12, 9)) + window_speedups = comparison["window_speedups"] + window_labels = [f"s{row['seed']}:{row['window_start']}-{row['window_end']}" for row in window_speedups] + axes[0].scatter( + window_labels, + [100 * row["speedup"] for row in window_speedups], + marker="o", + ) + axes[0].axhline(5, color="red", linestyle="--", label="5% run-level target") + axes[0].set_ylabel("window token-throughput speedup (%)") + axes[0].tick_params(axis="x", rotation=25) + axes[0].legend() + + pair_labels = [f"seed {row['seed']}" for row in paired] + speedups = [100 * row.get("perf/step_token_per_s:improvement", float("nan")) for row in paired] + pair_labels.append("geometric mean") + speedups.append(100 * (comparison["token_throughput_geomean_speedup"] or 0)) + axes[1].bar(pair_labels, speedups) + axes[1].axhline(5, color="red", linestyle="--", label="5% target") + axes[1].set_ylabel("paired run token-throughput speedup (%)") + axes[1].legend() + figure.suptitle("Task 21 preregistered windows and paired run summary") + finish("task21_window_summary.png") + return generated + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--run-dir", + action="append", + type=Path, + required=True, + help="Benchmark run directory. Repeat for paired baseline/experiment comparison.", + ) + parser.add_argument( + "--output-dir", + type=Path, + default=None, + help="Comparison output directory; required only when more than one --run-dir is supplied.", + ) + parser.add_argument( + "--steady-windows", + default="4-8,9-13,14-18", + help="Inclusive, non-overlapping step windows as START-END comma-separated ranges.", + ) + parser.add_argument("--expected-samples", type=int, default=256) + parser.add_argument( + "--expected-actor-chunks", + type=int, + default=2, + help="Expected actor fetch/forward chunks when the pipeline is enabled; producer put grouping is dynamic.", + ) + parser.add_argument( + "--expected-gpu-count", + type=int, + default=8, + help="Required distinct GPU indexes in NVML telemetry when enforcing targets.", + ) + parser.add_argument( + "--validate-only", + action="store_true", + help="Validate each run without requiring paired performance target checks.", + ) + parser.add_argument( + "--enforce-targets", + action="store_true", + help="Require the preregistered 5% throughput, 15% phase-1, and 80% overlap targets.", + ) + parser.add_argument( + "--no-plots", + action="store_true", + help="Do not generate comparison PNG files.", + ) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + args = build_parser().parse_args(argv) + try: + windows = _parse_windows(args.steady_windows) + if args.expected_samples <= 0 or args.expected_actor_chunks <= 0 or args.expected_gpu_count <= 0: + _fail("expected sample, actor chunk, and GPU counts must be positive") + if len(args.run_dir) > 1 and args.output_dir is None: + _fail("--output-dir is required when comparing multiple runs") + if args.validate_only and args.enforce_targets: + _fail("--validate-only and --enforce-targets are mutually exclusive") + + analyses = [ + analyze_run( + run_dir, + windows=windows, + expected_samples=args.expected_samples, + expected_actor_chunks=args.expected_actor_chunks, + require_reproducibility_artifacts=args.enforce_targets, + ) + for run_dir in args.run_dir + ] + result: dict[str, Any] = { + "runs": [analysis.summary for analysis in analyses], + "validation": "passed", + } + if len(analyses) > 1: + comparison = _build_comparison( + analyses, + windows=windows, + enforce_targets=args.enforce_targets, + expected_gpu_count=args.expected_gpu_count, + ) + result["comparison"] = comparison + output_dir = args.output_dir.resolve() + output_dir.mkdir(parents=True, exist_ok=True) + if not args.no_plots: + result["plots"] = _plot_comparison(analyses, comparison, output_dir, windows) + (output_dir / "comparison_summary.json").write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + _write_csv(output_dir / "paired_run_summary.csv", comparison["paired_runs"]) + _write_csv(output_dir / "window_speedup_summary.csv", comparison["window_speedups"]) + print(json.dumps(result, indent=2, sort_keys=True)) + return 0 + except BenchmarkValidationError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh index 00228498d..d1a453a42 100644 --- a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh +++ b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh @@ -6,7 +6,7 @@ # The Ray cluster is managed externally — do NOT kill ray or start a new cluster. # # Usage: -# bash scripts/training/multimodal/run-qwen35-9B-8xgpu-async.sh [async|sync] +# bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh [hybrid-async|sync] set -ex set -o pipefail @@ -30,6 +30,86 @@ MODEL_DIR="${MODEL_DIR:-${EXP_DIR}}" DATA_DIR="${DATA_DIR:-${EXP_DIR}}" NUM_ROLLOUT="${NUM_ROLLOUT:=200}" +HYBRID_PIPELINE_FORWARD="${HYBRID_PIPELINE_FORWARD:-0}" +HYBRID_PIPELINE_TRACE_DIR="${HYBRID_PIPELINE_TRACE_DIR:-}" +HYBRID_PIPELINE_FETCH_TIMEOUT_S="${HYBRID_PIPELINE_FETCH_TIMEOUT_S:-600}" +SGLANG_DETERMINISTIC_INFERENCE="${SGLANG_DETERMINISTIC_INFERENCE:-0}" +SEED="${SEED:-}" +ROLLOUT_SEED="${ROLLOUT_SEED:-}" + +case "${MODE}" in + hybrid-async|sync) ;; + *) + echo "MODE must be hybrid-async or sync, got ${MODE}" >&2 + exit 2 + ;; +esac +case "${HYBRID_PIPELINE_FORWARD}" in + 0|1) ;; + *) + echo "HYBRID_PIPELINE_FORWARD must be 0 or 1, got ${HYBRID_PIPELINE_FORWARD}" >&2 + exit 2 + ;; +esac +case "${SGLANG_DETERMINISTIC_INFERENCE}" in + 0|1) ;; + *) + echo "SGLANG_DETERMINISTIC_INFERENCE must be 0 or 1, got ${SGLANG_DETERMINISTIC_INFERENCE}" >&2 + exit 2 + ;; +esac +if [ "${MODE}" != "hybrid-async" ] && { + [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] || [ -n "${HYBRID_PIPELINE_TRACE_DIR}" ]; +}; then + echo "Hybrid pipeline forward/trace options require MODE=hybrid-async" >&2 + exit 2 +fi + +HYBRID_PIPELINE_ARGS=() +if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ]; then + HYBRID_PIPELINE_ARGS+=( + --hybrid-pipeline-forward + --hybrid-pipeline-fetch-timeout-s "${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" + ) +fi +if [ -n "${HYBRID_PIPELINE_TRACE_DIR}" ]; then + HYBRID_PIPELINE_ARGS+=( + --hybrid-pipeline-trace-dir "${HYBRID_PIPELINE_TRACE_DIR}" + ) +fi + +REPRO_ARGS=() +if [ -n "${SEED}" ]; then + REPRO_ARGS+=(--seed "${SEED}") +fi +if [ -n "${ROLLOUT_SEED}" ]; then + REPRO_ARGS+=(--rollout-seed "${ROLLOUT_SEED}") +fi + +DEBUG_ARGS=() +if [ -n "${SAVE_DEBUG_ROLLOUT_DATA:-}" ]; then + DEBUG_ARGS+=(--save-debug-rollout-data "${SAVE_DEBUG_ROLLOUT_DATA}") +fi +if [ -n "${LOAD_DEBUG_ROLLOUT_DATA:-}" ]; then + DEBUG_ARGS+=(--load-debug-rollout-data "${LOAD_DEBUG_ROLLOUT_DATA}") +fi +if [ -n "${SAVE_DEBUG_TRAIN_DATA:-}" ]; then + DEBUG_ARGS+=(--save-debug-train-data "${SAVE_DEBUG_TRAIN_DATA}") +fi +if [ "${SGLANG_DETERMINISTIC_INFERENCE}" = "1" ]; then + DEBUG_ARGS+=(--sglang-enable-deterministic-inference) +fi + +printf '%s\n' \ + "MODE=${MODE}" \ + "NUM_ROLLOUT=${NUM_ROLLOUT}" \ + "HYBRID_PIPELINE_FORWARD=${HYBRID_PIPELINE_FORWARD}" \ + "HYBRID_PIPELINE_TRACE_DIR=${HYBRID_PIPELINE_TRACE_DIR}" \ + "HYBRID_PIPELINE_FETCH_TIMEOUT_S=${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" \ + "SEED=${SEED}" \ + "ROLLOUT_SEED=${ROLLOUT_SEED}" \ + "SGLANG_DETERMINISTIC_INFERENCE=${SGLANG_DETERMINISTIC_INFERENCE}" + CKPT_ARGS=( --hf-checkpoint ${MODEL_DIR}/Qwen3.5-9B @@ -139,8 +219,9 @@ MISC_ARGS=( ) -mkdir -p log -if [ ${MODE} = "hybrid-async" ]; then +LOG_DIR="${EXP_DIR}/logs" +mkdir -p "${LOG_DIR}" +if [ "${MODE}" = "hybrid-async" ]; then ray job submit ${RAY_NO_WAIT:+--no-wait} --address="http://127.0.0.1:8265" \ --runtime-env-json="${RUNTIME_ENV_JSON}" \ -- python3 -m relax.entrypoints.train \ @@ -150,6 +231,9 @@ if [ ${MODE} = "hybrid-async" ]; then --num-iters-per-train-update 2 \ --balance-data \ --hybrid \ + "${HYBRID_PIPELINE_ARGS[@]}" \ + "${REPRO_ARGS[@]}" \ + "${DEBUG_ARGS[@]}" \ "${MODEL_ARGS[@]}" \ "${CKPT_ARGS[@]}" \ "${ROLLOUT_ARGS[@]}" \ @@ -158,7 +242,7 @@ if [ ${MODE} = "hybrid-async" ]; then "${WANDB_ARGS[@]}" \ "${PERF_ARGS[@]}" \ "${SGLANG_ARGS[@]}" \ - "${MISC_ARGS[@]}" 2>&1 | tee log/qwen35-9b-GRPO-gpu8-hybrid-async-${now}.log + "${MISC_ARGS[@]}" 2>&1 | tee "${LOG_DIR}/qwen35-9b-GRPO-gpu8-hybrid-async-${now}.log" else ray job submit ${RAY_NO_WAIT:+--no-wait} --address="http://127.0.0.1:8265" \ --runtime-env-json="${RUNTIME_ENV_JSON}" \ @@ -169,6 +253,8 @@ else --colocate \ --use-health-check \ --balance-data \ + "${REPRO_ARGS[@]}" \ + "${DEBUG_ARGS[@]}" \ "${MODEL_ARGS[@]}" \ "${CKPT_ARGS[@]}" \ "${ROLLOUT_ARGS[@]}" \ @@ -177,5 +263,5 @@ else "${WANDB_ARGS[@]}" \ "${PERF_ARGS[@]}" \ "${SGLANG_ARGS[@]}" \ - "${MISC_ARGS[@]}" 2>&1 | tee log/qwen35-9b-GRPO-gpu8-fully-sync-${now}.log + "${MISC_ARGS[@]}" 2>&1 | tee "${LOG_DIR}/qwen35-9b-GRPO-gpu8-fully-sync-${now}.log" fi diff --git a/tests/backends/megatron/test_data_vpp.py b/tests/backends/megatron/test_data_vpp.py index c2f8980aa..a4002b5a4 100644 --- a/tests/backends/megatron/test_data_vpp.py +++ b/tests/backends/megatron/test_data_vpp.py @@ -15,6 +15,7 @@ def _load_data_module(monkeypatch): training = types.ModuleType("megatron.training") global_vars = types.ModuleType("megatron.training.global_vars") tracking_utils = types.ModuleType("relax.utils.tracking_utils") + ray = types.ModuleType("ray") class _PackedSeqParams: pass @@ -31,6 +32,7 @@ class _PackedSeqParams: "megatron.training": training, "megatron.training.global_vars": global_vars, "relax.utils.tracking_utils": tracking_utils, + "ray": ray, } for name, module in modules.items(): monkeypatch.setitem(sys.modules, name, module) @@ -94,6 +96,58 @@ def test_rollout_minibatch_plan_rejects_non_divisible_prompt_groups(monkeypatch) data_module.build_rollout_minibatch_plan(args, dp_size=2) +def test_hybrid_forward_chunk_plan_matches_producer_granularity(monkeypatch): + data_module = _load_data_module(monkeypatch) + args = Namespace( + rollout_batch_size=32, + n_samples_per_prompt=8, + global_batch_size=256, + num_steps_per_rollout=None, + num_iters_per_train_update=2, + ) + rollout_plan = data_module.build_rollout_minibatch_plan(args, dp_size=1) + + chunk_plan = data_module.build_hybrid_forward_chunk_plan(args, rollout_plan, dp_size=1) + + assert chunk_plan.chunks_per_mini == 2 + assert chunk_plan.chunk_global_samples == 128 + assert chunk_plan.chunk_local_samples == 128 + assert [ + chunk_plan.transfer_queue_batch_index(mini_index, chunk_index) + for mini_index in range(3) + for chunk_index in range(chunk_plan.chunks_per_mini) + ] == [0, 1, 2, 3, 4, 5] + + +@pytest.mark.parametrize( + ("num_iters", "global_batch_size", "n_samples_per_prompt", "error"), + [ + (1, 256, 8, "num_iters_per_train_update >= 2"), + (3, 256, 8, "mini_global_samples must be divisible"), + (4, 24, 8, "must preserve complete prompt groups"), + ], +) +def test_hybrid_forward_chunk_plan_rejects_unsafe_boundaries( + monkeypatch, + num_iters, + global_batch_size, + n_samples_per_prompt, + error, +): + data_module = _load_data_module(monkeypatch) + args = Namespace( + rollout_batch_size=global_batch_size // n_samples_per_prompt, + n_samples_per_prompt=n_samples_per_prompt, + global_batch_size=global_batch_size, + num_steps_per_rollout=None, + num_iters_per_train_update=num_iters, + ) + rollout_plan = data_module.build_rollout_minibatch_plan(args, dp_size=1) + + with pytest.raises(ValueError, match=error): + data_module.build_hybrid_forward_chunk_plan(args, rollout_plan, dp_size=1) + + def test_concat_rollout_batches_preserves_order_and_scalar_metadata(monkeypatch): data_module = _load_data_module(monkeypatch) @@ -120,6 +174,117 @@ def test_concat_rollout_batches_preserves_order_and_scalar_metadata(monkeypatch) assert merged["weight_version"] == 7 +@pytest.mark.parametrize( + ("second_batch", "error"), + [ + ( + {"tokens": ["c"], "total_lengths": [3]}, + "schema mismatch", + ), + ( + { + "tokens": ["c", "unexpected"], + "total_lengths": [3], + "weight_version": 7, + }, + "Per-sample rollout field", + ), + ], +) +def test_concat_rollout_batches_rejects_incomplete_chunks(monkeypatch, second_batch, error): + data_module = _load_data_module(monkeypatch) + + with pytest.raises(ValueError, match=error): + data_module.concat_rollout_batches( + [ + { + "tokens": ["a", "b"], + "total_lengths": [1, 2], + "weight_version": 7, + }, + second_batch, + ] + ) + + +def test_canonicalize_rollout_chunks_reorders_every_sample_field(monkeypatch): + data_module = _load_data_module(monkeypatch) + chunks = [ + ( + { + "tokens": ["token-12", "token-10"], + "total_lengths": [12, 10], + "multimodal_train_inputs": [{"image": "12"}, {"image": "10"}], + "scores": torch.tensor([[12], [10]]), + "array": data_module.np.array([[12], [10]]), + "weight_version": 7, + }, + [12, 10], + ), + ( + { + "tokens": ["token-13", "token-11"], + "total_lengths": [13, 11], + "multimodal_train_inputs": [{"image": "13"}, {"image": "11"}], + "scores": torch.tensor([[13], [11]]), + "array": data_module.np.array([[13], [11]]), + "weight_version": 7, + }, + [13, 11], + ), + ] + + merged, global_indexes = data_module.canonicalize_rollout_chunks(chunks, expected_sample_count=4) + + assert global_indexes == [10, 11, 12, 13] + assert merged["tokens"] == ["token-10", "token-11", "token-12", "token-13"] + assert merged["total_lengths"] == [10, 11, 12, 13] + assert merged["multimodal_train_inputs"] == [ + {"image": "10"}, + {"image": "11"}, + {"image": "12"}, + {"image": "13"}, + ] + assert torch.equal(merged["scores"], torch.tensor([[10], [11], [12], [13]])) + assert merged["array"].tolist() == [[10], [11], [12], [13]] + assert merged["weight_version"] == 7 + + +@pytest.mark.parametrize( + ("chunks", "expected_count", "error"), + [ + ( + [ + ({"total_lengths": [1, 1]}, [10, 11]), + ({"total_lengths": [1, 1]}, [11, 12]), + ], + 4, + "contain duplicates", + ), + ( + [({"total_lengths": [1, 1]}, [10])], + 2, + "length mismatch", + ), + ( + [({"total_lengths": [1]}, [10])], + 2, + "sample count mismatch", + ), + ], +) +def test_canonicalize_rollout_chunks_rejects_invalid_metadata( + monkeypatch, + chunks, + expected_count, + error, +): + data_module = _load_data_module(monkeypatch) + + with pytest.raises((ValueError, TypeError), match=error): + data_module.canonicalize_rollout_chunks(chunks, expected_count) + + def test_get_data_iterator_uses_rollout_mini_boundaries_with_balance_data(monkeypatch): data_module = _load_data_module(monkeypatch) monkeypatch.setattr( diff --git a/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py new file mode 100644 index 000000000..52b798ffc --- /dev/null +++ b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py @@ -0,0 +1,177 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import importlib +from contextlib import nullcontext +from types import SimpleNamespace + +import pytest + + +try: + actor_module = importlib.import_module("relax.backends.megatron.actor") +except ImportError: + actor_module = None + +pytestmark = pytest.mark.skipif( + actor_module is None, + reason="Megatron/Ray/TransferQueue runtime dependencies are required for the actor wiring test", +) + + +def test_hybrid_pipeline_runtime_rechecks_supported_parallel_topology(monkeypatch): + assert actor_module is not None + actor = object.__new__(actor_module.MegatronTrainRayActor) + actor.args = SimpleNamespace( + num_iters_per_train_update=2, + n_samples_per_prompt=2, + expert_tensor_parallel_size=1, + offload_train=False, + offload_rollout=False, + compute_advantages_and_returns=True, + ) + actor.weights_backuper = SimpleNamespace(backup_tags={"actor"}) + rollout_plan = SimpleNamespace( + num_rollout_minis=1, + mini_global_samples=4, + mini_local_sample_request=4, + fixed_n_samples_per_prompt=2, + ) + monkeypatch.setattr(actor_module.mpu, "get_pipeline_model_parallel_world_size", lambda: 1) + monkeypatch.setattr(actor_module.mpu, "get_virtual_pipeline_model_parallel_world_size", lambda: None) + monkeypatch.setattr(actor_module.mpu, "get_tensor_model_parallel_world_size", lambda: 2) + monkeypatch.setattr(actor_module.mpu, "get_context_parallel_world_size", lambda: 2) + monkeypatch.setattr(actor_module.mpu, "get_expert_model_parallel_world_size", lambda: 1) + + chunk_plan = actor._validate_hybrid_pipeline_runtime(rollout_plan, dp_size=1) + + assert chunk_plan.chunks_per_mini == 2 + assert chunk_plan.chunk_local_samples == 2 + + monkeypatch.setattr(actor_module.mpu, "get_context_parallel_world_size", lambda: 1) + with pytest.raises(RuntimeError, match="requires TP=2, CP=2, EP=1, and ETP=1"): + actor._validate_hybrid_pipeline_runtime(rollout_plan, dp_size=1) + + +def _rollout_batch(start: int, count: int) -> dict: + values = list(range(start, start + count)) + return { + "tokens": [[value] for value in values], + "total_lengths": [1] * count, + "response_lengths": [1] * count, + "loss_masks": [[1] for _ in values], + "rollout_log_probs": [[0.0] for _ in values], + "rewards": [1.0] * count, + "raw_reward": [1.0] * count, + } + + +@pytest.mark.parametrize( + ("pipeline_enabled", "expected_fetch_sizes", "expected_forward_count"), + [ + (False, [4], 1), + (True, [2, 2], 2), + ], +) +def test_train_hybrid_wires_one_update_around_flagged_actor_chunks( + monkeypatch, + pipeline_enabled, + expected_fetch_sizes, + expected_forward_count, +): + assert actor_module is not None + events = [] + actor = object.__new__(actor_module.MegatronTrainRayActor) + actor.args = SimpleNamespace( + rollout_batch_size=2, + n_samples_per_prompt=2, + global_batch_size=4, + num_steps_per_rollout=None, + num_iters_per_train_update=2, + hybrid_pipeline_forward=pipeline_enabled, + hybrid_pipeline_trace_dir=None, + hybrid_pipeline_fetch_timeout_s=1.0, + use_rollout_routing_replay=False, + multimodal_keys={"image": "image"}, + use_opd=False, + debug_train_only=False, + compute_advantages_and_returns=True, + use_routing_replay=False, + ref_update_interval=None, + num_rollout=1, + save=None, + ) + actor._hybrid_pipeline_chunk_plan = SimpleNamespace( + chunks_per_mini=2, + chunk_local_samples=2, + transfer_queue_batch_index=lambda mini_index, chunk_index: mini_index * 2 + chunk_index, + ) + actor._active_model_tag = "actor" + actor.rollout_data_postprocess = None + actor.model = actor.optimizer = actor.opt_param_scheduler = None + actor.tokenizer = actor.flops_counter = None + actor.prof = SimpleNamespace(step=lambda **_kwargs: events.append("profile")) + actor.weights_backuper = SimpleNamespace( + backup_tags={"actor"}, + backup=lambda tag: events.append(("backup", tag)), + ) + + def get_data(_task, _rollout_id, _fields, expected_samples, batch_index): + events.append(("fetch", expected_samples, batch_index)) + start = batch_index * expected_samples + return ( + _rollout_batch(start, expected_samples), + SimpleNamespace(global_indexes=list(range(start, start + expected_samples))), + ) + + actor._get_data_from_transfer_queue = get_data + actor.all_consumed = lambda *_args, **_kwargs: False + actor._restore_hybrid_pipeline_actor = lambda **kwargs: events.append(("restore", kwargs["chunk_index"])) + actor._hybrid_actor_forward_without_switch = lambda _batch, **kwargs: events.append( + ("forward", kwargs["chunk_index"]) + ) + actor._hybrid_forward_subbatch = lambda _batch, **kwargs: events.append(("forward", kwargs["chunk_index"])) + actor._switch_model = lambda tag: events.append(("switch", tag)) + actor._wait_for_previous_eval = lambda: None + actor._check_services_health = lambda: None + actor.update_weights = lambda: events.append("update_weights") + actor._run_step_evaluation = lambda *_args, **_kwargs: None + + monkeypatch.setattr( + actor_module.mpu, + "get_data_parallel_world_size", + lambda **_kwargs: 1, + ) + monkeypatch.setattr(actor_module.mpu, "get_data_parallel_rank", lambda: 0) + monkeypatch.setattr(actor_module.mpu, "get_data_parallel_group", lambda **_kwargs: None) + monkeypatch.setattr(actor_module, "timer", lambda *_args, **_kwargs: nullcontext()) + monkeypatch.setattr(actor_module, "inverse_timer", lambda *_args, **_kwargs: nullcontext()) + monkeypatch.setattr(actor_module, "emit_hybrid_pipeline_event", lambda *_args, **_kwargs: None) + monkeypatch.setattr( + actor_module, + "compute_advantages_and_returns", + lambda _args, _batch: events.append("advantages"), + ) + monkeypatch.setattr(actor_module, "log_rollout_data", lambda *_args, **_kwargs: None) + monkeypatch.setattr(actor_module, "get_data_iterator", lambda *_args, **_kwargs: ([], [])) + monkeypatch.setattr(actor_module, "train", lambda *_args, **_kwargs: events.append("optimizer")) + monkeypatch.setattr(actor_module.train_dump_utils, "save_debug_train_data", lambda *_args, **_kwargs: None) + monkeypatch.setattr(actor_module, "Timer", lambda: SimpleNamespace(seq_lens=None)) + monkeypatch.setattr(actor_module, "log_perf_data", lambda *_args, **_kwargs: None) + monkeypatch.setattr(actor_module.tracking_utils, "flush_metrics", lambda *_args, **_kwargs: None) + monkeypatch.setattr(actor_module, "compute_rollout_step", lambda *_args, **_kwargs: 0) + monkeypatch.setattr(actor_module, "get_gloo_group", lambda: None) + + def all_gather_object(output, value, **_kwargs): + output[:] = [value] + + monkeypatch.setattr(actor_module.dist, "all_gather_object", all_gather_object) + monkeypatch.setattr(actor_module.dist, "barrier", lambda **_kwargs: None) + + actor.train_hybrid(rollout_id=0) + + assert [event[1] for event in events if isinstance(event, tuple) and event[0] == "fetch"] == (expected_fetch_sizes) + assert sum(isinstance(event, tuple) and event[0] == "forward" for event in events) == expected_forward_count + assert sum(isinstance(event, tuple) and event[0] == "restore" for event in events) == int(pipeline_enabled) + assert events.count("advantages") == 1 + assert events.count("optimizer") == 1 + assert events.count("update_weights") == 1 diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py new file mode 100644 index 000000000..7bfd36b3d --- /dev/null +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -0,0 +1,684 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import importlib.util +import json +import sys +from pathlib import Path + +import pytest + + +SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "tools" / "analyze_hybrid_pipeline_benchmark.py" +SPEC = importlib.util.spec_from_file_location("analyze_hybrid_pipeline_benchmark", SCRIPT_PATH) +analyzer = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +sys.modules[SPEC.name] = analyzer +SPEC.loader.exec_module(analyzer) + +FINGERPRINT_0 = "00000000000000000000000000000001" +FINGERPRINT_1 = "00000000000000000000000000000002" +FULL_FINGERPRINT = "00000000000000000000000000000003" + + +def _event( + event, + monotonic_ns, + *, + role, + rollout_id=4, + chunk_index=None, + sample_count=None, + fingerprint=None, + hostname="test-host", + pid=None, + global_rank=None, + **details, +): + row = { + "event": event, + "monotonic_ns": monotonic_ns, + "rollout_id": rollout_id, + "chunk_index": chunk_index, + "sample_count": sample_count, + "total_tokens": sample_count * 10 if sample_count is not None else None, + "response_tokens": sample_count * 4 if sample_count is not None else None, + "multimodal_tensor_bytes": sample_count * 100 if sample_count is not None else None, + "role": role, + "hostname": hostname, + "pid": (10 if role == "rollout" else 20) if pid is None else pid, + "global_rank": (-1 if role == "rollout" else 0) if global_rank is None else global_rank, + "cuda_visible_devices": "0,1,2,3", + "cuda_max_allocated_bytes": 1024, + "cuda_max_reserved_bytes": 2048, + "global_indexes_fingerprint": fingerprint, + } + row.update(details) + return row + + +def _write_run( + tmp_path, + *, + name="run", + pipeline_enabled=True, + hostname="test-host", + missing_event=None, + nonfinite=False, + producer_put_count=2, + producer_last_start_ns=700, +): + run_dir = tmp_path / name + timeline = run_dir / "timeline" + timeline.mkdir(parents=True) + (run_dir / "run_manifest.json").write_text( + json.dumps( + { + "condition": "experiment" if pipeline_enabled else "baseline", + "order": "B1" if pipeline_enabled else "A1", + "seed": 7, + "rollout_seed": 7, + "num_rollout": 20, + "max_staleness": 2, + "global_batch_size": 4, + "rollout_batch_size": 2, + "n_samples_per_prompt": 2, + "num_iters_per_train_update": 2, + "hybrid_pipeline_forward": int(pipeline_enabled), + "hybrid_pipeline_trace_dir": str(timeline), + "hybrid_pipeline_fetch_timeout_s": 600, + "git_commit": "a" * 40, + "git_branch": "perf/task21", + "git_status_porcelain": "", + "image_archive_sha256": "b" * 64, + "image_manifest_digest": "sha256:" + "c" * 64, + "image_id": "sha256:" + "d" * 64, + "transferqueue_commit": "e" * 40, + "python": "3.12.3", + "entrypoint": "bash target.sh hybrid-async", + } + ), + encoding="utf-8", + ) + + producer_shapes = { + 1: [(4, FULL_FINGERPRINT)], + 2: [(2, FINGERPRINT_0), (2, FINGERPRINT_1)], + # Fingerprints are additive, so three value-1 digests preserve + # the same full digest as the actor's value-1 + value-2 chunks. + 3: [(1, FINGERPRINT_0), (1, FINGERPRINT_0), (2, FINGERPRINT_0)], + } + if producer_put_count not in producer_shapes: + raise ValueError(f"unsupported producer_put_count={producer_put_count}") + producer = [] + for chunk_index, (sample_count, _) in enumerate(producer_shapes[producer_put_count]): + start_ns = producer_last_start_ns if chunk_index == producer_put_count - 1 else 100 + chunk_index * 100 + producer.append( + _event( + "tq_put_start", + start_ns, + role="rollout", + chunk_index=chunk_index, + sample_count=sample_count, + hostname=hostname, + event_id=f"put-{chunk_index}", + is_last=chunk_index == producer_put_count - 1, + ) + ) + for chunk_index, (sample_count, fingerprint) in enumerate(producer_shapes[producer_put_count]): + done_ns = 800 if chunk_index == producer_put_count - 1 else 150 + chunk_index * 100 + producer.append( + _event( + "tq_put_done", + done_ns, + role="rollout", + chunk_index=chunk_index, + sample_count=sample_count, + fingerprint=fingerprint, + hostname=hostname, + event_id=f"put-{chunk_index}", + is_last=chunk_index == producer_put_count - 1, + ) + ) + producer.sort(key=lambda row: row["monotonic_ns"]) + if nonfinite: + producer[0]["total_tokens"] = float("nan") + + if pipeline_enabled: + actor = [ + _event("actor_restore_start", 210, role="actor", chunk_index=0, sample_count=4), + _event("actor_restore_end", 300, role="actor", chunk_index=0, sample_count=4), + _event("chunk_fetch_start", 310, role="actor", chunk_index=0, sample_count=2), + _event( + "chunk_fetch_end", + 350, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event( + "actor_forward_start", + 400, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event( + "actor_forward_end", + 600, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event("chunk_fetch_start", 810, role="actor", chunk_index=1, sample_count=2), + _event( + "chunk_fetch_end", + 850, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + _event( + "actor_forward_start", + 860, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + _event( + "actor_forward_end", + 1000, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + ] + else: + actor = [ + _event("chunk_fetch_start", 810, role="actor", chunk_index=0, sample_count=4), + _event( + "chunk_fetch_end", + 850, + role="actor", + chunk_index=0, + sample_count=4, + fingerprint=FULL_FINGERPRINT, + ), + _event("actor_restore_start", 860, role="actor", chunk_index=0, sample_count=4), + _event("actor_restore_end", 900, role="actor", chunk_index=0, sample_count=4), + _event( + "actor_forward_start", + 910, + role="actor", + chunk_index=0, + sample_count=4, + fingerprint=FULL_FINGERPRINT, + ), + _event( + "actor_forward_end", + 1000, + role="actor", + chunk_index=0, + sample_count=4, + fingerprint=FULL_FINGERPRINT, + ), + ] + actor += [ + _event("advantages_start", 1010, role="actor", sample_count=4), + _event("advantages_end", 1020, role="actor", sample_count=4), + _event("optimizer_start", 1030, role="actor", sample_count=4), + _event("optimizer_end", 1040, role="actor", sample_count=4), + ] + if missing_event is not None: + actor = [row for row in actor if row["event"] != missing_event] + + for filename, rows in (("rollout.jsonl", producer), ("actor.jsonl", actor)): + (timeline / filename).write_text( + "".join(json.dumps(row) + "\n" for row in rows), + encoding="utf-8", + ) + return run_dir + + +def test_complete_pipeline_trace_is_validated_and_summarized(tmp_path): + run_dir = _write_run(tmp_path) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.summary["validation"] == "passed" + assert analysis.summary["trace"]["producer_overlap_rollout_ratio"] == 1.0 + assert analysis.trace_rows[0]["actor_fetch_count"] == 2 + assert analysis.trace_rows[0]["actor_restore_count"] == 1 + assert analysis.trace_rows[0]["actor_forward_samples"] == 4 + assert (run_dir / "analysis" / "summary.json").is_file() + assert (run_dir / "analysis" / "trace_events.csv").is_file() + + +@pytest.mark.parametrize("producer_put_count", [1, 2, 3]) +def test_producer_put_grouping_is_independent_from_actor_chunks(tmp_path, producer_put_count): + run_dir = _write_run(tmp_path, producer_put_count=producer_put_count) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.trace_rows[0]["producer_put_count"] == producer_put_count + assert analysis.trace_rows[0]["actor_fetch_count"] == 2 + assert analysis.trace_rows[0]["actor_forward_count"] == 2 + + +def test_delayed_put_done_is_not_strict_producer_overlap(tmp_path): + run_dir = _write_run(tmp_path, producer_last_start_ns=150) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + row = analysis.trace_rows[0] + assert row["first_forward_before_last_put_start"] is False + assert row["producer_overlap_s"] == 0 + assert row["first_forward_before_last_put_done"] is True + assert row["transfer_overlap_s"] > 0 + + +def test_baseline_trace_preserves_one_full_fetch_and_no_overlap(tmp_path): + run_dir = _write_run(tmp_path, pipeline_enabled=False) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.trace_rows[0]["actor_fetch_count"] == 1 + assert analysis.trace_rows[0]["actor_restore_count"] == 1 + assert analysis.trace_rows[0]["first_forward_before_last_put_start"] is False + + +@pytest.mark.parametrize( + ("kwargs", "error"), + [ + ({"missing_event": "optimizer_end"}, "optimizer_end"), + ({"nonfinite": True}, "non-finite"), + ], +) +def test_invalid_trace_returns_nonzero_validation(tmp_path, kwargs, error, capsys): + run_dir = _write_run(tmp_path, **kwargs) + + status = analyzer.main( + [ + "--run-dir", + str(run_dir), + "--steady-windows", + "4-4", + "--expected-samples", + "4", + "--expected-actor-chunks", + "2", + "--validate-only", + ] + ) + + assert status == 2 + assert error in capsys.readouterr().err + + +def test_cross_hostname_trace_is_rejected(tmp_path): + run_dir = _write_run(tmp_path) + actor_path = run_dir / "timeline" / "actor.jsonl" + rows = [json.loads(line) for line in actor_path.read_text(encoding="utf-8").splitlines()] + rows[0]["hostname"] = "another-host" + actor_path.write_text( + "".join(json.dumps(row) + "\n" for row in rows), + encoding="utf-8", + ) + + with pytest.raises(analyzer.BenchmarkValidationError, match="multiple hostnames"): + analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + +def test_dirty_manifest_and_missing_reproducibility_artifacts_fail_closed(tmp_path): + dirty_run = _write_run(tmp_path, name="dirty") + manifest_path = dirty_run / "run_manifest.json" + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["git_status_porcelain"] = " M relax/file.py" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(analyzer.BenchmarkValidationError, match="dirty working tree"): + analyzer.analyze_run( + dirty_run, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + incomplete_run = _write_run(tmp_path, name="missing-artifacts") + with pytest.raises(analyzer.BenchmarkValidationError, match="reproducibility artifacts"): + analyzer.analyze_run( + incomplete_run, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + require_reproducibility_artifacts=True, + ) + + +def test_invalid_or_overlapping_windows_are_rejected(): + with pytest.raises(analyzer.BenchmarkValidationError, match="overlap"): + analyzer._parse_windows("4-8,8-10") + with pytest.raises(analyzer.BenchmarkValidationError, match="0 <= START <= END"): + analyzer._parse_windows("5-4") + + +def test_global_index_fingerprint_combination_ignores_chunk_grouping(): + producer = [ + {"global_indexes_fingerprint": FINGERPRINT_0}, + {"global_indexes_fingerprint": FINGERPRINT_1}, + ] + regrouped_fetch = [{"global_indexes_fingerprint": FULL_FINGERPRINT}] + + assert analyzer._combine_global_index_fingerprints( + producer, + context="producer", + ) == analyzer._combine_global_index_fingerprints( + regrouped_fetch, + context="fetch", + ) + + +def test_aggregate_throughput_uses_token_sum_over_time_sum(): + rows = [ + {"tag": "perf/step_time", "step": 4, "value": 3.0}, + {"tag": "perf/step_time", "step": 5, "value": 1.0}, + {"tag": "perf/step_token_per_s", "step": 4, "value": 100.0}, + {"tag": "perf/step_token_per_s", "step": 5, "value": 200.0}, + ] + + result = analyzer._aggregate_throughput( + rows, + "perf/step_token_per_s", + ((4, 5),), + ) + + assert result == 125.0 + assert analyzer._aggregate_samples_per_second(rows, ((4, 5),), samples_per_step=256) == 128.0 + + +def test_nvml_utilization_uses_registered_steady_step_wall_time(): + scalar_rows = [ + { + "tag": "perf/step_time", + "step": 4, + "value": 2.0, + "wall_time": 100.0, + } + ] + intervals = analyzer._steady_wall_time_intervals(scalar_rows, ((4, 4),)) + nvml_rows = [ + { + "gpu_index": 0.0, + "wall_time": wall_time, + "gpu_util_percent": utilization, + "memory_used_mib": memory, + "power_w": 100.0, + } + for wall_time, utilization, memory in ( + (97.0, 0.0, 1000.0), + (99.0, 80.0, 2000.0), + (101.0, 0.0, 3000.0), + ) + ] + + summary = analyzer._nvml_summary(nvml_rows, steady_intervals=intervals) + + assert intervals == [(98.0, 100.0)] + assert summary["per_gpu"]["0"]["steady_sample_count"] == 1 + assert summary["steady_mean_gpu_util_percent"] == 80.0 + assert summary["steady_idle_ratio_below_10_percent"] == 0.0 + # Peak memory deliberately remains a full-run safety metric. + assert summary["peak_memory_used_mib"] == 3000.0 + + +def _comparison_analysis( + condition, + seed, + throughput, + phase1, + *, + overlap=True, + accuracy=0.5, + peak_vram_mib=10_000, +): + metrics = { + "perf/step_token_per_s": {"aggregate": throughput}, + "perf/step_resp_token_per_s": {"aggregate": throughput / 2}, + "perf/wall_clock_samples_per_s": {"aggregate": 12.8}, + "perf/step_time": {"mean": 20.0, "p95": 21.0}, + "perf/hybrid_phase1_time": {"mean": phase1}, + "rollout/raw_reward": {"mean": accuracy}, + "rollout/truncated": {"mean": 0.0}, + "train/loss": {"mean": 1.0}, + "train/grad_norm": {"mean": 0.5}, + } + trace_rows = [ + { + "rollout_id": rollout_id, + "first_forward_before_last_put_start": overlap, + "first_forward_before_last_put_done": overlap, + "producer_overlap_s": 1.0 if overlap else 0.0, + "transfer_overlap_s": 1.0 if overlap else 0.0, + "actor_fetch_samples": 256, + "actor_total_tokens": 10_000 + seed, + "actor_response_tokens": 5_000 + seed, + "actor_multimodal_tensor_bytes": 1_000_000 + seed, + "producer_lead_at_first_forward": 1.0, + } + for rollout_id in range(4, 19) + ] + scalar_values = { + "perf/step_token_per_s": throughput, + "perf/step_resp_token_per_s": throughput / 2, + "perf/step_time": 20.0, + "perf/hybrid_phase1_time": phase1, + "rollout/raw_reward": accuracy, + "rollout/truncated": 0.0, + "train/loss": 1.0, + "train/grad_norm": 0.5, + } + scalar_rows = [ + {"tag": tag, "step": step, "value": value} for step in range(4, 19) for tag, value in scalar_values.items() + ] + return analyzer.RunAnalysis( + run_dir=Path(f"/{condition}-{seed}"), + manifest={ + "condition": condition, + "order": ("A" if condition == "baseline" else "B") + str(seed), + "seed": seed, + "rollout_seed": seed, + "num_rollout": 20, + "max_staleness": 2, + "global_batch_size": 256, + "rollout_batch_size": 32, + "n_samples_per_prompt": 8, + "num_iters_per_train_update": 2, + "hybrid_pipeline_forward": condition == "experiment", + "hybrid_pipeline_fetch_timeout_s": 600, + "git_commit": "a" * 40, + "git_branch": "perf/task21", + "image_archive_sha256": "b" * 64, + "image_manifest_digest": "sha256:" + "c" * 64, + "image_id": "sha256:" + "d" * 64, + "transferqueue_commit": "e" * 40, + "python": "3.12.3", + "entrypoint": "bash target.sh hybrid-async", + }, + trace_rows=trace_rows, + actor_rank_rows=[], + scalar_rows=scalar_rows, + nvml_rows=[], + summary={ + "metrics": metrics, + "nvml": { + "gpu_count": 8, + "per_gpu": { + str(gpu_index): { + "steady_sample_count": 10, + "steady_mean_gpu_util_percent": 80.0, + "steady_idle_ratio_below_10_percent": 0.05, + } + for gpu_index in range(8) + }, + "steady_mean_gpu_util_percent": 80.0, + "steady_idle_ratio_below_10_percent": 0.05, + "peak_memory_used_mib": peak_vram_mib, + }, + }, + ) + + +def test_preregistered_targets_pass_and_fail_deterministically(): + passing = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + + comparison = analyzer._build_comparison( + passing, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + assert comparison["token_throughput_geomean_speedup"] == pytest.approx(0.06) + assert comparison["hybrid_phase1_geomean_reduction"] == pytest.approx(0.2) + assert comparison["experiment_steady_producer_overlap_ratio"] == 1.0 + assert len(comparison["window_speedups"]) == 6 + + failing = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 104, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 106.08, 8.16), + ] + with pytest.raises(analyzer.BenchmarkValidationError, match="below 5%"): + analyzer._build_comparison( + failing, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + +def test_comparison_rejects_mixed_candidate_commits(): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + analyses[1].manifest["git_commit"] = "f" * 40 + + with pytest.raises(analyzer.BenchmarkValidationError, match="git_commit"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + +def test_comparison_plot_bundle_is_generated(tmp_path): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + comparison = analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + generated = analyzer._plot_comparison( + analyses, + comparison, + tmp_path, + ((4, 8), (9, 13), (14, 18)), + ) + + assert {Path(path).name for path in generated} == { + "task21_correctness_quality.png", + "task21_gpu_util_vram.png", + "task21_phase1_overlap.png", + "task21_step_throughput.png", + "task21_window_summary.png", + } + assert all(Path(path).stat().st_size > 0 for path in generated) + + +@pytest.mark.parametrize( + ("guardrail", "error"), + [ + ("overlap", "at least 80% steady producer overlap"), + ("accuracy", "accuracy drop exceeds 3 percentage points"), + ("truncation", "truncation-rate increase exceeds 2 percentage points"), + ("staleness", "average producer-lead increase exceeds 0.25"), + ("staleness_max", "observed producer lead exceeds configured max_staleness=2"), + ("vram", "peak VRAM increased"), + ("workload", "actor_total_tokens changed by more than 1%"), + ], +) +def test_preregistered_guardrails_fail_closed(guardrail, error): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + experiment = analyses[1] + if guardrail == "overlap": + for row in experiment.trace_rows: + row["first_forward_before_last_put_start"] = False + elif guardrail == "accuracy": + experiment.summary["metrics"]["rollout/raw_reward"]["mean"] = 0.46 + elif guardrail == "truncation": + experiment.summary["metrics"]["rollout/truncated"]["mean"] = 0.03 + elif guardrail == "staleness": + for row in experiment.trace_rows: + row["producer_lead_at_first_forward"] = 1.3 + elif guardrail == "staleness_max": + experiment.trace_rows[0]["producer_lead_at_first_forward"] = 3.0 + elif guardrail == "vram": + experiment.summary["nvml"]["peak_memory_used_mib"] = 12_000 + elif guardrail == "workload": + for row in experiment.trace_rows: + row["actor_total_tokens"] *= 2 + + with pytest.raises(analyzer.BenchmarkValidationError, match=error): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) diff --git a/tests/utils/test_arguments_opd_teacher_colocate.py b/tests/utils/test_arguments_opd_teacher_colocate.py index b87721536..2b1f7c585 100644 --- a/tests/utils/test_arguments_opd_teacher_colocate.py +++ b/tests/utils/test_arguments_opd_teacher_colocate.py @@ -73,6 +73,211 @@ def test_sft_invalid_multimodal_strategy_option(arguments_module, argv, expected assert args.sft_invalid_multimodal_strategy == expected +def test_hybrid_pipeline_options_default_off(arguments_module): + arguments_module.RouterArgs = SimpleNamespace(add_cli_args=lambda parser, **_kwargs: parser) + parser = argparse.ArgumentParser() + arguments_module.get_slime_extra_args_provider()(parser) + + args = parser.parse_args([]) + + assert args.hybrid_pipeline_forward is False + assert args.hybrid_pipeline_trace_dir is None + assert args.hybrid_pipeline_fetch_timeout_s == 600.0 + + +def _hybrid_pipeline_args() -> SimpleNamespace: + return SimpleNamespace( + hybrid=True, + hybrid_pipeline_forward=True, + hybrid_pipeline_trace_dir="/tmp/trace", + hybrid_pipeline_fetch_timeout_s=600.0, + use_dynamic_batch_size=True, + multimodal_keys={"image": "image"}, + advantage_estimator="grpo", + compute_advantages_and_returns=True, + enable_weights_backuper=True, + kl_coef=0.0, + use_kl_loss=False, + use_opd=False, + keep_old_actor=False, + true_on_policy_mode=False, + use_rollout_logprobs=False, + get_mismatch_metrics=False, + use_routing_replay=False, + use_rollout_routing_replay=False, + use_agentic_rollout=False, + partial_rollout=False, + use_dynamic_global_batch_size=False, + use_critic=False, + per_rank_fetch=False, + attention_dropout=0.0, + hidden_dropout=0.0, + tensor_model_parallel_size=2, + pipeline_model_parallel_size=1, + context_parallel_size=2, + expert_model_parallel_size=1, + expert_tensor_parallel_size=1, + offload_train=False, + offload_rollout=False, + num_iters_per_train_update=2, + rollout_batch_size=32, + n_samples_per_prompt=8, + global_batch_size=256, + ) + + +def test_hybrid_pipeline_supported_configuration_is_accepted(arguments_module): + arguments_module._validate_hybrid_pipeline_args(_hybrid_pipeline_args()) + + +@pytest.mark.parametrize( + ("field", "value", "error"), + [ + ("hybrid", False, "only supported with --hybrid"), + ("use_dynamic_batch_size", False, "missing --use-dynamic-batch-size"), + ("multimodal_keys", None, "missing --multimodal-keys"), + ("enable_weights_backuper", False, "--disable-weights-backuper"), + ("kl_coef", 0.1, "--kl-coef"), + ("use_opd", True, "--use-opd"), + ("true_on_policy_mode", True, "--true-on-policy-mode"), + ("use_rollout_routing_replay", True, "--use-rollout-routing-replay"), + ("use_agentic_rollout", True, "--use-agentic-rollout"), + ("partial_rollout", True, "--partial-rollout"), + ("attention_dropout", 0.1, "--attention-dropout"), + ("tensor_model_parallel_size", 1, "--tensor-model-parallel-size"), + ("pipeline_model_parallel_size", 2, "--pipeline-model-parallel-size"), + ("context_parallel_size", 1, "--context-parallel-size"), + ("expert_model_parallel_size", 2, "--expert-model-parallel-size"), + ("expert_tensor_parallel_size", 2, "--expert-tensor-parallel-size"), + ("offload_train", True, "--offload-train"), + ("offload_rollout", True, "--offload-rollout"), + ("num_iters_per_train_update", 1, "must be >= 2"), + ("global_batch_size", 128, "exactly one optimizer mini per rollout"), + ], +) +def test_hybrid_pipeline_rejects_unsupported_configuration( + arguments_module, + field, + value, + error, +): + args = _hybrid_pipeline_args() + setattr(args, field, value) + + with pytest.raises(ValueError, match=error): + arguments_module._validate_hybrid_pipeline_args(args) + + +@pytest.mark.parametrize("timeout", [0, -1, float("nan"), float("inf")]) +def test_hybrid_pipeline_timeout_must_be_positive_and_finite(arguments_module, timeout): + args = _hybrid_pipeline_args() + args.hybrid_pipeline_fetch_timeout_s = timeout + + with pytest.raises(ValueError, match="finite value greater than 0"): + arguments_module._validate_hybrid_pipeline_args(args) + + +def _install_transfer_queue_contract(monkeypatch, *, global_indexes=True, custom_meta=True): + transfer_queue = ModuleType("transfer_queue") + + class BatchMeta: + __slots__ = ("global_indexes",) if global_indexes else ("partition_ids",) + + if custom_meta: + + class TransferQueueClient: + async def async_put(self, data, partition_id=None, custom_meta=None, is_last=False): + return None + + else: + + class TransferQueueClient: + async def async_put(self, data, partition_id=None, is_last=False): + return None + + transfer_queue.BatchMeta = BatchMeta + transfer_queue.TransferQueueClient = TransferQueueClient + monkeypatch.setitem(sys.modules, "transfer_queue", transfer_queue) + + +def test_hybrid_pipeline_transfer_queue_contract_is_checked_before_runtime( + arguments_module, + monkeypatch, +): + _install_transfer_queue_contract(monkeypatch) + monkeypatch.setattr("importlib.metadata.version", lambda _package: "0.1.10.dev0") + + arguments_module.check_hybrid_pipeline_transfer_queue_contract() + + +@pytest.mark.parametrize( + ("kwargs", "error"), + [ + ({"global_indexes": False}, "global_indexes"), + ({"custom_meta": False}, "custom_meta"), + ], +) +def test_hybrid_pipeline_transfer_queue_contract_fails_fast( + arguments_module, + monkeypatch, + kwargs, + error, +): + _install_transfer_queue_contract(monkeypatch, **kwargs) + monkeypatch.setattr("importlib.metadata.version", lambda _package: "0.1.10.dev0") + + with pytest.raises(RuntimeError, match=error): + arguments_module.check_hybrid_pipeline_transfer_queue_contract() + + +@pytest.mark.parametrize( + ("args", "expected"), + [ + ( + SimpleNamespace( + hybrid_pipeline_forward=False, + hybrid_pipeline_trace_dir="/tmp/timeline", + fully_async=True, + ), + "hybrid", + ), + ( + SimpleNamespace( + hybrid_pipeline_forward=False, + hybrid_pipeline_trace_dir=None, + fully_async=True, + ), + "version", + ), + ( + SimpleNamespace( + hybrid_pipeline_forward=False, + hybrid_pipeline_trace_dir=None, + fully_async=False, + ), + None, + ), + ], +) +def test_transfer_queue_runtime_selects_trace_contract_before_side_effects( + arguments_module, + monkeypatch, + args, + expected, +): + calls = [] + monkeypatch.setattr( + arguments_module, + "check_hybrid_pipeline_transfer_queue_contract", + lambda: calls.append("hybrid"), + ) + monkeypatch.setattr(arguments_module, "check_transfer_queue_version", lambda: calls.append("version")) + + arguments_module.check_transfer_queue_runtime(args) + + assert calls == ([] if expected is None else [expected]) + + def _opd_args() -> SimpleNamespace: return SimpleNamespace( loss_type="grpo", diff --git a/tests/utils/test_hybrid_forward_pipeline.py b/tests/utils/test_hybrid_forward_pipeline.py new file mode 100644 index 000000000..ec382265d --- /dev/null +++ b/tests/utils/test_hybrid_forward_pipeline.py @@ -0,0 +1,140 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import pytest + +from relax.utils.training.hybrid_forward_pipeline import ( + execute_hybrid_forward_mini, + fetch_exact_chunk_with_timeout, +) + + +def test_first_chunk_forward_runs_before_later_chunk_is_ready(): + events = [] + chunk_one_ready = False + + def restore_actor(batch_index): + events.append(("restore", batch_index)) + + def fetch_chunk(batch_index): + nonlocal chunk_one_ready + if batch_index == 1: + assert chunk_one_ready, "chunk 1 was fetched before chunk 0 forward completed" + events.append(("fetch", batch_index)) + return {"total_lengths": [batch_index + 1]}, [100 + batch_index] + + def forward_chunk(batch, batch_index, global_indexes): + nonlocal chunk_one_ready + events.append(("forward", batch_index, global_indexes)) + if batch_index == 0: + chunk_one_ready = True + + chunks = execute_hybrid_forward_mini( + chunks_per_mini=2, + batch_index_for_chunk=lambda chunk_index: chunk_index, + restore_actor=restore_actor, + fetch_chunk=fetch_chunk, + forward_chunk=forward_chunk, + ) + + assert events == [ + ("restore", 0), + ("fetch", 0), + ("forward", 0, [100]), + ("fetch", 1), + ("forward", 1, [101]), + ] + assert chunks == [ + ({"total_lengths": [1]}, [100]), + ({"total_lengths": [2]}, [101]), + ] + + +def test_restore_occurs_once_per_optimizer_mini(): + restore_calls = [] + + for mini_index in range(3): + execute_hybrid_forward_mini( + chunks_per_mini=2, + batch_index_for_chunk=lambda chunk_index, mini=mini_index: mini * 2 + chunk_index, + restore_actor=restore_calls.append, + fetch_chunk=lambda batch_index: ({"total_lengths": [1]}, [batch_index]), + forward_chunk=lambda batch, batch_index, global_indexes: None, + ) + + assert restore_calls == [0, 2, 4] + + +def test_invalid_chunk_count_fails_before_restore(): + restore_calls = [] + + with pytest.raises(ValueError, match="chunks_per_mini must be positive"): + execute_hybrid_forward_mini( + chunks_per_mini=0, + batch_index_for_chunk=lambda chunk_index: chunk_index, + restore_actor=restore_calls.append, + fetch_chunk=lambda batch_index: ({}, []), + forward_chunk=lambda batch, batch_index, global_indexes: None, + ) + + assert restore_calls == [] + + +def test_exact_fetch_retries_then_returns_complete_chunk(): + attempts = iter( + [ + (None, None), + ({"total_lengths": [1, 2]}, "metadata"), + ] + ) + now = [0.0] + + batch, metadata, elapsed = fetch_exact_chunk_with_timeout( + fetch_once=lambda: next(attempts), + expected_samples=2, + timeout_s=5, + error_context="rollout_id=7, mini_index=0, chunk_index=1", + clock=lambda: now[0], + sleep=lambda seconds: now.__setitem__(0, now[0] + seconds), + ) + + assert batch == {"total_lengths": [1, 2]} + assert metadata == "metadata" + assert elapsed == 0.1 + + +def test_exact_fetch_rejects_underfilled_chunk(): + with pytest.raises(RuntimeError, match="expected=2, last_returned=1"): + fetch_exact_chunk_with_timeout( + fetch_once=lambda: ({"total_lengths": [1]}, "metadata"), + expected_samples=2, + timeout_s=5, + error_context="rollout_id=7, mini_index=0, chunk_index=1", + ) + + +def test_exact_fetch_timeout_contains_actionable_context(): + now = [0.0] + + with pytest.raises( + TimeoutError, + match="rollout_id=7, mini_index=0, chunk_index=1, expected=2, last_returned=0", + ): + fetch_exact_chunk_with_timeout( + fetch_once=lambda: (None, None), + expected_samples=2, + timeout_s=0.2, + error_context="rollout_id=7, mini_index=0, chunk_index=1", + clock=lambda: now[0], + sleep=lambda seconds: now.__setitem__(0, now[0] + seconds), + ) + + +@pytest.mark.parametrize("timeout_s", [float("nan"), float("inf"), 0, -1]) +def test_exact_fetch_rejects_invalid_timeout(timeout_s): + with pytest.raises(ValueError, match="positive and finite"): + fetch_exact_chunk_with_timeout( + fetch_once=lambda: (None, None), + expected_samples=2, + timeout_s=timeout_s, + error_context="rollout_id=7", + ) diff --git a/tests/utils/test_hybrid_pipeline_trace.py b/tests/utils/test_hybrid_pipeline_trace.py new file mode 100644 index 000000000..9766ef60c --- /dev/null +++ b/tests/utils/test_hybrid_pipeline_trace.py @@ -0,0 +1,105 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import json +from types import SimpleNamespace + +import pytest +import torch + +from relax.utils.training import hybrid_pipeline_trace + + +def test_global_index_fingerprint_is_order_independent_and_composable(): + combined = hybrid_pipeline_trace.fingerprint_global_indexes([11, 12, 11]) + reordered = hybrid_pipeline_trace.fingerprint_global_indexes([12, 11, 11]) + first = hybrid_pipeline_trace.fingerprint_global_indexes([11]) + second = hybrid_pipeline_trace.fingerprint_global_indexes([12, 11]) + + assert combined == reordered + assert combined != hybrid_pipeline_trace.fingerprint_global_indexes([11, 12]) + assert int(combined, 16) == (int(first, 16) + int(second, 16)) % (1 << 128) + + +def test_trace_disabled_creates_no_files(tmp_path): + args = SimpleNamespace(hybrid_pipeline_trace_dir=None) + + record = hybrid_pipeline_trace.emit_hybrid_pipeline_event( + args, + "actor_forward_start", + rollout_id=1, + role="actor", + ) + + assert record is None + assert list(tmp_path.iterdir()) == [] + + +def test_trace_records_metrics_without_sample_content(tmp_path, monkeypatch): + hybrid_pipeline_trace._close_writers() + monkeypatch.setattr(hybrid_pipeline_trace.socket, "gethostname", lambda: "test-host") + monkeypatch.setattr(hybrid_pipeline_trace.os, "getpid", lambda: 123) + monkeypatch.setattr(hybrid_pipeline_trace, "_global_rank", lambda: 4) + monkeypatch.setattr(hybrid_pipeline_trace, "_cuda_memory_peaks", lambda: (1024, 2048)) + args = SimpleNamespace(hybrid_pipeline_trace_dir=str(tmp_path)) + batch = { + "total_lengths": [10, 20], + "response_lengths": [4, 5], + "multimodal_train_inputs": [ + {"pixel_values": torch.zeros((2, 3), dtype=torch.float32)}, + None, + ], + "prompt": "SECRET_SAMPLE_CONTENT", + } + + record = hybrid_pipeline_trace.emit_hybrid_pipeline_event( + args, + "chunk_fetch_end", + rollout_id=7, + role="actor", + chunk_index=1, + batch=batch, + global_indexes=[11, 12], + monotonic_ns=99, + details={"mini_index": 0, "is_last": False}, + ) + + assert record["sample_count"] == 2 + assert record["total_tokens"] == 30 + assert record["response_tokens"] == 9 + assert record["multimodal_tensor_bytes"] == 24 + assert record["global_indexes_fingerprint"] == hybrid_pipeline_trace.fingerprint_global_indexes([11, 12]) + assert record["cuda_max_allocated_bytes"] == 1024 + assert record["cuda_max_reserved_bytes"] == 2048 + + trace_files = list(tmp_path.glob("*.jsonl")) + assert [path.name for path in trace_files] == ["hybrid-pipeline-actor-test-host-pid123-rank4.jsonl"] + payload = trace_files[0].read_text(encoding="utf-8") + assert "SECRET_SAMPLE_CONTENT" not in payload + assert json.loads(payload)["event"] == "chunk_fetch_end" + hybrid_pipeline_trace._close_writers() + + +def test_trace_rejects_non_scalar_details(tmp_path): + args = SimpleNamespace(hybrid_pipeline_trace_dir=str(tmp_path)) + + with pytest.raises(TypeError, match="trace details must be scalar"): + hybrid_pipeline_trace.emit_hybrid_pipeline_event( + args, + "event", + rollout_id=1, + role="actor", + details={"sample": {"must": "not be serialized"}}, + ) + + +def test_trace_rejects_reserved_detail_fields(tmp_path): + args = SimpleNamespace(hybrid_pipeline_trace_dir=str(tmp_path)) + + with pytest.raises(ValueError, match="cannot replace record fields"): + hybrid_pipeline_trace.emit_hybrid_pipeline_event( + args, + "event", + rollout_id=1, + role="actor", + details={"event": "overwritten"}, + ) From aebc19347a580f90520678212f425099347b86fe Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Wed, 29 Jul 2026 16:10:36 +0800 Subject: [PATCH 02/21] test(benchmark): keep plotting checks portable Exercise the five-file plot contract without forcing the optional plotting stack into the minimal CI environment, and fail clearly when real benchmark plotting is requested without matplotlib. Constraint: GitHub's core Python test image intentionally omits matplotlib. Rejected: Add matplotlib to core test dependencies | it would expand the default environment for a benchmark-only output path. Confidence: high Scope-risk: narrow Directive: Keep requested benchmark artifacts fail-fast; never silently return an empty plot bundle. Tested: 101 passed and 3 skipped on host; 104 passed in the fixed Relax image; pre-commit run --all-files. Not-tested: 8-GPU multimodal smoke and paired performance A/B remain pending on the pinned dataset and idle topology. --- .../analyze_hybrid_pipeline_benchmark.py | 7 ++- .../test_analyze_hybrid_pipeline_benchmark.py | 57 ++++++++++++++++++- 2 files changed, 61 insertions(+), 3 deletions(-) diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py index b4d9609ae..dece35bdf 100644 --- a/scripts/tools/analyze_hybrid_pipeline_benchmark.py +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -1338,8 +1338,11 @@ def _plot_comparison( matplotlib.use("Agg") import matplotlib.pyplot as plt - except ImportError: - return [] + except ImportError as exc: + raise BenchmarkValidationError( + "comparison plot generation requires the optional dependency " + "'matplotlib'; install it in the benchmark environment" + ) from exc output_dir.mkdir(parents=True, exist_ok=True) generated = [] diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py index 7bfd36b3d..49c5a96f2 100644 --- a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -3,7 +3,9 @@ import importlib.util import json import sys +import types from pathlib import Path +from unittest.mock import MagicMock import pytest @@ -20,6 +22,46 @@ FULL_FINGERPRINT = "00000000000000000000000000000003" +def _install_fake_matplotlib(monkeypatch): + """Install the plotting API subset used by the analyzer for minimal CI.""" + matplotlib = types.ModuleType("matplotlib") + matplotlib.__path__ = [] + matplotlib.use = MagicMock() + pyplot = types.ModuleType("matplotlib.pyplot") + pyplot.tight_layout = MagicMock() + pyplot.close = MagicMock() + pyplot.figure = MagicMock() + pyplot.plot = MagicMock() + pyplot.xlabel = MagicMock() + pyplot.ylabel = MagicMock() + pyplot.legend = MagicMock() + pyplot.title = MagicMock() + pyplot.savefig = MagicMock(side_effect=lambda path, **_kwargs: Path(path).write_bytes(b"deterministic plot")) + + def axes(count): + return [MagicMock() for _ in range(count)] + + matrix_axes = {(row, column): MagicMock() for row in range(2) for column in range(2)} + for axis in matrix_axes.values(): + axis.get_legend_handles_labels.return_value = ([], []) + + class AxesGrid: + def __getitem__(self, key): + return matrix_axes[key] + + pyplot.subplots = MagicMock( + side_effect=[ + (MagicMock(), axes(2)), + (MagicMock(), axes(3)), + (MagicMock(), AxesGrid()), + (MagicMock(), axes(2)), + ] + ) + matplotlib.pyplot = pyplot + monkeypatch.setitem(sys.modules, "matplotlib", matplotlib) + monkeypatch.setitem(sys.modules, "matplotlib.pyplot", pyplot) + + def _event( event, monotonic_ns, @@ -608,7 +650,7 @@ def test_comparison_rejects_mixed_candidate_commits(): ) -def test_comparison_plot_bundle_is_generated(tmp_path): +def test_comparison_plot_bundle_is_generated(tmp_path, monkeypatch): analyses = [ _comparison_analysis("baseline", 1, 100, 10), _comparison_analysis("experiment", 1, 106, 8), @@ -621,6 +663,9 @@ def test_comparison_plot_bundle_is_generated(tmp_path): enforce_targets=True, ) + if importlib.util.find_spec("matplotlib") is None: + _install_fake_matplotlib(monkeypatch) + generated = analyzer._plot_comparison( analyses, comparison, @@ -638,6 +683,16 @@ def test_comparison_plot_bundle_is_generated(tmp_path): assert all(Path(path).stat().st_size > 0 for path in generated) +def test_comparison_plot_bundle_fails_fast_without_matplotlib(tmp_path, monkeypatch): + monkeypatch.setitem(sys.modules, "matplotlib", None) + + with pytest.raises( + analyzer.BenchmarkValidationError, + match="plot generation requires the optional dependency 'matplotlib'", + ): + analyzer._plot_comparison([], {}, tmp_path, ()) + + @pytest.mark.parametrize( ("guardrail", "error"), [ From 1350cfc8b9be78f701448a857cce8ce7a4b729ee Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Wed, 29 Jul 2026 17:03:52 +0800 Subject: [PATCH 03/21] fix(benchmark): preserve analyzer CLI operability Argparse interpolates percent signs while rendering help, so keep the preregistered target text literal and lock the public help path with a regression test. Constraint: Benchmark operators must be able to inspect the exact acceptance targets before launching expensive 8-GPU runs. Rejected: Removing percentages from help | That would hide the preregistered thresholds rather than fixing the CLI contract. Confidence: high Scope-risk: narrow Directive: Keep argparse help strings safe for percent interpolation and cover --help as an executable interface. Tested: 26 analyzer tests; 96 related host tests with 3 Megatron-only skips; 99 related tests in the fixed Relax image; targeted pre-commit hooks; direct --help smoke. Not-tested: 8-GPU Hybrid-async end-to-end smoke remains pending idle capacity. --- scripts/tools/analyze_hybrid_pipeline_benchmark.py | 2 +- .../scripts/test_analyze_hybrid_pipeline_benchmark.py | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py index dece35bdf..ae8283094 100644 --- a/scripts/tools/analyze_hybrid_pipeline_benchmark.py +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -1568,7 +1568,7 @@ def build_parser() -> argparse.ArgumentParser: parser.add_argument( "--enforce-targets", action="store_true", - help="Require the preregistered 5% throughput, 15% phase-1, and 80% overlap targets.", + help="Require the preregistered 5%% throughput, 15%% phase-1, and 80%% overlap targets.", ) parser.add_argument( "--no-plots", diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py index 49c5a96f2..50dbae952 100644 --- a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -22,6 +22,16 @@ FULL_FINGERPRINT = "00000000000000000000000000000003" +def test_parser_help_renders_percent_targets(capsys): + with pytest.raises(SystemExit) as exc_info: + analyzer.build_parser().parse_args(["--help"]) + + assert exc_info.value.code == 0 + output = capsys.readouterr().out + assert "--enforce-targets" in output + assert "5% throughput, 15% phase-1, and 80% overlap" in " ".join(output.split()) + + def _install_fake_matplotlib(monkeypatch): """Install the plotting API subset used by the analyzer for minimal CI.""" matplotlib = types.ModuleType("matplotlib") From f9ee04ee0d9cf5e4eafe6bb967b167af38cc3914 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Wed, 29 Jul 2026 17:36:27 +0800 Subject: [PATCH 04/21] fix(hybrid): preserve frozen rollout fields across chunk replay Use one shape-aware slicer for live-equivalent debug chunks so tensor, NumPy, tuple, and list fields stay aligned during off/on frozen-input parity. Constraint: Frozen replay must exercise the same 128-sample partitioning without duplicating unsliced per-sample arrays. Rejected: Converting every debug field to Python lists | That would change dtype/container semantics and hide replay mismatches. Confidence: high Scope-risk: narrow Directive: Route future debug-rollout partitioning through the shared shape-aware slicer and test new per-sample container types. Tested: 96 related host tests with 4 Megatron-only skips; 106 related tests in the fixed Relax image; targeted pre-commit hooks; py_compile; git diff --check. Not-tested: 8-GPU Hybrid-async smoke remains gated on all eight GPUs becoming idle. --- relax/backends/megatron/actor.py | 37 ++++++------------- .../test_hybrid_pipeline_actor_wiring.py | 29 +++++++++++++++ 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/relax/backends/megatron/actor.py b/relax/backends/megatron/actor.py index 032e39189..21319dde3 100644 --- a/relax/backends/megatron/actor.py +++ b/relax/backends/megatron/actor.py @@ -136,14 +136,12 @@ def _slice(value): return value if isinstance(value, (list, tuple)) and len(value) == num_samples: return value[start:end] - ndim = getattr(value, "ndim", None) - size = getattr(value, "size", None) - if ndim and ndim > 0 and callable(size): - try: - if int(size(0)) == num_samples: - return value[start:end] - except (TypeError, ValueError, IndexError): - pass + shape = getattr(value, "shape", None) + try: + if shape is not None and len(shape) > 0 and int(shape[0]) == num_samples: + return value[start:end] + except (TypeError, ValueError, IndexError): + pass return value return {k: _slice(v) for k, v in rollout_data.items()} @@ -1366,27 +1364,14 @@ def fetch_once(): @staticmethod def _split_rollout_batch(rollout_data: RolloutBatch, num_chunks: int) -> List[RolloutBatch]: - """Split a merged rollout batch (dict of per-sample lists) into at most - ``num_chunks`` roughly equal sub-batches along the sample dimension. - - Keys whose value is not a per-sample list are copied into every chunk - unchanged. Used by the debug_train_only path to feed the collected- - sub-batch forward loop (one global batch per chunk). - """ + """Split a debug rollout into roughly equal per-sample chunks.""" num_samples = len(rollout_data["tokens"]) num_chunks = max(1, min(num_chunks, num_samples)) chunk_size = (num_samples + num_chunks - 1) // num_chunks - chunks: List[RolloutBatch] = [] - for start in range(0, num_samples, chunk_size): - end = min(start + chunk_size, num_samples) - chunk: RolloutBatch = {} - for key, value in rollout_data.items(): - if isinstance(value, list) and len(value) == num_samples: - chunk[key] = value[start:end] - else: - chunk[key] = value - chunks.append(chunk) - return chunks + return [ + _slice_rollout_batch(rollout_data, start, min(start + chunk_size, num_samples)) + for start in range(0, num_samples, chunk_size) + ] def _use_streaming_fwd(self) -> bool: """Whether ref / actor_fwd forward should stream via token-budget diff --git a/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py index 52b798ffc..3908198e1 100644 --- a/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py +++ b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py @@ -4,7 +4,9 @@ from contextlib import nullcontext from types import SimpleNamespace +import numpy as np import pytest +import torch try: @@ -65,6 +67,33 @@ def _rollout_batch(start: int, count: int) -> dict: } +def test_debug_rollout_chunking_slices_every_per_sample_container(): + assert actor_module is not None + shared = {"source": "frozen-rollout"} + rollout_data = { + "tokens": [[0], [1], [2], [3]], + "total_lengths": [1, 1, 1, 1], + "tensor_field": torch.arange(8).reshape(4, 2), + "array_field": np.arange(12).reshape(4, 3), + "tuple_field": ("a", "b", "c", "d"), + "shared": shared, + } + + chunks = actor_module.MegatronTrainRayActor._split_rollout_batch(rollout_data, 2) + + assert len(chunks) == 2 + assert chunks[0]["tokens"] == [[0], [1]] + assert chunks[1]["tokens"] == [[2], [3]] + assert torch.equal(chunks[0]["tensor_field"], torch.tensor([[0, 1], [2, 3]])) + assert torch.equal(chunks[1]["tensor_field"], torch.tensor([[4, 5], [6, 7]])) + np.testing.assert_array_equal(chunks[0]["array_field"], np.arange(6).reshape(2, 3)) + np.testing.assert_array_equal(chunks[1]["array_field"], np.arange(6, 12).reshape(2, 3)) + assert chunks[0]["tuple_field"] == ("a", "b") + assert chunks[1]["tuple_field"] == ("c", "d") + assert chunks[0]["shared"] is shared + assert chunks[1]["shared"] is shared + + @pytest.mark.parametrize( ("pipeline_enabled", "expected_fetch_sizes", "expected_forward_count"), [ From 98ef5268867067218994cfa6b866d51004fecf94 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Wed, 29 Jul 2026 22:13:37 +0800 Subject: [PATCH 05/21] feat(training): allow reproducible benchmark variants without recipe edits Expose validated, default-preserving model, resource, length, checkpoint, and output controls so paired multimodal runs can be reproduced without temporary launcher rewrites. Constraint: Preserve the existing Qwen3.5 8-GPU recipe by default. Rejected: Maintain external text-replacement overlays | brittle across launcher changes and cannot reliably propagate MetricsService output paths. Confidence: high Scope-risk: moderate Directive: Keep benchmark controls opt-in and compare only identical workload configurations. Tested: bash -n; shellcheck; 99 related pytest cases passed with 4 environment skips. Not-tested: Registered 8-GPU Qwen3.5 E2E is pending an uncontended 8-GPU window. --- docs/en/guide/hybrid-training.md | 45 ++++ docs/zh/guide/hybrid-training.md | 43 ++++ ...n-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh | 198 +++++++++++++++--- .../test_multimodal_hybrid_async_launcher.py | 152 ++++++++++++++ 4 files changed, 412 insertions(+), 26 deletions(-) create mode 100644 tests/scripts/test_multimodal_hybrid_async_launcher.py diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index deebc116e..145be85b3 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -239,6 +239,51 @@ bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ hybrid-async ``` +The same launcher provides default-preserving experiment overrides so a smoke +run or paired benchmark does not require editing the script: + +| Environment variable | Default | Purpose | +| --- | --- | --- | +| `MODEL_CONFIG_FILE` | `qwen35-9B.sh` | Select the model-parallel configuration script | +| `MODEL_NAME` / `MODEL_RUN_NAME` | `Qwen3.5-9B` / `qwen35-9b` | Select the checkpoint subdirectory and log prefix | +| `MODEL_CHECKPOINT_DIR` / `REFERENCE_CHECKPOINT_DIR` | `${MODEL_DIR}/${MODEL_NAME}` | Pin actor and reference inputs explicitly | +| `CHECKPOINT_SAVE` | `1` | Set to `0` to omit all `--save*` arguments and use separate rollout/TensorBoard outputs | +| `ROLLOUT_RESULT_DIR` / `TENSORBOARD_DIR` | No override while saving; `${EXP_DIR}/rollout_result` and `${EXP_DIR}/tensorboard_log` without saving | Preserve raw results and scalars in no-save runs; `TENSORBOARD_DIR` is exported for MetricsService | +| `ROLLOUT_MAX_RESPONSE_LEN` / `ROLLOUT_MAX_PROMPT_LEN` / `ROLLOUT_MAX_CONTEXT_LEN` | `10240` / `2048` / `12288` | Pin generation and context limits | +| `ACTOR_MAX_TOKENS_PER_GPU` | `12288` | Bound dynamic-batch actor microbatch tokens | +| `HYBRID_ACTOR_GPUS` / `HYBRID_ROLLOUT_GPUS` | `4` / `4` | Build the Hybrid placement resource | +| `ROLLOUT_NUM_GPUS_PER_ENGINE` | `2` | Set each SGLang engine's tensor-parallel width | +| `SGLANG_MEM_FRACTION_STATIC` | `0.8` | Set SGLang's static memory fraction | + +Before submitting a Ray Job, the launcher validates positive integers, context +capacity, rollout-GPU divisibility, and that actor GPU counts are multiples of +the current TP=2, CP=2 topology. Invalid combinations fail before Ray workers +start instead of silently degrading. + +For example, this is a no-save Qwen3-VL-8B functional smoke for a constrained +machine. It is not an 8-GPU Qwen3.5 performance result: + +```bash +MODEL_CONFIG_FILE="${MODEL_CONFIG_DIR}/qwen3-vl-8B.sh" \ +MODEL_NAME=Qwen3-VL-8B-Instruct \ +MODEL_RUN_NAME=qwen3-vl-8b \ +CHECKPOINT_SAVE=0 \ +ROLLOUT_MAX_RESPONSE_LEN=512 \ +ROLLOUT_MAX_PROMPT_LEN=2048 \ +ROLLOUT_MAX_CONTEXT_LEN=2560 \ +ACTOR_MAX_TOKENS_PER_GPU=6144 \ +HYBRID_ACTOR_GPUS=4 \ +HYBRID_ROLLOUT_GPUS=1 \ +ROLLOUT_NUM_GPUS_PER_ENGINE=1 \ +HYBRID_PIPELINE_FORWARD=1 \ +bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ + hybrid-async +``` + +After changing the model, resource topology, or length limits, compare only +against a baseline with the identical configuration. Do not pool such runs +with the default 8-GPU recipe. + Trace files are separated by hostname, role, PID, and global rank. They contain timestamps, counts, token totals, multimodal tensor byte counts, CUDA peaks, and an irreversible global-index fingerprint; prompts, responses, images, and diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index 08267aa1f..d5e04832b 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -235,6 +235,49 @@ bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ hybrid-async ``` +同一脚本还提供默认行为不变的实验开关，避免为了 smoke 或配对基准临时修改 +脚本正文： + +| 环境变量 | 默认值 | 作用 | +| --- | --- | --- | +| `MODEL_CONFIG_FILE` | `qwen35-9B.sh` | 选择模型并行配置脚本 | +| `MODEL_NAME` / `MODEL_RUN_NAME` | `Qwen3.5-9B` / `qwen35-9b` | 选择 checkpoint 子目录和日志前缀 | +| `MODEL_CHECKPOINT_DIR` / `REFERENCE_CHECKPOINT_DIR` | `${MODEL_DIR}/${MODEL_NAME}` | 显式固定 actor 与 reference 输入 | +| `CHECKPOINT_SAVE` | `1` | 设为 `0` 时不传 `--save*`；改用独立 rollout 与 TensorBoard 输出目录 | +| `ROLLOUT_RESULT_DIR` / `TENSORBOARD_DIR` | 保存开启时不覆盖；关闭时为 `${EXP_DIR}/rollout_result` 和 `${EXP_DIR}/tensorboard_log` | 在 no-save 运行中保留原始结果和标量；`TENSORBOARD_DIR` 会导出给 MetricsService | +| `ROLLOUT_MAX_RESPONSE_LEN` / `ROLLOUT_MAX_PROMPT_LEN` / `ROLLOUT_MAX_CONTEXT_LEN` | `10240` / `2048` / `12288` | 固定生成和上下文上限 | +| `ACTOR_MAX_TOKENS_PER_GPU` | `12288` | 控制 dynamic-batch actor microbatch 的 token 上限 | +| `HYBRID_ACTOR_GPUS` / `HYBRID_ROLLOUT_GPUS` | `4` / `4` | 构造 Hybrid placement resource | +| `ROLLOUT_NUM_GPUS_PER_ENGINE` | `2` | 设置每个 SGLang engine 的 TP 数 | +| `SGLANG_MEM_FRACTION_STATIC` | `0.8` | 设置 SGLang 静态显存比例 | + +脚本会在提交 Ray Job 前校验正整数、上下文容量、rollout GPU 可整除关系， +以及 actor GPU 数是当前 TP=2、CP=2 拓扑的整数倍。不满足约束会直接退出， +不会在 Ray worker 内静默降级。 + +例如，以下命令用于资源受限环境中的 no-save Qwen3-VL-8B 功能 smoke； +它不是 8 卡 Qwen3.5 性能结果： + +```bash +MODEL_CONFIG_FILE="${MODEL_CONFIG_DIR}/qwen3-vl-8B.sh" \ +MODEL_NAME=Qwen3-VL-8B-Instruct \ +MODEL_RUN_NAME=qwen3-vl-8b \ +CHECKPOINT_SAVE=0 \ +ROLLOUT_MAX_RESPONSE_LEN=512 \ +ROLLOUT_MAX_PROMPT_LEN=2048 \ +ROLLOUT_MAX_CONTEXT_LEN=2560 \ +ACTOR_MAX_TOKENS_PER_GPU=6144 \ +HYBRID_ACTOR_GPUS=4 \ +HYBRID_ROLLOUT_GPUS=1 \ +ROLLOUT_NUM_GPUS_PER_ENGINE=1 \ +HYBRID_PIPELINE_FORWARD=1 \ +bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ + hybrid-async +``` + +资源拓扑、模型或长度上限变化后，只能与完全相同配置的 baseline 做配对 +比较；不得与默认 8 卡配方合并统计。 + Trace 按 hostname、role、PID 和 global rank 分文件。内容仅包括时间戳、 样本与 token 计数、多模态 tensor 字节数、CUDA 峰值以及不可逆的 global-index fingerprint；不会写 prompt、response、图片或样本 tensor。 diff --git a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh index d1a453a42..0d98f3d44 100644 --- a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh +++ b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh @@ -21,14 +21,29 @@ SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" if [ -z "${RELAX_ENTRYPOINT_MODE:-}" ]; then source "${SCRIPT_DIR}/../../entrypoint/local.sh" fi -source "${MODEL_CONFIG_DIR}/qwen35-9B.sh" -# source "${MODEL_CONFIG_DIR}/qwen3-vl-4B.sh" +MODEL_CONFIG_FILE="${MODEL_CONFIG_FILE:-${MODEL_CONFIG_DIR}/qwen35-9B.sh}" +if [ ! -f "${MODEL_CONFIG_FILE}" ]; then + echo "MODEL_CONFIG_FILE does not exist: ${MODEL_CONFIG_FILE}" >&2 + exit 2 +fi +source "${MODEL_CONFIG_FILE}" PROJECT_NAME="${PROJECT_NAME:=Relax/dev/fully_async_openr1mm}" EXP_DIR="${EXP_DIR:-${SCRIPT_DIR}/../../../../exps}" MODEL_DIR="${MODEL_DIR:-${EXP_DIR}}" DATA_DIR="${DATA_DIR:-${EXP_DIR}}" NUM_ROLLOUT="${NUM_ROLLOUT:=200}" +MODEL_NAME="${MODEL_NAME:-Qwen3.5-9B}" +MODEL_RUN_NAME="${MODEL_RUN_NAME:-qwen35-9b}" +MODEL_CHECKPOINT_DIR="${MODEL_CHECKPOINT_DIR:-${MODEL_DIR}/${MODEL_NAME}}" +REFERENCE_CHECKPOINT_DIR="${REFERENCE_CHECKPOINT_DIR:-${MODEL_CHECKPOINT_DIR}}" + +CHECKPOINT_SAVE="${CHECKPOINT_SAVE:-1}" +CHECKPOINT_DIR="${CHECKPOINT_DIR:-${EXP_DIR}/${MODEL_NAME}_mcore_8xgpu/}" +CHECKPOINT_SAVE_INTERVAL="${CHECKPOINT_SAVE_INTERVAL:-100}" +MAX_ACTOR_CKPT_TO_KEEP="${MAX_ACTOR_CKPT_TO_KEEP:-1}" +ROLLOUT_RESULT_DIR="${ROLLOUT_RESULT_DIR:-}" +TENSORBOARD_DIR="${TENSORBOARD_DIR:-}" HYBRID_PIPELINE_FORWARD="${HYBRID_PIPELINE_FORWARD:-0}" HYBRID_PIPELINE_TRACE_DIR="${HYBRID_PIPELINE_TRACE_DIR:-}" @@ -37,6 +52,34 @@ SGLANG_DETERMINISTIC_INFERENCE="${SGLANG_DETERMINISTIC_INFERENCE:-0}" SEED="${SEED:-}" ROLLOUT_SEED="${ROLLOUT_SEED:-}" +ROLLOUT_MAX_RESPONSE_LEN="${ROLLOUT_MAX_RESPONSE_LEN:-10240}" +ROLLOUT_MAX_PROMPT_LEN="${ROLLOUT_MAX_PROMPT_LEN:-2048}" +ROLLOUT_MAX_CONTEXT_LEN="${ROLLOUT_MAX_CONTEXT_LEN:-12288}" +ACTOR_MAX_TOKENS_PER_GPU="${ACTOR_MAX_TOKENS_PER_GPU:-12288}" +HYBRID_ACTOR_GPUS="${HYBRID_ACTOR_GPUS:-4}" +HYBRID_ROLLOUT_GPUS="${HYBRID_ROLLOUT_GPUS:-4}" +SYNC_GPUS="${SYNC_GPUS:-8}" +ROLLOUT_NUM_GPUS_PER_ENGINE="${ROLLOUT_NUM_GPUS_PER_ENGINE:-2}" +SGLANG_MEM_FRACTION_STATIC="${SGLANG_MEM_FRACTION_STATIC:-0.8}" + +require_positive_integer() { + local name="$1" + local value="$2" + if ! [[ "${value}" =~ ^[1-9][0-9]*$ ]]; then + echo "${name} must be a positive integer, got ${value}" >&2 + exit 2 + fi +} + +require_fraction() { + local name="$1" + local value="$2" + if ! [[ "${value}" =~ ^(0\.[0-9]*[1-9][0-9]*|1(\.0+)?)$ ]]; then + echo "${name} must be greater than 0 and at most 1, got ${value}" >&2 + exit 2 + fi +} + case "${MODE}" in hybrid-async|sync) ;; *) @@ -58,6 +101,13 @@ case "${SGLANG_DETERMINISTIC_INFERENCE}" in exit 2 ;; esac +case "${CHECKPOINT_SAVE}" in + 0|1) ;; + *) + echo "CHECKPOINT_SAVE must be 0 or 1, got ${CHECKPOINT_SAVE}" >&2 + exit 2 + ;; +esac if [ "${MODE}" != "hybrid-async" ] && { [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] || [ -n "${HYBRID_PIPELINE_TRACE_DIR}" ]; }; then @@ -65,6 +115,80 @@ if [ "${MODE}" != "hybrid-async" ] && { exit 2 fi +for item in \ + "NUM_ROLLOUT:${NUM_ROLLOUT}" \ + "HYBRID_PIPELINE_FETCH_TIMEOUT_S:${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" \ + "ROLLOUT_MAX_RESPONSE_LEN:${ROLLOUT_MAX_RESPONSE_LEN}" \ + "ROLLOUT_MAX_PROMPT_LEN:${ROLLOUT_MAX_PROMPT_LEN}" \ + "ROLLOUT_MAX_CONTEXT_LEN:${ROLLOUT_MAX_CONTEXT_LEN}" \ + "ACTOR_MAX_TOKENS_PER_GPU:${ACTOR_MAX_TOKENS_PER_GPU}" \ + "HYBRID_ACTOR_GPUS:${HYBRID_ACTOR_GPUS}" \ + "HYBRID_ROLLOUT_GPUS:${HYBRID_ROLLOUT_GPUS}" \ + "SYNC_GPUS:${SYNC_GPUS}" \ + "ROLLOUT_NUM_GPUS_PER_ENGINE:${ROLLOUT_NUM_GPUS_PER_ENGINE}"; do + require_positive_integer "${item%%:*}" "${item#*:}" +done +if [ "${CHECKPOINT_SAVE}" = "1" ]; then + require_positive_integer "CHECKPOINT_SAVE_INTERVAL" "${CHECKPOINT_SAVE_INTERVAL}" + require_positive_integer "MAX_ACTOR_CKPT_TO_KEEP" "${MAX_ACTOR_CKPT_TO_KEEP}" +fi +if (( ROLLOUT_MAX_CONTEXT_LEN < ROLLOUT_MAX_PROMPT_LEN + ROLLOUT_MAX_RESPONSE_LEN )); then + echo "ROLLOUT_MAX_CONTEXT_LEN must cover prompt + response limits" >&2 + exit 2 +fi +if (( HYBRID_ROLLOUT_GPUS % ROLLOUT_NUM_GPUS_PER_ENGINE != 0 )); then + echo "HYBRID_ROLLOUT_GPUS must be divisible by ROLLOUT_NUM_GPUS_PER_ENGINE" >&2 + exit 2 +fi +if (( SYNC_GPUS % ROLLOUT_NUM_GPUS_PER_ENGINE != 0 )); then + echo "SYNC_GPUS must be divisible by ROLLOUT_NUM_GPUS_PER_ENGINE" >&2 + exit 2 +fi +if (( HYBRID_ACTOR_GPUS % 4 != 0 || SYNC_GPUS % 4 != 0 )); then + echo "actor GPU counts must be multiples of TP(2) * CP(2)" >&2 + exit 2 +fi +if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] && [ "${HYBRID_ACTOR_GPUS}" != "4" ]; then + echo "HYBRID_PIPELINE_FORWARD currently requires TP=2, CP=2, DP=1 (4 actor GPUs)" >&2 + exit 2 +fi +require_fraction "SGLANG_MEM_FRACTION_STATIC" "${SGLANG_MEM_FRACTION_STATIC}" + +if [ "${CHECKPOINT_SAVE}" = "0" ]; then + ROLLOUT_RESULT_DIR="${ROLLOUT_RESULT_DIR:-${EXP_DIR}/rollout_result}" + TENSORBOARD_DIR="${TENSORBOARD_DIR:-${EXP_DIR}/tensorboard_log}" +fi +if [ -n "${TENSORBOARD_DIR}" ]; then + export TENSORBOARD_DIR + RUNTIME_ENV_JSON_INPUT="${RUNTIME_ENV_JSON:-}" + if [ -z "${RUNTIME_ENV_JSON_INPUT}" ]; then + RUNTIME_ENV_JSON_INPUT='{}' + fi + RUNTIME_ENV_JSON="$( + python3 - "${RUNTIME_ENV_JSON_INPUT}" "${TENSORBOARD_DIR}" <<'PY' +import json +import sys + +runtime_env = json.loads(sys.argv[1]) +if not isinstance(runtime_env, dict): + raise SystemExit("RUNTIME_ENV_JSON must decode to an object") +env_vars = runtime_env.setdefault("env_vars", {}) +if not isinstance(env_vars, dict): + raise SystemExit("RUNTIME_ENV_JSON env_vars must be an object") +env_vars["TENSORBOARD_DIR"] = sys.argv[2] +print(json.dumps(runtime_env, separators=(",", ":"), sort_keys=True)) +PY + )" +fi + +HYBRID_RESOURCE="{\"actor\": [1, ${HYBRID_ACTOR_GPUS}], \"rollout\": [1, ${HYBRID_ROLLOUT_GPUS}]}" +SYNC_RESOURCE="{\"actor\": [1, ${SYNC_GPUS}], \"rollout\": [1, ${SYNC_GPUS}]}" +if [ "${MODE}" = "hybrid-async" ]; then + RUN_GPU_COUNT=$((HYBRID_ACTOR_GPUS + HYBRID_ROLLOUT_GPUS)) +else + RUN_GPU_COUNT="${SYNC_GPUS}" +fi + HYBRID_PIPELINE_ARGS=() if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ]; then HYBRID_PIPELINE_ARGS+=( @@ -108,39 +232,59 @@ printf '%s\n' \ "HYBRID_PIPELINE_FETCH_TIMEOUT_S=${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" \ "SEED=${SEED}" \ "ROLLOUT_SEED=${ROLLOUT_SEED}" \ - "SGLANG_DETERMINISTIC_INFERENCE=${SGLANG_DETERMINISTIC_INFERENCE}" + "SGLANG_DETERMINISTIC_INFERENCE=${SGLANG_DETERMINISTIC_INFERENCE}" \ + "MODEL_CONFIG_FILE=${MODEL_CONFIG_FILE}" \ + "MODEL_CHECKPOINT_DIR=${MODEL_CHECKPOINT_DIR}" \ + "REFERENCE_CHECKPOINT_DIR=${REFERENCE_CHECKPOINT_DIR}" \ + "CHECKPOINT_SAVE=${CHECKPOINT_SAVE}" \ + "CHECKPOINT_DIR=${CHECKPOINT_DIR}" \ + "ROLLOUT_RESULT_DIR=${ROLLOUT_RESULT_DIR}" \ + "TENSORBOARD_DIR=${TENSORBOARD_DIR}" \ + "ACTOR_MAX_TOKENS_PER_GPU=${ACTOR_MAX_TOKENS_PER_GPU}" \ + "HYBRID_RESOURCE=${HYBRID_RESOURCE}" \ + "SYNC_RESOURCE=${SYNC_RESOURCE}" \ + "RUN_GPU_COUNT=${RUN_GPU_COUNT}" CKPT_ARGS=( - --hf-checkpoint ${MODEL_DIR}/Qwen3.5-9B - --ref-load ${MODEL_DIR}/Qwen3.5-9B - # --hf-checkpoint ${MODEL_DIR}/Qwen3-VL-4B-Instruct + --hf-checkpoint "${MODEL_CHECKPOINT_DIR}" + --ref-load "${REFERENCE_CHECKPOINT_DIR}" --megatron-to-hf-mode bridge --warm-hf-checkpoint-page-cache - # --ref-load ${MODEL_DIR}/Qwen3-VL-4B-Instruct - # --load ${EXP_DIR}/Qwen3.5-9B_mcore_8xgpu/ - --save ${EXP_DIR}/Qwen3.5-9B_mcore_8xgpu/ - --save-interval 100 - --max-actor-ckpt-to-keep 1 ) +if [ "${CHECKPOINT_SAVE}" = "1" ]; then + CKPT_ARGS+=( + --save "${CHECKPOINT_DIR}" + --save-interval "${CHECKPOINT_SAVE_INTERVAL}" + --max-actor-ckpt-to-keep "${MAX_ACTOR_CKPT_TO_KEEP}" + ) +fi + +OUTPUT_ARGS=() +if [ -n "${ROLLOUT_RESULT_DIR}" ]; then + OUTPUT_ARGS+=(--rollout-result-dir "${ROLLOUT_RESULT_DIR}") +fi +if [ -n "${TENSORBOARD_DIR}" ]; then + OUTPUT_ARGS+=(--tensorboard-dir "${TENSORBOARD_DIR}") +fi -PROMPT_SET=${DATA_DIR}/multimodal-open-r1-8k-verified/data/train-00000-of-00001_converted_noextract.parquet +PROMPT_SET="${PROMPT_SET:-${DATA_DIR}/multimodal-open-r1-8k-verified/data/train-00000-of-00001_converted_noextract.parquet}" SYSTEM_PROMPT="A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within and tags, respectively, i.e., reasoning process here answer here " ROLLOUT_ARGS=( - --prompt-data ${PROMPT_SET} + --prompt-data "${PROMPT_SET}" --input-key prompt --label-key label --apply-chat-template --rollout-shuffle --rm-type openr1mm - --num-rollout ${NUM_ROLLOUT} + --num-rollout "${NUM_ROLLOUT}" --rollout-batch-size 32 --n-samples-per-prompt 8 - --rollout-max-response-len 10240 - --rollout-max-prompt-len 2048 - --rollout-max-context-len 12288 + --rollout-max-response-len "${ROLLOUT_MAX_RESPONSE_LEN}" + --rollout-max-prompt-len "${ROLLOUT_MAX_PROMPT_LEN}" + --rollout-max-context-len "${ROLLOUT_MAX_CONTEXT_LEN}" --rollout-temperature 0.8 --global-batch-size 256 --multimodal-keys '{"image":"image"}' @@ -164,7 +308,7 @@ PERF_ARGS=( # --micro-batch-size 16 # --qkv-format bshd --use-dynamic-batch-size - --max-tokens-per-gpu 12288 + --max-tokens-per-gpu "${ACTOR_MAX_TOKENS_PER_GPU}" --no-rope-fusion ) @@ -198,13 +342,13 @@ WANDB_ARGS=( --use-tensorboard --use-clearml --use-metrics-service - --tb-project-name ${PROJECT_NAME} - --tb-experiment-name qwen35-9b-GRPO-gpu8-${MODE}-${now} + --tb-project-name "${PROJECT_NAME}" + --tb-experiment-name "${MODEL_RUN_NAME}-GRPO-gpu${RUN_GPU_COUNT}-${MODE}-${now}" ) SGLANG_ARGS=( - --rollout-num-gpus-per-engine 2 - --sglang-mem-fraction-static 0.8 + --rollout-num-gpus-per-engine "${ROLLOUT_NUM_GPUS_PER_ENGINE}" + --sglang-mem-fraction-static "${SGLANG_MEM_FRACTION_STATIC}" ) MISC_ARGS=( @@ -225,7 +369,7 @@ if [ "${MODE}" = "hybrid-async" ]; then ray job submit ${RAY_NO_WAIT:+--no-wait} --address="http://127.0.0.1:8265" \ --runtime-env-json="${RUNTIME_ENV_JSON}" \ -- python3 -m relax.entrypoints.train \ - --resource '{"actor": [1, 4], "rollout": [1, 4]}'\ + --resource "${HYBRID_RESOURCE}" \ --max-staleness 2 \ --num-data-storage-units 1 \ --num-iters-per-train-update 2 \ @@ -236,18 +380,19 @@ if [ "${MODE}" = "hybrid-async" ]; then "${DEBUG_ARGS[@]}" \ "${MODEL_ARGS[@]}" \ "${CKPT_ARGS[@]}" \ + "${OUTPUT_ARGS[@]}" \ "${ROLLOUT_ARGS[@]}" \ "${OPTIMIZER_ARGS[@]}" \ "${GRPO_ARGS[@]}" \ "${WANDB_ARGS[@]}" \ "${PERF_ARGS[@]}" \ "${SGLANG_ARGS[@]}" \ - "${MISC_ARGS[@]}" 2>&1 | tee "${LOG_DIR}/qwen35-9b-GRPO-gpu8-hybrid-async-${now}.log" + "${MISC_ARGS[@]}" 2>&1 | tee "${LOG_DIR}/${MODEL_RUN_NAME}-GRPO-gpu${RUN_GPU_COUNT}-hybrid-async-${now}.log" else ray job submit ${RAY_NO_WAIT:+--no-wait} --address="http://127.0.0.1:8265" \ --runtime-env-json="${RUNTIME_ENV_JSON}" \ -- python3 -m relax.entrypoints.train \ - --resource '{"actor": [1, 8], "rollout": [1, 8]}'\ + --resource "${SYNC_RESOURCE}" \ --max-staleness 0 \ --num-data-storage-units 1 \ --colocate \ @@ -257,11 +402,12 @@ else "${DEBUG_ARGS[@]}" \ "${MODEL_ARGS[@]}" \ "${CKPT_ARGS[@]}" \ + "${OUTPUT_ARGS[@]}" \ "${ROLLOUT_ARGS[@]}" \ "${OPTIMIZER_ARGS[@]}" \ "${GRPO_ARGS[@]}" \ "${WANDB_ARGS[@]}" \ "${PERF_ARGS[@]}" \ "${SGLANG_ARGS[@]}" \ - "${MISC_ARGS[@]}" 2>&1 | tee "${LOG_DIR}/qwen35-9b-GRPO-gpu8-fully-sync-${now}.log" + "${MISC_ARGS[@]}" 2>&1 | tee "${LOG_DIR}/${MODEL_RUN_NAME}-GRPO-gpu${RUN_GPU_COUNT}-fully-sync-${now}.log" fi diff --git a/tests/scripts/test_multimodal_hybrid_async_launcher.py b/tests/scripts/test_multimodal_hybrid_async_launcher.py new file mode 100644 index 000000000..f496633db --- /dev/null +++ b/tests/scripts/test_multimodal_hybrid_async_launcher.py @@ -0,0 +1,152 @@ +# Copyright (c) 2026 Relax Authors. All Rights Reserved. + +import json +import os +import subprocess +from pathlib import Path + + +REPO_ROOT = Path(__file__).resolve().parents[2] +LAUNCHER = ( + REPO_ROOT + / "scripts" + / "training" + / "multimodal" + / "run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh" +) + + +def _argument_value(arguments: list[str], flag: str) -> str: + if flag in arguments: + index = arguments.index(flag) + return arguments[index + 1] + prefix = f"{flag}=" + for argument in arguments: + if argument.startswith(prefix): + return argument.removeprefix(prefix) + raise ValueError(f"{flag!r} is not present") + + +def _run_launcher( + tmp_path: Path, + *, + overrides: dict[str, str] | None = None, +) -> tuple[subprocess.CompletedProcess[str], list[str], Path, Path]: + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + capture_path = tmp_path / "ray-arguments.txt" + tensorboard_capture_path = tmp_path / "ray-tensorboard-dir.txt" + fake_ray = bin_dir / "ray" + fake_ray.write_text( + """#!/usr/bin/env bash +set -e +printf '%s\\n' "$@" > "${RAY_CAPTURE}" +printf '%s' "${TENSORBOARD_DIR:-}" > "${RAY_TENSORBOARD_CAPTURE}" +""", + encoding="utf-8", + ) + fake_ray.chmod(0o755) + + model_config = tmp_path / "test-model.sh" + model_config.write_text( + "MODEL_ARGS=(--test-model-arg test-model-value)\n", + encoding="utf-8", + ) + + env = { + "PATH": f"{bin_dir}:{os.environ['PATH']}", + "HOME": str(tmp_path), + "RELAX_ENTRYPOINT_MODE": "ray-job", + "MODEL_CONFIG_DIR": str(tmp_path), + "MODEL_CONFIG_FILE": str(model_config), + "RUNTIME_ENV_JSON": "{}", + "RAY_CAPTURE": str(capture_path), + "RAY_TENSORBOARD_CAPTURE": str(tensorboard_capture_path), + "EXP_DIR": str(tmp_path / "exp"), + "MODEL_DIR": str(tmp_path / "models"), + "DATA_DIR": str(tmp_path / "data"), + "PROJECT_NAME": "Relax/test-launcher", + "NUM_ROLLOUT": "2", + } + if overrides: + env.update(overrides) + + result = subprocess.run( + ["bash", str(LAUNCHER), "hybrid-async"], + cwd=REPO_ROOT, + env=env, + check=False, + capture_output=True, + text=True, + ) + arguments = capture_path.read_text(encoding="utf-8").splitlines() if capture_path.exists() else [] + return result, arguments, capture_path, tensorboard_capture_path + + +def test_launcher_preserves_default_qwen35_recipe(tmp_path): + result, arguments, _, tensorboard_capture = _run_launcher(tmp_path) + + assert result.returncode == 0, result.stderr + assert _argument_value(arguments, "--resource") == '{"actor": [1, 4], "rollout": [1, 4]}' + assert _argument_value(arguments, "--hf-checkpoint") == str(tmp_path / "models" / "Qwen3.5-9B") + assert _argument_value(arguments, "--ref-load") == str(tmp_path / "models" / "Qwen3.5-9B") + assert _argument_value(arguments, "--rollout-max-response-len") == "10240" + assert _argument_value(arguments, "--rollout-max-context-len") == "12288" + assert _argument_value(arguments, "--max-tokens-per-gpu") == "12288" + assert _argument_value(arguments, "--rollout-num-gpus-per-engine") == "2" + assert _argument_value(arguments, "--save") == f"{tmp_path / 'exp' / 'Qwen3.5-9B_mcore_8xgpu'}/" + assert _argument_value(arguments, "--save-interval") == "100" + assert "--rollout-result-dir" not in arguments + assert "--tensorboard-dir" not in arguments + assert tensorboard_capture.read_text(encoding="utf-8") == "" + assert _argument_value(arguments, "--test-model-arg") == "test-model-value" + + +def test_launcher_supports_no_save_memory_safe_smoke_configuration(tmp_path): + result, arguments, _, tensorboard_capture = _run_launcher( + tmp_path, + overrides={ + "MODEL_NAME": "Qwen3-VL-8B-Instruct", + "MODEL_RUN_NAME": "qwen3-vl-8b", + "CHECKPOINT_SAVE": "0", + "ROLLOUT_MAX_RESPONSE_LEN": "512", + "ROLLOUT_MAX_PROMPT_LEN": "2048", + "ROLLOUT_MAX_CONTEXT_LEN": "2560", + "ACTOR_MAX_TOKENS_PER_GPU": "6144", + "HYBRID_ROLLOUT_GPUS": "1", + "ROLLOUT_NUM_GPUS_PER_ENGINE": "1", + }, + ) + + assert result.returncode == 0, result.stderr + assert _argument_value(arguments, "--resource") == '{"actor": [1, 4], "rollout": [1, 1]}' + assert _argument_value(arguments, "--hf-checkpoint") == str( + tmp_path / "models" / "Qwen3-VL-8B-Instruct" + ) + assert _argument_value(arguments, "--rollout-max-response-len") == "512" + assert _argument_value(arguments, "--rollout-max-context-len") == "2560" + assert _argument_value(arguments, "--max-tokens-per-gpu") == "6144" + assert _argument_value(arguments, "--rollout-num-gpus-per-engine") == "1" + assert "--save" not in arguments + assert "--save-interval" not in arguments + assert _argument_value(arguments, "--rollout-result-dir") == str(tmp_path / "exp" / "rollout_result") + expected_tensorboard_dir = tmp_path / "exp" / "tensorboard_log" + assert _argument_value(arguments, "--tensorboard-dir") == str(expected_tensorboard_dir) + assert tensorboard_capture.read_text(encoding="utf-8") == str(expected_tensorboard_dir) + runtime_env = json.loads(_argument_value(arguments, "--runtime-env-json")) + assert runtime_env["env_vars"]["TENSORBOARD_DIR"] == str(expected_tensorboard_dir) + log_names = [path.name for path in (tmp_path / "exp" / "logs").iterdir()] + assert len(log_names) == 1 + assert log_names[0].startswith("qwen3-vl-8b-GRPO-gpu5-hybrid-async-") + + +def test_launcher_rejects_invalid_checkpoint_switch_before_ray(tmp_path): + result, arguments, capture_path, _ = _run_launcher( + tmp_path, + overrides={"CHECKPOINT_SAVE": "2"}, + ) + + assert result.returncode == 2 + assert "CHECKPOINT_SAVE must be 0 or 1" in result.stderr + assert arguments == [] + assert not capture_path.exists() From 2a921ad645c94da901a282c8c47870f53a614477 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Wed, 29 Jul 2026 23:35:25 +0800 Subject: [PATCH 06/21] fix(benchmark): reject incomparable hybrid performance runs Use the rollout-side truncation ratio and require paired runs to match the complete model, data, length, resource, deterministic-inference, and debug workload configuration before computing statistics. Keep TensorBoard parsing usable with newer host protobuf runtimes. Constraint: Task 21 evidence must compare identical workloads and report an actual truncation rate. Rejected: Rely only on matching commit and batch sizes | model, context, and GPU topology mismatches can still invalidate the result. Confidence: high Scope-risk: narrow Directive: Extend COMPARISON_WORKLOAD_MANIFEST_FIELDS whenever the benchmark runner adds a behavior-affecting setting. Tested: 100 relevant pytest cases passed, 4 environment-dependent cases skipped; ruff, py_compile, bash -n, shellcheck with source exclusions, git diff --check; host smoke-pair analysis passed. --- docs/en/guide/hybrid-training.md | 8 +++ docs/zh/guide/hybrid-training.md | 7 +++ .../analyze_hybrid_pipeline_benchmark.py | 44 ++++++++++++++-- .../test_analyze_hybrid_pipeline_benchmark.py | 51 +++++++++++++++++-- 4 files changed, 103 insertions(+), 7 deletions(-) diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index 145be85b3..3d3e07b7a 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -322,6 +322,9 @@ checks per-run strict producer overlap, step-time p95, eight-GPU NVML coverage, peak VRAM, token/multimodal-byte workload, and the raw-reward, truncation-rate, and staleness guardrails. It uses `sum(step_tokens) / sum(step_time)` for aggregate throughput rather than averaging per-step rates. +The truncation guardrail reads the rollout-side +`rollout/truncated_ratio`; the training-side `rollout/truncated` scalar is not +used because chunk aggregation can sum that value more than once per step. GPU utilization and the below-10% idle ratio use only 500 ms NVML samples whose wall time falls inside the registered steady step intervals reconstructed from TensorBoard `perf/step_time`; sampled peak VRAM remains a full-run safety @@ -332,6 +335,11 @@ dependency freeze, wheel hash, and launcher log for every run. The manifest also records and cross-checks `max_staleness`, global/rollout batch sizes, samples per prompt, and actor chunk count, so CLI expectations cannot silently disagree with the measured workload. +Before any paired statistics are calculated, the analyzer also requires +identical model/config/data paths, prompt/response/context limits, actor token +budget, actor/rollout resource topology, SGLang determinism and memory +settings, physical-to-container GPU mapping, checkpoint mode, and debug +capture/replay settings. Missing workload fields or any mismatch fail closed. The staleness curve is the trace-derived producer lead at the first actor forward: the largest completed producer rollout ID minus the actor rollout ID at that timestamp. The current rollout is considered ready once its actor fetch diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index d5e04832b..8be7343ae 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -312,6 +312,9 @@ fetch/forward 次数仍严格固定。严格 producer 重叠定义为首个 acto step-time p95、8 卡 NVML 覆盖、峰值显存、token/多模态字节工作量和 raw-reward、截断率、staleness 非劣护栏。aggregate throughput 使用 `sum(step_tokens) / sum(step_time)`，不会对 per-step rate 做简单平均。 +截断率护栏读取 rollout 侧的 `rollout/truncated_ratio`；训练侧 +`rollout/truncated` 在分块聚合时可能被一个 step 累加多次，因此不作为截断率 +统计口径。 GPU utilization 与低于 10% 的 idle ratio 只使用墙钟时间落入预注册稳态 step 区间的 500 ms NVML 样本；区间由 TensorBoard `perf/step_time` 的 step end wall time 和 duration 重建。sampled peak VRAM 仍使用 full-run @@ -320,6 +323,10 @@ step end wall time 和 duration 重建。sampled peak VRAM 仍使用 full-run 身份完全一致，并存在非空的输入 hash、依赖 freeze、wheel hash 和 launcher log。 manifest 还会记录并交叉校验 `max_staleness`、global/rollout batch size、 每 prompt 样本数和 actor chunk 数，避免分析器 CLI 与实际工作负载静默不一致。 +在计算任何配对统计前，分析器还要求模型、模型配置、数据路径、prompt/response/ +context 上限、actor token 预算、actor/rollout 资源拓扑、SGLang 确定性与显存 +配置、物理卡到容器卡映射、checkpoint 模式以及 debug 捕获/回放配置完全一致； +字段缺失或取值不同都会 fail closed。 staleness 曲线来自 trace：在 actor 首次 forward 时，用已完成 put 的最大 producer rollout ID 减去当前 actor rollout ID；若 producer 的 trace 写入稍晚， 已完成的 actor fetch 本身可证明当前 rollout 已 ready。该值不得超过 manifest diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py index ae8283094..b1fc3b0ed 100644 --- a/scripts/tools/analyze_hybrid_pipeline_benchmark.py +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -8,6 +8,7 @@ import csv import json import math +import os import statistics import sys from collections import defaultdict @@ -58,7 +59,7 @@ COMPARISON_PERFORMANCE_TAGS = PERFORMANCE_TAGS + ("perf/wall_clock_samples_per_s",) CORRECTNESS_GUARDRAIL_TAGS = ( "rollout/raw_reward", - "rollout/truncated", + "rollout/truncated_ratio", "train/loss", "train/grad_norm", ) @@ -112,6 +113,29 @@ "python", "entrypoint", ) +COMPARISON_WORKLOAD_MANIFEST_FIELDS = ( + "schema_version", + "baseline_commit", + "model_variant", + "model_name", + "model_dir", + "model_config_file", + "data_file", + "rollout_max_response_len", + "rollout_max_prompt_len", + "rollout_max_context_len", + "actor_max_tokens_per_gpu", + "resource", + "rollout_num_gpus_per_engine", + "physical_gpu_indices", + "container_cuda_visible_devices", + "checkpoint_save", + "sglang_deterministic_inference", + "sglang_mem_fraction_static", + "load_debug_rollout_data", + "save_debug_rollout_data", + "save_debug_train_data", +) REPRODUCIBILITY_ARTIFACTS = ( "pip-freeze.txt", "inputs.sha256", @@ -667,6 +691,9 @@ def _load_tensorboard_scalars(run_dir: Path) -> list[dict[str, Any]]: event_paths = sorted(path for path in run_dir.rglob("events.out.tfevents.*") if path.is_file()) if not event_paths: return [] + # TensorBoard 2.10's generated protobuf bindings need the pure-Python + # compatibility path when the host has a newer protobuf runtime. + os.environ.setdefault("PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION", "python") try: from tensorboard.backend.event_processing.event_accumulator import EventAccumulator except ImportError: @@ -1026,7 +1053,16 @@ def _build_comparison( if set(by_condition) != {"baseline", "experiment"}: _fail(f"comparison requires conditions named 'baseline' and 'experiment', got {sorted(by_condition)}") - for field in COMPARISON_FIXED_MANIFEST_FIELDS: + comparison_fields = COMPARISON_FIXED_MANIFEST_FIELDS + COMPARISON_WORKLOAD_MANIFEST_FIELDS + missing_workload_fields = { + str(analysis.run_dir): sorted(set(COMPARISON_WORKLOAD_MANIFEST_FIELDS) - set(analysis.manifest)) + for analysis in analyses + if set(COMPARISON_WORKLOAD_MANIFEST_FIELDS) - set(analysis.manifest) + } + if missing_workload_fields: + _fail(f"comparison manifests are missing workload fields: {missing_workload_fields}") + + for field in comparison_fields: values = {json.dumps(analysis.manifest[field], sort_keys=True) for analysis in analyses} if len(values) != 1: _fail(f"comparison requires identical manifest field {field!r}, got {sorted(values)}") @@ -1306,9 +1342,9 @@ def _build_comparison( if statistics.fmean(accuracy_deltas) < -0.02: _fail(f"mean paired raw-reward/accuracy drop exceeds 2 percentage points: {accuracy_deltas}") - truncation_deltas = [row.get("rollout/truncated:delta") for row in paired] + truncation_deltas = [row.get("rollout/truncated_ratio:delta") for row in paired] if any(value is None for value in truncation_deltas): - _fail("correctness targets require rollout/truncated for every paired run") + _fail("correctness targets require rollout/truncated_ratio for every paired run") if any(value > 0.02 for value in truncation_deltas): _fail(f"a paired truncation-rate increase exceeds 2 percentage points: {truncation_deltas}") diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py index 50dbae952..ad57896d4 100644 --- a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -530,7 +530,7 @@ def _comparison_analysis( "perf/step_time": {"mean": 20.0, "p95": 21.0}, "perf/hybrid_phase1_time": {"mean": phase1}, "rollout/raw_reward": {"mean": accuracy}, - "rollout/truncated": {"mean": 0.0}, + "rollout/truncated_ratio": {"mean": 0.0}, "train/loss": {"mean": 1.0}, "train/grad_norm": {"mean": 0.5}, } @@ -555,7 +555,7 @@ def _comparison_analysis( "perf/step_time": 20.0, "perf/hybrid_phase1_time": phase1, "rollout/raw_reward": accuracy, - "rollout/truncated": 0.0, + "rollout/truncated_ratio": 0.0, "train/loss": 1.0, "train/grad_norm": 0.5, } @@ -585,6 +585,27 @@ def _comparison_analysis( "transferqueue_commit": "e" * 40, "python": "3.12.3", "entrypoint": "bash target.sh hybrid-async", + "schema_version": 1, + "baseline_commit": "f" * 40, + "model_variant": "qwen3vl8b", + "model_name": "Qwen3-VL-8B-Instruct", + "model_dir": "/data01/LWX/Qwen3-VL-8B-Instruct", + "model_config_file": "/workspace/Relax/scripts/models/qwen3-vl-8B.sh", + "data_file": "/data01/LWX/openr1mm/train.parquet", + "rollout_max_response_len": 1024, + "rollout_max_prompt_len": 2048, + "rollout_max_context_len": 3072, + "actor_max_tokens_per_gpu": 6144, + "resource": {"actor": [1, 4], "rollout": [1, 1]}, + "rollout_num_gpus_per_engine": 1, + "physical_gpu_indices": [0, 1, 2, 3, 5], + "container_cuda_visible_devices": [0, 1, 2, 3, 4], + "checkpoint_save": False, + "sglang_deterministic_inference": 1, + "sglang_mem_fraction_static": 0.8, + "load_debug_rollout_data": None, + "save_debug_rollout_data": None, + "save_debug_train_data": None, }, trace_rows=trace_rows, actor_rank_rows=[], @@ -660,6 +681,30 @@ def test_comparison_rejects_mixed_candidate_commits(): ) +def test_comparison_rejects_mixed_or_incomplete_workload_manifests(): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + ] + analyses[1].manifest["rollout_max_response_len"] = 2048 + + with pytest.raises(analyzer.BenchmarkValidationError, match="rollout_max_response_len"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=False, + ) + + analyses[1].manifest["rollout_max_response_len"] = 1024 + del analyses[1].manifest["physical_gpu_indices"] + with pytest.raises(analyzer.BenchmarkValidationError, match="missing workload fields"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=False, + ) + + def test_comparison_plot_bundle_is_generated(tmp_path, monkeypatch): analyses = [ _comparison_analysis("baseline", 1, 100, 10), @@ -729,7 +774,7 @@ def test_preregistered_guardrails_fail_closed(guardrail, error): elif guardrail == "accuracy": experiment.summary["metrics"]["rollout/raw_reward"]["mean"] = 0.46 elif guardrail == "truncation": - experiment.summary["metrics"]["rollout/truncated"]["mean"] = 0.03 + experiment.summary["metrics"]["rollout/truncated_ratio"]["mean"] = 0.03 elif guardrail == "staleness": for row in experiment.trace_rows: row["producer_lead_at_first_forward"] = 1.3 From 5198f4f5ce8e1e2e4a593832012ba8bfbee4e5e1 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Thu, 30 Jul 2026 01:08:24 +0800 Subject: [PATCH 07/21] fix(benchmark): make hybrid stage evidence fail closed Expose prompt-aligned stage count without changing the default recipe, and require paired deterministic runs to prove identical samples, workload, static inputs, and successful exits before accepting performance targets. Constraint: Task 21 must support a four-stage 256-sample mini and reproducible paired first-step evidence on the available five-GPU topology. Rejected: Accept aggregate workload within one percent under deterministic inference | it can hide different samples or generated lengths. Confidence: high Scope-risk: narrow Directive: Keep baseline and experiment NUM_ITERS_PER_TRAIN_UPDATE identical and label window 0-0 as fresh-process first-step evidence. Tested: 104 relevant pytest cases passed, 4 environment-dependent cases skipped; ruff, py_compile, bash -n, shellcheck warning gate, git diff --check. --- docs/en/guide/hybrid-training.md | 35 +++++++-- docs/zh/guide/hybrid-training.md | 29 +++++-- .../analyze_hybrid_pipeline_benchmark.py | 75 ++++++++++++++++++- ...n-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh | 9 ++- tests/backends/megatron/test_data_vpp.py | 23 ++++++ .../test_analyze_hybrid_pipeline_benchmark.py | 36 ++++++++- .../test_multimodal_hybrid_async_launcher.py | 30 +++++--- 7 files changed, 214 insertions(+), 23 deletions(-) diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index 3d3e07b7a..24e698b76 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -235,6 +235,7 @@ The reference launcher exposes the options as environment variables: HYBRID_PIPELINE_FORWARD=1 \ HYBRID_PIPELINE_TRACE_DIR=/data01/LWX/relax-task21/runs/smoke/timeline \ HYBRID_PIPELINE_FETCH_TIMEOUT_S=600 \ +NUM_ITERS_PER_TRAIN_UPDATE=4 \ bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ hybrid-async ``` @@ -249,6 +250,7 @@ run or paired benchmark does not require editing the script: | `MODEL_CHECKPOINT_DIR` / `REFERENCE_CHECKPOINT_DIR` | `${MODEL_DIR}/${MODEL_NAME}` | Pin actor and reference inputs explicitly | | `CHECKPOINT_SAVE` | `1` | Set to `0` to omit all `--save*` arguments and use separate rollout/TensorBoard outputs | | `ROLLOUT_RESULT_DIR` / `TENSORBOARD_DIR` | No override while saving; `${EXP_DIR}/rollout_result` and `${EXP_DIR}/tensorboard_log` without saving | Preserve raw results and scalars in no-save runs; `TENSORBOARD_DIR` is exported for MetricsService | +| `NUM_ITERS_PER_TRAIN_UPDATE` | `2` | Set the prompt-group-aligned producer and actor chunk count for each optimizer mini | | `ROLLOUT_MAX_RESPONSE_LEN` / `ROLLOUT_MAX_PROMPT_LEN` / `ROLLOUT_MAX_CONTEXT_LEN` | `10240` / `2048` / `12288` | Pin generation and context limits | | `ACTOR_MAX_TOKENS_PER_GPU` | `12288` | Bound dynamic-batch actor microbatch tokens | | `HYBRID_ACTOR_GPUS` / `HYBRID_ROLLOUT_GPUS` | `4` / `4` | Build the Hybrid placement resource | @@ -260,8 +262,17 @@ capacity, rollout-GPU divisibility, and that actor GPU counts are multiples of the current TP=2, CP=2 topology. Invalid combinations fail before Ray workers start instead of silently degrading. -For example, this is a no-save Qwen3-VL-8B functional smoke for a constrained -machine. It is not an 8-GPU Qwen3.5 performance result: +With the reference `global_batch_size=256` and +`n_samples_per_prompt=8`, setting `NUM_ITERS_PER_TRAIN_UPDATE=4` creates four +64-sample stages. Each stage therefore contains eight complete prompt groups. +Baseline and experiment runs in a pair must use the same value: the baseline +waits for and forwards the full mini, while the optional pipeline fetches and +forwards the four stages incrementally. + +For example, this is a no-save Qwen3-VL-8B configuration for a constrained +machine. It may be used for a resource-qualified smoke or paired performance +benchmark, but its results must be labeled separately from the default 8-GPU +Qwen3.5 recipe: ```bash MODEL_CONFIG_FILE="${MODEL_CONFIG_DIR}/qwen3-vl-8B.sh" \ @@ -272,6 +283,7 @@ ROLLOUT_MAX_RESPONSE_LEN=512 \ ROLLOUT_MAX_PROMPT_LEN=2048 \ ROLLOUT_MAX_CONTEXT_LEN=2560 \ ACTOR_MAX_TOKENS_PER_GPU=6144 \ +NUM_ITERS_PER_TRAIN_UPDATE=4 \ HYBRID_ACTOR_GPUS=4 \ HYBRID_ROLLOUT_GPUS=1 \ ROLLOUT_NUM_GPUS_PER_ENGINE=1 \ @@ -319,7 +331,8 @@ before the producer starts its final put. The final put completion is retained only as a transfer-stage diagnostic because its trace write can lose a scheduling race with the consumer. With `--enforce-targets`, the analyzer also checks per-run strict producer overlap, step-time p95, -eight-GPU NVML coverage, peak VRAM, token/multimodal-byte workload, and the +the configured expected-GPU NVML coverage, peak VRAM, +token/multimodal-byte workload, and the raw-reward, truncation-rate, and staleness guardrails. It uses `sum(step_tokens) / sum(step_time)` for aggregate throughput rather than averaging per-step rates. The truncation guardrail reads the rollout-side @@ -330,8 +343,9 @@ whose wall time falls inside the registered steady step intervals reconstructed from TensorBoard `perf/step_time`; sampled peak VRAM remains a full-run safety metric. The strict comparison additionally requires a clean run manifest, identical -candidate/image/TransferQueue identities, and non-empty input hashes, -dependency freeze, wheel hash, and launcher log for every run. +candidate/image/TransferQueue identities, verified static-input hashes, +dependency freeze, wheel hash, launcher log, and zero training/validation/final +exit-status artifacts for every run. The manifest also records and cross-checks `max_staleness`, global/rollout batch sizes, samples per prompt, and actor chunk count, so CLI expectations cannot silently disagree with the measured workload. @@ -340,6 +354,11 @@ identical model/config/data paths, prompt/response/context limits, actor token budget, actor/rollout resource topology, SGLang determinism and memory settings, physical-to-container GPU mapping, checkpoint mode, and debug capture/replay settings. Missing workload fields or any mismatch fail closed. +Paired global-index fingerprints must match exactly. When deterministic SGLang +inference is enabled, total, response, and multimodal-byte workloads must also +match exactly rather than within a tolerance. The comparison JSON reports the +mean, median, range, population standard deviation, and coefficient of +variation across paired repeats. The staleness curve is the trace-derived producer lead at the first actor forward: the largest completed producer rollout ID minus the actor rollout ID at that timestamp. The current rollout is considered ready once its actor fetch @@ -354,6 +373,12 @@ Before widening the support matrix, add collective-order and restore-count tests for the new DP/PP/VPP or role graph, then rerun frozen-input parity, multimodal smoke, and paired performance measurements. +`--steady-windows 0-0` deliberately measures a fresh-process first optimizer +step. It is useful when only one training step fits the fixed resource window, +but it is not a steady-state throughput claim. Label it as a paired first-step +benchmark, balance launch order across at least two seeds, and use later +multi-step windows whenever resources permit. + ______________________________________________________________________ ## Quick Start diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index 8be7343ae..34e330c1c 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -231,6 +231,7 @@ chunk 大小必须精确重建 optimizer mini，并且是 `n_samples_per_prompt` HYBRID_PIPELINE_FORWARD=1 \ HYBRID_PIPELINE_TRACE_DIR=/data01/LWX/relax-task21/runs/smoke/timeline \ HYBRID_PIPELINE_FETCH_TIMEOUT_S=600 \ +NUM_ITERS_PER_TRAIN_UPDATE=4 \ bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ hybrid-async ``` @@ -245,6 +246,7 @@ bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ | `MODEL_CHECKPOINT_DIR` / `REFERENCE_CHECKPOINT_DIR` | `${MODEL_DIR}/${MODEL_NAME}` | 显式固定 actor 与 reference 输入 | | `CHECKPOINT_SAVE` | `1` | 设为 `0` 时不传 `--save*`；改用独立 rollout 与 TensorBoard 输出目录 | | `ROLLOUT_RESULT_DIR` / `TENSORBOARD_DIR` | 保存开启时不覆盖；关闭时为 `${EXP_DIR}/rollout_result` 和 `${EXP_DIR}/tensorboard_log` | 在 no-save 运行中保留原始结果和标量；`TENSORBOARD_DIR` 会导出给 MetricsService | +| `NUM_ITERS_PER_TRAIN_UPDATE` | `2` | 设置每个 optimizer mini 中按完整 prompt group 对齐的 producer/actor 分块数 | | `ROLLOUT_MAX_RESPONSE_LEN` / `ROLLOUT_MAX_PROMPT_LEN` / `ROLLOUT_MAX_CONTEXT_LEN` | `10240` / `2048` / `12288` | 固定生成和上下文上限 | | `ACTOR_MAX_TOKENS_PER_GPU` | `12288` | 控制 dynamic-batch actor microbatch 的 token 上限 | | `HYBRID_ACTOR_GPUS` / `HYBRID_ROLLOUT_GPUS` | `4` / `4` | 构造 Hybrid placement resource | @@ -255,8 +257,15 @@ bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ 以及 actor GPU 数是当前 TP=2、CP=2 拓扑的整数倍。不满足约束会直接退出， 不会在 Ray worker 内静默降级。 -例如，以下命令用于资源受限环境中的 no-save Qwen3-VL-8B 功能 smoke； -它不是 8 卡 Qwen3.5 性能结果： +参考配置的 `global_batch_size=256`、`n_samples_per_prompt=8`。将 +`NUM_ITERS_PER_TRAIN_UPDATE` 设为 `4` 时，每个 optimizer mini 会形成四个 +64-sample stage，每个 stage 包含八组完整 prompt。成对比较中的 baseline 与 +experiment 必须使用同一个值：baseline 等完整 mini 到齐后统一 forward， +实验开关开启时则依次 fetch/forward 四个 stage。 + +例如，以下命令是资源受限环境中的 no-save Qwen3-VL-8B 配置，可用于明确 +标注资源条件的 smoke 或配对性能实验，但结果不能与默认 8 卡 Qwen3.5 配方 +合并： ```bash MODEL_CONFIG_FILE="${MODEL_CONFIG_DIR}/qwen3-vl-8B.sh" \ @@ -267,6 +276,7 @@ ROLLOUT_MAX_RESPONSE_LEN=512 \ ROLLOUT_MAX_PROMPT_LEN=2048 \ ROLLOUT_MAX_CONTEXT_LEN=2560 \ ACTOR_MAX_TOKENS_PER_GPU=6144 \ +NUM_ITERS_PER_TRAIN_UPDATE=4 \ HYBRID_ACTOR_GPUS=4 \ HYBRID_ROLLOUT_GPUS=1 \ ROLLOUT_NUM_GPUS_PER_ENGINE=1 \ @@ -309,7 +319,7 @@ fetch/forward 次数仍严格固定。严格 producer 重叠定义为首个 acto 早于 producer 最后一次 put 的开始时刻；最后一次 put 的完成时刻仅作为传输 阶段诊断，因为其 trace 写入可能在调度上晚于 consumer。`--enforce-targets` 还会检查每次实验的严格 producer 重叠比例、 -step-time p95、8 卡 NVML 覆盖、峰值显存、token/多模态字节工作量和 +step-time p95、命令指定数量的 GPU NVML 覆盖、峰值显存、token/多模态字节工作量和 raw-reward、截断率、staleness 非劣护栏。aggregate throughput 使用 `sum(step_tokens) / sum(step_time)`，不会对 per-step rate 做简单平均。 截断率护栏读取 rollout 侧的 `rollout/truncated_ratio`；训练侧 @@ -320,13 +330,17 @@ step 区间的 500 ms NVML 样本；区间由 TensorBoard `perf/step_time` 的 step end wall time 和 duration 重建。sampled peak VRAM 仍使用 full-run 安全口径。 严格比较还要求每次 run 的 manifest 为 clean tree，candidate/image/TransferQueue -身份完全一致，并存在非空的输入 hash、依赖 freeze、wheel hash 和 launcher log。 +身份完全一致，并存在已校验的静态输入 hash、依赖 freeze、wheel hash、 +launcher log，以及值为 0 的训练、验证和最终退出状态文件。 manifest 还会记录并交叉校验 `max_staleness`、global/rollout batch size、 每 prompt 样本数和 actor chunk 数，避免分析器 CLI 与实际工作负载静默不一致。 在计算任何配对统计前，分析器还要求模型、模型配置、数据路径、prompt/response/ context 上限、actor token 预算、actor/rollout 资源拓扑、SGLang 确定性与显存 配置、物理卡到容器卡映射、checkpoint 模式以及 debug 捕获/回放配置完全一致； -字段缺失或取值不同都会 fail closed。 +字段缺失或取值不同都会 fail closed。成对 run 的 global-index fingerprint +必须精确一致；启用 SGLang 确定性推理时，总 token、response token 与多模态 +tensor 字节数也必须精确一致，不使用容差。comparison JSON 同时报告重复实验 +的均值、中位数、范围、总体标准差和变异系数。 staleness 曲线来自 trace：在 actor 首次 forward 时，用已完成 put 的最大 producer rollout ID 减去当前 actor rollout ID；若 producer 的 trace 写入稍晚， 已完成的 actor fetch 本身可证明当前 rollout 已 ready。该值不得超过 manifest @@ -337,6 +351,11 @@ producer rollout ID 减去当前 actor rollout ID；若 producer 的 trace 写 DP/PP/VPP 或 role graph，必须先新增 collective-order 与 restore-count 测试， 再重跑 frozen-input parity、多模态 smoke 和成对性能实验。 +`--steady-windows 0-0` 明确统计“每次启动新进程后的第一个 optimizer step”。 +它适用于固定资源窗口只能容纳单步训练的场景，但不能表述为稳态吞吐。报告中 +应称为配对 first-step benchmark，至少使用两个 seed 并平衡启动顺序；资源允许 +时仍应补充后续多 step 的稳态窗口。 + ______________________________________________________________________ ## 快速开始 diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py index b1fc3b0ed..874ff0345 100644 --- a/scripts/tools/analyze_hybrid_pipeline_benchmark.py +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -141,6 +141,12 @@ "inputs.sha256", "transferqueue-wheel.sha256", "logs/launcher.log", + "manifests/static-input-verification.log", +) +EXIT_STATUS_ARTIFACTS = ( + "training_exit_status.txt", + "validation_exit_status.txt", + "exit_status.txt", ) @@ -308,6 +314,16 @@ def _require_reproducibility_artifacts(run_dir: Path) -> None: missing.append(relative_path) if missing: _fail(f"{run_dir} is missing non-empty reproducibility artifacts {missing}") + for relative_path in EXIT_STATUS_ARTIFACTS: + path = run_dir / relative_path + if not path.is_file(): + _fail(f"{run_dir} is missing exit-status artifact {relative_path}") + try: + status = path.read_text(encoding="utf-8").strip() + except OSError as exc: + _fail(f"cannot read {path}: {exc}") + if status != "0": + _fail(f"{run_dir} has non-zero or invalid {relative_path}: {status!r}") def _load_trace_rows(run_dir: Path) -> list[dict[str, Any]]: @@ -1037,6 +1053,23 @@ def _geometric_mean(values: Iterable[float]) -> float: return math.exp(statistics.fmean(math.log(value) for value in values)) +def _distribution_summary(values: Iterable[float]) -> dict[str, float | int]: + samples = list(values) + if not samples: + _fail("distribution summary requires at least one value") + mean = statistics.fmean(samples) + stddev = statistics.pstdev(samples) + return { + "count": len(samples), + "mean": mean, + "median": statistics.median(samples), + "min": min(samples), + "max": max(samples), + "population_stddev": stddev, + "coefficient_of_variation": stddev / abs(mean) if mean else 0.0, + } + + def _build_comparison( analyses: Sequence[RunAnalysis], *, @@ -1126,6 +1159,22 @@ def _build_comparison( window_speedups = [] for seed, pair in sorted(by_seed.items(), key=lambda item: str(item[0])): row: dict[str, Any] = {"seed": seed} + baseline_fingerprints = { + int(trace_row["rollout_id"]): trace_row["producer_global_indexes_fingerprint"] + for trace_row in pair["baseline"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + } + experiment_fingerprints = { + int(trace_row["rollout_id"]): trace_row["producer_global_indexes_fingerprint"] + for trace_row in pair["experiment"].trace_rows + if int(trace_row["rollout_id"]) in steady_steps + } + if baseline_fingerprints != experiment_fingerprints: + _fail( + f"seed {seed} producer global-index fingerprints differ between " + f"baseline and experiment: baseline={baseline_fingerprints}, " + f"experiment={experiment_fingerprints}" + ) for tag in COMPARISON_PERFORMANCE_TAGS: baseline_metrics = pair["baseline"].summary["metrics"].get(tag, {}) experiment_metrics = pair["experiment"].summary["metrics"].get(tag, {}) @@ -1268,6 +1317,24 @@ def _build_comparison( else None ), "experiment_steady_producer_overlap_ratio_by_run": experiment_producer_overlap_by_run, + "distributions": { + "baseline_step_token_per_s": _distribution_summary( + row["perf/step_token_per_s:baseline"] for row in paired if "perf/step_token_per_s:baseline" in row + ), + "experiment_step_token_per_s": _distribution_summary( + row["perf/step_token_per_s:experiment"] for row in paired if "perf/step_token_per_s:experiment" in row + ), + "paired_step_token_per_s_speedup": _distribution_summary( + row["perf/step_token_per_s:improvement"] + for row in paired + if "perf/step_token_per_s:improvement" in row + ), + "paired_hybrid_phase1_reduction": _distribution_summary( + row["perf/hybrid_phase1_time:improvement"] + for row in paired + if "perf/hybrid_phase1_time:improvement" in row + ), + }, } if enforce_targets: @@ -1313,6 +1380,7 @@ def _build_comparison( seed = row["seed"] if row["actor_fetch_samples:baseline"] != row["actor_fetch_samples:experiment"]: _fail(f"seed {seed} actor fetch sample count changed") + deterministic = int(by_seed[seed]["baseline"].manifest["sglang_deterministic_inference"]) == 1 for field in ( "actor_total_tokens", "actor_response_tokens", @@ -1323,7 +1391,12 @@ def _build_comparison( if baseline_total <= 0 or experiment_total <= 0: _fail(f"seed {seed} {field} must be positive, got {baseline_total}, {experiment_total}") relative_delta = row[f"{field}:relative_delta"] - if abs(relative_delta) > 0.01: + if deterministic and baseline_total != experiment_total: + _fail( + f"seed {seed} {field} must match exactly under deterministic " + f"inference, got {baseline_total} and {experiment_total}" + ) + if not deterministic and abs(relative_delta) > 0.01: _fail(f"seed {seed} {field} changed by more than 1%: {relative_delta:.4%}") vram_baseline = row.get("nvml_peak_memory_mib:baseline") diff --git a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh index 0d98f3d44..f590b9ad7 100644 --- a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh +++ b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh @@ -48,6 +48,7 @@ TENSORBOARD_DIR="${TENSORBOARD_DIR:-}" HYBRID_PIPELINE_FORWARD="${HYBRID_PIPELINE_FORWARD:-0}" HYBRID_PIPELINE_TRACE_DIR="${HYBRID_PIPELINE_TRACE_DIR:-}" HYBRID_PIPELINE_FETCH_TIMEOUT_S="${HYBRID_PIPELINE_FETCH_TIMEOUT_S:-600}" +NUM_ITERS_PER_TRAIN_UPDATE="${NUM_ITERS_PER_TRAIN_UPDATE:-2}" SGLANG_DETERMINISTIC_INFERENCE="${SGLANG_DETERMINISTIC_INFERENCE:-0}" SEED="${SEED:-}" ROLLOUT_SEED="${ROLLOUT_SEED:-}" @@ -118,6 +119,7 @@ fi for item in \ "NUM_ROLLOUT:${NUM_ROLLOUT}" \ "HYBRID_PIPELINE_FETCH_TIMEOUT_S:${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" \ + "NUM_ITERS_PER_TRAIN_UPDATE:${NUM_ITERS_PER_TRAIN_UPDATE}" \ "ROLLOUT_MAX_RESPONSE_LEN:${ROLLOUT_MAX_RESPONSE_LEN}" \ "ROLLOUT_MAX_PROMPT_LEN:${ROLLOUT_MAX_PROMPT_LEN}" \ "ROLLOUT_MAX_CONTEXT_LEN:${ROLLOUT_MAX_CONTEXT_LEN}" \ @@ -128,6 +130,10 @@ for item in \ "ROLLOUT_NUM_GPUS_PER_ENGINE:${ROLLOUT_NUM_GPUS_PER_ENGINE}"; do require_positive_integer "${item%%:*}" "${item#*:}" done +if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] && (( NUM_ITERS_PER_TRAIN_UPDATE < 2 )); then + echo "HYBRID_PIPELINE_FORWARD requires NUM_ITERS_PER_TRAIN_UPDATE >= 2" >&2 + exit 2 +fi if [ "${CHECKPOINT_SAVE}" = "1" ]; then require_positive_integer "CHECKPOINT_SAVE_INTERVAL" "${CHECKPOINT_SAVE_INTERVAL}" require_positive_integer "MAX_ACTOR_CKPT_TO_KEEP" "${MAX_ACTOR_CKPT_TO_KEEP}" @@ -230,6 +236,7 @@ printf '%s\n' \ "HYBRID_PIPELINE_FORWARD=${HYBRID_PIPELINE_FORWARD}" \ "HYBRID_PIPELINE_TRACE_DIR=${HYBRID_PIPELINE_TRACE_DIR}" \ "HYBRID_PIPELINE_FETCH_TIMEOUT_S=${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" \ + "NUM_ITERS_PER_TRAIN_UPDATE=${NUM_ITERS_PER_TRAIN_UPDATE}" \ "SEED=${SEED}" \ "ROLLOUT_SEED=${ROLLOUT_SEED}" \ "SGLANG_DETERMINISTIC_INFERENCE=${SGLANG_DETERMINISTIC_INFERENCE}" \ @@ -372,7 +379,7 @@ if [ "${MODE}" = "hybrid-async" ]; then --resource "${HYBRID_RESOURCE}" \ --max-staleness 2 \ --num-data-storage-units 1 \ - --num-iters-per-train-update 2 \ + --num-iters-per-train-update "${NUM_ITERS_PER_TRAIN_UPDATE}" \ --balance-data \ --hybrid \ "${HYBRID_PIPELINE_ARGS[@]}" \ diff --git a/tests/backends/megatron/test_data_vpp.py b/tests/backends/megatron/test_data_vpp.py index a4002b5a4..ae419ecbe 100644 --- a/tests/backends/megatron/test_data_vpp.py +++ b/tests/backends/megatron/test_data_vpp.py @@ -119,6 +119,29 @@ def test_hybrid_forward_chunk_plan_matches_producer_granularity(monkeypatch): ] == [0, 1, 2, 3, 4, 5] +def test_hybrid_forward_chunk_plan_supports_four_prompt_aligned_stages(monkeypatch): + data_module = _load_data_module(monkeypatch) + args = Namespace( + rollout_batch_size=32, + n_samples_per_prompt=8, + global_batch_size=256, + num_steps_per_rollout=None, + num_iters_per_train_update=4, + ) + rollout_plan = data_module.build_rollout_minibatch_plan(args, dp_size=1) + + chunk_plan = data_module.build_hybrid_forward_chunk_plan(args, rollout_plan, dp_size=1) + + assert chunk_plan.chunks_per_mini == 4 + assert chunk_plan.chunk_global_samples == 64 + assert chunk_plan.chunk_local_samples == 64 + assert [ + chunk_plan.transfer_queue_batch_index(mini_index, chunk_index) + for mini_index in range(2) + for chunk_index in range(chunk_plan.chunks_per_mini) + ] == list(range(8)) + + @pytest.mark.parametrize( ("num_iters", "global_batch_size", "n_samples_per_prompt", "error"), [ diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py index ad57896d4..ba56c0786 100644 --- a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -546,6 +546,7 @@ def _comparison_analysis( "actor_response_tokens": 5_000 + seed, "actor_multimodal_tensor_bytes": 1_000_000 + seed, "producer_lead_at_first_forward": 1.0, + "producer_global_indexes_fingerprint": f"{seed:032x}", } for rollout_id in range(4, 19) ] @@ -649,6 +650,8 @@ def test_preregistered_targets_pass_and_fail_deterministically(): assert comparison["hybrid_phase1_geomean_reduction"] == pytest.approx(0.2) assert comparison["experiment_steady_producer_overlap_ratio"] == 1.0 assert len(comparison["window_speedups"]) == 6 + assert comparison["distributions"]["paired_step_token_per_s_speedup"]["count"] == 2 + assert comparison["distributions"]["baseline_step_token_per_s"]["coefficient_of_variation"] > 0 failing = [ _comparison_analysis("baseline", 1, 100, 10), @@ -705,6 +708,21 @@ def test_comparison_rejects_mixed_or_incomplete_workload_manifests(): ) +def test_comparison_rejects_changed_global_index_fingerprint(): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + ] + analyses[1].trace_rows[0]["producer_global_indexes_fingerprint"] = "f" * 32 + + with pytest.raises(analyzer.BenchmarkValidationError, match="global-index fingerprints differ"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=False, + ) + + def test_comparison_plot_bundle_is_generated(tmp_path, monkeypatch): analyses = [ _comparison_analysis("baseline", 1, 100, 10), @@ -757,7 +775,7 @@ def test_comparison_plot_bundle_fails_fast_without_matplotlib(tmp_path, monkeypa ("staleness", "average producer-lead increase exceeds 0.25"), ("staleness_max", "observed producer lead exceeds configured max_staleness=2"), ("vram", "peak VRAM increased"), - ("workload", "actor_total_tokens changed by more than 1%"), + ("workload", "actor_total_tokens must match exactly"), ], ) def test_preregistered_guardrails_fail_closed(guardrail, error): @@ -792,3 +810,19 @@ def test_preregistered_guardrails_fail_closed(guardrail, error): windows=((4, 8), (9, 13), (14, 18)), enforce_targets=True, ) + + +def test_reproducibility_artifacts_require_all_zero_exit_statuses(tmp_path): + run_dir = tmp_path / "run" + for relative_path in analyzer.REPRODUCIBILITY_ARTIFACTS: + path = run_dir / relative_path + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("captured\n", encoding="utf-8") + for relative_path in analyzer.EXIT_STATUS_ARTIFACTS: + (run_dir / relative_path).write_text("0\n", encoding="utf-8") + + analyzer._require_reproducibility_artifacts(run_dir) + + (run_dir / "validation_exit_status.txt").write_text("2\n", encoding="utf-8") + with pytest.raises(analyzer.BenchmarkValidationError, match="non-zero or invalid"): + analyzer._require_reproducibility_artifacts(run_dir) diff --git a/tests/scripts/test_multimodal_hybrid_async_launcher.py b/tests/scripts/test_multimodal_hybrid_async_launcher.py index f496633db..e82e4c2f9 100644 --- a/tests/scripts/test_multimodal_hybrid_async_launcher.py +++ b/tests/scripts/test_multimodal_hybrid_async_launcher.py @@ -7,13 +7,7 @@ REPO_ROOT = Path(__file__).resolve().parents[2] -LAUNCHER = ( - REPO_ROOT - / "scripts" - / "training" - / "multimodal" - / "run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh" -) +LAUNCHER = REPO_ROOT / "scripts" / "training" / "multimodal" / "run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh" def _argument_value(arguments: list[str], flag: str) -> str: @@ -94,6 +88,7 @@ def test_launcher_preserves_default_qwen35_recipe(tmp_path): assert _argument_value(arguments, "--rollout-max-context-len") == "12288" assert _argument_value(arguments, "--max-tokens-per-gpu") == "12288" assert _argument_value(arguments, "--rollout-num-gpus-per-engine") == "2" + assert _argument_value(arguments, "--num-iters-per-train-update") == "2" assert _argument_value(arguments, "--save") == f"{tmp_path / 'exp' / 'Qwen3.5-9B_mcore_8xgpu'}/" assert _argument_value(arguments, "--save-interval") == "100" assert "--rollout-result-dir" not in arguments @@ -115,18 +110,18 @@ def test_launcher_supports_no_save_memory_safe_smoke_configuration(tmp_path): "ACTOR_MAX_TOKENS_PER_GPU": "6144", "HYBRID_ROLLOUT_GPUS": "1", "ROLLOUT_NUM_GPUS_PER_ENGINE": "1", + "NUM_ITERS_PER_TRAIN_UPDATE": "4", }, ) assert result.returncode == 0, result.stderr assert _argument_value(arguments, "--resource") == '{"actor": [1, 4], "rollout": [1, 1]}' - assert _argument_value(arguments, "--hf-checkpoint") == str( - tmp_path / "models" / "Qwen3-VL-8B-Instruct" - ) + assert _argument_value(arguments, "--hf-checkpoint") == str(tmp_path / "models" / "Qwen3-VL-8B-Instruct") assert _argument_value(arguments, "--rollout-max-response-len") == "512" assert _argument_value(arguments, "--rollout-max-context-len") == "2560" assert _argument_value(arguments, "--max-tokens-per-gpu") == "6144" assert _argument_value(arguments, "--rollout-num-gpus-per-engine") == "1" + assert _argument_value(arguments, "--num-iters-per-train-update") == "4" assert "--save" not in arguments assert "--save-interval" not in arguments assert _argument_value(arguments, "--rollout-result-dir") == str(tmp_path / "exp" / "rollout_result") @@ -150,3 +145,18 @@ def test_launcher_rejects_invalid_checkpoint_switch_before_ray(tmp_path): assert "CHECKPOINT_SAVE must be 0 or 1" in result.stderr assert arguments == [] assert not capture_path.exists() + + +def test_launcher_rejects_invalid_actor_chunk_count_before_ray(tmp_path): + result, arguments, capture_path, _ = _run_launcher( + tmp_path, + overrides={ + "HYBRID_PIPELINE_FORWARD": "1", + "NUM_ITERS_PER_TRAIN_UPDATE": "1", + }, + ) + + assert result.returncode == 2 + assert "requires NUM_ITERS_PER_TRAIN_UPDATE >= 2" in result.stderr + assert arguments == [] + assert not capture_path.exists() From 74c746fd7a7fc9da928d9081f15bd31eba821be3 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Thu, 30 Jul 2026 01:11:57 +0800 Subject: [PATCH 08/21] fix(benchmark): bind formal runs to producer stages and hardware Require the formal analyzer to observe four equal producer puts, and reject paired results captured on different hosts or GPU inventories. Constraint: The claimed optimization is a four-by-64 producer/actor pipeline measured on one fixed hardware topology. Rejected: Treat producer put grouping as diagnostic in the formal protocol | actor-only chunk evidence does not prove end-to-end staging. Confidence: high Scope-risk: narrow Directive: Pass --expected-producer-chunks only for protocols that pre-register producer grouping. Tested: 105 relevant pytest cases passed, 4 environment-dependent cases skipped; ruff, py_compile, bash -n, shellcheck warning gate, git diff --check. --- docs/en/guide/hybrid-training.md | 4 ++ docs/zh/guide/hybrid-training.md | 9 +++- .../analyze_hybrid_pipeline_benchmark.py | 48 +++++++++++++++++-- .../test_analyze_hybrid_pipeline_benchmark.py | 25 ++++++++++ 4 files changed, 80 insertions(+), 6 deletions(-) diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index 24e698b76..3580f2b73 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -349,11 +349,15 @@ exit-status artifacts for every run. The manifest also records and cross-checks `max_staleness`, global/rollout batch sizes, samples per prompt, and actor chunk count, so CLI expectations cannot silently disagree with the measured workload. +The registered four-stage protocol additionally requires exactly four producer +puts of 64 samples for every rollout; regrouped producer puts fail validation. Before any paired statistics are calculated, the analyzer also requires identical model/config/data paths, prompt/response/context limits, actor token budget, actor/rollout resource topology, SGLang determinism and memory settings, physical-to-container GPU mapping, checkpoint mode, and debug capture/replay settings. Missing workload fields or any mismatch fail closed. +The hostname and a SHA-256 fingerprint over GPU UUID, model, PCI address, and +driver version must also match across the comparison. Paired global-index fingerprints must match exactly. When deterministic SGLang inference is enabled, total, response, and multimodal-byte workloads must also match exactly rather than within a tolerance. The comparison JSON reports the diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index 34e330c1c..de175e5db 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -334,11 +334,16 @@ step end wall time 和 duration 重建。sampled peak VRAM 仍使用 full-run launcher log，以及值为 0 的训练、验证和最终退出状态文件。 manifest 还会记录并交叉校验 `max_staleness`、global/rollout batch size、 每 prompt 样本数和 actor chunk 数，避免分析器 CLI 与实际工作负载静默不一致。 +注册的四阶段协议还要求每个 rollout 恰好执行四次、每次 64 sample 的 producer +put；producer 重新分组会导致验证失败。 在计算任何配对统计前，分析器还要求模型、模型配置、数据路径、prompt/response/ context 上限、actor token 预算、actor/rollout 资源拓扑、SGLang 确定性与显存 配置、物理卡到容器卡映射、checkpoint 模式以及 debug 捕获/回放配置完全一致； -字段缺失或取值不同都会 fail closed。成对 run 的 global-index fingerprint -必须精确一致；启用 SGLang 确定性推理时，总 token、response token 与多模态 +字段缺失或取值不同都会 fail closed。 +此外还要求 hostname，以及由 GPU UUID、型号、PCI 地址和驱动版本生成的 +SHA-256 硬件指纹完全一致。 +成对 run 的 global-index fingerprint 必须精确一致；启用 SGLang 确定性推理时， +总 token、response token 与多模态 tensor 字节数也必须精确一致，不使用容差。comparison JSON 同时报告重复实验 的均值、中位数、范围、总体标准差和变异系数。 staleness 曲线来自 trace：在 actor 首次 forward 时，用已完成 put 的最大 diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py index 874ff0345..0e0e8c9c5 100644 --- a/scripts/tools/analyze_hybrid_pipeline_benchmark.py +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -73,6 +73,7 @@ "staleness", ) RUN_MANIFEST_REQUIRED_FIELDS = { + "hostname", "condition", "order", "seed", @@ -97,6 +98,7 @@ "entrypoint", } COMPARISON_FIXED_MANIFEST_FIELDS = ( + "hostname", "num_rollout", "max_staleness", "global_batch_size", @@ -129,6 +131,7 @@ "rollout_num_gpus_per_engine", "physical_gpu_indices", "container_cuda_visible_devices", + "gpu_hardware_fingerprint", "checkpoint_save", "sglang_deterministic_inference", "sglang_mem_fraction_static", @@ -277,7 +280,7 @@ def _validate_run_manifest( if trace_dir != expected_trace_dir: _fail(f"{run_dir} manifest trace directory is {trace_dir}, expected {expected_trace_dir}") - for key in ("condition", "order", "git_branch", "python", "entrypoint"): + for key in ("hostname", "condition", "order", "git_branch", "python", "entrypoint"): if not isinstance(manifest[key], str) or not manifest[key].strip(): _fail(f"{run_dir} manifest field {key!r} must be a non-empty string") @@ -304,6 +307,12 @@ def _validate_run_manifest( length=40, context=f"{run_dir} transferqueue_commit", ) + if "gpu_hardware_fingerprint" in manifest: + _validate_hex_digest( + manifest["gpu_hardware_fingerprint"], + length=64, + context=f"{run_dir} gpu_hardware_fingerprint", + ) def _require_reproducibility_artifacts(run_dir: Path) -> None: @@ -443,6 +452,7 @@ def _analyze_trace( pipeline_enabled: bool, expected_samples: int, expected_actor_chunks: int, + expected_producer_chunks: int | None, ) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]: producer_by_rollout: dict[int, list[dict[str, Any]]] = defaultdict(list) actor_by_rollout_stream: dict[tuple[int, tuple[str, int, int]], list[dict[str, Any]]] = defaultdict(list) @@ -480,6 +490,22 @@ def _analyze_trace( put_done = [end for _, end in put_pairs] if any(row["sample_count"] is None for row in put_done): _fail(f"rollout_id={rollout_id} producer put is missing sample_count") + if expected_producer_chunks is not None: + if expected_samples % expected_producer_chunks != 0: + _fail( + "expected_samples must be divisible by expected_producer_chunks, " + f"got {expected_samples=} and {expected_producer_chunks=}" + ) + expected_producer_chunk_samples = expected_samples // expected_producer_chunks + producer_chunk_samples = [int(row["sample_count"] or 0) for row in put_done] + if len(put_pairs) != expected_producer_chunks or any( + sample_count != expected_producer_chunk_samples for sample_count in producer_chunk_samples + ): + _fail( + f"rollout_id={rollout_id} expected {expected_producer_chunks} producer " + f"chunks of {expected_producer_chunk_samples} samples, got " + f"{producer_chunk_samples}" + ) producer_samples = sum(int(row["sample_count"]) for row in put_done) if producer_samples != expected_samples: _fail( @@ -969,6 +995,7 @@ def analyze_run( windows: Sequence[tuple[int, int]] = DEFAULT_WINDOWS, expected_samples: int = 256, expected_actor_chunks: int = 2, + expected_producer_chunks: int | None = None, write_outputs: bool = True, require_reproducibility_artifacts: bool = False, ) -> RunAnalysis: @@ -1004,6 +1031,7 @@ def analyze_run( pipeline_enabled=pipeline_enabled, expected_samples=expected_samples, expected_actor_chunks=expected_actor_chunks, + expected_producer_chunks=expected_producer_chunks, ) scalar_rows = _load_tensorboard_scalars(run_dir) nvml_rows = _parse_nvml_rows(run_dir) @@ -1661,7 +1689,13 @@ def build_parser() -> argparse.ArgumentParser: "--expected-actor-chunks", type=int, default=2, - help="Expected actor fetch/forward chunks when the pipeline is enabled; producer put grouping is dynamic.", + help="Expected actor fetch/forward chunks when the pipeline is enabled.", + ) + parser.add_argument( + "--expected-producer-chunks", + type=int, + default=None, + help="Require an exact producer put count and equal sample count per put.", ) parser.add_argument( "--expected-gpu-count", @@ -1691,8 +1725,13 @@ def main(argv: Sequence[str] | None = None) -> int: args = build_parser().parse_args(argv) try: windows = _parse_windows(args.steady_windows) - if args.expected_samples <= 0 or args.expected_actor_chunks <= 0 or args.expected_gpu_count <= 0: - _fail("expected sample, actor chunk, and GPU counts must be positive") + if ( + args.expected_samples <= 0 + or args.expected_actor_chunks <= 0 + or args.expected_gpu_count <= 0 + or (args.expected_producer_chunks is not None and args.expected_producer_chunks <= 0) + ): + _fail("expected sample, actor chunk, producer chunk, and GPU counts must be positive") if len(args.run_dir) > 1 and args.output_dir is None: _fail("--output-dir is required when comparing multiple runs") if args.validate_only and args.enforce_targets: @@ -1704,6 +1743,7 @@ def main(argv: Sequence[str] | None = None) -> int: windows=windows, expected_samples=args.expected_samples, expected_actor_chunks=args.expected_actor_chunks, + expected_producer_chunks=args.expected_producer_chunks, require_reproducibility_artifacts=args.enforce_targets, ) for run_dir in args.run_dir diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py index ba56c0786..21f30dc36 100644 --- a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -126,6 +126,7 @@ def _write_run( json.dumps( { "condition": "experiment" if pipeline_enabled else "baseline", + "hostname": hostname, "order": "B1" if pipeline_enabled else "A1", "seed": 7, "rollout_seed": 7, @@ -332,6 +333,28 @@ def test_producer_put_grouping_is_independent_from_actor_chunks(tmp_path, produc assert analysis.trace_rows[0]["actor_forward_count"] == 2 +def test_registered_producer_stage_shape_is_enforced(tmp_path): + run_dir = _write_run(tmp_path, producer_put_count=2) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + expected_producer_chunks=2, + ) + assert analysis.trace_rows[0]["producer_put_count"] == 2 + + with pytest.raises(analyzer.BenchmarkValidationError, match="expected 4 producer chunks"): + analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + expected_producer_chunks=4, + ) + + def test_delayed_put_done_is_not_strict_producer_overlap(tmp_path): run_dir = _write_run(tmp_path, producer_last_start_ns=150) @@ -566,6 +589,7 @@ def _comparison_analysis( return analyzer.RunAnalysis( run_dir=Path(f"/{condition}-{seed}"), manifest={ + "hostname": "test-host", "condition": condition, "order": ("A" if condition == "baseline" else "B") + str(seed), "seed": seed, @@ -601,6 +625,7 @@ def _comparison_analysis( "rollout_num_gpus_per_engine": 1, "physical_gpu_indices": [0, 1, 2, 3, 5], "container_cuda_visible_devices": [0, 1, 2, 3, 4], + "gpu_hardware_fingerprint": "9" * 64, "checkpoint_save": False, "sglang_deterministic_inference": 1, "sglang_mem_fraction_static": 0.8, From 2cd88e4962166ae6383921b632741c73805d4af9 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Thu, 30 Jul 2026 03:06:50 +0800 Subject: [PATCH 09/21] fix(hybrid): preserve policy numerics across chunk overlap Replay each chunk's dynamic microbatch schedule during the single merged optimizer step so batch-shape-dependent BF16 rounding cannot create a false PPO ratio before weights change. Constraint: Hybrid overlap remains limited to actor-only dynamic batching with one optimizer mini and TP2/CP2/DP1. Rejected: Recompute one full-batch training schedule | It changes packed kernel shapes relative to old-policy forward and produced artificial KL. Confidence: high Scope-risk: moderate Directive: Keep old-policy and training-forward microbatch grouping identical when extending supported topologies. Tested: 111 targeted pytest passed, 4 dependency-gated skipped; Ruff check/format, py_compile, bash -n, git diff --check. Not-tested: Full GPU frozen-rollout parity is run immediately after this commit. --- docs/en/guide/hybrid-training.md | 17 ++++- docs/zh/guide/hybrid-training.md | 14 +++- relax/backends/megatron/actor.py | 73 ++++++++++++++----- .../utils/training/hybrid_forward_pipeline.py | 68 ++++++++++++++++- .../test_hybrid_pipeline_actor_wiring.py | 35 ++++++++- tests/utils/test_hybrid_forward_pipeline.py | 32 ++++++++ 6 files changed, 212 insertions(+), 27 deletions(-) diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index 3580f2b73..7f74eb022 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -198,7 +198,10 @@ global-index fingerprint are conserved. The optional path: 2. fetches and forwards chunk 0 while rollout can continue producing chunk 1; 3. fetches and forwards chunk 1; 4. orders every per-sample field by `BatchMeta.global_indexes`; -5. computes advantages once over all 256 samples and performs one optimizer +5. records each chunk's dynamically balanced microbatch schedule, translates + it to the canonical merged-batch indexes, and replays the same sample + grouping and order during training; +6. computes advantages once over all 256 samples and performs one optimizer step. It does not change producer transfer policy, multimodal preprocessing, pixel @@ -206,6 +209,16 @@ tensor values, GRPO group boundaries, reward normalization, or optimizer semantics. The additional actor fetch is intended to expose rollout/actor overlap, not to reduce work. +The schedule replay is a correctness requirement, not a performance tuning +heuristic. Packed multimodal kernels can produce batch-shape-dependent BF16 +rounding differences. Comparing chunked old-policy log-probs with a differently +packed full-batch training forward would therefore create a non-zero PPO ratio +before any weight update. Replaying the exact chunk microbatches keeps the +old-policy and training forward shapes aligned while retaining one optimizer +update over the complete mini. For this reason, the switch requires +`--use-dynamic-batch-size`; an incompatible batch mode fails during actor +startup. + The first implementation is intentionally limited and fails fast instead of silently falling back: @@ -217,7 +230,7 @@ silently falling back: | Parallel topology | TP=2, DP=1, PP=1, VPP=1, CP=2, EP=1, ETP=1 | | Offload | `offload_train=False`, `offload_rollout=False` | | Dropout | Attention and hidden dropout both zero | -| Batch policy | Exactly one fixed optimizer mini per rollout (`rollout_batch_size * n_samples_per_prompt == global_batch_size`); no partial or dynamic-global batch | +| Batch policy | Dynamic microbatching; exactly one fixed optimizer mini per rollout (`rollout_batch_size * n_samples_per_prompt == global_batch_size`); no partial or dynamic-global batch | | Log-prob source | Actor-computed log-prob; no true-on-policy or rollout-log-prob shortcut | | TensorBackuper | Normal enabled backuper with only the `actor` tag | diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index de175e5db..5359f14b8 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -198,12 +198,22 @@ global-index fingerprint 完整守恒即可。打开开关后： 2. chunk 0 ready 后立即 fetch 和 forward，此时 rollout 可继续产生 chunk 1； 3. 再 fetch 和 forward chunk 1； 4. 按 `BatchMeta.global_indexes` 对所有 per-sample 字段恢复确定顺序； -5. 在完整 256 samples 上只计算一次 advantage，并只执行一次 optimizer step。 +5. 记录每个 chunk 的动态均衡 microbatch 调度，将其映射为合并 batch 中的 + canonical index，并在训练 forward/backward 中回放相同的样本分组与顺序； +6. 在完整 256 samples 上只计算一次 advantage，并只执行一次 optimizer step。 该路径不修改 producer 传输策略、多模态预处理、pixel tensor 数值、GRPO group 边界、reward normalization 或 optimizer 语义。多出的一次 actor fetch 用于暴露 rollout/actor 重叠窗口，而不是减少工作量。 +microbatch 调度回放是正确性约束，而不是性能调参。多模态 packed kernel +在 BF16 下可能因 batch shape 不同产生数值舍入差异；如果 old-policy +log-prob 按 chunk 计算、训练 forward 却按另一种完整 batch 分组，就会在 +权重尚未更新时产生非零 PPO ratio。回放完全相同的 chunk microbatch，可在 +保持整个 optimizer mini 只更新一次的同时，对齐 old-policy 与训练 forward +的计算形状。因此该开关强制要求 `--use-dynamic-batch-size`；batch 模式不兼容 +时会在 actor 启动阶段直接失败。 + 首版支持范围有意收窄；不支持的组合会 fail fast，不会静默回退： | 维度 | 打开开关时支持的范围 | @@ -214,7 +224,7 @@ group 边界、reward normalization 或 optimizer 语义。多出的一次 actor | 并行拓扑 | TP=2、DP=1、PP=1、VPP=1、CP=2、EP=1、ETP=1 | | Offload | `offload_train=False`、`offload_rollout=False` | | Dropout | attention/hidden dropout 均为 0 | -| Batch 策略 | 每次 rollout 恰好一个固定 optimizer mini（`rollout_batch_size * n_samples_per_prompt == global_batch_size`）；不支持 partial/dynamic-global batch | +| Batch 策略 | 使用 dynamic microbatch；每次 rollout 恰好一个固定 optimizer mini（`rollout_batch_size * n_samples_per_prompt == global_batch_size`）；不支持 partial/dynamic-global batch | | Log-prob 来源 | actor 计算；不支持 true-on-policy 或 rollout-log-prob 快捷路径 | | TensorBackuper | 启用普通 backuper，且只有 `actor` tag | diff --git a/relax/backends/megatron/actor.py b/relax/backends/megatron/actor.py index 21319dde3..a1f617677 100644 --- a/relax/backends/megatron/actor.py +++ b/relax/backends/megatron/actor.py @@ -64,6 +64,7 @@ from relax.utils.training import train_dump_utils from relax.utils.training.data_fields import build_data_fields from relax.utils.training.hybrid_forward_pipeline import ( + canonicalize_hybrid_microbatch_schedule, execute_hybrid_forward_mini, fetch_exact_chunk_with_timeout, ) @@ -1220,6 +1221,8 @@ def _validate_hybrid_pipeline_runtime(self, rollout_plan, dp_size: int): ) if not self.args.compute_advantages_and_returns: raise RuntimeError("--hybrid-pipeline-forward requires compute_advantages_and_returns=True") + if not self.args.use_dynamic_batch_size: + raise RuntimeError("--hybrid-pipeline-forward requires use_dynamic_batch_size=True") return build_hybrid_forward_chunk_plan(self.args, rollout_plan, dp_size) def _restore_hybrid_pipeline_actor( @@ -1261,7 +1264,7 @@ def _hybrid_actor_forward_without_switch( rollout_id: int, chunk_index: int, global_indexes: list[int], - ) -> None: + ) -> list[list[int]]: if self._active_model_tag != "actor": raise RuntimeError( "--hybrid-pipeline-forward requires the actor model to remain active before " @@ -1269,6 +1272,24 @@ def _hybrid_actor_forward_without_switch( ) data_iterator, num_microbatches = get_data_iterator(self.args, self.model, sub_batch) + if len(data_iterator) != 1 or len(num_microbatches) != 1: + raise RuntimeError( + "--hybrid-pipeline-forward expected one PP/VPP iterator and one optimizer step " + f"per chunk, got iterators={len(data_iterator)}, steps={num_microbatches}" + ) + microbatch_indices = data_iterator[0].micro_batch_indices + if microbatch_indices is None: + raise RuntimeError("--hybrid-pipeline-forward requires an explicit dynamic microbatch schedule") + forward_schedule = [list(indices) for indices in microbatch_indices] + flattened_schedule = [index for indices in forward_schedule for index in indices] + if len(forward_schedule) != num_microbatches[0] or sorted(flattened_schedule) != list( + range(len(sub_batch["total_lengths"])) + ): + raise RuntimeError( + "--hybrid-pipeline-forward produced an invalid chunk microbatch schedule: " + f"chunk_index={chunk_index}, microbatches={len(forward_schedule)}, " + f"expected_microbatches={num_microbatches[0]}, samples={len(flattened_schedule)}" + ) emit_hybrid_pipeline_event( self.args, "actor_forward_start", @@ -1292,6 +1313,7 @@ def _hybrid_actor_forward_without_switch( raise RuntimeError( f"--hybrid-pipeline-forward actor tag changed during chunk forward: {self._active_model_tag!r}" ) + return forward_schedule def _fetch_hybrid_pipeline_chunk( self, @@ -1494,6 +1516,21 @@ def phase1_timer(): collected_batches: list[RolloutBatch] = [] collected_global_indexes: list[int] = [] rollout_mini_local_sample_counts: list[int] = [] + pipeline_chunk_schedules: list[tuple[list[int], list[list[int]]]] = [] + + def forward_pipeline_chunk( + sub_batch: RolloutBatch, + tq_batch_index: int, + global_indexes: list[int], + ) -> None: + forward_schedule = self._hybrid_actor_forward_without_switch( + sub_batch, + rollout_id=rollout_id, + chunk_index=tq_batch_index, + global_indexes=global_indexes, + ) + pipeline_chunk_schedules.append((list(global_indexes), forward_schedule)) + if self.args.debug_train_only: # Bypass the transfer queue and load the offline debug rollout dump # directly (mirrors `train`'s debug_train_only path). The dump holds @@ -1524,14 +1561,7 @@ def fetch_debug_chunk(tq_batch_index): sample_count=batch_size, ), fetch_chunk=fetch_debug_chunk, - forward_chunk=lambda sub_batch, tq_batch_index, global_indexes: ( - self._hybrid_actor_forward_without_switch( - sub_batch, - rollout_id=rollout_id, - chunk_index=tq_batch_index, - global_indexes=global_indexes, - ) - ), + forward_chunk=forward_pipeline_chunk, ) mini_batch, canonical_indexes = canonicalize_rollout_chunks( mini_chunks, @@ -1594,14 +1624,7 @@ def fetch_pipeline_chunk(tq_batch_index): sample_count=batch_size, ), fetch_chunk=fetch_pipeline_chunk, - forward_chunk=lambda sub_batch, tq_batch_index, global_indexes: ( - self._hybrid_actor_forward_without_switch( - sub_batch, - rollout_id=rollout_id, - chunk_index=tq_batch_index, - global_indexes=global_indexes, - ) - ), + forward_chunk=forward_pipeline_chunk, ) mini_batch, canonical_indexes = canonicalize_rollout_chunks( @@ -1704,6 +1727,10 @@ def fetch_pipeline_chunk(tq_batch_index): # ── Phase 2: Merge sub-batches and compute advantages with correct global normalization ── if pipeline_enabled: rollout_data = concat_rollout_batches(collected_batches) + pipeline_train_microbatch_indices = canonicalize_hybrid_microbatch_schedule( + pipeline_chunk_schedules, + collected_global_indexes, + ) else: # Keep the flag-off merge path byte-for-byte compatible with the # pre-optimization implementation. @@ -1744,7 +1771,17 @@ def fetch_pipeline_chunk(tq_batch_index): log_rollout_data(rollout_id, self.args, rollout_data) # ── Phase 3: Train on the full merged batch ── - data_iterator, num_microbatches = get_data_iterator(self.args, self.model, rollout_data) + if pipeline_enabled: + data_iterator = [ + DataIterator( + rollout_data, + micro_batch_indices=pipeline_train_microbatch_indices, + max_tokens_per_gpu=self.args.max_tokens_per_gpu, + ) + ] + num_microbatches = [len(pipeline_train_microbatch_indices)] + else: + data_iterator, num_microbatches = get_data_iterator(self.args, self.model, rollout_data) if self.args.use_routing_replay: os.environ["ROUTING_REPLAY_STAGE"] = "replay_backward" with timer("actor_train"): diff --git a/relax/utils/training/hybrid_forward_pipeline.py b/relax/utils/training/hybrid_forward_pipeline.py index 8996d1921..d3ed029b0 100644 --- a/relax/utils/training/hybrid_forward_pipeline.py +++ b/relax/utils/training/hybrid_forward_pipeline.py @@ -2,7 +2,7 @@ import math import time -from collections.abc import Callable +from collections.abc import Callable, Sequence from typing import Any @@ -30,6 +30,72 @@ def execute_hybrid_forward_mini( return chunks +def canonicalize_hybrid_microbatch_schedule( + chunk_schedules: Sequence[tuple[Sequence[int], Sequence[Sequence[int]]]], + canonical_global_indexes: Sequence[int], +) -> list[list[int]]: + """Translate chunk-local forward schedules into merged-batch indexes. + + The actor old-logprob forward chooses its dynamic microbatches independently + for each producer chunk. Training must replay those exact sample groups and + their order; otherwise batch-shape-dependent numerics create an artificial + PPO ratio even though the weights did not change. + """ + canonical_indexes = list(canonical_global_indexes) + if not canonical_indexes: + raise ValueError("canonical_global_indexes must not be empty") + if not all(type(index) is int for index in canonical_indexes): + raise TypeError("canonical_global_indexes must contain only int values") + if len(set(canonical_indexes)) != len(canonical_indexes): + raise ValueError("canonical_global_indexes must not contain duplicates") + + canonical_positions = {index: position for position, index in enumerate(canonical_indexes)} + observed_global_indexes: list[int] = [] + merged_schedule: list[list[int]] = [] + + for chunk_index, (global_indexes, local_schedule) in enumerate(chunk_schedules): + chunk_global_indexes = list(global_indexes) + if not chunk_global_indexes: + raise ValueError(f"chunk {chunk_index} global_indexes must not be empty") + if not all(type(index) is int for index in chunk_global_indexes): + raise TypeError(f"chunk {chunk_index} global_indexes must contain only int values") + + flattened_local_indexes: list[int] = [] + for microbatch_index, local_indexes in enumerate(local_schedule): + normalized_local_indexes = list(local_indexes) + if not normalized_local_indexes: + raise ValueError(f"chunk {chunk_index} microbatch {microbatch_index} must not be empty") + if not all(type(index) is int for index in normalized_local_indexes): + raise TypeError(f"chunk {chunk_index} microbatch {microbatch_index} must contain only int indexes") + try: + merged_schedule.append( + [ + canonical_positions[chunk_global_indexes[local_index]] + for local_index in normalized_local_indexes + ] + ) + except IndexError as exc: + raise ValueError( + f"chunk {chunk_index} microbatch {microbatch_index} contains an out-of-range local index" + ) from exc + except KeyError as exc: + raise ValueError( + f"chunk {chunk_index} references global index {exc.args[0]} outside canonical_global_indexes" + ) from exc + flattened_local_indexes.extend(normalized_local_indexes) + + expected_local_indexes = list(range(len(chunk_global_indexes))) + if sorted(flattened_local_indexes) != expected_local_indexes: + raise ValueError(f"chunk {chunk_index} microbatch schedule must cover each local sample exactly once") + observed_global_indexes.extend(chunk_global_indexes) + + if sorted(observed_global_indexes) != sorted(canonical_indexes): + raise ValueError("chunk schedules must cover each canonical global index exactly once") + if sorted(index for microbatch in merged_schedule for index in microbatch) != list(range(len(canonical_indexes))): + raise ValueError("merged microbatch schedule must cover the merged batch exactly once") + return merged_schedule + + def fetch_exact_chunk_with_timeout( *, fetch_once: Callable[[], tuple[Any | None, Any]], diff --git a/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py index 3908198e1..096e4a360 100644 --- a/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py +++ b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py @@ -30,6 +30,7 @@ def test_hybrid_pipeline_runtime_rechecks_supported_parallel_topology(monkeypatc offload_train=False, offload_rollout=False, compute_advantages_and_returns=True, + use_dynamic_batch_size=True, ) actor.weights_backuper = SimpleNamespace(backup_tags={"actor"}) rollout_plan = SimpleNamespace( @@ -49,6 +50,11 @@ def test_hybrid_pipeline_runtime_rechecks_supported_parallel_topology(monkeypatc assert chunk_plan.chunks_per_mini == 2 assert chunk_plan.chunk_local_samples == 2 + actor.args.use_dynamic_batch_size = False + with pytest.raises(RuntimeError, match="requires use_dynamic_batch_size=True"): + actor._validate_hybrid_pipeline_runtime(rollout_plan, dp_size=1) + actor.args.use_dynamic_batch_size = True + monkeypatch.setattr(actor_module.mpu, "get_context_parallel_world_size", lambda: 1) with pytest.raises(RuntimeError, match="requires TP=2, CP=2, EP=1, and ETP=1"): actor._validate_hybrid_pipeline_runtime(rollout_plan, dp_size=1) @@ -155,9 +161,12 @@ def get_data(_task, _rollout_id, _fields, expected_samples, batch_index): actor._get_data_from_transfer_queue = get_data actor.all_consumed = lambda *_args, **_kwargs: False actor._restore_hybrid_pipeline_actor = lambda **kwargs: events.append(("restore", kwargs["chunk_index"])) - actor._hybrid_actor_forward_without_switch = lambda _batch, **kwargs: events.append( - ("forward", kwargs["chunk_index"]) - ) + + def forward_chunk(batch, **kwargs): + events.append(("forward", kwargs["chunk_index"])) + return [list(range(len(batch["total_lengths"])))] + + actor._hybrid_actor_forward_without_switch = forward_chunk actor._hybrid_forward_subbatch = lambda _batch, **kwargs: events.append(("forward", kwargs["chunk_index"])) actor._switch_model = lambda tag: events.append(("switch", tag)) actor._wait_for_previous_eval = lambda: None @@ -182,7 +191,18 @@ def get_data(_task, _rollout_id, _fields, expected_samples, batch_index): ) monkeypatch.setattr(actor_module, "log_rollout_data", lambda *_args, **_kwargs: None) monkeypatch.setattr(actor_module, "get_data_iterator", lambda *_args, **_kwargs: ([], [])) - monkeypatch.setattr(actor_module, "train", lambda *_args, **_kwargs: events.append("optimizer")) + + def train(_rollout_id, _model, _optimizer, _scheduler, data_iterator, num_microbatches): + events.append( + ( + "optimizer_schedule", + data_iterator[0].micro_batch_indices if pipeline_enabled else None, + num_microbatches, + ) + ) + events.append("optimizer") + + monkeypatch.setattr(actor_module, "train", train) monkeypatch.setattr(actor_module.train_dump_utils, "save_debug_train_data", lambda *_args, **_kwargs: None) monkeypatch.setattr(actor_module, "Timer", lambda: SimpleNamespace(seq_lens=None)) monkeypatch.setattr(actor_module, "log_perf_data", lambda *_args, **_kwargs: None) @@ -204,3 +224,10 @@ def all_gather_object(output, value, **_kwargs): assert events.count("advantages") == 1 assert events.count("optimizer") == 1 assert events.count("update_weights") == 1 + optimizer_schedule = next( + event for event in events if isinstance(event, tuple) and event[0] == "optimizer_schedule" + ) + if pipeline_enabled: + assert optimizer_schedule[1:] == ([[0, 1], [2, 3]], [2]) + else: + assert optimizer_schedule[1:] == (None, []) diff --git a/tests/utils/test_hybrid_forward_pipeline.py b/tests/utils/test_hybrid_forward_pipeline.py index ec382265d..d3f4428be 100644 --- a/tests/utils/test_hybrid_forward_pipeline.py +++ b/tests/utils/test_hybrid_forward_pipeline.py @@ -3,6 +3,7 @@ import pytest from relax.utils.training.hybrid_forward_pipeline import ( + canonicalize_hybrid_microbatch_schedule, execute_hybrid_forward_mini, fetch_exact_chunk_with_timeout, ) @@ -64,6 +65,37 @@ def test_restore_occurs_once_per_optimizer_mini(): assert restore_calls == [0, 2, 4] +def test_chunk_microbatch_schedule_is_replayed_on_canonical_batch(): + schedule = canonicalize_hybrid_microbatch_schedule( + [ + ([12, 10], [[1], [0]]), + ([13, 11], [[0, 1]]), + ], + [10, 11, 12, 13], + ) + + assert schedule == [[0], [2], [3, 1]] + + +@pytest.mark.parametrize( + ("chunk_schedules", "canonical_indexes", "error"), + [ + ([([10, 11], [[0]])], [10, 11], "cover each local sample exactly once"), + ([([10, 11], [[0, 0], [1]])], [10, 11], "cover each local sample exactly once"), + ([([10, 11], [[0, 2]])], [10, 11], "out-of-range local index"), + ([([10], [[0]])], [10, 11], "cover each canonical global index exactly once"), + ([([10, 12], [[0, 1]])], [10, 11], "outside canonical_global_indexes"), + ], +) +def test_chunk_microbatch_schedule_rejects_incomplete_or_invalid_coverage( + chunk_schedules, + canonical_indexes, + error, +): + with pytest.raises(ValueError, match=error): + canonicalize_hybrid_microbatch_schedule(chunk_schedules, canonical_indexes) + + def test_invalid_chunk_count_fails_before_restore(): restore_calls = [] From 320f5a95c72270222608142a8c7d4733be8988fd Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Thu, 30 Jul 2026 03:30:47 +0800 Subject: [PATCH 10/21] feat(hybrid): isolate producer overlap in matched benchmarks Add a no-overlap control that fetches every identical actor chunk before forwarding while preserving chunk-local dynamic schedules and one merged optimizer update. Benchmark validation now proves the requested ordering and rejects same-weight PPO KL or clipping drift. Constraint: Formal Task 21 comparisons must differ only by hybrid_pipeline_overlap; both arms use chunk forwarding. Rejected: Use the legacy full-batch flag-off path as the performance baseline | Its different packed microbatch shapes confound overlap with BF16 numerical and kernel-shape effects. Confidence: high Scope-risk: moderate Directive: Keep the no-overlap arm schedule-identical whenever chunk execution or topology support changes. Tested: 119 targeted pytest passed, 5 dependency-gated skipped; Ruff check/format, py_compile, bash -n, shellcheck error-level, git diff --check. Not-tested: Frozen GPU no-overlap versus overlap parity and formal paired throughput runs follow this commit. --- docs/en/guide/hybrid-training.md | 31 ++++- docs/zh/guide/hybrid-training.md | 27 ++++- relax/backends/megatron/actor.py | 2 + relax/utils/arguments.py | 20 +++- .../utils/training/hybrid_forward_pipeline.py | 21 ++-- .../analyze_hybrid_pipeline_benchmark.py | 70 ++++++++++- ...n-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh | 16 +++ .../test_hybrid_pipeline_actor_wiring.py | 16 ++- .../test_analyze_hybrid_pipeline_benchmark.py | 112 +++++++++++++++++- .../test_multimodal_hybrid_async_launcher.py | 26 ++++ .../test_arguments_opd_teacher_colocate.py | 15 +++ tests/utils/test_hybrid_forward_pipeline.py | 29 +++++ 12 files changed, 355 insertions(+), 30 deletions(-) diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index 7f74eb022..7681ef39f 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -182,6 +182,7 @@ instead of waiting for a complete optimizer mini: | Option | Default | Purpose | | --- | --- | --- | | `--hybrid-pipeline-forward` | off | Fetch and forward each fixed actor chunk as soon as enough samples are ready | +| `--hybrid-pipeline-overlap` / `--no-hybrid-pipeline-overlap` | on | Forward immediately, or fetch all identical chunks first for a schedule-matched performance control | | `--hybrid-pipeline-trace-dir PATH` | unset | Write content-free producer, fetch, restore, forward, advantage, and optimizer events | | `--hybrid-pipeline-fetch-timeout-s SECONDS` | `600` | Fail an incomplete chunk wait with rollout/mini/chunk context | @@ -219,6 +220,14 @@ update over the complete mini. For this reason, the switch requires `--use-dynamic-batch-size`; an incompatible batch mode fails during actor startup. +`--no-hybrid-pipeline-overlap` changes only the ordering of the same chunk +operations: the actor fetches every chunk before starting any chunk forward. +It preserves the chunk-local dynamic schedules and their single merged +optimizer update. This is the registered baseline for a causal performance +comparison. Omitting `--hybrid-pipeline-forward` remains the compatibility +rollback, but its full-batch packing is not a schedule-matched performance +control. + The first implementation is intentionally limited and fails fast instead of silently falling back: @@ -264,6 +273,7 @@ run or paired benchmark does not require editing the script: | `CHECKPOINT_SAVE` | `1` | Set to `0` to omit all `--save*` arguments and use separate rollout/TensorBoard outputs | | `ROLLOUT_RESULT_DIR` / `TENSORBOARD_DIR` | No override while saving; `${EXP_DIR}/rollout_result` and `${EXP_DIR}/tensorboard_log` without saving | Preserve raw results and scalars in no-save runs; `TENSORBOARD_DIR` is exported for MetricsService | | `NUM_ITERS_PER_TRAIN_UPDATE` | `2` | Set the prompt-group-aligned producer and actor chunk count for each optimizer mini | +| `HYBRID_PIPELINE_OVERLAP` | `1` | Set to `0` only with `HYBRID_PIPELINE_FORWARD=1` to run the schedule-matched no-overlap control | | `ROLLOUT_MAX_RESPONSE_LEN` / `ROLLOUT_MAX_PROMPT_LEN` / `ROLLOUT_MAX_CONTEXT_LEN` | `10240` / `2048` / `12288` | Pin generation and context limits | | `ACTOR_MAX_TOKENS_PER_GPU` | `12288` | Bound dynamic-batch actor microbatch tokens | | `HYBRID_ACTOR_GPUS` / `HYBRID_ROLLOUT_GPUS` | `4` / `4` | Build the Hybrid placement resource | @@ -278,9 +288,10 @@ start instead of silently degrading. With the reference `global_batch_size=256` and `n_samples_per_prompt=8`, setting `NUM_ITERS_PER_TRAIN_UPDATE=4` creates four 64-sample stages. Each stage therefore contains eight complete prompt groups. -Baseline and experiment runs in a pair must use the same value: the baseline -waits for and forwards the full mini, while the optional pipeline fetches and -forwards the four stages incrementally. +Baseline and experiment runs in a strict pair both enable chunk forwarding and +use the same value. The baseline sets `HYBRID_PIPELINE_OVERLAP=0`, so it fetches +all four stages before their forwards; the experiment sets it to `1`, so each +ready stage is forwarded while later stages are still produced. For example, this is a no-save Qwen3-VL-8B configuration for a constrained machine. It may be used for a resource-qualified smoke or paired performance @@ -335,6 +346,12 @@ python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ --enforce-targets ``` +For every paired seed, launch the baseline with +`HYBRID_PIPELINE_FORWARD=1 HYBRID_PIPELINE_OVERLAP=0` and the experiment with +`HYBRID_PIPELINE_FORWARD=1 HYBRID_PIPELINE_OVERLAP=1`. All other manifest +fields, including candidate commit, model, data, limits, topology, seed, image, +and hardware fingerprint, must match. + The analyzer validates event closure, one restore and optimizer step, sample conservation, same-host monotonic timing, finite metrics, producer/fetch fingerprints, steady-step coverage, and the registered performance thresholds. @@ -343,10 +360,14 @@ strictly fixed. Strict producer overlap means the first actor forward starts before the producer starts its final put. The final put completion is retained only as a transfer-stage diagnostic because its trace write can lose a scheduling race with the consumer. With `--enforce-targets`, the analyzer also -checks per-run strict producer overlap, step-time p95, +checks that the baseline fetched all chunks before its first forward, that the +experiment forwarded before its final fetch completed, per-run strict producer +overlap, step-time p95, the configured expected-GPU NVML coverage, peak VRAM, token/multimodal-byte workload, and the -raw-reward, truncation-rate, and staleness guardrails. It uses `sum(step_tokens) / +raw-reward, truncation-rate, staleness, same-weight PPO KL, and policy +clip-fraction guardrails. Same-weight `abs(train/ppo_kl)` and +`abs(train/pg_clipfrac)` must each remain at or below `1e-7`. It uses `sum(step_tokens) / sum(step_time)` for aggregate throughput rather than averaging per-step rates. The truncation guardrail reads the rollout-side `rollout/truncated_ratio`; the training-side `rollout/truncated` scalar is not diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index 5359f14b8..b2dc33361 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -184,6 +184,7 @@ Hybrid 可选地从 TransferQueue 按固定 sample count 增量请求 actor chun | 参数 | 默认值 | 作用 | | --- | --- | --- | | `--hybrid-pipeline-forward` | 关闭 | 足量 sample ready 后立即 fetch 固定 actor chunk 并执行 forward | +| `--hybrid-pipeline-overlap` / `--no-hybrid-pipeline-overlap` | 开启 | chunk ready 后立即 forward；或先 fetch 全部相同 chunk，作为调度匹配的性能对照 | | `--hybrid-pipeline-trace-dir PATH` | 未设置 | 记录不含样本内容的 producer、fetch、restore、forward、advantage 和 optimizer 事件 | | `--hybrid-pipeline-fetch-timeout-s SECONDS` | `600` | chunk 未完整到达时，带 rollout/mini/chunk 上下文终止等待 | @@ -214,6 +215,12 @@ log-prob 按 chunk 计算、训练 forward 却按另一种完整 batch 分组， 的计算形状。因此该开关强制要求 `--use-dynamic-batch-size`；batch 模式不兼容 时会在 actor 启动阶段直接失败。 +`--no-hybrid-pipeline-overlap` 只改变同一组 chunk 操作的执行顺序：actor +先 fetch 完所有 chunk，再开始任何 chunk forward。chunk 内动态 microbatch +调度及合并后的单次 optimizer update 均保持不变，因此它是因果性能比较的 +注册 baseline。完全去掉 `--hybrid-pipeline-forward` 仍是兼容性回滚方式， +但 full-batch packing 不属于调度匹配的性能对照。 + 首版支持范围有意收窄；不支持的组合会 fail fast，不会静默回退： | 维度 | 打开开关时支持的范围 | @@ -257,6 +264,7 @@ bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ | `CHECKPOINT_SAVE` | `1` | 设为 `0` 时不传 `--save*`；改用独立 rollout 与 TensorBoard 输出目录 | | `ROLLOUT_RESULT_DIR` / `TENSORBOARD_DIR` | 保存开启时不覆盖；关闭时为 `${EXP_DIR}/rollout_result` 和 `${EXP_DIR}/tensorboard_log` | 在 no-save 运行中保留原始结果和标量；`TENSORBOARD_DIR` 会导出给 MetricsService | | `NUM_ITERS_PER_TRAIN_UPDATE` | `2` | 设置每个 optimizer mini 中按完整 prompt group 对齐的 producer/actor 分块数 | +| `HYBRID_PIPELINE_OVERLAP` | `1` | 仅在 `HYBRID_PIPELINE_FORWARD=1` 时设为 `0`，运行调度匹配的 no-overlap 对照 | | `ROLLOUT_MAX_RESPONSE_LEN` / `ROLLOUT_MAX_PROMPT_LEN` / `ROLLOUT_MAX_CONTEXT_LEN` | `10240` / `2048` / `12288` | 固定生成和上下文上限 | | `ACTOR_MAX_TOKENS_PER_GPU` | `12288` | 控制 dynamic-batch actor microbatch 的 token 上限 | | `HYBRID_ACTOR_GPUS` / `HYBRID_ROLLOUT_GPUS` | `4` / `4` | 构造 Hybrid placement resource | @@ -269,9 +277,10 @@ bash scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh \ 参考配置的 `global_batch_size=256`、`n_samples_per_prompt=8`。将 `NUM_ITERS_PER_TRAIN_UPDATE` 设为 `4` 时，每个 optimizer mini 会形成四个 -64-sample stage，每个 stage 包含八组完整 prompt。成对比较中的 baseline 与 -experiment 必须使用同一个值：baseline 等完整 mini 到齐后统一 forward， -实验开关开启时则依次 fetch/forward 四个 stage。 +64-sample stage，每个 stage 包含八组完整 prompt。严格配对中的 baseline 与 +experiment 都启用 chunk forward 并使用相同分块数。baseline 设置 +`HYBRID_PIPELINE_OVERLAP=0`，先 fetch 完四个 stage 再依次 forward；experiment +设置为 `1`，每个 stage ready 后立即 forward，并与后续 stage 的生成重叠。 例如，以下命令是资源受限环境中的 no-save Qwen3-VL-8B 配置，可用于明确 标注资源条件的 smoke 或配对性能实验，但结果不能与默认 8 卡 Qwen3.5 配方 @@ -310,6 +319,11 @@ python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ --validate-only ``` +每个配对 seed 中，baseline 使用 +`HYBRID_PIPELINE_FORWARD=1 HYBRID_PIPELINE_OVERLAP=0`，experiment 使用 +`HYBRID_PIPELINE_FORWARD=1 HYBRID_PIPELINE_OVERLAP=1`。candidate commit、 +模型、数据、长度上限、拓扑、seed、镜像及硬件指纹等其余 manifest 字段必须一致。 + 比较两次 baseline 和两次 experiment，并生成注册的 CSV、JSON 与曲线： ```bash @@ -328,9 +342,12 @@ python scripts/tools/analyze_hybrid_pipeline_benchmark.py \ fetch/forward 次数仍严格固定。严格 producer 重叠定义为首个 actor forward 早于 producer 最后一次 put 的开始时刻；最后一次 put 的完成时刻仅作为传输 阶段诊断，因为其 trace 写入可能在调度上晚于 consumer。`--enforce-targets` -还会检查每次实验的严格 producer 重叠比例、 +还会检查 baseline 在首次 forward 前已 fetch 完全部 chunk、experiment 在 +最后一次 fetch 完成前已开始 forward、每次实验的严格 producer 重叠比例、 step-time p95、命令指定数量的 GPU NVML 覆盖、峰值显存、token/多模态字节工作量和 -raw-reward、截断率、staleness 非劣护栏。aggregate throughput 使用 +raw-reward、截断率、staleness、同权重 PPO KL 与 policy clip fraction 护栏。 +同权重 `abs(train/ppo_kl)` 与 `abs(train/pg_clipfrac)` 都必须不超过 `1e-7`。 +aggregate throughput 使用 `sum(step_tokens) / sum(step_time)`，不会对 per-step rate 做简单平均。 截断率护栏读取 rollout 侧的 `rollout/truncated_ratio`；训练侧 `rollout/truncated` 在分块聚合时可能被一个 step 累加多次，因此不作为截断率 diff --git a/relax/backends/megatron/actor.py b/relax/backends/megatron/actor.py index a1f617677..2ea99c2d9 100644 --- a/relax/backends/megatron/actor.py +++ b/relax/backends/megatron/actor.py @@ -1562,6 +1562,7 @@ def fetch_debug_chunk(tq_batch_index): ), fetch_chunk=fetch_debug_chunk, forward_chunk=forward_pipeline_chunk, + overlap_producer=bool(getattr(self.args, "hybrid_pipeline_overlap", True)), ) mini_batch, canonical_indexes = canonicalize_rollout_chunks( mini_chunks, @@ -1625,6 +1626,7 @@ def fetch_pipeline_chunk(tq_batch_index): ), fetch_chunk=fetch_pipeline_chunk, forward_chunk=forward_pipeline_chunk, + overlap_producer=bool(getattr(self.args, "hybrid_pipeline_overlap", True)), ) mini_batch, canonical_indexes = canonicalize_rollout_chunks( diff --git a/relax/utils/arguments.py b/relax/utils/arguments.py index 353d4f813..cc8b9af91 100644 --- a/relax/utils/arguments.py +++ b/relax/utils/arguments.py @@ -237,6 +237,16 @@ def add_serve_arguments(parser): "Default: disabled." ), ) + parser.add_argument( + "--hybrid-pipeline-overlap", + action=argparse.BooleanOptionalAction, + default=True, + help=( + "When Hybrid pipeline forwarding is enabled, forward each chunk immediately " + "instead of waiting until all chunks have been fetched. Disable only to run a " + "schedule-matched no-overlap performance control. Default: enabled." + ), + ) parser.add_argument( "--hybrid-pipeline-trace-dir", type=str, @@ -2842,10 +2852,16 @@ def _normalize_sync_ppo_kl_args(args) -> bool: def _validate_hybrid_pipeline_args(args) -> None: enabled = bool(getattr(args, "hybrid_pipeline_forward", False)) + overlap_enabled = bool(getattr(args, "hybrid_pipeline_overlap", True)) trace_enabled = bool(getattr(args, "hybrid_pipeline_trace_dir", None)) - if (enabled or trace_enabled) and not getattr(args, "hybrid", False): - raise ValueError("--hybrid-pipeline-forward and --hybrid-pipeline-trace-dir are only supported with --hybrid.") + if (enabled or trace_enabled or not overlap_enabled) and not getattr(args, "hybrid", False): + raise ValueError( + "--hybrid-pipeline-forward, its overlap control, and --hybrid-pipeline-trace-dir " + "are only supported with --hybrid." + ) if not enabled: + if not overlap_enabled: + raise ValueError("--no-hybrid-pipeline-overlap requires --hybrid-pipeline-forward.") return timeout = getattr(args, "hybrid_pipeline_fetch_timeout_s", 600.0) diff --git a/relax/utils/training/hybrid_forward_pipeline.py b/relax/utils/training/hybrid_forward_pipeline.py index d3ed029b0..4a4b902ba 100644 --- a/relax/utils/training/hybrid_forward_pipeline.py +++ b/relax/utils/training/hybrid_forward_pipeline.py @@ -13,20 +13,27 @@ def execute_hybrid_forward_mini( restore_actor: Callable[[int], None], fetch_chunk: Callable[[int], tuple[Any, list[int]]], forward_chunk: Callable[[Any, int, list[int]], None], + overlap_producer: bool = True, ) -> list[tuple[Any, list[int]]]: - """Restore once, then fetch and forward each fixed actor chunk in order.""" + """Restore once, then execute a matched chunk schedule with optional overlap.""" if chunks_per_mini <= 0: raise ValueError(f"chunks_per_mini must be positive, got {chunks_per_mini}") + if type(overlap_producer) is not bool: + raise TypeError(f"overlap_producer must be bool, got {overlap_producer!r}") - first_batch_index = batch_index_for_chunk(0) - restore_actor(first_batch_index) + batch_indexes = [batch_index_for_chunk(chunk_index) for chunk_index in range(chunks_per_mini)] + restore_actor(batch_indexes[0]) - chunks = [] - for chunk_index in range(chunks_per_mini): - batch_index = batch_index_for_chunk(chunk_index) + chunks: list[tuple[Any, list[int]]] = [] + for batch_index in batch_indexes: batch, global_indexes = fetch_chunk(batch_index) - forward_chunk(batch, batch_index, global_indexes) chunks.append((batch, global_indexes)) + if overlap_producer: + forward_chunk(batch, batch_index, global_indexes) + + if not overlap_producer: + for (batch, global_indexes), batch_index in zip(chunks, batch_indexes, strict=True): + forward_chunk(batch, batch_index, global_indexes) return chunks diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py index 0e0e8c9c5..3e2c3b9ea 100644 --- a/scripts/tools/analyze_hybrid_pipeline_benchmark.py +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -62,6 +62,8 @@ "rollout/truncated_ratio", "train/loss", "train/grad_norm", + "train/ppo_kl", + "train/pg_clipfrac", ) QUALITY_TAG_FRAGMENTS = ( "raw_reward", @@ -236,6 +238,13 @@ def _validate_run_manifest( ): if type(manifest[key]) is not int: _fail(f"{run_dir} manifest field {key!r} must be an integer, got {manifest[key]!r}") + if "hybrid_pipeline_overlap" in manifest and ( + type(manifest["hybrid_pipeline_overlap"]) is not int or manifest["hybrid_pipeline_overlap"] not in (0, 1) + ): + _fail( + f"{run_dir} manifest field 'hybrid_pipeline_overlap' must be integer 0 or 1, " + f"got {manifest['hybrid_pipeline_overlap']!r}" + ) if manifest["seed"] != manifest["rollout_seed"]: _fail( f"{run_dir} must use the same paired Megatron/rollout seed, got " @@ -450,6 +459,7 @@ def _analyze_trace( rows: Sequence[dict[str, Any]], *, pipeline_enabled: bool, + pipeline_overlap_enabled: bool, expected_samples: int, expected_actor_chunks: int, expected_producer_chunks: int | None, @@ -622,8 +632,17 @@ def _analyze_trace( ) last_forward_ns = max(row["monotonic_ns"] for row in forward_end) first_forward_ns = min(row["monotonic_ns"] for row in forward_start) + last_fetch_end_ns = max(row["monotonic_ns"] for row in fetch_end) last_put_start_ns = max(row["monotonic_ns"] for row in put_start) last_put_done_ns = max(row["monotonic_ns"] for row in put_done) + chunk_schedule_overlapped = first_forward_ns < last_fetch_end_ns + if pipeline_enabled and chunk_schedule_overlapped != pipeline_overlap_enabled: + expected_order = ( + "the first forward before the final fetch completed" + if pipeline_overlap_enabled + else "all fetches before the first forward" + ) + _fail(f"{context} does not implement the requested overlap mode: expected {expected_order}") ready_rollout_ids = [ int(row["rollout_id"]) for row in rows @@ -643,6 +662,7 @@ def _analyze_trace( "pid": stream[1], "global_rank": stream[2], "pipeline_enabled": pipeline_enabled, + "pipeline_overlap_enabled": pipeline_overlap_enabled, "producer_samples": producer_samples, "actor_fetch_samples": sum(int(row["sample_count"]) for row in fetch_end), "actor_forward_samples": sum(int(row["sample_count"]) for row in forward_start), @@ -659,6 +679,8 @@ def _analyze_trace( # Diagnostic only: async_put may have made data visible # before the producer coroutine records tq_put_done. "first_forward_before_last_put_done": first_forward_ns < last_put_done_ns, + "first_forward_before_last_fetch_end": chunk_schedule_overlapped, + "all_chunks_fetched_before_first_forward": not chunk_schedule_overlapped, "producer_overlap_s": max(0, last_put_start_ns - first_forward_ns) / 1e9, "transfer_overlap_s": max(0, last_put_done_ns - first_forward_ns) / 1e9, "producer_lead_at_first_forward": producer_lead, @@ -706,15 +728,19 @@ def _analyze_trace( producer_overlap_count = sum(bool(row["first_forward_before_last_put_start"]) for row in rollout_rows) transfer_overlap_count = sum(bool(row["first_forward_before_last_put_done"]) for row in rollout_rows) + chunk_schedule_overlap_count = sum(bool(row["first_forward_before_last_fetch_end"]) for row in rollout_rows) summary = { "hostname": next(iter({row["hostname"] for row in rows})), "pipeline_enabled": pipeline_enabled, + "pipeline_overlap_enabled": pipeline_overlap_enabled, "rollout_count": len(rollout_rows), "actor_stream_count": len({_stream_key(row) for row in rows if row["role"] == "actor"}), "producer_overlap_rollout_count": producer_overlap_count, "producer_overlap_rollout_ratio": producer_overlap_count / len(rollout_rows), "transfer_overlap_rollout_count": transfer_overlap_count, "transfer_overlap_rollout_ratio": transfer_overlap_count / len(rollout_rows), + "chunk_schedule_overlap_rollout_count": chunk_schedule_overlap_count, + "chunk_schedule_overlap_rollout_ratio": chunk_schedule_overlap_count / len(rollout_rows), "mean_phase1_s": statistics.fmean(row["phase1_s"] for row in rollout_rows), "mean_producer_overlap_s": statistics.fmean(row["producer_overlap_s"] for row in rollout_rows), "mean_transfer_overlap_s": statistics.fmean(row["transfer_overlap_s"] for row in rollout_rows), @@ -1009,6 +1035,10 @@ def analyze_run( if type(pipeline_flag) is not int or pipeline_flag not in (0, 1): _fail(f"{run_dir} hybrid_pipeline_forward must be integer 0 or 1, got {pipeline_flag!r}") pipeline_enabled = bool(pipeline_flag) + overlap_flag = manifest.get("hybrid_pipeline_overlap", pipeline_flag) + if type(overlap_flag) is not int or overlap_flag not in (0, 1): + _fail(f"{run_dir} hybrid_pipeline_overlap must be integer 0 or 1, got {overlap_flag!r}") + pipeline_overlap_enabled = bool(overlap_flag) if manifest["global_batch_size"] != expected_samples: _fail( f"{run_dir} expected_samples={expected_samples} disagrees with " @@ -1022,13 +1052,14 @@ def analyze_run( condition = str(manifest["condition"]) if condition not in {"baseline", "experiment"}: _fail(f"{run_dir} has unsupported condition {condition!r}; expected 'baseline' or 'experiment'") - if condition == "baseline" and pipeline_enabled: - _fail(f"{run_dir} is labeled baseline but hybrid_pipeline_forward is enabled") - if condition == "experiment" and not pipeline_enabled: - _fail(f"{run_dir} is labeled experiment but hybrid_pipeline_forward is disabled") + if condition == "baseline" and pipeline_enabled and pipeline_overlap_enabled: + _fail(f"{run_dir} is labeled baseline but both chunk forwarding and producer overlap are enabled") + if condition == "experiment" and (not pipeline_enabled or not pipeline_overlap_enabled): + _fail(f"{run_dir} is labeled experiment but chunk forwarding with producer overlap is not enabled") rollout_rows, actor_rank_rows, trace_summary = _analyze_trace( trace_rows, pipeline_enabled=pipeline_enabled, + pipeline_overlap_enabled=pipeline_overlap_enabled, expected_samples=expected_samples, expected_actor_chunks=expected_actor_chunks, expected_producer_chunks=expected_producer_chunks, @@ -1045,6 +1076,7 @@ def analyze_run( "condition": manifest["condition"], "seed": manifest["seed"], "hybrid_pipeline_forward": pipeline_enabled, + "hybrid_pipeline_overlap": pipeline_overlap_enabled, "steady_windows": [list(window) for window in windows], "trace": trace_summary, "metrics": metrics, @@ -1141,6 +1173,28 @@ def _build_comparison( steady_steps = _stable_steps(windows) if enforce_targets: + missing_overlap_mode = [ + str(analysis.run_dir) for analysis in analyses if "hybrid_pipeline_overlap" not in analysis.manifest + ] + if missing_overlap_mode: + _fail( + f"performance targets require an explicit hybrid_pipeline_overlap manifest field: {missing_overlap_mode}" + ) + invalid_modes = { + str(analysis.run_dir): { + "condition": analysis.manifest["condition"], + "hybrid_pipeline_forward": analysis.manifest["hybrid_pipeline_forward"], + "hybrid_pipeline_overlap": analysis.manifest["hybrid_pipeline_overlap"], + } + for analysis in analyses + if analysis.manifest["hybrid_pipeline_forward"] not in (1, True) + or bool(analysis.manifest["hybrid_pipeline_overlap"]) != (analysis.manifest["condition"] == "experiment") + } + if invalid_modes: + _fail( + "performance targets require schedule-matched chunk forwarding with overlap disabled " + f"for baseline and enabled for experiment: {invalid_modes}" + ) required_tags = PERFORMANCE_TAGS + CORRECTNESS_GUARDRAIL_TAGS for analysis in analyses: context = f"{analysis.manifest['condition']} seed={analysis.manifest['seed']}" @@ -1449,6 +1503,14 @@ def _build_comparison( if any(value > 0.02 for value in truncation_deltas): _fail(f"a paired truncation-rate increase exceeds 2 percentage points: {truncation_deltas}") + for analysis in analyses: + ppo_kl = analysis.summary["metrics"]["train/ppo_kl"]["mean"] + pg_clipfrac = analysis.summary["metrics"]["train/pg_clipfrac"]["mean"] + if abs(ppo_kl) > 1e-7: + _fail(f"{analysis.run_dir} same-weight train/ppo_kl exceeds 1e-7: {ppo_kl}") + if abs(pg_clipfrac) > 1e-7: + _fail(f"{analysis.run_dir} same-weight train/pg_clipfrac exceeds 1e-7: {pg_clipfrac}") + staleness_deltas = [row["producer_lead_at_first_forward:delta"] for row in paired] if any(value > 0.25 for value in staleness_deltas): _fail(f"a paired average producer-lead increase exceeds 0.25: {staleness_deltas}") diff --git a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh index f590b9ad7..6fbfb9481 100644 --- a/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh +++ b/scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh @@ -46,6 +46,7 @@ ROLLOUT_RESULT_DIR="${ROLLOUT_RESULT_DIR:-}" TENSORBOARD_DIR="${TENSORBOARD_DIR:-}" HYBRID_PIPELINE_FORWARD="${HYBRID_PIPELINE_FORWARD:-0}" +HYBRID_PIPELINE_OVERLAP="${HYBRID_PIPELINE_OVERLAP:-1}" HYBRID_PIPELINE_TRACE_DIR="${HYBRID_PIPELINE_TRACE_DIR:-}" HYBRID_PIPELINE_FETCH_TIMEOUT_S="${HYBRID_PIPELINE_FETCH_TIMEOUT_S:-600}" NUM_ITERS_PER_TRAIN_UPDATE="${NUM_ITERS_PER_TRAIN_UPDATE:-2}" @@ -95,6 +96,13 @@ case "${HYBRID_PIPELINE_FORWARD}" in exit 2 ;; esac +case "${HYBRID_PIPELINE_OVERLAP}" in + 0|1) ;; + *) + echo "HYBRID_PIPELINE_OVERLAP must be 0 or 1, got ${HYBRID_PIPELINE_OVERLAP}" >&2 + exit 2 + ;; +esac case "${SGLANG_DETERMINISTIC_INFERENCE}" in 0|1) ;; *) @@ -134,6 +142,10 @@ if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ] && (( NUM_ITERS_PER_TRAIN_UPDATE < 2 ) echo "HYBRID_PIPELINE_FORWARD requires NUM_ITERS_PER_TRAIN_UPDATE >= 2" >&2 exit 2 fi +if [ "${HYBRID_PIPELINE_FORWARD}" = "0" ] && [ "${HYBRID_PIPELINE_OVERLAP}" = "0" ]; then + echo "HYBRID_PIPELINE_OVERLAP=0 requires HYBRID_PIPELINE_FORWARD=1" >&2 + exit 2 +fi if [ "${CHECKPOINT_SAVE}" = "1" ]; then require_positive_integer "CHECKPOINT_SAVE_INTERVAL" "${CHECKPOINT_SAVE_INTERVAL}" require_positive_integer "MAX_ACTOR_CKPT_TO_KEEP" "${MAX_ACTOR_CKPT_TO_KEEP}" @@ -201,6 +213,9 @@ if [ "${HYBRID_PIPELINE_FORWARD}" = "1" ]; then --hybrid-pipeline-forward --hybrid-pipeline-fetch-timeout-s "${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" ) + if [ "${HYBRID_PIPELINE_OVERLAP}" = "0" ]; then + HYBRID_PIPELINE_ARGS+=(--no-hybrid-pipeline-overlap) + fi fi if [ -n "${HYBRID_PIPELINE_TRACE_DIR}" ]; then HYBRID_PIPELINE_ARGS+=( @@ -234,6 +249,7 @@ printf '%s\n' \ "MODE=${MODE}" \ "NUM_ROLLOUT=${NUM_ROLLOUT}" \ "HYBRID_PIPELINE_FORWARD=${HYBRID_PIPELINE_FORWARD}" \ + "HYBRID_PIPELINE_OVERLAP=${HYBRID_PIPELINE_OVERLAP}" \ "HYBRID_PIPELINE_TRACE_DIR=${HYBRID_PIPELINE_TRACE_DIR}" \ "HYBRID_PIPELINE_FETCH_TIMEOUT_S=${HYBRID_PIPELINE_FETCH_TIMEOUT_S}" \ "NUM_ITERS_PER_TRAIN_UPDATE=${NUM_ITERS_PER_TRAIN_UPDATE}" \ diff --git a/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py index 096e4a360..d9d3456d0 100644 --- a/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py +++ b/tests/backends/megatron/test_hybrid_pipeline_actor_wiring.py @@ -101,15 +101,17 @@ def test_debug_rollout_chunking_slices_every_per_sample_container(): @pytest.mark.parametrize( - ("pipeline_enabled", "expected_fetch_sizes", "expected_forward_count"), + ("pipeline_enabled", "pipeline_overlap", "expected_fetch_sizes", "expected_forward_count"), [ - (False, [4], 1), - (True, [2, 2], 2), + (False, True, [4], 1), + (True, True, [2, 2], 2), + (True, False, [2, 2], 2), ], ) def test_train_hybrid_wires_one_update_around_flagged_actor_chunks( monkeypatch, pipeline_enabled, + pipeline_overlap, expected_fetch_sizes, expected_forward_count, ): @@ -123,6 +125,7 @@ def test_train_hybrid_wires_one_update_around_flagged_actor_chunks( num_steps_per_rollout=None, num_iters_per_train_update=2, hybrid_pipeline_forward=pipeline_enabled, + hybrid_pipeline_overlap=pipeline_overlap, hybrid_pipeline_trace_dir=None, hybrid_pipeline_fetch_timeout_s=1.0, use_rollout_routing_replay=False, @@ -221,6 +224,13 @@ def all_gather_object(output, value, **_kwargs): assert [event[1] for event in events if isinstance(event, tuple) and event[0] == "fetch"] == (expected_fetch_sizes) assert sum(isinstance(event, tuple) and event[0] == "forward" for event in events) == expected_forward_count assert sum(isinstance(event, tuple) and event[0] == "restore" for event in events) == int(pipeline_enabled) + fetch_forward_events = [ + event[0] for event in events if isinstance(event, tuple) and event[0] in {"fetch", "forward"} + ] + if pipeline_enabled: + assert fetch_forward_events == ( + ["fetch", "forward", "fetch", "forward"] if pipeline_overlap else ["fetch", "fetch", "forward", "forward"] + ) assert events.count("advantages") == 1 assert events.count("optimizer") == 1 assert events.count("update_weights") == 1 diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py index 21f30dc36..e486180ca 100644 --- a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -113,19 +113,22 @@ def _write_run( *, name="run", pipeline_enabled=True, + pipeline_overlap=None, hostname="test-host", missing_event=None, nonfinite=False, producer_put_count=2, producer_last_start_ns=700, ): + if pipeline_overlap is None: + pipeline_overlap = pipeline_enabled run_dir = tmp_path / name timeline = run_dir / "timeline" timeline.mkdir(parents=True) (run_dir / "run_manifest.json").write_text( json.dumps( { - "condition": "experiment" if pipeline_enabled else "baseline", + "condition": "experiment" if pipeline_enabled and pipeline_overlap else "baseline", "hostname": hostname, "order": "B1" if pipeline_enabled else "A1", "seed": 7, @@ -137,6 +140,7 @@ def _write_run( "n_samples_per_prompt": 2, "num_iters_per_train_update": 2, "hybrid_pipeline_forward": int(pipeline_enabled), + "hybrid_pipeline_overlap": int(pipeline_overlap), "hybrid_pipeline_trace_dir": str(timeline), "hybrid_pipeline_fetch_timeout_s": 600, "git_commit": "a" * 40, @@ -196,7 +200,7 @@ def _write_run( if nonfinite: producer[0]["total_tokens"] = float("nan") - if pipeline_enabled: + if pipeline_enabled and pipeline_overlap: actor = [ _event("actor_restore_start", 210, role="actor", chunk_index=0, sample_count=4), _event("actor_restore_end", 300, role="actor", chunk_index=0, sample_count=4), @@ -251,6 +255,61 @@ def _write_run( fingerprint=FINGERPRINT_1, ), ] + elif pipeline_enabled: + actor = [ + _event("actor_restore_start", 210, role="actor", chunk_index=0, sample_count=4), + _event("actor_restore_end", 300, role="actor", chunk_index=0, sample_count=4), + _event("chunk_fetch_start", 310, role="actor", chunk_index=0, sample_count=2), + _event( + "chunk_fetch_end", + 350, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event("chunk_fetch_start", 810, role="actor", chunk_index=1, sample_count=2), + _event( + "chunk_fetch_end", + 850, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + _event( + "actor_forward_start", + 860, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event( + "actor_forward_end", + 900, + role="actor", + chunk_index=0, + sample_count=2, + fingerprint=FINGERPRINT_0, + ), + _event( + "actor_forward_start", + 910, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + _event( + "actor_forward_end", + 1000, + role="actor", + chunk_index=1, + sample_count=2, + fingerprint=FINGERPRINT_1, + ), + ] else: actor = [ _event("chunk_fetch_start", 810, role="actor", chunk_index=0, sample_count=4), @@ -387,6 +446,21 @@ def test_baseline_trace_preserves_one_full_fetch_and_no_overlap(tmp_path): assert analysis.trace_rows[0]["first_forward_before_last_put_start"] is False +def test_schedule_matched_baseline_fetches_all_chunks_before_forward(tmp_path): + run_dir = _write_run(tmp_path, pipeline_enabled=True, pipeline_overlap=False) + + analysis = analyzer.analyze_run( + run_dir, + windows=((4, 4),), + expected_samples=4, + expected_actor_chunks=2, + ) + + assert analysis.trace_rows[0]["actor_fetch_count"] == 2 + assert analysis.trace_rows[0]["all_chunks_fetched_before_first_forward"] is True + assert analysis.summary["hybrid_pipeline_overlap"] is False + + @pytest.mark.parametrize( ("kwargs", "error"), [ @@ -542,10 +616,12 @@ def _comparison_analysis( throughput, phase1, *, - overlap=True, + overlap=None, accuracy=0.5, peak_vram_mib=10_000, ): + if overlap is None: + overlap = condition == "experiment" metrics = { "perf/step_token_per_s": {"aggregate": throughput}, "perf/step_resp_token_per_s": {"aggregate": throughput / 2}, @@ -556,6 +632,8 @@ def _comparison_analysis( "rollout/truncated_ratio": {"mean": 0.0}, "train/loss": {"mean": 1.0}, "train/grad_norm": {"mean": 0.5}, + "train/ppo_kl": {"mean": 0.0}, + "train/pg_clipfrac": {"mean": 0.0}, } trace_rows = [ { @@ -582,6 +660,8 @@ def _comparison_analysis( "rollout/truncated_ratio": 0.0, "train/loss": 1.0, "train/grad_norm": 0.5, + "train/ppo_kl": 0.0, + "train/pg_clipfrac": 0.0, } scalar_rows = [ {"tag": tag, "step": step, "value": value} for step in range(4, 19) for tag, value in scalar_values.items() @@ -600,7 +680,8 @@ def _comparison_analysis( "rollout_batch_size": 32, "n_samples_per_prompt": 8, "num_iters_per_train_update": 2, - "hybrid_pipeline_forward": condition == "experiment", + "hybrid_pipeline_forward": True, + "hybrid_pipeline_overlap": condition == "experiment", "hybrid_pipeline_fetch_timeout_s": 600, "git_commit": "a" * 40, "git_branch": "perf/task21", @@ -692,6 +773,23 @@ def test_preregistered_targets_pass_and_fail_deterministically(): ) +def test_performance_targets_require_schedule_matched_overlap_modes(): + analyses = [ + _comparison_analysis("baseline", 1, 100, 10), + _comparison_analysis("experiment", 1, 106, 8), + _comparison_analysis("baseline", 2, 102, 10.2), + _comparison_analysis("experiment", 2, 108.12, 8.16), + ] + analyses[0].manifest["hybrid_pipeline_forward"] = False + + with pytest.raises(analyzer.BenchmarkValidationError, match="schedule-matched chunk forwarding"): + analyzer._build_comparison( + analyses, + windows=((4, 8), (9, 13), (14, 18)), + enforce_targets=True, + ) + + def test_comparison_rejects_mixed_candidate_commits(): analyses = [ _comparison_analysis("baseline", 1, 100, 10), @@ -801,6 +899,8 @@ def test_comparison_plot_bundle_fails_fast_without_matplotlib(tmp_path, monkeypa ("staleness_max", "observed producer lead exceeds configured max_staleness=2"), ("vram", "peak VRAM increased"), ("workload", "actor_total_tokens must match exactly"), + ("ppo_kl", "same-weight train/ppo_kl exceeds"), + ("pg_clipfrac", "same-weight train/pg_clipfrac exceeds"), ], ) def test_preregistered_guardrails_fail_closed(guardrail, error): @@ -828,6 +928,10 @@ def test_preregistered_guardrails_fail_closed(guardrail, error): elif guardrail == "workload": for row in experiment.trace_rows: row["actor_total_tokens"] *= 2 + elif guardrail == "ppo_kl": + experiment.summary["metrics"]["train/ppo_kl"]["mean"] = 1e-4 + elif guardrail == "pg_clipfrac": + experiment.summary["metrics"]["train/pg_clipfrac"]["mean"] = 1e-4 with pytest.raises(analyzer.BenchmarkValidationError, match=error): analyzer._build_comparison( diff --git a/tests/scripts/test_multimodal_hybrid_async_launcher.py b/tests/scripts/test_multimodal_hybrid_async_launcher.py index e82e4c2f9..967c5e233 100644 --- a/tests/scripts/test_multimodal_hybrid_async_launcher.py +++ b/tests/scripts/test_multimodal_hybrid_async_launcher.py @@ -160,3 +160,29 @@ def test_launcher_rejects_invalid_actor_chunk_count_before_ray(tmp_path): assert "requires NUM_ITERS_PER_TRAIN_UPDATE >= 2" in result.stderr assert arguments == [] assert not capture_path.exists() + + +def test_launcher_exposes_schedule_matched_no_overlap_control(tmp_path): + result, arguments, _, _ = _run_launcher( + tmp_path, + overrides={ + "HYBRID_PIPELINE_FORWARD": "1", + "HYBRID_PIPELINE_OVERLAP": "0", + }, + ) + + assert result.returncode == 0, result.stderr + assert "--hybrid-pipeline-forward" in arguments + assert "--no-hybrid-pipeline-overlap" in arguments + + +def test_launcher_rejects_overlap_control_without_pipeline_forward(tmp_path): + result, arguments, capture_path, _ = _run_launcher( + tmp_path, + overrides={"HYBRID_PIPELINE_OVERLAP": "0"}, + ) + + assert result.returncode == 2 + assert "requires HYBRID_PIPELINE_FORWARD=1" in result.stderr + assert arguments == [] + assert not capture_path.exists() diff --git a/tests/utils/test_arguments_opd_teacher_colocate.py b/tests/utils/test_arguments_opd_teacher_colocate.py index 2b1f7c585..5cdf6ce52 100644 --- a/tests/utils/test_arguments_opd_teacher_colocate.py +++ b/tests/utils/test_arguments_opd_teacher_colocate.py @@ -81,14 +81,20 @@ def test_hybrid_pipeline_options_default_off(arguments_module): args = parser.parse_args([]) assert args.hybrid_pipeline_forward is False + assert args.hybrid_pipeline_overlap is True assert args.hybrid_pipeline_trace_dir is None assert args.hybrid_pipeline_fetch_timeout_s == 600.0 + args = parser.parse_args(["--hybrid-pipeline-forward", "--no-hybrid-pipeline-overlap"]) + assert args.hybrid_pipeline_forward is True + assert args.hybrid_pipeline_overlap is False + def _hybrid_pipeline_args() -> SimpleNamespace: return SimpleNamespace( hybrid=True, hybrid_pipeline_forward=True, + hybrid_pipeline_overlap=True, hybrid_pipeline_trace_dir="/tmp/trace", hybrid_pipeline_fetch_timeout_s=600.0, use_dynamic_batch_size=True, @@ -130,6 +136,15 @@ def test_hybrid_pipeline_supported_configuration_is_accepted(arguments_module): arguments_module._validate_hybrid_pipeline_args(_hybrid_pipeline_args()) +def test_hybrid_pipeline_overlap_control_requires_chunk_forward(arguments_module): + args = _hybrid_pipeline_args() + args.hybrid_pipeline_forward = False + args.hybrid_pipeline_overlap = False + + with pytest.raises(ValueError, match="requires --hybrid-pipeline-forward"): + arguments_module._validate_hybrid_pipeline_args(args) + + @pytest.mark.parametrize( ("field", "value", "error"), [ diff --git a/tests/utils/test_hybrid_forward_pipeline.py b/tests/utils/test_hybrid_forward_pipeline.py index d3f4428be..e679a106c 100644 --- a/tests/utils/test_hybrid_forward_pipeline.py +++ b/tests/utils/test_hybrid_forward_pipeline.py @@ -65,6 +65,35 @@ def test_restore_occurs_once_per_optimizer_mini(): assert restore_calls == [0, 2, 4] +def test_no_overlap_control_fetches_all_chunks_before_matched_forwards(): + events = [] + + chunks = execute_hybrid_forward_mini( + chunks_per_mini=3, + batch_index_for_chunk=lambda chunk_index: 10 + chunk_index, + restore_actor=lambda batch_index: events.append(("restore", batch_index)), + fetch_chunk=lambda batch_index: ( + events.append(("fetch", batch_index)) or {"total_lengths": [batch_index]}, + [100 + batch_index], + ), + forward_chunk=lambda _batch, batch_index, global_indexes: events.append( + ("forward", batch_index, global_indexes) + ), + overlap_producer=False, + ) + + assert events == [ + ("restore", 10), + ("fetch", 10), + ("fetch", 11), + ("fetch", 12), + ("forward", 10, [110]), + ("forward", 11, [111]), + ("forward", 12, [112]), + ] + assert [indexes for _, indexes in chunks] == [[110], [111], [112]] + + def test_chunk_microbatch_schedule_is_replayed_on_canonical_batch(): schedule = canonicalize_hybrid_microbatch_schedule( [ From 0af92992cb70d1e8b2ca1998bfc28ed60ff76e3c Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Sun, 2 Aug 2026 10:32:17 +0800 Subject: [PATCH 11/21] chore(hybrid): keep pipeline docs reproducible under CI Apply the repository docformatter layout so local and GitHub pre-commit checks produce no working-tree mutation. Constraint: The project-wide pre-commit job requires docformatter-idempotent docstrings. Rejected: Disable or bypass docformatter | That would weaken an existing repository gate. Confidence: high Scope-risk: narrow Directive: Run the repository pre-commit hooks, not only Ruff, when editing docstrings. Tested: pre-commit on the changed file; 119 targeted pytest passed with 6 dependency-gated skips; git diff --check. --- relax/utils/training/hybrid_forward_pipeline.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/relax/utils/training/hybrid_forward_pipeline.py b/relax/utils/training/hybrid_forward_pipeline.py index 4a4b902ba..e20af3f6e 100644 --- a/relax/utils/training/hybrid_forward_pipeline.py +++ b/relax/utils/training/hybrid_forward_pipeline.py @@ -15,7 +15,8 @@ def execute_hybrid_forward_mini( forward_chunk: Callable[[Any, int, list[int]], None], overlap_producer: bool = True, ) -> list[tuple[Any, list[int]]]: - """Restore once, then execute a matched chunk schedule with optional overlap.""" + """Restore once, then execute a matched chunk schedule with optional + overlap.""" if chunks_per_mini <= 0: raise ValueError(f"chunks_per_mini must be positive, got {chunks_per_mini}") if type(overlap_producer) is not bool: @@ -43,10 +44,10 @@ def canonicalize_hybrid_microbatch_schedule( ) -> list[list[int]]: """Translate chunk-local forward schedules into merged-batch indexes. - The actor old-logprob forward chooses its dynamic microbatches independently - for each producer chunk. Training must replay those exact sample groups and - their order; otherwise batch-shape-dependent numerics create an artificial - PPO ratio even though the weights did not change. + The actor old-logprob forward chooses its dynamic microbatches + independently for each producer chunk. Training must replay those exact + sample groups and their order; otherwise batch-shape-dependent numerics + create an artificial PPO ratio even though the weights did not change. """ canonical_indexes = list(canonical_global_indexes) if not canonical_indexes: From 78d3fe0d50f01d1e5a05fcaefdc211f9350df9e8 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Tue, 11 Aug 2026 17:17:43 +0800 Subject: [PATCH 12/21] keep hybrid timing valid after upstream cleanup Constraint: Upstream no longer imports contextlib.nullcontext in actor.py Confidence: high Scope-risk: narrow Tested: 121 targeted tests; Ruff; compileall; pre-commit; shell syntax and ShellCheck Not-tested: final 5-GPU parity and ABBA pending idle GPUs --- relax/backends/megatron/actor.py | 1 + 1 file changed, 1 insertion(+) diff --git a/relax/backends/megatron/actor.py b/relax/backends/megatron/actor.py index 2ea99c2d9..58b7d9f98 100644 --- a/relax/backends/megatron/actor.py +++ b/relax/backends/megatron/actor.py @@ -6,6 +6,7 @@ import socket import time from argparse import Namespace +from contextlib import nullcontext from functools import partial from typing import Any, List From 780c742793fea54acdb28cb40ffb591fb909ba51 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Tue, 11 Aug 2026 19:51:18 +0800 Subject: [PATCH 13/21] Accept valid asynchronous producer regrouping The producer may expose prompt-group-aligned puts with unequal sizes while the actor still consumes the fixed registered schedule. Validate event closure, aggregate samples, and fingerprints instead of coupling producer puts to actor chunks. Constraint: Producer completion order can yield 64,64,72,56 for a valid 256-sample rollout. Rejected: Require four equal producer puts | async producer grouping is not part of the actor-forward contract. Confidence: high Scope-risk: narrow Directive: Keep actor chunk counts strict; producer put counts remain diagnostic. Tested: 33 analyzer tests; Ruff on changed Python; git diff --check Not-tested: Full GPU benchmark reruns after this analyzer-only correction --- docs/en/guide/hybrid-training.md | 6 ++-- docs/zh/guide/hybrid-training.md | 5 ++-- .../analyze_hybrid_pipeline_benchmark.py | 29 +------------------ .../test_analyze_hybrid_pipeline_benchmark.py | 22 -------------- 4 files changed, 8 insertions(+), 54 deletions(-) diff --git a/docs/en/guide/hybrid-training.md b/docs/en/guide/hybrid-training.md index 7681ef39f..b4fdfce84 100644 --- a/docs/en/guide/hybrid-training.md +++ b/docs/en/guide/hybrid-training.md @@ -383,8 +383,10 @@ exit-status artifacts for every run. The manifest also records and cross-checks `max_staleness`, global/rollout batch sizes, samples per prompt, and actor chunk count, so CLI expectations cannot silently disagree with the measured workload. -The registered four-stage protocol additionally requires exactly four producer -puts of 64 samples for every rollout; regrouped producer puts fail validation. +The registered four-stage protocol requires exactly four actor fetch/forward +chunks of 64 samples. Producer puts may be regrouped by asynchronous completion; +their events must close and their aggregate sample count and fingerprint must +match the actor-consumed workload exactly. Before any paired statistics are calculated, the analyzer also requires identical model/config/data paths, prompt/response/context limits, actor token budget, actor/rollout resource topology, SGLang determinism and memory diff --git a/docs/zh/guide/hybrid-training.md b/docs/zh/guide/hybrid-training.md index b2dc33361..f244ef3b4 100644 --- a/docs/zh/guide/hybrid-training.md +++ b/docs/zh/guide/hybrid-training.md @@ -361,8 +361,9 @@ step end wall time 和 duration 重建。sampled peak VRAM 仍使用 full-run launcher log，以及值为 0 的训练、验证和最终退出状态文件。 manifest 还会记录并交叉校验 `max_staleness`、global/rollout batch size、 每 prompt 样本数和 actor chunk 数，避免分析器 CLI 与实际工作负载静默不一致。 -注册的四阶段协议还要求每个 rollout 恰好执行四次、每次 64 sample 的 producer -put；producer 重新分组会导致验证失败。 +注册的四阶段协议要求 actor 恰好执行四次、每次 64 sample 的 fetch/forward。 +producer put 可因异步完成顺序重新分组，但事件必须闭合，且聚合后的 sample 总数与 +fingerprint 必须和 actor 实际消费的工作量精确一致。 在计算任何配对统计前，分析器还要求模型、模型配置、数据路径、prompt/response/ context 上限、actor token 预算、actor/rollout 资源拓扑、SGLang 确定性与显存 配置、物理卡到容器卡映射、checkpoint 模式以及 debug 捕获/回放配置完全一致； diff --git a/scripts/tools/analyze_hybrid_pipeline_benchmark.py b/scripts/tools/analyze_hybrid_pipeline_benchmark.py index 3e2c3b9ea..c78b25ee7 100644 --- a/scripts/tools/analyze_hybrid_pipeline_benchmark.py +++ b/scripts/tools/analyze_hybrid_pipeline_benchmark.py @@ -462,7 +462,6 @@ def _analyze_trace( pipeline_overlap_enabled: bool, expected_samples: int, expected_actor_chunks: int, - expected_producer_chunks: int | None, ) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]: producer_by_rollout: dict[int, list[dict[str, Any]]] = defaultdict(list) actor_by_rollout_stream: dict[tuple[int, tuple[str, int, int]], list[dict[str, Any]]] = defaultdict(list) @@ -500,22 +499,6 @@ def _analyze_trace( put_done = [end for _, end in put_pairs] if any(row["sample_count"] is None for row in put_done): _fail(f"rollout_id={rollout_id} producer put is missing sample_count") - if expected_producer_chunks is not None: - if expected_samples % expected_producer_chunks != 0: - _fail( - "expected_samples must be divisible by expected_producer_chunks, " - f"got {expected_samples=} and {expected_producer_chunks=}" - ) - expected_producer_chunk_samples = expected_samples // expected_producer_chunks - producer_chunk_samples = [int(row["sample_count"] or 0) for row in put_done] - if len(put_pairs) != expected_producer_chunks or any( - sample_count != expected_producer_chunk_samples for sample_count in producer_chunk_samples - ): - _fail( - f"rollout_id={rollout_id} expected {expected_producer_chunks} producer " - f"chunks of {expected_producer_chunk_samples} samples, got " - f"{producer_chunk_samples}" - ) producer_samples = sum(int(row["sample_count"]) for row in put_done) if producer_samples != expected_samples: _fail( @@ -1021,7 +1004,6 @@ def analyze_run( windows: Sequence[tuple[int, int]] = DEFAULT_WINDOWS, expected_samples: int = 256, expected_actor_chunks: int = 2, - expected_producer_chunks: int | None = None, write_outputs: bool = True, require_reproducibility_artifacts: bool = False, ) -> RunAnalysis: @@ -1062,7 +1044,6 @@ def analyze_run( pipeline_overlap_enabled=pipeline_overlap_enabled, expected_samples=expected_samples, expected_actor_chunks=expected_actor_chunks, - expected_producer_chunks=expected_producer_chunks, ) scalar_rows = _load_tensorboard_scalars(run_dir) nvml_rows = _parse_nvml_rows(run_dir) @@ -1753,12 +1734,6 @@ def build_parser() -> argparse.ArgumentParser: default=2, help="Expected actor fetch/forward chunks when the pipeline is enabled.", ) - parser.add_argument( - "--expected-producer-chunks", - type=int, - default=None, - help="Require an exact producer put count and equal sample count per put.", - ) parser.add_argument( "--expected-gpu-count", type=int, @@ -1791,9 +1766,8 @@ def main(argv: Sequence[str] | None = None) -> int: args.expected_samples <= 0 or args.expected_actor_chunks <= 0 or args.expected_gpu_count <= 0 - or (args.expected_producer_chunks is not None and args.expected_producer_chunks <= 0) ): - _fail("expected sample, actor chunk, producer chunk, and GPU counts must be positive") + _fail("expected sample, actor chunk, and GPU counts must be positive") if len(args.run_dir) > 1 and args.output_dir is None: _fail("--output-dir is required when comparing multiple runs") if args.validate_only and args.enforce_targets: @@ -1805,7 +1779,6 @@ def main(argv: Sequence[str] | None = None) -> int: windows=windows, expected_samples=args.expected_samples, expected_actor_chunks=args.expected_actor_chunks, - expected_producer_chunks=args.expected_producer_chunks, require_reproducibility_artifacts=args.enforce_targets, ) for run_dir in args.run_dir diff --git a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py index e486180ca..11f9e9c19 100644 --- a/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py +++ b/tests/scripts/test_analyze_hybrid_pipeline_benchmark.py @@ -392,28 +392,6 @@ def test_producer_put_grouping_is_independent_from_actor_chunks(tmp_path, produc assert analysis.trace_rows[0]["actor_forward_count"] == 2 -def test_registered_producer_stage_shape_is_enforced(tmp_path): - run_dir = _write_run(tmp_path, producer_put_count=2) - - analysis = analyzer.analyze_run( - run_dir, - windows=((4, 4),), - expected_samples=4, - expected_actor_chunks=2, - expected_producer_chunks=2, - ) - assert analysis.trace_rows[0]["producer_put_count"] == 2 - - with pytest.raises(analyzer.BenchmarkValidationError, match="expected 4 producer chunks"): - analyzer.analyze_run( - run_dir, - windows=((4, 4),), - expected_samples=4, - expected_actor_chunks=2, - expected_producer_chunks=4, - ) - - def test_delayed_put_done_is_not_strict_producer_overlap(tmp_path): run_dir = _write_run(tmp_path, producer_last_start_ns=150) From e71d30e7ef622943cafd140cd747818f76b4a007 Mon Sep 17 00:00:00 2001 From: Pinoeer-kingxi <13022943007@163.com> Date: Tue, 11 Aug 2026 21:49:53 +0800 Subject: [PATCH 14/21] Make Task 21 GPU evidence reviewable Publish the preregistered four-pair benchmark curves and compact data, and render quality guardrails by paired seed so one-step runs remain interpretable. Constraint: Formal training evidence was collected at parent commit 780c742 on physical GPUs 1,2,3,4,6. Rejected: Rerunning eight successful GPU jobs after an analyzer-only presentation change | the performance path is unchanged and the existing raw artifacts were reanalyzed successfully. Confidence: high Scope-risk: narrow Directive: Keep performance claims tied to the immutable raw manifests and disclose asynchronous microbatch-order effects separately from deterministic replay parity. Tested: 33 analyzer tests; ruff check; git diff --check; formal analyzer --enforce-targets passed on 8 real GPU runs. Not-tested: No additional GPU training was run after this analyzer/report-only commit. --- .../README.md | 49 +++++++++++++++ .../SHA256SUMS | 10 +++ .../formal-paired-results.csv | 5 ++ .../paired_run_summary.csv | 5 ++ .../task21_correctness_quality.png | Bin 0 -> 325723 bytes .../task21_gpu_util_vram.png | Bin 0 -> 369238 bytes .../task21_phase1_overlap.png | Bin 0 -> 234079 bytes .../task21_step_throughput.png | Bin 0 -> 159848 bytes .../task21_window_summary.png | Bin 0 -> 111356 bytes .../window_speedup_summary.csv | 5 ++ .../analyze_hybrid_pipeline_benchmark.py | 59 +++++++++--------- 11 files changed, 102 insertions(+), 31 deletions(-) create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/README.md create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/SHA256SUMS create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/task21_correctness_quality.png create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/task21_gpu_util_vram.png create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/task21_phase1_overlap.png create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/task21_step_throughput.png create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/task21_window_summary.png create mode 100644 docs/assets/task21-hybrid-mm-forward-pipeline/window_speedup_summary.csv diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/README.md b/docs/assets/task21-hybrid-mm-forward-pipeline/README.md new file mode 100644 index 000000000..dbf33be73 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/README.md @@ -0,0 +1,49 @@ +# Task 21: Hybrid-async multimodal pipeline — formal GPU results + +## Protocol + +- Performance-code commit: `780c742793fea54acdb28cb40ffb591fb909ba51`. +- Base commit: `9a5674afde12f608698ab4f60cdb9849a0eb6cb3`. +- Hardware: physical RTX A6000 GPUs `1,2,3,4,6`; four actor GPUs (`TP=2`, `CP=2`) plus one rollout GPU. +- Workload: Qwen3-VL-8B-Instruct, OpenR1-Multimodal, 256 generated samples per run, response cap 1024, one real optimizer step. +- Design: paired seeds `20260811`–`20260814`, ABBAABBA order; baseline and experiment differ only in `--hybrid-pipeline-overlap`. +- Statistics: four fresh-process paired runs. The formal analyzer validated every run and all preregistered targets. + +## Result summary + +| Metric | Baseline | Experiment | Result | +|---|---:|---:|---:| +| step token throughput, arithmetic mean | 575.68 token/s | 644.69 token/s | paired geometric-mean speedup **12.01%** | +| hybrid phase-1 time, arithmetic mean | 165.38 s | 130.38 s | paired geometric-mean reduction **21.21%** | +| end-to-end step time, arithmetic mean | 342.24 s | 305.61 s | **10.70% lower** | +| producer/actor overlap | 0/4 runs | 4/4 runs | **100%** experiment overlap | +| steady GPU utilization, arithmetic mean | 40.01% | 44.37% | +4.36 percentage points | +| sampled peak VRAM, maximum | 47,380 MiB | 47,140 MiB | no regression | + +Every paired run exceeded the preregistered 5% throughput target: `10.22%`, `12.21%`, `12.80%`, and `12.82%`. Every pair also reduced phase-1 time by `19.11%`–`23.53%`. + +The throughput plot shows both total-token and response-token throughput for each matched seed. The experiment point is above its baseline point in every pair. + +![Step throughput](task21_step_throughput.png) + +The phase-1 plot shows the producer/actor overlap interval, phase-1 latency, and producer lead. The experiment overlaps producer work with actor forward in all four runs, while the baseline deliberately disables that overlap. + +![Phase-1 overlap](task21_phase1_overlap.png) + +The GPU plot combines the 500 ms NVML time series with steady-window utilization, idle ratio, and sampled peak VRAM. Utilization improves without a peak-memory regression. + +![GPU utilization and VRAM](task21_gpu_util_vram.png) + +The quality plot pairs reward, response length, truncation, loss, gradient norm, and PPO KL by seed. Workload, reward, response length, and truncation are exact within each pair; loss differs by at most `6.21e-5`, PPO KL remains within `2.37e-11`, and every value is finite. Seed `20260814` shows a disclosed gradient-norm difference (`1.0033` vs `0.4775`) caused by different asynchronous microbatch grouping; the deterministic replay parity run controls grouping and bounds the patched-path gradient-norm difference to `0.207%`. + +![Correctness and quality](task21_correctness_quality.png) + +The window summary shows each paired throughput speedup and the geometric mean against the preregistered 5% threshold. + +![Window and paired summary](task21_window_summary.png) + +## Correctness evidence + +The final replay parity dataset contains 1,024 samples across four actor ranks. Baseline/control/experiment agree exactly for tokens, response lengths, loss masks, rewards, raw rewards, advantages, returns, truncation flags, multimodal tensors, and dynamic-microbatch schedules. Loss is exactly `0.03446025028824806`; experiment log-probability maximum absolute difference is `2.3842e-7`; gradient-norm symmetric relative difference is `0.207%` (below the BF16 0.5% guardrail). + +Full raw logs, manifests, TensorBoard events, timeline JSONL, NVML CSV, summaries, and generated figures are stored under `/data01/LWX/relax-task21/`. Compact paired data and checksums are included in this directory. diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/SHA256SUMS b/docs/assets/task21-hybrid-mm-forward-pipeline/SHA256SUMS new file mode 100644 index 000000000..172e31328 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/SHA256SUMS @@ -0,0 +1,10 @@ +5016e5fc77093f2df3d85942e8abf95999738c75191c17a37176d53bbfc8a4b3 /data01/LWX/relax-task21/comparisons/FORMAL-FINAL8-Q3VL8B-R1024-C4-DET-GPU12346-FIRSTSTEP/comparison_summary.json +6e72053891db350c2863bbf3508188a3740d13d176cd86eaf0b56261d3875938 /data01/LWX/relax-task21/comparisons/FORMAL-FINAL8-Q3VL8B-R1024-C4-DET-GPU12346-FIRSTSTEP/console-final.log +025300b86959bc7c0dfb7dd941989423be873a3b8b06452c1697a8886da73018 docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv +da9b541bc2fbd26d5a4730eb9096d3b152d9470d2dd81c03f5104b4a62d03306 docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv +7408c899219b63619335ebf42f6374421df46a664b7d22c4a45ed03567b38f5e docs/assets/task21-hybrid-mm-forward-pipeline/task21_correctness_quality.png +38c6f272370cb0e12abb1d9a403596681ae4f150f30624e35d0b85943adf8c91 docs/assets/task21-hybrid-mm-forward-pipeline/task21_gpu_util_vram.png +e1256d0fdd7b5b22f81411ba8e20224f26f58e0955142352a37b492aa9b6a708 docs/assets/task21-hybrid-mm-forward-pipeline/task21_phase1_overlap.png +bbad23dc4059ffc0610786b4d042b9af37558a8753cb298e5bf5429e44011134 docs/assets/task21-hybrid-mm-forward-pipeline/task21_step_throughput.png +c989433666edfcecbe5d0761c0e03b41d104772a232c15fb6abdfe5c6699bd0a docs/assets/task21-hybrid-mm-forward-pipeline/task21_window_summary.png +ea8cfbd0958c16f54ef799aecf9a9bc45601b7861e8dd01ba7dd8a8ed033eb64 docs/assets/task21-hybrid-mm-forward-pipeline/window_speedup_summary.csv diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv new file mode 100644 index 000000000..7f00a2b90 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/formal-paired-results.csv @@ -0,0 +1,5 @@ +seed,baseline_step_token_per_s,experiment_step_token_per_s,throughput_speedup_percent,baseline_phase1_s,experiment_phase1_s,phase1_reduction_percent,baseline_step_time_s,experiment_step_time_s,baseline_peak_vram_mib,experiment_peak_vram_mib,raw_reward,truncated_ratio,baseline_loss,experiment_loss,baseline_grad_norm,experiment_grad_norm +20260811,597.4306030273438,658.458984375,10.215141480601897,169.09609985351562,136.78187561035156,19.10997608528948,348.0404357910156,315.78277587890625,47380.0,47140.0,0.296875,0.1875,0.013692490756511688,0.013677351176738739,0.4145539104938507,0.4117335081100464 +20260812,576.8270874023438,647.2715454101562,12.21240464365998,171.17022705078125,136.36746215820312,20.33225374074731,353.73858642578125,315.24017333984375,46818.0,46898.0,0.41796875,0.10546875,0.012462313286960125,0.012400214560329914,0.44756120443344116,0.4443718492984772 +20260813,559.2531127929688,630.8582763671875,12.803713012273654,162.420166015625,124.19634246826172,23.53391483646563,342.5908508300781,303.7052917480469,46912.0,47104.0,0.35546875,0.11328125,-0.021680761128664017,-0.021680761128664017,0.6372919082641602,0.6365648508071899 +20260814,569.1944580078125,642.1531372070312,12.817882917302992,158.83006286621094,124.18817901611328,21.81065928260567,324.609619140625,287.7288818359375,46916.0,46824.0,0.38671875,0.1640625,0.004201771225780249,0.004198123700916767,1.0032663345336914,0.4775027334690094 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv b/docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv new file mode 100644 index 000000000..44cc84d57 --- /dev/null +++ b/docs/assets/task21-hybrid-mm-forward-pipeline/paired_run_summary.csv @@ -0,0 +1,5 @@ +actor_fetch_samples:baseline,actor_fetch_samples:experiment,actor_fetch_samples:relative_delta,actor_multimodal_tensor_bytes:baseline,actor_multimodal_tensor_bytes:experiment,actor_multimodal_tensor_bytes:relative_delta,actor_response_tokens:baseline,actor_response_tokens:experiment,actor_response_tokens:relative_delta,actor_total_tokens:baseline,actor_total_tokens:experiment,actor_total_tokens:relative_delta,nvml_peak_memory_mib:baseline,nvml_peak_memory_mib:delta,nvml_peak_memory_mib:experiment,perf/hybrid_phase1_time:baseline,perf/hybrid_phase1_time:experiment,perf/hybrid_phase1_time:improvement,perf/step_resp_token_per_s:baseline,perf/step_resp_token_per_s:experiment,perf/step_resp_token_per_s:improvement,perf/step_time:baseline,perf/step_time:experiment,perf/step_time:improvement,perf/step_time:p95_baseline,perf/step_time:p95_experiment,perf/step_time:p95_regression,perf/step_token_per_s:baseline,perf/step_token_per_s:experiment,perf/step_token_per_s:improvement,perf/wall_clock_samples_per_s:baseline,perf/wall_clock_samples_per_s:experiment,perf/wall_clock_samples_per_s:improvement,producer_lead_at_first_forward:baseline,producer_lead_at_first_forward:delta,producer_lead_at_first_forward:experiment,rollout/raw_reward:baseline,rollout/raw_reward:delta,rollout/raw_reward:experiment,rollout/truncated_ratio:baseline,rollout/truncated_ratio:delta,rollout/truncated_ratio:experiment,seed,train/grad_norm:baseline,train/grad_norm:delta,train/grad_norm:experiment,train/loss:baseline,train/loss:delta,train/loss:experiment,train/pg_clipfrac:baseline,train/pg_clipfrac:delta,train/pg_clipfrac:experiment,train/ppo_kl:baseline,train/ppo_kl:delta,train/ppo_kl:experiment +256,256,0.0,1327110144,1327110144,0.0,119338,119338,0.0,207930,207930,0.0,47380.0,-240.0,47140.0,169.09609985351562,136.78187561035156,0.19109976085289482,342.8854675292969,377.91168212890625,0.10215135348836757,348.0404357910156,315.78277587890625,0.09268365567579862,348.0404357910156,315.78277587890625,-0.09268365567579862,597.4306030273438,658.458984375,0.10215141480601897,0.7355467172030487,0.8106838610417711,0.10215142299109847,0.0,0.0,0.0,0.296875,0.0,0.296875,0.1875,0.0,0.1875,20260811,0.4145539104938507,-0.0028204023838043213,0.4117335081100464,0.013692490756511688,-1.5139579772949219e-05,0.013677351176738739,0.0,0.0,0.0,-2.8720626137762606e-12,2.8720626137762606e-12,0.0 +256,256,0.0,1468078080,1468078080,0.0,109958,109958,0.0,204046,204046,0.0,46818.0,80.0,46898.0,171.17022705078125,136.36746215820312,0.20332253740747308,310.8453674316406,348.80706787109375,0.12212406687322264,353.73858642578125,315.24017333984375,0.10883294772823704,353.73858642578125,315.24017333984375,-0.10883294772823704,576.8270874023438,647.2715454101562,0.1221240464365998,0.7236982614383574,0.8120792387841379,0.12212407028603667,0.0,0.0,0.0,0.41796875,0.0,0.41796875,0.10546875,0.0,0.10546875,20260812,0.44756120443344116,-0.0031893551349639893,0.4443718492984772,0.012462313286960125,-6.209872663021088e-05,0.012400214560329914,0.0,0.0,0.0,0.0,0.0,0.0 +256,256,0.0,1581717504,1581717504,0.0,94331,94331,0.0,191595,191595,0.0,46912.0,192.0,47104.0,162.420166015625,124.19634246826172,0.2353391483646563,275.345947265625,310.6004638671875,0.12803717269734372,342.5908508300781,303.7052917480469,0.1135043711407171,342.5908508300781,303.7052917480469,-0.1135043711407171,559.2531127929688,630.8582763671875,0.12803713012273654,0.7472470422946981,0.842922421688908,0.1280371469927848,0.0,0.0,0.0,0.35546875,0.0,0.35546875,0.11328125,0.0,0.11328125,20260813,0.6372919082641602,-0.0007270574569702148,0.6365648508071899,-0.021680761128664017,0.0,-0.021680761128664017,0.0,0.0,0.0,-2.2905178188176167e-12,2.2905178188176167e-12,0.0 +256,256,0.0,958666752,958666752,0.0,109918,109918,0.0,184766,184766,0.0,46916.0,-92.0,46824.0,158.83006286621094,124.18817901611328,0.2181065928260567,338.615966796875,382.0193786621094,0.12817886963750502,324.609619140625,287.7288818359375,0.11361566364646236,324.609619140625,287.7288818359375,-0.11361566364646236,569.1944580078125,642.1531372070312,0.12817882917302992,0.7886395993986166,0.8897264618223858,0.12817878090429868,0.0,0.0,0.0,0.38671875,0.0,0.38671875,0.1640625,0.0,0.1640625,20260814,1.0032663345336914,-0.525763601064682,0.4775027334690094,0.004201771225780249,-3.6475248634815216e-06,0.004198123700916767,0.0,0.0,0.0,-2.20278361523496e-11,2.3654630120981124e-11,1.6267939686315236e-12 diff --git a/docs/assets/task21-hybrid-mm-forward-pipeline/task21_correctness_quality.png b/docs/assets/task21-hybrid-mm-forward-pipeline/task21_correctness_quality.png new file mode 100644 index 0000000000000000000000000000000000000000..da9f12c369f372347de463585a6e558870f1a218 GIT binary patch literal 325723 zcmeFZcU;eH|2O__ph8L|N=iry4LdX>l2B=q<{2u{RN7Ls(-x&cvhN6xg zmQl5f{Ly5q$*%Ej@vuX$ni303Mm0OP5TB;l_{yDE{SFE|T65gnYg?G+$`gvsr8{?B ztodSiyk_+aK{<9g)?DAsiN!p|LrsV7)1~N$Cqll7|qT9 z_(adXT>{1b`MZ@1%IR1C$0zb9)!(6H|Kk%6k8%Cqi~RSA{@*A0KUe1egEeWKxc~2# zP#G+kXqHu0W-a&U@u8txoUnf_zX`QY($RA{_il;`+kTGe?Af!sjy}}+)&244<;&Z3 z90qnRTX!I*x=Yn%X0juIfss*WdVE04c2GO3+_J4?(T}DVoYe5Ik5U?WkqfV>^BBDh zOAQbG{G}?NP&jd->~Xa7nN-Qe%xgWx49gP)S5Dgw`C?fF5faq>WJ^^ z>qUR&xNj_3QJ#4EaShko6;d;N0&RkK?{7>0_wp?kEo@AaVeuV1M&61PsV1-3uwg^i z#^Rc1ruAuUQMS!8=H{Z_K0X|EVq#)1+sk}Ls*du?$jXkK%&Q3Ct*8tYW?i|`&CQKA zLj3sg;{wGs`d5l(cT;*ZlOuLrp&B)7mn+Y{`uMjbC^r@Vtpp~+TzfA2MmtX()BpPP z!p98jUKOg?m4+@vAa`dC$I5k5&c6Ni>2!j}tk>){=aaA<^gZIo)z3c<7Iuva+%P zts>q{Xr1ZnOP4NXT>2`m7$w26c=2L!BVqLv*X-;)j!sT(9UXL6u3Yiu+NF@T0)JVy zT9k^~Z+r9k^WCmN?)<2U%1#sI+*cCz)0?<9WM2swANZEMf7yIdj9cvSZ*djLTzAl( z@4x}=?IUtY79HhSgi@dKY)siXIQ0ALQkmHFBBP>A_p1EQV%qg@6;-A2@bDP^_3IUW zEekJi!04#;PUC9&b9vS;=O63skg%vIKbN`ROZHr8KCGa?u&p;^KNnMjmykXANK9UN5?9ms&$wPR?qLbRCcMbdmc9uF;-5ckbEcCN8YR zBe;vVYk&RsjVaASKQld4#IS7H;X{XL^R5nf?`gK9yYM0M<;xpOViOa+-rw6pM+F51 z4R=Sov{yYiDu3!!MRojfeN$86&wuc#!$U=X+DAGnUcY~jhp@)VVdEYvdbf&-ik^W1t*NQ$-u?R@*bX&+s;N;me&H=BDCmmkQBqPuUXQQuvO5tGFKcT# z((`{ZjgF3nJXaeVJAwLv`(x$eVp_g@IbMx^V~&NWxOmdf-)dJQZ>``pcy)6*#W&#` z;O}4FkY!3k`K;QeGWGR^?a;cy+3~YfG4`XrvGEp-#}f4g{NF!D2cVMVk2jdEK46z& zmoFY2Ti`I!UgELzQ&rW=M*RahxwHDfwV(775)yc@n{=*qaXd5o_Wa$*cI_NjTsJM{JomfXvg5SC#*Orx z+qH{3Dgu;MRq2I=g*DHfecp2R z<6&D_r|;~!a$iYRRrR-z5)G;@E%maYA+N-2YGO!)V%fTFn;O?~1A{nGi}u4PkFsr9 zHF&wJww-yHVWy`S^C|K4W{-iLJ9oMpe}2TcX3ZLfitV%Z2OKArdcI^nP~>G3P-HUZ z^W~Q>n#K;O*==7k4TVfKSbu$gFZ^`q{QJ9%NB!3-H|T`cWAPR+xwMc6Gl{L|{x)1h z*pXLV-KISE>cCyi@7>*xt$kho%xa`?u39Caq0`jAoxx4-;zdD<1sg;o@n@yu%@2|L zHmnmKI(W6`)B4Q8G9S*jDar3XMD6F8dtqc^A{2Sx+7q4p-6GBcsNsv2ELjpE`TbqQ zhPqS7kF%lT$D40OHC?}B$LZCQjyaSxS;mca)h?&rzk7$r<<23j&T!_;8SzCNPoF+9Rj6uedZNeSrSaUB->6yS zd{FXtC!5;+=RLoFb>}!tY^RD}zKlImo{)fsN(zRjmlq9{lb2^O-1-umNm@l^CE9}f z>BlG63ku4o4FEfcUi`%O&xH%JM~^PS%8?G=e(pA^JNCl{K0XRvfB#@}fzR>q?J~N$ zy2fjJdwYj}{ycQ`>h5NTp@S$ZGH1_nkB^UA+PF;hZg!k(yVX;h%z1O!8ve9(pYpF+ zQJa!uMhCuej*N_as;j$+7T4I=7?YkJFg-D3I5F5n${Z=O!WwBYR^1;g4c$FFUcP<1 z^w_aul*04peJw>&@7}*3L4CjiXL)=wss!btbT?|%Ap-;c=4;(NRB?_)M`^9jL;LX` ziNiCeRa9ELx;$oP9N$Gt@ur^5K7vNz=;(O+&K*xw;g>I8(#YOo4mCa7^lKLIOYFjj zwUjhIPSm7EaOaigwQhWT0s@Qi=*m2na+?2{bIPCUORl}Fp$w$NLQk{dsQ+O!9(_Z@ z8~7FOw{CS5U#F9omoGL=FCgWKv;(7YwWxlF)?ymP`v=yPm6eGC?ewP?F&{m8bmOK? z`q&Ap#SG{Re}BKH5FMRYz9m2?-N8uh`;O5{;q>C>=5;I99VialcDB+n!0hXjMU>u^ zD*=*DQ>qea2OY-s8lD;BGVl<}$wxXkIBeX!nOwv0_;`7}uFJVzXW`?a=_l}_^Yim# zVq>q9oBr^DxuK!qVoTvc*Pzhz^Ka+o_qMj{tQ(F<>+15#$jIz9Z>0oI+*NJ+QlEPE z7WP66YG4nZKCsm?s(ol^a?-9b&t~a|4({SO;#snC zbNk>us$TQhV?-}Q;0!G-Z48<&8X^Hx7posK0&kIT!>exl+4WY4CwzY2Bz!o1wr}|Y z^5szq>v}G|g>nI5b`=U2VR0 z?b;zFrQ7c^BO}+w%K5Bn|2(_ra^r2&Y|Ac%kprAtHDfZa^!8UJC~tUKRwgHZ@}#=v zsXSWW!~Cqk$nK;V=ogff9I~*9LZM{9Y7S0jc_mZKQ-1uoA z)FxqBtkf+jza~cvEo_91Izqd8KR<4-6hYVISh;fLKb60J{&WX0dYkf2MML*qb+i|1 zDkD}TyUOX)9OK!Sn{18t?MJBUw)_Ebt>8+QGp= z(VaV0HS3-kpZ?L9%XatfU3o>tQq(A6jm`Z0w3Mjhr1?&h8hL#w!h&909jNr%v}Fq; zfcY!X7QIWCeDS^9sI-ZtUse5!ii&bnT%)3R&VRgr&d@Nl`*q-k_R8(a8tLlx&A9HK zy3~8`GI57|5)xcghVf?s>)vX)v?i?Es&ri`s#v6G#w{d-JwEdvK(+v>IoI6WYY}2g z-GF@-x6V$qXlOd42sGwjV_WG(zhuQa_u$}Q?I?6b{ra?Xl2 zEACw;AtjlkV`CimZPnOUtukmSbON>v^RWS{q7>9m+Rx48D&`7((3rZ0_Zd>HX1{go z*4Cb$1hOdF+Gq|QJm_Jxd-v`yg--jaR#2QA`!SOUC$+JusmV#t3IH-1@YD!PeBY&( zLY}<5JPt__ITO=JZ>27qbmJWtKDhh)Gkbe`w{~I$a zj=4k|U%W^tAXC5m#+C8Ob_-N~m+HULQj!2SH@DceO`@VoY57`D&r_~TIk)qPh@@9` z9IzXvu=p65nB3yyHxcNEN~E4!^s}_AO#j^-F@mE(t8l#mXMW#!E`E!dhn5096+R)T z{2kkSu~N{*od99+W8XhSEu~(6_)v~M8(PeZkbE`iW96Hp-QmM$?jZtlKv z-!*hG8>{44=p%k&+mLrdx$3{V={5AVa__aD8!6IH&8@6XYgW8|ozPsc`^MtshtTNQ z!+?!8tY1$Knqyzhb?Sv(=$he z4&_bxym;XduOX6S)4z%;uC2{_C*>M8XE+BoCuTo-(PJ!RRzLTNO}xt}Xb(rQ{X-Xa zk=Y6h85Rf`iztAgz+PGVbHEvkwxu@YzPGioP(Gt>U*^TBQKek=+kfr8m0+@^xmMk+ zKYsXbIsJ$+kRXV?9+Q!wv^DsCa9j&pJhs+NcXvuq z>)L~@O)qRkp5`CO+I7+>p*^qu=021*R(h@TlD@`lvS25Mm=)J+VP)+~En9%*Fb&C`Bi^jVrwxg$yeHIe)&rG1rP= z`Eof+OL0p}%hDF30A8uZ6#4+)u3a1`c!^5U9Bn89OP4MM%y{|j8+UraIM)&XwO&9E zF@OO=+_x6GZK3{OtyXy-pz_G~dsj+6eoSx7i-?G@tiV$DK@DsNrU17pNm5JBFsT*l z>FFtMYT^NtD+fbhzdXQ~WjmxZGR?F9YR$!;$JNzW-CWKsFt>gTsHFT|#17L-`QAYS zYqZXu)yPY=ei6o*O?NF2gc`I3C1q=Nfuiw=d%IJIzZjVXhhQDV{Ie;5Q{$J?_p%Ad zG9M|`fjm_;t)bd##v}2;rBrR?(1mEHo5jV_K=~_im#8ND!bEhLQOVFP4X~N-h#B5& zd~T#>rlAZxb7#i|=F5#a%DqGF~91yCR16|*Q)1c zXNmfA_W7m2W3C|KOI?F%58`Ij*kR+AD)2omcF{R zg5$CkK~%HT6CLP711X-|&yG zJA8Teue$K@z8n-RHRrJ4 zkDsIIh%6YaV}Ky{3JM%GAk(lyJ>gVuopgK!xPD5lnbXFV$3hzt+Z}CeY%JPK86)>u zEp{0H@e zoSdARXU-ftb?TJkij*^1OpJ_-U6*gE)b=L>%EraTS+*>#bxNW_6`z zHWFO{{8!Rz;TvrlY5d*_MsY9CJE^$w2G{b|M6EVyQa}OF+KD9*9e+3 z^uWEnz_-BDH-KckgcX_!ow%juCO5b~Nmo-_zIHFm3SP-lY*2DTlfQqj-@JKARaMoO zJR9NXmz%uNY4qQQZ8MtNzfc=%Zj1i4FcOyC+ii9TZ*tkPW#+xro8rIm%l~<;7KZoX z!?+!)MvmieaprXgJn5D&^X=QWrlZ_XR#i2mWb?;!+S(ZqBS(i?cu@htvr14mstTQ^ z`=a!={QmuW^^Wt)z;~BWlcS$cWjg-03?0)wheaZr zdmrP<$~;6mpMjyFzJY;Tc8l|iN~gm|k81Sh)$CS_Os@e_DXrbn-ri2G^G8b&FB*;< z6ci$J1F*cFZOcv?sQd}Fm|s|UDWr}Z%PuD9h8%TgUtBfg`1srax`#KY0UBzk7UZXf z)kifnG-f14L{1qW>`jvKt_khe(fa)OR0ZVTz+d=6;-!&sm~y$% zC%R+Dt6RLzp1}4jixw?nU}i1{WX&{sy9&41ft$T?@7~!gmzI_Dh@qBK@ee>EzJz4; z^!O;fo=>p~!Mac!h4Q0(LhzKxO=AywyHHTbOQFt%Jn#LIxdDBQf-JOAKp+my5kh`q zO-?0IQPGq4TC~$sdbmRDxhRDx^-#bYS}N__3xUhGRF%`x(!gOm1x=He8~;4jyL;C$ z`;$-KD;4fePW7iND=8^;iQ$LJhODft-lp`UzvA^5H#Bg)dGn^HKAo2o>8{M{L1xD1 z-`%+mYEDBzwYdT1|M?lGIwEn=2x=vsM}-PC5uP=)x_b5bWapd@)6%6QSPyi9T5WMx zS_*4DpBv!kX(+(cvb(!XlWJ~_<~@A;*bvx1^zxN|{@M6E-8gA)%TLwszaDyDl`eBW zm7-fz`0gK8H6l<`tzb5U!gADMf4-cXovxMi&&tZ0Trvfj`>1{t1fL%uD8{@tU3nT^ zJ9f^i2AZ0y)zfwSuxDrzn)^g3A|X(lytFG<@`5pO)tGkt__6gexMi{e%x zp%U8`r(5ZTQ%vvg?mFF2fi>>4-*!-N>)Q_>Rt0Ydj;n$sX=w;}P=db@{pEnm+*&jX z&gkdq-uPjS8QJf?YjzBS6|3c@-|d73z{n;*3I5*D4YSD`)HK%~uhY_{kw?;TFQgiZ zLXmrg(q&Zn^vM%Zn?BVMn=|=#nI)+Ss%FqSh?3zNlqwAiEXkT9{Pk;jVt3&!g!UUk zbD~^#nbzrz_0=+EZ_9-~1*pYutDV9`tgkZGD!{Xm!rV30)7 z!5X*`7|6Pkjm_fsPb1(`ic(KKqt+(1xZ&&9HkcY@d!c2D!U`m)>7e~+Ld|b8@IE+N zJ1FT>r&d4|vg?krt-SOLYgGF9@$swT2M_W@JO1=~CuaOa#@=3{*?HP5;t8a%lB%jq zl}y@AE~Sc4yRf>5hu9vj>nxM5Q=q4>fBoUZjf4w;>B;?>n-z1OvCE7)%>NQgrww`> z^mtO+Af3tnRTzl`g!V=$%())8(wwhQp{@q+p*%Ug&>K%xA9ptimI(`E1r+%WXg{a! zACR%O-pRjV0}a*O+zfXJYYa;e^sWS0Xr#%uB^FK*X$$ZXu=wa8&lv(4wllG+Kr)qs z`-tv&Yx}vp82kpv;WU&luhcrUErqCkEnsL{wR2cVb4pF!23SCcs=s>m>ee^Gn{%vt zgYK}y;L&~g=~Kp;0iiy)Y2~n#$m`J1xwZ%_)7;wnFl15G)WF-f`Q~T}pM0}byLtcq zIv4_ePz3lkZCXSi9vDGeYpW|BSLA+MW|)6p(BK44Mkb6~U=w-cTlnz~V9wkK59d6S zec3N3XRmg?9V?;r8TvPJ+&=sCRobNYUAPE6e6sb%ic+;>0;ldTb8?zBDKpU1b3OOs zs{W@K8M~>`E%u{5*U^$nAS$BV?zJ6U3vt0nhkN<*-`yR3-%NL4KNG*~ORm-C@gHAr zK!E2&#k^5iD4C{PiFgF5&gKyxI#Uh@RBF&@7E>hW@L?9$mn0Zc$NDP~ybJSHdFF zO#Ib?=UH;6a;oypo523f>(|GsI02^?s#t`Dg@Q)yQmsspC%Mt7Tx)bxC{{7vWkuQX; zjK>bvKZ=WHOvHO@Lx1HY>#zLt=g(}bM0B>Yb8;@lci~FmyoQ+;{qtrel*{wy)dw}f zPD=I<pV7@VCWg;KEf~fN zQC;9wET+~Tv}Z@*&B)HK08nNh4-5%$A09S`@rYal9ni4()vHJF9zx0|zGM7=@Sd|M%L^Lf*;Ykpdo0)=BZ%W2!Q4p`FC}8>cPw57$i*!{&*X|IJ;Na7_W`Mc%@(rJfp>-Xunx*Ba6c`Kxo4NYuH1I!S%T6uUWE3GtIH8ac>VDsr^dRV%UTZ< z0r`=;ARG}il1}jJqjxc0R`&P7Y( zlaySirKLsGeZ&!T2P+SOgR|l`hzoCLxA)nzXQaD=rh5-16XFV1NUR9^iNS53p9`c7 z;lY@9f8e=z@nZ3tH^&;ozGD%#U)2OJN*)iBuPh&dAo_8~LO_M-8yF%9v*1mlg3F-A zm$vvod21iHg8dS5c9tQb*=C6=t~fR(h9WQ5aBI4DQvS!={NS9gus>Sc+ud4Q_2S2p z`askfu+AOKNR5W>L%B|VJ$*PT{e+wpn@HumcctiaFpls&OYQ9JJcmZmSV%hHVpZHC zqIY~AgmX^b7M8V`)6^)FUntfHQBTm>P&;n<`1s^3QbX4|8M&8+vYQ>N*9T7dynE+P z_HYs1#Z<0ZwU7#Pi1q`N0e)xzK%bwG@qvF@m3nqh?#N&1d-15m5`8dhI_$W@k|hSbLHpp1)7RKcPZrMdBL*D z;W}x-#=f4@of^S@CLe7mX-tovIiD0CDMcC7hyOcDg*5@-r=us3R~ zTDQ(0Li2`g+m-`OWaah!O#v#O_^e_}{nsyDeh#F|d3s>a!$*(Wv2%|=Cx(RWzUg>a zf^{T(BqI9})gCFSD&SV^du&}=T&97+JKQT+utTW-iG|O5c=CUF9!dI;gC+q+N~5=J z-I{^k1iX-J9Z7N(XqbDQrgjpVq%qjj;|Wqnq@sUqCr|mNjVP9V%x(+M+0&CN42gtR zfKJL4nVFZcpbBUbwP4%Af?bMcLL9;Z`!Tn`KxIF_aHKcT$Vl)C3LLbbzxBZ6lHziHV7@qwyc;Q5x}Yy>!lFE4m-%C(y8lMGS7TtuJqw3Lc7^GAs3D#V=bes^?06Ssd3w zlGJ1vT#aS*Y6eQXFy!wm;ab*9CxuFN3k=-;2Dvbm9rKs0e8YCMN14IR6|&H)=4PFy zwtoYviS9dTZmyUi|8s!{>(E9nWU~1A<8FQjqa;Q?)X&y|fz&~B@{4Nx=V_h^3kS!h z!GWzOKJ2yZWI$Akgiufm^?Ix0#m(X#JjgU}J7nGSi3Z9eOhjT>HQRM8N2ae{p|RF0 zRxMc_w>8=GAfPhkf{6(eo|{PFx4K^^d>)=F42P(-L7nL1Vk1w1~Z2yajuTE z?{2S`#mg1_=QRr@4U$Z34p_=Py}d8dds6NJ-IACWuj4RHp6w7byd&h|O5VL&hDN9P z^!#!(n9%zgFoW?SojMwcfg6r`cmE`OH{~-TB>45Pz_oh!hAKz=3jaAaR))o-cj3Yk zIy$=5Yu3E{`gIM-OIlk$E3r^d(WJqpL&7@5LZAB+h1R>fIaYRwhtuTbq%?mZi9>AJ zvPIlTUtgcdu5fy3feGQ(yh4vf0s;E(Vl=VJd}b^T)`|e+f)JH-p8I_Pr90+WumID9 zo)i#penCMM3`78!7&Hl@BUe{fSFZJ$fAgHrx4F$Lc>N2x*id+hTv=@LHUG*lhxbPOqRtDZ83zSSof{a>4l~U zwBZAnO-+5izrW`r6~MjcQ|`-`lr#NKaZYA-@Vpp0Ez+LyTQpXdu z`}Ka01?sW*x5CwUNjsaJCg#I;_(4G1GBAOW;WpV&UAn|Roj-R~qrRxxn9;OCUtJCR z2e*Zo#|pe6$MLTh$djbXth{v{M;77f&dp6*M?P94(tG3jqu~p!2@=vw}vK`2h^7t5RyJXG4TD(v18X z7b-)x&{DDR$;2y=T&Z+A2fv3fsQCIegxd_7GBQL7nX_~XkxcJ;8!pWA_Ul(IB5+Fy zbML;C@X1tWzFE{Ly}P&P6vYRer-C5m^-qsZaOXNaM$kxxg?{~p4YI3uU($@2u*UD# zuunfXbsvB$Ds;Ps-275CFbL7%61|HQTJZsmw_w-!!u<{1p+CE_eF~6wNFzv(ibx+Tv zTf$^?An_}p%LrDD!JD8>Bwq^Ea6Md&XC<)uM2#xh5zvY?dK>no$VCcS%i9qVUa-Jp z@HP;!K_XWv&r;qdasN$Ztr^_-^ne^FSI}t`s0b0Hp*+muPsv_$An8!Sl;E(in_g_2 zOHk9`a~#D2qL;aSjsFDVHFyZ6fIR-lyz+~Qu>vbY)+H(>NhARJpeQbe%vN^QF`zER{_7 zgp@S04Hn}+wEi(NY6H+G4iz5lFg(Vv{mk>dT3J>@Rn;A6TN=?Z)M^`TaXV#E5R+f`|ewj0DvXcB+?;?DL0TrheWdnCz_g*W`ze9e4QI z>1FrlTwT~lxLY>@0#-==nX!T%B_C-Ul%~WDkm@`;MO-$shG#DkB_+XG#1ByT^uROV zvQ(WH7~))j6}CatLv(XpN@{9o`6Sp933!2tlW;O@ zz`)2zUtO134*cv9L{fE}#?(B7e-P;ntLm_`v*eE-Kg^L`1O(C#ERvN|;NdV#++o5W~8Y5PcGg6l5%Ikr$@mV3J8d0W7P5rvTs%y@wrEjB% z*opN@DT#A}^RIzf%n!8*4$fi7rjQHcGX|gs;$7Wb!OKh`)>YKl-4B;m$WMzSnh}WX z<{5=&z&UX$3h5M`^p___J|*^8Z*JJ5lr(~aEU=Ke`mih+cqFV@C3u>q+6ndyb4&#?1BuI0ukRmyb51BDB z(vFso@h+MAL!8f?0;5LI{j{bC3@dV*6j4sXuNj;YLUNu6MbjNj3)4!YPXf~@x{wsw z2N(U3GH=6muk0_F9(;ihD%5oV@9wNjenIOiWRNh(B#O$3Eo2b!q*g~~6uQS!={E`w zMT81yeVG($SUu4g=|`3_+^K`_~D84$k@@p<61;mFB6Chjtq!|#_k8^f zSg4;`NW_q5=wXD6v^2OprH4w2Xmnt*A>g8<8i6O+0Es&b^H^lnklA5IGeq9;dU&`- zgiX;@pD>gP)oUpb0$ZV;E}}ey{hk?rKGOJShL{=_w3|-F%6F)$AFWL;k`J|+YjOOw zoYI4Kdk6KVL;?XN&0Nb(R4aA@B*@?`;pI-~qY(wMB1o5D<5>6AFe0kDBxa^D3o-*( zv_e#PSlBA4OUp1`^Gg)G=8MkV|9{3Ub~rAauMa4oGkoE zF(NJlE|F=wcOJP9BqvicnxPztBKd(FtVKu|Nuy*ud*=S=(RsO7;lBVEgd@X|(Rl?d zMMEJHi@xzl^Xk;7@d4+V)zt9s?&w!Y1f~^07np3oWWeh(-MbLg!So*wA2|OZl5x3T zkW}KiTuZq}DM|Ix9Q8$y5+t4{>3kyzUy|_|F5}b{jP%I_UaR1B(Z2pPDYC&?iiI`M zWZtfpe-Ahu8ivu1V>3Ecmf0$(sP2i0TU>+a>C}>zim8TLi(f>9xqV#18$$-Mh!51R zOUE0hT<49aAy^F5*KN*_JWIK|poGgQvTr%p4Va8z?{m$gwUrWoVC+~N>w40hOnriw zKQfJH|2E`U$A{OirLooR@~sU{1*|&=o0R?>(DA{P2@g46}?!=GJ3yxYq9urVDqmcyCRW-Ze}80%=O2JB>RoaCqgJSmh$Nzg05PKoHo&;vYZu znYd7$OK&7$-NWacUitgxd?7*41R_GAxt^SC8e#J`?_&YT`_34-RtI>nd*pwa@IQOpR7DW`PXJT5C36%ubz)+yk8tG2I2iPIoK zsMk9iMjMEuQC(`!!J<*;qI`!r&Y=lJk!uti^jmHFUO2+zr{I6#~H(xErn@9kHhn z!}Yvx2&Vfwu0}LDxL5{^@ZLf`;m`uWGpJ@Cx%?t|RD!_~47ej?U&@DD>-_8sy zVz3o!LMV4tHRGOMoe!0Ot=`Gsi4MfdT96`SpRU#HC~LG*O-NQvEJVj9E@j^ewfD9d z9Bd^i&r02qRS94Ep^v|1Y8euiSb-Sp~IG)LuT zr>ngjyA^c*r=SI5#_UG`}VZcjGJm_6IP1qCq zpt?^vPHNR3Rk+(~G1#~dnM2Ry{E(BbTh499LxBEjX=8F6si)F6B&VjDEMh!^Cj$>V z>D~BI6_VQ)7e|DW7Mamf31~;q<$`;GxmI8*U=c-ZS8$`7@H>I5jqX<`zV26Hg)6XF z>B%-#EjIS*k2LeolXXiSviKITMpV0}mT9@Pfx9I8A7L>{}z;q{nm5l*Y@hn)g9{IvdypMwvMhldx=RSzML zKLT79Kj7o-O$I=6$9Khw*Gt#d>xSYPSJw+8wfk=0D`f6+LpA%{lAHxRJrnxop+20@ubwL1%}w;7-a zO5y&Oq(lzAN&^gKp!LyGus=PZh6zp7Vp{CZy?Zcq+U2YI_F%$ke%`kgh=aJ=+B}JH zqDOk;z~BXWvhyd@VshI(Or4;ashx_MSx@O0I_vb*Y4A2Btp#xvhq`$Vy>EZ#;UcOt0$hQ!u*okm%y;^JZo zm?$+^OrDJ$(Gb;v}7_D}Cod&lX4?Do%A-YfM;*C>bsX0IQUFT{Lo{J_X1je8Vt=)fhnX8F<{s0I`PSdbAhj3EC&t!hg zc(osJvo7P2lboA0gd_Nt2O~HV&#$yFQ>&$%^K&*aQ_Hw^%4@Ix2eB-K4}(mj0|T#P zfQCl!Yu$zFhsV?oh6?V2i1yX zw*0YUGQf^l9wb)aIMHN_aF}(jQ^dV{WKfctJ3E&%x=LE5+I@vFkIf~Qjl1w|L<5k1 z^RNkXs%~UzZr{E&m>ee84^T#8o+vuP2Y3@XKPH2ATAt-DAc1`O?~EdeBzJns!K^Wd z6_QgKemF2XIslp$Sr&xcDyY-YQan677KgfEIFLLjXfcs^kRBkE9~zrnw4~EN&-Lba zC_n!-;-(LMLzo?!()m`nwJ>>!QbSG*AVDuGB_$;TE`Y?E8=w*R44ROXjlVwZA0jg@ zBt*GlMFn_*FVf;|s4E&=05xfQH8RY;?QSe|I@?zF0@-RXu^Si@xyj6PX5iNn3c}%? zyLXAA0cWfPTZO1Mm?+!;9SQi443Z-=;Rg%kRYk?oN3-aABZvkX;)RmTCiXHyf+P<^ zmLs`=k0Zd$r&r73{j)&0k-4$B4U`!Z*=1zBAD4o~$AF8ySd+ktOxT9cn6G6c$z3SR z?HwIe03sIL{V6G1i3|n|6p-Uly{E;I#C#Jw^Ah{GR41nDALg;#L!NQ>v0<2m{}`()R*35)^IxX;jBXFEfeBiO}-22M;!oY+?%XpX%pz(f`78 zFC*<^X@f-1h8^WdP`&lbd?I8uhu%>F1q)VO+^B?n5p;u)aVs*Yo(EapLput(Vk~?T zgQ!^9TS5avSUuu4x-(oT;>4nwbHt{~mBb9&*07^Tj)XaEgEjKVIv<^e5pjq13r5)1 zUE}{&rvK8TpETwnZc*|nR`zXv1*t;j)8l3y15mq5?ZF{atRdTDUK#NicF2v7tP4Bu zT`6S6#Ac}>e$LW~%o<_Pw)ri|>`;8m)`&lR*U$A=i~c9V2II|u7W80BmO~RX3^2fh z6~AGzg?~GO6gh?o0vf?Z{}rQ8MoVsk^&@!c3K64o%w>#cV5GQ_L5~rzFn5qDAv0YG z|0mXijan2R8PgDYvDi(ipkJ4V3|71T`Cenf|GZD%qRy%D3z z>AE8IlAAWkWjueb{-x06&zz87ZXO{JN$ch(gH>0Ma)E<5D!|Cve*9R5K{@Dpv6zLp zv(HKZp(~xiaT%NSHxcdR{xdf%vUtThUqE#0l5nj)dm$0{o0I4cx0|#A|^uz z*q9qCOSK|Ph&<8}XrOJKovKl%!!ixZm`IN0FRv&5QJmaTMI1XpLqlUE`DQGxIyo02 zvaee4Zsmdkf&0^P%F!X zgN!$$0v9(n5>E>M0kVP1^Pw1P<G*lPSTgTgl9y1#p)-)O!NsxQyq{3|Z10wL%}oN8D^;;M*-SmX3LF>#9}683w>OkdYs#{R+z)k;6$?OE6OK*{F#))CH;t+kF2| zcu8miIyg8AG}~I91;2omHBiBEwCX548E0qbK4y=ElO&`K(oI9rEnWtI5UekZA=0TqqbJ*aL~3}1a20Fjufnr07UA<%E4?08 z>yi|Dp6tTd0Mv04Ox!z#e)qs(80mwbk~bofgmQZe;DWsM2M=zbBF>X0P#x^F49vOa zym*nkUzG$1w%II^{x{e$|Idf7p$%kOb)TG^ngaf0#;7xRku3HG85Nsf1+__`d9OC| z$4N-zc&?a1zJ;gxzxkKk0CSlmxQ2LPejU80_dlfa?@nEqL zVN|Ht3f2@`ocw!`SC3Sd*ydOCcNV81hXoy@kNL*Jk0hLIJt$tg>cg7_lFT~$c=upn z1ogT2ShTpepP!o5#=noL?}Djm@*qpkrkDHHEC8IiKQ-l%IKC&~zduQQzv_+u_wzF+ zjyKWNJO|dA0lFsP+51v1{$zX?NfNsbzdbRjspkgfV7DW>?FTan#GwtQilq(mQ6zCe zlo0@`qv$lG?xGTr2`tEziC5SDy~Y3fhsaVdWLooXBoHlz1h+Ai(^-jzcoTk4aZSy$ zGlnD~g}56n_2tVKKKNg0Cy}H!M}duZLOnFZEE|?jviep^&(w4&#niEorRBH(IxHDs zvb4qY@9+AP=r`RNetyv%Bmqf^0ogfo%nP8i5Yrfg9dwAZzZ~nY7lZJ>7Kdb18eVG3 z;~+;b0J&vi>Mwo_0|uYq@-E}vy$muLEMPK}2u|k%Wtf}+GT9k?YPjgnYTfv2^s&O?xc~gF$*J0WdC*Di!=lxZkLOpeuxU3i< zzKGaqI0yuOsX!P$e>v*NEk1^vYrh0=mXmhx9Nc1nizUGO?GFMZOHgD()2QHx2u7qd zJ!ZG++Ck1{XEi3!r zkeOYyVwT6yF0%gOraY{~4|AZw(cq&o;Vk&xLFwV{!C>Lt`)epN?MI24)@_aGz2W#^EX%NNZa5hm8r`!N9F+kZuMLnwq!A{Ep~*j3&APr%nKPMqXT>b9B-uT1qLURgprYBeYmanHd^7Luu|CY$cu9aY&_Oh^DR zAzE8oV>0|lCy6dnI{}_Ef*CFn-6s>$W3-Eg;R4c9#4kmkRevEhzuA*`@w=%I#b=nF zj4Sz;%Tka61Q81ZK%fLh97+q`9)|4OejUy#1n+bPE-oK0+sLT z3qu1TaE>T=sMARt8z_>3YH^<420fPm2=$yO>-4R1$b)r0cf{-Vep4wg_hXj(^~Vpc z{@2AWMdg)z`l8yTPnzdO23qtvnvlk)=5Zbg?KlA|5DFXSXBgayH;;NIxlTxu$29t+ zJ#v|HE#h1;c(SkGyph47Kv5CF*S4(GLl~iKXcwMp9pv$#_OJhfPXKpgIwG(sU5I-ZO)%OGRs;+ zOHOW>hOA4@QAqRO`7}N5_%CwsRq!dvm)K@v@Rv=>IM=TA#V9=)Q~-L$XqsYtLq&PH z(6;2u2F$EgZ_8aKgmP}V+)!UhW4B(pRw>A5@S}f>-H`I+{_XA+{cmg8HnOtr+6&vJ zrQT!nCeOa$>gb5-=uP}W+(SLhp#6MZ)nwP1fxd?5W>_gnX{fa7#dF)}a|Bj{Gz8IEZ(wZ2B+ z5t|v?bho3W^_Ia!BGtDUhCL^_=3cz?THw)1wsu2*&y!qUbv%&T5GYRy%;fa+tCkjC zBw{RWW@cw;s3-}WB&F!;_zW1SA$72850mqX_+SpE%>aIq(I^y3l1K&K&N^hs#v0fs z=dtMJ!I^uOn_Z7vt@yK8S^rhe#=Q&skUM{3Ap;qAlN`HQ6NXqV9Yu&xCr2W>5R!cI zk94k)6xKW!ug5CJ*@IzW?U@+tt9*tu$WCx_@5sotgfl5Rujf~Jigh9@tCHLGb>~Lf zl^-vs#r|#Es(;tO;lnjb1v?o~6UnOrMndUxsGGNN>0FE#} zC}6^%lBxk_XvqIOiEv?ThpJRQZUAGjIDCkgqDTT7Q>jdFcF`zF2}D~YiI8W{{;~C_ z{P2M^6PSuP?(1QYChEL9oIX$l5)@NlGsbPIimzj34c#_Aw|upNWpYg&mT!*dy`<&j z(S_I}Mg%d2K+0rXGtL%x35j<;*xzyZaP3P6-i7Lwgjw|;-?CRg0qJ9w+jO}p&jT#9 z4aAf~;^f#3_1|h-{wUTDK)5xZ{KI&;rO*#~4FnxyPDM*jA};d@0{9uP2PBk0f=LJ( z$B!XoV!CM^e_Z%E=c`A?8ZN{q3a-Sj8+%?kiNy_Bni5 zWYOPeSj!U`uwJDgd=TfXC8xK;i=WT61O<`l0J7JOXR(7iaAy=nM|XqCudTP&3reIc zGN1atQTH*muYFkx1c-wRge>W3MRAV(!15ES?2IUf15Bsh6%RQp?{(b3to8d(s-1Dd z1F!nL2EDr%g!i6NA8I+4#06y)t?+20LAa+p$KMV1&^a<7@f(?5LaHA#;c88>N*wvt zjG`(S<3@g7B>O%_kzfrfB^Dwn_VyAcDzy{mzy{BMm9UYkvnUYSw&?z*o|ok><;G;N zjcPMe!+G`Jdx%EKI#|F!a1i8r3iFnbz>ke{3dsdiM zQ*%75Dy7!leDt`AZ{5)&{KXS$U;NI}StWl@T7mw7Cro21|4R&#(O$b@zBKJzH}p0f zp|=Xx;(^K7k|`KGlhD`5k)4n%FPfS@K%5qO>9gwED-Yu02u&yFDuA6`BxgR_4lct8 zkH`K@fC6%E9~llounALCRX9wCShrwYCEd?2d|XHL3WSR)P;SaPisq(-u`+MjHeXu; z5sb)L7VsV)#>dA3Qr!_ZV#0t`=y-C{WWuHSsQ_BNeR|uyC)Rn~JS@{-v!-EH^Q-si zw;qgf@7FUpuO(ZBx14wqo24#N(B(<|vt64@K2>BIN1bubhAEC$Q-Q|zkAM2vY-ZEC zl$RL9My?3cZJ6$madgbr*2T$n6+ro!m;=3daV9(OC3GM%`3uYObyft@Vg9^Q%&SY7Y(Y;sC5b;m3J{x{XYNKNdn^*I<)-PFC zmLCSjc_>jdEw@GceD+p1>c^;qZ|wu2?1Pt1fb(<0EEjd2vHPn56QLHGp5G~{S$2qc zb^}@UrBld>lj%niqr=j90Am72_Z-13fubyhT#X(^gE>!n3Q2xIg7wJCk;6@qGyQ~P zOv!<2m|De&Xs+)b$}JqEhPdDgGM*nT$wLB!Se==;Z)8--hzU8v?(R*we?0hijqCH z{Q&PD--mwF7LR|?T04xLyJ%u@cmCzdN8A+WrK6DYlg7++v&$bR9P8JPmq*TeA|f50 z2?30F2nMqt2*}=L|F+#zuhAlH+8BpJqqR{&?l1CkRtGr@;bs9$-EH```Gd z($JtGQQAo|N+fM9710(=DKxZXmWFm|NJNPg4Ih%EC21-NWh5g_l+u#+x*x9(=lp;F z>%Q;nx?R^f=Q_vf+volMyvFnSSWl{B#%1VeMb*`XAy(grC61Ds{pgoM4yg4}r0q}*J2(f;J|E%ah)VKc>s zB5TL~pg4+`{kf}whpjV&)c`YDL)B8iG^+qNOyZ%605EvT`#THDx+p!0}YEDPx6 z^z7s&tax1H%#fHTZAD*@^U|>aZ84FGVm)-Dl91yOgj;G8d1~@1!NTqyb7cBh zjaqT=ZSmAJ*B`6%TIn0@y#*GLiTL<$nHWGxYqkzZ+yugHa;M7@O!M0a^hETd~E7 z6Xnxdg?aEV2ae9RQ@6q8aqj8`;@)^J5l=XIcXxM^QnBZ{((3&5UFJx46@l>%RWElO z;T_2xg&+1rnl(UqZMTuKUo;FxrYy@a-Tu^o_ejr{6(X&-m!_D?R2W@cC)CLO$204v z?7s1Ur9uPNBPwf`h?eix$!hJVPz3cwABX2ukL{lK^m)e&@U&&R1OkR?-Z>S4;H@^& z-UMAI*WZp49pN0pj@Ow-A4(G)URJI2s8LI zY(J0}p&)};78WKigYiLzUK}!}abZ;h6b1-O1U4s67FyFs1Kwii-~c7I9ObYOrDCEz zKqkSOo&vmr$Pht&M1z*jmZ+?G^=h}|l#UC~l48$JdLfn5B0{FR*xD0ko;+QJqW;3^ zlbe+!Ql&=Qm?=jGlq<$>br)YQY)ls!RWYQz-O~eVOcWJ(EsWhVW{Pfq_wG+M*WX$RtKtd+;*b2i8c<>4ML|({d7B814T!xHMYpuR zY?7jPror#?$5TN60NK|?Ln!#)?^HgmctxIaTL{~{Z_v!S8OvghYmxV&%|dirwf*<< zA7|fvpt~bzuohXklFPTsPW5h?J-Jb1BHONJ&6?6fDY`E+h4h3<1#`Z5cfpjHEeWSE zu?rrnd-duSlxjq>gJQwZAW$E9%*gZfoc{AuZf3WkVL0415Vj5@Dcm04bJxk8s9zv$ z56{ofe;J}4DSL4=_f7_5W0GCh>+eENCr@sCd4Y-Iw6R9~$O6lv$LqsqT)kb_`=cnmav7M; zTRB9oP_~~O4`|?#fqDtr$FRuA#fTb+fbks4Lv4d}93@Lo4Lqx7rVu|DN#@4!e-Ije zB0qv?;|PfZBMCb|%AY@f4u!c_9v(-od!y=N6T#`}Y1X)1Jl;D~!mRnl!j9cyFr%wnNUV#rlTtwvNOFF|&+2->{%71o57g7u1$-&Yg zm{Awql$dgaKKZJwJw13j6);VEvw9p|Hb1>W(}whBj7ccb54hi_n_r+C%#scL`X`n8 z-XgqBJpO3Q%2R%};og;kl)$GF!h=PLV#6cm$Ip&h^a@Z9pV1H#^HD5tIvDtlZQAT^ zT*|DN!ECdzBBRb__q(|{k!2od%v>9PF5k`mBG*g<*u>5Tq|UQbh$WhG6EZ$9VGK01 z31vq7om7aXNs55I9Zr)PY(8Ve`hp52DrXV~LAE90!$|(XD#R)A0b)7U@F_9VsW~}$ zZ%a4c^DbSBr9ugbA{!1ZnH>*^y+1N@bKTng#yjmUE^YkSKK$#N7R!kn=iPH(1t%89 zCW}jwmX!LW?{$5)uNqVKb;TsW30|xDep(aZAffB`?c3J@n(T_V`@S?^odm zx}<#L+&MZ#(L`dAl6VeWq_wD3Nt&V8mkcyiu=qWA$s639cE7HLbHqI%BEoj(Ic9}) z2ixHQ^=nIl8v^y42Q7A4G2ifhTKCFUU7RDVYvjW-W2c9%ocB{d?j6{KV~gs52K-VU zB|?*Xv21hxp($QhV-nr$)4w6;>E;_uQ zA}tH}x`zp#jLzcNA&_vCKwu^#A6rXSTM6w!tP7~YORlC+aiD+dhJVL=+n2t({Wx!{ zo9<=sSn~;K);#ZKY5pl;`r+ZW~~E7T8@aq4&WM>Ro;=$B(H5va@uT_yy7(9Y&w3ccf0?0PwZNPk86JER>mfRD9FqwmUr$wT>s#t z^nSk`mywG$f^AKLgnQrIVg|g69_T!vRCrXoiKe071+HFPYy_vG35-yw;UcfZ+a>u2 zk6fBrA$snTE2CV1G>ufRaE++}eIt}Tknt`6f^d#8QqYFj+*3zsM*wW%X@J3okT*Cf zH6Rp#9+fB@ajuWRKtOQzf>UXI`e(J@KiGEsCqvq(POD1M(wJQTbBWLL4(^|s{ZjG9 zg6p_H<(_VC=!tb}{3jj8H&NOUpUw~vKbJ8}9R z{0j83s=AtCetsU2xdQOQ(gBHYE%W6A_62KMg7rZINTGbYm7C^~fOJj=&vwa4nJ5<)}se>kfY+QQykvPt=c)}!F+*HU;) zXc*CNA8!46$McW;ZQcTeAYB8ocOkq+W~3OBhLeM>i=nJYnzr*BhftG|tC^XS=$c89(t(bn zmh2~-TzS?w-HAI8KV|N}K{}lrKOAw|S_~T*B-0)Ytyr{be#!W$^6-^!YmT-2+5CL8 zgk|Ay_Z#;r>8o3|S~xtS-JkNgqA%+x6(kA|BGI$Rm_RR+clrs^>PT6PiuO?75VHM9 z&fxa#l#Mph`#DUsv>3?cgGQaa@uX>V5#-t=k4|E~!OU|vVC9ivDkPXj1+)bd#HNHz z!*xW=9WPy^(J&G?LFvRrn~aBr@5d(j`6ERxjg6#Fzy9!HIfYX|z-rHKR3pI)_j!j; zKhU35tT?;naruyGcem|=@4C-S8_%UlSg0g2X=#toGi{t(gE%0pyCQTNaP2cwDDa}A zPH%Zp+!?v@v)Y+*l@PS7mvE~Hu!r8um zV6rdxg}?tl3UPnZ&$Y$*0rE<9ze1i$*k!Kg-BH{#x#V%|6pPYhAD1W^>}I>8?`p!G zxY11%Pdt$ds&?Gax*#x#BzX}dBhjY;XCoP&4MziTbCdrI+=j%$iT0I{a9CVbFu@@v zj%qN6CnQgWy##3uA!^AFrcBgBOo!`}WSfBSfyWtlkZq^R9_&*R_YVp23l+VLUz6v1 z7SgfE#gSJ6-xCSDp$gjh8S(WvUIk(A^)NJ|S3Pd%Yhd#Yji$@XD3pV%_c}e1r2s4~ zkn=aF*|2OdDGZmAT-{vq?sH%6?QVK|KXoA(CrYByEb7NUt~7J`Of5H-2B|1oa^^R@?LCJ*SS*pPV`Nak!c87 z|9|ArGO_nKWIw%o-dpZTf&>AYG zUYpN*`c0usyCT(gRnMUNZ*7{2U`;t(uuEukFdEVTg>9CB` zYHwOW{vi)WUB0m_SZFOz4A==r;|0G=0qLaTsXLc%Yhj;PINy`{6y1`t(z%|_q zn$b0uDjyd8bP71BH0wXJ=$Verp+~ge<5CP_21b*m*vPH8;Wgbr$#Wu~6<`i3?=np{ z+*yD}&yoMWdUb}9sHyz%U&{O&`0R8K^vGNYk(4`RlfSX@;1S`;g-J=1R5jy&uRz<( z?(gS%^rZEuUhESFH^lbqr`-7R)+HB`wI9F)vA*W)1WRDQ}03p`ryMudDn;q7Iu7U{UJQ1&r{zXMq)$31oS2Hi< z`GmEtLY5@zI`X#=&XZK0L7P1VyG(bn4d%WI9jOx?cAeV#aFJ@Bfv(A-xZ870lWAU__b#PI2X1nP%zn zI=VNn`r0$Nz4dl}v2({-)5D>T#clgS!d<^|`F!_~*U|gCxKD5q0amugi&w8h<|8N4 z)^brOQ$dmvxpVA-Hf6^wsc{Z1Pi0el%?}4z>@s1GW4`9;-R*fhL&eJSFgx(0e*MJP)icr$=UW)K6lCsh zQX=19dNSx7j(cDnNH7=~mgPQ&#(|y!`w!EaHCe1Ngf4oQwX1N&hU>Fp&4@z6R0ui> zp+Da=-C6!NcoCpvMNr%KHI+PG$-? z>r&|~O?f>nQ=a>|uQj@LiaI&kCfx|CU3eW{yWrt>X`Pc+z-(^Q+YiRxj+c9WZCaE(WL+z0!#8p@_BQ9&q5G9% z9(AJY#LeCE@Oi-Yyuerrl4HW^=C_FQH~5U1LUJ~To{TObR57|k)D9%q20?-=F=)i` z^Rv1-1`7E*Rn=!!x3+q;S!cSRfsW>~|7r;dv+Sh^Lh{o(ZD14HH4)tDu}O$AD0AOo zsrt~K?$r2_H;m`P7sAu4H2-_R6wr8&_cucZuQI!e8qZE;zEgSRdED4;d$Z$V=kq6p zQ$y`W6^z&G)pXzgd1UywsiAgTS!Qi7IT$0TYow%DNC`^%HqsYrx*>Rnjshg#yZIUkvsw=g+ux*Un zeZ^S(+WZRd;&#ih-h!{5GhM0wN)dzr3(z)XB}<7X%hyyv-hd^|KUpRgpKo3f@8aa* zTsayO!{YZ{;dV#)nuAu$>#AJZWv!JT?vHsxd++AEQB-CWs$GPRhn|9Q7RYSnnFL%z zN*UBtq?;tI5d5ZO>e@j_K3Tu4MHfjS5~ce_#8<5H;Tt%HnwBN4smNMP{_nF!L9wK_X&amEP6OL3r!uHrn#d+P9E%FUc=H=u* zWAw1qyjqxl+5m7#(Y5njlVAIvzxrWHC1;3Pz&)GN&V-V1=>IZpCI4ys+yk?h2Z_i1|WvIR4ZX)llf!Yo*Fp%FxTXOgv2u)>RQdeCpZ;?C31v|o}DRuPyy)ti=f)M6a9rQ z(bfjvgk?Pj;Jsg<$0J1<12 zhxeWcp1B%IKD2XxZ&mkzFjuR?=C}4DLPhJMw7ah}cy+%&ImSpl0`k&efnX*9sAvQg zVDk@Pr4Z$=n{`399tDa=GZo2SlmcnGKEO_TQZ@K zfO?LZk~ES%|K?3|aNX_I7haY!mLE2~ux7R><1)~^8+qlTL)Tujd@QF6Wt?(}{u}ev zcb=3v*jXsrAu4`B8z)I^F}cXo)YW%zr04jGieoR9>uuqmSNF1E+O1Lh$UCBGcIc;> zxc-vI(~ZFkaUNpoG8VMOZoHf1uOKej6G@vV!Zpp@Fx)2NWub0HIca+-y`S>4T9@ll zWS!qiPy(Sv`8|4wX9W%#%voXBt`>g?^Gzs$B-pv6A~md}Vbca2ot21BHY;pvaR&Xl+`B{5)-$F!Z$if3hR*gn1W1e zU;g6jb}Ms#NJU0u@7c-9Kwm@=oe7*ZNm#G6V|&)(z?o}-I?7|ubdRkP6dd{U(-c4s z(fnKkXijo?pai%Dj;U`+c;7bQ*Tjg4uS*s&JW$CQB>e`BMTUKKCF&A63t_6^CWu5B zPc$;XGS5QzPg2`lRQyerhHvx}IsP=_TF-deNV81S!)<%lyQT&RY@3!2dc8DotheBS z`}fA~%V*s;oBHVNX?KJWE={)PO&3eXq zR*|vRNo}Cz%E5l|L)N-aLhF_ca*prTx!dJL30x~@@AGr_r0Q_!mzOcLTlEso!6TDy zO!NEcw^t?zP9<$`+A7kkaA26fVzkV1Y231z_J#SZLCy#lNI~R@9tGu}bWS(6Fo3%X z0CeR0MahNq(I6O0Nri>j3$u^+p=}~#>ip-%ERZ84f@T(lPA%?*lCd|twx_4jU7<@b z`gn?+en|lHOlf<=J?Xw=f2K2Pe4Xt_RX*?D$Cyz2!zDVAoO}GMUe~?1BI9XJv6#_u zr-|Nk(AxRw^JjVrvT7Rc=;J8I)-Eovm$E)K`^-V%alrZeT1OJ!^nH)No)p%!!J}%! zfN)ZGnv$TsqJzTA94qQtIq38=p;kxvLLmnt#80J%ng4uYpUp4s8e`7Q$xqk_s`#G{ zheX67hRmV|upFF&BnJ+0%^NIkz{H0-m=c(uzZoMSC=_tuB*q6j1OpaH>JUOMuySG3 z%+$;^g~%31ax@r9)K;J*4V=I7)xl$|iNrO6VUUnJFe5lEp=yOv`{I7a9me-C`y~tx z3KIF2v3h297f@Y7Iba!<@XDF_2|;#q8jSWaZ|^rG3M}Ukq+?!~lm_JrGiIETfdFq1 zn4H%N`v{JioppDA^@LB50HArDZXuG5z&^P$*!jV8{pq z=kRtal@DpS1ZN?NA^b2vmP;S}yq+M4kf?f*VHOP-#0rCkNW@f{ai9G!zAuN7GKD-= z!2V%KKeoPy&;v*qLh)6Qy5SbhEQz5#*d|r4U+-}rOGYA^3Zhkrng<0cbWQYv*B69t zpY-yIzh)|(v+;HPlP7WYJY2?QCkA?#%GJwg)XpW%nnaz8Uu@kopmU(^@^jw#nber^ zsiJdsGaaulchlNexFCKlq;8>x1SDR1bXSYWbrBkfNhaygOF6yELX^76VeA7SF$}B88vMPrH(J3d#ar ztL-?|hw)Rlva|K>1z>rhFX;r2L+wL(WVf)Z3Zo#l*QjFtloE{%IkNoz_sR;hwsy5ld04shq zBI1ZF=h34Ygh)x7E}&?HUjSo4mQZe@|CjTY!!c>@bG=`h*8G%R9G+B5@wm?$;LmEh z?d7C?3|I4RZVry(xTLt^ya3U?zwMav6iwa?{xO?$TUhoiV}2N~Q6|1(gHiEI_oPJ~ zpFbo=WbP!oVWJ4cX@r}3J8(dIR8p|ruD5)|0=+}xogw7(BV$k;s&4Q&y8iP4hLa0e zf}}`?=jR-d$Oe^4@iBbi4yY{Q$HG&Ri~T}!auj~B8}hO%cZD&(<^n_(wre;5UuFx82elJSiZ9UU;) zXFg$JVPvG=V%)VVN@bq_B< z;bUxU9UxGV`Y9tgC?bkANDQeCfy)xI^G-QLM&=3ZHpp9@eD&bPROi}@2P5@=l?lj? ze-seZ&_TA6hR@KUQ*G*JZSv_jSbwb_KKpt^!h7iY*CVf-u9mi!Of;$+TUxS0Pfv86 zJ2SX3tBMTmMCmqB)3^-_nH&fg8om=~jm%%$d+p4n6EOJ_OZ}@7tg)M9APxsdD1L}g zG=NZU0jIlA??^;3K;pszJU*K?H*PxmQ5IpH^nlr^Q)z$-h$ECtTQTn@9X1wA^I7bR zYoAjLVscGv81|7Fx&>h-Y1^mAiw=k_qkitPo2JW*xu$=0XX&RCf5LwLTF0{KXQoR6 zo7mZ|z1_7J{hu4`z9cKi#ig3?Ha*;M&g{E|rYvV+_f(tu@tNw;yT9i%bj;`7h@lez z90x8FOn(tlR8&MAu^s6qKvGDezyUlQB$`)0pMe!a5j|5V?s0Nv5-Q_7Ni391Aj8uQ ztxt}_NKKnzhoukBK@>KQ^4Z^Pi!>@=Rri7Tg>wR|`_h_35e%|Jr_HOoF@%(S5*7#4wqQA`Nxgx!HKnLy5nRvk}4RK5gGLrwsWz2IYCz8po&0hwX($e9Ol z3IrUOodl)BB8*Ohef@5gCR?Z`9lC82Xo(`L_~ZaT8flU>N=Jd|E)>;n^CWOeLfBvv zk}P$YnpBC#7>Wy$?nzc0UNeBqTUCu&MpB$`semgXgS*hsv#_$7SZ$cpU&hKvW?O)E z6WOp~x7+L0_-mXrIzY9^_k>Liq)GrJH;M|P8O3}xyPQ;UG>XvCYCtIF;!hIMXbm^D zHWtEV^dZy%s*K&!dt29xKHBi0;IhbkX-Y)jvO$I2+9x6x?hAMDj+`F5LdicjoHCc9 zzU9N_bd$vkHtp3Lj;;I?Qn#*LqLO}puz$G2l{W57v|c+0u3Y1$o9)Z#bZq`*qkhwQ zM!GBnX9r}SwCNg{dGmex)L?5Q5V-^2p2vSrY}>tOPZ)ftgrtP)l$3)jmwgoc*PgQylH@F=zfksD*E;#~{UH{$XJ1xBrk48qVrGKZc>mNE0q zZg7ssY4M`WC#Mip7YFu`%q0TTty|&^;h`Ui;~hxRO$rQ8*n__`>$iiTh#*!4?fHF^ ztB9bxueopC*xF^R#J~t&7y|0Zv;=A{axqBnh@NWFKu6|tar+jo3fG*CMwc#&PUhO# z3ykw5i)rQLYF`q+{Yh_I>^%SB<%!&@$wEx0g7^Q9h!WzXKpb)nc??9AyT8Q#Bx$82 z9{(z4dRV4HtxDod(I1fp)S)&SxueqKL+!cJFDJ8f@7^a|_V>Nv#!|~rWD{8kiD@8} zP(}9?2MP^q{okQnv-+_3?UcPJjY&^9` z&FwWNbRy&@4U!a>IU+kH<5B;<6W&+XfF@SJ$qsT3g{_gV4Z?@W(;$n|gi$@tsXHZl zPfVE{j0y5laV6(s6|T@Nl)ts+PQE$!&D>Bu(ri%@dz*urmf3}+220Bscd6WqD%9rZ z7Z&0u?^PW3ncK|5dvsS0_Pla8*)Y(&TY!f^Cx*{8# z_INFB-7wU{FNq~Z*$#Uqyb~1Z8wn$rh7I**9OR$rD5f}dxKKXadocMu700~U{oNa{ zI3Tr-><{Ub1gIy8-_`und{RCZI5|r0+m=Pa)|2Qvker;5S6H~wV)l3AeAEv|>TgBq zY>~erYd-L;;|