Task.17】DeepEyes processor 动态 patch

docs/.vitepress/config.mts

@@ -269,7 +269,8 @@ export default defineConfig({
                 { text: 'Metrics Service', link: '/en/guide/metrics-service-detailed' },
                 { text: 'Notification System', link: '/en/guide/notification-system' },
                 { text: 'Update Weights Pipeline', link: '/en/guide/update-weights-pipeline' },
-                { text: 'Low-Rank Adaptation (LoRA) Training', link: '/en/guide/low-rank-adaptation-training' }
+                { text: 'Low-Rank Adaptation (LoRA) Training', link: '/en/guide/low-rank-adaptation-training' },
+                { text: 'SGLang Runtime Patches', link: '/en/guide/sglang-patches' }
               ]
             },
             {
@@ -378,7 +379,8 @@ export default defineConfig({
                 { text: 'Metrics 服务', link: '/zh/guide/metrics-service-detailed' },
                 { text: '通知系统', link: '/zh/guide/notification-system' },
                 { text: '权重更新流水线优化', link: '/zh/guide/update-weights-pipeline' },
-                { text: '低秩适配（LoRA）训练', link: '/zh/guide/low-rank-adaptation-training' }
+                { text: '低秩适配（LoRA）训练', link: '/zh/guide/low-rank-adaptation-training' },
+                { text: 'SGLang 运行时补丁', link: '/zh/guide/sglang-patches' }
               ]
             },
             {

docs/en/guide/sglang-patches.md

@@ -0,0 +1,72 @@
+# SGLang Runtime Patches
+
+Relax injects runtime enhancements into SGLang at engine start-up via monkey-patching, rather than editing SGLang source or `cp`-overwriting system files. Each patch is gated by its own env flag, applied in `relax/backends/sglang/sglang_engine.py::_launch_server_with_patches`, and any combination may be enabled independently.
+
+## Overview
+
+| Patch | env flag | Injection point | Purpose |
+|-------|----------|-----------------|---------|
+| Routing Replay | `RELAX_OPTIMIZE_ROUTING_REPLAY=1` | scheduler subprocess (`_patched_run_scheduler_process`) | async D→H copy of routed-experts buffers, removing default-stream sync |
+| OPD Pre-expanded | `RELAX_OPD_PREEXPANDED_PATCH=1` | main process | accept `opd_preexpanded_raw` pre-expanded `input_ids`+raw images, skip decode→retokenize |
+| DeepEyes Qwen-VL | `RELAX_DEEPEYES_QWEN_VL_PATCH=1` | main process | support DeepEyes multi-turn pre-tokenized `input_ids`: collapse consecutive `<|image_pad|>` and restore the original token sequence + mrope |
+
+## How the DeepEyes Qwen-VL patch reaches the SGLang subprocess
+
+The primary switch for this patch is the CLI argument `--deepeyes-qwen-vl-patch` (default `False`, i.e. stock SGLang Qwen-VL behavior). The propagation chain:
+
+1. The training script passes `--deepeyes-qwen-vl-patch` (e.g. `examples/deepeyes/run_deepeyes_r3.sh`, gated by the `DEEPEYES_PATCH` env var, default `0` = off / stock SGLang behavior; DeepEyes multi-turn training requires `DEEPEYES_PATCH=1` to enable it).
+2. `relax/backends/sglang/sglang_engine.py::_init_normal` sees `args.deepeyes_qwen_vl_patch` truthy and sets `os.environ["RELAX_DEEPEYES_QWEN_VL_PATCH"] = "1"` (same pattern as `--optimize-routing-replay`).
+3. `_init_normal` spawns the SGLang subprocess via `multiprocessing.Process(target=_launch_server_with_patches, start_method='spawn')`, which inherits the parent's `os.environ`.
+4. `_launch_server_with_patches` checks the env flag and, if set, calls `apply_deepeyes_qwen_vl_patch`.
+
+> You may also skip the CLI arg and `export RELAX_DEEPEYES_QWEN_VL_PATCH=1` directly, forwarding it via `RELAX_PROPAGATE_ENV_VARS` (the equivalent programmatic path, matching the OPD patch). Both hit the same env flag; either works.
+
+## DeepEyes Qwen-VL Patch
+
+**Module**: `relax/backends/sglang/patches/qwen_vl_patch.py`
+
+**Why**: DeepEyes multi-turn VLM rollout (`examples/deepeyes/rollout.py::_run_inference_step`) sends **pre-tokenized** `input_ids: list[int]` to SGLang instead of text. The stock `QwenVLImageProcessor.process_mm_data_async` decodes→retokenizes, causing:
+
+- **A. Token drift**: decode→retokenize is not lossless; token count may change, misaligning mrope positions.
+- **B. N×M explosion**: the N accumulated `<|image_pad|>` tokens are re-expanded by `load_mm_data` into N×M.
+
+**The two real changes injected** (all other upstream logic untouched):
+
+1. `_strip_image_token`: collapse consecutive `<|image_pad|>` into a single placeholder before `load_mm_data`.
+2. `original_input_ids` save/restore: save the caller's original `input_ids`, let the mm pipeline build `pixel_values`/grids as usual, then restore the token sequence and **recompute mrope from the restored `input_ids`**.
+
+> **Why re-implement the method body instead of a plain wrapper**: mrope must be recomputed from the restored original `input_ids`, while the original method computes mrope internally from the retokenized sequence. A plain wrapper that only rewrites `input_ids` in the return value would misalign `input_ids` and mrope — exactly the bug this patch fixes. Recomputing mrope requires the method-internal artifact `ret` (carrying `image_grid_thw`/`video_grid_thw`/`second_per_grid_ts`), unavailable outside. The patch therefore rewrites `process_mm_data_async`, **delegating to upstream primitives** `self.load_mm_data` / `self.process_and_combine_mm_data` / `MRotaryEmbedding.get_rope_index` / the upstream module-level `preprocess_video`, inlining the two changes. Unrelated helpers (`smart_resize`, `preprocess_video`, `get_mm_data`, `__init__`, …) are **not** copied — they stay upstream.
+
+**Idempotent**: `_PATCH_FLAG = "_relax_deepeyes_patched"` marks a patched class; a second call is a no-op.
+
+**No effect on normal requests**: when `input_text` is a `str`, `_strip_image_token` passes through, `original_input_ids` stays `None`, and the method follows the original path.
+
+## SGLang upgrade checklist
+
+After upgrading SGLang, verify each patch's upstream dependencies still exist:
+
+### DeepEyes Qwen-VL Patch
+- `sglang.srt.multimodal.processors.qwen_vl.QwenVLImageProcessor` exists and has:
+  - `process_mm_data_async` (the method being replaced)
+  - `load_mm_data` (instance method, signature `prompt, image_data, video_data, audio_data, multimodal_tokens`)
+  - `process_and_combine_mm_data` (instance method, returns `(mm_items, input_ids, ret)`)
+- `sglang.srt.multimodal.processors.qwen_vl.preprocess_video` (module-level async function)
+- `sglang.srt.layers.rotary_embedding.MRotaryEmbedding.get_rope_index` (keyword args: `spatial_merge_size, image_token_id, video_token_id, vision_start_token_id, model_type, tokens_per_second, input_ids, image_grid_thw, video_grid_thw, second_per_grid_ts, use_audio_in_video, audio_seqlens, audio_token_id, audio_start_token_id, position_id_per_seconds`)
+- `sglang.srt.managers.schedule_batch.MultimodalProcessorOutput.from_dict` (sglang ≥0.5.12; if absent, the patch falls back to returning a dict)
+
+**Fail-fast**: `apply_qwen_vl_patches` calls `_verify_upstream_api` before binding; any missing symbol raises `RuntimeError` stating "Relax requires sglang 0.5.12.post1". `apply_deepeyes_qwen_vl_patch` additionally wraps this in try/except so a patch failure never blocks engine start-up (warning only).
+
+### OPD Pre-expanded Patch
+- Also replaces `QwenVLImageProcessor.process_mm_data_async`; depends on `self._processor.image_processor`, `self.get_mm_items_offset`, `MRotaryEmbedding.get_rope_index`, `MultimodalProcessorOutput`. See `relax/utils/opd/opd_sglang_patch.py`.
+
+### Routing Replay Patch
+- Replaces methods on `_RoutedExpertsCapturerReal`. See `relax/backends/sglang/routing_replay_patch.py`.
+
+## Verification
+
+- Unit tests: `pytest tests/backends/sglang/test_qwen_vl_patch.py -v` (stubs sglang, no GPU needed).
+- End-to-end: run `examples/deepeyes/run_deepeyes_r3.sh` in an environment with the target SGLang version + GPU; output must be token-for-token identical to the old `cp` overlay at `temperature=0`.
+
+## Deprecated
+
+`examples/deepeyes/qwen_vl.py` (the old whole-file `cp` overlay onto `/sgl-workspace/...`) has been deleted and replaced by this patch. Clean up any remaining `/sgl-workspace` references in your environment.

docs/zh/guide/sglang-patches.md

@@ -0,0 +1,72 @@
+# SGLang 运行时 Patch 说明
+
+Relax 在 SGLang 引擎启动时通过 monkey-patch 注入若干运行时增强，而非修改 SGLang 源码或 `cp` 覆盖系统文件。每个 patch 由独立 env flag 控制，在 `relax/backends/sglang/sglang_engine.py::_launch_server_with_patches` 中按 flag 应用，互斥组合任意启用。
+
+## 总览
+
+| Patch | env flag | 注入位置 | 作用 |
+|-------|----------|----------|------|
+| Routing Replay | `RELAX_OPTIMIZE_ROUTING_REPLAY=1` | scheduler 子进程 (`_patched_run_scheduler_process`) | routed-experts buffer 的异步 D→H 拷贝，消除默认流同步 |
+| OPD Pre-expanded | `RELAX_OPD_PREEXPANDED_PATCH=1` | 主进程 | 接收 `opd_preexpanded_raw` 格式的预展开 `input_ids`+原始图片，跳过 decode→retokenize |
+| DeepEyes Qwen-VL | `RELAX_DEEPEYES_QWEN_VL_PATCH=1` | 主进程 | 支持 DeepEyes 多轮预分词 `input_ids`，折叠连续 `<|image_pad|>` 并恢复原始 token 序列与 mrope |
+
+## env flag 如何到达 SGLang 子进程
+
+DeepEyes Qwen-VL patch 的主开关是 CLI 参数 `--deepeyes-qwen-vl-patch`（默认 `False`，即使用上游 SGLang 原生行为）。其传递链路：
+
+1. 训练脚本传 `--deepeyes-qwen-vl-patch`（如 `examples/deepeyes/run_deepeyes_r3.sh`，受 `DEEPEYES_PATCH` 环境变量控制，默认 `0` 即关闭，走 stock SGLang 行为；DeepEyes 多轮训练需手动 `DEEPEYES_PATCH=1` 开启）。
+2. `relax/backends/sglang/sglang_engine.py::_init_normal` 检测到 `args.deepeyes_qwen_vl_patch` 为真，设置 `os.environ["RELAX_DEEPEYES_QWEN_VL_PATCH"] = "1"`（与 `--optimize-routing-replay` 同模式）。
+3. `_init_normal` 通过 `multiprocessing.Process(target=_launch_server_with_patches, start_method='spawn')` 拉起 SGLang 子进程，子进程继承父进程 `os.environ`。
+4. `_launch_server_with_patches` 检查 env flag，命中则调用 `apply_deepeyes_qwen_vl_patch`。
+
+> 也可不传 CLI 参数、直接 `export RELAX_DEEPEYES_QWEN_VL_PATCH=1` 并通过 `RELAX_PROPAGATE_ENV_VARS` 透传（程序化场景的等价路径，与 OPD patch 一致）。两者命中同一个 env flag，任选其一。
+
+## DeepEyes Qwen-VL Patch
+
+**模块**：`relax/backends/sglang/patches/qwen_vl_patch.py`
+
+**为什么需要**：DeepEyes 多轮 VLM rollout（`examples/deepeyes/rollout.py::_run_inference_step`）向 SGLang 发送预分词的 `input_ids: list[int]`，而非文本。SGLang 原生 `QwenVLImageProcessor.process_mm_data_async` 会 decode→retokenize，导致两个问题：
+
+- **A. token 漂移**：decode→retokenize 非无损，token 数可能变化，mrope position 错位。
+- **B. N×M 展开**：多轮累积的 N 个 `<|image_pad|>` 被 `load_mm_data` 再次展开成 N×M。
+
+**注入的两处真实差异**（其余上游逻辑不动）：
+
+1. `_strip_image_token`：`load_mm_data` 前把连续 `<|image_pad|>` 折叠为单个占位符。
+2. `original_input_ids` 保存/恢复：保存调用方原始 `input_ids`，让 mm 管线照常产出 `pixel_values`/grid，再恢复 token 序列并**用恢复后的 `input_ids` 重算 mrope**。
+
+> **为什么是重写方法体而非纯包装器**：mrope 必须用恢复后的原始 `input_ids` 重算，而原始方法内部已用 retokenize 序列算好 mrope。纯包装器只改返回值的 `input_ids` 会让 input_ids 与 mrope 错位——正是要修的 bug。重算 mrope 需要方法内部中间产物 `ret`（含 `image_grid_thw`/`video_grid_thw`/`second_per_grid_ts`），外层拿不到。因此 patch 重写 `process_mm_data_async`，**委托上游原语** `self.load_mm_data` / `self.process_and_combine_mm_data` / `MRotaryEmbedding.get_rope_index` / 上游模块级 `preprocess_video`，把两处改动内联。`smart_resize`、`preprocess_video`、`get_mm_data`、`__init__` 等无关函数**不复制**，留在上游。
+
+**幂等**：`_PATCH_FLAG = "_relax_deepeyes_patched"` 标记已 patch 的类，重复调用为 no-op。
+
+**对普通请求无影响**：`input_text` 为 `str` 时，`_strip_image_token` 直通、`original_input_ids` 保持 `None`，方法走原路径。
+
+## SGLang 升级检查清单
+
+升级 SGLang 版本后，逐项校验各 patch 依赖的上游 API 仍存在：
+
+### DeepEyes Qwen-VL Patch
+- `sglang.srt.multimodal.processors.qwen_vl.QwenVLImageProcessor` 存在，且含：
+  - `process_mm_data_async`（被替换的目标方法）
+  - `load_mm_data`（实例方法，签名 `prompt, image_data, video_data, audio_data, multimodal_tokens`）
+  - `process_and_combine_mm_data`（实例方法，返回 `(mm_items, input_ids, ret)`）
+- `sglang.srt.multimodal.processors.qwen_vl.preprocess_video`（模块级 async 函数）
+- `sglang.srt.layers.rotary_embedding.MRotaryEmbedding.get_rope_index`（关键字参数：`spatial_merge_size, image_token_id, video_token_id, vision_start_token_id, model_type, tokens_per_second, input_ids, image_grid_thw, video_grid_thw, second_per_grid_ts, use_audio_in_video, audio_seqlens, audio_token_id, audio_start_token_id, position_id_per_seconds`）
+- `sglang.srt.managers.schedule_batch.MultimodalProcessorOutput.from_dict`（sglang ≥0.5.12；缺失时 patch 自动回退为返回 dict）
+
+**fail-fast**：`apply_qwen_vl_patches` 在绑定前调用 `_verify_upstream_api`，缺失任一符号即 `raise RuntimeError`，提示 "Relax requires sglang 0.5.12.post1"。`apply_deepeyes_qwen_vl_patch` 额外包一层 try/except，使 patch 失败不阻断引擎启动（仅 warning）。
+
+### OPD Pre-expanded Patch
+- 同样替换 `QwenVLImageProcessor.process_mm_data_async`，依赖 `self._processor.image_processor`、`self.get_mm_items_offset`、`MRotaryEmbedding.get_rope_index`、`MultimodalProcessorOutput`。详见 `relax/utils/opd/opd_sglang_patch.py`。
+
+### Routing Replay Patch
+- 替换 `sglang.srt.managers.io_struct`（或对应模块）的 `_RoutedExpertsCapturerReal` 方法。详见 `relax/backends/sglang/routing_replay_patch.py`。
+
+## 验证
+
+- 单测：`pytest tests/backends/sglang/test_qwen_vl_patch.py -v`（stub sglang，无需 GPU）。
+- 端到端：在装有目标 SGLang 版本 + GPU 的环境运行 `examples/deepeyes/run_deepeyes_r3.sh`，`temperature=0` 下与旧 `cp` 覆盖方案逐 token 一致。
+
+## 已弃用
+
+`examples/deepeyes/qwen_vl.py`（整文件 `cp` 覆盖 `/sgl-workspace/...` 的旧方案）已删除，由本 patch 取代。如发现环境中仍存在 `/sgl-workspace` 引用，请清理。

examples/deepeyes/openai_judge_service.sh

@@ -0,0 +1,86 @@
+#!/bin/bash
+# Remote OpenAI-compatible LLM Judge 服务：使用外部托管的 OpenAI 兼容 endpoint 作为
+# DeepEyes 的 LLM judge，替代 sglang_judge_service.sh 本地起 Qwen2.5-1.5B 的做法。
+#
+# 用法: source "$(dirname "${BASH_SOURCE[0]}")/openai_judge_service.sh"
+#
+# 与 sglang_judge_service.sh 的区别：本脚本不在本地启动任何服务，仅把 judge
+# endpoint 通过环境变量（及 Ray runtime_env）透传给 reward worker。reward 代码
+# （reward_deepeyes.py::_get_judge_client）原样消费 DEEPEYES_JUDGE_* 变量，无需改动。
+#
+# 可在调用前覆盖以下变量切换 endpoint / 模型 / api-key：
+#   DEEPEYES_JUDGE_BASE_URL  (默认 http://29.160.40.138:8021/v1)
+#   DEEPEYES_JUDGE_API_KEY   (默认 EMPTY)
+#   DEEPEYES_JUDGE_MODELS    (默认 DeepSeek-V4-Flash-safety-t2)
+# 依赖: TIMESTAMP 需在 source 前已定义（用于日志命名，可选）
+
+# 设置默认值（允许外部覆盖）
+DEEPEYES_JUDGE_BASE_URL="${DEEPEYES_JUDGE_BASE_URL:-http://29.160.40.138:8021/v1}"
+DEEPEYES_JUDGE_API_KEY="${DEEPEYES_JUDGE_API_KEY:-EMPTY}"
+DEEPEYES_JUDGE_MODELS="${DEEPEYES_JUDGE_MODELS:-DeepSeek-V4-Flash-safety-t2}"
+
+# 检查必要依赖是否存在
+if ! command -v curl &> /dev/null; then
+    echo "Error: curl is required but not installed."
+    exit 1
+fi
+
+# 健康检查：远程 endpoint 不可达时 fail fast，避免训练跑到 reward 阶段才崩
+echo "Checking remote LLM judge service at ${DEEPEYES_JUDGE_BASE_URL} ..."
+http_status=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 \
+    -H "Authorization: Bearer ${DEEPEYES_JUDGE_API_KEY}" \
+    "${DEEPEYES_JUDGE_BASE_URL}/models" 2>/dev/null || echo "000")
+if [ "$http_status" != "200" ] && [ "$http_status" != "204" ]; then
+    echo "Error: Remote LLM judge service not reachable at ${DEEPEYES_JUDGE_BASE_URL} (HTTP ${http_status})."
+    echo "Set DEEPEYES_JUDGE_BASE_URL / DEEPEYES_JUDGE_API_KEY to a reachable OpenAI-compatible endpoint."
+    exit 1
+fi
+echo "Remote LLM judge service is ready (models endpoint HTTP ${http_status})."
+echo "  base_url: ${DEEPEYES_JUDGE_BASE_URL}"
+echo "  models:   ${DEEPEYES_JUDGE_MODELS}"
+
+# 导出到当前 shell（供非 Ray 路径 / 直接调用 reward 使用）
+export DEEPEYES_JUDGE_API_KEY
+export DEEPEYES_JUDGE_BASE_URL
+export DEEPEYES_JUDGE_MODELS
+
+# 注入 Ray runtime_env，使 reward worker（Ray actor）继承这些环境变量。
+# 与 sglang_judge_service.sh 保持完全一致的注入方式。
+if [ -n "${RUNTIME_ENV_JSON:-}" ]; then
+    json_escape() {
+        local value="${1:-}"
+        value=${value//\\/\\\\}
+        value=${value//\"/\\\"}
+        value=${value//$'\n'/\\n}
+        value=${value//$'\r'/\\r}
+        value=${value//$'\t'/\\t}
+        printf '%s' "$value"
+    }
+
+    runtime_env_prefix="${RUNTIME_ENV_JSON%$'\n}\n}'}"
+    export RUNTIME_ENV_JSON="${runtime_env_prefix},
+   \"DEEPEYES_JUDGE_API_KEY\": \"$(json_escape "${DEEPEYES_JUDGE_API_KEY}")\",
+   \"DEEPEYES_JUDGE_BASE_URL\": \"$(json_escape "${DEEPEYES_JUDGE_BASE_URL}")\",
+   \"DEEPEYES_JUDGE_MODELS\": \"$(json_escape "${DEEPEYES_JUDGE_MODELS}")\"
+}
+}"
+fi
+
+# 包装 ray：若调用方走 ray job submit 且未显式传 --runtime-env-json，则补上。
+# 与 sglang_judge_service.sh 一致。
+ray() {
+    if [ "$1" = "job" ] && [ "$2" = "submit" ] && [ -n "${RUNTIME_ENV_JSON:-}" ]; then
+        local arg=""
+        for arg in "$@"; do
+            if [ "$arg" = "--runtime-env-json" ] || [[ "$arg" == --runtime-env-json=* ]]; then
+                command ray "$@"
+                return
+            fi
+        done
+        command ray job submit ${RAY_NO_WAIT:+--no-wait} --runtime-env-json="${RUNTIME_ENV_JSON}" "${@:3}"
+        return
+    fi
+    command ray "$@"
+}
+
+echo "Remote LLM judge service is fully ready for use."