Description
Observed on vLLM 0.25 + ModelExpress 0.4.1.
For models whose MTP (speculative draft) weights live in the same s3:// checkpoint
as the main weights, loading the MTP weights fails. ModelStreamerStrategy is gated
on MX_MODEL_URI (is_available) but resolves the actual streaming source as
model_config.model_weights or model_config.model — and for the MTP draft neither
points at the checkpoint.
The target model loads fine (vLLM's runai pull sets model_weights=s3://…). But for
method="mtp", vLLM sets SpeculativeConfig.model = target_model_config.model (the
local snapshot dir) and builds draft_model_config = ModelConfig(model=<snapshot>, model_weights=None). The MTP weights are in the checkpoint MX_MODEL_URI
points at, but the strategy never uses it.
Reproduction
Run vLLM with s3://path/to/GLM-5.2 as the model and MX_MODEL_URI set, plus
--speculative-config={"method":"mtp","num_speculative_tokens":5,"moe_backend":"triton"}.
The runai streamer pulls the main weights successfully, then fails on the MTP draft:
Cannot find any safetensors model weights with /root/.cache/vllm/assets/model_streamer/
Proposed fix
When the primary source (model_weights or model) yields no streamable weights,
fall back to streaming from MX_MODEL_URI (the gate env the strategy already reads),
into the existing draft module (without re-initializing it).
Description
Observed on vLLM 0.25 + ModelExpress 0.4.1.
For models whose MTP (speculative draft) weights live in the same s3:// checkpoint
as the main weights, loading the MTP weights fails.
ModelStreamerStrategyis gatedon
MX_MODEL_URI(is_available) but resolves the actual streaming source asmodel_config.model_weights or model_config.model— and for the MTP draft neitherpoints at the checkpoint.
The target model loads fine (vLLM's runai pull sets
model_weights=s3://…). But formethod="mtp", vLLM setsSpeculativeConfig.model = target_model_config.model(thelocal snapshot dir) and builds
draft_model_config = ModelConfig(model=<snapshot>, model_weights=None). The MTP weights are in the checkpointMX_MODEL_URIpoints at, but the strategy never uses it.
Reproduction
Run vLLM with
s3://path/to/GLM-5.2as the model andMX_MODEL_URIset, plus--speculative-config={"method":"mtp","num_speculative_tokens":5,"moe_backend":"triton"}.The runai streamer pulls the main weights successfully, then fails on the MTP draft:
Cannot find any safetensors model weights with /root/.cache/vllm/assets/model_streamer/
Proposed fix
When the primary source (
model_weights or model) yields no streamable weights,fall back to streaming from
MX_MODEL_URI(the gate env the strategy already reads),into the existing draft module (without re-initializing it).