Upstream a vLLM pre-resolution hook and remove the Hugging Face snapshot patch

[zhengluo-nv]

Summary

PR #592 integrates the ModelExpress server cache into the vLLM loading strategy while preserving P2P-first weight loading. vLLM resolves the Hugging Face snapshot before the custom model loader runs, so the current implementation patches huggingface_hub.snapshot_download() at runtime to install non-weight metadata first.

Track the upstream work needed to replace that global monkey patch with a supported vLLM extension point.

Related: #569, #592.

Current workaround

When MODEL_EXPRESS_NO_SHARED_STORAGE=1 and a ModelExpress server address is configured, ModelExpress patches the snapshot_download bindings used by vLLM. The wrapper:

1. asks ModelExpress Server to install the repository metadata into the local Hugging Face cache;
2. does not fetch weight files, preserving RDMA/P2P as the first weight source;
3. delegates to the original Hugging Face function so normal resolution behavior and errors remain intact.

Multiple huggingface_hub namespaces must be patched because the binding used by HfApi.snapshot_download() differs across versions. This is process-global and therefore carries compatibility and maintenance risk.

Desired upstream capability

vLLM should expose a supported hook before remote/local model snapshot resolution, allowing a model-loader plugin to materialize the configuration, tokenizer, and other non-weight repository files before vLLM resolves ModelConfig. The hook must run early enough for offline workers, while leaving weight acquisition to the selected model loader.

The interface should preserve the original model identifier and requested revision and work in the relevant frontend and EngineCore process boundaries.

Work items

• Confirm the appropriate vLLM ownership point and extension API with upstream maintainers.
• Propose and implement the smallest general-purpose pre-resolution hook upstream.
• Add upstream coverage for an offline worker whose plugin materializes metadata locally.
• Adopt the released vLLM API in ModelExpress.
• Remove patch_hf_snapshot_prefetch and its process-global Hugging Face mutations once the minimum supported vLLM version contains the hook.
• Document the compatibility boundary for older vLLM releases.

Acceptance criteria

• ModelExpress can bootstrap an offline vLLM worker from a Hugging Face repository ID without modifying huggingface_hub globals.
• Configuration/tokenizer metadata is available before vLLM model resolution.
• Weight loading remains P2P first, followed by server cache on a miss.
• The requested revision is preserved across the pre-resolution hook.
• Existing behavior remains a no-op when server-cache mode is disabled.
• The runtime patch is removed for supported vLLM versions.

[scydas]

Survey of what vLLM exposes today, against c296851a7d (just after v0.27.2rc0). Short version: the capability this issue asks for already exists upstream, hardcoded to one source — so the proposal can be "generalize an existing step" rather than "add a new hook".

vLLM already pre-materializes metadata, for one source

ModelConfig.maybe_pull_model_tokenizer_for_runai() performs the same split this issue describes:

object_storage_model.pull_files(model, allow_pattern=["*.model", "*.py", "*.json"])  # metadata only
self.model_weights = model             # remembered so the loader fetches weights later
self.model = object_storage_model.dir  # vLLM resolves locally from here on

runai_streamer_loader and sharded_state_loader read model_weights back at weight-load time. So "materialize metadata early, leave weights to the selected loader" is already a supported pattern with a dedicated ModelConfig field. It is just dispatched on the s3:// / gs:// / az:// schemes, which a plain Hugging Face repo ID cannot match. Two independent integrations needing the same step is the argument that it is a category worth generalizing, with Run:AI as the first implementation and ModelExpress as the second.

Timing is not the obstacle — the missing piece is an attachment point

EngineArgs.__post_init__()
    load_general_plugins()                   # plugins are live from here on
    get_model_path() -> snapshot_download()  # HF_HUB_OFFLINE branch; our patch intercepts here
EngineArgs.create_engine_config()
    create_model_config() -> ModelConfig.__post_init__()
        maybe_model_redirect()
        maybe_pull_model_tokenizer_for_runai()   # <- the precedent: official, source-specific
        resolve_revision()

load_general_plugins() also runs in v1/engine/core.py and v1/worker/worker_base.py, so the frontend and EngineCore boundaries this issue calls out are already covered. A plugin is live before anything resolves the model; it simply has no supported surface to act on, which is why the patch replaces huggingface_hub globals instead.

One detail for whoever implements it

resolve_revision() runs immediately after the pull step, with local_files_only when HF_HUB_OFFLINE is set, and offline resolution branches on the shape of the revision: huggingface_hub tests it against REGEX_COMMIT_HASH (^[0-9a-f]{40}$, lowercase only) and goes straight to snapshots/<sha>/, falling back to refs/<revision> otherwise. A hook that installs a pinned snapshot under its lowercase commit SHA therefore needs no refs/ entry; a branch or tag pin does.