Add OpenVINO (NPU/GPU) and QNN (NPU) recipes for Gemma 4 E2B#527
Draft
justinchuby wants to merge 7 commits into
Draft
Add OpenVINO (NPU/GPU) and QNN (NPU) recipes for Gemma 4 E2B#527justinchuby wants to merge 7 commits into
justinchuby wants to merge 7 commits into
Conversation
Add openvino/{npu,gpu}/config.json recipes that build the portable
mobius ONNX for google/gemma-4-E2B-it and INT4-quantize the decoder with
K-Quant, targeting the OpenVINO EP. The generated genai_config selects the
OpenVINO EP with the matching device_type (NPU / GPU); unsupported ops fall
back to the CPU EP automatically.
Wire the two recipes into info.yml (adding the OpenVINOExecutionProvider EP
and npu device), document them in the README, and ignore the generated
openvino/*/models artifacts.
Requires OpenVINO EP support in MobiusBuilder (microsoft/Olive#2553) and the
mobius openvino EP (onnxruntime/mobius#388).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds two new OpenVINO-targeted Olive recipes for google/gemma-4-E2B-it (NPU + GPU) alongside documentation and metadata updates so the recipes are discoverable and their outputs are ignored by git.
Changes:
- Add
openvino/npu/config.jsonandopenvino/gpu/config.jsonrecipes usingMobiusBuilder(fp16)+OnnxKQuantQuantization(INT4). - Register OpenVINO EP + NPU device in
info.ymland add the new recipes. - Document OpenVINO recipes/build commands and ignore generated
openvino/*/modelsoutputs.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| google-gemma-4-E2B-it/README.md | Documents OpenVINO install note, recipes, and build commands |
| google-gemma-4-E2B-it/openvino/npu/config.json | New Mobius→KQuant INT4 recipe targeting OpenVINO EP on NPU |
| google-gemma-4-E2B-it/openvino/gpu/config.json | New Mobius→KQuant INT4 recipe targeting OpenVINO EP on GPU |
| google-gemma-4-E2B-it/info.yml | Adds OpenVINO EP + NPU device + registers the new recipes |
| google-gemma-4-E2B-it/.gitignore | Ignores generated OpenVINO recipe model outputs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| |--------|-----------------| | ||
| | CPU | `pip install onnxruntime-genai` | | ||
| | GPU (CUDA) | `pip install onnxruntime-genai-cuda` | | ||
| | OpenVINO (NPU / GPU) | `pip install onnxruntime-genai` + [OpenVINO EP](https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html) | |
Comment on lines
+48
to
+49
| The OpenVINO EP compiles the graph for the target device at load time; ops it | ||
| does not support fall back to the CPU EP automatically. This requires the |
Add a two-step QNN/HTP flow: qnn/build builds the portable onnx-standard multi-component model via mobius, then per-component configs compile each part to an HTP context binary. The decoder uses the full HTP LLM flow (GraphSurgeries -> OnnxStaticQuantization -> SplitModel -> StaticLLM -> EPContextBinaryGenerator -> ComposeOnnxModels); the vision/audio/embedding encoders compile via HTP fp16. The mobius build and decoder GraphSurgeries were validated on CPU. The ModelBuilder-specific RemoveRopeMultiCache surgery is omitted because it does not match mobius's single-RotaryEmbedding decoder. All HTP-specific steps (static quant for HTP, context-binary generation) require the QNN SDK and a Snapdragon target and are NOT verified on hardware — the README marks this recipe as experimental with the remaining bring-up TODOs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
The repo-root .gitignore ignores `build/` (Python build artifacts), which silently matched the QNN step-1 recipe dir `qnn/build/`, leaving qnn/build/config.json untracked. Since info.yml, the README, and all four downstream QNN component configs depend on it (their OnnxModel input paths point at qnn/build/models/<component>/model.onnx), the QNN recipe set was unrunnable from a fresh clone. Re-include it via the recipe-local .gitignore and add the file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Instead of leaving encoder weights in FP16 and relying only on the decoder's
static quantization, quantize all components to INT4 with K-Quant in the build
step and lower the resulting MatMulNBits ops to standard-ONNX INT4 QDQ via
MatMulNBitsToQDQ. This yields HTP-friendly QDQ weights (DequantizeLinear +
MatMul, zero com.microsoft ops) for both the decoder and the encoders.
- qnn/build: add OnnxKQuantQuantization(int4) after MobiusBuilder.
- qnn/decoder: prepend MatMulNBitsToQDQ; exclude the QDQ nodes from the
activation static-quant pass.
- qnn/{vision,audio,embedding}: prepend MatMulNBitsToQDQ before the
context-binary step (FP16 activations on HTP).
Validated on CPU: K-Quant → MatMulNBitsToQDQ converts all 276 decoder
MatMulNBits to INT4 QDQ with zero com.microsoft ops remaining. HTP-specific
steps remain hardware-only (documented).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
justinchuby
changed the title
The OpenVINO ONNX graph is device-independent and mobius now emits a default OpenVINO device_type of NPU, so a separate openvino/gpu recipe (which would still emit device_type=NPU) is misleading. Keep a single openvino/npu recipe and document that a different OpenVINO device can be selected by editing device_type in the generated genai_config.json (no rebuild needed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
The OpenVINO EP (onnxruntime-openvino 1.24) fails to load the mobius export because it does not support the opset-24 ai.onnx::Attention op. Document this so users know the recipe currently requires a newer OpenVINO (or an Attention-decomposition pass), and point to the prebuilt HF package for version/hardware testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
… blockers Add MatMulNBitsToQDQ after K-Quant so the INT4 weights become standard-ONNX QDQ (DequantizeLinear + MatMul) instead of the com.microsoft MatMulNBits op, which the OpenVINO frontend cannot consume. Combined with the mobius openvino EP change (no SkipSimplifiedLayerNormalization fusion), this removes two classes of unconvertible ops. Document the two remaining OpenVINO-side blockers (opset-24 RMSNormalization unsupported; Attention-24 KV-cache shape inference), which are pending OpenVINO frontend fixes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the existing mobius-based recipe for
google/gemma-4-E2B-itto two additional execution providers: OpenVINO and QNN / Snapdragon HTP. Both are experimental / not runnable on today's runtimes — see the per-EP status below.OpenVINO —
openvino/npu/config.jsonMobiusBuilder(fp16)→OnnxKQuantQuantization(INT4 K-Quant), then agenai_config.jsonwhoseprovider_optionsselect the OpenVINO EP. The OpenVINO graph is device-independent, so mobius emits a defaultdevice_typeofNPU; a different OpenVINO device (GPU/CPU) can be selected downstream by editingdevice_typein the generated genai_config — no rebuild needed.openvino/npuproduces an INT4 decoder (~1.5 GB) + embedding/vision/audio components, withgenai_configprovider_options = [{"OpenVINO": {"device_type": "NPU"}}]on every component.onnxruntime-openvino1.24 (OpenVINO 2024/2025): the mobius export uses ONNX opset 24 ops — notablyai.onnx::Attention(plusRMSNormalization,RotaryEmbedding) — and the OpenVINO EP's ONNX frontend does not yet support the opset-24Attentionop. Loading fails at session init (it does not fall back to CPU):openvino/defaultandonnx-standardmobius builds — all emit the opset-24Attentionop. The model does load/run on the plain CPU EP (opset-24 supported there).So this recipe needs an OpenVINO build whose ONNX frontend implements opset-24
Attention(a newer OpenVINO), or a graph pass that decomposesAttentioninto OpenVINO-supported primitives. A prebuilt INT4 package is on the Hub for hardware/version testing:justinchuby/gemma-4-E2B-it-ONNX(openvino/npu).Depends on: microsoft/Olive#2553 (
MobiusBuilderOpenVINO routing), onnxruntime/mobius#388 (mobiusopenvinoEP).QNN / Snapdragon HTP —⚠️ experimental, untested on hardware
qnn/*A two-step flow (the QNN/HTP LLM passes operate on a single decoder, but mobius emits a multi-component package):
qnn/build/config.json—MobiusBuilder(fp32, onnx-standard)→OnnxKQuantQuantization(int4).qnn/decoder/config.json:MatMulNBitsToQDQ→GraphSurgeries→OnnxStaticQuantization→SplitModel→StaticLLM→EPContextBinaryGenerator→ComposeOnnxModels.qnn/{vision_encoder,audio_encoder,embedding}/config.json:MatMulNBitsToQDQ→EPContextBinaryGenerator(enable_htp_fp16_precision)→ComposeOnnxModels.INT4 K-Quant weights are lowered to standard-ONNX INT4 QDQ (
DequantizeLinear+MatMul) viaMatMulNBitsToQDQ.Validation status: building an HTP context binary requires the QNN SDK (
onnxruntime-qnn) and a Snapdragon target; it cannot run on x86/CUDA. CPU-validated here: mobiusonnx-standardbuild (decoder 100%ai.onnx), K-Quant →MatMulNBitsToQDQ(all 276 decoderMatMulNBits→ INT4 QDQ, zerocom.microsoftops), and decoderGraphSurgeries. The ModelBuilder-specificRemoveRopeMultiCachesurgery is omitted (it crashes on the mobius decoder, which uses a singleRotaryEmbedding).Also changed
info.yml: add the OpenVINO + QNN recipes, theOpenVINOExecutionProvider/QNNExecutionProviderEPs, and thenpudevice.README.md: document both EP paths (incl. the OpenVINO opset-24 limitation)..gitignore: ignore generatedopenvino/*/modelsandqnn/*/models(and re-include theqnn/build/recipe config the repo-rootbuild/pattern would otherwise hide).Note
This PR is best kept as draft / not-for-merge until at least the OpenVINO opset-24
Attentionsupport lands (or anAttention-decomposition pass is added), since neither EP path is runnable on current released runtimes.