feat(model-manager): load nvfp4 Qwen3-VL text encoders (Krea-2, Ideogram 4)

docs/src/content/docs/Users Guide/03.Models/02.model-families.mdx

@@ -1,7 +1,7 @@
 ---
 title: Model Formats and Families
 description: An overview of the models InvokeAI supports — checkpoint, Diffusers and quantized formats, the components that make up a model, and the model families with their licenses.
-lastUpdated: 2026-09-30
+lastUpdated: 2026-10-11
 sidebar:
   order: 2
 ---
@@ -48,7 +48,7 @@ InvokeAI recognizes the format from the file's contents, so you install them lik
 - **NF4** (bitsandbytes): 4-bit, CUDA only.
 - **SDNQ**: see [SDNQ Quantization](/configuration/optimization/sdnq-quantization/).
 - **int8** (`int8_convrot`, from ComfyUI repackages): the weights stay int8 in memory on every device, including Apple Silicon, and need no setting to turn on. Supported for FLUX.1, FLUX.2 Klein, Z-Image, Krea-2, Ideogram 4, MiniMax H3, LTX-2 and the PiD decoders, and for the Qwen3 8B, Qwen3-VL and LTX-2 Gemma encoders.
-- **nvfp4** (4-bit, in ComfyUI's checkpoint format, such as Comfy-Org's builds; other nvfp4 exports such as AWQ are refused): the weights stay packed in memory and are unpacked per layer, so no special GPU is needed. Supported for the Z-Image, Krea-2, Qwen-Image and LTX-2 transformers and for the Qwen3, Qwen2.5-VL and FLUX.2 Mistral encoders; the Ideogram 4 nvfp4 files are refused at install.
+- **nvfp4** (4-bit, in ComfyUI's checkpoint format, such as Comfy-Org's builds; other nvfp4 exports such as AWQ are refused): the weights stay packed in memory and are unpacked per layer, so no special GPU is needed. Supported for the Z-Image, Krea-2, Qwen-Image and LTX-2 transformers and for the Qwen3, Qwen3-VL 4B/8B, Qwen2.5-VL and FLUX.2 Mistral encoders; the Ideogram 4 nvfp4 transformer files are refused at install.
 - **fp8** (plain or `fp8_scaled`): InvokeAI keeps the weights in fp8 when [FP8 Storage](/configuration/optimization/fp8-storage/) is on, which installation does for you, or when `fp8_compute` is enabled.
 - **MXFP8** (fp8 with one scale per block of 32 weights): loaded by unpacking to BF16. Turn FP8 Storage off for these; see the [FP8 Storage caution](/configuration/optimization/fp8-storage/).
 

docs/src/content/docs/Users Guide/03.Models/03.model-format-support.mdx

@@ -1,7 +1,7 @@
 ---
 title: Model Format Support
 description: Which file formats and quantizations InvokeAI loads for every model family, text encoder, VAE and adapter.
-lastUpdated: 2026-10-10
+lastUpdated: 2026-10-11
 sidebar:
   order: 3
 ---
@@ -57,7 +57,7 @@ Ministral 3B encoder. Others are recognized by their layer names and fail, or mi
 | T5-XXL | FLUX.1, SD 3 | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ (and bitsandbytes int8) |
 | CLIP-L / CLIP-G | SD, SDXL, FLUX.1, SD 3 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | CLIP-L inside a FLUX.1 SDNQ pipeline |
 | Qwen3 0.6B / 4B / 8B | Z-Image, FLUX.2 Klein, Anima | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Qwen3-VL 4B / 8B | Krea-2, Ideogram 4 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ |
+| Qwen3-VL 4B / 8B | Krea-2, Ideogram 4 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ |
 | Qwen3-VL 32B (truncated) | MiniMax H3 | in the Diffusers folder | ✓ | L | ✓ | L | ✗ | ✗ |
 | Qwen3.5 4B | Anima 3.8B | ✗ | ✓ | L | L | L | ✗ | ✗ |
 | Qwen2.5-VL | Qwen-Image | ✓ | ✓ | ✓ | L | ✓ | ✗ | ✗ |
@@ -68,7 +68,11 @@ Ministral 3B encoder. Others are recognized by their layer names and fail, or mi
 | UMT5 | Wan 2.2 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
 
 The Qwen3-VL GGUF encoder is the language model only; a separate `mmproj` vision file is refused, and split
-multi-part GGUFs have to be joined before installing. The Gemma-4 encoder installs only as a folder: its
+multi-part GGUFs have to be joined before installing. Qwen3-VL nvfp4 single files keep their layers packed: ComfyUI's
+`qwen3vl_8b_nvfp4`, and repacks that name their nvfp4 layers the same way, in `.comfy_quant` markers or the
+`_quantization_metadata` header. A single file with nvfp4 layers named neither way, with AWQ layers, or with packed
+weights missing nvfp4's per-tensor `weight_scale_2` fails to install with a message; quantized encoder folders (AWQ,
+ModelOpt, llm-compressor) are not covered by this. The Gemma-4 encoder installs only as a folder: its
 `config.json` and tokenizer files next to exactly one weight file, bf16 or int8. A standalone `.safetensors` file is
 not recognized.
 

docs/src/content/docs/Users Guide/03.Models/Local Models/ideogram4.mdx

@@ -1,7 +1,7 @@
 ---
 title: Ideogram 4
 description: Generate images with the Ideogram 4 text-to-image models.
-lastUpdated: 2026-10-08
+lastUpdated: 2026-10-11
 sidebar:
   order: 6
 ---
@@ -33,7 +33,7 @@ Comfy-Org's [`Comfy-Org/Ideogram-4`](https://huggingface.co/Comfy-Org/Ideogram-4
 - `text_encoders/qwen3vl_8b_fp8_scaled.safetensors` — the Qwen3-VL **8B** encoder (not the 4B one Krea-2 uses). A llama.cpp GGUF of the 8B encoder works as well; see [Qwen3-VL GGUF encoders](/users-guide/models/local-models/krea-2/#qwen3-vl-gguf-encoders).
 - `vae/flux2-vae.safetensors` — the 32-channel VAE, shared with FLUX.2, which is why it installs under the FLUX.2 base
 
-Pick the unconditional branch, the encoder and the VAE in the **Components** section next to the model — and pick the unconditional branch of the *same* build as the model. The `nvfp4` repackage in that repo is **not** supported yet and is refused at install.
+Pick the unconditional branch, the encoder and the VAE in the **Components** section next to the model — and pick the unconditional branch of the *same* build as the model. The `nvfp4` transformers in that repo are **not** supported yet and are refused at install; its nvfp4 encoder, `text_encoders/qwen3vl_8b_nvfp4.safetensors`, loads and keeps its layers packed.
 
 Which branch a file holds is read from the file itself, and from the filename for a build that does not record it, so keep these names as published.
 

docs/src/content/docs/Users Guide/03.Models/Local Models/krea-2.mdx

@@ -1,7 +1,7 @@
 ---
 title: Krea-2
 description: Generate images with the Krea-2 text-to-image models (Turbo and Raw), including the GGUF / single-file workflow, the conditioning enhancers and training-free style reference.
-lastUpdated: 2026-09-30
+lastUpdated: 2026-10-11
 sidebar:
   order: 7
 ---
@@ -70,7 +70,8 @@ the bundled components.
 :::note[Qwen3-VL encoder]
 Single-file Qwen3-VL encoders bundle only the weights. Their tokenizer and architecture config ship
 with InvokeAI, so these encoders load without network access. A single-file fp8 encoder is kept
-resident in fp8 (roughly half the VRAM of the bf16 encoder).
+resident in fp8 (roughly half the VRAM of the bf16 encoder), and an nvfp4 encoder keeps its layers packed
+at 4 bits.
 :::
 
 ### Qwen3-VL GGUF encoders

invokeai/backend/model_manager/configs/qwen3_vl_encoder.py

@@ -15,6 +15,10 @@
 )
 from invokeai.backend.model_manager.model_on_disk import ModelOnDisk
 from invokeai.backend.model_manager.taxonomy import BaseModelType, ModelFormat, ModelType, Qwen3VLVariantType
+from invokeai.backend.model_manager.util.qwen3_vl import drop_qwen3vl_unloaded_keys
+from invokeai.backend.quantization.fp8_scaled import parse_quantization_metadata
+from invokeai.backend.quantization.int8_convrot import read_comfy_quant_markers
+from invokeai.backend.quantization.nvfp4 import WEIGHT_SCALE_2_SUFFIX, find_nvfp4_layers
 
 # The two Qwen3-VL encoders v7 can build, by language-model width. Both have 36 layers, so the
 # width alone identifies the variant. Anything else -- including MiniMax H3's truncated
@@ -94,6 +98,25 @@ def _variant_from_checkpoint_shape(state_dict: dict[str | int, Any]) -> Qwen3VLV
     return variant
 
 
+def _raise_for_undecodable_nvfp4_layers(mod: ModelOnDisk, state_dict: dict[str | int, Any]) -> None:
+    """Refuse at install time an nvfp4 layer the loader would refuse at the first generation.
+
+    The loader's own check, over the tensors the loader reads -- so a visual tower or LM head it drops cannot
+    get a file refused. It turns away AWQ builds, packed weights without nvfp4's per-tensor global scale,
+    malformed layers, and layers neither a ComfyUI ``.comfy_quant`` marker nor the safetensors header names as
+    nvfp4 (ModelOpt-style exports among them). Identification holds meta tensors, whose markers have no bytes to
+    parse, so those are read from the file -- and only from a file that has an nvfp4 layer to name.
+    """
+    consumed = drop_qwen3vl_unloaded_keys(state_dict)
+    names = parse_quantization_metadata(mod.metadata())
+    if any(isinstance(key, str) and key.endswith(WEIGHT_SCALE_2_SUFFIX) for key in consumed):
+        names = {**names, **read_comfy_quant_markers(mod.path)}
+    try:
+        find_nvfp4_layers(consumed, header_layers=names)
+    except ValueError as e:
+        raise InvalidMatchError(f"this Qwen3-VL encoder cannot be loaded: {e}") from e
+
+
 class Qwen3VLEncoder_Qwen3VLEncoder_Config(Config_Base):
     """Configuration for standalone Qwen3-VL text encoder models (diffusers-like directory format).
 
@@ -210,6 +233,9 @@ def from_model_on_disk(cls, mod: ModelOnDisk, override_fields: dict[str, Any]) -
         state_dict = mod.load_state_dict()
         if not _is_qwen3_vl_encoder_state_dict(state_dict):
             raise NotAMatchError("state dict does not look like a single-file Qwen3-VL encoder")
+        # Ahead of the variant, whose width is read off the embedding: an undecodable repack that packs the
+        # embedding too would fail there instead, and register as an unknown model with the reason only in the log.
+        _raise_for_undecodable_nvfp4_layers(mod, state_dict)
         variant = override_fields.pop("variant", None) or _variant_from_checkpoint_shape(state_dict)
 
         return cls(**override_fields, variant=variant)

invokeai/backend/model_manager/load/model_loaders/krea2.py

@@ -34,15 +34,16 @@
 )
 from invokeai.backend.model_manager.util.llamacpp_keys import convert_llamacpp_decoder_keys
 from invokeai.backend.model_manager.util.qwen3_vl import (
+    drop_qwen3vl_unloaded_keys,
     drop_qwen3vl_visual_tower,
-    drop_qwen3vl_visual_tower_keys,
     normalize_qwen3vl_rope_config,
     qwen3vl_target_key,
 )
 from invokeai.backend.quantization.fp8_scaled import (
     INPUT_SCALE_SUFFIXES,
     attach_fp8_scales,
     cast_state_dict,
+    count_fp8_weights,
     dequantize_fp8_scaled,
     detach_layer_sidechannel,
     extract_comfy_quant_hints,
@@ -866,9 +867,10 @@ def _load_te_config(self, config: _Qwen3VLSingleFileConfig) -> Any:
 class Qwen3VLEncoderCheckpointLoader(_Qwen3VLEncoderSingleFileLoader[Qwen3VLEncoder_Checkpoint_Config]):
     """Loads a single-file Qwen3-VL encoder checkpoint (e.g. ComfyUI ``qwen3vl_4b_bf16`` / ``_fp8_scaled``).
 
-    The checkpoint bundles the language model + visual tower; the tower is dropped on the way in (see
-    ``drop_qwen3vl_visual_tower_keys``). ComfyUI 'scaled fp8' weights are dequantized to the compute
-    dtype on load.
+    The checkpoint bundles the language model + visual tower; the tower is dropped on the way in, with the
+    LM head (see ``drop_qwen3vl_unloaded_keys``). ComfyUI 'scaled fp8' weights are kept for the fp8 matmul or fp8
+    storage where the device has either, and folded into the compute dtype otherwise. ``int8_tensorwise``
+    layers stay int8 and Comfy's ``nvfp4`` layers stay packed, both dequantized per forward.
     """
 
     CONFIG_CLASS = Qwen3VLEncoder_Checkpoint_Config
@@ -886,7 +888,19 @@ def _load_text_encoder(self, config: Qwen3VLEncoder_Checkpoint_Config) -> AnyMod
         # Ahead of every pass that reads tensors, so the tower is never dequantized, cast, copied or
         # reserved for. (The file *header* is still read below for quantization metadata, which can
         # therefore still name visual layers; those hints match no surviving key and are ignored.)
-        sd = drop_qwen3vl_visual_tower_keys(sd)
+        # The LM head goes with it: this model has none, and a packed one would name no module.
+        sd = drop_qwen3vl_unloaded_keys(sd)
+        metadata = read_safetensors_metadata(model_path, self._logger)
+
+        # Comfy's nvfp4 build packs every language-model projection and names the layers in `.comfy_quant`
+        # markers (a header in some repacks). Take them out before either side-channel branch reads a scale:
+        # the int8 branch strips every remaining marker -- one of the two things that name an nvfp4 layer --
+        # and the fp8 extraction pops every `weight_scale` and discards those whose weight is not float8,
+        # nvfp4's block scales included. `install_nvfp4_layers` puts them back, packed, under the paths the
+        # remap gives their weights.
+        nvfp4_payloads = pop_nvfp4_layers(sd, header_layers=parse_quantization_metadata(metadata))
+        nvfp4_payloads = _remap_qwen3vl_singlefile_keys(nvfp4_payloads)
+
         # Same one-or-the-other split as the transformer above, and here it also guards the fp8
         # *detection*: an int8 layer ships a `.weight_scale` too, so probing for scales without
         # ruling out int8 first would keep the encoder "fp8-resident" over weights that were never
@@ -913,7 +927,6 @@ def _load_text_encoder(self, config: Qwen3VLEncoder_Checkpoint_Config) -> AnyMod
             # only what this path already left dense -- work with no saving on the layers that matter.
             use_fp8_storage = False
         else:
-            metadata = read_safetensors_metadata(model_path, self._logger)
             # Per-layer markers must be read before extract_fp8_scaled_layers() drops them, and before
             # the key remap, which would not carry a ".comfy_quant" suffix to a sensible destination.
             layer_hints = {**extract_comfy_quant_hints(sd), **parse_quantization_metadata(metadata)}
@@ -942,19 +955,22 @@ def _load_text_encoder(self, config: Qwen3VLEncoder_Checkpoint_Config) -> AnyMod
             # for the byte count the file already had.
             use_fp8_storage = source_is_fp8 and _device_supports_fp8_storage(self._torch_device, self._logger)
 
+        # Every layer that arrived scaled, before the split folds the ones the matmul cannot take. The storage
+        # pass below must leave those alone: cast again, they would be re-quantized *without* their scale.
+        scaled_paths = set(fp8_layers)
+
         te_config = self._load_te_config(config)
         with accelerate.init_empty_weights():
             model = Qwen3VLModel._from_config(te_config)
         # Its weights were dropped from the state dict above, so the module has to go too -- left in
         # place it would stay on the meta device and `_reject_incomplete_load` would rightly refuse.
         drop_qwen3vl_visual_tower(model)
+        # `Qwen3VLModel` declares no precision-sensitive modules, but read them rather than assume: in the
+        # int8 split they are also what keeps a marker on a non-Linear (a "quantize everything" repack's 1-D
+        # norms) out of `swap_in_int8_linears`, and the nvfp4 install decodes rather than packs under them.
+        skip_patterns = _model_declared_skip_patterns(model)
 
         if int8_markers:
-            # `Qwen3VLModel` declares no precision-sensitive modules, but read them rather than
-            # assume: the split is also what keeps a marker on a non-Linear (a "quantize
-            # everything" repack's 1-D norms) out of `swap_in_int8_linears`.
-            skip_patterns = _model_declared_skip_patterns(model)
-
             quantized = install_int8_convrot_layers(
                 model,
                 sd,
@@ -963,6 +979,9 @@ def _load_text_encoder(self, config: Qwen3VLEncoder_Checkpoint_Config) -> AnyMod
                 architecture="Qwen3-VL encoder",
                 reserve=self._ram_cache.make_room,
                 skip_patterns=skip_patterns,
+                # The nvfp4 layers taken out above are held beside the int8 ones, and this is the
+                # branch's only reservation.
+                extra_reserved_bytes=predict_nvfp4_install_size(model, nvfp4_payloads, model_dtype, skip_patterns),
             )
             self._logger.info(
                 f"Qwen3-VL encoder: kept {len(quantized)} of {len(int8_markers)} layer(s) in int8 "
@@ -999,21 +1018,27 @@ def _load_text_encoder(self, config: Qwen3VLEncoder_Checkpoint_Config) -> AnyMod
                 keep_fp8=keep_fp8,
                 model=model,
                 fp8_layers=fp8_layers,
-                nvfp4_payloads={},
+                nvfp4_payloads=nvfp4_payloads,
+                skip_patterns=skip_patterns,
             )
             if fp8_layers and not keep_fp8:
                 # Neither consumer wants them packed: fold the scales into the weights. Both consumers
                 # were resolved above, next to the matmul probe, because this fold is irreversible.
                 dequantize_fp8_scaled(sd, fp8_layers, model_dtype)
                 fp8_layers = {}
-            fp8_layers = split_fp8_scaled_layers(sd, fp8_layers, model_dtype, model=model)
-            # No `skip_patterns` here on purpose: this model declares none, and the storage pass below
-            # applies `_FP8_DEFAULT_SKIP_PATTERNS` itself. Those two lists used to have to not intersect
-            # on a layer class with no fp8-capable wrapper -- a `pos_embed` or a `patch_embed.proj`
-            # would arrive fp8 from the state dict, be skipped by the cast pass, and then raise on the
-            # first forward. `_apply_fp8_to_nn_module` now restores the compute dtype on the modules it
-            # skips, so the two lists are independent again.
-            cast_state_dict(sd, model_dtype, keep_fp8=keep_fp8, model=model)
+            # The model's own patterns (none, today) go to the reservation, the split, the cast and the nvfp4
+            # install alike, so the prediction answers for what each of them then does. They are independent
+            # of `_FP8_DEFAULT_SKIP_PATTERNS`, which the storage pass below applies itself: a `pos_embed` or
+            # `patch_embed.proj` arriving fp8 and skipped there used to raise on the first forward, and
+            # `_apply_fp8_to_nn_module` now restores the compute dtype on the modules it skips.
+            fp8_layers = split_fp8_scaled_layers(sd, fp8_layers, model_dtype, model=model, skip_patterns=skip_patterns)
+            cast_state_dict(sd, model_dtype, keep_fp8=keep_fp8, model=model, skip_patterns=skip_patterns)
+
+        if nvfp4_payloads:
+            packed = install_nvfp4_layers(model, sd, nvfp4_payloads, model_dtype, skip_patterns)
+            self._logger.info(
+                f"Qwen3-VL encoder '{config.name}': kept {packed} of {len(nvfp4_payloads)} nvfp4 layer(s) packed."
+            )
 
         load_state_dict_ignoring_extras(
             model, sd, source="Qwen3-VL encoder checkpoint", assign=True, allow_missing=True
@@ -1074,7 +1099,22 @@ def _load_text_encoder(self, config: Qwen3VLEncoder_Checkpoint_Config) -> AnyMod
         if use_fp8_storage:
             # `model.dtype` now reports the float8 storage dtype; `_apply_fp8_to_nn_module` records
             # the real compute dtype so callers can recover it via `get_model_compute_dtype`.
-            self._apply_fp8_to_nn_module(model, storage_dtype=torch.float8_e4m3fn, compute_dtype=model_dtype)
+            #
+            # A scaled layer reaches this branch only when the split folded it -- every one it had, or the
+            # attach branch above would have run. The Linears among them go to fp8 storage like any other,
+            # which is what this pass is for: an MXFP8 build's block scales are folded, and its language model
+            # is held in fp8 rather than at twice the size. A folded embedding is not: Comfy's nvfp4 8B build
+            # scales its token table, and cast again it would lose that scale for the accuracy cost measured
+            # above for exactly this table -- the reason the attach branch leaves every embedding alone.
+            self._apply_fp8_to_nn_module(
+                model,
+                storage_dtype=torch.float8_e4m3fn,
+                compute_dtype=model_dtype,
+                skip=lambda name, module: name in scaled_paths and not isinstance(module, torch.nn.Linear),
+            )
+        # Counted rather than assumed: on that nvfp4 build every Linear is packed and the embedding skipped, so
+        # the pass leaves nothing in fp8, and saying it did would misdescribe the encoder.
+        if use_fp8_storage and count_fp8_weights(model):
             self._logger.info(
                 f"FP8 layerwise casting enabled for Qwen3-VL encoder '{config.name}' "
                 f"(storage=float8_e4m3fn, compute={model_dtype})."