feat(model-manager): load nvfp4 Qwen3-VL text encoders (Krea-2, Ideogram 4)

[Pfannkuchensack]

Summary

Single-file nvfp4 builds of the Qwen3-VL 4B/8B text encoder (Krea-2, Ideogram 4) installed, then failed at the first generation with size mismatch ... [out, in/2], after the model cache had already been cleared for them: the checkpoint loader passed nvfp4_payloads={} and never took the packed layers out. They now load packed (NVFP4Linear, dequantized per forward), the way the Qwen3, Qwen2.5-VL and Mistral encoder loaders already do, and nvfp4 layers the loader cannot decode are refused at install.

File	Before	After
Comfy-Org qwen3vl_8b_nvfp4.safetensors (Ideogram 4)	installs, generation fails	252/252 layers packed, 4.80 GiB resident (the fp8_scaled build: 7.63 GiB)
Community 4B nvfp4 (mixed nvfp4/fp8/bf16)	installs, generation fails	224/224 layers packed, 2.79 GiB resident (fp8_scaled 4.11, bf16 7.49 GiB)
AWQ, unnamed (ModelOpt-style) or missing weight_scale_2	installs, generation fails	refused at install with the reason

• Loader (Qwen3VLEncoderCheckpointLoader): nvfp4 layers leave the state dict before the int8 branch (which strips every remaining .comfy_quant marker) and the fp8 extraction (which discards non-fp8 block scales), are remapped to the transformers paths, counted in the branch's one reservation, and installed packed. The LM head is dropped with the visual tower before any tensor work (drop_qwen3vl_unloaded_keys); the 8B fp8 builds no longer fold a 151936x4096 head only to discard it.
• Scaled embedding: the official 8B build ships its token table as scaled fp8. Folded, it is no longer re-quantized without its scale by the fp8 storage pass; folded Linears (e.g. an MXFP8 build's) still go to fp8 storage as before.
• Install check: Qwen3VLEncoder_Checkpoint_Config runs the loader's own validator, find_nvfp4_layers (made public), over the keys the loader reads, with the .comfy_quant markers read from the file, since identification only holds meta tensors.
• Fold peak: dequantize_fp8_scaled folds tensors above 2^24 elements a slab of rows at a time. On the 8B embedding the RAM peak drops from 5.80 to 1.13 GiB, bit-identical result.
• Docs: format matrix, Krea-2, Ideogram 4 and model-families pages.

Related Issues / Discussions

Closes #9735

QA Instructions

Automated (Windows, CPU):

• pytest -n 4 tests/backend/model_manager tests/backend/quantization tests/app/invocations tests/model_identification: 4809 passed, 159 skipped, 7 xfailed. New tests: tests/backend/model_manager/load/test_qwen3_vl_encoder_nvfp4_loader.py (a real toy Qwen3VLModel with the official layout, marker- and header-named, in fold / fp8-compute / fp8-storage modes; an int8 neighbour; MXFP8 under fp8 storage; refusals before the loader's reservation), new cases in tests/backend/model_manager/configs/test_qwen3_vl_encoder_config.py, and TestFoldInSlabs in tests/backend/quantization/test_fp8_scaled.py.
• ruff@0.11.2 check and format --check on the changed files: clean.
• Docs: pnpm -C docs test, pnpm -C docs build, check-deploy-output pass; scripts/generate_docs_json.py leaves docs/src/generated unchanged.
• All seven local Qwen3-VL single files identify with the right variant (8B/4B nvfp4, 8B/4B fp8_scaled, 4B bf16, two 4B int8).

Manual, through the real loader with the model cache's custom layers (RTX 4090; RX 9060 XT with ROCm 10 on Windows):

Comparison (6 prompts)	Conditioning cosine	Relative L2
8B nvfp4 vs 8B fp8_scaled (Ideogram 4 encode path)	0.9966–0.9980	0.081–0.113
4B nvfp4 vs 4B bf16 (Krea-2 encode path)	0.9900–0.9941	0.108–0.141
4B fp8_scaled vs 4B bf16, for reference	0.9989–0.9994	0.035–0.047
8B nvfp4, RTX 4090 vs RX 9060 XT	≥ 0.99996	–

• Decoded weights reach cosine 0.995 against the other builds' weights (4-bit block quantization's floor), so the 4B's lower conditioning cosine comes from that community quant, not the decode.
• Images at the same seed, through the server: Krea-2 Turbo (fp8_scaled) and Ideogram 4 (GGUF Q4_0 pair) give near-identical compositions with the fp8 and the nvfp4 encoder, and are bit-identical across repeated runs. Ideogram 4 renders its "Image blocked by safety filter" placeholder for some prompts with either encoder.
• Speed, warm and interleaved fp8/nvfp4 (Krea-2, 3 runs each): sessions 14.17/14.60/13.84 s vs 14.21/14.14/13.81 s; encoder node 0.65–0.69 s vs 0.66–0.70 s.
• A synthetic text-encoder LoRA on packed layers applies as a sidecar (output changes) and unpatches bit-identically.
• Not available: MPS and XPU were not tested. There is no FP4 compute path; packed layers dequantize per forward on every device, Blackwell included.

Review

Material findings resolved:

• An earlier draft skipped every folded scaled layer in the fp8 storage pass, which would have kept MXFP8 encoders' Linears in bf16 (about +3.6 GB resident on the 4B, +6.9 GB on the 8B). The skip now covers folded non-Linear modules only, with a regression test.
• The 5.8 GiB float32 transient when folding the 8B scaled embedding is gone: row-slab fold.
• The install check reuses the loader's validator instead of its own predicates, so unnamed layers are refused at install too.
• Coverage was added for the int8+nvfp4 branch, for fp8/int8 files still installing, and for the log line.

Remaining limitations:

• A valid repack that also packs the embedding would not be identified, because the variant is read from the embedding's width. No such file is known.
• A file whose header names a layer nvfp4 while its marker carries no readable format is refused at install, although it would load.
• The install-time refusal of unnamed layers reuses the loader's message. Its hint about a header warning only applies at load time.
• Quantized Qwen3-VL encoder folders (AWQ, ModelOpt, llm-compressor) are out of scope.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}