Repository navigation
feat(flux3): load quantized FLUX.3 single files - #10093
Pfannkuchensack wants to merge 7 commits into
Conversation
FLUX.3 Klein and Klein Base now load Comfy-Org's int8_convrot and nvfp4 files and BFL's fp8r and nvfp4 variants; one rule set refuses at install what the loader cannot run. BFL's nvfp4 is re-laid out to Comfy's packing after the reservation, and the int8 build's fused qkv is split into the model's projections.
github-actions
Bot
added
python
| int8_markers = {path: marker for path, marker in markers.items() if marker.get("format") == INT8_TENSORWISE_FORMAT} | ||
| reject_unmarked_int8_weights(state_dict, int8_markers, "FLUX.3") | ||
| for path, marker in int8_markers.items(): | ||
| group_size = marker.get("convrot_groupsize", CONVROT_GROUP_SIZE) | ||
| try: | ||
| if marker.get("convrot"): | ||
| check_hadamard_size(int(group_size)) |
There was a problem hiding this comment.
[P2] Unsupported int8 layouts pass installation validation, then fail on first generation.
Validation checks marker presence and power-of-four rotation groups, but omits required scales, supported scale shapes, and divisibility against the input width.
The fresh reviewer reproduced three cases on pinned HEAD: a missing weight_scale, an unsupported blockwise scale, and a rotation group larger than the input width. The identification validator accepted each; the production loader rejected each. Users consequently get an installed model that cannot generate.
Reuse existing int8 layout validation in the shared checker and add installation-refusal coverage. This is a bounded validation defect affecting malformed or unsupported repacks; it does not demonstrate a failure in the four published builds.
…hecks Install-time validation now reuses check_int8_layer_layout, so a missing scale, blockwise scale or oversized rotation group is refused at install.
… check Split the rotation-width rule out of check_int8_layer_layout so the swap runs it after the target-shape check; install runs both.
|
Fixed: install validation now uses the loader's own checks. |
# Conflicts: # docs/src/content/docs/Users Guide/03.Models/Local Models/flux-3.mdx # invokeai/backend/flux3/denoise_working_memory.py
# Conflicts: # docs/src/content/docs/Users Guide/03.Models/Local Models/flux-3.mdx
Summary
The quantized FLUX.3 Image Klein / Klein Base single files were refused at install. They now load:
flux3-image-klein{,-base}-int8-convrot)flux3-image-klein{,-base}-nvfp4)variants/nvfp4(NVIDIA ModelOpt packing)variants/fp8r(per-row scaled fp8)invokeai/backend/flux3/checkpoint_quantization.py). A file the loader cannot run is refused at install with a reason: torchao, bitsandbytes, fp8 scales in an unknown spelling, int8 without markers, unnamed or half-stored nvfp4, int8 mixed with another scheme, packed input embedders.feef60b0…. BFL's variants install by URL (both are namedmodel.safetensors); the docs give pinned fp8r URLs.Related Issues / Discussions
Follow-up to #9883 and #10084.
QA Instructions
CPU, through the production loader on the real Klein files: every build identifies as one FLUX.3 Klein match; reserved bytes equal resident bytes; dequantized layers against the bf16 file: int8 worst cosine 0.99995 (rel. error 1 %), nvfp4 worst cosine 0.9966 (rel. error 8 %, inherent to fp4), fp8r worst cosine 0.99965 (rel. error 2.6 %).
pytest tests/backend/flux3 tests/backend/quantization tests/backend/model_manager/configs tests/backend/model_manager/load tests/model_identificationand the starter tests — pass.GPU, RTX 4090, FLUX.3 Klein at 1024², 16 steps, three prompts × three seeds. Each build starts from an empty model cache with one warm-up image. The builds ran once in each order, and the table averages both orders, because the card slows a little over a series:
int8_compute/fp8_computeint8_computefp8_computefp8_computeon, the scaled-fp8 text encoder runs on fp8 compute as well, so that run's images, bf16 included, differ from the default run. Its PSNR values compare against the bf16 images of the same run.Docs:
pnpm buildindocs/builds all 360 pages, and the link validator reports no broken link.Review
Compatibility / Rollout
Checklist
What's Newcopy (if doing a release after this PR)Need help on this PR? Tag
@codesmith-botwith what you need.