Getting only noise output

[girzai]

Hi there, I currently convert my finetuned Z base image to int8 using :

ctq
-i /workspace/runpod-slim/ComfyUI/models/diffusion_models/zib.safetensors
-o /workspace/runpod-slim/ComfyUI/models/diffusion_models/zib.INT8.safetensors
--int8 --simple
--scaling_mode row
--convrot
--zimage
--comfy_quant
--save-quant-metadata
--verbose VERBOSE

But on ComfyUI 0.27 Im getting only noise as input

Is my ctq command wrong for the conversion ? Thanks

[YellowRoseCx]

what nodes are you using to load the model?

[girzai]

what nodes are you using to load the model?

A simple load diffusion model, was guessing comfyUI with the latest update automatically accept this quant now but Im probably wrong

[hum-ma]

I also get only noise from some Krea2 community checkpoints, although they don't have that kind of a pattern but instead completely random colored pixels even from the very first step preview. Some other checkpoints work fine when quantized with the same options --save-quant-metadata --low-memory --flux2 --simple. So in this case they are fp8 instead of int8, and I'm testing with ComfyUI 0.26.0

Also tried with --krea2 but that produces much larger files. With the flux2 profile the resulting files are over a gigabyte smaller, run much faster on 4GB VRAM and allow higher resolutions without OOM. I originally found this in Winnougan's instructions and the fp8mixed model file in that repo is also clearly smaller and faster than the fp8scaled in the Comfy-Org repo but I do wonder why some checkpoints fail to work correctly when quantized in this way.

[hum-ma]

It seems my noise problem was solved by adding blocks to the excluded layers. The problematic models were already partially quantized so it was not only redundant to re-quantize most of the layers but doing so broke the model. In fact it is specifically the tproj layer that is critical to quantize for Krea2 to be optimal with low VRAM. Some of the other layers such as first, last, projector and tmlp seem to be important to exclude (i.e. keep them at high precision) or quality suffers noticeably.

Oh, and looking at the ctq options more carefully I realized that the --flux2 filter wasn't actually doing anything for Krea2, as none of the layer names match.

[girzai]

It seems my noise problem was solved by adding blocks to the excluded layers. The problematic models were already partially quantized so it was not only redundant to re-quantize most of the layers but doing so broke the model. In fact it is specifically the tproj layer that is critical to quantize for Krea2 to be optimal with low VRAM. Some of the other layers such as first, last, projector and tmlp seem to be important to exclude (i.e. keep them at high precision) or quality suffers noticeably.

Oh, and looking at the ctq options more carefully I realized that the --flux2 filter wasn't actually doing anything for Krea2, as none of the layer names match.

Thanks for the update! That's really interesting.

Could you explain how you added blocks to the excluded layers? Did you modify the --exclude-layers regex manually, or did you make changes directly in the code?

I'm trying to convert a fully fine-tuned Z-Image model to INT8, so understanding exactly how you excluded those layers would be really helpful.

[hum-ma]

Did you modify the --exclude-layers regex manually,

Yes, created the exclusion list manually and removed the model filter option.

With most checkpoints I now use it like this: --exclude-layers "(first|last|project|tmlp|model.blocks.)" so not only plain "blocks" because that would also exclude layers like txtfusion.refiner_blocks which I often do want to quantize.

For Z-Image the names are different of course, I think you have "layers" instead of "blocks" there. But if your finetuned model is 16-bit and you exclude the main layers, the quantization probably won't do much to reduce size or increase speed. For me it worked because I downloaded models in which most layers were already 8-bit precision and really only needed to quantize the single text projection layer (and txtfusion in some cases).