Controlnet training method for playgroundV2.5 model #8596
Description
When I try to train a controlnet model for playgroundV2.5, but there were some issues happened.
The general training code in diffusers for controlnet sdxl is not fit for EDM model, training is ok, loss looks ok, but the result is broke, after 400 steps the image looks like snow pieces. although I tied to set the scheduler to EDM type....
I think is mostly due to the EDM mean and EDM std inside the VAE config and the vae scale factor is also different from normal sdxl model.
I think something should change for encode the images.
How can I fix it?