Comfy Nodes or Wrapper

[MaisonMeta]

Spent the day try to make it into a comfy wrapper for the community but I failed so far.
Tried to just use it as a model + Lora in regular wrokflows, it works but do not apply changes as per paper. I must be doing something wrong...

Would love if someone can help turn this into a comfy node.

[prepstorm]

Could you share it somehow? Would love to dig into this. Also, I tried a quantized flux version, however I think it needs a bit of more work to load GGUFs properly.

[primecai]

Does it work if you use one of the long-prompt examples in this demo page?

[prepstorm]

I am in the process of attempting to quantizing the models to GGUF. Just give me a moment. However, since we don't have any wrapper nodes for it yet we still cannot really test the quants. @MaisonMeta would you mind sharing your node code somehow so I can take a look at it?

[MaisonMeta]

@prepstorm - Yes, I am going to push it Github. Also note, it is not yet working unfortunately but wouldnt mind some help :)
Sharing it shortly...

[prepstorm]

@MaisonMeta @primecai I have started to upload my quants on HF here:
https://huggingface.co/prepstorm/diffusion-self-distillation-gguf

This is my first GGUF quants so I am not sure how well they perform. But I will keep uploading a few more versions. Also, I was not sure what license these go under, but if you require me to change it just hit me up and I will fix it.

[MaisonMeta]

@prepstorm Amazing work. Here is the public repo of the wrapper https://github.com/MaisonMeta/DSD_comfy_node
Hope you can fix it!

[prepstorm]

Thank you! I will dig into it soon.

[prepstorm]

The GGUFs might be bugged. Might look into this asap

[prepstorm]

After messing around with this all day I noticed that my quants didn't work properly. However, I decided to do a merge at least with the LORA and got this result:

https://huggingface.co/prepstorm/diffusion-self-distillation-merge

Just for fun I loaded it up in Comfy with an img2img Flux workflow and even though some things seems to work with "This character..." prompting, some of it is not consistent. This is probably not the way to use the model, but at least you don't need the LORA since it is embedded into it.

I suppose we have to wait for @primecai to update this in the proper way.

Here is a link to my workflow: https://ibb.co/VcfHSh1c

Edit: Also, it seems to work better with anime styles for some reason.

[primecai]

Thank you so much for the work, I will look into it ASAP. One thing I do notice is the the Comfy does not have prompt enhancement set up --- if that is the case, we will have to write very detailed prompts since the model is trained on those. I am not familiar with Comfy but I suppose it should be portable to Gemini or something.

[prepstorm]

Thank you so much for the work, I will look into it ASAP. One thing I do notice is the the Comfy does not have prompt enhancement set up --- if that is the case, we will have to write very detailed prompts since the model is trained on those. I am not familiar with Comfy but I suppose it should be portable to Gemini or something.

No problem. Sure, but if it all comes down to the prompting I will have to kind of benchmark my merge just to see what the difference of that and regular I2I might be, if there is any. And since it sort of works in Comfy, maybe there might be other alternatives besides Gemini available? Maybe Florence might work?

[MaisonMeta]

Thank you so much for the work, I will look into it ASAP. One thing I do notice is the the Comfy does not have prompt enhancement set up --- if that is the case, we will have to write very detailed prompts since the model is trained on those. I am not familiar with Comfy but I suppose it should be portable to Gemini or something.

No, they do, They have any LLMs node you can think of, from Florence to Gemini or any other APIs as well.
I can prepare you one quickly for LLM prompt enhancements (if you want).

**UPDATE: https://github.com/MaisonMeta/DSD_comfy_node/blob/main/example_DSD_workflow.json here is the updated workflow and added a string entry on the node to merge the LLM prompts to the node prompts.

Here is the node to install for Gemini API :
https://github.com/Visionatrix/ComfyUI-Gemini
or this one:
https://github.com/ZHO-ZHO-ZHO/ComfyUI-Gemini

Quick qn: On some of your examples, there's weather changes, with just simple prompts like: "Dramatic sky"
Does these needs prompt enhancements as well ?

[primecai]

Thanks! This is very helpful, I'll look into it.
For the weather change/relighting model, the current released model does not support those yet. We have a separate model for "no pose change" effects but that model requires some further alpha testing, it is not robust enough yet.

[prepstorm]

This is how far I have gotten today, just using regular I2I

https://ibb.co/kggpb9bG

Not sure if I am hallucinating or if it really is influenced by my merge that I posted before. Or does someone know if this here is just regular I2I that could be achieved by Flux Dev or something?

Things that I noted: Being just a short prompt it seems to figure out the appearance of the character that I meant and the environment that I wanted to change, that is why I am a bit confused.

[alexisrolland]

Totally up voting this, it would be fantastic to have ComfyUI nodes for this!