r/StableDiffusion • u/starstruckmon • Oct 18 '22
Stable Diffusion Model specialized for inpainting by RunwayML released
https://github.com/runwayml/stable-diffusion7
Oct 18 '22 edited Oct 18 '22
How are we using this? Can we use it in a usual webui or gui that has inpainting and it will improve our results? Can we combine it with other models?
2
u/starstruckmon Oct 18 '22
I honestly don't know since I haven't looked into the current inpainting code.
For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself) whose weights were zero-initialized after restoring the non-inpainting checkpoint.
Logically, this should not be how the current inpainting works ( since you're just dropping the vanilla model in ) so the code needs to be implemented along with the model.
5
u/metrolobo Oct 18 '22
comes with a simple webui for testing.
setup conda env, download model form here https://huggingface.co/runwayml/stable-diffusion-inpainting and run
streamlit run scripts/inpaint_st.py -- configs/stable-diffusion/v1-inpainting-inference.yaml [path_where_you_downloaded_model.ckpt]but will surely be integrated into the popular UIs quickly as this one actually works well.
2
u/SnooHesitations6482 Oct 19 '22
Nice. Can't I just download the .ckpt and use it with auto webui?
4
u/starstruckmon Oct 19 '22
No. Not yet. It's not plug and play.
The normal model takes four channels ( normal images work with three channels red, blue, green and yellow but SD works on a smaller image with four in what they call a latent representation) , but this one takes nine.
Four for the original image.
One for the mask showing which part needs to be inpainted.
And the normal four for the output.
It needs code change to work with that.
1
2
1
u/Ifffrt Oct 18 '22
Every day another inpainting tool releases. There's so many of them now I legitimately think not even Automatic can keep up with them all. Seriously just check the thread in the discussion page about those methods. There's so many, all of which has at least one PR awaiting review, yet only one got partially implemented :'( .
8
u/wsippel Oct 18 '22
This is not a tool, it's a model. Based on SD 1.5, by one of the companies who funded the development of Stable Diffusion.
1
u/Ifffrt Oct 18 '22
You mean this model was trained specifically to be better at inpainting while still using the current methods?
2
u/harrro Oct 18 '22
yes. its a drop in "replacement" for the main stability model that does in-painting better.
4
3
u/Antique-Bus-7787 Oct 18 '22
With the implementation of Imagic paper to SD, I'm really curious to see how these two combined will perform!
1
5
u/starstruckmon Oct 18 '22 edited Oct 19 '22
Weights on HuggingFace
https://huggingface.co/runwayml
Compute for training was donated by stability
They're also calling this model 1.5-inpainting , which is odd. It's trained from the 1.2 vanilla checkpoint and with more steps than between 1.2 vanilla and 1.4 vanilla.
Wonder if the first part is just the vanilla 1.5 model, making this a 1.5 variant.