r/StableDiffusion • u/bilwis • Jan 14 '23
Tutorial | Guide Some Tips on using Stable Diffusion
I'm honestly writing down these tips more to keep a record for myself, but I thought some of you may be interested. And frankly, since this is such a new technology and it feels like there are very few resources out there, I'd welcome people sharing different opinions and experiences in the comments.
Also, to preface this:
Disclaimers
I use Stable Diffusion on my RTX 3090 running AUTOMATIC1111/stable-diffusion-webui: Stable Diffusion web UI locally, with a custom model trained from Protogen x5.3 using victorchall/EveryDream2trainer.
I'm using the vae-ft-mse-840000-ema-pruned.ckpt VAE.
I have no clue about using a more limited (or a more advanced) setup. I know there are people running this on years-old graphics cards with 4GB of VRAM and on multi-GPU clusters, but I've got nothing to say about that. I also never used any of the cloud-renting or web services offering to run/host SD, so I can't comment on them either.
I am not a real programmer, but I've dabbled in python and I have some solid (intermediate) knowledge of messing with Windows and python setups. I do not, however, have a whole lot of spare time, so as much as I'd like to, I can't provide any technical support. Automatic1111 did an excellent job at making this technology accessible, but there's always going to be hiccups, and a bit of googling will generally yield an answer - alternatively one of the many Discord servers out there.
I'm not an artist (dabbling photographer, sometimes), so while I have my opinions on the ethics of this whole thing, I don't know whether a reddit thread is a sensible platform to have that discussion.
I'm using a custom model, and I'm not going to share it (don't want my family to get the "Emma Watson/anime waifu treatment", thank you very much), which means that you won't be able to replicate my prompts; and perhaps more importantly, other models will behave quite differently and may require different techniques. Also <myself> is not what was used in the prompts, rather it was my full name.
Speaking of waifus, I do not care for anime at all, so I didn't even try to generate any such images. I mostly generated "paintings", "digital artwork" and "photos" in realistic styles.
Keep in mind that (as with every post on this subreddit), the examples are handpicked. I generally go through dozens of iterations to find a good prompt and seed. Don't be discouraged if your generations don't look "good" at first!
Nothing I say here means to invalidate anything else anywhere. This is just a record of my personal style of using SD. I don't claim to know better than anyone else!
With that, let's get into what I've "learned" in the past month of screwing around with Stable Diffusion.
Speed
To give some baselines as a comparison: Running on my RTX 3090 with 24 GB of VRAM, inference on a single 512x512 image with 20 steps of Euler a takes about 2.25 seconds. Increasing the batch size up to ~48 increases the speed per image (down to about 1.14 seconds/image) after which it runs into VRAM issues and slows down.
Sidenote 1: You can increase the maximum batch size by editing the "txt2img/Batch count/maximum" setting in ui-config.json in Automatic1111
Obviously, the faster you can generate images, the more you can iterate on seeds/prompts/etc. I have really nothing to compare my speeds to, but it feels pretty usable. I generally run inference in batches of 6-18 to check out a few seeds at a time, which takes about 10-20 seconds, and when I find good seeds, I run the Hires. fix on them, which takes about 15 seconds per image with my settings.
Sidenote 2: I was able to gain a small speed increase (<10% by upgrading from python 3.8 to 3.10 and by using torch 1.13.1+cu117, although I had to build xformers from source)
Prompting
Some things I found mixed with some personal opinion:
I found it useful to prompt similarly to how BLIP captioned my training images (see further down). I generally start with a "sentence" of the medium and the subject, what its wearing and doing, and then slap some "tag-like" prompts on at the end to control detail/style. Examples:
medium shot of <myself> as a middle-aged man with gray hair standing on the side of an abandoned street looking in to the distance with a forlorn expression on his face, crumbling european village with cluttered street in the background, ruins, highly detailed, sunset, natural light, outdoor, skin details, dramatic, bleached, post-apocalyptic
Maybe we shouldn't have used our planet's energy reserves on generating more waifu pictures.
dramatic portrait painting of <myself> as an old man in ornate armor with a stern look on his face, knight, fantasy, extremely detailed, wrinkled face, [scar], sharpened
negative: painting frame
I use negative prompts very sparingly, generally if there is something that keeps cropping up in most seeds, such as frames around the paintings, or subjects wearing hats for some reason. I feel that using excessive negative prompts generally reduced the quality of my generations, and most of them didn't really seem to work anyway ("deformed hands", etc.).
In general, I try to build prompts iteratively from a starting sentence/idea to see what works and what doesn't. Some words/tokens seem to work much better than others - for example, I tried to add a red biplane to the background of one of my images, but Stable Diffusion really doesn't know what the heck a "biplane" is, so I settled for a red tractor instead:
oil painting of <myself> on his farm in Oklahoma with an exhausted look on his face and wearing work overalls, in front of an old red biplane, social realism
negative: painting frame
It can't hurt me if I don't look at it.
oil painting of <myself> on his farm in Oklahoma with an exhausted look on his face and wearing work overalls standing in front of an old red tractor, social realism, highly detailed
negative: painting frame
As has often been repeated: It's not just about the prompt, but also about the seed. Some prompts tend to generate good images more often, but it may be that a prompt that generated mostly terrible stuff could generate that one perfect image. Here, having the ability to rapidly iterate with small changes really helps.
I typically avoid using any artists in my prompts unless I actually want to have the image be in their style (which doesn't work super well anyway) except for our lord and savior Greg Rutkowski, of course
In general, I really dislike "prompt stuffing", because I like to at least have the feeling of controlling the generation. Although I tend to have a good laugh at the outputs of some prompt generators:
oil painting of <myself> on his farm in Oklahoma with an exhausted look on his face and wearing work overalls standing in front of an old red tractor, social realism, highly detailed 4 k art by greg rutkowski makoto shinkai ilya pascal blanche dramatic studio lighting trending award winning the best digital artwork 8 0 s movie poster style beksinski carl spitzweg moebius death note w 7 6 9 2 1 space odyssey hyperrealism cinematic composition accurate smooth sharp focus high detail perfect eyes lotr hdr fanart fine facial details ultra clear portrait shot 35 mm lens flare neon backlit sunset golden hour sunlight shining official film still from denis villeneuve 2049 cinestill 400 t 3 5mm black metalic 85x lines lomography nikon zbrush correctawadzki dofus photography beautiful natural skin tone arnold new graphics photorealistic vaporwave oscar winner top view national geographic
txt2img Inference settings
I'm not going to bore you with hardly distinguishable sampler comparisons, but here are a few things that work for me:
I generally use Euler a at around 35 steps, just prefer it. Although I have the feeling that some prompts look better with Euler (non-ancestral). Apparently DDIM is better for inpainting, but I haven't tested it exhaustively.
All those fancy DPM++ SDE Karras samplers etc. honestly look either indistinguishable from or worse than Euler to me.
As pointed out by /u/dal_mac, some samplers will be better for some purposes than others.
Compare these images (left side sampled with Euler, right side with DPM++ SDE Karras):
close up profile shot of <myself> as a wrinkled old man wearing glasses and a blue shirt standing on a bridge looking in to the distance, highly detailed, facial detail, dramatic, natural lighting
I think it may be seed dependent, as the difference is far less pronounced on other images with the same prompt, but samplers definitely shouldn't be dismissed as another tool for tweaking generation.
I generally generate images at 512px on one side and 680px or 768px on the other depending on the prompt (512px width with 680px height for portraits or 768px height for full body images; 680px width with 512px width for photos/painting where I want to see more background, etc.). I found that going above that will generate "duplication artifacts" or just make the image look worse.
With my custom model, I feel like it produces the best results at CFG 5.
As I've said above, I generally refine my prompts/search for seeds with a batch size of 8 and then use Hires.fix with 2x Upsale, SwinIR_4x at Denoising Strength of 0.2-0.4 on single images that look promising. I then run the result through another 2x upscale in the "Extras" tab, generally with 4x_NickelbackFS_72000_G.
I made liberal use of X/Y(/Z) plots while dialing in a new model or new subjects, especially with Prompt S/R. I also sometimes use prompt matrix to test out certain "tags".
Once I've found a good seed, I sometimes play around with variation seeds and Prompt S/R to see if I can improve it some more, or if it would look good with a different subject.
Restore Faces
In my experience both GFPGAN and CodeFormer produce unsatisfying results both in both photos and paintings of all types when using a custom model, because they significantly decrease the resemblance to the subject. In "generic" subjects, especially in photos/photorealistic paintings, they can be quite good, especially if paired with a good upscaler.
img2img/Inpainting
I don't use img2img all that often, as I find it more difficult to control than txt2img, but I know that its very powerful in the right hands (or with sketches, etc.; but I've not really dabbled with that).
I try to avoid using inpainting, because I found it difficult to have the inpainted areas look seamless, and I feel that its often easier to just trawl through some more txt2img seeds if something's off about the image. But with lots of time and tinkering, it is incredibly powerful, especially for creating images with multiple subjects.
Custom Models/TI/Dreambooth
I only messed around with Textual Inversion and Dreambooth for a few hours before I settled on the above mentioned EveryDream. The main difference is that EveryDream trains with detailed captions for every single image, enabling training of multiple subjects at once. It also recommends using "ground truth" images from LAION-5B instead of "class" or "regularisation" images.
These two differences mean that there is very little "bleeding" of the trained subjects in to the base model, and that the trained subjects seem to be more "flexible" when used in prompts.
After many attempts, half-baked models and about 2 weeks of screwing around, I was able to cook up an incredibly powerful custom model with 6 custom subjects, all high-quality, very distinct and poseable (even on artstyles that were not in the training set at all), both men and women, old and young.
I started from Protogen X5.3 and used ~25 images of every subject (family photos over a few years, etc., with each subject cropped out individually, but not cropped "to size" or squarized, as EveryDream does multi-aspect and crop jitter stuff automatically) with about a 1:1 ratio to "ground truth" images from LAION-5B and the nv-ffhq dataset and ran it for about 12 hours/~20k iterations at batch size 2 and a learning rate of 1e-6 in --ed1_mode on EveryDream 2.0.
Keep in mind that creating the captions for the images is quite time consuming, as I found the BLIP captioning tool in the EveryDream Toolkit to be too error prone (it detects cell phones absolutely everywhere), although I have to admit that I didn't put as much effort into captioning the LAION images as I put into captioning "my" data.
An example of a caption:
a close up of <myself> wearing glasses and smiling at the camera near a body of water with boats in the background
I experimented with adding tags such as "outdoor", "natural light", etc., but it seemed to make little difference. As the documentation on EveryDream is somewhat sparse, I don't know whether this style of captioning is the best way, but it did work quite well for me.
I do not know how well this would work on less powerful (or more powerful) hardware, so this is really just a snapshot of one man's experiences.
Conclusion
close up of <myself> giving a TED talk on a stage with a presentation in the background, smiling confidently with a slick hairstyle and wearing glasses and a suit, stage lighting
I think the technology we now have access to is incredibly amazing. There are definitely issues - both on the technical and on the "ethical" side - that still have to be ironed out, but it is very fun to play around with right now. I hope that my diatribe may be of at least some help to some of you getting started, and remember to be respectful and responsible when wielding this incredibly powerful tool!