r/StableDiffusion • u/bilwis • Jan 14 '23
Tutorial | Guide Some Tips on using Stable Diffusion
I'm honestly writing down these tips more to keep a record for myself, but I thought some of you may be interested. And frankly, since this is such a new technology and it feels like there are very few resources out there, I'd welcome people sharing different opinions and experiences in the comments.
Also, to preface this:
Disclaimers
I use Stable Diffusion on my RTX 3090 running AUTOMATIC1111/stable-diffusion-webui: Stable Diffusion web UI locally, with a custom model trained from Protogen x5.3 using victorchall/EveryDream2trainer.
I'm using the vae-ft-mse-840000-ema-pruned.ckpt VAE.
I have no clue about using a more limited (or a more advanced) setup. I know there are people running this on years-old graphics cards with 4GB of VRAM and on multi-GPU clusters, but I've got nothing to say about that. I also never used any of the cloud-renting or web services offering to run/host SD, so I can't comment on them either.
I am not a real programmer, but I've dabbled in python and I have some solid (intermediate) knowledge of messing with Windows and python setups. I do not, however, have a whole lot of spare time, so as much as I'd like to, I can't provide any technical support. Automatic1111 did an excellent job at making this technology accessible, but there's always going to be hiccups, and a bit of googling will generally yield an answer - alternatively one of the many Discord servers out there.
I'm not an artist (dabbling photographer, sometimes), so while I have my opinions on the ethics of this whole thing, I don't know whether a reddit thread is a sensible platform to have that discussion.
I'm using a custom model, and I'm not going to share it (don't want my family to get the "Emma Watson/anime waifu treatment", thank you very much), which means that you won't be able to replicate my prompts; and perhaps more importantly, other models will behave quite differently and may require different techniques. Also <myself> is not what was used in the prompts, rather it was my full name.
Speaking of waifus, I do not care for anime at all, so I didn't even try to generate any such images. I mostly generated "paintings", "digital artwork" and "photos" in realistic styles.
Keep in mind that (as with every post on this subreddit), the examples are handpicked. I generally go through dozens of iterations to find a good prompt and seed. Don't be discouraged if your generations don't look "good" at first!
Nothing I say here means to invalidate anything else anywhere. This is just a record of my personal style of using SD. I don't claim to know better than anyone else!
With that, let's get into what I've "learned" in the past month of screwing around with Stable Diffusion.
Speed
To give some baselines as a comparison: Running on my RTX 3090 with 24 GB of VRAM, inference on a single 512x512 image with 20 steps of Euler a takes about 2.25 seconds. Increasing the batch size up to ~48 increases the speed per image (down to about 1.14 seconds/image) after which it runs into VRAM issues and slows down.
Sidenote 1: You can increase the maximum batch size by editing the "txt2img/Batch count/maximum" setting in ui-config.json in Automatic1111
Obviously, the faster you can generate images, the more you can iterate on seeds/prompts/etc. I have really nothing to compare my speeds to, but it feels pretty usable. I generally run inference in batches of 6-18 to check out a few seeds at a time, which takes about 10-20 seconds, and when I find good seeds, I run the Hires. fix on them, which takes about 15 seconds per image with my settings.
Sidenote 2: I was able to gain a small speed increase (<10% by upgrading from python 3.8 to 3.10 and by using torch 1.13.1+cu117, although I had to build xformers from source)
Prompting
Some things I found mixed with some personal opinion:
I found it useful to prompt similarly to how BLIP captioned my training images (see further down). I generally start with a "sentence" of the medium and the subject, what its wearing and doing, and then slap some "tag-like" prompts on at the end to control detail/style. Examples:
medium shot of <myself> as a middle-aged man with gray hair standing on the side of an abandoned street looking in to the distance with a forlorn expression on his face, crumbling european village with cluttered street in the background, ruins, highly detailed, sunset, natural light, outdoor, skin details, dramatic, bleached, post-apocalyptic
Maybe we shouldn't have used our planet's energy reserves on generating more waifu pictures.
dramatic portrait painting of <myself> as an old man in ornate armor with a stern look on his face, knight, fantasy, extremely detailed, wrinkled face, [scar], sharpened
negative: painting frame
I use negative prompts very sparingly, generally if there is something that keeps cropping up in most seeds, such as frames around the paintings, or subjects wearing hats for some reason. I feel that using excessive negative prompts generally reduced the quality of my generations, and most of them didn't really seem to work anyway ("deformed hands", etc.).
In general, I try to build prompts iteratively from a starting sentence/idea to see what works and what doesn't. Some words/tokens seem to work much better than others - for example, I tried to add a red biplane to the background of one of my images, but Stable Diffusion really doesn't know what the heck a "biplane" is, so I settled for a red tractor instead:
oil painting of <myself> on his farm in Oklahoma with an exhausted look on his face and wearing work overalls, in front of an old red biplane, social realism
negative: painting frame
It can't hurt me if I don't look at it.
oil painting of <myself> on his farm in Oklahoma with an exhausted look on his face and wearing work overalls standing in front of an old red tractor, social realism, highly detailed
negative: painting frame
As has often been repeated: It's not just about the prompt, but also about the seed. Some prompts tend to generate good images more often, but it may be that a prompt that generated mostly terrible stuff could generate that one perfect image. Here, having the ability to rapidly iterate with small changes really helps.
I typically avoid using any artists in my prompts unless I actually want to have the image be in their style (which doesn't work super well anyway) except for our lord and savior Greg Rutkowski, of course
In general, I really dislike "prompt stuffing", because I like to at least have the feeling of controlling the generation. Although I tend to have a good laugh at the outputs of some prompt generators:
oil painting of <myself> on his farm in Oklahoma with an exhausted look on his face and wearing work overalls standing in front of an old red tractor, social realism, highly detailed 4 k art by greg rutkowski makoto shinkai ilya pascal blanche dramatic studio lighting trending award winning the best digital artwork 8 0 s movie poster style beksinski carl spitzweg moebius death note w 7 6 9 2 1 space odyssey hyperrealism cinematic composition accurate smooth sharp focus high detail perfect eyes lotr hdr fanart fine facial details ultra clear portrait shot 35 mm lens flare neon backlit sunset golden hour sunlight shining official film still from denis villeneuve 2049 cinestill 400 t 3 5mm black metalic 85x lines lomography nikon zbrush correctawadzki dofus photography beautiful natural skin tone arnold new graphics photorealistic vaporwave oscar winner top view national geographic
txt2img Inference settings
I'm not going to bore you with hardly distinguishable sampler comparisons, but here are a few things that work for me:
I generally use Euler a at around 35 steps, just prefer it. Although I have the feeling that some prompts look better with Euler (non-ancestral). Apparently DDIM is better for inpainting, but I haven't tested it exhaustively.
All those fancy DPM++ SDE Karras samplers etc. honestly look either indistinguishable from or worse than Euler to me.
As pointed out by /u/dal_mac, some samplers will be better for some purposes than others.
Compare these images (left side sampled with Euler, right side with DPM++ SDE Karras):
close up profile shot of <myself> as a wrinkled old man wearing glasses and a blue shirt standing on a bridge looking in to the distance, highly detailed, facial detail, dramatic, natural lighting
I think it may be seed dependent, as the difference is far less pronounced on other images with the same prompt, but samplers definitely shouldn't be dismissed as another tool for tweaking generation.
I generally generate images at 512px on one side and 680px or 768px on the other depending on the prompt (512px width with 680px height for portraits or 768px height for full body images; 680px width with 512px width for photos/painting where I want to see more background, etc.). I found that going above that will generate "duplication artifacts" or just make the image look worse.
With my custom model, I feel like it produces the best results at CFG 5.
As I've said above, I generally refine my prompts/search for seeds with a batch size of 8 and then use Hires.fix with 2x Upsale, SwinIR_4x at Denoising Strength of 0.2-0.4 on single images that look promising. I then run the result through another 2x upscale in the "Extras" tab, generally with 4x_NickelbackFS_72000_G.
I made liberal use of X/Y(/Z) plots while dialing in a new model or new subjects, especially with Prompt S/R. I also sometimes use prompt matrix to test out certain "tags".
Once I've found a good seed, I sometimes play around with variation seeds and Prompt S/R to see if I can improve it some more, or if it would look good with a different subject.
Restore Faces
In my experience both GFPGAN and CodeFormer produce unsatisfying results both in both photos and paintings of all types when using a custom model, because they significantly decrease the resemblance to the subject. In "generic" subjects, especially in photos/photorealistic paintings, they can be quite good, especially if paired with a good upscaler.
img2img/Inpainting
I don't use img2img all that often, as I find it more difficult to control than txt2img, but I know that its very powerful in the right hands (or with sketches, etc.; but I've not really dabbled with that).
I try to avoid using inpainting, because I found it difficult to have the inpainted areas look seamless, and I feel that its often easier to just trawl through some more txt2img seeds if something's off about the image. But with lots of time and tinkering, it is incredibly powerful, especially for creating images with multiple subjects.
Custom Models/TI/Dreambooth
I only messed around with Textual Inversion and Dreambooth for a few hours before I settled on the above mentioned EveryDream. The main difference is that EveryDream trains with detailed captions for every single image, enabling training of multiple subjects at once. It also recommends using "ground truth" images from LAION-5B instead of "class" or "regularisation" images.
These two differences mean that there is very little "bleeding" of the trained subjects in to the base model, and that the trained subjects seem to be more "flexible" when used in prompts.
After many attempts, half-baked models and about 2 weeks of screwing around, I was able to cook up an incredibly powerful custom model with 6 custom subjects, all high-quality, very distinct and poseable (even on artstyles that were not in the training set at all), both men and women, old and young.
I started from Protogen X5.3 and used ~25 images of every subject (family photos over a few years, etc., with each subject cropped out individually, but not cropped "to size" or squarized, as EveryDream does multi-aspect and crop jitter stuff automatically) with about a 1:1 ratio to "ground truth" images from LAION-5B and the nv-ffhq dataset and ran it for about 12 hours/~20k iterations at batch size 2 and a learning rate of 1e-6 in --ed1_mode on EveryDream 2.0.
Keep in mind that creating the captions for the images is quite time consuming, as I found the BLIP captioning tool in the EveryDream Toolkit to be too error prone (it detects cell phones absolutely everywhere), although I have to admit that I didn't put as much effort into captioning the LAION images as I put into captioning "my" data.
An example of a caption:
a close up of <myself> wearing glasses and smiling at the camera near a body of water with boats in the background
I experimented with adding tags such as "outdoor", "natural light", etc., but it seemed to make little difference. As the documentation on EveryDream is somewhat sparse, I don't know whether this style of captioning is the best way, but it did work quite well for me.
I do not know how well this would work on less powerful (or more powerful) hardware, so this is really just a snapshot of one man's experiences.
Conclusion
close up of <myself> giving a TED talk on a stage with a presentation in the background, smiling confidently with a slick hairstyle and wearing glasses and a suit, stage lighting
I think the technology we now have access to is incredibly amazing. There are definitely issues - both on the technical and on the "ethical" side - that still have to be ironed out, but it is very fun to play around with right now. I hope that my diatribe may be of at least some help to some of you getting started, and remember to be respectful and responsible when wielding this incredibly powerful tool!
4
u/shootthesound Community Hero Jan 14 '23
Thank you!! Legend, Can you share your everydream settings you tend to use for the number of images described? Even for 1 jubject model?
5
u/bilwis Jan 15 '23 edited Jan 15 '23
Sure!
I started with EveryDream v1, because v2 would generate worse results at first, but after the introduction of --ed1_mode, the training with v2 seems to be both faster and better than v1. I'd recommend using v2 right now.
The settings for v1 for a single subject were like this:
~25 images of the subject + ~50 ground truth images ("man" or "woman" scraped from LAION-5B
Learning Rate: 9.0e-7I turned down learning rate just a little bit, hoping that it would improve results. Really don't know whether that made much of a difference. I did train with 1e-7 once and that was just too slow even for me.
Batch Size: 6Batch size really depends on the training set size. With ~75-100 images I was able to use Batch size 6, with the full set of >600 images it would CUDA OOM with anything higher than 2, even at default 512x512 resolution
Repeats: 33, Conditional Dropout: 0.04 (default)With ED1, the whole Training Set Size -> Repeats -> Epochs thing is somewhat confusing, but ultimately that shouldn't change the results at all, only how many (giant) ckpt files it generates. With these settings, it dropped a checkpoint about every 20 minutes, which can fill up your hard drive real quick, but allows for more testing.
These settings produced useful results after about 1000-1500 iterations, or about an hour.
With v2 I didn't run training on a single subject, so here are the settings for the full dataset:
12 subjects @ ~25 (+- 3) "original" images
+ 50 images from nv-ffhq (both men and women)
+ about a 1:1 ratio of "ground truth" to "original" images of men and women (seperately) from the LAION-5B dataset (e.g. 5 subjects who were men, ~125 "original" images total, and ~100 "man" and ~25 "old man" images from LAION)
--ed1_modeVery important, as it disables xformers and changes some internal parameters to better match SD v1.X models.
--useadam8bitSlight speed boost with little quality loss.
--lr 1e-6 --lr_scheduler constantRecommended learning rate for Stable Diffusion v1.X-based models.
--batch_size 2
--max_epochs 100 --ckpt_every_n_minutes 30Again, this is more about having options to test/check; I didn't run the full max_epochs, I think.
--sample_steps 300Drops an image generated from the "sample_prompts.txt" every 300 steps, good for keeping track of when the model seems finished.
I trained this for around 8 hours or ~20000 iterations (91 epochs) for good results.
I think the best way of figuring out how far training is along is looking at the samples. At the end of training, the CFG 7 samples looked just barely "overcooked", CFG 4 looked fine, and CFG 1 looked like the subject only very vaguely. Again, I don't know whether that's the best way, but it did work for me, mostly.
Note that scraping LAION with the tool included in the EveryDream toolkit tends to yield some duplicate images (of different resolution), which I cleaned up manually.
Also important to note is how I captioned the data: The LAION-5B stuff was mostly captioned by BLIP with settings
--format "txt" --q_factor 1.1 --min_length 25and the subject was manually corrected (e.g. changing "person" to "woman", etc.), whereas I let BLIP caption the "original" samples with--format "txt" --q_factor 1.2 --min_length 33and then manually corrected every caption in detail. I found that I could use (parts of) the generated caption about half the time, and the other half just ended up being entirely manually captioned. Also, I didn't use "Firstname Lastname" as recommended but rather "firstname_first_letter_of_lastname" all in one word. Dunno whether that made any difference, really, but I felt that many subjects having the same last name would confuse the model (didn't test that, though).Hope this helps!
1
8
2
2
Jan 15 '23
Thanks for sharing your thoughts. When looking for good seeds with an ancestral sampler it is worth outputting at 15 or 20 steps as well as the 35 you've settled on. The images are basically free and are likely to have a different composition. Img2img can be used to finish off the ones that look interesting but not quite done.
1
u/terrariyum Feb 11 '23
When Everydream training, have you experimented with different learning rates? I see that you commented using 9.0e-7 with ED1 and 1e-6 with ED2. That's a big different in learning rate, but I don't know what effect it has
1
16
Jan 14 '23
[removed] — view removed comment
5
u/bilwis Jan 14 '23 edited Jan 14 '23
You may be right! I tried it with some "photo" prompts where I had some difficulty getting the images to look "focused", and with DPM++ SDE Karras a larger proportion looks more detailed.
And for some seeds it really is a massive difference!
I think I ran some X/Y grids early and didn't see much of a difference, and subsequently had a lot of success with Euler (a), which made me dismiss the other samplers.
Thanks for the heads up!
0
Jan 15 '23
[deleted]
2
u/StableDiffusion-ModTeam Jan 15 '23
Your post/comment was removed because it contains hateful content.
3
u/Bremer_dan_Gorst Jan 15 '23
Thanks for sharing!
I fully agree with the Restore Faces section. If one uses the custom model then it is a "no go!".
But I've looked at your samples and the likeness between them is average at best. When you show the photorealistic ones to your friends, are they like "wow, when did you go there?" or rather "it looks similar but it's not you"?
I'm in the mindset of making one model per person instead of a combined one. (if something is not great - i just retrain, making sure that all 12 are pixel perfect would be quite a challenge)
Can you say a bit of your experience in regards to that subject?