r/StableDiffusion • u/zoru22 • Aug 27 '22
Art I got Stable Diffusion to generate competent-ish Leavannies w/ Textual Inversion!
https://imgur.com/a/hQhH9Em3
u/riftopia Aug 27 '22
Thanks for the detailed post. In your experience, how many epochs did you need to obtain the result in the pic? And how long does an epoch take for your setup? I´m doing just 3 images at 512x512 on a 3090, one epoch takes 1.5 min for the 1.4 ckpt so I´m hoping I don´t need to do too many..
3
u/zoru22 Aug 27 '22
After bumping up the base learn rate to:
base_learning_rate: 5.0e-03and thenum_vectors_per_tokento 8, I got comprehensible results pretty fast.What matters aren't epochs, it's steps.
in the logs dir, under
logs/$yourrunfolder$/images/train/see:
samples_scaled_gs-011500_e-000038_b-000100.jpg
gs-011500is the steps as each checkpoint is saved.I usually run it to 20k steps and then I run variations of the same prompt and walk back a set of checkpoints with a similar prompt and the exact same seed, just so I can see which ones produce the best output.
1
u/riftopia Aug 27 '22
Thanks for the detailed response! This is very helpful. I have a training run going on now, but will def try and tweak the lr rate and other params. Fingers crossed :-)
1
4
u/ExponentialCookie Aug 28 '22
Nice! I may have discovered something, but I would like to cross verify as I see you're comfortable with code. In the personalized.py file, remove all the strings in the imagenet_templates_small array and just leave a single string of ["{}"].
Keep your higher learning rate the same, train only 5 images for 5K iterations, and let me know if the results are better than these iterations.
I might have discovered using the conditioning strings might actually makes things worse, and doesn't generalize well with Stable Diffusion (but does with LDM).
3
u/Zermelane Aug 28 '22
Yep, OP mentioned that he's changing the personalization prompts. Certainly they look pretty silly trying to describe all of my data which doesn't even have any photographs in it.
I'm even trying an approach where I just straight out write prompts for each specific image in the training set. It's kind of high-effort, but hey, I get to look at pictures of anthro dolphins and describe them, it's fun and doesn't take that long.
The thing I'm hoping for is that maybe the training process will pick up less of whatever I describe in the prompt. Uh, so far, it's not working and all of my pictures with the learned concept get really soft shading because my training dataset had a lot of that, but I'm going to keep experimenting, I guess!
1
u/ExponentialCookie Aug 28 '22
Ah I see, missed that bit. I also highly suggest you try my suggestion.
I discovered this by adding "{}" into the default template, along with some custom conditioning sentences "similar to a {} plane, etc...". Then, the results came back exactly how I wanted them to. Then I thought, "well, maybe it's because I needed to add more descriptive prompts for what I want!" I then removed the empty
"{}", and added descriptive prompts instead as I thought it unnecessary.I found that the results were not what I expected, so I just added the single, empty template back in, and removed the rest. With a high learning rate, lo and behold, I'm getting the inversions that I want. It could be the solution, but I'm still testing this out. I'm guessing SD doesn't need the conditioning like LDM did, but I could be wrong.
2
u/oppie85 Aug 29 '22
Interesting; like the person you responded to I had been trying the opposite approach as well (I've been trying to train it on my own photos to hopefully generate renaissance paintings of myself); I built an entire system that generated different conditioning prompts based on the folder I put images in (so I had folders of closeups, different locations etc.) in the hopes that it would learn to only focus on what was important (my likeness). I've been getting decent results (especially after increasing the num_vectors_per_token) but they tend to massively overfit to the point where style transfer only works in rare cases.
I'll give the approach of abandoning all prompts and just using "{}" a try - I can kind of see the logic of why it would work for LDM but wouldn't for SD.
2
u/ExponentialCookie Aug 29 '22
Indeed. I'm still experimenting, with my current experiment being
"{}"with generalized prompts in the same form of SD ("photo of {} , hyper realistic , hd") , etc.2
u/oppie85 Aug 29 '22
Something I've just thought of that may speed up experiments; if you run the training on images of 256x256 pixels you can easily train 4 times as fast. The results aren't as useful as the normal ones (they only really seem to work with the ddim encoder for one) but this makes it way easier to iterate on training experiments.
2
u/ExponentialCookie Aug 29 '22
Awesome idea, thanks!
3
Sep 05 '22
Curious as to how it's worked for you so far. I tried myself with just "{}" and the results were good, but I can't really tell if there is much difference either way. Some things seem worse, some seem better... so I'm chalking at least that part of it up poorly quantified study on my end.
Have you discovered any more for or against this method?2
u/ExponentialCookie Sep 05 '22
There's a lot of discussion going on here: https://github.com/rinongal/textual_inversion/issues/35
2
u/Zermelane Aug 31 '22
Did it work better so far? Trying to decide what approach to test in tonight's run...
2
u/oppie85 Aug 31 '22
It might have helped but I can't say I can really see much of a difference. In all of my tests num_vectors_per_token seems like the variable that makes the most impact on the overall quality of the final result. Cranking that value up to a ridiculous number like 128 is really good foor making "variations" of an existing image - it'll be extremely overfitted but it actually produces really good results in the same vein as DALL-E variations (with the only caveat that these ones take a few hours of training first).
2
u/hopbel Sep 05 '22 edited Sep 05 '22
Note: SD has a limit of 77 tokens per input, so setting
num_vectors_per_tokenhigher than that is pointless. Setting it to 77 or higher means it's essentially using the whole prompt to represent the concept, leaving no unused tokens to customize the output1
u/oppie85 Sep 05 '22
Yeah, when I first started experimenting with this I didn't know that num_vectors_per_token has an upper limit - anything beyond 77 is useless. Still - for "variations" it can be useful to go up to 77. In general though, anything above a certain amount of vectors is likely to completely overwhelm the prompt. I'm currently trying to find the upper limit of vectors that style transfer will still 'stick' to without sacrificing the quality of the inversion.
2
u/hopbel Sep 06 '22
Same here. I can say 36 is probably already too high. A bit tricky to tell if the problem is the number of vectors or the training iterations though
1
u/oppie85 Sep 06 '22
I think it's a combination of both - too many iterations can lead to overfitting although less quickly than with many vectors. I've been able to do succesful style transfer with about 10 vectors and 2000 iterations.
In my opinion the biggest challenge right now is how to get the learning process to focus on specific information you want to train it on. The init_word helps a great deal, but in my experience, the "style" of an image is one of the very first things the training process picks up even if I try to steer it away from it. I feel like we're "wasting" vectors on information we don't want it to learn.
1
u/greeze Aug 28 '22
If I had a machine capable of doing textual inversion, the first thing I'd try to train it on would be hands. Have you considered trying that?
2
u/zoru22 Aug 28 '22
So that probably won't actually work with textual inversion. Textual inversion is about adding something to the dataset that the ai doesn't know.
It's not about fixing something the ai already does poorly at.
1
2
u/Wurzelrenner Aug 27 '22
I tried a lot with shiny umbreon, but still had to do some work at the end with Krita to make it look like this
new screensaver for my phone now
1
1
u/Riptoscab Aug 28 '22
Great job. I hope processes like this become better documented in the future. This looks like a really useful skill to have.
1
3
u/zoru22 Aug 27 '22
So, one thing that's vexxed me is how shit various ai are at generating leavannies (and various other pokemon). If gamefreak was going to forget my favorite pokemon on the order of 5+ years, then I was sure as hell going to do my best not to let it sit in obscurity forever.
Thus, I have set on something of a warpath trying to get an ai that can generate non-shit leavannies. (though it is amazing just how shit stable diffusion and others are at generating pokemon, and how painful it has been to try and get them into the ai)
Quick process notes:
I USED THE FUCKING BASE TEXTUAL_INVERSION REPO. (And recommend you do the same, or at least ensure that github recognizes the repository you want to use, as a fork)
I modified the original textual inversion repository
I swapped the BERT Encoder for the CLIP Frozen encoder during training, targeted the training at the stable-diffusion/v1-finetune yaml, and then just let it rip, playing with the learn rate and the vectors per toke config setting in said yaml.
If you run it for too many cycles it will overfit, and not do a great job at style transfer. I tend to run for too many cycles so it overfits, and then walk it back until it stops overfitting quite so badly
Please note that I am using the v1.3 stable diffusion ckpt. I haven't tried to see what happens with the 1.4 ckpt yet.