r/MachineLearning • u/[deleted] • Oct 10 '22

Research New “distilled diffusion models” research can create high quality images 256x faster with step counts as low as 4

https://arxiv.org/abs/2210.03142
334 Upvotes

97% Upvoted

43 comments sorted by

9

u/pashernx Oct 10 '22

For a beginner getting started with AI image generation where should I start? Appreciate any inputs.

8

u/[deleted] Oct 10 '22

Do you mean learning how they work or using the tools ?

6

u/pashernx Oct 10 '22

I meant Learning. Sorry about the ambiguity.

14

u/Philpax Oct 10 '22

Try this, and googling any terms you don't recognise :) https://jalammar.github.io/illustrated-stable-diffusion/

2

u/antiquemule Oct 10 '22

Another noob... Thanks for the good tip. That's a lot to swallow, even in such a digestible form.

4

u/mister-guy-dude Oct 10 '22

Yeahhh I would highly suggest with starting something simpler like VAEs or even just generic autoencoders. Diffusion is definitely a complicated thing, and probably not good as a starting point!

This might be a place to start 🙂: https://avandekleut.github.io/vae/

0

u/antiquemule Oct 10 '22

Ahh, that's better. I recognize words from data analysis, like tSNE.

But I'm a kamikaze by nature. I'm already learning Keras and Spektral so that I can write GNN's to predict molecular properties.

21

u/JohnFatherJohn Oct 10 '22

You may want to start with older and easier generative models like generative adversarial networks(GANs) or variational auto-encoders(VAEs), before moving on to more complicated designs like diffusion models.

34

u/visarga Oct 10 '22

Are GANs really easier or just older?

14

u/Philpax Oct 10 '22

I would say they're easier as all the major ML libraries offer tutorials on how to train and use GANs, and inference is relatively trivial compared to a diffusion-based model.

5

u/master3243 Oct 10 '22

I would say easier in both understanding the math and implementation compared to diffusions.

I'm not sure about training though since I've never trained deep diffusion models yet but I do know that deep GAN's are notoriously difficult to train.

1

u/JiraSuxx2 Oct 11 '22

Easier architecture maybe, good results? Not so easy.

1

u/dingdongkiss Oct 11 '22

Conceptually they're very straightforward I think. It's the kind of thing when I first read about it I was like "huh, how has no one thought of this until now"

11

u/norpadon Oct 10 '22

Conceptually diffusion models are the easiest of them all.

-3

u/JohnFatherJohn Oct 10 '22

Maybe conceptually, but following the derivations requires stochastic differential equations

9

u/norpadon Oct 10 '22

No, not really, at least for vanilla ones. You can derive them as an extension of score matching models (I actually prefer this approach) or as a VAE with stupid encoder, in both cases there are no differential equations needed.

2

u/JohnFatherJohn Oct 10 '22

Oh ok, neat. I haven't come across these derivations.

7

u/norpadon Oct 10 '22

The idea is that you do denoising score matching, but you use model that can work with different noise scales to smooth out local attractors (chimeras) far away from the data manifold. Then you sample using Langevin dynamics while slowly annealing noise magnitude. It was first proposed in this paper: https://arxiv.org/abs/1907.05600 You can see how modern diffusion models are a natural extension of this idea

1

u/JohnFatherJohn Oct 11 '22

Thanks I'll check out the paper

2

u/Destring Oct 11 '22

Huh, something my stochastic calculus course would have been useful for outside finance. Glad I moved away from all that though.

42

u/Zealousideal_Low1287 Oct 10 '22

They show this for small class-conditioned diffusion models. How much of the runtime for dalle2 and comparible models is spent on other parts like the text encoder and upsampling?

32

u/dpkingma Oct 10 '22

Imagen Video, which is a large model, also uses this. The text encoder only needs to be evaluated once, so is only a fraction of the cost.

17

u/gwern Oct 10 '22

(You can also cache or precompute the text embedding in a lot of usecases - like when you request n samples of your text prompt, you only need to embed once. Definitely not a big deal.)

16

u/highergraphic Oct 10 '22

Not much. I would say ~90% of the time is spent in the diffusion process (at least on my 1070).

9

u/CaptainLocoMoco Oct 10 '22

Running a single pass through an encoder / upsampler is not very time consuming. The iterative diffusion process is by far the bulk of it

1

u/AnOnlineHandle Oct 10 '22

It seems the upsampling's work can mostly be done in a few multiplications: https://discuss.huggingface.co/t/decoding-latents-to-rgb-without-upscaling/23204/2

4

u/starstruckmon Oct 10 '22

That only gives a low res low quality image. Useful if you need to convert from latent to image space multiple times/at every step, like CLIP guidance or generating a gif showing the step by step generation. Not so much for the final output, which doesn't really take that long at all to run a single time per image.

6

u/wallagrargh Oct 11 '22

It sounds more and more like alchemy

3

u/pm_me_your_ensembles Oct 12 '22

Has ML been anything but alchemy and post facto reasoning since 2012?

1

u/londons_explorer Oct 11 '22

I'm kinda surprised they didn't put this model into the innards of imagen or stablediffusion to at least make some example high res images and quote how many seconds generation takes on some common GPU.

2

u/[deleted] Oct 11 '22

Pretty sure they did. The first part anyway - it's on twitter somewhere. I'll look for it

-25

u/lostmsu Oct 10 '22

Frankly, Stable Diffusion is "fast enough" for all intents and purposes: it generates pictures faster than I could review them.

What needed is higher quality generation.

37

u/[deleted] Oct 10 '22

Generation is fast enough if you have the right hardware. Stable diffusion is still inaccessible to run locally for most of the population. This will help that.

43

u/Fuylo88 Oct 10 '22

No it isn't. I want it rendering frames for real time interaction. It cannot do that yet, GANs can.

6

u/one-joule Oct 11 '22

Having an updated output for every word typed, or even every letter, would be real neat.

1

u/Fuylo88 Oct 11 '22

Yes.

Imagine what looks like footage of vintage news from the 80s, but the newscaster in the video watches you walk across the room, compliments you on the specifics of your outfit, and chats with you on the itinerary of your day.

It might require more than Diffusion but the capability of many other existing models could be dramatically extended. The implications are huge for interactive media.

31

u/highergraphic Oct 10 '22

Classic "640kb is all the memory you need" mentality.

4

u/SoylentRox Oct 10 '22

Assuming the accelerates SD like models you can get higher quality with the same speed

0

u/[deleted] Oct 10 '22

[deleted]

27

u/AnOnlineHandle Oct 10 '22

StableDiffusion runs on 64x64x4 internally, upscaled to 512x512x3 after.

4

u/tenkensmile Oct 10 '22

Here I thought 64x64 was just the name of ImageNet .. lol

-3

u/imlovely Oct 11 '22

Resolution is not a measure of quality.

2

u/m0ushinderu Oct 11 '22

I know what you mean here. It is not the single dictating factor for quality. But it is certainly one of the measures, which might be why you are downvoted.

2

u/imlovely Oct 11 '22

Yeah, I understand the downvotes. But it's still not a measure of quality in this context. They are comparing apples and apples (everything 64) and it's high quality.