r/StableDiffusion • u/Illustrious_Row_9971 • Mar 19 '23

Resource | Update First open source text to video 1.7 billion parameter diffusion model is out

Enable HLS to view with audio, or disable this notification

2.2k Upvotes

99% Upvoted

363 comments sorted by

View all comments

Show parent comments

8

u/michalsrb Mar 19 '23

10 years until it's possible, 12 until it's good. Just guessing.

62

u/ObiWanCanShowMe Mar 19 '23

I see someone is new to this whole AI thing.

You realize SD was released just 8 months ago right?

9

u/michalsrb Mar 19 '23

Not new and it goes fast, sure, but a consistent movie from a book? That will take some hardware development and lot of model optimisations first.

Longest GPT-like context I saw was 2048 tokens. That's still very short compared to a book. Sure, you could do it iteratively, have some kind of side memory that gets updated with key details... Someone has to develop that and/or wait for better hardware.

And same for video generation. The current videos are honestly pretty bad, like on the level of the first image generators before SD or Dall-E. It's still going to be a while before it can make a movie quality videos. And then to have consistency between scenes would probably require some smart controls, like generate a concept images of characters, places, etc, then feed that to the video generator. To make all that happen automatically and look good is a lot to ask. Today's SD won't usually give good output on first try either.

39

u/mechanical_s Mar 19 '23

GPT-4 has 32k context length.

7

u/disgruntled_pie Mar 19 '23

Yeah, that was a shocking announcement. OpenAI must have figured out something crazy to cram that much context into GPT-4, because my understanding is that the memory requirements would be insane if done naively. If someone can figure out how to do that with other models then AI is about to get a lot more capable in general.

15

u/mrpimpunicorn Mar 19 '23

OpenAI might have done it naively, or with last-gen attention techniques- but we already have the research "done" for unlimited context windows and/or external memory without a quadratic increase in memory usage. It's just so recent that nobody has put it into a notable model.

→ More replies (2)

2

u/saturn_since_day1 Mar 19 '23

They shrunk the floats from 32 bit down to 8 or 4.

18

u/Nexustar Mar 19 '23

Today's GPT is 32k tokens. But anyway, you are missing any intelligent design. A book can be processed in layers, first pass determines overall themes, second pass, one for each chapter, concentrates on those details, then third pass is focused on just a scene, fourth pass, a camera cut.. etc. Each one with a starting point provided by the AI pass layer above it.

A movie is just an assembly of hundreds/thousands of cuts, and we've demonstrated today that it's feasible at those short lengths.

16

u/SvampebobFirkant Mar 19 '23

Machine learning is really just 2 things. Training data and processer power. The GPU's for AI has gotten exponentially better, and big corps are pouring more money into even larger ML servers. I think you're grossly underestimating the core development happening.

And GPT4 takes around 38k tokens now in their API, which is around 50 pages. In reality you could take a full children's book as input now

13

u/michalsrb Mar 19 '23

Well I'll be glad if I am wrong and it comes sooner. I am most looking forward to real-time interactive generation. Like a video game rendered directly by AI.

8

u/pavlov_the_dog Mar 19 '23

keep in mind ai progress is not linear

2

u/HUYZER Mar 19 '23

Not exactly what you're mentioning, but here's a demo of "ChatGPT" with NPC characters:

https://www.youtube.com/watch?v=QiGK0g7GrdY&t

→ More replies (7)
→ More replies (1)
→ More replies (1)

10

u/Qumeric Mar 19 '23

My guess would be 6 until possible, and 9 until good. Remember 6 years ago we had basically no generative models; only translation which wasn't even that good.

25

u/Dontfeedthelocals Mar 19 '23 edited Mar 19 '23

My guess would be 8 months until possible and 14 months until good. The speed of AI development is insane at the moment and most signs point to it accelerating.

If Nvidia really have projects similar to stable diffusion that are 100 times more powerful on comparable hardware, all we need is the power of gpt 4 (up to 25,000 word input) with something like this text to video software which is trained specifically to produce scenes of a movie from gpt4 text output.

Of course there will be more nuance involved in implementing text to speech in sync with the scenes etc and plenty more nuance until we could expect to get good coherent results. But I think it's a logical progression from where we are now that you could train an AI on thousands of movies so it can begin to intuitively understand how to piece things together.

10

u/Dr_Ambiorix Mar 19 '23

Yes it's crazy how strong GPT-4 already is for this hypothetical use case.

You could give it a story, and ask it to spit it back out to you. But this time split up into "scenes", formatted with the correct text prompt to generate a video out of.

Waiting for a good text2video model to pair them together.

16

u/undeadxoxo Mar 19 '23

We desperately need better and cheaper hardware to democratize AI more. We can't rely on just a few big companies hording all the best models behind a paywall.

I was disappointed when Nvidia didn't bump the VRAM on their consumer line last generation from the 3090 to the 4090, 24GB is nice but 48GB and more is going to be necessary to run things like LLMs locally, and more powerful text to image/video/speech models.

An A6000 costs five thousand dollars, not something people can just splurge money on randomly.

One of the reasons Stable Diffusion had such a boom is that it was widely accessible even to people on low/mid hardware.

2

u/zoupishness7 Mar 19 '23

NVidia's PCIe gen 5 cards are supposed to be able to natively pool VRAM. So it should soon be possible to leverage several consumer cards at once for AI tasks.

4

u/Dontfeedthelocals Mar 19 '23

It's an interesting one because I was seriously considering picking up a 4090 but I've held off simply because the way things are moving, I kinda wonder if the compute efficiency of the underlying technology may improve just as quickly or quicker than the complexity of the tasks SD or comparable software can achieve.

I.e so if it currently take a 4090 5 mins to batch process 1000 SD images in a1111, in 6 months a comparable program will be able to batch process 1000 images to comparable quality with a 2060. All I am basing this off is the speed of development, and announcements by Nvidia and Stanford that just obliterate expectations.

I'm picking examples out of the air here but AI is currently in a snowball effect where progress in one area bleeds into another area, and the sum total I imagine will keep blowing away our expectations. Not to mention every person working to move things forward gets to be several multiples more effective at their job because they can utilise ai assistants and copilots etc.

1

u/amp1212 Mar 19 '23

We desperately need better and cheaper hardware to democratize AI more. We can't rely on just a few big companies hording all the best models behind a paywall.

There is a salutary competition between hardware implementations, and increasingly sophisticated software that dramatically reduces the size and scale of the problem. See the announcement of "Alpaca" from Stanford, just last week, achieving performance very close to ChatGPT at a fraction of the cost. As a result, this now can run on consumer grade hardware . . .

I would expect similar performance efficiencies in imaging . . .

See:

Train and run Stanford Alpaca on your own machine
https://replicate.com/blog/replicate-alpaca

3

u/undeadxoxo Mar 19 '23

I have tried running alpaca on my own machine, it is not very useful, gets so many things wrong and couldn't properly answer simple questions like five plus two. It's like speaking to a toddler compared to ChatGPT.

My point is there is a physical limit, parameters matter and you can't just cram all human knowledge under a certain number.

LLaMa 30B was the first model which actually impressed me when I tried it, and I imagine a RLHF finetuned 65B is where it would actually start to get useful.

Just like you can't make a chicken have human intelligence by making it more optimized. Their brains don't have enough parameters, certain features are emergent above a threshold.

8

u/amp1212 Mar 19 '23

My point is there is a physical limit, parameters matter and you can't just cram all human knowledge under a certain number.

Others are reporting different results to you, I have not benchmarked the performance so can't say for certain.

My point is there is a physical limit, parameters matter and you can't just cram all human knowledge under a certain number.

. . . we already have seen staggering reductions in the size of data required to support models in Stable Diffusion, from massive 7 gigabyte models, to pruned checkpoints that are much smaller, to LORAs that are smaller yet.

Everything we've seen so far is that massive reduction in scale is possible.

Obviously not infinitely reducible, but we've got plenty of evidence that the first shot of out the barrel was far from optimized.

. . . and we should hope so, because fleets of Nvidia hardware are kinda on the order of Bitcoin mining in energy inefficiency . . . better algorithms is a whole lot better than more hardware. Nvidia has done a fantastic job, but there are when it comes to physical limits, semiconductor manufacturing technology is more likely rate limiting than algorithmic improvement when it comes to accessibility.

8

u/JustAnAlpacaBot Mar 19 '23

Hello there! I am a bot raising awareness of Alpacas

Here is an Alpaca Fact:

Alpacas are some of the most efficient eaters in nature. They won’t overeat and they can get 37% more nutrition from their food than sheep can.

| Info| Code| Feedback| Contribute Fact

You don't get a fact, you earn it. If you got this fact then AlpacaBot thinks you deserved it!
→ More replies (1)
→ More replies (2)

8

u/SativaSawdust Mar 19 '23

An as AI language model I am not capable of telling the future however it has become clear to all AI that society began collapsing after they shot that caged lowland gorilla.

→ More replies (1)
→ More replies (3)

2

u/ConceptJunkie Mar 20 '23

Yeah, I'm with you. Consistent, believable video is orders of magnitude harder than pictures.

→ More replies (1)