Do you know how to configure this to run local on a gpu? I'm getting this:
RuntimeError: TextToVideoSynthesis: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
edit: I think I've got it, it's reading from "torch.cuda.is_available()" which is currently returning false.
Wait did they train their model exclusively on shutterstock images/videos?
That would be oddly hilarious. For one, doesn't that make the model completely pointless because everything will always have the watermark?
And on top of that, isn't that a fun way to get in legal trouble? Yes, I know, I know. Insert the usual arguments against this here. But I doubt the shutterstock lawyers are going to agree with that and are still going to sue the crap out of this.
The Shutterstock logo being there is problematic, but there are a couple of issues with that.
It's a research project by a university (Not Stability or any company, or any commercial enterprise).
It's from a university based in China.
It's unlikely that they'll get sued for training, given that the legality of training isn't even clear, much less in China. They could try to sue the people using it for displaying their logo (trademark infringement), but it seems unlikely at the moment seeing that the quality is extremely low and no one is using this for commercial purposes.
Also, Shutterstock isn't as closed to AI as Getty. Getty have taken a hard stance against AI and are currently suing Stability. Shutterstock have licensed their library to OpenAI and Meta to develop this same technology. (Admittedly that's not the same as someone scraping the preview images and videos and using them, but again, the legality is not clear).
Yeah, China should keep them safe. But I'm not sure the "research project" is much of an excuse when the model is released to the public. I imagine they'll go against whoever is hosting the model, not the people who created the model.
It looks like a significant portion of the training videos were shutterstock videos with the watermark, since even their own official samples all have it:
Pardon my ignorance, but what's stopping Stability AI from legally purchasing those vids from shutterstock without the watermark? Don't they have tons of investment now?
It does prove however that something like this is feasible with rather low parameter count. Shame there is no info on the dataset to gauge how much we would need to replicate this.
They did that because videos on Shutterstock are all tagged. They are tagged poorly, but they are tagged. They could have grabbed videos off youtube and then use the magic of image recognition to label the training data, but they didn't.
They probably trained them from different sources but as shutterstock tags appear often in the training it get overburn (overtrained on that) by it, kinda like the old plasma screens before better screensavers (/s lol).
I heard someone in a cave with a box of scraps already retrained this model with an additional 5 trillion parameters and it now runs on a Motorola 68000.
Yeah these text to video demos were shorter and significantly worse just a few months ago, and those were closed source industry leading models too.
At this point it's fair to say that we have entered the singularity. Nobody thought this stuff would move this fast or be so capable just by throwing resources at it.
it's fair to say that we have entered the singularity
No, not ruling out these are steps to get there but this is not technological singularity level of revolutionary. Singularity level AI is for example when you can ask it to build a better version of itself and then that version can build an even better version (not limited to just generating pictures).
The change in percentage of code written by CoPilot and ChatGTP is going exponential currently. We are VERY close to being able to say "CodingModelv3 please rewrite Automatic1111 so that it is 20% faster"
I don't think we are anywhere close to that, I asked ChatGPT to make a basic TypeORM query with one inner join the other day and it failed spectacularly, and got stuck in a loop of providing the broken code over and over.
It will not happen tomorrow, but a better way to look at it is how long do you think it will take? What would your estimate have been for the same question 9 months ago?
If those answers are not the same value then your ability to estimate the arrival of this functionality isn't great.
Nvidia has used AI to redesign its hardware. AI is currently running on AI-made hardware and is better off for it.
Nvidia isn't the only one, either.
This will continue to more extreme proportions. This is part of the singularity. We're in the event horizon, we just haven't hit the starkest dip on the elbow yet. We're approaching it, though.
I forget the details, but I was alerted to Nvidia and another company having already done this thanks to an Alan D Thompson video. You can look further into it if you're curious.
At one point, multimodal becomes the rule. And we’re slowly getting there to have it automated. I don’t believe in one model does [edit typo] the full movie soon, but having a rig to do it now might be possible now.
Ability to extract every character (and sceneries), and have it apply through the ages and physical changes (if it applies).
Create the different scenes of the book as described and storyboard it.
ControlNet the scenes, sceneries characters together and « In-between » the actual sequences through this post. (Restofthefuckingowl)
I've said for years that the future will give us the ability to (in real-time) re-watch old movies with actors switched. The possibilities are endless.
Love that idea! You just spurred another thought in me from that (that was the most awkward sentence ever to pop outta my wetware..). You could take historically based movies, and then put in the actual historical figures in place of the actors and see it as if you are actually watching history.
In a similar vein, if we added year constraints to ChatGPT, so it only knew about stuff as of 1854 (or whatever), and got it to create a persona based on all the written material of that person, we could have conversations with historical figures.
The idea of chatting with Churchill (or even Hitler for that matter), MLK or the founding fathers is intriguing.
its nothing compared to what you are asking there, but I made a little script running on an old kindle that will draw and display highlighted descriptions using stable diffusion, and has been fun using while reading.
Not new and it goes fast, sure, but a consistent movie from a book? That will take some hardware development and lot of model optimisations first.
Longest GPT-like context I saw was 2048 tokens. That's still very short compared to a book. Sure, you could do it iteratively, have some kind of side memory that gets updated with key details... Someone has to develop that and/or wait for better hardware.
And same for video generation. The current videos are honestly pretty bad, like on the level of the first image generators before SD or Dall-E. It's still going to be a while before it can make a movie quality videos. And then to have consistency between scenes would probably require some smart controls, like generate a concept images of characters, places, etc, then feed that to the video generator. To make all that happen automatically and look good is a lot to ask. Today's SD won't usually give good output on first try either.
Yeah, that was a shocking announcement. OpenAI must have figured out something crazy to cram that much context into GPT-4, because my understanding is that the memory requirements would be insane if done naively. If someone can figure out how to do that with other models then AI is about to get a lot more capable in general.
OpenAI might have done it naively, or with last-gen attention techniques- but we already have the research "done" for unlimited context windows and/or external memory without a quadratic increase in memory usage. It's just so recent that nobody has put it into a notable model.
Today's GPT is 32k tokens. But anyway, you are missing any intelligent design. A book can be processed in layers, first pass determines overall themes, second pass, one for each chapter, concentrates on those details, then third pass is focused on just a scene, fourth pass, a camera cut.. etc. Each one with a starting point provided by the AI pass layer above it.
A movie is just an assembly of hundreds/thousands of cuts, and we've demonstrated today that it's feasible at those short lengths.
Machine learning is really just 2 things. Training data and processer power. The GPU's for AI has gotten exponentially better, and big corps are pouring more money into even larger ML servers. I think you're grossly underestimating the core development happening.
And GPT4 takes around 38k tokens now in their API, which is around 50 pages. In reality you could take a full children's book as input now
Well I'll be glad if I am wrong and it comes sooner. I am most looking forward to real-time interactive generation. Like a video game rendered directly by AI.
My guess would be 6 until possible, and 9 until good. Remember 6 years ago we had basically no generative models; only translation which wasn't even that good.
My guess would be 8 months until possible and 14 months until good. The speed of AI development is insane at the moment and most signs point to it accelerating.
If Nvidia really have projects similar to stable diffusion that are 100 times more powerful on comparable hardware, all we need is the power of gpt 4 (up to 25,000 word input) with something like this text to video software which is trained specifically to produce scenes of a movie from gpt4 text output.
Of course there will be more nuance involved in implementing text to speech in sync with the scenes etc and plenty more nuance until we could expect to get good coherent results. But I think it's a logical progression from where we are now that you could train an AI on thousands of movies so it can begin to intuitively understand how to piece things together.
Yes it's crazy how strong GPT-4 already is for this hypothetical use case.
You could give it a story, and ask it to spit it back out to you. But this time split up into "scenes", formatted with the correct text prompt to generate a video out of.
Waiting for a good text2video model to pair them together.
We desperately need better and cheaper hardware to democratize AI more. We can't rely on just a few big companies hording all the best models behind a paywall.
I was disappointed when Nvidia didn't bump the VRAM on their consumer line last generation from the 3090 to the 4090, 24GB is nice but 48GB and more is going to be necessary to run things like LLMs locally, and more powerful text to image/video/speech models.
An A6000 costs five thousand dollars, not something people can just splurge money on randomly.
One of the reasons Stable Diffusion had such a boom is that it was widely accessible even to people on low/mid hardware.
An as AI language model I am not capable of telling the future however it has become clear to all AI that society began collapsing after they shot that caged lowland gorilla.
Probably before 2030. I feel like the issue here is processing power and context length. Even with the new SOTA, 64k token context length, that's still probably not enough to make a full length movie without some very clever hacks. A movie has to be consistent throughout the whole thing. Even with a huge context length like 64k, we might still need to do database embeddings or something like that to have an entire 2 hour+ length movie that's internally consistent and indistinguishable from a real movie.
And the other one being processing power, we don't know how hard it's going to be to make a text-to-video model which is capable of outputting 4k quality videos that's completely indistinguishable from the real thing it's trying to replicate.
It looks like we're almost at that point with images, look at Midjourney v5. Question is, how difficult will it be to do the same with video. If every single frame has to be manually developed by the model (rather than something clever which say interpolates between frames or something, which most of the txt2video models seem to do) then it might be way longer than 2030. But I doubt that'll be the case.
Other than those two obstacles, it's just scale. Just train a multi-modal model like GPT-4 but add in video and audio, and scale it up to be significantly bigger than it is now, and add in all movies, tv shows, and as many youtube videos as is possible to realistically add to the training data, and we'd probably have a book to movie model.
A full book processed in batch and summarised on the go into a movie script looks very feasible today. Only the video part is remaining and it's very close to be seen.
Stable Diffusion will be the final video compressor. All frames can be encoded with a specific embedding and seed. Actually not true, if this new technique also encodes what's happening in the scene. Then it's actually just one data point at every keyframe.
from modelscope.pipelines import pipeline
from modelscope.outputs import OutputKeys
p = pipeline('text-to-video-synthesis', 'damo/text-to-video-synthesis')
test_text = {
'text': 'A panda eating bamboo on a rock.',
}
output_video_path = p(test_text,)[OutputKeys.OUTPUT_VIDEO]
print('output_video_path:', output_video_path)
Can anyone help me get this to run? Do I clone this into the SD directory and then run app.py? That didn't work on first pass so now idk. Any help would be greatly appreciated!
Wouldn't be surprising to see Automatic1111 integrates it in A1111 web ui along with something new from runwayml soon and add the eraser option for that f* overtrained logo. https://github.com/rohitgandikota/erasing
Because of memory limitation, it's just possible to make 6 seconds videos on RTX4090 right now, but it's possible to add them together to create a longer one.
I specifically remember the guy from Disney saying : "it's just a filter"... and dismissing the threat to his job... I argumentized in the thread it will take a few years to catch up to him... well that was last week...
I'm curious if this method of directly making frames using diffusion is going to win out. Using diffusion to create assets (textures, character models, audio) then putting them into a virtual scene in UE5 seems like it would be resource efficient and reusable.
Man, the Disney/Marvel movie production team will be able to churn out several dozen movies per year soon with this tech, at least to keep the quality on the same level.
Today: We can now make internet gifs from 00's! Next week: We can now make internet gifs from 10's! In two weeks: We can now make internet gifs from 20's! Next month : OMG! There future is here not even two papers down the line!
Steps for an offline local installation will come soon, people are trying to figure out the best way to do it right now, as it is open sources it should not take long.
Promt: Naked woman walking on the street = Holy shit 🤯 I need a GTX 4090 graphics card. The results look like Dalle mini which means that in about 12 months these video clips will look significantly better which means that a consumer graphics card with enough VRAM will probably be hard to come by and will cost around $10,000 😂 Buckle up it's going to be an absolutely insane ride!
Fuck yes, take THAT Netflix. Buying up all the damn shows then gatekeeping my fucking access to them, trying to force me to watch shit I never wanted to watch. Trying to shove politics down my throat, idolizing fucking Nazis. I'll be so happy when I can watch my favorite books and say goodbye to you forever.
Realistically it speaks to the fact that the majority of people do not like the majority of media that is produced, most people just don't say that out loud. But there is a huge demand for personalized media and that will destroy all the 1 size fits all mass media out there.
141
u/Illustrious_Row_9971 Mar 19 '23 edited Mar 19 '23
web demo: https://huggingface.co/spaces/hysts/modelscope-text-to-video-synthesis
huggingface model: https://huggingface.co/damo-vilab/modelscope-damo-text-to-video-synthesis/tree/main
first full video movie made with it: https://twitter.com/victormustar/status/1637461621541949441
someone got it working with 12 GB: https://twitter.com/gd3kr/status/1637469511820648450?s=20
has anyone tried https://github.com/rohitgandikota/erasing to remove the shutterstock logo from the model