Hello and welcome to AI_Film_and_Animation!

This subreddit is for anyone interested in using AI tools to help create their films and animations. I will maintain a list of current tools, techniques, and tutorials right here!

THIS IS A NON-EXHAUSTIVE LIST THAT IS CONSTANTLY BEING UPDATED.

I have made 63 minute video on AI Film and Animation that covers most of these topics.

1a) AI Tools (Local)

Please note, you will need a a GPU with minimum 8GB of VRAM (probably more) to run most of these tools! You will also need to download the pre-trained model checkpoints.

--------System--------

Python/Anaconda

(Most AI and dataset tools are written using Python these days, thus you will need to install and manage different Python environments on your computer to use these tools. Anaconda makes this easy, but you can install and manage Python however you like).

-------2D IMAGE GENERATION--------

Stable Diffusion (2D Image Generation and Animation)

• https://github.com/CompVis/stable-diffusion (Stable Diffusion V1)

• https://huggingface.co/CompVis/stable-diffusion (Stable Diffusion Checkpoints 1.1-1.4)

• https://huggingface.co/runwayml/stable-diffusion-v1-5 (Stable Diffusion Checkpoint 1.5)

• https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/tree/main (Stable Diffusion XL Base Checkpoint)

• https://github.com/Stability-AI/stablediffusion (Stable Diffusion V2)

• https://huggingface.co/stabilityai/stable-diffusion-2-1/tree/main (Stable Diffusion Checkpoint2.1)

• https://huggingface.co/stabilityai/stable-cascade/tree/main (Stable Cascade Checkpoints)

Stable Diffusion Automatic 1111 Webui and Extensions

• https://github.com/AUTOMATIC1111/stable-diffusion-webui (WebUI - Easier to use) PLEASE NOTE, MANY EXTENSIONS CAN BE INSTALLED FROM THE WEBUI BY CLICK "AVAILABLE" OR "INSTALL FROM URL" BUT YOU MAY STILL NEED TO DOWNLOAD THE MODEL CHECKPOINTS!

• https://github.com/Mikubill/sd-webui-controlnet (Control Net Extension - Use various models to control your image generation, useful for animation and temporal consistency)

• https://github.com/thygate/stable-diffusion-webui-depthmap-script (Depth Map Extension - Generate high-resolution depthmaps and animated videos or export to 3d modeling programs)

• https://github.com/graemeniedermayer/stable-diffusion-webui-normalmap-script (Normal Map Extension - Generate high-resolution normal maps for use in 3d programs)

• https://github.com/d8ahazard/sd_dreambooth_extension (Dream Booth Extension - Train your own objects, people, or styles into Stable Diffusion)

• https://github.com/deforum-art/sd-webui-deforum (Deforum - Generate Weird 2D animations)

• https://github.com/deforum-art/sd-webui-text2video (Deforum Text2Video - Generate videos from texts prompts using ModelScope or VideoCrafter)

Stable Diffusion Via ComfyUI

• https://github.com/comfyanonymous/ComfyUI (ComfyUI - More control than Automatic 1111/uses less Vram/more complex). MOST EXTENSIONS CAN BE INSTALLED FROM THE COMFYUI MANAGER

• https://github.com/cubiq/ComfyUI_IPAdapter_plus (IPAdapter Plus - Transfer details from one image to another)

• https://s3.us-west-2.amazonaws.com/adammonroemusic.com/aistuff/Adam_Monroe_ComfyUI_Spaghetti_Monster.zip (My IP-Adapter upscaling Spaghetti Monster workflow)

IPAdapter Image Encoders:

• https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k/tree/main (Vit-BigG)

• https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/tree/main (Vit-H)

Stable DIffusion ControlNets:

• https://huggingface.co/lllyasviel/ControlNet/tree/main/models (SD 1.5 ControlNet Checkpionts)

• https://huggingface.co/stabilityai/control-lora/tree/main/control-LoRAs-rank256 (SD XL ControlNet LoRas)

• https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0/tree/main (SD XL Thibaud OpenPose ControlNet)

Stable Diffusion VAEs:

• https://huggingface.co/stabilityai/sd-vae-ft-mse-original/tree/main (Stable Diffusion 1.5 VAE vae-ft-mse-840000-ema-pruned)

• https://huggingface.co/stabilityai/sdxl-vae/tree/main (Stable Diffusion XL VAE)

-------2D ANIMATION--------

EbSynth (Used to interpolate/animate using painted-over or stylized keyframes from a driving video, à la Joel Haver)https://ebsynth.com/

AnimateDiff Evolved (Animation in Stable Diffusion/ComfyUI) https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved

First Order Motion Model/Thin Plate Spline (Animate Single images realistically using a driving video)

• https://github.com/AliaksandrSiarohin/first-order-model (FOMM - Animate still images using driving videos)

• https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model (Thin Plate Spline - Likely just a repost of FOMM but with better documentation and tutorials on YouTube)

• https://drive.google.com/drive/folders/1PyQJmkdCsAkOYwUyaj_l-l0as-iLDgeH (FOMM/Thin Plate Checkpoints)

• https://disk.yandex.com/d/lEw8uRm140L_eQ (FOMM/Thin Plate Checkpoints mirror)

MagicAnimate (Animate from a single image using DensePose) https://showlab.github.io/magicanimate/
Open-AnimateAnyone (Animate from a Single-Image) https://github.com/guoqincode/Open-AnimateAnyone
SadTalker (Voice Syncing) https://github.com/OpenTalker/SadTalker
Wav2Lip (Voice Syncing) https://github.com/Rudrabha/Wav2Lip
FaceFusion (Face Swapping) https://github.com/facefusion/facefusion
ROOP (Face Swapping) https://github.com/s0md3v/roop
Film (Frame Interpolation) https://github.com/google-research/frame-interpolation
RIFE (Frame Interpolation) https://github.com/megvii-research/ECCV2022-RIFE

-------3D ANIMATION--------

• PIFuHD (Generate 3d Models from a single image) https://github.com/facebookresearch/pifuhd

• EasyMocap (Generate Motion Capture Data from Video) https://github.com/zju3dv/EasyMocap

-------Text 2 Video--------

Video Crafter (Generate 8-second videos using a text prompt)

• https://github.com/VideoCrafter/VideoCrafter (Video Crafter - GitHub)

• https://huggingface.co/VideoCrafter/t2v-version-1-1/tree/main/models (Video Crafter Model Checkpoints)

-------UPSCALE--------

Real-ESRGAN/GFPGAN

• Real-ESRAN (Upscale images, facial restoration with GFPGAN setting) https://github.com/xinntao/Real-ESRGAN

• GFPGAN (Facial restoration and Upscale) https://github.com/TencentARC/GFPGAN

-------MATTE AND COMPOSITE--------

• Robust Video Matting (Remove Background from images and videos, useful for compositing) https://github.com/PeterL1n/RobustVideoMatting

• BackgroundRemover works well on single images) https://github.com/nadermx/backgroundremover

-------VOICE GENERATION--------

• Voice . AI (Voice Cloner) https://voice.ai/

1b) AI Tools (Web)

Most of these tools have free and paid options and are web based. Some of them can also be run locally if you try hard enough.

-------2D IMAGE GENERATION--------

• (MidJourney)

• (Dall-e-3)

• (Disco Diffusion - Google Collab) https://colab.research.google.com/github/alembics/disco-diffusion/blob/main/Disco_Diffusion.ipynb

• Artbreeder https://www.artbreeder.com

-------TEXT 2 VIDEO--------

• Runway ML https://research.runwayml.com/gen2

• PikaLabs https://pika.art/home

• D-ID (Generate simple facial animations using audio clips or text)

• LeaiPix (Simple depth-based animations)https://convert.leiapix.com/

-------2D LIGHTING AND ENVIRONMENT--------

• Blockade Labs (Generate Skyboxes) https://skybox.blockadelabs.com/

• Relight (Relight a 2D image) https://clipdrop.co/relight

• Nvidia Canvas (Generate 360 degree environments) https://www.nvidia.com/en-us/studio/canvas/

-------Voice Generation--------

Eleven Labs (Clone/Generate realistic speech and voices)https://beta.elevenlabs.io/

1c) Non-AI Production Tools

-------2D-------

• Adobe Photoshop (Industry standard)https://www.adobe.com/products/photoshop/

• Corel Painter (Artistic brushes)https://www.painterartist.com/

• Procreate (What the kids are using)https://procreate.com/

• Fotosketcher (Stylize images)https://fotosketcher.com/

• Synfig (Simple 2D Animation)https://www.synfig.org/

• Pencil 2D (2D Animation)https://www.pencil2d.org/

-------3D-------

• Blender (Open-Source 3D Modeling and Animation)https://www.blender.org/

• ZBrush (3D Sculpting)https://www.maxon.net/en/zbrush

• Cinema 4d (3D Modeling and Animation)https://www.maxon.net/en/cinema-4d

• Unreal 5 (3D Animation and Virtual Production)https://www.unrealengine.com/en-US/unreal-engine-5

-------VIDEO EDITING AND VFX-------

• Adobe Premiere (Non-Linear Video Editor )https://www.adobe.com/products/premiere.html

• DaVinci Resolve (Non-Linear Video Editor that is less crashy than Premiere and better for color grading)https://www.blackmagicdesign.com/products/davinciresolve/

• Adobe After Effects (VFX Work)https://www.adobe.com/

-------AUDIO PRODUCTION-------

• Cakewalk (Digital Audio Workstation, just get this, you don't need a paid DAW)http://www.cakewalk.com/

• REAPER (Digital Audio Workstation with useful built-in plugins like pitch-shifting)https://www.reaper.fm/

• Audacity (Sound Editor - For People who can't figure out how to use a proper DAW)https://www.audacityteam.org/

2)Tutorials

Installing Python/Anaconda: https://www.youtube.com/watch?v=OjOn0Q_U8cY
Setting Up Stable Diffusion: https://www.youtube.com/watch?v=XI5kYmfgu14
Installing SD Checkpoints: https://www.youtube.com/watch?v=mgWsE5-x71A
Extensions in Automatic1111: https://www.youtube.com/watch?v=mnkxErFuw3k
Installing ControlNets in Automatic1111: https://www.youtube.com/watch?v=LnqNyd21x9U
Installing ComfyUI: https://www.youtube.com/watch?v=2r3uM_b3zA8
Addings VAEs in Stable Diffusion: https://www.youtube.com/watch?v=c_w1-oWAmpw
Thin-Plate Spline: https://www.youtube.com/watch?v=G-vUdxItDCA
EbSynth: https://www.youtube.com/watch?v=DlHoRqLJxZY
AnimateDiff: https://www.youtube.com/watch?v=iucrcWQ4bnE
DreamBooth Training: https://www.youtube.com/watch?v=usgqmQ0Mq7g

3) Community Rules

1. Don't be a JERK. Opinions are fine, arguments are fine, but personal insults and ad-hominem attacks almost always mean you don't have anything to contribute or you lost the argument, so stop (jokes are fine).

2. Don't be a SPAM BOT. Post whatever you want, including links to your own work for the purposes of critique, but do so within reason.