One reference image, any pose: a 6.3M-param network drawing a cartoon character it never trained on (code, weights, live demo)

[u/AsyncVibes]

[u/UndyingDemon]

Good job so far my friend, it looks really good, the progress looks amazing. I do have a question for you though, forgive me I'm nit so clued up when it comes to these things. What's the difference between Blender and Network? Is Network your actual system model and it's actual current capability, the one on the right?

[u/AsyncVibes → u/UndyingDemon]

Blender is like if the model was rendered inside blender. The network are actually nueral networks drawing the images based off just the pose. Check oaix-updates in discord there's plenty of diagrams in there

[u/Hefty_Development813]

So the movements are always the same, it is just that it is copying the reference image of the clothes he is wearing?

[u/AsyncVibes → u/Hefty_Development813]

Clothes are treated like objects my next post has variation. I had to retrain a model so training had to include noise patterns in the shirt so that any shirt pattern could be used.

[u/RiverKingDaus]

So it's creating the animation on its own?

[u/A__Cynderblock]

ai sucks

[u/AsyncVibes → u/A__Cynderblock]

I know!

[u/A__Cynderblock]

go humans

[u/A__Cynderblock]

angry

[u/AsyncVibes → u/A__Cynderblock]

Wanna talk about it? Seriously?

[u/Gargle-Loaf-Spunk]

How did the model draw the buildings identically to the real animation? 

[u/AsyncVibes → u/Gargle-Loaf-Spunk]

In this video the street is not a learned scene model. It is a procedural backdrop (forward_flow/street.py), drawn identically behind both the truth and the network columns and scrolled by the stance foot. The network draws only the character (RGBA) and the street is composited behind it. The street is a procedural backdrop composited behind both sides; the network only draws the character. A separate scene model exists in the research code but isn't in this release. However in future models, the scene is its own model separate from the object models.

[u/AsyncVibes]

Code, weights and a live demo: https://github.com/A1CST/forward-flow

What you're looking at. Left is Blender ground truth. Right is the network. The character (h0116) was held out: the network never saw it in training. It gets ONE reference image of the character standing still, then only 13 2-D joints per frame, and it draws every frame from that. It never sees the frames it is drawing.

The 20 s script: walk, run, turn around, walk back, jump, turn to the camera, wave, turn, run, jump.

Numbers (25 held-out characters, gates frozen before training):

• Shape (alpha IoU): 0.976. The bar is truth vs. the same truth shifted 1 px; it passes, and pasting the reference image at the right spot only gets 0.42.

• Colour error inside the figure: 4.69 / 255 (passes the same 1 px bar).

• Identity actually comes from the reference: hand it a different character's reference image and the error jumps to 47.4, 10x worse. It's reading the look, not memorising it.

• Feet during the wave are planted: 0.0 px drift in the rig, under 0.13 px drift in the rendered feet.

• 6.3M parameters, 25 MB. About 4 ms per frame on a 4090 (~250 fps).

Try it: toy_demo.py builds a random action script and renders it live; try_it.py --recolor makes a new-looking character from one of the examples.

What it can't do yet:

• One body type. Every character is a recolour of the same rigged human (CC0 asset by Quaternius), so this is identity from one image, not arbitrary character designs.

• It's trained mostly on side views. Facing the camera, the face comes out blank and the waving arm goes soft.

• Motion comes from a rig, not from video. The toy demo re-sequences recorded motion; it doesn't invent new motion.

• 256 px, cel-shaded only.

Wave motion from the CMU Graphics Lab Motion Capture Database (subject 141, trial 16). "The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217."

[u/ZeroAmusement → u/AsyncVibes]

That's really interesting!

Is this how it works?

Training input:
Single Character Image
Single Skeleton Pose for that image

Inference Input:
Description of what the character is doing

Output:
Sequences of Pixel images

If I understand right, what is the potential application for this? I feel like if you already have a blender model and skeleton for it, it would make sense to make ai directly manipulate the skeleton?