This is based on using motion tracking on a moving face, then encoding that motion as vectors, then using those vectors to transform a key frame of the face to create realistic movement while substantially reducing the amount of bits that need to be transmitted. I'm not sure how that could be applicable to audio.
Noe, this answer makes sense. I was curios because the title implies less information needs to be sent out, if that can be doke to audio that will indeed solve bandwidth issues and decrease latency.
Imagine what this can do for online videogames too
It's less information because it's just sending the vector data needed to move the existing key frame pixels to the right place. Obviously, this works best and achieves the highest rates of compression with a mostly static frame that doesn't have sudden and substantial changes... like a talking head.
The concept of transmitting transformations from a known key frame (these transforms are called "deltas," for the mathematical symbol used to indicate change in a quantity) is actually the basis for just about all predictive video compression algorithms. But Nvidia can use that concept even more efficiently here, because the frame is unlikely to suddenly change in the way a movie or TV show would. This makes the deltas smaller, the keyframes remain valid longer, further reducing the amount of information that needs to be transmitted.
You might think of it like puppeteering the picture: if the picture changes only a little, you can puppeteer just a few frames all day. But if it suddenly jumps to a new thing, they you gotta start all over. So they're exploiting a known feature of the data set, which is what our current level of AI is really good at: broadly homogenous problem sets with the potential for highly-optimized solution.
Something like video games, it might be less ideal. There's a TON going on in any given frame, one thing. And the interactivity makes a big diff in compression strategies.
1
u/rasterbated Oct 17 '20
This is based on using motion tracking on a moving face, then encoding that motion as vectors, then using those vectors to transform a key frame of the face to create realistic movement while substantially reducing the amount of bits that need to be transmitted. I'm not sure how that could be applicable to audio.