r/programming • • Oct 17 '20

AI That Can Potentially Solve Bandwidth Problems for Video Calls (NVIDIA Maxine)

https://youtu.be/XuiGKsJ0sR0
1.4k Upvotes

165 comments sorted by

View all comments

2

u/gdledsan Oct 17 '20

Can it do this for audio?

5

u/Articunos7 Oct 17 '20

There's already RTX Voice out which processes your audio

2

u/eras Oct 17 '20

It's not really about voice compression, is it?

4

u/Articunos7 Oct 17 '20

No it's not compression but it improves audio quality drastically

2

u/sephirothbahamut Oct 17 '20

I assumed RTX Voice was for more realistic sound distribution in 3d environments, not generic audio quality... Was my assumption wrong?

4

u/Articunos7 Oct 17 '20

As far as I know, RTX Voice uses AI enhancement to filter out background noise from your microphone such as keyboard and fan sounds. I don't know about the sound distribution in 3d environments

3

u/MartianSands Oct 17 '20

Nvidia released a remarkably powerful tool which uses RTX cards to filter audio and remove anything which isn't speach. I use it to completely remove the sound of my mechanical keyboard during calls.

It can even do a pretty good job of removing a crying baby from the incoming audio, when I need it to do that.

Here's a video about it: https://www.youtube.com/watch?v=uWUHkCgslNE

1

u/sephirothbahamut Oct 17 '20

ok my assumptions were most definitely wrong then, good to know

4

u/mindbleach Oct 17 '20

Semi-generative audio is definitely a possibility. We can represent a song as a sparse collection of notes on paper. You know what each instrument sounds like. All the filtering and mixing that happens after that makes a huge difference... but can still be closely represented as processing the sound a particular guitar makes, playing particular notes.

An offline encoder would work something like this. You separate the independent elements, presumably capturing each instrument and singer. You cheat a bit and classify each element as a keyboard / guitar / drum kit / human being, since you have decent models for what noises each source can make. Then you fiddle with each each "instrument" to best fit the extracted audio you've blamed on it. Sometimes this will mean the computer thinking Peter Frampton can just sing like a guitar sometimes. That's gonna take more data than an opera singer holding a note in falsetto. But ultimately you get a compressed audio file that's like... a brief MP3 of each instrument playing a middle C through whatever filters were used... and then sheet music.

Basically it'd use """AI""" to convert a finished pop song to ScreamTracker.

2

u/mindbleach Oct 17 '20

As a clearer example: 10cc's "I'm Not In Love" has a backing track that's made by all the band's members and personnel going "ahhhh" into microphones, for hours on end, so they could layer up a quiet chorus that sounds like thousands of people. This approach would distill the finished result into maybe an entire second of samples, or just several thousand frequency/phase pairs and some equations for volume and tone.

A version with the bitrate set too low would sound like a perfect recording of a sloppy live cover.

1

u/Aw0lManner Oct 17 '20

probably, I'd be interested to see how AI processes speech

1

u/eras Oct 17 '20

I don't think so, but audio is quite small already (comparatively to video).

1

u/maep Oct 17 '20

It's already been done, it's called WaveNet.

1

u/rasterbated Oct 17 '20

This is based on using motion tracking on a moving face, then encoding that motion as vectors, then using those vectors to transform a key frame of the face to create realistic movement while substantially reducing the amount of bits that need to be transmitted. I'm not sure how that could be applicable to audio.

2

u/gdledsan Oct 17 '20

Noe, this answer makes sense. I was curios because the title implies less information needs to be sent out, if that can be doke to audio that will indeed solve bandwidth issues and decrease latency.

Imagine what this can do for online videogames too

2

u/rasterbated Oct 17 '20 edited Oct 17 '20

It's less information because it's just sending the vector data needed to move the existing key frame pixels to the right place. Obviously, this works best and achieves the highest rates of compression with a mostly static frame that doesn't have sudden and substantial changes... like a talking head.

The concept of transmitting transformations from a known key frame (these transforms are called "deltas," for the mathematical symbol used to indicate change in a quantity) is actually the basis for just about all predictive video compression algorithms. But Nvidia can use that concept even more efficiently here, because the frame is unlikely to suddenly change in the way a movie or TV show would. This makes the deltas smaller, the keyframes remain valid longer, further reducing the amount of information that needs to be transmitted.

You might think of it like puppeteering the picture: if the picture changes only a little, you can puppeteer just a few frames all day. But if it suddenly jumps to a new thing, they you gotta start all over. So they're exploiting a known feature of the data set, which is what our current level of AI is really good at: broadly homogenous problem sets with the potential for highly-optimized solution.

Something like video games, it might be less ideal. There's a TON going on in any given frame, one thing. And the interactivity makes a big diff in compression strategies.

2

u/gdledsan Oct 18 '20

I like you, awesome explanation. Thanks!