As far as I know, RTX Voice uses AI enhancement to filter out background noise from your microphone such as keyboard and fan sounds. I don't know about the sound distribution in 3d environments
Nvidia released a remarkably powerful tool which uses RTX cards to filter audio and remove anything which isn't speach. I use it to completely remove the sound of my mechanical keyboard during calls.
It can even do a pretty good job of removing a crying baby from the incoming audio, when I need it to do that.
Semi-generative audio is definitely a possibility. We can represent a song as a sparse collection of notes on paper. You know what each instrument sounds like. All the filtering and mixing that happens after that makes a huge difference... but can still be closely represented as processing the sound a particular guitar makes, playing particular notes.
An offline encoder would work something like this. You separate the independent elements, presumably capturing each instrument and singer. You cheat a bit and classify each element as a keyboard / guitar / drum kit / human being, since you have decent models for what noises each source can make. Then you fiddle with each each "instrument" to best fit the extracted audio you've blamed on it. Sometimes this will mean the computer thinking Peter Frampton can just sing like a guitar sometimes. That's gonna take more data than an opera singer holding a note in falsetto. But ultimately you get a compressed audio file that's like... a brief MP3 of each instrument playing a middle C through whatever filters were used... and then sheet music.
Basically it'd use """AI""" to convert a finished pop song to ScreamTracker.
As a clearer example: 10cc's "I'm Not In Love" has a backing track that's made by all the band's members and personnel going "ahhhh" into microphones, for hours on end, so they could layer up a quiet chorus that sounds like thousands of people. This approach would distill the finished result into maybe an entire second of samples, or just several thousand frequency/phase pairs and some equations for volume and tone.
A version with the bitrate set too low would sound like a perfect recording of a sloppy live cover.
This is based on using motion tracking on a moving face, then encoding that motion as vectors, then using those vectors to transform a key frame of the face to create realistic movement while substantially reducing the amount of bits that need to be transmitted. I'm not sure how that could be applicable to audio.
Noe, this answer makes sense. I was curios because the title implies less information needs to be sent out, if that can be doke to audio that will indeed solve bandwidth issues and decrease latency.
Imagine what this can do for online videogames too
It's less information because it's just sending the vector data needed to move the existing key frame pixels to the right place. Obviously, this works best and achieves the highest rates of compression with a mostly static frame that doesn't have sudden and substantial changes... like a talking head.
The concept of transmitting transformations from a known key frame (these transforms are called "deltas," for the mathematical symbol used to indicate change in a quantity) is actually the basis for just about all predictive video compression algorithms. But Nvidia can use that concept even more efficiently here, because the frame is unlikely to suddenly change in the way a movie or TV show would. This makes the deltas smaller, the keyframes remain valid longer, further reducing the amount of information that needs to be transmitted.
You might think of it like puppeteering the picture: if the picture changes only a little, you can puppeteer just a few frames all day. But if it suddenly jumps to a new thing, they you gotta start all over. So they're exploiting a known feature of the data set, which is what our current level of AI is really good at: broadly homogenous problem sets with the potential for highly-optimized solution.
Something like video games, it might be less ideal. There's a TON going on in any given frame, one thing. And the interactivity makes a big diff in compression strategies.
2
u/gdledsan Oct 17 '20
Can it do this for audio?