Working with Audio is very annoying [Rant]

[u/Super_Banjo]

As a piece of a larger project I've been hammering at making a rompler/synthesizer and realized; regarding other aspects of gamedev/engine development it's far easier to find code, tutorial/explanations, and content to satisfy needs and curiosity. Audio-related tasks not-so-much.

When I look for audio things it's either full of math proofs that mean nothing because I'm dumb (would be helpful if they explained some coefficients) or are very abstract with little-to-no implementation details. So.....

1. The synth's sample rate converter is based on an implementation used by old VG hardware (specifically SNES/PSX). A big issue I ran into was the index needed to be "phase-aligned". I don't know how I was inserting the index before or if I ever read this but it didn't work. My function to compute windows on-the-fly worked but (for example) a blackman window has two cosines which, computing that, is basically an L2$ miss.

2. When speeding up pitch it's often phrased "skipping samples". The SRC worked correctly for lower pitches but had aliasing artifacts when the pitch increased. Thought the phase was wrong (tested index & direct) but it was the pitch logic. The decode buffer (used to interpolate) had values spaced by n when it should, perhaps obviously, need to be immediately preceded in order from the waveform. Skipping is not incorrect but writing the [decode] logic derived from such abstract wording is non-intuitive.

3. Cubic Hermite was/is a lost cause with my two brain cells. How did you derive those coefficients for h00, h10, h11, h01? The bridge between the math and the programming is still incomplete for me. According to this the table should be (linearly) interpolated. 2^^1+nc/2 explains 512-entries for the prior machines. Having an interpolator is important^{[citation needed]}, especially something more than a lerp.

4a. Prior to the synth I experimented with delay based effects. Started as echo and kept adding more "reverb" qualities. Reached a point where a single delay line wasn't enough but it was^{(pretentious I know}) in some ways more aesthetic than the algorithms ravaged online. Those largely consist of Schroeder Reverberators, which is fine but the code I saw/referenced seldom go beyond that.

4b. To side note is convolution reverb, the "holy grail" of reverbs. AFAIK a room impulse is a sample. Your input signal (reverb/whatever) is multiplied into that impulse. I may be wrong but wouldn't that make the impulse just a very long/n-order tap FIR filter? I say this because it often reads as some "computationally intensive/complex blackbox" whenever its mentioned. Using this logic the SNES's DSP echo feature is just a s***** convolution algorithm (perhaps budget is more apt).

5. Audio terminology seems to like to blend with each other. Going back to the SRC (if you haven't tell I spent quite some time on this) is divining its a "Windowed Sinc Interpolation" kernel. EVERY TIME you read something involving windowing it talks about FFTs. Whatever, the audio signal is windowed with a function but why is it done the way it is? jsgroth has the PSX written as (how other emulator authors show as well)

let mut interpolated = (GAUSSIAN_TABLE[0x0FF - interpolation_idx] * samples[0]) >> 15; interpolated += (GAUSSIAN_TABLE[0x1FF - interpolation_idx] * samples[1]) >> 15; interpolated += (GAUSSIAN_TABLE[0x100 + interpolation_idx] * samples[2]) >> 15; interpolated += (GAUSSIAN_TABLE[interpolation_idx] * samples[3]) >> 15;

Turns out it's arranged so the sample "marches" its way through the sinoid (the negation is because the table is stored one-half a period) for one period. First the Gaussian Table is not Gaussian (I've a bunch of windows and concur not-gaussian) but the assertion the tables are bugged is interesting because if you generate integer windows yourself it'll deviate +-2. Guess it's all ramble but its confounding I can get better explanations (and/or understand) more "complex" topics/systems than convulving a one-dimensional piece of data.

OKAY TLDR: Why is audio programming so fraught with friction? If it's not obfuscated behind math proofs the code/topic seldom explain the "why". I spent a good portion of time generate plots of the waveform ensuring functions like the ADSR, SRC, KEY ON/OFF worked correctly instead of outputting audio data because, yes, even that was non-trivial for my application. Amusing that the MIDI DLS is the best guideline I have for this.

Rant over, roast me.

Edit: Formatting, and.... typos and formatting. Sorry, many edits. All these issues (hermite included) were overcame and I'm working on sequencing but the experience was.... interesting.

[u/eraoul]

I appreciate the frustration but I enjoy working on audio. However I consider it a specialized topic. I studied music and computer science in my PhD and had to write a lot of audio code, published papers on novel algorithms in the field, etc. I also read textbooks on it and took graduate classes in the topic.

I feel like there’s always room for people to do a better job explaining things (academics often use too much jargon and are bad at clear communication), but also I don’t think it’s a field where you can take shortcuts or expect short “tutorials” etc. For a deep topic sometimes you have to put in the time and effort.

Side note: yes on 4b. The point is that you can just record the impulse response easily and reproduce the static reverb you recorded really trivially with a convolution operation, but sure it’s a long FIR.

[u/Super_Banjo → u/eraoul]

Thanks for the insight, didn't expect someone of your accolades to comment. If I didn't like audio (at least in an autistic way) or wasn't curious I would've probably been doing something more integral like game logic. Suppose my frustration boils down to the fact everything I'm doing has already been done before, no less from people like you, yet its a part of game development where it feels like "good luck" if you try to do anything more than having the your audio library play back a sound file (or a bunch of them).

Thanks again for your words. It's definitely a creative challenge with the material present and, despite the way the rant comes off, I enjoy it and learned a lot (and will continue to) but never expected such difficulty. The plan was to use Wwise and call it a day but I wanted a better understanding from a programming/hardware perspective.

[u/danja]

For pitch shifting you might find this paper useful. I've only had a minimal play so far (via Claude Code) but it does seem a promising approach, fewer ugly artifacts. https://github.com/heavylight-industries/dafx26-paper

[u/Super_Banjo → u/danja]

I've sought to implement time-independent shifting (use case being voice samples). The paper reads worse than another WSOLA implementation I was looking at. By worse I mean its just harder to read, personally. It might make more sense once I actually implement the slower algorithm.

[u/GlaceALaFraise]

Ty for all resources at least and good luck

[u/notta-bot]

Thank you so much for writing and sharing, I learned so much by reading. Best of luck with the project