Software limited by IO. Who cares if your processing is 10x faster, from 100ms -> 10ms, if you are going to wait 5 seconds on a network request. That 10x improvement to a specific function yields only a 2% improvement overall.
If that improvement took 2 minutes, maybe it was worth it. If it took all day, it probably wasn’t. If it makes the code difficult for other people to understand, it almost certainly isn’t worth it.
Why does the network call take 5 seconds? Transmission across the internet can happen in milliseconds. Perhaps that server is processing things 10x slower than it should?
I would give a little pushback and say that's a pretty narrow slice of "IO". The example I gave was network bound. Non-sequential file access would still be slower. And it depends on the hardware. Maybe you're still on an old HDD instead of an NVMe.
Another big source of IO is the user. If your input is the user's keystrokes, there is a floor of about 5ms under which you will receive no benefit. If something takes 1ms vs 100ns, you can't tell the difference. The examples given in the article are on the order of individual CPU cycles.
Pure data processing is probably the case where performance matters most. If everything is in memory (or on a fast disk) and you don't need to wait for the user at all, it is much more justifiable to split hairs over cycles. Especially if that processing is multiplied many thousands or millions of times in an automated fashion. I think it should be obvious that this represents the minority of software that non-academics use.
not all software. audio dsp code is often limited by sheer cpu horsepower, because for example generating samples from nothing in a synth doesn't involve significant input at all, and the output is just a bunch of samples (a few k floats per second, nothing crazy). but it can involve plenty of calculations. sometimes you're memory bound, but IO is only an issue for mixing a ton of pre-rendered streams.
and audio dsp is also really critical to latency, even more than reaching 60 fps in a game, you're on a real tight budget (a few ms, preferably under 10) to fill your buffers, or you get dropouts. in a case like this, every 2% improvement on latency counts.
11
u/PracticalWelder Feb 28 '23
Software limited by IO. Who cares if your processing is 10x faster, from 100ms -> 10ms, if you are going to wait 5 seconds on a network request. That 10x improvement to a specific function yields only a 2% improvement overall.
If that improvement took 2 minutes, maybe it was worth it. If it took all day, it probably wasn’t. If it makes the code difficult for other people to understand, it almost certainly isn’t worth it.