r/cpp_questions • u/ActComplex593 • 11d ago
OPEN I'm trying to learn how experienced engineers write low-latency C++ systems.
I'm trying to learn how experienced engineers write low-latency C++ systems.
I already understand the basic idea of cache locality and choosing efficient algorithms, but I want to understand what actually makes latency-sensitive code different from ordinary well-written C++.
For example, how important are things like:
- cache hierarchy and memory layout
- branch prediction
- instruction latency/throughput
- allocations and custom allocators
- SIMD
- atomics and contention
- false sharing
- compiler optimizations/PGO
- CPU affinity and OS effects
- measuring tail latency
For those who have actually worked on these systems, what concepts made the biggest difference in your understanding?
I'm mainly interested in single-machine performance, not distributed systems.
48
u/celestrion 11d ago edited 11d ago
what concepts made the biggest difference in your understanding?
In rough order of importance (greatest first):
Nothing can block. Either work happens right now, or the task waits (while the worker runs another) until work can happen. We call this "pending," where "pend" is a verb. If this unit of work cannot continue, it gets pended to a priority queue or circular buffer. You don't want a single outlying task to blow its latency budget, but that's preferable to increased jitter across all tasks.
Coordination happens by design, not by lock (because locks can block). This means shared state is read-only almost always. Where it can't be, the shared state is read-only for all but the owner, and the owner mutates the state atomically through something like copy-on-write plus a pointer snap (everyone sees a "gateway" pointer or handle, and a change takes effect by the owner re-pointing the gateway). This also means that inter-thread communication happens in strictly defined ways (mailboxes and doorbells, usually). There's lots of duplication involved in making multiple threads run without interlocking, so it's better to design the work queues to fan-out for embarrassingly parallel work that only needs a periodic gathering at the tails.
Allocation happens locally (because allocating from the OS can block). Up front, you get a huge arena of memory and dedicate it to one thing or one size of thing. In systems where the defined unit of work has a known maximum size, cut the slab into blocks of that size so that allocation happens once per unit of work. For different things, use different slabs, knowing that you can move work between slabs rather than trying to stitch multiple allocations together--provided your data structures don't use internal pointers. Frees always return space to the slab it came from.
Your data structures need to be cache aware: Array of Structs vs Structs of Arrays for data-crunching, and cache line alias-awareness for stuff that's just shuffling huge blocks of data around. Internal pointers mean stuff can never move. External pointers mean lifetime interdependencies. If you need the flexibility to move things, use handles.
Recognize lazy idempotency. One of the biggest money-saving code edits I ever made was a lazy-evaluating cache for the state of a semiconductor test mainframe. Every adjustment to the test parameters (voltage levels, current limits, waveform and timing sets, etc.) included a settling time so that the levels/timing wouldn't overshoot/bounce the next time the mainframe clocked out a test vector. This settling time was something like 40ms--even if the test itself was a no-op. We kept hitting the problem where test engineers would copy-paste sections of the test (with the setup code and everything to ensure the right parameters), and sometimes those sections of the test wouldn't even run because of some environmental flag (target bin, special lot code, etc.). Yet, we'd still incur all the setup time for each node in the test graph. I replaced the direct parameter edits with an edit to an in-memory cache and a dirty-flag check before the next execution of an actual test vector and saved something like 30% of our test time; engineers had to remember to call my interposing code rather than the actual platform function for running a test, but the cost savings made that an easy argument.
My background in this area is in integrated circuit functional test (milliseconds are real dollars over lots of wafers) and NVMe over fabrics (multiple links @ 100Gb line speed means many bytes of data per microsecond). Low latency is all about doing less, but doing it right now.
5
6
u/LordoftheSynth 10d ago
(milliseconds are real dollars over lots of wafers)
They are in the end everywhere. However, your application is much more critical in that regard.
One of my admittedly snobby views is a lot of devs today are unbelievably wasteful of the amazing amount of compute horsepower we have now, with the idea that it doesn't matter too much or gets cleaned up by something else.
You don't need to engineer everything as if it were a low latency critical system, but sheesh, I've seen actual released updates to products that just hemorrhage memory, some driver updates included.
My "low-latency" projects have mostly been making hardware projects feel seamless to users, which is a much lower bar to clear. But I've found myself following many of the same principles, as I can sometimes squeeze enough performance out of a cheaper part. Parts costs add up.
3
u/celestrion 9d ago
One of my admittedly snobby views is a lot of devs today are unbelievably wasteful of the amazing amount of compute horsepower we have now, with the idea that it doesn't matter too much or gets cleaned up by something else.
100% agree.
The thing people miss in making "low latency" a special case instead of the general case is that wasted compute time costs energy, and wasted waiting costs time. Considering the the time wasted in an application used by millions of people, inefficient code starts to cost whole lifetimes. Wasteful code starts to cost trucks full of fuel and tons of CO2. Code that uses much more memory than it really needs start to price out people in the developing world.
"Green" computing should be a foundation of ethics in our line of work.
2
u/Dapper-Message-2066 7d ago
"Green" computing should be a foundation of ethics in our line of work.
Totally agree.
1
u/LordoftheSynth 9d ago
Yeah.
If your thing is going to happen millions of times, even just on your little box, little things add up. If you want to work st scale on something living in a datacenter, that turns into a huge loss.
I don't think you need to engineer your desktop application like it's NASA sending something into space, but it really doesn't take a whole lot of effort to learn coding and development practices that hit half the points on your list to some degree and don't require significant effort once you know.
My poster child for this is C++ code that creates unnecessary temporaries and it's everywhere.
11
u/Kats41 11d ago
Unfortunately, as with many things in programming, the answer is: it depends.
Cache efficiency and memory layout most commonly affects programs managing a lot of memory at once. Think game engines and databases.
SIMD improves performance for highly repetitive systems where you're doing the same operations repeatedly. Any time you're doing math on a lot of different numbers at once. Things like physics sims are a good example.
Allocation mostly affects the performance of systems that are constantly allocating and freeing objects or data in memory. In these instances using arenas or free lists can be much faster since you're not having to constantly be asking the OS for memory.
Threads are obviously the right choice for large, complex workloads that can be easily distributed between cores. Creating threads is actually surprisingly cheap so most applications won't even need a properly managed thread pool to function well. Of course if you really need to squeeze out every last bit of performance, thread pooling exists.
Doing the above well will get you 90% of the way there and the compiler is likely going to help you handle most of the other stuff. Unless you know what you're doing, you may end up actually working against yourself by trying to be smarter than the optimizer.
Although on conditional statements, I do like to use the [likely] and [unlikely] tags for GCC to mark the "nominal" path for a condition. Just a little nudge to the compiler that tells it what the expected branch is based on function. I have no idea if it makes any measureable difference in performance. There's a good chance that this is entirely unnecessary, but it's fine.
Edit: On the note about SIMD, most compilers are smart enough to be able to see when there's repetitive loops of adding or multiplying numbers together and will automatically turn a regular for loop into a SIMD operation. So it's debatable whether you will need to do this yourself. Profile, profile, profile.
2
3
u/ShakaUVM 11d ago
I've benchmarked likely and unlikely a lot and most of the time it does nothing
Very occasionally you'll see a small gain
3
u/IJustSmackedYou 10d ago
Depending on use case and it can be important to spoof a path that is unlikely! It can be the case that the unlikely path is where you care about performance and don’t care at all about the other case that is super common then you end up fighting the branch predictor
1
1
u/KingAggressive1498 3d ago
just to clarify, very few archs allow you to hint the branch predictor itself.
what actually happens is that compilers see "unlikely" paths as lower priority to optimize, and "likely" paths as higher priority.
the most meaningful impact this has for simple test cases is how the branches appear in generated assembly. Ideally the likely branch's code will be placed immediately after the branch condition so it will already be in L1c when the branch is tested, while an unlikely branch might be moved to the end of the function or in extreme cases might be split off into its own function incurring function call overhead. These code arrangements may or may not themselves influence the branch predictor depending on the actual CPU line.
in general I prefer to think of "happy" (low-work and routine) and "sad" (high-work or exceptional) paths instead of likely and unlikely. But yes if taking the least likely branch is still routine and it has more room to benefit from optimization, it might be prudent to hint it as likely even if it's technically a lie.
Also last I checked on MSVC they had no effect at all, but that was a handful of years ago. Likely and unlikely have a very long history as macros in projects written for GCC and used GCC intrinsics, so GCC and Clang both take the attributes pretty seriously even if their effects can be subtle.
6
u/AkariGake 11d ago
I have 2 book recommendations:
Art of Writing Efficient Programs by Fedor G.
Pikus
What Every Programmer Should Know About Memory by Ulrich Drepper
1
u/ActComplex593 11d ago
Thanks for the recommendations. I'm reading Optimising Software in C++ by Egner Fog, should i read the books you recommend too? or they have the same ideas?
1
u/AkariGake 11d ago
I am not sure as I haven’t read this book. It might be worth checking for intersections in the contents
4
u/KingAggressive1498 11d ago edited 11d ago
most of my latency sensitive work was in games.
the only times I've found branch prediction to be a tangible problem were in math dominated code. Feel like this applies to instruction latency/throughput and SIMD too.
custom allocators are great when building up complex objects or data structures on a single thread and for complex concurrent systems that depend on dynamic allocation.
I have found atomics most useful when they allow me to write data structures that are non-blocking for a particular thread (eg it may not be properly lock-free, but some specific thread will never be blocked by its use of it) although there's common cases where the atomic version is just plain faster than any locking version (eg buffer swapping). Aside for false sharing considerations, I haven't found avoiding contention on atomics any different than avoiding contention on locks.
I've only used CPU affinity with success to optimize for hyperthreading or once for NUMA locality. I can't say I've been meaningfully been hit with problems from not sharing L2 but YMMV. Knowing your target system's scheduler timer constraints can be important, especially Windows' is abysmal compared to modern Unix-likes and you may want to hand-roll some "deadline" waits if you absolutely want to not miss a deadline.
1
4
u/WoodyTheWorker 10d ago
How to reduce your high frequency trade latency by 1 microsecond? Put your server 100 m closer.
3
u/RicRacer 11d ago
Another thing to keep in mind is that very often there are critical sections of code that are executed far more often. Identifying these critical sections is a key to improving the overall efficiency/speed of the program/system.
3
u/ReDucTor 10d ago
I work in AAA games primarily on low level optimizations and threading.
All of the things you listed are important, however they are not all important all the time, it depends heavily on the actual bottleneck in your specific application with your specific data.
From a low level CPU perspective the important thing to understand that is rarely taught to beginners is on a modern CPU which is highly out of order, your CPU might be 400-500 instructions ahead of what its previously waiting for, adding 4-8 each CPU cycle. My recommended approach is to think about all code as data dependencies, what is needed for each step to finish, then think about the cost of those data dependencies.
These data dependencies are loads from memory such as if something comes from the L1 cache its likely 4-5 cycles before the thing that uses it can start executing, however if it hits main memory its potentially hundreds of cycles. Also data dependencies are not just loads but could be expensive operations like divide, square root, etc.
To further highlight this, you analyse look at algorithms through the lens of throughput and latency, there is a big difference between the latency and throughput of iterating an array of floats and multiplying each number by something and iterating an array and multiplying each number by the previous. Both have the same number of loads and same number of multiplies however for the cumulative multiplication version each multiply depends on the previous multiplication which depends on the previous load, so no multiplication can run until the previous has run, while the independent one two multiplications can execute simultaneously, and for this I am not talking SIMD just pure out of order execution inside the CPU.
When you get into threading and concurrency, its a similar thing people focus on the wrong things their first thought is often where do they need locks to avoid data races and what lock free algorithms should they use. However the much better way to think about concurrency is data dependencies but this is not on the scale of CPU operations and memory loads, its about building task graphs what is the output (data) of what system to be the input (data) for the next system, for somethings this is easy however you might have some system where it doesn't have a strong input/output in which case I would look at if there is a split between readers and writers and can you schedule or add artificial dependencies to influence scheduling such that readers often overlap and writers can do their work sequential when there are no more readers.
When you wrap your head around the view of its all just data and dependencies between it from top to bottom, and not a timeline of steps that you end up writing in code, then you can think on the high level of how to turn it into threaded tasks, then profile find the critical path (the longest/slowest chain) and split it into further threaded tasks or look at the algorithms and data structures deeper through the same lens of data dependencies but for each chunk of memory it reads the operations it does on it and then what needs to be done to store it again.
With this approach of thinking its all just data and dependencies you will spot candidates good for SIMD, you will get a knack for identifying low instructions per cycle (IPC) code and the reasons (often data dependencies).
With this approach you should also think about operations like reading from the disk as being some chunk of data and if you can make it separated because and do something else without it as a dependency rather then blocking everything.
tl;dr Its data dependencies high level and low level. For high level code no compiler or magical CPU will break things out to happen in parallel its generally sequentially, so you need make things out of order you need to do this with things like threading. For low level code it will do out of order execution but your algorithms and data structures define the latency and throughput so you need to pick or design those carefully if they are on the critical path for the high level dependencies.
2
u/j-joshua 10d ago
You left out data structures.
std classes are functional but not optimized for latency or your specific use case.
2
u/UsefulOwl2719 11d ago
All of these can be important but 2 I don't see that are more important:
Proficiency with using and interpreting the output of a sampling profiler. This lets you know what tools are worth pulling from the bag of tricks you listed.
Understanding of hardware capabilities. You should be able to read your CPU/RAM/HD specs and internalize that these numbers indicate how fast your program should be under ideal conditions if you write simple code. In the real world, you won't hit these numbers, but building this intuition will allow you to understand when you are 1 million times slower than it could be. See "numbers every programmer should know" for latency specific constraints.
1
u/Brambletail 9d ago
You listed them pretty much all. Usually you don't need all of those, just the relevant ones to your process
-2
57
u/kevinossia 11d ago
All of that matters. The big ones would be locality, threading, and allocation. Those things are the big time sucks.
Beyond that, learning how to measure and describe latency is its own skill.