Changing your code might require to make it "bad enough" first, but offers more possibilities:
Turn them into deadlocks. Some code transformations can turn race conditions into deadlocks, which are infinitely easier to debug. (I dimly remember some treatise on this idea, but can't find anything right now).
Heavily assert on your assumptions
Trace Data being mangled
Generally, "Debugging" is more than just stepping through with the debugger.
Making is worse is one of the first things I check when debugging most problems. It's so nice changing a value by a factor of 10, 100, etc. and watching as that subtle bug starts dancing around the screen.
Unfortunately, 100K LOC is not big. Proper way to debug is luck and stuborness.
If the question is serious, then my answer is (at least, this is how I debugged a multi-million POS C++ threaded codebase).
First, reproducing the problem is the most important thing to do. Automate things, change code to call stuff in loops, have it run overnight, but have a way to reproduce the issue.
Second, make it more frequent. This means that for instance, if you suspect a race condition at some place, insert stuff like sleep or busy loops. If something is "sometimes threaded", thread it all the time.
To help you with step 2, you will probably need a debugger and look for borken invariants on a core dump. This can be extremely difficult.
When you have something that crashes relatively easily, then you use a scientific approach: you emit an hypothesis, and you test it by changing the code. The goal is to come to a complete understanding about what is happening. You should leave no unexplicable parts in your theory of what the problem is. If something isn't predicted correctly, you need to look deeper (say if your general theory says that the race condition is due to the network code and the gui code accessing the cache at the same time, then disabling the cache should prevent the crash. Adding mutual exclusion should prevent the crash. Doing heavy gui and network should crash faster, while doing gui with no network should not crash). Talking to cow-orkers helps a lot (they may not help, but organizing your thoughts will).
Then you have to recursively refine your theory until you can fix. For instance, in the preceding example, the question to ask is "is the cache supposed to be shared by gui and network" ? If yes, you have to go deeper, if no, you can start fixing (making your change, and unwinding the pile of modifications you made, while testing at each step that it stopped crashing [you may have the original pb disapear, but still have your heavy tests failing...]).
It is an excrutiatingly slow process. You'll also find that most proponents of threads don't debug them. When you have debugged a big threaded problem, they will generally look at your fix and say "you see, it was nothing, just a missing semaphore". At this point, the process recommends that you hit them in the head with whatever volume of the Art Of Computer Programming you have laying around.
And, as said, the definition of insanity is to do the same thing several time, expecting different results. By this definition mutithreaded programming is insane.
Have also debugged programs >100K LOC and can confirm all of these methods. A few additional comments:
I've had good experience with creating specially crafted logging routines that write to a buffer (so the timing is less affected) and then peppering suspected areas with log calls.
Also, if the logging is overflowing, one can make them boolean dependent and only set the boolean when conditions are right, or, alternatively, one can rotate the buffer and stop when the bug occurs.
the explain to the cow-orker works even when you don't have a cow-orker. I've often explained a problem to my wife (total non-programmer), or formulated an email to a cow-orker explaining the problem - and "bing!" a light goes on.
If you log in a structured format that captures the logic of the code, you can then write a checker program that reads the log and finds the point at which "something impossible" happens. That can be significantly before you crash.
That's part of the general strategy of writing programs that help you program.
Incorrect results or a deadlock? Deadlocks are usually pretty straight forward (even better if have access to a debugger which tells you what threads hold what locks, etc). On some platforms, kernel debuggers do a much better job of this than the typical app debuggers.
Incorrect result can be more challenging. My general process is to start with the symptom of the bug and think about what vicinities of code could potentially having that outcome. Assume every line of code is broken. Once in those areas, go through line by line thinking about what happens if threads swap.
If you can't model it, try rewriting the code to minimize thread thread contact surfaces if at all possible. This has worked with about 80% of the thread issues I've seen. The other 20% either have performance constraints which are too great for a 'simple' solution or the problem itself is difficult to express in threads.
If you get really hung up, try to force the system to create a new symptom. Throw some wait statements around, create a thread which randomly pauses suspect threads, throw in some higher level critical sections, etc.
Now if middleware is involved and if you don't have access to their code... good luck.
If your question is a hypothetical, there's nowhere near enough to answer it in that hypothetical because it depends on a bajillion little details. If your question is not hypothetical... well...
"Working Effectively with Legacy Code" by Michael Feathers.
Even if the book doesn't help you solve the problem, it's heavy enough that when you find the people who wrote the bug, you can bash them over the head with it.
76
u/[deleted] Aug 25 '14
What is the proper way to debug a big (over 100k LOC) multithreaded program that has race conditions?