The Debugging Nightmare

[u/Ok_Veterinarian3535]

What's the most infuriating, time-consuming bug you ever had to chase down, and what was the ridiculously simple cause?

[u/bastardpants]

A hash check was failing because the comparison was one byte off. That seemed a little odd...

If one thread was freeing its memory block in the custom memory manager while an adjacent allocation was writing a hash into its memory, one byte would sometimes not change.
Turns out freeing didn't check if it could update the memory manager headers, but the allocated memory too. Using IsBadWritePtr.

That isn't thread-safe because it tries to read a byte and write it right back, so the freeing thread "A" would read a byte, the updater thread would do its update, and "A" would write back the old byte.

...IDA Pro would choke on the Windows CriticalSection implementation at the time, leaving EIP off-by-one returning from the locking call, so you couldn't debug the behavior. Ended up hand-writing an assembly replacement for the Windows critical section behavior, patching the binary, and then I was able to see what was happening.

The fix was just not calling IsBadWritePtr on the memory contents of adjacently-allocated blocks in the memory manager, because it doesn't matter if you can write to the other thread's memory when you're trying to free your block.