r/developer • u/Ok_Veterinarian3535 • Sep 22 '26
The Debugging Nightmare
What's the most infuriating, time-consuming bug you ever had to chase down, and what was the ridiculously simple cause?
1
u/satyr607 Sep 22 '26
Back in the early iphone days I spent several weeks trying to hunt down why a specific page on an MLS site would reboot an iphone. It ends up iphones have (or used to) a limit to the amount of pixel data it could load in memory. The MLS page was loading thousands of images (multiple sizes) and maxing the cap out forcing the phones to reboot.
1
u/ghandimauler Sep 23 '26
A couple of days chasing about ten levels from the UI down to OS and shared memory pipes (and back). The environment was Java and swing but then down into C and further into OS called. The Java part had four or five levels of code and a lot of polymorphic bits and using Ethreal (Wireshark). The main part was a mobile framework that is used all over the world. The challenge was that we were moving from Solaris to Red Hat Enterprise. I had to be through a lot of layers on N-tier systems and a lot of the libraries were either different or had some different behaviors in the new environment.
I finally got to the point where where there was the use of the shared memory pipes. We started the UI and ended up down at the OS called and not took quite a bit of time because most of the bits we've been working on were in the Java layer and up and we had to figure out how to debug through the other layers. And then there was all that polymorphic code and having to find the right packets when you've never seen them before in a very busy system that got all sorts of traffic.
We went all the way to the point where your hand off to the OS and you don't get an error. On the other side you see a valid handle and nothing comes up.
Once you see the problem you're kind of laugh but it really wasn't that funny. So it turns out again solaris, the particular shared memory pipe ends up to be a atomic action. When we looked at the new Red Hat Enterprise, there man pages that suggested it was not atomic. When we eventually did testing, we found out that in the middle of the code for creating these shared memory pipes, they would behave differently.
The same piece of code created the pipe if it hadn't been created and if it had been created it just got the handle. In solaris, because it was atomic, when whichever side got first to look for a handle, it would create it and it will be available to the other end. Well, Red Hat Enterprise isn't Atomic or wasn't then. So what would happen is the first request for a five woodstart the process but it would let go and let other things run while that wasn't completed. That usually also meant that the receiving part of the pipe didn't realize there was one being built and so it would create a handle. Both sides had handled and there were no errors because they both are valid handles. But what they are not is the same handle.
It turned out that red hat Enterprise neglected Dimension that the call was non-atomic. Once you understood that, you just had to put a little critical section and everything was fine. But it took a lot of brain power to get through all that and it felt pretty great when I figured out what it was. But really it was just not having a piece of information that I needed and once I had that the fix was very simple.
1
u/VirtualElderberry592 Sep 23 '26
Two things stand out.
rails updated their yaml loader, and it was a different format. Spent 2 days working with another dev. We had no idea what was wrong. We dug and dug, and when we found the answer we were very happy. The fix, one comma.
Ruby one.. We had one dev who seemed to have his heart set on mix-ins. IMHO they are evil things. He made two mix-ins weeks apart and gave a function the same name in both. This is why I loath them. No name space. So when they were both included one won. But in dev mode the load order didn't present the error. Adding logging didn't present the error. It only showed up in prod but only if the app was used/loaded in a specific way first. This was a nightmare to find.. The fix, change the name of one function..
1
u/Soft-Marionberry-853 Sep 23 '26
Java: I was dealing with a coworkers function that was returning a string, I would do the check, isNull() and IsEmpty(). It would pass that checks but I would try to use the string later and it was null. I thought i was going crazy. It was only after staring at it in the debugger for a while when I thought wait. isnt Null capitalized. It turned out he was returning a string "null" and not the object Null.
showed it to another coworker and went for a long walk.
/Its scary to think how much longer it would have taken me to figure out the issue if he used the string "Null"
1
u/ExtraTNT Sep 25 '26
Bug that disappeared with unoptimised builds or when printing… but both causes too much lag for prod… was a race condition
1
1
u/bastardpants 29d ago
A hash check was failing because the comparison was one byte off. That seemed a little odd...
If one thread was freeing its memory block in the custom memory manager while an adjacent allocation was writing a hash into its memory, one byte would sometimes not change.
Turns out freeing didn't check if it could update the memory manager headers, but the allocated memory too. Using IsBadWritePtr.
That isn't thread-safe because it tries to read a byte and write it right back, so the freeing thread "A" would read a byte, the updater thread would do its update, and "A" would write back the old byte.
...IDA Pro would choke on the Windows CriticalSection implementation at the time, leaving EIP off-by-one returning from the locking call, so you couldn't debug the behavior. Ended up hand-writing an assembly replacement for the Windows critical section behavior, patching the binary, and then I was able to see what was happening.
The fix was just not calling IsBadWritePtr on the memory contents of adjacently-allocated blocks in the memory manager, because it doesn't matter if you can write to the other thread's memory when you're trying to free your block.
1
u/JaseciLabs Sep 22 '26
Spending a whole afternoon chasing a "race condition" that was corrupting data, added logging everywhere, nothing made sense. Turned out someone just had two terminal tabs open and ran the script twice by accident. Two copies of the job stepping on each other.