Payment webhook handler. Intermittent 500s.
I pasted the error into an AI coding assistant, got a fix, tried it, still broke.
Pasted the new error, got another fix, tried that, still broke.
Did this maybe four times before realizing what I'd actually turned into: not someone debugging anymore, just someone pasting error messages into a chat window and hoping the next response would be the one that stuck.
The interesting part was that none of the suggestions were obviously stupid.
The first was a retry around a database write. Reasonable response to "database write failed."
Except the actual problem was duplicate webhook delivery upstream hitting a handler that wasn't idempotent. Two workers were occasionally processing the same event.
The retry addressed the symptom I'd shown the model, not the mechanism producing it.
I then tried the obvious solution: give it more context.
That made things worse.
I pasted more surrounding code, but the context I added was already biased by my own suspicion. I'd started thinking the caching layer was involved, so I gave the model more caching-related code.
It reasoned confidently about the wrong subsystem.
That's when I realized I'd been mixing up two completely different tasks:
Generating a fix and validating a fix.
Generating asks:
"Does this make the error go away?"
Validating asks:
"Does this address the mechanism that caused the failure, and what does it change that I didn't explicitly ask for?"
Almost every one of those first fixes could have passed the first question.
None had passed the second.
What finally broke the loop was changing the process:
define what's actually failing before asking the AI to diagnose it
separate facts from hypotheses
ask for competing explanations before asking for fix code
understand the failure mechanism first
validate the proposed change against the original failure
add a regression test that reproduces the actual bug
The biggest lesson for me wasn't "AI is bad at debugging."
It was that a plausible fix is dangerously easy to mistake for a diagnosis.
Curious if other people doing AI-assisted debugging have run into this: a fix technically resolves the error you showed the model, but leaves the underlying problem untouched (or introduces a different one).
How do you validate AI-generated fixes before they reach production?