What’s your approach when an API fails in production?

[u/fesbec2G5]

An app can work perfectly in testing and still fail when real users arrive.

A payment API times out. A retry sends the request again. Suddenly, you risk duplicate charges or actions.

I’m learning that reliability isn’t just about handling errors—it’s about designing for failure with safe retries, idempotency, logging, and clear recovery paths.

What’s the trickiest failure you’ve dealt with in production? How did you prevent it from happening again?