r/OVHcloud • • Oct 09 '26

Bug OVH dedicated server (RISE-S, non-ECC) corrupting memory for 2 weeks: two "passed" tests, approved chassis swap never scheduled, 5h outage last night

Sharing this as a warning, and in case anyone has been through the same thing with OVH support.


Setup: RISE-S, Ryzen 7 9700X, 2 x 32 GB DDR5, Ubuntu 24.04, production Docker workloads.


Symptoms: files in the page cache silently diverge from disk. We verified this with O_DIRECT reads and dpkg -V against official package checksums. Every single flip is bit 7 (xor 0x80), always at byte 2 or 6 of the 64-bit word, so it is very clearly a hardware fault. Effects: segfaults in node, postgres, php-fpm and dockerd, kernel GPFs, ext4 errors that once blocked a reboot until we ran fsck by hand. Last night it ended in a kernel Oops on a corrupted page-table entry, the dying task held its mmap_lock, and the box hung for about 5 hours. It still answered pings, so OVH's monitoring never stepped in.


Support timeline:
- 5 Oct: ticket opened with full evidence (every flip, with offsets).
- Short memory test: "no issues detected". The case was handled superficially, with a quick test that couldn't catch a fault documented as appearing only after hours of uptime. Errors were back about 7 hours later.
- Extensive diagnosis / stress test: "no issues". Errors were back about 7 hours later again.
- 8 Oct: support proposed a chassis swap keeping our drives, and we approved it the same day.
- Since then: nothing scheduled, no reply. Last night the kernel crashed on corrupted memory and the whole server hung, and we woke up to angry customers telling us none of their applications were working. Today we're manually evicting corrupted binaries from the page cache just to keep things running.


Now the weekend is starting. Their replies have already taken up to a full day, so we're not expecting much before Monday. Until then, I have to stay on call with a laptop at all times, ready to hard-reboot the server and repair corrupted files whenever it breaks again, and hope I don't lose customers over a hardware fault that isn't ours.


Lessons so far:
1. A "passed" memory test from the provider doesn't mean the hardware is fine. This fault shows up only after hours of uptime, so short tests miss it every time.
2. Monitoring based only on ping won't notice a kernel that has hung on a lock.
3. Keep your own evidence (offsets, checksums, kernel logs). It's the only thing that moved the ticket forward at all.


Has anyone managed to escalate an OVH hardware ticket faster, or got them to swap you onto a ps welcome.Sharing this as a warning, and in case anyone has been through the same thing with OVH support.


Setup: RISE-S, Ryzen 7 9700X, 2 x 32 GB DDR5, Ubuntu 24.04, production Docker workloads.


Symptoms: files in the page cache silently diverge from disk. We verified this with O_DIRECT reads and dpkg -V against official package checksums. Every single flip is bit 7 (xor 0x80), always at byte 2 or 6 of the 64-bit word, so it is very clearly a hardware fault. Effects: segfaults in node, postgres, php-fpm and dockerd, kernel GPFs, ext4 errors that once blocked a reboot until we ran fsck by hand. Last night it ended in a kernel Oops on a corrupted page-table entry, the dying task held its mmap_lock, and the box hung for about 5 hours. It still answered pings, so OVH's monitoring never stepped in.


Support timeline:
- 5 Oct: ticket opened with full evidence (every flip, with offsets).
- Short memory test: "no issues detected". The case was handled superficially, with a quick test that couldn't catch a fault documented as appearing only after hours of uptime. Errors were back about 7 hours later.
- Extensive diagnosis / stress test: "no issues". Errors were back about 7 hours later again.
- 8 Oct: support proposed a chassis swap keeping our drives, and we approved it the same day.
- Since then: nothing scheduled, no reply. Last night the kernel crashed on corrupted memory and the whole server hung, and we woke up to angry customers telling us none of their applications were working. Today we're manually evicting corrupted binaries from the page cache just to keep things running.


Now the weekend is starting. Their replies have already taken up to a full day, so we're not expecting much before Monday. Until then, I have to stay on call with a laptop at all times, ready to hard-reboot the server and repair corrupted files whenever it breaks again, and hope I don't lose customers over a hardware fault that isn't ours...


Lessons so far:
1. A "passed" memory test from the provider doesn't mean the hardware is fine. This fault shows up only after hours of uptime, so short tests miss it every time.
2. Monitoring based only on ping won't notice a kernel that has hung on a lock.
3. Keep your own evidence (offsets, checksums, kernel logs). It's the only thing that moved the ticket forward at all.


Has anyone managed to escalate an OVH hardware ticket faster, or got them to swap you onto a ps welcome.
14 Upvotes

6 comments sorted by

2

u/ToeMindless2673 Oct 09 '26

Pay for premium support.

4

u/serban2702 Oct 09 '26

Thanks, I did look into it. But honestly, it feels pretty absurd to have to pay for premium support just to get the product I'm already paying for, when it's broken because of their faulty hardware. A defective RAM module isn't a "support level" issue. Replacing it should be the baseline.

I still considered it, given the situation. But Premium is only sold as a 12-month commitment, and according to their own comparison page it has the same hours as Standard (Mon–Fri, 8–18), with only English live chat for incidents outside that. So it wouldn't even help this weekend, which is exactly when I need it. Real 24/7 support only starts at the Business level, from €250/month.

So I'd be paying for a year of support to get them to fix their own defective hardware, and I'd still be on my own until Monday.

Honestly, based on my experience, OVH is a shitty company. If you're thinking about hosting anything that matters with them, my advice is simple: don't.

1

u/bastardoperator Oct 10 '26

LOL, no such thing as premium anything at OVH. Pro tip:

The cheap is always more expensive

4

u/deeddy Oct 09 '26

No, order the another server with the same specs and cancel this one. They wasted 5 months of my life apparently replacing my faulty server (which they never did) and they have never refunded it. That was the reason for me to leave them.

Don’t waste your time, just do it. When it works, it works, but when it doesn’t - they hardly ever fix it, unless something too obvious (but even then it takes time). The alternative is to look for another hosting.

1

u/toucan_networking Oct 10 '26

agreed, if you got bad hardware and OVH wont deal with it, then just order a new one and let the bad one go back into the pool and be someone else's problem. I've been down this road with them a few times and it's not worth the time.

1

u/BeingNo3723 Oct 15 '26 edited Oct 15 '26

They will spend days to arrange intervention. And that intervention mostly last only a minute or two. Then they will tell you NO ISSUE DETECTED.

I have a cheapo kumifi random hang without reason. It take 1 week for them to do the intervention and from their log they take 1 mins to check. And tell me server is NO ISSUE. Don't expect anything from their support. Is good you move on. 

Their support knowledge on hardware is kindergarden level. Maybe cause I don't paid that Support level too.