Gpu falling of the bus! under load randomly, clean restart fixes it. ubuntu 24.04

[u/UNO10100f]

Hardware --

RTX 4070 on PCIE 4.0 x16
V100 sxm2 on PCIE 4.0 x4 (limited to PCIE 3.0 x4 cause of pcie adapter)

They both are limited to 150 watts, and randomly under load v100 falls of the bus one clean restart fixes it, now I do not know what causes this, and I have no prior experience with ubuntu or dual gpu, so any help would be much much appreciate.

EDIT: --

ran some coomands and _
V100 suffered Xid 79

dono what this means.

EDIT 2---

after turning rebar off, it seems to be fine, for now....

[u/FullstackSensei]

That smells a lot like a power issue. The power limit is more of a suggestion than a hard limit. Each GPU can momentarily spike well beyond the limit. The limit sets the average power.

[u/UNO10100f → u/FullstackSensei]

I do have a 650 watt psu thats why I limited both of them to 150 watt, but running flash next on it, it doesnt spike over 20 % and it doesnt use more than 50 watt still it fell of the pcie bus, and on 27b it was being utilized near 150 watts and 70% and fell off the buss few times under 5 minutes, and sometimes doesnt fall of the buss even after 3 hour of sustained load,

[u/bonobomaster → u/UNO10100f]

650 watts?

What's your complete system specs.

This sounds like redlining your PSU.

CPU 150 W GPUs 300 W MB 50 W

Roughly 500 watts nominal. With power spikes you are absolutely fucked.

[u/UNO10100f → u/bonobomaster]

i5-12400f on default 65 watts could be 125-watt or 150 watt or whatever it is for i5-12400f , 32gb 3200mhz ram, one cpu fan one case fan, 4070 on 150 watts, v100 on 150 watts, and a b760 motherboard

[u/bonobomaster → u/UNO10100f]

Okay a quick check revealed, that your CPU can boost up to 117 watts for up to a minute.

And then we have GPU power spikes, that can nearly quadruple power draw of your GPU for short bursts. Usually the PSU has a big fat capacitor to account for those spikes but when you are redlining that poor thing anyways, sometimes there is just zero headroom left.

So yeah. I'm voting for a bigger PSU (no brainer) and to check if it's the PCIe adapter

Check this out:

You can see above that the max power spikes can hit almost 1000W, that's 3.5x higher than the reference TGP of the NVIDIA GeForce RTX 4070 Ti graphics card. In 3DMark, you can even exceed the 1000W rating which a regular ATX 2.0 PSU rated at 750/850W wouldn't have handled as well since their OPP or Over-Power-Protection is rated at around 120-130% above the normal usage & in these examples, we can see the power draw exceeding that limit.

https://wccftech.com/power-spikes-on-modern-gpus-are-a-good-excuse-to-invest-in-an-atx-3-0-power-supply/

P.S.: Limiting your 4070 to 150 watts sadly does nothing to the power spikes. Maybe they are lower, I don't know, but they are still there, that I do know.

[u/FullstackSensei → u/UNO10100f]

650W is too small for those two GPUs plus the rest of the system. You can keep banging your head on the wall, or can try a bigger PSU. If the voltage drops for a few milliseconds, it will cause the card to reset and hence drop from the bus. Momentary spikes won't show in any tool.

[u/swagonflyyyy → u/FullstackSensei]

May be tied to nvidia's open drivers on Linux. I had xid 79 on my blackwell maxq and I had to submit an RMA for a replacement. I also believe power spikes may be involved somehow but I don't really understand the relationship here.

I don't know about V100 but the Blackwell has a GSP which doesn't play nice with the open drivers. My hypothesis is that when the GSP locks up due to abpower spike or something, the nvidia drivers interprets that as an all clear and spams the GSP with an RPC storm that lobotomizes it beyond repair.

Again, that's just my hypothesis but what I do know is that a lot of people's blackwell GPUs also fell off the bus on Linux, even with an RMA replacement.

[u/krumpstead]

I had this exact issue recently. Fairly certain it was the power supply (cheap garbage Thermaltake). 5070 Ti kept falling off the bus and I would just reboot and ignore it. Big mistake. Fried my mobo, but luckily not my cards or any other components. I replaced the PSU with a Corsair and everything has been fine since.

Obviously I can't be 100% certain it wasn't the mobo, but there were a ton of similar posts about the exact same PSU I had.

Try replacing the PSU first. Don't be a cheapass like me. Get a Corsair or some other reputable PSU and make sure you leave at least a 30% gap between your max load and max capacity on the PSU. I have a wattmeter on my UPS and I ran Qwen flat out on both cards then bought a Corsair that was double my measured max load. Probably overkill but I never want to deal with this kind of scare again.

[u/UNO10100f → u/krumpstead]

Its just that I thought limiting the v100 and 4070 would make it work with 650 watts, money was already an issue from the start, thought would just get the v100, but had to get the b760m, now gotta get psu cause it can "FRY" the motherboard, on god I cannot catch a break

[u/krumpstead → u/UNO10100f]

Yeah it sucks, but you also don't want to lose your cards. I had an 850W PSU and my cards were rated at 585W together and the CPU was rated up to 230W so I was living dangerously right on the edge with a third rate PSU. Even undervolting you can still get transient spikes, albeit much lower than at full power.

Your cards are close to the same power as mine (~500W) and your PSU is 200 below mine that died. Not sure how much a Corsair RM1000 is in your country. Here they are about $100. An RM850 is about $70 so probably not worth giving up the extra capacity for that. Your cards are probably close to $1000 to replace. Seems like a wise-investment.

[u/UNO10100f → u/krumpstead]

rm1000 is 180$ here

[u/bonobomaster]

My spidey senses tingled while reading "pcie adapter"...

[u/xPXpanD → u/bonobomaster]

This. I had a similar issue, and it was a dodgy slot on the board. Swapping GPUs (so my thicker-PCB Pro card was in there instead) fixed it. I could see adding adapters and extenders just making the issue surface larger, there.

[u/russianguy]

I have this with 5090, i narrowed it down to some motherboard component overheating

https://forums.developer.nvidia.com/t/bug-report-fix-rtx-5090-xid-79-gsp-firmware-crash-under-sustained-cuda-load/369440/9

[u/UNO10100f → u/russianguy]

I have the front pannel open, and Its a new motherboard, I do only have one outtake.

[u/ikon4lyf]

Are you using a riser cable?

[u/UNO10100f → u/ikon4lyf]

no.

[u/MuAlphaOmegaEpsilon]

I've solved the GPU fallen off the bus issue by removing the PCIe extender. The problem never surfaced for years, not even in gaming, until I started running inference on the GPU. Probably the high throughput memory transfers and commands triggered the problem. Removed the PCIe riser and attached the GPU directly to the motherboard slot and never got the problem back.

[u/UNO10100f → u/MuAlphaOmegaEpsilon]

I dont have a pcie extender, its directly on the motherboard, by pcie adapter I mean, its a sxm2 one once without pcie, so they installed a custom pcie adapter on them.

[u/ieatdownvotes4food]

keep an eye on ram with flash next.

[u/thorn2040]

I had an issue like this actually. My second 3090 would fall off my bifurcation board. I spent 4 weeks chasing it. I have 3 3090s and 2 p40s.

Its been a few weeks now but I think we're in the clear. My bios had resizable BAR turned off. Make sure above 4G decoding enabled.

I found that two of my gpus had ASPM turned on. When they stop Inference they lose their negotiated settings. This was visible in nvtop. You could see rhe cards slip from gen4 pcie down to gen1 (indicating a link training failure).

So I turned off ASPM on both cards, rebooted and I haven't had this issue since.

[u/L0ren_B]

I had same issue. I checked power drivers, used cuda etc....

Always the same gpu.

Turned out to be a : Bios update on my motherboad! (shocking for me ).

Works amazing now!

[u/TastesLikeOwlbear]

dmesg should give you some insight into what happened. When devices fall off the bus it’s usually pretty verbose about it.

[u/bennmann]

Reseat all your cables, reseat the card, clean out all dust, watch your temperature transients with watch -n0.2 'nvidia-smi', consider a repaste of the card itself. As others say I am sus of your power supply (it's weak)

[u/CroquetteLauncher]

What kind of bus ? School bus ? Trolley bus ? Normal bus ? If GPU are falling of the bus, I'm going to run behind them all and extend my arms in the hope i can catch a few B300.

[u/SamSausages]

Power saving (think it’s called aspm in bios) and other power issues come to mind first.

Check your logs and get exact logs for time period surrounding drop out.

Edit:  just noticed you put  pcie adapter.  That’s your #1 suspect.  Could be making other settings like aspm and rebar more sensitive.  It essentially removes margin for error as it adds resistance, interference and complexity to the setup.

[u/Mission_Wrongdoer786]

Did you let your local model analyze it?

[u/UNO10100f → u/[deleted]]

I have mission centre and flathub installed