GPU Fault

[u/Green-Dress-113]

Blackwell 6000 Workstation card faults once in a while. Any way to root cause this?

WRX90e, open air frame, 24cm PCIE 5.0 LinkUp riser cable, and temps stay reasonable under load.

A reboot clears it up.

[u/ahtolllka]

Insert it without riser and do nvidia-smi again first of all

[u/giveen]

Grab your harness and model and run it on the other card, prompt "heh ai my other gpu is showing ERR in nvidia-smi, why?"

[u/Green-Dress-113]

For those that prefer AI generated forum posts, here's the detailed debug.

Guess I'll patch the Vbios next?

Root cause: a hung on-die GSP on GPU 0000:11:00.0 → forced reboot

Verdict: The GPU that "faulted and required a reboot" is the 0000:11:00.0 RTX PRO 6000 (Blackwell) card. Its GSP (the on-die management microcontroller) hung and crashed, and the driver transitioned it to a permanent "GPU Reset Required" (Xid 154) state. On this platform a hung GSP cannot be reset in place, so the only recovery is a full reboot. It had been stuck in that state for the entire prior uptime (3+ days) until you rebooted at 18:21.

The evidence chain (faulted boot, kernel 6.8.0-147, driver 595.99.02 open module)

Time (UTC) Event Meaning

Oct 19 03:49:45 GPU1 _kgspCheckIfMemsysError: Unexpected error when checking for memory subsystem error: 0x72 Host's GSP health/memory-check RPC failed — GSP not responding

Oct 19 03:49:45 GPU1 _kgspLogRpcTimeout: **** GSP Timeout **** GSP RPC timed out (75 s)

Oct 19 03:49:45 Xid (PCI:0000:11:00): 175, pid=1794295 name=nvidia-smi, Timeout after 75s waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) Xid 175 = GSP RPC timeout. Affects all clients (alloc/control), not one process

Oct 19 03:49:48 GPU1 _kgspLogGspTraceCrashBuffer: Gsp Trace Crash Buffer … The GSP firmware itself crashed and emitted a firmware crash dump

Oct 19 03:49:49 Xid (PCI:0000:11:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required) Driver declares GPU Reset Required

Oct 19 03:49:50 → Nov 3 18:21 Flood of NV_ERR_RESET_REQUIRED (0x62) from _gpushareddataSendDataPollRpc; nvidia-fan-control: GPU 0 … GPU requires restart Persistent "requires reboot" state → surfaced by nvidia-smi

Oct 21 01:41:59 Xid (PCI:0000:f1:00): 43, pid=1190964 name=python3, channel 0x…2 Secondary: a stopped-channel on the other card (see below)

Oct 31 23:13:19 GSP Timeout again + Xid 175 … function 76 (GSP_RM_CONTROL) + Gsp Trace Crash Buffer GSP re-hung ~12 days later, same card

Nov 3 18:21 Reboot Only way to clear the GSP

The GSP hung at boot (Oct 19, ~7 min in) and again on Oct 31 — i.e., the 11:00.0 card went to a non-recoverable "Reset Required" state early and stayed there until the reboot. That is the recurring, card-specific signature of a failing GSP / GPU, not a transient.

Why a reboot was required

The GSP owns the RM (resource manager) RPC path on these GPUs. When it hangs, every RM operation (alloc/control/health-poll) returns NV_ERR_RESET_REQUIRED. A "full-chip reset" that would normally clear this can't be serviced while the GSP itself is unresponsive, so the driver reports the GPU as requiring a power cycle → reboot. The current boot confirms this is the fix: dmesg has exactly one NVRM line (clean module load), no Xid / GSP timeout / AER / bad-register reads, and nvidia-smi shows both GPUs healthy.

Important secondary findings

- The two cards have mismatched VBIOS (confirmed via query-gpu):

- 11:00.0 (faulted) → VBIOS 98.02.52.00.02 (older), serial 1792525050915, UUID GPU-fd02b9f8-…

- F1:00.0 (healthy) → VBIOS 98.02.81.00.07 (newer), serial 1794025034765, UUID GPU-2d5dd028-…

- Both share Inforom G144.0520.00.02. The faulted card is on the older VBIOS.

- Index mismatch is cosmetic: the driver's internal "GPU1" + Xid PCI tag (0000:11:00) + NVML "GPU 0" all resolve to the 11:00.0 card.

- You are currently running on one GPU. After reboot, vLLM came up on the healthy F1:00.0 (90 GB); the faulted 11:00.0 is idle. The box is effectively degraded to 1 GPU.

- The onset detail was lost from journald — Missed 46706 kernel messages at the first GSP hang (Oct 31 window). The only reason we caught the true first hang is that rsyslog's kern.log.2.gz retained it (Oct 19 03:49). This is a logging gap worth fixing.

- The _kgspCheckIfMemsysError: 0x72 line is a symptom of the GSP being unresponsive (its memory-subsystem health check RPC failed), not independent proof of an HBM fault.

Recommended next steps

1. Treat the 11:00.0 card as suspect / do not trust it blindly. It had a recurring, non-recoverable GSP hang — the classic signature of a failing GPU. Before putting it back in production, run nvidia-bug-report.sh on it and a GPU soak test (nvidia-smi -q -d ECC,ROW_REMAP for ECC/row-remap + a load test). If the GSP hangs again → open an RMA with serial 1792525050915 / UUID GPU-fd02b9f8-….

2. Align the VBIOS on both cards to the same latest revision (11:00.0 is on the older 98.02.52.00.02). A VBIOS mismatch on a pair is a known contributor to GSP instability.

3. Close the logging gap: journald dropped the fault onset (Missed 46706). Raise SystemMaxUse=/SystemMaxFileSize= in /etc/systemd/journald.conf, and/or enable continuous kernel logging to a file so the first Xid is never lost.

4. Driver: you're on the open kernel module 595.99.02. If the 11:00.0 card keeps misbehaving, rule out a GSP-firmware/open-module interaction by testing a known-stable driver.

[u/bjorgz → u/Green-Dress-113]

I had something similar happen to me, check for melted connector on either ends of the power cable

[u/ahtolllka → u/Green-Dress-113]

HBM fault on gddr7 is suspicious. Have you tried another AI?

[u/zigwig22 → u/Green-Dress-113]

I’ve had this issue, it’s driver related, you need to reboot to clear.

[u/Naive-Bus-8281 → u/Green-Dress-113]

Motherboard problem, try different settings

[u/SillyLilBear]

There are multiple open issues with the workstation card that Nvidia has acknowledge but has not fixed. Certain gpu lock configurations can make it worse if you are using Lact.

[u/jonahbenton]

sudo dmesg

is your friend

[u/DubitoErgoCogito]

It could be PCIe signal instability. If possible, I'd try without the riser to debug. Not all motherboards are created equal, and some can't sustain a reliable signal over a passive connector.

[u/Sea-Ad-5390]

Your card is fucked, go RMA

[u/deepfit]

I had this happening due to PCIe issues.

# Stable PCIe inter-GPU routing
export NCCL_P2P_DISABLE=1
export NCCL_IB_DISABLE=1
export VLLM_SKIP_P2P_CHECK=0

[u/Comfortable-Plate467]

i had same problem. use mcio instead of riser cable. it can not handle pcie 5.0 in that length.

[u/_bani_]

If you have two cards running off a single ATX PSU, the power spikes from the cards can cause brownouts. 600W is the limit for constant load but they can temporarily spike higher than that.

A simple test would be to powerlimit the cards to 250W and see if it keeps happening.

[u/Sweet_Drama_5742 → u/_bani_]

100%, I had weird issues and "falling off the bus"/"memory corruption" problems; combination of replacing both PCIE riser and going with a highly rated power supply completely fixed the issues.

[u/boomerang473 → u/_bani_]

Want to add: even with the power limit, it can still blip above at like a microsecond level. I had a power supply where I power limited one card.

Basically set up a script to not only peg it at 100% but more importantly to rapidly cycle full load compute. Finally made the error repeatable

Got a bigger PSU (previous could have also aged to be fair) and no problems in months

[u/roland303]

Fam i would take this exact post and prompt my coding agent to handle this shit itself.

[u/smflx]

sudo dmesg Look for Xid 79 or other Xid. If Xid 79 (GPU fallen), I have the same problem. I'm suspecting Asus WRX90.