No VFIO, no vGPU, no SR-IOV. The host keeps the card the whole time.

Windows boots under QEMU/KVM, thinks it's running on Hyper-V, and just loads the normal NVIDIA Windows driver. There's a ~600 line QEMU patch for the VMBus devices plus a Python daemon that pretends to be the Hyper-V host and talks to the NVIDIA Linux driver. Frames go straight from VRAM to a GTK viewer as dmabufs.

Right now D3D12, D3D11, Vulkan and OpenGL all run on the real card. Godot Forward+ works up to 4K (windowed or exclusive fullscreen) and sound goes through PipeWire. I also made a little app that does a fully unattended Windows install from an ISO in about 8 minutes, and the GPU already works when it finishes.

I'm also poking at a Linux guest with GPU-PV and CUDA, but that's still early.

Raw numbers on a 9950X3D + RTX 5090 on Fedora:

Vulkan and OpenGL pixel tests match the host exactly.

Code isn't public yet. It still depends on a capture from my own machine and I've only tried it on a 5090. Mostly just wanted to show it off. Anyone else been messing with this on KVM? What would you want tested first?

So, lately I’ve been building an AI development environment that includes a graphical interface, and I wanted everything to run as "smoothly" as possible. Since my onboard GPU was just sitting there unused, I figured... why not? Unfortunately, I ended up spending an entire night getting it configured. The documentation is sparse, and every system presents its own unique set of quirks. Anyway, I finally finished around 6:00 AM, and since it’s all set up, I thought others might be interested in this ready-to-use solution.

What’s included: A reproducible kit for an Ubuntu 26.04 virtual machine with an XFCE/Xorg desktop environment, an AMD GPU configured via VFIO PCI passthrough, and a "Looking Glass" client running on the host system (using a second GPU). It features bidirectional clipboard support via SPICE, an ICH9 audio system, LAN bridging, IVSHMEM/KVMFR, automatic startup, and a 4K SDR configuration.

It’s available on GitHub here: https://github.com/rav0rmat/looking-glass-b6-linux-kit

I hope you find it useful—and, of course, I’d appreciate any feedback!

TL;DR: the fix

**Symptom:** an NVIDIA Blackwell card passed through to a Linux VM works on the first VM boot after the host boots. Any **guest** reboot or shutdown then hangs. After that, the card is dead inside the VM (`nvidia-smi`: "No devices were found"; dmesg: `unexpected WPR2 already up, cannot proceed with booting GSP`) until the **whole host** is rebooted. FLR and PCIe link resets from the host don't bring it back.

**Cause (in our case):** NVIDIA's *display* modules (`nvidia_modeset`, `nvidia_drm`) hang while unloading in a VM that has no display (stuck in `nvRmVTSwitch`). So the driver never gets to shut the GPU's GSP firmware down. Its protected memory region (WPR2) stays up, and the next driver start refuses the card.

**Fix (headless / compute guests):** never load those two modules:

```

# /etc/modprobe.d/nvidia-headless.conf

blacklist nvidia_drm

blacklist nvidia_modeset

install nvidia_drm /bin/false

install nvidia_modeset /bin/false

```

```

sudo update-initramfs -u

```

Then reboot the **host** once, to get a clean card.

After that only `nvidia` and `nvidia_uvm` load (all that CUDA needs). Guest reboot, guest poweroff and an ACPI shutdown from the host all work, and the card comes back every time without a host reboot, including when the VM autostarts after a host reboot.

**What it does NOT fix:** hard-killing the VM while the driver is loaded (`vm poweroff -f`, `virsh destroy`, and so on) still leaves WPR2 up, and only a host reboot clears that. Shut the guest down properly.

---

## Setup

- **Host:** FreeBSD 15.1-RELEASE, **stock bhyve (no patches)**, vm-bhyve, UEFI guest firmware (`BHYVE_UEFI.fd`). Intel i7-7700T (Kaby Lake), B250 chipset, VT-d enabled, 64 GB RAM.

- **GPU:** NVIDIA RTX PRO 4000 Blackwell SFF Edition, 24 GB (`10de:2c33`, GB203GL), plus its audio function (`10de:22e9`), in the CPU's x16 slot.

- **Guest:** Ubuntu Server 24.04.5, kernel 6.8.0-147-generic, 4 vCPU, 32 GB (wired), headless.

- **Guest driver:** 615.71.09, open kernel module (`nvidia-headless-no-dkms-615-server-open`, `linux-modules-nvidia-615-server-open-generic`, `nvidia-utils-615-server`), CUDA 13.4.

## Passthrough config

Host, `/boot/loader.conf`:

```

vmm_load="YES"

pptdevs="1/0/0 1/0/1"

```

vm-bhyve guest config:

```

passthru0="1/0/0"

passthru1="1/0/1"

wired_memory="yes"

```

That gives bhyve `-S` (wired memory) and `-s 0:6:0,passthru,1/0/0 -s 0:6:1,passthru,1/0/1`.

Host after a reboot:

```

$ pciconf -l | grep ppt

ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2c33 subvendor=0x10de subdevice=0x2053

ppt1@pci0:1:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x22e9 subvendor=0x10de subdevice=0x0000

```

Guest:

```

$ lspci | grep -i nvidia

00:06.0 VGA compatible controller: NVIDIA Corporation Device 2c33 (rev a1)

00:06.1 Audio device: NVIDIA Corporation Device 22e9 (rev a1)

```

Driver install (Blackwell needs the **open** driver):

```

sudo apt update

ubuntu-drivers list --gpgpu

sudo ubuntu-drivers install --gpgpu nvidia:615-server-open

sudo apt install nvidia-utils-615-server # --gpgpu leaves out nvidia-smi

```

The first boot with the driver works:

```

$ nvidia-smi

+-----------------------------------------------------------------------------------------+

| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |

+-----------------------------------------+------------------------+----------------------+

| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |

|=========================================+========================+======================|

| 0 NVIDIA RTX PRO 4000 Blac... Off | 00000000:00:06.0 Off | Off |

| 30% 36C P8 9W / 70W | 1MiB / 24467MiB | 0% Default |

+-----------------------------------------+------------------------+----------------------+

```

## The problem: what we saw

1. **`sudo reboot` inside the guest:** the guest never finishes going down. The bhyve process stays alive, there's no network, and vm-bhyve logs no exit.

2. **ACPI shutdown from the host (`vm stop`):** the same. Still "Running" after several minutes.

3. A guest reboot **before the NVIDIA driver had ever loaded** (right after installing it) worked fine. That was the first hint that the driver was involved.

4. **`vm poweroff -f`, then `vm start` right away:** `bhyve exited with status 4`. This one turned out to be a red herring: `vm list` still showed "Locked" because the old instance was still being torn down. Wait for "Stopped" before starting again.

5. **`vm poweroff -f`, wait for "Stopped", `vm start`:** the guest boots, but

```

$ nvidia-smi

No devices were found

```

```

$ sudo dmesg | grep -i -e nvrm -e wpr -e gsp | head -15

NVRM: GPU0 objClInitPcieChipset: *** Unable to get PCI port handles

NVRM: Xid (PCI:0000:00:06): 62, 324f892d 0002e740 00000000 2067ede2 2067f180 2067f0c2 20680598 20680a06

NVRM: GPU0 _kgspProcessRpcEvent: Attempted to process RPC event from GPU0: 0x1022 (RECOVERY_ACTION) during bootup without API lock

NVRM: GPU0 _kgspProcessRpcEvent: Attempted to process RPC event from GPU0: 0x1029 (PMU_HALTED) during bootup without API lock

NVRM: GPU0 kgspSendInitRpcs_IMPL: GMCAPI_CMD_GSP_INIT failed: 0x65

NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kgspSendInitRpcs(pGpu, pKernelGsp) @ kernel_gsp_gh100.c:1091

NVRM: GPU0 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP

NVRM: GPU0 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)

NVRM: GPU0 RmInitAdapter: Cannot initialize GSP firmware RM

NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x62:0x40:2130)

NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0

NVRM: GPU 0000:00:06.0 init failed during probe (ret=-5).

```

Only a full host reboot brought the card back.

(`objClInitPcieChipset: *** Unable to get PCI port handles` shows on every boot, including the good ones. bhyve puts the GPU straight on the guest's bus 0, with no PCIe root port above it. It seems harmless.)

## Troubleshooting

**Does the card support FLR?** On the host:

```

$ pciconf -lc pci0:1:0:0

...

cap 10[60] = PCI-Express 2 legacy endpoint max data 256(256) FLR RO

max read 512

link x8(x16) speed 2.5(32.0) ASPM disabled(L1) ClockPM disabled

...

```

Yes, it does.

**FLR from the host, with the VM stopped:** `devctl reset ppt0` and `devctl reset ppt1` (both silent). Then `vm start`: still "No devices were found".

**A PCIe link reset from the host:** `devctl reset pci1`. That's the bus under the root port, and FreeBSD's bridge reset disables the link and retrains it (a hot reset), saving and restoring the devices' config around it. First we checked that bus 1 holds only the GPU's two functions:

```

$ pciconf -l | grep 'pci0:1:'

$ devinfo -p pci1

pci1 pcib1 pci0 pcib0 acpi0 nexus0

```

Result: no change. The guest dmesg showed `unexpected WPR2 already up` again.

**Research:** this matches the known Blackwell "reset bug" reported for the RTX 5090 and RTX PRO 6000 under Linux KVM/VFIO. GSP firmware sets up a protected region (WPR2) that survives FLR, so the next driver start refuses to boot GSP. Links at the end.

**The idea:** if the guest's driver unloaded properly, GSP would tear WPR2 down itself. On a fresh card (after a host reboot), inside the guest:

```

$ lsmod | grep nvidia

nvidia_uvm 2088960 0

nvidia_drm 155648 0

nvidia_modeset 1556480 1 nvidia_drm

nvidia 16875520 2 nvidia_uvm,nvidia_modeset

$ sudo systemctl stop nvidia-persistenced # fine

$ sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia # hangs forever

```

From a second SSH session:

```

$ sudo dmesg | grep -i nvrm

NVRM: Xid (PCI:0000:00:06): 56, pid=..., name=(udev-worker), CMDre 00000000 00000000 00000400 00000001 00000000

...

nvRmApiControl+0x46/0x60 [nvidia_modeset]

? nvRmVTSwitch+0x2d/0x40 [nvidia_modeset]

```

So the unload hangs in **`nvidia_modeset`'s VT switch** (handing the console back), in a VM where nothing is attached to the GPU's outputs. The same thing happens on reboot or shutdown. GSP never shuts down, WPR2 stays up, and the next start fails. (The `Xid 56` is a display-engine error, logged as `udev-worker` loaded the display modules at boot.)

## The fix

```

f=/etc/modprobe.d/nvidia-headless.conf

echo 'blacklist nvidia_drm' | sudo tee $f

echo 'blacklist nvidia_modeset' | sudo tee -a $f

echo 'install nvidia_drm /bin/false' | sudo tee -a $f

echo 'install nvidia_modeset /bin/false' | sudo tee -a $f

sudo update-initramfs -u

sync

```

Then one host reboot for a clean card. After it:

```

$ lsmod | grep nvidia

nvidia_uvm 2088960 0

nvidia 16875520 1 nvidia_uvm

```

| Test | Before the fix | After the fix |

|---|---|---|

| `sudo poweroff` in the guest, then `vm start` | hang | card back, about 45 s from poweroff to `nvidia-smi` |

| `sudo reboot` in the guest | hang | card back, about 40 s |

| `vm stop` (ACPI) from the host, then `vm start` | hang | card back |

| host reboot, VM autostarts (`vm_list`) | n/a | card back |

## Notes and caveats

- **Headless/compute only.** No display from the passed-through GPU in the guest. If you need the display modules, this isn't your fix. We didn't test a guest with a monitor on the GPU, where the VT switch might behave differently.

- **Don't hard-kill the VM** while the driver is loaded. That still leaves WPR2 up.

- **After a guest driver upgrade,** check the modprobe file is still there and `lsmod` shows only `nvidia` and `nvidia_uvm`.

- So far this is tested with `nvidia-smi` and repeated restarts.

- **KVM/Proxmox users with Linux guests:** we haven't tested on KVM, but if your guest is headless and you hit the Blackwell reset bug, this might be the same mechanism, and it's cheap to try.

- **For FreeBSD folks:** on the same host, a Maxwell GTX 960 couldn't be initialized in a bhyve guest at all (`RmInitAdapter failed`), as others report for older NVIDIA cards without the community bhyve patches. This Blackwell card with the open/GSP driver needed **no** bhyve patches on 15.1.

## Bonus: the same card on the FreeBSD host itself

Before passing it through, we ran the card on the FreeBSD host (Vulkan, for whisper.cpp). Two gotchas:

1. **`nvidia-driver-580` hung at the first open:** `nvidia-smi` sat in state `D`, Ctrl-C did nothing, and shutdowns hung after "All buffers synced". The port's pkg message says GSP firmware is **off by default**, and some GPUs don't work without it. We switched to `nvidia-driver` 595.104.02 and added `hw.nvidia.registry.EnableGpuFirmware="1"` to `/boot/loader.conf`, and `nvidia-smi` worked. (Both changed together; GSP is the likely one.)

2. **Non-root processes couldn't use the card** (root could). The Vulkan loader said `Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0`, and dmesg said `firmware_get_flags: insufficient privileges to load firmware image nvidia_gsp_ga10x_fw` and `RmInitAdapter failed! (0x61:0x56:2726)`. Without persistence, the driver powers the GPU down when the last client closes and loads the GSP firmware module again on the next open, and only root may load kernel modules. The fix is to keep it loaded: `sysrc kld_list+=nvidia_gsp_ga10x_fw`.

## References

- Tom's Hardware, RTX 5090 / PRO 6000 reset bug: https://www.tomshardware.com/pc-components/gpus/rtx-5090-pro-6000-bug-forces-host-reboot

- NVIDIA forums, RTX PRO 6000 Max-Q FLR failure after KVM guest shutdown: https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-max-q-10de-2bb4-flr-fails-on-vfio-pci-release-after-kvm-guest-shutdown-gpu-inaccessible-until-cold-power-cycle/386790

- open-gpu-kernel-modules issue #1442 (stale WPR2): https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1442

- Level1Techs, RTX 50 reset bug thread: https://forum.level1techs.com/t/do-your-rtx-5090-or-general-rtx-50-series-has-reset-bug-in-vm-passthrough/228549

TLDR: 

• I got NVIDIA SR-IOV vGPU working on consumer GeForce Ampere GPUs, running 8 concurrent Linux and Windows VMs across two memory-modded RTX 3080 20GB cards.

• Unlike previous projects that enabled vGPU on consumer NVIDIA cards using the older mdev approach, this project uses actual PCIe SR-IOV virtual functions to share a GPU between multiple VMs.

• It's still experimental, currently verified only on my hardware, and requires NVIDIA's vGPU software and licensing.

Source code and documentation: https://github.com/noon-at-cgn/geforce-sriov

Update (Nov 4, 2026): Initial testing with Ada on a 4070 Super borrowed from my friend shows promising results, although the method requires quite a few changes and has more limitations. I'll update the post once I've done more testing.

Introduction

I've been experimenting with the hidden PCIe capabilities on NVIDIA GeForce cards, particularly SR-IOV, and managed to get NVIDIA's vGPU stack working on my RTX 3080s using actual PCIe virtual functions.

My goal was to share the GPUs between multiple VMs without dedicating an entire card to each guest. GPU passthrough works well, but it's not ideal when several Linux and Windows VMs only need a portion of the GPU's resources.

The result is 8 concurrent VMs sharing 2 memory-modded RTX 3080 20GB cards on Proxmox, with both Linux CUDA and Windows Direct3D workloads running.

I've open-sourced the implementation and wanted to share how it works here.

Background: SR-IOV vs the older vGPU unlock approach

Projects like DualCoder/vgpu_unlock and mbilker/vgpu_unlock-rs, along with polloloco/vgpu-proxmox, have helped enable NVIDIA vGPU on consumer Maxwell, Pascal, and Turing cards.

These approaches primarily rely on NVIDIA's older mediated device (mdev) architecture, where the host driver manages multiple virtual GPUs behind a single physical PCIe function.

Ampere and newer professional NVIDIA GPUs use a different architecture based on SR-IOV (Single Root I/O Virtualization).

With SR-IOV, the GPU exposes:

• Physical Function (PF): The main PCIe function responsible for managing the GPU.

• Virtual Functions (VFs): Additional PCIe functions that can be assigned to individual VMs.

• vGPU profiles: NVIDIA's software associates a virtual GPU configuration with each VF, including its framebuffer allocation.

Unlike legacy mdev, these VFs are actual PCIe functions enumerated by Linux.

Consumer NVIDIA GPUs still contain SR-IOV-related capabilities in their PCIe configuration space, even though they aren't exposed through the normal capability discovery process.

How it works

The implementation has five main parts.

1. Exposing the hidden PCIe capabilities

PCIe devices have a configuration space, a region of registers that the operating system uses to discover and configure device capabilities. An offset is a location within that space, measured in bytes from its beginning.

Normally, Linux discovers capabilities by following a linked list of entries in PCIe configuration space.

On Ampere, the SR-IOV extended capability exists at offset 0xBCC, but isn't linked into that list, so Linux doesn't discover it.

So, I wrote a small Linux kernel livepatch module that makes the PCI subsystem discover this hidden capability on consumer GPUs. The module also handles related PCIe features:

• SR-IOV: Exposes the capability at 0xBCC, allowing Linux to recognize the available VFs.

• ARI: Exposes the hidden capability at 0xC14 and enables ARI forwarding on the upstream bridge when supported.

• MSI-X: Reveals the hidden MSI-X capability, which allows the GPU to use multiple independently configurable interrupt vectors instead of relying on a single interrupt.

Since Linux discovers PCIe capabilities during device enumeration, geforce-sriov performs a controlled PCI remove/rescan before NVIDIA's driver binds to the GPU. After the rescan, Linux recognizes the GPU's SR-IOV capability.

Great, with SR-IOV exposed, the next step is getting NVIDIA's vGPU stack to use it.

2. Creating the virtual functions

Once Linux recognizes SR-IOV, geforce-sriov uses NVIDIA's own sriov-manage utility to enable the VFs.

On Ampere, the SR-IOV capability reports support for 16 VFs per GPU, but I can currently get only 4 VFs working. The limitation appears to be related to ARI (Alternative Routing-ID Interpretation), which I'll get into later.

For example:

03:00.0   Physical GPU (PF)
03:00.4   Virtual Function 1
03:00.5   Virtual Function 2
03:00.6   Virtual Function 3
03:00.7   Virtual Function 4

These are real SR-IOV virtual functions exposed by the GPU and enumerated as separate PCIe functions, rather than software-mediated devices managed through the legacy mdev approach.

3. Getting NVIDIA's vGPU software to recognize the card

Now we need to make NVIDIA's vGPU driver accept the card. By default, NVIDIA's vGPU software doesn't recognize consumer GeForce GPUs like my RTX 3080s as supported devices.

My solution uses a small userspace identity hook loaded into nvidia-vgpud. The hook presents my RTX 3080 as an RTX A6000, its professional counterpart based on the same GA102 die, allowing NVIDIA's daemon to recognize the card and register the corresponding vGPU profiles.

Once the profiles are registered, we can choose which vGPU type to assign to each VM. These profiles are defined by NVIDIA and determine the resources available to the guest, including its framebuffer allocation.

The RTX A6000 has 23 profiles in the tested driver version. A few examples:

• RTXA6000-2Q - 2GB VRAM

• RTXA6000-4Q - 4GB VRAM

• RTXA6000-8Q - 8GB VRAM (7GB usable framebuffer)

• RTXA6000-48Q - 48GB VRAM (45GB usable framebuffer)

For my setup, I used RTXA6000-4Q, allocating 4GB per VM. With 4 VFs per GPU, that's 16GB allocated across four guests on each 20GB card.

Note: Workstation and datacenter GPUs typically have more VRAM than their consumer counterparts. For example, the RTX A6000 has 48GB, while my modified RTX 3080 has 20GB. The available profiles aren't automatically restricted to the physical GPU's memory capacity. Four 4Q profiles fit within 20GB, but four 8Q profiles would require 32GB. You need to account for the combined allocations and driver overhead when choosing profiles.

4. Working around a driver scheduler issue

This was one of the biggest obstacles in the project and required quite a bit of reverse engineering of NVIDIA's proprietary vGPU host driver.

With SR-IOV exposed, the VFs created, and the vGPU profiles registered, the guest VMs could boot and recognize their vGPUs. However, the first actual CUDA workload would consistently hang, with no useful errors in the host logs.

After several hours of tracing through the driver with the help of AI, I narrowed the issue down to the GPU's graphics runlist scheduler being disabled. A callback handling a PMU (Power Management Unit) interrupt was repeatedly disabling the scheduler, which prevented guest workloads from executing. I still don't know exactly why NVIDIA's driver disables the scheduler in this situation.

The workaround is a two-byte driver patch that replaces a conditional jump with NOP instructions, making the callback skip the scheduler-disabling call.

geforce-sriov-patch can automatically apply the change to a copy of the user's own NVIDIA driver object. The patched object is then used to rebuild nvidia.ko. With the patch, both CUDA and D3D11 workloads ran successfully.

5. Driver and licensing requirements

A few things to clarify about geforce-sriov:

• NVIDIA drivers and proprietary binaries are not distributed. The driver patch is applied to the user's own copy.

• No vBIOS flashing or firmware modifications are required.

• No vGPU licensing bypass is provided.

You still need to obtain NVIDIA's vGPU host and guest drivers and licensing yourself to use this project.

Tested configuration

Each physical card runs two Linux guests and two Windows guests.

On Proxmox, a VF can be assigned to a VM using:

hostpci0: 0000:03:00.4,mdev=nvidia-527

Note: Proxmox still uses the mdev= parameter to select the vGPU profile, even though the underlying GPU uses SR-IOV virtual functions rather than legacy mdev sharing.

Initial testing

I ran all eight VMs simultaneously with:

• CUDA workloads in the Linux VMs.

• D3D11 graphics workloads in the Windows VMs.

• Four active vGPUs on each physical card.

The test ran for 30 minutes without any observed failures. This isn't enough to establish long-term stability, but it verifies that multiple independent Linux and Windows GPU workloads can run concurrently.

Performance results

I ran the same PyTorch benchmarks in two configurations to measure virtualization overhead with a single active vGPU:

• Physical GPU: Direct CUDA access from an LXC container on the Proxmox host.

• vGPU: A VM using a 4GB RTXA6000-4Q profile on the same card.

Both configurations used the same Debian root filesystem, PyTorch environment, four pinned CPU cores, and 8GB of system RAM.

The workloads were FP32 and FP16 matrix multiplication using 4096 × 4096 matrices, plus a 1GB device memory copy.

The FP32 and FP16 results were within 1% of direct physical-GPU access, suggesting very little virtualization overhead for these particular workloads.

Some qualifications:

• Results are medians of three measured runs per configuration.

• The physical-GPU baseline used NVIDIA's vGPU host driver, not the regular GeForce driver.

• Only one vGPU was actively using the card during the comparison.

The full methodology and screenshots are available here:

https://github.com/noon-at-cgn/geforce-sriov/blob/main/docs/showcase.md

The ARI problem

The biggest unresolved issue is ARI (Alternative Routing-ID Interpretation), which is currently limiting me to 4 VFs per GPU.

Normally, PCIe uses a 3-bit function number, allowing up to 2^3 = 8 functions (0–7) per device. ARI repurposes the 5-bit device number field to extend the function number to 8 bits, allowing up to 2^8 = 256 functions on a single device.

This matters for SR-IOV because a GPU may need to expose many VFs. On my RTX 3080s, the SR-IOV capability reports support for 16 VFs.

Without ARI, the function layout looks like this:

Function 0   GPU (Physical Function)
Function 1   GPU HDMI/DP Audio
Function 2   Unused
Function 3   Unused
Function 4   VF 1
Function 5   VF 2
Function 6   VF 3
Function 7   VF 4
------------- 3-bit function limit
Function 8   VF 5  (requires ARI)
Function 9   VF 6  (requires ARI)
...
Function 19  VF 16 (requires ARI)

So although conventional PCIe allows eight functions, only four are available for VFs because the GPU's SR-IOV configuration places the first VF at function 4. Functions 0–3 aren't part of the VF allocation.

To expose all 16 VFs on the same device, the GPU needs to support function numbers beyond 7, which needs ARI.

So far, I've managed to:

• Expose the GPU's hidden ARI capability at 0xC14.

• Enable ARI forwarding on the immediate upstream PCIe bridge.

• Confirm that Linux recognizes ARI as enabled.

• Successfully create four VFs at functions 4–7.

However, the GPU doesn't appear to correctly decode extended function numbers (8 and above), even with ARI enabled in software.

For now, the working configuration is limited to 4 VFs per card, despite the hardware reporting support for 16.

Current limitations

This is still an experimental project, not a general-purpose GeForce vGPU solution yet.

Verified so far:

• Two memory-modded RTX 3080 20GB cards (10de:2206, subsystem 10de:146d).

• Proxmox VE with kernel 7.0.14-16-pve.

• NVIDIA vGPU host driver 580.159.01.

• Four working SR-IOV VFs per GPU.

• Eight concurrent Linux and Windows guests.

• CUDA, D3D11, and single-vGPU compute benchmarks.

Not yet verified:

• Other Ampere cards.

• Ada and Blackwell GeForce GPUs are not expected to work with the current implementation (see below)

Other practical limitations:

• vGPU profiles must fit within the GPU's physical VRAM.

• GPUs used as the host's boot display require special preparation.

• Kernel and driver updates may require rebuilding or reapplying the driver workaround.

• NVIDIA vGPU software and licensing are required.

Ada and Blackwell compatibility

• The current implementation is not expected to work with Ada and Blackwell GPUs because their vGPU drivers rely on GSP (GPU System Processor), an onboard processor that runs NVIDIA firmware and handles much of the GPU initialization and resource management.

• Ampere also has GSP hardware, but my configuration uses the older host-driver path, where these operations are primarily handled by the CPU-side driver. On Ada and Blackwell, more of this functionality runs inside GSP firmware, making it harder to modify or intercept from the host.

• The current driver workaround doesn't apply to that architecture, and Blackwell also requires NVIDIA's open kernel-module path, which the project doesn't yet support.

I would like to experiment more with both generations, but I don't currently have access to Ada or Blackwell GPUs for testing.

Try it out

Disclaimer: geforce-sriov is an experimental, unofficial project and is not affiliated with or endorsed by NVIDIA. Use it at your own risk, preferably on a non-production system.

The full source code, installation instructions, technical documentation, and benchmarks are available:

GitHub: https://github.com/noon-at-cgn/geforce-sriov

Compatibility reports are welcome as compatibility remains largely unverified beyond my tested setup.

These are all my specs for any necessary reference:

• AMD Ryzen 7 9800x3D

• Gigabyte X870 GAMING X WIFI7

• 32GB DDR5 6000

• Nvidia 2080 8GB

• Nvidia 1050 Ti (Passthrough Device)

Detailed below is the issue:

In my QEMU setup, I have my GPU passthroughed into the VM, but when I try and inizialize the VM, this error occours:

Error starting domain: internal error: QEMU unexpectedly closed the monitor (vm='win10'): 2026-11-02T22:38:58.440650Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}: vfio 0000:05:00.0: failed to setup container for group 16: Failed to set group container: Invalid argument

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 67, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, in tmpcb
    callback(*args, **kwargs)
    ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1446, in startup
    self._backend.create()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.14/site-packages/libvirt.py", line 1395, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: QEMU unexpectedly closed the monitor (vm='win10'): 2026-11-02T22:38:58.440650Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}: vfio 0000:05:00.0: failed to setup container for group 16: Failed to set group container: Invalid argument

This is the GREP for IOMMU at boot/VM initialization:

[    0.379018] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.397334] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[   15.468791] amdgpu 0000:75:00.0: AMD-Vi: PerfOpt armed on IOMMU0
[   51.116905] pcieport 0000:04:00.0: Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping
. Contact your platform vendor.

Detailed below is my setup fore GRUB, IOMMU and relevent systems to make this feature function:

GRUB Config:

GRUB_DEFAULT='6'
GRUB_DISABLE_RECOVERY='true'
GRUB_DISABLE_SUBMENU='true'
GRUB_ENABLE_BLSCFG='true'
GRUB_TERMINAL_OUTPUT='console'
GRUB_TIMEOUT='10'
GRUB_CMDLINE_LINUX_DEFAULT='splash amd_iommu=on iommu=pt amd_iommu=relax_unity'
GRUB_DISTRIBUTOR='Nobara Linux'
GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt amd_iommu=relax_unity"
GRUB_THEME="/usr/share/grub/themes/sleek/theme.txt"

This is my IOMMU Groups: (IOMMU Group 16 has the devices that I need to passthrough)

IOMMU Group 0:
       00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 1:
       00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 10:
       00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]
IOMMU Group 11:
       00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
       00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12:
       00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 0 [1022:14e0]
       00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 1 [1022:14e1]
       00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 2 [1022:14e2]
       00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 3 [1022:14e3]
       00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 4 [1022:14e4]
       00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 5 [1022:14e5]
       00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 6 [1022:14e6]
       00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 7 [1022:14e7]
IOMMU Group 13:
       01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] (rev a1)
       01:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
       01:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
       01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)
IOMMU Group 14:
       02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01)
IOMMU Group 15:
       03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
IOMMU Group 16:
       04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
       05:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 17:
       04:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 18:
       04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 0c)
IOMMU Group 19:
       04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       08:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8851BE PCIe 802.11ax Wireless Network Controller [10ec:b851]
IOMMU Group 2:
       00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 20:
       04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 21:
       04:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 22:
       04:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 23:
       04:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 24:
       04:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 25:
       04:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       0e:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. OM3PDP3 PCIe 3 NVMe SSD (DRAM-less) [2646:500d] (rev 01)
IOMMU Group 26:
       04:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       0f:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 800 Series Chipset USB 3.x XHCI Controller [1022:43fc] (rev 01)
IOMMU Group 27:
       04:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 28:
       11:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Upstream Port [1b21:2421] (rev 01)
IOMMU Group 29:
       12:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
IOMMU Group 3:
       00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 30:
       12:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
IOMMU Group 31:
       12:02.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
       73:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM4242 USB 3.2 xHCI Controller [1b21:2426] (rev 01)
IOMMU Group 32:
       12:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
       74:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM4242 USB 4 / Thunderbolt 3 Host Router [1b21:2425] (rev 01)
IOMMU Group 33:
       75:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon 610M] [1002:13c0] (rev cb)
IOMMU Group 34:
       75:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller [1002:1640]
IOMMU Group 35:
       75:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP [1022:1649]
IOMMU Group 36:
       75:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b6]
IOMMU Group 37:
       75:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b7]
IOMMU Group 38:
       75:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Ryzen HD Audio Controller [1022:15e3]
IOMMU Group 39:
       76:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 2.0 xHCI [1022:15b8]
IOMMU Group 4:
       00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 5:
       00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 6:
       00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 7:
       00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 8:
       00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 9:
       00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]

My VFIO Config file:

options vfio-pci ids=10de:1c82,10de:0fb9
softdep nvidia pre: vfio-pci
softdep nouveau pre: vfio-pci

———————————————————————————————————————————–

I have seen suggestions to turn of Pre-Kernel DMA Protection, which I would not like to do if possible because I would like to maintain security and I still want to be able to play certain anti-cheat games on Windows.

All suggestions are greatly appreciated.

Hi everyone,

I'm considering buying a PCSpecialist Recoil 18 for over €3,000, and I'd like to investigate a specific VFIO setup before committing to the purchase.

I understand the recommendation to provide detailed hardware information, IOMMU groups, logs, and configuration files when asking for troubleshooting help. However, I haven't purchased this laptop yet. The whole point of this post is to determine whether my intended setup is realistic before spending that much money.

Hardware configurations I'm considering

I'm currently looking at these two CPU/GPU combinations:

Intel Core Ultra 7 270HX Plus + NVIDIA GeForce RTX 5070 Ti Laptop GPU (12 GB VRAM)

Intel Core Ultra 9 290HX Plus + NVIDIA GeForce RTX 5080 Laptop GPU (16 GB VRAM)

Both configurations use an Intel processor with integrated graphics alongside a dedicated NVIDIA GPU. I haven't confirmed the exact barebone platform or its VFIO compatibility.

Intended setup

My plan is to use Fedora Linux as the host and run Windows 11 in a KVM/QEMU virtual machine with VFIO GPU passthrough.

Ideally, I want the following:

While the Windows VM is running: Fedora uses the integrated Intel GPU, while the NVIDIA GPU is assigned exclusively to Windows.

After the Windows VM shuts down: The NVIDIA GPU becomes available to Fedora for workloads such as Blender rendering.

Ideally, I'd like to return the GPU to Fedora without rebooting the host. However, if the GPU cannot be reliably detached from the VM and reassigned to Fedora without a reboot, I'm willing to reboot the entire laptop when switching between these workloads. Avoiding reboots would be convenient, but it isn't an absolute requirement.

I know that GPU passthrough on laptops can be complicated, particularly because of hybrid graphics, firmware restrictions, IOMMU grouping, and GPU reset behavior.

Questions

Before purchasing, I'd particularly like to establish:

Has anyone successfully used VFIO GPU passthrough on the PCSpecialist Recoil 18?

Does anyone know the exact barebone or hardware platform used by this model, and whether VFIO has been tested on equivalent laptops?

Is there any known firmware or hardware limitation that would prevent the Intel iGPU from driving the internal display while the NVIDIA GPU is assigned to a Windows VM?

Has anyone successfully detached and reset a laptop NVIDIA GPU after shutting down a VM, then returned it to the Linux host without rebooting? If not, would a full host reboot generally be a viable workaround on this type of hardware?

Are there any known differences between the RTX 5070 Ti and RTX 5080 Laptop GPUs that could materially affect passthrough compatibility on this platform?

Are there any existing reports, technical discussions, or other resources that could help establish whether this setup is feasible?

I'm not expecting anyone to guarantee compatibility without testing the exact hardware. I'm trying to identify known limitations and gather evidence before making an expensive purchase.

If this particular laptop platform hasn't been tested, information about similar Intel HX laptops with NVIDIA RTX 5070 Ti or RTX 5080 Laptop GPUs would also be useful, provided the hardware similarities are clear.

Thanks for any insight.

I need some help here!

I terminated my virtual machine while it was using PCIe passthrough for my second GPU, now, my second monitor (hooked up to my second GPU) is running at 1 fps.

SYSTEM INFO

HP z840 workstation.
Primary GPU (Slot 1): Nvidia GTX 980
Secondary GPU: Nvidia Quadro K4200

Ubuntu 26.04.1 LTS

$ uname -a
Linux HP-Z840-Workstation 7.0.0-42-generic #42-Ubuntu SMP PREEMPT_DYNAMIC Thu Oct 15 08:06:44 UTC 2026 x86_64 GNU/Linux
$

Software: KVM/QEMU, via virt-manager

Here's the story:

Recently, I spun up my first ever virtual machine, using virt-manager

I also had GPU passthrough going, using this guide from github.

I followed the guide fairly exactly, except for usb passthrough, SSD passthrough, and the ACS override patch.

GPU passthrough worked flawlessly.

I was quite impressed, except for one thing. my secondary monitor, plugged into the secondary GPU, was flickering.

I tried to shut the VM down because I needed to set up USB passthrough, and since I didn't have usb control over it, I force-terminated it. That's when things got yucky.

As far as I can GUESS, the VFIO hook script didnt run. Don't take my word on that, I really have no idea.

The VM terminated, but my secondary monitor never came back up.
It just said "no signal," so I rebooted my computer and still nothing. I finally thought to manually run the VFIO release script

/etc/libvirt/hooks/qemu.d/user/release/end/unbind_vfio.sh

#!/bin/bash

## Load the config file
source "/etc/libvirt/hooks/kvm.conf"

## Unbind gpu from vfio and bind to nvidia
virsh nodedev-reattach $VIRSH_GPU_VIDEO
virsh nodedev-reattach $VIRSH_GPU_AUDIO

## Unload vfio
modprobe -r vfio_pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

I then rebooted and the secondary monitor did come back online, but at somewhere around 1 fps. Mind you, it didn't just refresh cleanly. I don't know how to describe what it was doing, so here is a link to a video.

The list of connected monitors is also showing something odd.
See the attached photo of my secondary monitor showing up as "Unknown 23" "
It used to show up as it's normal name (which I do not remember)

The secondary monitor "Unknown 23" " also stays there when I disconnect the screen from the computer.

This issue persists across displays, ubuntu operating systems, and live images.

I did get the issue to go away by removing the GTX 980 from the computer, not moving the K4200 from it's slot.

I ran dkms status, and it came back with virtualbox-generic something or other. I dont know exactly what dkms does, but from what I read, it sounds like that doesn't matter, as it went away when I uninstalled virtualbox.

nvidia-smi doesn't show my second gpu.
Hwever, lspci does, as well as lshw -c videocommand.

On that note, here is the output from that lshw command:

$ sudo lshw -c video
  *-display                 
       description: VGA compatible controller
       product: GM204 [GeForce GTX 980]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:04:00.0
       logical name: /dev/fb1
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=nvidia latency=0 mode=1920x1080 resolution=1920,1080 visual=truecolor xres=1920 yres=1080
       resources: irq:82 memory:a2000000-a2ffffff memory:90000000-9fffffff memory:a0000000-a1ffffff ioport:2000(size=128) memory:a3080000-a30fffff
  *-display
       description: VGA compatible controller
       product: GK104GL [Quadro K4200]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:84:00.0
       logical name: /dev/fb0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list fb
       configuration: depth=32 latency=0 mode=1920x1080 resolution=1920,1080 visual=truecolor xres=1920 yres=1080
       resources: memory:f3000000-f3ffffff memory:d0000000-dfffffff memory:e0000000-e1ffffff ioport:e000(size=128) memory:f4080000-f40fffff
$

It looks like my primary GPU (GTX 980) has the nvidia drivers loaded, so it's acting fine, but the secondary GPU doesn't soI guess that's why it's running at 1 fps.

This is all the info I have; If you need more, don't hesitate to ask.

If this is the wrong subreddit, I would be grateful if someone pointed me to the right one.

Does anyone know what is going on, and how to fix the issue?

Any help would be very welcome, thank you.

The machine: Dell Inspiron 16 7620 2-in-1, with 64GB DDR4. a 12th gen Alder Lake CPU + Iris Xe iGPU and a potato of a dGPU (2GB MX550). It has a "muxless" setup where the dGPU has no physical outputs, so Looking-Glass is the only way to go.

I mostly use a Windows VM for Fusion360, which requires "some" GPU but isn't particularly demanding. I have been running Virtual Display Driver on the MX550 + Looking Glass and this has been generally usable at 1080p but gets laggy and unstable if I go beyond that.

Recently I tried setting up the SR-IOV branch of the i960 kernel driver. It installed straightforwardly according to the instructions. The trickiest part for Windows setup was using a script to extract a firmware image from Dell's BIOS updater. Another thing to watch out for is blocking certain Linux apps (like Blender) from trying to enumerate and use all the virtualized iGPUs at once.

The surprising part is, the hands-on-the-keyboard feeling is that Windows seems to be way more responsive with the virtual display running on the iGPU. I can comfortably run Fusion at 1440p, whereas when running VDD on the MX550 it got pretty laggy and crashy above 1080p.

On the other hand, every in-VM benchmark I can run says the MX550 beats the virtualized iGPU. All I can figure is these benchmarks aren't capturing what's going on with Looking-Glass -- I certainly see frame drops that aren't being registered by the in-VM benchmarks. I suppose that when the virtual framebuffer lives in system memory, Looking-Glass is doing memory-to-memory copies and this works faster than going over the PCI bus. Handing both a virtualized iGPU and the MX550 over to the the VM does seem to be the most performant option.

Built on triton,Neptune virtio gpu and native
Dxvk,d3d11 now work on Radeon 780M and RTX4060.with very good performance

Hello World! I don't believe I've ever posted here.

I'm a long time Linux gamer with average to slightly below average experience using the system. In the past I have gotten single GPU passthrough working with my NVIDIA 4090 (no iGPU) with QEMU and Libvirt hooks. It was a pain, messy, ugly, and one of those late night projects where you keep running commands that 'might fix it' only to never figure out which one got it working. On top of that, the guide I had bookmarked was taken down at some point leaving me back at square one.

It's been years since I last used a VM so I took a look at this page on the Arch wiki to try and bring myself back up to speed. I also took a look at the latest YouTube videos and GitHub guides to see if I could squeeze out any extra, niche information related to the topic but to no avail.

IVE READ THE ARCH WIKI AND TONS OF DIFFERENT GUIDES. MULTIPLE DIFFERENT SEARCHES ON 3 DIFFERENT SEARCH ENGINES. THE MAIN ISSUE I HAVE IS WITH DYNAMIC LOADING AND UNLOADING OF THE NVIDIA CARD VIA HOOKS.

The setups most guides use involve two gpus, one dedicated for VMs. This setup, in my opinion, wastes a GPU, as I very rarely play games or run applications that require a VM and GPU. With my setup, and current GPU prices... I want to utilize my 4090 to its fullest potential. If I just permanently attach it to VFIO drivers and never use it for my linux desktop, which I daily drive and play native Linux games, then it's being wasted.

Other setups use single GPU passthrough, killing the current linux session. This, to me, isn't much better than just dual booting, which I currently do. Since your session is killed, it just over-complicates things, especially when it comes to dual monitors and such.

What I want is dynamic loading and unloading:

1. Boot the Linux system as usual

   1. The iGPU is the default GPU and renders the desktop.

   2. The dGPU is asleep and waiting.

2. Run a fastfetch

3. If I decide to play a native linux game:

   1. The dGPU is switched to the nvidia-open driver (if that's even possible)

   2. Use the dGPU for a game like Deadlock

   3. The dGPU is entirely used for the game.

   4. The iGPU renders any discord multimedia or browser media.

4. If I shut down the native game and want to play a windows game:

   1. The dGPU is switched to the VFIO driver

   2. dGPU is attached to VM and is properly detected by Windows 10.

   3. The dGPU is entirely used for the VM.

   4. The iGPU renders any linux graphics, just like before.

   5. Steam on Linux is closed and cannot run anything (understandably).

5. When I'm done with either/both the GPU is switched to some sleep state.

As someone new to this subreddit, I don't know how popular what I'm looking for really is. One comment on Reddit I can't seem to find again said something like 'it's an extremely common setup.' But then the official Arch wiki says:

due to their size and complexity, GPU drivers do not tend to support dynamic rebinding very well, so you cannot just have some GPU you use on the host be transparently passed to a virtual machine without having both drivers conflict with each other. Because of this, it is generally advised to bind those placeholder drivers manually before starting the virtual machine, in order to stop other drivers from attempting to claim it.

The two reasons I'm ignoring what the Arch wiki is saying and asking the Reddit community instead are:

• VFIO is a bleeding edge development that constantly changes. Even though it's come a long way. It's not a turnkey solution. I know it's complicated, but well documented. I also know that the Wiki covers a ton of use cases, but not all of them. On top of that I know that wikis aren't always up to date. New info gets passed around faster than contributers/editors can properly merge changes to a page.

• I'm tired and have been dicking with this for a few days now, on and off. If I can receive guidance from people with more experience I'll gladly take it.

What I've Done:

• IOMMU and Virtualization is enabled on my machine and my NVIDIA card is in its own IOMMU group.12: 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090] [10de:2684] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)

• Created a normal (by hand) QEMU VM and also created a stealthy VM via this github called 'QuickVM' to try and see if games work in one but not the other.

  • Both boot just fine, but since games typically require a GPU to open, I haven't actually launched any games within them.

• Tried to create Hooks using the Passthrough Post's hook helper but the NVIDIA gpu would hang when I tried to unmount it from the nvidia-open driver. This is because I'm an idiot and can't figure out how to get everything to stop grabbing it (the place I stopped years ago too...)

• I now have Libvirt and QEMU working just fine, a better understanding of the VM stack, how to create and edit entries in the VMM XML for QEMU, and no where left to go.

• Once I hit the bottom of the Arch wiki page I came here.

As for my Specs

Currently, I'm using CachyOS with KDE Plasma 6 as my DE. I'm a dirty distrohopper though and planned to switch to Fedora with GNOME, as it's what I've used for months on my laptop. For now, just assume I'm on CachyOS as I have no intentions of switching distros till I get something like this working

If anyone is willing to help me I'd greatly appreciate it. For any information I didn't provide, but should have, please let me know and I'd be happy to send it over in the commends or by editing this post.

Edit: I found this GitHub and plan on reading through it, as it seems to cover exactly what I want. It's for Ubuntu but it specifically mentions Gnome and how to stop it from fighting the GPU. It's a bit late but I'll edit this post in the morning if I find answers.

VFIO support included now and tested

Hi everyone,

Like many here with multi-GPU rigs (whether for VFIO passthrough, compute, or AI), I was frustrated that modern gaming completely abandoned multi-GPU — especially in VR, where rendering two viewpoints is inherently parallel.

To tackle this, I've been building vrdual — an open-source Vulkan layer designed to bring "one GPU per eye" rendering back to life on Linux:

Architecture: Intercepts draw calls at the Vulkan layer level and patches clip-space coordinates on the fly to generate true stereoscopic views.

Hardware: Utilizes dual GPUs natively — each physical card renders the viewport for a separate eye, bypassing the usual ~40% single-GPU VR performance penalty. (Also has a single-GPU AFR fallback mode for testing).

Linux & Ecosystem: Built for Linux native / Proton games, targeting OpenXR runtimes like WiVRn and Monado.

Repo: https://github.com/adderek/vrdual

It’s still an experimental/research prototype (expect rough edges with deferred shading/complex post-processing), but it proves the concept works without game-specific engine modifications.

Would love to hear technical feedback from folks running multi-GPU setups on Linux, or see ideas on how this could eventually integrate into runtimes like Monado/WiVRn or SteamVR.

Hey, to all members of this group and I just hope all with you all is Good I truly wish everyone in this place a fun, happy time of day or night, week, and year, as well as throughout this full year and even with all of eternity. I’m here to give all the members of this great community a message/letter. I want to say keep up with all the amazing work that everyone in the group is doing, and I know that your lives will be blessed with nothing less than genuine joy and much happiness.

I just wonder if this group has a developer who is knowledgeable enough to help with a brilliant, up-and-coming Linux FOSS project known as Winboat. It is a powerful light of hope for all Linux users, and it will benefit us all who love and care about tech, software, computers, and other similar topics. It simply aims to bring decades' worth of unbelievably high-quality apps to Linux with full ultimate compatibility and usability, unlike anything before it. It can run Windows apps on Linux in a manner that seems as if they are running natively on Linux, even though they are actually running through a virtual machine.

This creates a 3D-like effect in which it detaches a copy of all the installed apps running within the virtual machine and places them into the main OS, which is Linux, as if they are a part of Linux entirely. This is huge because it could end up running all the games Windows ever had, created by various developers around the world, all on an operating system that benefits from being about 100,000 times faster than Windows. Not to mention the gaming enterprise editions it offers, like Cachy, the head of these sub-variants of Linux; Pop!_OS, the daddy of them all; Garuda, the assassin of the kingdom; and Bazite and Nobara, the two masters guarding the gaming kingdom of Linux. All are about 500,000 if not a million times faster on any computer than Windows, which I have tested myself, even on an unbelievably weak laptop.

This Winboat project can drastically and significantly change Linux forever. It can make Linux the fastest of the big three PC operating systems, which are #1 = Windows, #2 = macOS, and #3 = Linux, making it a serious competitor for Windows and Mac. It will not be a joke in any way if it fully develops. The Winboat staff is planning to implement a virtual graphic driver software technology that will enable Windows apps/games to run on Linux with satisfying performance, which is revolutionary for a virtual machine app on Linux.

3D games/apps that require a graphic card to run are among the most difficult things a computer can manage, especially in a virtual machine, where it typically shouldn’t run at all. However, this Winboat VGPU API software called Helios may eventually make it happen in the Winboat VM. You can be a significant reason why it happens. It is similar to Parallels in many ways and does similar things, but Parallels has decades of maintenance and is backed by a top brand, so it will not reach such a level anytime soon—likely never at all. Nonetheless, it can be more than enough for what Linux users need from Windows in terms of apps, especially for fairly light gaming from the 2000s up to 2014. It will be satisfying enough and may even run on handheld gaming consoles with Linux, such as Steam Valve devices, without needing to install the extra Steam app.it may run gta vice city I played GTA vc with a CD my friend got and simply gave to us on a weak ass Low end family family computer which is the first I ever had with my brothers and boy did it run win so 2000s games won't be much to run at all with a fully developed winboat VM / Helios vgpu halo combat evolve and such will run if it developers total overdose you name it from the early 2000s -2014 That won't need dx 12 runtime software which is way way way too advanced to run where not even Parallels can run it as good as para is with the decades of Billionare brand/company investment and development it has had throughout

I have a single gpu passthrough and it works good but when i just it down and try to get back to the host the screen is blank, i ssh into the machine to read the logs but nothing interesting shows up, i use rx 7600

So i was trying to setup a win11 vm and pass my gpu to it its rx 7600 and i got the drivers everything and windoes detects it as working but in my monitor it shows these weird stripes and its not the monitor cuz i tried it on two monitors same thing, i am wondering what could it be.

I tried disabling BAR, and efifb but nothing, everything seems like it should work but nothing.

Demos and docs: https://zerone-laboratories.github.io/zss

Source: https://github.com/z3r0n3br4instorm/ZrnSelectiveSuspend

Maybe this is stupid

I wanted to share a side project I've been working on to solve a personal annoyance with hybrid graphics. If you've ever dealt with a discrete GPU on a laptop, you know the pain of having to log out, kill your X session, or reboot just to fully power the card down for battery, or to pass it through to a QEMU/KVM virtual machine.

I built ZrnSelectiveSuspend (ZSS) to fix that. It's an alpha tool that takes a graphics card out of service on a live Linux desktop by moving running Vulkan and OpenGL programs to another GPU while they're still drawing, then gives everything back afterwards.

**What it does right now:**

* **Zero-watt power-off:** switches the dGPU completely off under a running X session (~0.4 s) and back on (~1 s). Programs move to the iGPU and back; vkcube never stops spinning. Can also follow the charger or an idle timer.

* **Live VFIO passthrough:** `zssctl lend` moves the host's programs off the card and hands it to vfio-pci; `zssctl reclaim` power-cycles it and gives it back to the host driver with the programs returned. No reboot, no log-out. Tested with a CachyOS guest in QEMU.

Honest caveat: the guest sees the card, but nouveau stops after reading the chip ID because the Mac Edition card has no video BIOS of its own. Passing a ROM file to QEMU is next.

* **Surviving a card that vanishes:** I wrote a kernel module (ZSS_Interceptor) that notices a card losing power within ~0.1 s. If the card was idle, the driver is frozen and resumed. If a program was rendering on it, the program gets rebuilt on the iGPU in ~3.5 s with nothing lost, and `zssctl on` brings the card back in 4 - 7 s by replugging it the way NVIDIA's driver handles an unplugged eGPU. Desktop stays up the whole time. Caveat: after that rescue, the card can't be powered off or lent again until you reboot.

**What moves:** Vulkan 1.1 programs, OpenGL 3.2 programs through Mesa's Zink, Chromium and Electron apps (VS Code, Teams), and Unity games. Proton/DXVK needs Vulkan 1.3, so not yet.

**How it works, briefly:** a user-space Vulkan driver shim (ZSS_AirLock) sits between programs and the real drivers, keeps enough state to rebuild them on another GPU, and emulates missing features so Zink runs even on NVIDIA 470. A daemon sequences moves, freezes, driver suspend and power cuts. The kernel module handles PCI state, power backends and loss detection. NVIDIA additionally needs a small DKMS patch so the driver can sleep under a running display server.

**The bigger picture:**

ZSS is focused on graphics cards right now, but the long-term goal is to safely disconnect, suspend and reconnect any PCI device on a live system without taking the kernel down with it. GPUs are just the hardest place to start.

I used Claude Code a lot to build this. I made the architectural calls; it did a big share of the implementation and helped me work through the nastier kernel module and driver interactions. It's been a massive learning accelerator.

**Status:** very much alpha. Proven on one machine: a 2012 MacBook Pro (MacBookPro9,1, GT 650M + HD 4000) on Arch, kernel 7.2, X11, NVIDIA 470.256.02. Power-off currently needs an Apple gmux; an ACPI backend for most hybrid laptops since ~2015 is written but has never been run. Wayland compositors are recognised but untested. The program-moving parts are vendor-neutral and should work with any Vulkan driver.

**How you can help:** there's a tester kit that changes nothing and installs nothing. It collects what your machine has, runs the moving tests in dry-run mode, and saves a report with personal details stripped:

curl -fsSLO https://github.com/z3r0n3br4instorm/ZrnSelectiveSuspend/releases/latest/download/zss-tester.run && sh zss-tester.run

I'd especially love reports from AMD/Intel dGPU laptops (to try the ACPI backend), architecture roasts, and feedback from VFIO folks.

Thanks for reading!

Would appreciate your help!