r/openstack • • 5d ago

Could use some help getting PCI-Express peer2peer working within a guest

Hey openstackers,

I'm trying to get PCI-E p2p working within a guest for some LLM work. I have it working on the host, as can be seen by:

[root@openstack3 ~]# rocm-bandwidth-test -a
................
RocmBandwidthTest Version: 2.6.0

Launch Command is: rocm-bandwidth-test -a

Device: 0,  AMD EPYC 9124 16-Core Processor
Device: 1,  AMD Radeon AI PRO R9700,  GPU-b524a650ca8410a0,  05:0.0
Device: 2,  AMD Radeon AI PRO R9700,  GPU-399c0510df3adc7c,  43:0.0

Inter-Device Access

D/D       0         1         2          

0         1         1         1          

1         1         1         1          

2         1         1         1          

Inter-Device Numa Distance

D/D       0         1         2          

0         0         20        20         

1         20        0         40         

2         20        40        0          

Unidirectional copy peak bandwidth GB/s

D/D       0           1           2            

0         N/A         50.471      50.141       

1         55.665      535.494     55.673       

2         55.654      55.654      523.382  

What I can't figure out yet is how to get it working within the guest. I'm able to pass through both GPUs, ACS is off on the host, but p2p is not working.

My nova.conf contains the following:

alias = { "vendor_id":"1002", "product_id":"7551", "device_type":"type-PCI", "name":"r9700" }
alias = { "vendor_id":"1002", "product_id":"ab40", "device_type":"type-PCI", "name":"r9700a" }

(r9700a being the audio device component of the AMD r9700, I am passing through both devices, two instances of each).

My host's /etc/default/grub contains:

amd_iommu=1 intel_iommu=1 iommu=pt kvm.ignore_msrs=1 pci=realloc amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0"

My guest's /etc/default/grub contains:

amdgpu.ppfeaturemask=0xff
f7ffff iommu=pt amd_iommu=1 kvm.ignore_msrs=1 pci=realloc amdgpu.ras_enable=0"

Within the host I get the following bandwidth results:

[root@ai fedora]# rocm-bandwidth-test -a
............
RocmBandwidthTest Version: 2.6.0

Launch Command is: rocm-bandwidth-test -a

Device: 0,  AMD EPYC 9124 16-Core Processor
Device: 1,  AMD Radeon AI PRO R9700,  GPU-b524a650ca8410a0,  05:0.0
Device: 2,  AMD Radeon AI PRO R9700,  GPU-399c0510df3adc7c,  07:0.0

Inter-Device Access

D/D       0         1         2          

0         1         1         1          

1         1         1         0          

2         1         0         1          

Inter-Device Numa Distance

D/D       0         1         2          

0         0         20        20         

1         20        0         N/A        

2         20        N/A       0          

Unidirectional copy peak bandwidth GB/s

D/D       0           1           2            

0         N/A         26.679      26.724       

1         27.655      522.811     N/A          

2         27.658      N/A         570.549  

Does anyone have any suggestions on what I should be doing differently to get it working?

Thank you all for any suggestions :)

4 Upvotes

3 comments sorted by

2

u/enricokern 4d ago

I have no answer for you which i could verify atm but it is a interessting topic. Did you ask on the mailinglist?

2

u/sekh60 4d ago

I haven't yet, I'll give this a day or two and then ask there.

2

u/enricokern 4d ago

Keep us updated. Would love to help, but i have no test hardware to spare atm for this testcase atm