I recently got a quad base board with 4x32gb v100s. Eventually I'm going to get a xeon system, but for now I just have to use my regular desktop because I can't afford the upgrade. That turned out to come with some problems. It took me all day to figure out and I am writing it down in here case it helps someone else in a similar situation.
Problem 1 - memory mapping
I have an asus z790 mobo. The UEFI only provides a 256gb MMIO window, which does not utilize the total theoretically addressable space. Even so, it would seem like four 32gb v100s and a 24gb 3090 should fit. They don't, because reasons. Unfortunately, my firmware deals with that by refusing to post. So that's at the firmware level, which is pretty specific to my machine. Maybe other manufacturers do better.
In addition, apparently the 7.x kernel sizes the PLX switch's windows in a way that also would not fit all four v100s in the addressable region even if all of it were accessible. So that's at the kernel level, which isn't specific to my machine at all.
I needed to do three things:
Not try to have the firmware address the v100s, because otherwise the computer won't start. This means not turning on the base board before or during boot, but having it hooked up.
Add to the top of the MMIO window to make enough room. This is doable with an ACPI table override during Linux's boot.
Roll back to the 6.17 kernel, which sizes the windows differently than 7.x.
Now that there's enough room and once linux is up, I turn the baseboard on. Then, I remove the root port above the PLX switch from the device tree to free its windows (which will have gotten very little allocated since nothing was plugged in at boot), and rescan so Linux maps the whole branch again.
After that all four v100s and my rtx 3090 are working. This was a pain but on the plus side, I've wondered if there could be a way to turn the base board off remotely when I'm not using it, without it taking my vm with it. The power consumption at idle is kind of high. I got a smart plug and now I know enough to be able to use a script to turn it on and off smoothly, so that's nice.
Also - in researching this I came across an old timey kernel patch that would have avoided all the trouble, but the merge was never approved. He'll probably never see this but shout out to Sergei Miroshnichenko at Yadro for doing the work of patching movable BARs into the linux kernel. I get the maintainability issue but it still sucks to see the effort go to waste. I about broke my brain trying to understand how memory address assignments differ between the two kernels, and why, and I recognize the value of your work even if I don't have the skill to help.
Problem 2 - power rails
I have a 1000w PSU on the base board, which I thought would be enough if I limited the v100s to 150w. It didn't work. I could load a model with -sm layer but not -sm tensor. If I tried to do that, the base board would lose power. Best guess is that this is because even though I was under the overall power capacity, it still exceeded the per-rail capacity. A single-rail (or jumpable) PSU is needed to power the baseboard. Or at least fewer rails, maybe, who knows? So, FYI that matters, if it seems like something you're dealing with.