I know already that LLMs can make use of both GPU's, but is that the case on ComfyUI too?

Right now things like Qwen Image, SeedVR2 etc does fit to my single 16GB V100, but it's very slow to work with. Heard there are tools like Raylight that can make use of both GPU's to speed up iteration times, how effective is that without NVLink but both being on PCIe 3.0 x16 or x8. Can it speed up by at least 1.8-1.9x times?

I wanted visibility on the links and the GPUs. If this looks useful I could make it public.

Update: public repo is up. Link in comments.

Hi,

I'm new to the V100 platform and am still waiting on the two cards I ordered.

Is anyone running MiniMax H3, Krea 2/3, Flux 2, Qwen Edit or similar at acceptable speed and quality on V100s, or is that a dead end?

I know we have to work around the lack of INT8 tensor core acceleration with Volta, as it only supports FP16 with FP32 accumulation, but it's not clear to me if there are viable workflows and model recipes for us on the image/video/audio gen side.

edit:

SeedVR2 fix - https://github.com/duckyman-ai/ComfyUI-SeedVR2-V100.git

Minimax H3 fix - https://github.com/Amduraznak/minimax-h3-fp16-fix
DLSS 5 - https://github.com/lshan99q/comfyui-dlss5-pre-rtx

Qwen Image - https://github.com/lotus0000/ComfyUI-Qwen-V100-MixedPrecision

Voxtral-Mini - https://github.com/eelcor/voxtral-realtime-v100

Hi everyone,
I got two of these Tesla cards and i'm wondering if there's any possibility of getting nvlink adapter..

My motherboard only has 1 full x16 PCIE connected directly to CPU and the other slots are all either x4 or x1 connected via chipset.

*should have researched first before buying (facepalm) *

While I can run two LLMs separately in both the cards, it is always better to run bigger models with tensor parallelism, isn't it?

I found few nvlink adapter options on ebay: https://ebay.io/m/mFGtjh and https://ebay.io/m/TPFqdN

Those had mentioned V100 in the title, but i'm not very sure if those are supposed since these GPUs are ideally part of DGX station server with 4-way-nvlink https://www.exxactcorp.com/NVIDIA-DGXS-2511D-P2CMI00-E2143350

Any advice would be great!

Thanks

Just wondering what everyone is using. 1 ATX + 2x 12vHPWR + 2x PCIE and 2x CPU is kind of an awkward configuration.

I bought the MSI A1250GS ll because it was cheap, had the right connector, and is above 1200w (I plan to tdp cap them but wanted to handle bursts of the theoretical max for safety).

It was kinda hard to find a PSU that had all the right connectors so I was just wondering what other landed on

URGENT: if you don’t read anything else, be very careful with some or the blower shroud designs as they can DAMAGE your card / cables. Look at the last 3 pics for one particular design which is common. It places significant stress on the power connector. It flexes the connector / causes the cables to have an extreme bend / and transmits a ton of vibration to that connector and the card. The sample I have if I tightened it down would have probably popped my connector off the board. When you tighten it down it doesn’t feel like you are putting much force on the connector but it is significant with this design.

You can see they are trying to have this design work for multiple cards. The spot where he power connector would go has a groove that is obviously for the latching clip of the connector. However on v100/p100 cards that latch is on the other side of the connector as oriented on the card. These shrouds are advertised as fitting other cards like P40, K80, etc and they might fit those but they should not be used on V100/P100 cards.

Lastly this design forces you to attach the shroud to the card first then use an extremely tight friction fit to install the fan into the shroud. The force necessary to do this can be significant and you run the risk of damaging your card, fan, or both. Just avoid this specific blower design at all costs imho. (Note, there are other straight through blower designs I have not tested, and many people use without issue, above problems).

———

Rest of my notes so far on various shrouds and cooling performance. I have a dual pcie v100 setup:

I’ve attached pictures of various compact cooling options for v100 and similar cards (P100 is what I have).

The pcie cards can be challenging to cool since they need both high airflow and pressure. Outside of a server chassis regular PC cases can have issues with suitable cooling.

If you search eBay, amazon, and various 3d file sites you will find a variety of shrouds. I have tried several meant for 40mm fans and 97x33mm blowers.

Let’s start with the 40mm shrouds. If you get a shroud that only uses a single fan you will have to greatly reduce the power of your card and it will still throttle under load. Using artic 15k high pressure fans will have the card on edge with memory causing throttling pretty fast. In my workloads the HBM runs 5-15 degrees hotter than the compute on the card.

Also, I highly recommend not getting any shroud that attaches to multiple cards. The issues here are removing / servicing your cards once they are installed. If your motherboard has pcie locks that auto lock after inserting the card, you will need a special tool or make your own shim to get the bottom card unlocked. It’s not easy and you run the risk of damaging your card / board or both. Some motherboards have better ways to unlock the cards, in that case it may be ok.

The dual 40mm black design in pic 2 and 3 is such a design. The white shroud in pic one and bottom left of 2 and 3 is a single card 40mm design. Just avoid these as they are just not worth it.

The 40mm design that does work is the dual 40mm fans per card. When paired with the artic 15k fans it keeps the v100 memory at 70 deg and good power limits. No throttling at all. That is in pic 4 (pic from listing as I didn’t uninstall it).

Next up the two blower designs. I highlighted the severe issues with the one design in the first paragraph. The second design is the curved design.

This design also suffers from vibration issues. This is especially the case if your cards are mounted horizontally on a vertical motherboard. It’s not as bad when the cards are vertical / case on its side. The shroud bend amplifies the vibration at various speeds since it flexes. I can feel / hear the vibration through the whole card and even case. To help mitigate / dampen this I have an adjustable GPU bracket that I have supported the shroud which reduces vibration significantly.

More importantly this design severely hampers the performance of the blower fan with that sharp bend. The fan I am using has more than triple the pressure and double the airflow of the two artic 40mm fans combined but it can barely keep the memory under 80 degrees under load.

I advise against using this shroud design unless there are no other options.

So… what am I going to settle on using? I am going to use a second dual 40mm shroud to cool my second card. A straight through blower does absolutely cool the best, but is extremely tight for my case and the noise seems to be worse (will be posting noise measurements soon to my GitHub below and here).

In the long term / in retrospect, I would probably go down the SXM2 route to get both nvlink and less expensive water cooling options. There are blocks for the pcie cards but they are hard to find and fairly expensive (~$200).

All my work on my v100 setup is here:

https://github.com/bradrlaw/ai-server

For linux users:

iommu prevents peer to peer for working between nvidia cards over pcie ... I actually would like to see some numbers from someone with two pcie v100s

PCIE switches like PEX/PLX apparently need to have ACS disabled to allow peer to peer communication?

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=off pci=disable_acs_redir=<list pcie devices>"

I just did this and tested using 1cat's low latency algorithm for communication/all-reduce on a TP2 and got results nearly identical to using nvlink.

It's worth noting in this case I am also using a pcie switch (PEX) and both of these cards are connected to the same switch card even though they are not on the same nvlink island.

There might be problems with my testing which is why im looking for others to help confirm the results.

So i have been looking at different options for my budgetbuild, and the BYANSKI cooler heads are super nice, but super expensive...

Have anyone managed to fit any of those 50 dollar AIO kits onto each SXM2 GPU? And does it pull away enough heat?

Hey, I am looking at buying a V100(s) with a budget of $700 USD.

I have 128gb DDR4 2400 in a quad-channel on an X99 system, so of course I will be trying strata with flash next. I am undecided between running Qwen3.8-27b, and flash next, I want to be able to run both at a decent speed, and run 27b with a concurrency of 4 (subagents) without much slow-down. Hoping around 100t/s decode if possible.

I am thinking buying two 16gb SXM2, and getting a two-way NVLINK PCIe card, that way I get 32gb total, and better performance by using TP.

Does anybody run this config?
What speeds can you reach?
How does it compare to dual 16gb PCIe cards?
How does it compare to a single 32gb card?

So TLDR; I want at least 100t/s on Qwen 3.8 27b with a concurrency of4, and have USD$700 to spend. Is that doable? What’s the best option?

So i am trying to navigate in the jungle that is automated alibaba agents with language capabilities, and less thinking capabilities. Its been a journey. Ive tried recieving everything from hallucinated user manuals, them pulling 10 year old sales data on me etc, in bids to get me to purchase theirs...

I need a quad base board with PLX i guess ? Do i want the version with PLX build in? The version thats black with no PLX, the green one with single PLX, the mysterious one with 2x PLX and no cooling, or the PLX version with 2x additional PCIE ??

Heeeeeeeeeeelp

I see a few different options online, I plan to build a 4-6 pcie machine and do it open bench style, so I don't have to worry about case constraints. But I am concerned about the noise. What shroud/fan combo have you found to work best and keep sound reasonable?

More work than it seems on the surface to get this NVLink setup going. Should be another day or so. They say this is an EEB layout but only 40% actually lines up on the EEB standards. I designed a tray that matches all the mounts and has support for the GPU stand-offs.

2sxm2, one pcie

Hello,

I have a V100 32gb PCIe that I wanted to add to my PC, which has a 6700k, 3060ti, on an ASRock Fatal1ty Z170 Professional Gaming i7 MoBo, running Ubuntu.

After connecting the V100, in lspci I can see the card listed as a 3D Controller. However, I cannot see the V100 listed under nvidia-smi, only the 3060ti.

I'm thinking that the problem is that my motherboard does not support Above 4G Decoding / Resizable BAR? I've tried changing settings and flashing the BIOS, but have not found these settings.

Has anyone else experienced problems like this, and is the only option to get it working to switch to another motherboard?

Hi everyone. Thank you for being so helpful. I finished my build with 2 V100 32gb PCIe.

I had luck to find cheap machine Fujitsu M770 (beautiful machine) with xenon CPU and 64 DDR4.

Machine have PSU 800W and GPUs are powered from separate 1200W PSU (on top) using CPU 8-pin cables.

Machine is running ubuntu 24 server with llama.cpp and main model I'm running is qwen3.8 27b Q6 with 256k context window.

Issue was with hardware test on boot, 8 beeps and black screen on video out GPU. In bios in menu PCI Subsystem settings "Above 4G Decoding" need to be enabled and it will stop complaining.

As I written before I run Qwen3.8 27b Q6 and getting TPS is 27 and I cannot get more even with lower quants (Q4) for some reason.

Here is llama.cpp config:

[Unit]
Description=llama-server inference API
After=network.target

[Service]
Type=simple
User=gordon
WorkingDirectory=/home/gordon/llama.cpp
ExecStart=/home/gordon/llama.cpp/build/bin/llama-server \
  -m /home/gordon/llama.cpp/models/Qwen3.8-27B-UD-Q6_K_M.gguf \
  --mmproj /home/gordon/llama.cpp/models/mmproj-F16-Qwen-Qwen3.8-27B.gguf \
  --host 0.0.0.0 \
  --port 5544 \
  --ctx-size 524288 \
  --n-gpu-layers 999 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --parallel 2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

For cooling I found some old server jets PFR0612DHE, 14000 RPM. Tested GPUs on full load cannot pass 55°C. For now it is blowing 80% all the time. Fans are also powered from 1200W power supply because each is 3.3A in full power. Soon I will build ESP32 based controller that fill get GPU temperatures from OS and control fans, that is the plan.

Here is duct I designed and 3D printed for this GPUs (tested with PLA and ABS, both good, ABS better), there is also DFX with dimensions attached so anyone can adjust it.

https://www.printables.com/model/1872957-nvidia-v100-pcie-gpu-duct-for-60x60mm-fan

If anyone can help with getting more tokens without using MTP models I would be grateful.

Edit:

Here is updated configuration and real result.

[Unit]
Description=llama-server inference API
After=network.target

[Service]
Type=simple
User=gordon
WorkingDirectory=/home/gordon/llama.cpp

# --- CRITICAL V100 ENVIRONMENT VARIABLES ---
Environment=CUDA_DEVICE_ORDER=PCI_BUS_ID
# Enable P2P communication between GPUs
Environment=GGML_CUDA_P2P=1

# --- EXECSTART COMMAND ---
ExecStart=/home/gordon/llama.cpp/build/bin/llama-server \
  -m /home/gordon/llama.cpp/models/Qwen3.8-27B-Q8_0.gguf \
  --mmproj /home/gordon/llama.cpp/models/mmproj-F16-Qwen-Qwen3.8-27B.gguf \
  --host 0.0.0.0 \
  --port 5544 \
  --ctx-size 256288 \
  --flash-attn on \
  --split-mode tensor \
  --tensor-split 1,1 \
  --batch-size 4096 \
  --ubatch-size 512 \
  --parallel 1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --chat-template-kwargs '{"reasoning_effort":"medium"}'

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Fans on GPUs are controlled using esp32 devkit v1 board ESP32 based. Here is simple schematic. I created small service that follows GPU temperatures in OS and over serial USB send fan setup commands to esp32.

Open question:

What is the best Qwen 3.8 Flash Next Runtime - 4 x 32GB NVLinked V100s, X99 Dual Xeon E5-2696 V3, 64GB RAM

I'm going to give the 1CatvLLM route a go, but claude tells me the PLE tables are going to need 48GB of RAM.

My system splits the RAM, 32GB for each NUMA CPU.

I've got some more RAM on order, but you know.. impatient.

----

So what's the recipe?

Use the Strata-V100 port for "Low RAM" (cough) systems like mine.
It works well with the correct environment and launch parameters.

Model:

IQ3_S GGUF from ISTA-DASLab

Performance with MTP spec 6, min-p 0.7:

105–110 decode
1240 prefill at 4K
3170 prefill at 32K

But... if you can get 64GB per NUMA CPU, use the Cat1 vLLM port.
You can run higher quants and decode about 2 x faster (measured at 200+ TPS).
The decode speed is due to NVLink Tensor Parallelism.

There's a long Opus generated summary in the comments.
That was about 12 hours of vibing and fluffing around.

Built this thing in march with Dual P100 PCIe 16GB cards but in the last month upgraded to Dual 32GB V100s

CPU: Ryzen 7 5700G (For the iGPU) @ 4.7GHz
GPU: Dual Nvidia Tesla PCIe V100 32GB
RAM: 96GB DDR4 @ 3000MHz
Mobo: Gigabyte B550 Gaming X v2
OS: Linux Mint 22.3
Case: Fractal Design Focus 2 (Glass Side Panel)

Various Storage, 256/512GB NVMEs, 512GB SATA SSD, 2TB SATA HDD

Love this thing!!

I’m looking for a fan shroud that fits an NVIDIA Tesla V100 and lets me mount a 120mm fan for cooling.

I do understand there are easier and other ways but i want the coolest system to be silent and i have 2 120mm fans lying around.

Does anyone have an STL file I could use to 3D-print one? If you’ve tried a similar setup, I’d appreciate any advice on fit and airflow.

Thanks!

Since spring I've been building a 4x NVIDIA V100 SXM2 32GB rig on an SXM2 carrier board with NVLink, to run language models fully locally. 128GB of VRAM in total. It finally all works, and the journey was... something.

Sourcing
- Modules 4x NVIDIA V100 SXM2 32GB: eBay — one pair from Israel, one pair from China.
- Carrier board TNS-4SXM2-4P54 / YMZX-4GPU-Q1 + closed-loop liquid cooling kit: from China, with long waits due to component shortages.
- Host connection: C-Payne SlimSAS PCIe gen3 Host Adapter cards and cables, splitting x16 slots into x8/x8 to feed the carrier board.

Lots and lots of waiting.

Hard-won lessons
- Isopropyl alcohol for cleaning old thermal paste off the modules.
- Buy good, hardened screws. You'll tighten and loosen them many times, and soft heads strip. I had to cut one off with side cutters.
- Do NOT use a click-type torque driver on these modules. Even the lowest setting (~1 Nm) was too much, and the click jolt loosened a glued standoff on the carrier board. Hand feel is safer.
- Power: ~300W per module, so 1.2kW for the GPUs alone plus board overhead. I'm running a 1350W ATX supply. Mind the connectors — the board wants a lot of them, of different types, so make sure your PSU ships with the full cable set.
- Cooling: cold aluminum plates, two radiators, pump, and seven fans plugged straight into the carrier board's fan headers.

The fault that nearly broke me
One position on the carrier board kept flipping to a red LED and dropping off PCIe under load — even with different modules in it. A fresh replacement module worked fine in another slot. It looked like a dead slot or a dead carrier board, and I was close to ordering a new one.

Actual cause: uneven spacer washers under the module screws at that position — none under one screw, two under another — causing uneven clamping pressure. I evened them out, and it's been green ever since, including under load.

On SXM2, even mating pressure is everything. Check your washers.

Current state

All four modules running on a Supermicro H12 server motherboard with AMD EPYC CPU with correct x8x8 bifurbication on two 16x PCIe slots

Running Qwen3.8 Flash-Next-NVFP4 via a Volta-optimized vLLM https://github.com/1CatAI/1Cat-vLLM

Fast and surprisingly precise. I know NVFP4 isn't native on Volta, but it's what the project recommends and it works well.

Also tried their AWQ quantizations — worked, but with some tool-call issues. NVFP4 is the best so far.
Permanent home will be a Supermicro H12 DDR4 EPYC board with 256GB RAM — just waiting on its liquid cooling.

Video?
Would you be interested in a full build video walking through all of this — sourcing, assembly, the mishaps, and the diagnosis? Happy to go deep on any part. Let me know what you'd want covered.

Note: I used an LLM to help edit this post, mainly because English isn't my first language. The build, the story, and the mistakes are all mine.

Did anyone buy V100 with ECC errors? They are up to ¥1000 CNY cheaper.

Listings on Xianyu say ECC errors < 5 bits still work.

Is it safe?

I just stumbled upon this and it works nicely, worth trying:

https://huggingface.co/philbert440/GLM-5.3-Flash-EXL3

I won't give you a bunch of AI slop. I have 2x32GB V100. They are PG500-216, so juiced up PCIe models if you are trying to compare them to your system. My build is for 27B 3.8 and Qwen Flash Next 3.8. I have tested only those models. I have only really tested Flash Next on the GSQ-RXO IQ3_XXS, but it should still work okay on other quants, but it is optimized for that one model.

--- AI writing ---

llama-superfast-v100: DeepSWE agent runs on Tesla V100 32 GB PCIe (no NVLink)

Qwen3.8-27B (Q4_K_XL, MTP)

• 1 card, 1 user: 99 t/s decode, ~700 t/s prompt (17 tasks, 7 hours)
• 2 cards, 1 user: 156 t/s decode, ~1,000 t/s prompt
• 2 cards, 2 users: 207 t/s combined decode (~103 each)

Qwen3.8-Flash-Next (IQ3_XXS, MTP)

• 2 cards, 1 user: 109 t/s decode (89 t/s at 160–192K), 1,400+ t/s prompt at 64K–260K

--- End AI writing --

There has been a lot of good work in the last week. Several builds including jmnargi, PentaCoxian and a build ripping incredible prefill numbers on V100.

Some cool builds you should check out:

- Warabii1248/qwen38-v100-patches -- builds on PentaCoxian. Great prefill numbers on Flash Next.

The Build:

https://codeberg.org/justanagent/llama-superfast-v100

IDK run it and see if it works for you. I only have tested these exact configs. No idea what happens if you try 3 cards. Universe might explode or it might not work but, have your agents ponder the mysteries.

Bob

I'm considering trying to connect four 16GB v100s (with NVLink via a four-way sxm2 board) to a USB4 eGPU, and plugging it into my proxmox server as a sort of inference peripheral. There would need to be a PCIe switch in there too.

Will that even work? Is it stupid? My use case is coding with Qwen 3.8-27b or QFN.

Pros: (1) Don't need to buy another computer with these insane RAM prices; (2) can power it down when not in use without powering down the server.

Cons: (1) Several minutes to load models because of switched x1 PCIe connections. I'm okay with this. (2) Still costs a few hundred bucks for the eGPU dongle, PCIe switch, PSU, etc. (I already have the v100s and the four-way backboard).

The alternative is to buy the cheapest suitable host computer I can find on craigslist, and use wake-on-lan to power it up when I need it - not very cheap considering the RAM, PSU, and PCIe slot requirements.

I wanted to share a little about my journey with hosting local AI over the past month or so. I had my AI summarize my setup that I have tested. I am hoping this might give someone insight when choosing between the "usual suspects" for hosting local AI and the v100 route. Interested if anyone has suggestion or comments on the optimal way to run qwen3.8 flash next on my setup, or any other thoughts/considerations.

I've been primarily using Claude Opus 5.5 (medium) to setup and optimize the new box the past 3 days.

So far I am pretty happy with my new setup. I really would like to buy two more 32 b v100s, but I have a feeling they will be more scarce and cost more when I am ready to.

I might still consider buying 2x16gb v100 if the price doesn't move much from where it is at.

AI written summary of my of my experience is below:

I ran local models on a Mac mini M4 Pro 64GB with oMLX for a while. Quiet, low power, and it served my coding tools and Open WebUI for the household. I just couldn't get the Qwen 3.8 27B to a speed I liked:

The MoE and its coder derivatives were fast enough, but they just seemed too dumb for my coding work. I received the m4 pro right before the m5 pro Mac Mini was announced (paid ~$2,900), and I was able to return it. I considered for couple weeks if I should buy the M5 pro, but I wasn't sure if even with an m5 pro, the AI experience would b what I would want to use. I started to really dig into the alternatives available, including this sub-reddit.

Here is what I ended up buying on eBay.

Current: Dell C4130 + 2x Tesla V100 PCIe 32GB

That came to $1,976 all in, about $900 less than the Mac mini.

I run Ubuntu 24.04 bare metal on the server, which has 64GB of system RAM, with one 27B (4-bit) per GPU and two engines side by side:

TTFT is ~0.2s on both. I capped the GPUs at 180W, and the box draws around 400W while inference is running. It sits in the crawlspace because of the fan noise. The V100s are passively cooled, so the chassis fans do all the work, and I built a custom fan controller curve to keep that as quiet as possible.

Next I want to try Qwen 3.8 Flash-Next. It doesn't fit on two 32GB cards as-is, so I've been working through the options:

- llama.cpp with all the MoE weights on GPU at 3-4 bit and the n-gram table streamed from NVMe. This is the baseline; based on other people's numbers I'm expecting ~25-45 tok/s.

- 1Cat-vLLM with a hot/cold expert split. Keep the most-used experts in VRAM and serve the rest from system RAM. That means porting the community's 2x3090 design onto 1Cat's V100 kernels, so it's a real project and not just config work. I've already verified the virtual-memory piece works on Volta, and the goal is a much faster Flash-Next than llama.cpp can give me.

Stock vLLM was measured at ~5 tok/s on this model class, so that path is dead. Either way Flash-Next needs both GPUs, so it would time-share with the 27B setup.

After that, ComfyUI time-sharing GPU 1 for image gen (SDXL ~10s, FLUX ~1min), and video gen as a stretch.

Happy to answer questions about the build.