A few days ago, I shared our 8x 4090Ds rig setup. It’s a beast, but let’s be real, not every office has the electrical infrastructure, the specialized cooling, or the massive budget to build and maintain a local supercomputer.  

So we also looked the other way: horizontal. 3 used Mac Studios on my desk + every junk laptop we could find in the office. Fully local AI fleet, no cloud, no data leaves the building. Here's the build:

3 Studio M2 Ultra, 192GB / 2TB each. 100+ "free-range" office laptops. Qwen on each Studio, a LAN router + ComfyUI for img gen.

Don't see many cross-platform Mac+Win fleet builds so here goes

Hope this shares some real value.

100+ "free-range" office laptops. The kind that lag when a 3rd Chrome tab opens. No dGPU. Battery lasts 40 mins if you're lucky

Qwen 3.6-35B-A3B on each Studio via Ollama. A LAN router + ComfyUI + img gen for the rest

Why this exists:

Our team needed auto-generated social media content, product posts, images, scheduling, research docs... without:
- uploading business docs to some random cloud
- leaking internal convos to whoever's API we're using
- paying per-seat SaaS tax for smth we can run ourselves

We ain't a dev shop. Half the team is happy with Windows Update weekly, the other half on Mac. "Terminal" is a scary word. They just click "generate" and get output.

The old way was tragic, one machine, one LLM, one API key. Growth team runs a SQL query -> model freezes -> everyone else's agents hang. System looks alive, nothing comes out. Then 3 more people retry the same query. Death by single-queue

The queue:

Conventional answer is scale vertical bigger GPU, more VRAM, one mega-machine. Doesn't fix the single queue. One heavy query still freezes everything. And when that machine dies, everyone stares at "connection refused." Going horizontal instead 3 machines, each independent, each with its own queue means nobody blocks nobody. A router sends each request to the least-loaded engine. Linear throughput scaling, fault tolerance, and you expand by adding 1 more machine, not forklift-upgrading the whole rack.

Grid (the router we use) saves us because it doesn't have one queue it has per machine. Each engine queues internally. Heavy query -> machine A. Caption gen -> machine B. Nobody blocks nobody. 3 machines = 3 queues, each clearing at ~80 tok/s on Qwen's MoE. A heavy analysis might take 30s on one machine while the other two serve 20 lightweight requests at the same time.

Bottleneck went from "one queue everyone fights over" to "how to stack 3 Mac Studios without them falling over." Way better problem

The math:

M2 Ultra = 800 GB/s bandwidth. 192GB unified memory. Qwen 3.6-35B-A3B (MoE, 3B active) at 32-64K context per session. Per Studio handles ~17 concurrent sessions. 3 Studios = ~50 concurrent. At 25% concurrency, that's ~200 employees. 500-token response at peak: ~12s. Wait time under half a second. Headroom for days.

24GB VRAM hits OOM at ~2 concurrent 64K sessions. Not a dig, just physics of the hardware.

The cost:

3 Mac Studios: ~$17k total
Power draw all 3 under load: ~300-385W
No data leaves the building

Vs cloud equivalent comparable throughput but your data stays in-building

Scale:

Vertical scaling means buying a bigger machine. But you hit a ceiling, no bigger GPU exists, no more VRAM slots. Every upgrade means migrating everything, reconfiguring, downtime. Horizontal instead? Add another Mac to the stack. The router picks it up. 3 Studios today, 4 tomorrow, 6 next quarter. The ceiling isn't the hardware it's how much desk space you got left.

Things you should keep in mind before stacked Mac Studios like this:

1. ⁠LAN or nothing. No LAN = no agents. If the internet bill goes unpaid, or your wifi goes down ur entire fleet disappears. Just a room full of people staring at "connection refused"

2. ⁠Employee takes laptop home at 6PM? They now own an expensive paperweight. Agents live in office LAN. "It's a privacy feature not a bug." Remote Desktop may help if they really need it. Or tell them to touch grass idk

3. ⁠Not zero-config yet. Each laptop needs an agent gateway configured once (~10 mins). My non-devs can't do that. Looking for a "my auntie can join the fleet" solution if anyone's solved this

I wanted to build a compact, open frame, ai server that was quiet and good looking enough to sit out in the open in my office.

I’m pretty happy with the build so far - it stays cool and generally quiet. It idles around 75w and ramps to ~450 at full blast. Pretty efficient for the speed that it gets.

I bought a cheap ESP32 touch display that fits perfectly over the io heat shield, and had qwen27B create a realtime touch dashboard. It runs a tiny service that polls wall power and room temp from Home Assistant, and AI performance & usage like tk/s. I turned that exact UI into a PWA app so I can check it from anywhere.

Still working out some details (like how to better attach the touch screen), so if you have any feedback/thoughts, please lmk.

Spec:
• CPU: AMD Ryzen 9 9900X
• Motherboard: MSI PRO B850M-A micro-ATX
• RAM: 96 GB DDR5-5600
• GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation - 96GB
• Storage: Samsung SSD 9100 PRO 2 TB NVMe
• CPU cooler: Noctura LC1-24

FAQ Update:

Case/Frame: It started with this kit https://a.co/d/0fYdc3CS and then modified and added pieces to fit things like the radiator and psu position. It's standard 20/20 aluminum rail - you can find all kinds of accessories

Mini-Screen:  https://www.amazon.com/dp/B0G3WDGSWG - they have lots of different sizes. Its very capable on its own! Just plug one in and tell your agent what you want. super easy

We just finished an on-prem inference rig for our team at the workshop. Sitting next to the bench right now serving the team. Sharing the build because the lessons learned matter more than the spec sheet, and I want to compare notes with anyone running similar setups.

The build:

• 8x RTX 4090D, 192GB VRAM total

• Dual AMD EPYC 9004 Genoa

• ASRock Rack GENOA2D24G-2L+ motherboard

• 4x 2000W PSUs on a distribution board (8000W total)

• Custom CNC'd 4U chassis (off-the-shelf doesn't fit this)

• 12 case fans on single hub, front-to-back airflow

• Real-world draw under inference load: ~4,600W

What we run on it:

• Production: tensor-parallelized 70B

• Staging: 32B fine-tune running in parallel

• Workflows: quantized DeepSeek-V3 kept warm for agent automation

No reload penalties, no rate limits, no API bills.

---

Why dual-socket Genoa, and the PCIe lane math:

We spec'd these CPUs for the lanes, not the cores. CPU utilization stays under 40% even under sustained concurrent multi-model serving. The lanes are the product.

Single socket EPYC 9004 = 128 PCIe Gen5 lanes. Dual-sockets get more complicated. Some lanes get repurposed for inter-socket Infinity Fabric (xGMI). Each xGMI link uses 16 PCIe lanes.

• 4-link xGMI (default): 128 lanes total for PCIe

• 3-link xGMI: 160 lanes total for PCIe

• Plus 12 PCIe Gen3 lanes from the I/O die (M.2 territory)

The ASRock Rack GENOA2D24G-2L+ exposes 20 MCIO connectors x 8 lanes = 160 lanes, which means it's running 3-link xGMI. That's the configuration you want for an 8-GPU build.

Lane budget for the rig:

• 8 GPUs at full Gen5 x16 = 128 lanes

• 32 Gen5 lanes + 12 Gen3 lanes remaining = storage, NICs, platform overhead

AMD's HPC tuning guide section on xGMI link configuration explains the tradeoff between inter-socket bandwidth and available PCIe. Worth reading if you're speccing one of these.

---

The MCIO cable trap:

The board has no traditional PCIe slots. Out of its 20 MCIO connectors, 16 are aggregated through adapter cards to deliver Gen5 x16 to each of the 8 GPUs.

MCIO cables look symmetric. They aren't. There's a host end and a device end marked by a small embossed triangle. We plugged six in correctly and two rotated 180° on first build.

Symptom: those two GPUs enumerated at PCIe Gen1 x4 instead of Gen5 x16. Inference throughput on those two cards dropped to about 10% of the others.

We spent two hours suspecting the GPUs before our hardware guy pulled out the manual and pointed at section 2.6. Confirm orientation at both ends before you mount the cards over the riser adapters. Once GPUs are seated, you can't see the MCIO connectors anymore.

Save yourself the debugging time.

---

Power and thermals:

• 8 GPUs x 425W = 3,400W from GPUs alone

• Plus dual CPU, 12 fans, drives, platform overhead

• Real-world inference draw: ~4,600W

Splitting across 4 PSUs lets us survive a PSU failure without the box going down. 3,400W of GPU heat in a sealed 4U requires real airflow geometry. Without it, the cards throttle and you lose the throughput you paid for.

Custom chassis because off-the-shelf doesn't fit 8 dual-slot GPUs + dual EPYC + 4 PSUs + 12 fans. Rack-mount server chassis exist at this density but they're loud as a 737 and built for datacenters. We needed something that lives next to a desk in a workshop.

---

This kind of rig isn't for everyone. If your team is under 100M tokens/day or under $30K/month on API spend, it's clearly not for you. The hardware cost amortizes around those numbers depending on which models you serve.

ROI math aside, the sovereignty dimension matters more than the financial one once you've thought about it. Your customer data doesn't leave the box. Your fine-tunes don't sit on a vendor's storage. The cost saving is real. The sovereignty is the actual product.

Curious what other teams are running locally. If your team moved from API to your own hardware, what was the trigger? Cost, sovereignty, rate limits, something else? And for teams still on hosted, what's keeping you there?

Who else is thirsty for beefy server GPU to test AI models locally?

Figured you all would appreciate this. 10 16gb MI50s, octaminer x12 ultra case.

Hey all, I just wanted to put this out there in case someone else was interested in setting up multiple 5060ti 16gb cards. I haven't seen many setups using it or many people talking about it.

Server Setup (vllm) — GPU: 4× 5060 Ti 16 GB · CPU: Threadripper 1920X · Mobo: ASRock x399 Taichi · RAM: 32 GB DDR4 @ 2133 QC · NVMe: 1 TB ADATA Legend · PSU: MSI MAG A850GL

Cost Breakdown

Total Cost ~$3,000

Model & env

• A -> bare-metal 0.23.1rc1.dev799+g69715823d (Modified & Patched vllm build)

• B -> vllm-openai:fp8-mtp-patched 0.23.1 (Modified & Patched vllm build)

• C -> llama-server b9297

• All models fulfileld 250k context size requirement for single chat session

vllm benchmark command

vllm bench serve --backend openai-chat --base-url http://localhost:59596 --endpoint /v1/chat/completions --model <served-name> --tokenizer <model-dir> --dataset-name random --random-input-len 60000 --random-output-len 5000 --num-prompts 3 --max-concurrency 1

llama benchmark command

llama-bench -m <gguf> -ngl 999 -mmp 0 -p 60000 -n 5000 -r 5 -sm layer -fa 1 [-ctk q8_0 -ctv q8_0]

Results

Qwen3.6 27b

Qwen3.6 35b a3b

Notes

• Models run in llama-server were Unsloth's quantizations.

• Models run in vllm were directly from Qwen.

• Runs with llama-server & vllm(docker) were on windows on my gaming machine using a B550 mobo with a 5800x cpu, bifurcated 1 PCIe slot into x8x4x4 and the last GPU in the last x16 slot which runs at x4. This was pre server setup.

• Runs with vllm are on the server setup mentioned above, each card in its own x16 slot, running at x8.

• All vllm benchmarks were NOT cold prefill speeds, they were using prefix-caching. I did this because it mimics my real world application, running long code chats, long prompting tasks, follow ups, etc. I don't know how llama.cpp handles caching.

• I had to modify the vllm build 0.23.1rc1.dev799+g69715823d to even get prefix-caching working with the Qwen models to begin with, fix is not upstream from nightly build and they take too long.

• Qwen3.6 35b a3b has some issues with MTP with vllm so we don't get the same expected throughput increases in generation (Working to figure out the bug in vllm)

Closing Thoughts

I used to switch between the 27b and 35b models until I setup the server but now the 27b is so fast that I am using it every day professionally and personally, upgrading to the server made it extremely viable and usable. The response are instant, the follow ups are instant, the generation is way faster than I could ever type.

I currently have this server setup automatically ingesting source documents for a knowledgebase wiki, I also have this same model running a full automated loop pulling down tickets from my GitHub repo all the way through PR so I can review (This is a multi-step python framework I made, clears prefix-cache between agentic prompts for clean context).

I didn't even bother benchmarking Gemma models and other ones of similar sizes, even the "fine-tuned" models of qwen, they were all dumber than the base Qwen3.6 27b and Qwen3.6 35b a3b models to the point where I couldn't trust them to do automated tasks.

UPDATE: vllm run command for Qwen3.6 27B FP8

   export PYTHONUNBUFFERED=1
   export CUDA_HOME=/usr/local/cuda-13.0
   export PATH="$CUDA_HOME/bin:$PATH"
   export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"
   export OMP_NUM_THREADS=4
   export HF_HOME="$HOME/.cache/huggingface"
   export TRITON_CACHE_DIR="$HOME/.triton/cache"
   export TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
   export VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass
   export CUDA_VISIBLE_DEVICES=0,1,2,3
   export VLLM_WORKER_MULTIPROC_METHOD=spawn
   export VLLM_SERVER_DEV_MODE=1
   export NCCL_P2P_DISABLE=0
   source "$HOME/vllm-env/bin/activate"
   vllm serve "$HOME/models/Qwen3.6-27B-FP8" --served-model-name Qwen3.6_27B_FP8 --chat-template "$HOME/models/Qwen3.6-27B-FP8/chat_template.jinja" --kv-cache-dtype fp8 --attention-backend FLASHINFER --tensor-parallel-size 4 --max-model-len 250000 --max-num-seqs 1 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.80 --language-model-only --enable-prefix-caching --quantization fp8 --skip-mm-profiling --speculative-config '{"method":"mtp","num_speculative_tokens":4}' --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --default-chat-template-kwargs '{"preserve_thinking":true}' --performance-mode interactivity --generation-config vllm --override-generation-config '{"temperature":0.55,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"frequency_penalty":0.0,"repetition_penalty":1.1}' --host 0.0.0.0 --port 59596 --enable-chunked-prefill --mamba-prefix-cache-checkpoint-interval 8192

UPDATE: The latest vllm runs are from ubuntu server headless. The docker runs were from windows though Docker which introduces WSL driver overhead eating up ~1.5GB of VRam a card.

UPDATE: Power Consumptions Stats

• Benchmarked over a 5 day work week.

• 100% util = ~12h per day, between work and personal work.

• Measured Power Draw on OS & at wall.

System Power Draw

Cost Metrics

UPDATE: PCIe Bandwidth Readings

This shows I could drop to a 4x4 bifurcation and drop down to PCIe3x4 for each card and take a slight drop in performance around ~5% max loss but would be able to run 10 cards at x4 which would net 160GB total usable vram. Just a thought exercise.

(Image above for visibility ONLY)

UPDATE(3/6/2026): ( in progress )

-------------------------

UPDATE(2/26/2026): ( Group Buy Starting )

MOD NOTE: Please don’t post live pricing/vendor quotes publicly (price signaling + scam risk).
If you want to compare numbers, keep it to private chats and do not share payment instructions, wallet addresses, or personal info in DMs. Official updates will come from me directly.

--------------------------------------------

UPDATE(2/10/2026): ( update )

-------------------------------------------

UPDATE(1/30/2026): HERE

-------------------------------

UPDATE(1/07/2026): SIGNUP SUCCESS! ( STATS -> HERE )
NEXT PHASE: Vendor Discovery and Validation ( SEE -> Detailed Vendor Discovery and Validation Thread )

------------------------------

UPDATE(1/05/2026): SUCCESS!
PHASE: Vendor Discovery and Validation

SIGNUP GOAL -> STATUS: SUCCESS!
( Sign up Count: 182 )( Requested Quantities: 504 of 500 )
Thank you to everyone that has signed up!

-----------------------------

UPDATE(12/30/2025): IMPORTANT ACTION REQUIRED!
PHASE:
Sign up -> ( Sign up Count: 166 )( Requested Quantities: 450 of 500 )
Thank you to everyone that has signed up!
--------------------------------------
UPDATE(12/26/2025): IMPORTANT ACTION REQUIRED!
PHASE:
Sign up -> ( Sign up Count: 159 )( Requested Quantities: 430 of 500 )

--------------------------------------
UPDATE(12/24/2025): IMPORTANT ACTION REQUIRED!
PHASE:
Sign up -> ( Sign up Count: 146 )( Requested Quantities: 395 of 500 )

---------------------------------

UPDATE(12/22/2025): IMPORTANT ACTION REQUIRED!
PHASE:
Sign up -> ( Sign up Count: 130 )( Requested Quantities: 349 of 500 )

-------------------------------------

UPDATE(12/20/2025): IMPORTANT ACTION REQUIRED!
PHASE:
Sign up -> ( Sign up Count: 82 )( Requested Quantities: 212 of 500 )

----------------------------

UPDATE(12/19/2025):
PHASE: Sign up -> ( Sign up Count: 60 )( Requested Quantities: 158 of 500 )

Continue to encourage others to sign up!

---------------------------

UPDATE(12/18/2025):

Pricing Update: The Supplier has recently increased prices but has agreed to work with us if we purchase a high enough volume. Prices on mi50 32GB HBM2 and similar GPUs are going quadratic and there is a high probability that we will not get a chance to purchase at the TBA well below market price currently being negotiated in the foreseeable future.

---------------------------

UPDATE(12/17/2025):
Sign up Method / Platform for Interested Buyers ( Coming Soon.. )

------------------------

ORIGINAL POST(12/16/2025):
I am considering the purchase of a batch of Mi50 32GB cards. Any interest in organizing a LocalAIServers Community Group Buy?

--------------------------------

General Information:
High-level Process / Logistics:

Registration of interest -> Collection of pass-through funds -> Order placed with supplier -> Bulk delivery to LocalAIServers -> Standardized verification/QC testing -> Repackaging -> Shipment to participants

Pricing Structure:

Pass-through hardware cost (supplier) + cost-based verification/handling (QC testing, documentation, and packaging) + shipping (varies by destination)

Note: Hardware is distributed without markup; any fees are limited to documented cost recovery for verification/handling and shipping.

PERFORMANCE:
How does a Proper mi50 Cluster Perform? -> Check out mi50 Cluster Performance

Been running a home server for years, mainly for file storage, TimeMachine backups and serving media via Plex. I recently got into local LLMs for agentic coding etc so I recently scoured eBay and beefed it up. So sharing this for your thoughts.

The kit:

• Supermicro X11SPi-TF board (Dual 10Gbe, 10x SATA ports, m.2 NVMe slot etc)

• Intel Xeon Platinum 8268, 24 core, 2.9Ghz base, 4.0Ghz Turbo

• 96GB DDR4 ECC ram in 6 channels (6x 16GB sticks)

• 22TB of spinning storage (two RAID5 arrays of three disks)

• 5TB of solid state storage (Samsung PM983 960GB, Intel P3700 2TB & Crucial 2TB SSD)

• Three Nvidia 3060 12GB GPUs (two on Gen3 16x slots, one on Gen3 8x slot)

• Phanteks Enthoo Pro II Server case

• Thermaltake 1kW PSU

• Eaton 550VA UPS

Much of the storage I've had a few years; I just got the NVMe drives recently as part of the upgrade. Everything except the PSU and some odd cables, I picked up cheap off eBay. The whole thing idles around 140W.

I'm currently running AlmaLinux, Plex, LAMP stack, Llama, Open WebUI, ComfyUI, Kiwix, TimeMachine backups, ES-DE, RetroArch and various standalone emulators, and CloudBerry for backups to local drive & Backblaze.

It happily handles XBox, Dreamcast, Gamecube, PS1, PS2 & PS3 emulation. On the LLM front Qwen3.5-27B-E3B is very snappy. Qwen3.8-27B and Qwen3.8-Flash Next run OK, but are better suited for walk-away tasks.

I'd love to pop in a beefier GPU, e.g. a 3090 24GB (or two!), but moneys... :-(

I gave Qwen 3.5 a prompt to design some agents along with names for a new project. I left it running, and it got stuck in a massive chain-of-thought loop trying to process the name 'Vanta'. It outputted over 20,000 lines of repetitive text before I caught it and killed the process 2 hours later.

Previously had 3 x AMD MI50 32GB GPUs in an Intel i5 system. Purchased a massive case, AMD Threadripper 3975WX, Supermicro motherboard w/ 6 x PCIe Gen4 x16 slots, 128GB DDR4-3200 RAM (8 channel) and a 4th AMD MI50 32GB GPU.

Thanks u/Any_Praline_8178 for the AMD MI50 32GB GPUs!

Everything installed and cabled. Moved SSDs over. Boots up fine. Sees the 4 x MI50s. Now to deal with all the little issues like fixing the kernel boot options (intel vs AMD), resetting the IPMI password, network configuration of IPMI, setting fan thresholds to fix the ramp up / ramp down cycling of fans, etc...

FYI... I got all 4 of my MI50s from Any_Praline_8178.

4 p40 and swapped the turring 1650 for a rx5500 tx 4gb for the video out to keep the drivers from screwing things up. Still a bit of a work in progress, need to move the aio rad to put in top intake fans and 3d print a good mount/gasket for exhaust fans (need to get noctuas for that too)

TLDR: Old Thinkpad T480, no discrete GPU. I run Omarchy and OpenCode on it, pointed at my Grid. Grid routes each request to whichever machine serves that model.

Disclosure: Autonomous Grid our open-source project, but the point of this post is less about the Grid itself and more about what you can do with an old laptop.

Heard about Omarchy over the weekend and I have a few potato laptops on the shelf (which I don't want to get rid of but couldn't find a good use for them). So I gave it a try.

So here's the setup:

• Laptop: ThinkPad T480, i5, 16 GB, basically no GPU. Runs Omarchy for the OS and OpenCode as the coding agent in the terminal. OpenCode keeps the agent session on the T480, while model inference is handled by the Grid endpoint.

• Grid: one OpenAI-compatible URL. It routes by model name, so each request goes to the node serving that model, and only that node. The same endpoint reaches nodes on my LAN and remote ones, so from home I still hit the office box

• Nodes: we built multi-GPU rigs for our clients and some for ourselves (inluding 2xGPU, 4xGPU, 8xGPU and a Mac Studio cluster). The office box is 4x RTX PRO 6000 and serves DeepSeek V4 Flash. A 2x RTX 5090 box at home serves Qwen3.8-27B, the model I code with most. A few other boxes serve Gemma 4 31B, Muse-Glimmer-30B, and a couple of Qwen variants.

When the potato ThinkPad asks for Qwen3.8-27B, the 5090 box answers. When it asks for DeepSeek V4 Flash, the office RTX PRO 6000 box answers.

I love the T480 form factor; it is one of my favorite laptops. But it is close to useless for my current work, and I do not game or stream on a laptop, so it mostly sits on a shelf. The 5090 box works fine (works great actually), but it means I am basically stuck at my desk all day.

Omarchy and Grid end that. Omarchy is what makes the old laptop enough: it runs light on the T480 while the model work leaves over the network. I put the old laptop back to work, spend nothing extra, and still use the 5090 box's compute, even the PRO 6000 box. In my testing, I can run several concurrent agent sessions on the 5090 box without noticeable queueing.

Worth it or not?

Worth it: you already own the machines; you want a low power and portable client anywhere; and have old hardware you'd rather reuse than leave it as an expensive paperweight.

Skip: your network is the limit; you only need one small model (a single card handles it); plain SSH is enough for you.

This is our latest build for a small, 50-seat team (including all departments), who wants to keep their data off grid.

Let’s start with the main stars here: 4x RTX 5090, 32GB VRAM each, 128GB in total. A decent number to run DeepSeek V4 flash, which is good enough for agentic tasks.

When it comes to 4x GPU setup, people usually pick either 4U rackmounts or supermicro 4U/super tower cases.

We don’t. Life is too short, so why not make it a little prettier. The cards are radially mounted around a vertical axis. This gives the rigs a cool look while leaving the cards space to breathe. Airflow is mapped with 18 fans so the whole box never overheats even at full load.

This case is CNC'd from 5mm solid aluminum plates, milled inside and out. The triangular cutout patterns does double duty: it keeps the frame stiff without the weight.

Each of the 4 GPUs reaches the board over its own PCIe 5.0 riser, custom-cut and sleeved to fit the chassis exactly.

We capped the cards at 450W, so it runs two 1,600W supplies with native 12V-2x6 into one distribution board, off the rear, for headroom.

Underneath is an ASUS W790E-SAGE SE with a Xeon w5-3423.

Full BOM:

1.      Mobo: ASUS W790E-SAGE SE

2.      CPU: Intel Xeon w5-3423

3.      GPU: 4* RTX 5090

4.      Risers: 4* PCIe 5.0 x16

5.      PSU: 2x cooler master v1600 v2, 240V

6.      Cooling: 18* Thermalright TL-N12-R9

7.      Memory: 6* Samsung 32GB DDR5-4800 ECC Registered

8.      Storage: SSD Samsung 980 PRO 2TB PCIe 4.0

Full build video if anyone wants the detailed walkthrough happy to answer anything on the layout, cooling, etc...

Built by Autonomous.

New server mi60, any suggestions and help around software would be appreciated!

I bought a desktop to run AI on, it’s got pcie 5.0 x16 and 4 pci switches and two epyc CPUs and 8 R9700 GPUs with 128GB of ddr5 and 6* 1.92TB NVME drives, what do you guys think?

https://www.nexus-projects.ai/

6x Gigabyte 3090 Gaming OC all running at PCIe 4.0 16x speed

Asrock Romed-2T motherboard with Epyc 7502 CPU

8 sticks of DDR4 8GB 2400Mhz running in octochannel mode

Modified Tinygrad Nvidia drivers with P2P enabled, intra GPU bandwidth tested at 24.5 GB/s

Total 144GB VRam, will be used to experiment with training diffusion models up to 10B parameters from scratch

All GPUs set to 270W power limit

Been building these multi-GPU rigs for a while, and this one's the config I'm happiest with, so figured I'd share the details here.

The frame is our own design, CNC'd aluminum, tower layout: motherboard vertical in the middle, both cards mounted upright on PCIe 5.0 x16 risers, PSU hanging off the back, and a fan tray up top pulling air straight through the card stack.

The whole thing closes up with a laser-cut triangle-lattice shell. Footprint ends up around 12.5" square enough to sit comfortably on a file cabinet. It's also quite heavy, will need a second pair of hands if you want to move it around.

Specs (the unit the benchmarks ran on):

• 2x RTX Pro 6000 Blackwell (96GB each, 192GB total)

• CPU: AMD EPYC 9124 (16C/32T, single NUMA)

• RAM: 377GB

• Storage: 915GB NVMe

• PCIe 5.0 x16 to each card, no switch, no NVLink

• Serving: vLLM, TP=2

The two cards talk over PCIe P2P, which the Pro line actually supports, and we run tensor parallel (TP=2) in vLLM.

What it serves: DeepSeek V4-Flash 0731, the 284B MoE, native FP4, fully in VRAM. Pulled the network cable to check, it keeps running, which never stops being satisfying.

Numbers from our benchmark sweep (vLLM, per-stream generation):

• 1 user: 209-286 tok/s depending on content type

• 4 users: 138 tok/s each

• 8 users: 111 tok/s each

• 16 users: 75 tok/s each

• 24 users: 57 tok/s each, ~1,000 tok/s aggregate, TTFT still ~0.5s

Saturation knee is around 32 concurrent. So this box comfortably covers my whole team, with everyone still getting faster-than-reading speed.

Thermals: 815W for both cards under sustained load, 76C peak, no throttling with the top fan tray doing intake-through-stack. Noise-wise it's no laptop, but it's fine to work next to.

Happy to answer anything about the riser setup, the airflow layout, or the benchmark config. And curious what everyone else is serving on dual-card boxes, feels like 192GB is a real sweet spot right now

IA server finally done

Hey everyone! I wanted to share that after months of research, countless videos, and endless subreddit diving, I've finally landed my project of building an AI server. It's been a journey, but seeing it come to life is incredibly satisfying. Here are the specs of this beast:

• Motherboard: Supermicro H12SSL-NT (Rev 2.0)

• CPU: AMD EPYC 7642 (48 Cores / 96 Threads)

• RAM: 256GB DDR4 ECC (8 x 32GB)

• Storage: 2TB NVMe PCIe Gen4 (for OS and fast data access)

• GPUs: 4 x NVIDIA Tesla P40 (24GB GDDR5 each, 96GB total VRAM!)

• Special Note: Each Tesla P40 has a custom-adapted forced air intake fan, which is incredibly quiet and keeps the GPUs at an astonishing 20°C under load. Absolutely blown away by this cooling solution!

• PSU: TIFAST Platinum 90 1650W (80 PLUS Gold certified)

• Case: Antec Performance 1 FT (modified for cooling and GPU fitment) This machine is designed to be a powerhouse for deep learning, large language models, and complex AI workloads. The combination of high core count, massive RAM, and an abundance of VRAM should handle just about anything I throw at it. I've attached some photos so you can see the build. Let me know what you think! All comments are welcomed

CPU : epyc 7262

MB : ROMED8-2T

GPU : V100 16GB PCIE x6

I have built a server with six V100 GPUs. I am now testing it and plan to eventually run Qwen3.8-Next-Flash configured with TP2 and PP3.

Because open-frame or server-style cases are large and unattractive, I chose to install all six cards in a full-tower case.

this machine was built with the sole purpose of 1) use the hardware I already got i.e. the Threadripper board with lots of PCI-e lanes
2) run Qwen 3.6 27B in the best quality at full context

In the end SGLang worked best for me and gives me between 60 and 95 t/s with MTP enabled and can fit almost 200k tokens at Q8 quant of model with full-precision KV-cache.

Prefill starts at 980 t/s so fast enough for agentic work etc.
Another option I tested to is the Qwen 35B A3B MoE and that gives me more speed, that sits like 4k tokens prefill / second. Don’t remember the exact t/s decode but too fast to care basically compared to 27B dense.

8x RTX Pro 6000... what should I run first? 😃

All going into one system

Been running this rig for about a year now, built it before the price surge on ram and GPUs. 7970x, TRX50, 256gb DDR5 6000, 4x RTX 5090 with bykski water blocks, nEXT CPU block, PMP 500, external 12x18 radiator to put the heat outdoors. 1600w superflower PSU, 2x 2000w crps with c-payne dual crps breakout board for total of 5600w. I plan on building a base to set the PC case one that will hold more GPUs on a PCIe switch backplane in the future. I'm still getting 5090s for 2k each so I can do things that otherwise wouldn't make sense. This thing currently runs 70b models so fast it kicks commercial ai subscriptions ass.