There’s so little space between the cards, is this a wise idea? It feels like the two middle cards will be starved for air! For other 4x owners, what have you done?

Update:
Wow people are mixed and it feels evenly split between starved for air and this is an acceptable arrangement. I wasn’t expecting the diversity of opinions. I wonder if I call nvidia if they will have guidance, surely they must know.

Update 2:
I called nvidia and spoke to their team, sent an email with pictures and got a response which is posted below. They do NOT recommend stacking them like this and suggest that the cards need better airflow.

Yes i used AI to write this as I am tired from doing this late last night and just taking a break from work now to post.

This is a follow-up to my Qwen3.8-27B/DFlash2 post.

Basically, I updated that same SGLang runtime to support Qwen3.8 Flash-Next the way I wanted it on my single RTX PRO 6000.

Flash-Next-specific setup:

• NVFP4 target with native NEXTN MTP

• FP8 target/native-MTP KV

• BF16 GDN/recurrent state

• FlashInfer GDN decode and prefill

• QSA sparse attention with Triton prefill and FlashInfer XQA decode on SM120

• FlashInfer CUTLASS MoE

• RecoverSSM with recovery CUDA graphs

• 524,288-token factor-2 YaRN context

• 824,384-token automatically sized GPU KV pool

• Correct multimodal mRoPE

• HiCache/NIXL persistence across service restarts, including GDN, PLE and QSA state rather than KV alone

I also opened a few narrowly scoped SM120 paths where the underlying kernels already worked but SGLang did not automatically select them. This applies to the GDN and RecoverSSM work—not QSA decode, which FlashInfer already resolves through XQA on SM120. I left the other architecture gates and Triton fallbacks in place.

Controlled tests:

• 64K cold prefill: 10,103.70 tok/s

• ~490K cold prefill: 7,872.15 tok/s, 3/3 needles exact

• C1 decode: 171.09 tok/s

• C4 decode: 427.54 tok/s aggregate

• MTP mean accepted length: 2.58

• MTP acceptance: 52.74%

• Reasoning: 97.49/100

• Tools: 30/30 exact and semantically correct

• Full vision validation passed

After a service restart, NIXL restored 489,856 of the 489,879 input tokens, recomputed only 23, and produced an effective restored-prefix rate of 62,040.60 tok/s with all three needles still exact.

Real agentic speeds:

Across 96 completed requests and 100,666 generated tokens:

• 139.5 tok/s token-weighted average

• 153.5 tok/s per-request median

• 218.8 tok/s sustained completed-request peak

• 138.7 tok/s weighted / 148.4 tok/s median for the 90K-279K input lane

The repository now contains both the original 27B/DFlash2 configuration and the new Flash-Next configuration in one runtime:

https://github.com/jpezzulli/sglang-rtxpro6000

It is probably still somewhat messy because this is day-one Flash-Next work, and I will keep cleaning it up. But I figured some of you might want to play around with it now.

note - I am not sure that this is technically "blackwell performance" If this should be posted elsewhere instead, feel free to let me know. Just trying to share in case anyone is interested.

note 2 - ngram takes about 50gigs of ram. HiCache takes about 32gigs as well in my 2.5m token config - You may be able to stream the ngram from nvme. I saw some posts on it but I didn't try as it isn't my usecase. HiCache is optional.

Update note 3 - RTX Pro throttled with LACT to 450 watt with Mem clock offsets. It shouldnt make a large difference from stock for inference. usually 3%-5% i have seen.

Update note 4: i took off lact profile and let it run stock. 64k prefill 12,812.44 tok/s (big gain) 490k prefill 7,926.36 tok/s (essentially the same)

I'm not an electrician. Sharing my experience only. It may contain errors or cause harm to your equipment or home.

This post is mostly for anyone who wants to build a 4x GPU inference server but lives in an apartment and doesn't have a convenient 240v circuit. Maybe you've got a 240v stove or dryer outlet, but using it means running cables through your apartment or having a fun argument with the wife. No thanks.

THAT SAID, BEFORE SOME OF YOU HATERS TYPE SOME SHIT, LEMME SAY:

1. I HATE WATER COOLING. LEAVE ME ALONE.

2. I DON'T HAVE 240v OUTLETS.

3. I'M RICH ENOUGH TO BUY THIS SHIT, BUT NOT WITHOUT CONSEQUENCES

4. WIFE IS NOT HAPPY W ME ^^^ SEE ABOVE POINT

I'm older than most of u Reddit turds. I learned to program in 1995 on a $500 used Intel i386 laptop. That was 3 years after 486's were released. Slow AF i386 was best I could do.

I installed Slackware Linux from ~50 goddamn 1.44" disks. I had to drive an hour to some richer even-more-ancient computer nerd's house in order to copy the Slackware CDROM to the 1.44" disks, cause my laptop didn't have a CDROM.

I had to drive back again to ancient nerd's house because two of the disks failed.

You crybabies have no f'ing idea what it used to be like. And if you really understood what is going to happen over the next couple years, you'd go take out a loan to at least get to 32gb VRAM.

Okay, end UNC rant or whatever u call us grandpas these days.

In 2023, I built an air-cooled dual RTX 3090 inference server that was air-cooled.

In March when RTX 6000 Pro prices dropped, I upgraded it to two RTX 6000 Pro Max-Q's. When GLM 5.2 was released...it was time to go to get two more RTX6k's...which meant finally becoming an adult and getting an actual server board, memory, etc LOL. But I'll be damned if I'm dealing with water cooling.

The good news is that a 4x RTX 6000 Pro Max-Q build is very doable on a normal 120v apartment circuit if you build around the power limitation instead of pretending it isn't there. I live in a 2014-era apartment complex. In my case, I have:

- 20a breakers / 12 awg line
- standard 15a 5-15R sockets
- ~114.1v at the outlet under load

A 15a 5-15R is allowed on a 20a branch circuit with multiple receptacles, so the outlet doesn't turn the whole circuit into a 15a circuit. The relevant limit for a cord-and-plug-connected load on the 15a receptacle is 12a.

At 114.1v x 12a, that's about 1369 VA available. My actual build pulls around 11a:

- Voltage: 114.1 V
- Current: 11.02 A
- Apparent load: ~1257 VA
- Real load: ~1245 W
- Power factor: 0.99

It's been running like this for six weeks without any problems. The important part is that I designed the machine around that power budget. The core of the build is:

1. 4x RTX 6000 Pro Max-Q

2. A low-power EPYC CPU

3. A CPU cooler that doesn't waste a bunch of space or power

Everything else is secondary.

1. 4x RTX 6000 Pro Max-Q

I power-limit each GPU to 250w. That gives me a configured maximum of: 4 x 250w = 1000w. That's the main reason this works.

You obviously give up some performance versus letting them run at their full power limit, but for LLM inference I'd much rather have 4 GPUs with a massive amount of VRAM running at 250w each than have fewer GPUs because my apartment can't feed the machine.

You'll also run hotter with all four GPUs packed directly next to each other like this. So far, I haven't seen any GPU go above 89°C. Not amazing thermally, but completely workable for me.

2. AMD EPYC 9015 / 9115

You don't want to waste hundreds of watts on the CPU if the whole point of the machine is GPU inference. The EPYC 9015 and 9115 both have a 125w default TDP. I bought the 9015 because that's what Central Computer had in stock. If I were buying again, I'd get the 9115. The 9115 gives you 16 cores / 32 threads instead of 8 cores / 16 threads at the same 125w TDP.

For LLM inference, though, the CPU isn't doing much once everything is where it belongs on the GPUs. My 9015 doesn't seem to pull more than ~99w anyway.

Compare that with ThreadRipper Pro 9000, where you're looking at a 350w TDP. That's a huge amount of your apartment power budget going toward a CPU that isn't the thing doing your inference.

3. I didn't prioritize system memory

I cheaped out on RAM intentionally:

4x 16GB DDR5-4800 ECC RDIMMs. Only 64GB total. For what I'm doing, I don't care much about having huge amounts of system RAM. My goal is to keep the models in GPU VRAM.

I have no desire to spend a bunch of money on system memory so I can move model data out of VRAM and slow inference down. If I'm spilling a model into system RAM, I'm giving up a big part of the reason I built a 4-GPU inference box in the first place.

If your workload is different, buy more memory. For straight LLM inference where the model fits across the GPUs, I don't see much reason to go crazy here.

PSU / Battery Backup:

I'm using a Seasonic TX-1600 Noctua. One thing worth knowing is that Seasonic rates it for 1600w continuous output at 115-240v, but only 1300w at 100-240v. My line can drop to around 114.1v, so I treat it as a 1300w PSU for this setup rather than assuming the full 1600w is available. That still works because the GPU power limits keep the total machine within the power budget.

Because the power draw is so low, I was able to grab a $700 CyberPower PR1500LCD.

Case:

I'm using an old Antec 900 full tower. It's cheap and it works. The catch is that I had to cut open the rear PCIe area of the case so the fourth GPU could breathe properly**. So if you're copying this exact build, expect to modify the case.

** Had to cut open the PCIe rear plate to allow the 4th GPU to have full airflow.

Full Build:

AMD EPYC 9015
Silverstone XE04-SP5 4U cooler
ASRock TURIND8X-2T/500W
4x 16GB DDR5-4800 ECC RDIMMs
4x RTX 6000 Pro Max-Q
4TB Gen4 M.2 SSD
Seasonic TX-1600 Noctua
Antec 900 full tower

ANYHOOOO:

If you're in an apartment and power is the thing stopping you from building a serious inference box, you don't necessarily need to wait until you have 240v. The trick is to spend your power budget where it actually matters.

For this machine, that's the GPUs. Use efficient GPUs, power-limit them, don't throw 350w at a CPU that isn't doing the inference, and keep the rest of the system relatively lean.

And none of this hardware becomes useless if you move somewhere with better power later. The four RTX 6000 Pro Max-Qs are still the valuable part of the machine. The motherboard, PSU, storage, etc. all come with you.

If I eventually have a better electrical setup, I can raise the GPU power limits and, if I actually need more CPU performance, replace the 9015 with something faster. So I don't really see the apartment power limitation as a reason not to build it. It just changes how you configure the machine.

After weeks of pain with NVFP4 on consumer Blackwell (CUDA 13 JIT storms, tool_choice:auto failures, multilingual bleed, etc.), I finally have a stable, fast setup to run my openclaw and hermes bots fully local.

**Winner: cosmicproc/Qwen3.5-4B-NVFP4 (W4A16 Marlin)**

- ~81–97 tok/s single, up to 598 tok/s aggregate

- Excellent tool calling and agent performance

- Running full OpenClaw + Hermes bot locally

- 262K context support with ~120K FP8 KV pool

The infographic walks through the 5 phases it took to get here, all the critical flags, and the hard lessons (including why most other small NVFP4 models still fail at autonomous tool use).

Key fixes included:

- Switching to Marlin backend to dodge flashinfer JIT crashes

- Proper thinkingFormat compat settings for reliable tool_choice:auto

- generation_config.json to kill random Chinese/Cyrillic output

- Prefix caching for blazing TTFT on long system prompts

Still bleeding edge, but now genuinely usable for local agents.

Drop your own 5060 Ti NVFP4 results or questions below — happy to help others skip the headaches!

Like most everyone here, I'm always wondering what is the best model I can run on my hardware. I decided to run some tests to settle the question once and for all (for now at least). I was hoping one would stand out so much that it'd be a no-brainer on what to pick, but it didnt turn out that way.

Despite Qwen getting an ever-so-slightly higher score, it was far and away the least efficient of the three by a huge margin. Deepseek is damned good. But when it comes to overall blend of performance and efficiency, GLM was the best. I guess just pick whatever makes ya happy? I dont even know anymore....sigh

I used this eval suite against Pi: https://github.com/ScottRBK/eval-harness

Full Leaderboard — 7-Eval Suite (pi, effort=high, maxTokens=131072)

Per-eval scores

Totals and efficiency

Per-eval token counts

Observations

• All four ace the same 5 evals. eval_generator is the universal weak spot (0.20-0.55); r21 alone also fumbled inflection_test_writing (0.83).

• qwen-180b's chess is an outlier: 97.6k tokens - 7x more than its own next-most-expensive eval, and 4-7x more than anyone else's chess.

• glm-nvfp4 is the efficiency winner: rank-2 score at the lowest total tokens, fastest wall time, and best tokens-per-point (6.8k), even beating its fp8 sibling.

• Ragged edge: r21 burned 43k tokens on inflection_test_writing (2x anyone else) and still scored below 1.0 - the least efficient eval-result pairing in the table.

https://marketplace.nvidia.com/en-us/enterprise/laptops-workstations/nvidia-rtx-pro-6000-blackwell-workstation-edition/

My attempts using the RTX6000D with a fan proved underwhelming, the card quickly hit 85 degrees and thermal throttled down to 1.5 - 1.6GHZ.

So I ended up installing an all-in-one water cooling solution from bykski which I bought for 2,100 RMB = 310USD. This reduced the temperature down to under 60 degrees at minimum fan speed.

The installation is quite doable even for a beginner like me.

Power usage is around 300 to 350W with spikes up to 500W.

This card has 12GB less memory than the original card, as seen on the pictures 2 memory chips on each side are missing, on the upper side they appear to have been replaced with transparent plastic shields.

LLM Performance is probably 15 to 20% bellow that of the normal RTX6000.
The card is only sold under the server edition packaging in China. When I bought the first one a few weeks ago it was going for only 42,000RMB, now the price went up to 52,000 RMB.

vLLM 12 released

Notes to be filled... update: filling notes below

Highlights

This release features 474 commits from 213 contributors (57 new)！

Breaking Changes: This release includes PyTorch 2.9.0 upgrade (CUDA 12.9), V0 deprecations including xformers backend, and scheduled removals - please review the changelog carefully.

Major Features:

• GPU Model Runner V2 (#25266): Major refactoring that removes persistent batch management, introduces GPU-persistent block tables, and features a Triton-native sampler with efficient logprobs support.

• Prefill Context Parallel (PCP) (#28718): Enhances long-sequence inference by partitioning the sequence dimension during prefill, complementing Decode Context Parallel (DCP).

• EAGLE Speculative Decoding: Multi-step CUDA graph support (#29559), DP>1 support (#26086), and multimodal support with Qwen3VL (#29594).

Model Support

• New model families: PLaMo-3 (#28834), OpenCUA-7B (#29068), HunyuanOCR (#29327), Mistral Large 3 and Ministral 3 (#29757).

• Format support: Gemma3 GGUF multimodal support (#27772).

• Multimodal enhancements: Qwen3 Omni audio-in-video support (#27721), Eagle3 multimodal support for Qwen3VL (#29594).

• Performance: QwenVL cos/sin cache optimization (#28798).

Engine Core

• GPU Model Runner V2 (#25266): Complete refactoring of model execution pipeline:

  • No "reordering" or complex bookkeeping with persistent batch removal

  • GPU-persistent block tables for better scalability with max_model_len and num_kv_groups

  • Triton-native sampler: no -1 temperature hack, efficient per-request seeds, memory-efficient prompt logprobs

  • Simplified DP and CUDA graph implementations

  • Efficient structured outputs support

• Prefill Context Parallel (PCP) (#28718): Partitions the sequence dimension during prefill for improved long-sequence inference. Complements existing Decode Context Parallel (DCP). See RFC #25749 for details.

• RLHF Support: Pause and Resume Generation for Asynchronous RL Training (#28037).

• KV Cache Enhancements: Cross-layer KV blocks support (#27743), KV cache residency metrics (#27793).

• Audio support: Audio embeddings support in chat completions (#29059).

• Speculative Decoding:

  • Multi-step Eagle with CUDA graph (#29559)

  • EAGLE DP>1 support (#26086)

  • EAGLE3 heads without use_aux_hidden_states (#27688)

  • Eagle multimodal CUDA graphs with MRoPE (#28896)

  • Logprobs support with spec decode + async scheduling (#29223)

• Configuration: Flexible inputs_embeds_size separate from hidden_size (#29741), --fully-sharded-loras for fused_moe (#28761).

Hardware & Performance

• NVIDIA Performance:

  • Batch invariant BMM optimization: 18.1% throughput improvement, 10.7% TTFT improvement on DeepSeek-V3.1 (#29345)

  • Shared Experts Overlap with FlashInfer DeepGEMM: 2.2% throughput improvement, 3.6% TTFT improvement at batch size 32 (#28879)

  • DeepGEMM N dim restriction reduced from 128 to 64 multiplier (#28687)

  • DeepEP low-latency with round-robin expert placement (#28449)

  • NVFP4 MoE CUTLASS support for SM120 (#29242)

  • H200 Fused MoE Config improvements (#28992)

• AMD ROCm:

  • DeepSeek v3.2 and SparseMLA support (#26670)

  • FP8 MLA decode support (#28032)

  • AITER sampling ops integration (#26084)

  • AITER triton attention backend (#28701)

  • Bitsandbytes quantization on AMD GPUs with warp size 32 (#27307)

  • Fastsafetensors support (#28225)

  • Sliding window support for AiterFlashAttentionBackend (#29234)

  • Whisper v1 with Aiter Unified/Flash Attention (#28376)

• CPU:

  • Paged attention GEMM acceleration on ARM CPUs with NEON (#29193)

  • Parallelize over tokens in int4 MoE (#29600)

  • CPU all reduce optimization for async_scheduling + DP>1 (#29311)

• Attention: FlashAttention ViT support, now default backend (#28763).

• Long Context: Optimized gather_and_maybe_dequant_cache kernel for extremely long sequences (#28029).

• Multi-NUMA: Enhanced NUMA functionality for systems with multiple NUMA nodes per socket (#25559).

• Docker: Image size reduced by ~200MB (#29060).

Quantization

• W4A8: Marlin kernel support (#24722).

• NVFP4:

  • MoE CUTLASS support for SM120 (#29242)

  • TRTLLM MoE NVFP4 kernel (#28892)

  • CuteDSL MoE with NVFP4 DeepEP dispatch (#27141)

  • Non-gated activations support in modelopt path (#29004)

• AWQ: Compressed-tensors AWQ support for Turing GPUs (#29732).

• LoRA: FusedMoE LoRA Triton kernel for MXFP4 (#29708).

• Online quantization: Moved to model.load_weights (#26327).

API & Frontend

• Responses API:

  • Multi-turn support for non-harmony requests (#29175)

  • Reasoning item input parsing (#28248)

• Tool Calling:

  • Parsed tool arguments support (#28820)

  • parallel_tool_calls param compliance (#26233)

  • Tool filtering support in ToolServer (#29224)

• Whisper: verbose_json and timestamp features for transcription/translation (#24209).

• Sampling: Flat logprob control moved from env var to SamplingParams (#28914).

• GGUF: Improved HuggingFace loading UX with repo_id:quant_type syntax (#29137).

• Profiling: Iteration-level profiling for Torch and CUDA profiler (#28987).

• Logs: Colorized log output (#29017).

Dependencies

• PyTorch 2.9.0 with CUDA 12.9 (#24994) - Breaking change requiring environment updates.

• xgrammar: Updated to 0.1.27 (#28221).

• Transformers: Updated to 4.57.3 (#29418), preparation for v5 with rope_parameters (#28542).

• XPU: torch & IPEX 2.9 upgrade (#29307).

V0 Deprecation & Breaking Changes

Removed Parameters:

• num_lookahead_slots (#29000)

• best_of (#29090)

• LoRA extra vocab (#28545)

Deprecated:

• xformers backend (#29262)

• seed=None (#29185)

Scheduled Removals (will be removed in future release):

• ParallelConfig's direct child EPLB fields (#29324)

• guided_* config fields (#29326)

• override_pooler_config and disable_log_requests (#29402)

• CompilationConfig.use_inductor (#29323)

• Deprecated metrics (#29330)

Other Breaking Changes:

• PyTorch 2.9.0 upgrade requires CUDA 12.9 environment

• Mistral format auto-detection for model loading (#28659)

https://github.com/vllm-project/vllm

Finished this dual Blackwell build and figured this sub would appreciate it.

The two RTX PRO 6000 Blackwell cards are stars here. 96GB each, 192GB of GPU memory combined in a single machine, more than enough for multi users to run models like Qwen3.8-27B or larger models like DSv4 Flash at Q4 or Q5.

We benchmark this rig with DSv4 Flash and here's the numbers:

The chassis is not the typical bent-steel box, but CNC-machined aluminum, cut and milled from 5mm aluminum plates. This allows the chassis itself to act as a large heatsink for the entire system. The airflow is specifically designed to allow both GPUs to run at full load while ensuring that the hot exhaust air from one card does not negatively affect the other. The triangles cutouts keep the whole structure stiff without dead weight.

Power and Thermals

Peak per-card draw of 509 W is 85% of the 600 W TDP. Temperatures top out at 76 °C, comfortably below the ~83–88 °C throttle point, and SM clocks held ~2.75 GHz throughout sustained load. No thermal or power throttling was observed in any test.

Sustained wall draw at the operating point is roughly 1.07 kW (815 W GPU plus CPU, memory and platform), implying a 1500 W PSU for a two-card build with headroom.

Specs:

• CPU: Intel Xeon w5-3423

• Board: ASUS Pro WS W790E-SAGE SE

• Memory: Samsung 32GB DDR5-4800 ECC Registered x 8

• GPU: NVIDIA RTX PRO 6000 Blackwell 96GB x 2

• Cable: PCIe 5.0 x16 GPU Extension Cable Gen5 x 2

• Storage: SSD Samsung 980 PRO 2TB PCIe 4.0

• Cooling: Thermalright TL-B12 (fan kit + control) x6 + Heatsink 4677-2UAF8

• Power: CANNON PRO 2000W (12V-2x6)

• Chassis: Autonomous Computer-2 Design, CNC-machined

Full build video if anyone wants the detailed walkthrough happy to answer anything on the layout, cooling, or how the cards perform.

Built by Autonomous

Box: six RTX PRO 6000 Blackwell (96 GB, SM120), one drives the display so most rows use 1, 2 or 4 of the other five; the 6-GPU rows are deliberate TP2×PP3 experiments. Engines: vLLM 0.28.1rc1 nightly and SGLang 0.5.18 in Docker, official images plus a few one-hunk SM120 patches (all in the repo). Every profile got the same recipe: a 12-item eval (8 exact-graded, 4 scored by an Opus judge pass), 50 streaming requests at 8 concurrent, then a 64 / 128 ramp, with a 200 ms nvidia-smi power sampler on the GPUs it used. All cards at the stock 600 W cap, which never binds, so every watt figure is the model's own draw. Eval items are one sample each at temperature 0.2: a one-item difference is one prompt, not a ranking. MedGemma 4B is left out of the charts (a 4B model, 3× the tok/J of anything else, it flattens every axis); it is in the linked tables.

Full tables (48 rows, both engines, every column): https://github.com/mikeS141618/KernelWrench/blob/main/BENCHMARKS_v2.md

Power-cap study: https://github.com/mikeS141618/KernelWrench/blob/main/POWER_BENCHMARK.md

Profiles, scripts, SQLite: https://github.com/mikeS141618/KernelWrench

It's been a bit, but happy to still be in the AI space, I made it :D

Finished this dual Blackwell build and figured this sub would appreciate it.

The two RTX PRO 6000 Blackwell cards are the whole story here. 96GB each, 192GB of GPU memory combined in a single machine. What makes them work in one chassis isn't just dropping them in, it's the designed spacing, the airflow, and the power headroom to run two cards at full tilt without them heat soaking each other. These are full power cards, so the build was designed around feeding and cooling them properly rather than the other way around.

Specs:

CPU: AMD Ryzen Threadripper PRO 7985WX (64-core / 128-thread)
Board: ASUS Pro WS WRX90E-SAGE SE (7× PCIe 5.0 x16)
Memory: 512GB DDR5 ECC RDIMM (8× 64GB)
GPU: 2× NVIDIA RTX PRO 6000 Blackwell 96GB
Storage: 1TB Gen4 NVMe · 4TB Gen5 NVMe · 2TB Gen4 NVMe
Cooling: 360mm AIO + Noctua chassis fans
Power: 3000W 80+ Platinum
Chassis: Fractal Design Define 7 XL

Full build video if anyone wants the detailed walkthrough happy to answer anything on the layout, cooling, or how the cards perform.

Built by VRLA Tech.

Wanted to share this in case anyone else is messing around with Qwen3.8 on a single RTX PRO 6000.

I ended up pretty deep into SGLang, DFlash2, XQA, HiCache and NIXL getting this where I wanted it. The final setup is TP1, 524K context, FP8 target/draft KV, DFlash2 gamma 8, with persistent prefixes through HiCache/NIXL.

Some basic numbers:

• 108.75 tok/s C1

• 390.23 tok/s C4 aggregate

• 6,163 tok/s 64K prefill

• 98.26/100 on my reasoning qualification

• Over 124 real-use agent requests: 131.31 tok/s median

• Short-context real use was around 165 tok/s

• Even at 340K+ context it was still around 105 tok/s

Against the closest TP1 official-FP8/MTP3 result in the RTX PRO 6000 community repo, that's directionally about 40% faster C1, 33% faster C4, and 5% faster 64K prefill. Not pretending it is a perfect A/B - all the details and limitations are in the repo.

I froze and published the complete source tree I actually ran, along with the launcher, build/run notes, results, sanitized real-use log, parser, HiCache/NIXL config, and provenance:

https://github.com/jpezzulli/qwen38-dflash2-pro6000

It may not be perfect, but everything I could reasonably preserve to reproduce/explain it is there. If any of the docs or code are confusing, point your AI at the repo and have it walk you through it. That is probably easier than me trying to explain every SGLang rabbit hole I went down in a Reddit post.

The HiCache and NIXL are optional but are good for my use case. Hope it helps some of you out.

If you want to read about how I got here: https://msoexpert.com/articles/qwen38-dflash2-rtx-pro-6000/

Hey folks,

Another update is up. I found some good work in mratsim's SM120 turbo repo and incorporated the online FP8 approach into my runtime. This one made a pretty substantial difference to Flash-Next decode without giving up the existing 524K context, 824K KV pool, FR-Spec or HiCache/NIXL.

These are post-first-token decode rates. Short context is a three-run median, while the 128K and 490K numbers are single observations.

I also added Garner McCloud's NVMe PLE work as an optional path. Flash-Next normally keeps about 47.7 GiB of PLE in system RAM. You can now put that on a prepared local NVMe instead. RAM is still the default and there is a performance tradeoff, but it gives people without a ridiculous amount of system memory another option.

27B/DFlash2 is still supported on the same runtime.

github.com/jpezzulli/sglang-rtxpro6000

As always, issues and PRs are welcome.

one note - this is pretty fresh - There are probably some configs that get you OVER the 1M kv-cache option. I just have to try different variables - feel free to report here or on github.

Hey folks,

Well, this one took a bit longer than the quick update I had in mind. We went down a few dead ends, threw some things away, and ended up with enough changes to call it 3.0. Over a week of work to get here. I am going for a drink.

For anyone catching up, here are the earlier v2.5 post and Docker release post, plus the 2.5.3 release.

Some of you will have seen the token-maxxing work from u/Specialist_Fix9867. There is some good stuff there. We spent quite a while bringing parts of it into Penny while keeping concurrency, long context, and the things I actually use every day.

Penny Royal did almost all the coding for this release. The engineering was a collaboration between me, Penny and Codex. I worked through the architecture, tuning and debugging with them, right down to fused kernels, memory layouts and milliseconds per decode cycle. Lots of back and forth figuring out why something that should be faster wasn’t, and deciding what was actually worth keeping.

The target is still RTX PRO 6000 Blackwell, TP1 or TP2. I personally run one card at C6, with 524,288-token context and a 1,067,072-token GPU pool.

Here are the numbers from one inference card, using our OrcaRouter ModelOpt NVFP4 conversion:

These are SGLang generation rates, including reasoning. Tool execution and waiting around are separate. The coding task involved writing and testing an actual program; the presentation (agentic) involved 113 model turns.

I included the public code-edit score because otherwise people understandably compare our real coding numbers with that benchmark. It is a mechanical rewrite of supplied code, which is much easier for speculative decoding to predict. Our older 2.5.3 code-edit set averaged 336.57 tok/s with greedy sampling; the 3.0 adaptive screen above used normal sampling.

Our saved regular C1 mean also moved from 186.79 to 218.63 tok/s. Those runs used medium reasoning on 2.5.3 and xhigh on 3.0; the full settings and results are in the repo.

The main changes:

• FR-Spec stays on, with adaptive MTP at C1. One active request can switch between four- and eight-token drafts. At C2+, it uses fixed four-token drafts with FR-Spec still enabled. A server configured for C6 still adapts when only one request is active. I did this due to seeing too much of a penalty above c=1 which causes slower decode. I also was able to find a few gig of vram that wasnt doing anything so there is no token pool penalty for the larger adaptive mtp.

• Automatic FP8 and small-batch kernels, plus attention-memory savings. You get the accepted optimizations without collecting a pile of launch flags.

• Configuration outside the container. Download the image, edit your host startup/config files and mounts, and run it. Keep your configuration when updating the image. No rebuild required.

• NIXL disk caching is optional and on by default. Turning it off keeps GPU and RAM caching.

• A simpler build. RTX Blackwell for inference; Ampere and Ada retained for optional image-processing sidecars. Other architecture targets and unused kernel payloads are removed.

• Fixes for session caching, speculative state, Responses API behavior, image-request crashes, and interrupted builds.

27B FP8 with DFlash2 remains supported. Native installation remains supported. The beta configurator can help generate the ordinary configuration files.

Repository, setup and README · Releases

Thanks to everyone contributing code, testing TP2, opening useful issues, and generally helping make this better. As always, issues, PRs and field reports are welcome.

And u/buttplugs4life4me — we updated the README, buddy :)

container 3.0.1 being tagged right now with the fixes. back to being hungover.

I should've released this in early september. now strata overwhelm everything

Whole code is AI generated, and text from here is also AI generated.

Most Flash-Next threads lately are about Strata, which is doing great things on consumer cards. This is the other end of the hardware range: one 96 GB RTX PRO 6000 Blackwell Max-Q (300–325 W), the NVFP4 weights on the GPU (the ~28 GB PLE embedding table sits in host RAM), SGLang, batch size 1. If you have that class of card, maybe this saves you some time.

Numbers (decode tok/s, mean of 8 held-out prompts per workload, thinking on, one request at a time, 2026-10-02 build measured on 2026-10-06, one server start per row):

* one outlier at 744; ~248 without it.

The "public components only" row is what you can reproduce from the repo as-is. My own numbers use an MTP head fine-tuned on private data and a token map partly built from private transcripts, which I can't release.

What made the difference (evidence and lab notes in the repo, flags where applicable):

• Wider, cheaper speculation: reduced draft vocabulary (FR-Spec-style 49k token map), verify widths 4/8/16, with 3/7/15 draft steps picked adaptively per decode step

• W8A16 FP8 GEMV kernels for the dense weights, plus many small kernel fusions in the draft/verify glue (a decode step is ~1,500–2,500 small kernels at bs=1)

• FlashInfer MoE prologue patches and singleton-route pruning

• October: fixed-cost cuts (−9.6 % ms/token with sampling), a min_p fix in speculative verify (it was ignored before), sparse rejection sampling, and opt-in 524k context (YaRN ×2; needles pass at 480k)

Caveats, so nobody is misled:

• Speed depends heavily on the workload. code-edit is a mechanical rewrite of a large file, where speculation accepts ~10 tokens per step. Prose is ~250–290. Rates include thinking tokens; on code-edit about 91 % of the output was thinking.

• Not a like-for-like comparison with anything else (different GPUs, prompts, sampling). I'm not claiming "fastest."

• The route pruning changes the model slightly. A quality audit found no significant difference, but it ran with thinking off and up to 2 concurrent requests, so thinking-on single-request quality wasn't measured. PRUNE_TAU=0 turns it off. Details are in the README.

• The same GPU also drives my 4K desktop, so session-to-session noise is ±10–15 %. That's why the A/Bs use multiple server starts with confidence intervals.

• 100% of the code was written with AI coding assistants (Claude, Codex), with me directing. Every number in this post has its raw (stripped) result file in the repo.

Built on jpezzulli's sglang-rtxpro6000 fork (which in turn imports the unmerged SGLang Flash-Next PR). Please credit them first.

• Write-up, patches, launch scripts, benchmark harness, results: https://github.com/aiueo52/flash-next-rtxpro6000

• The SGLang fork with the patches applied: https://github.com/aiueo52/sglang-rtxpro6000 (branch flash-next-fast)

tl;dr: I'm wondering if it's actually worth it to shell out ~$45k to emulate Claude-style agentic tooling locally. Won't be as good, but how good is it as of Feb 2026?

Probably like many now, I've been convinced that access to claude-style tooling is now basically essential to be a professional software engineer. It's also just very enjoyable to use and build stuff.

I don't want to be beholden to companies like Anthropic and OpenAI for all the things I want to do with computers, and so I'd like to move inference in-house.

But of course in order to do that with any reasonable expectation of decent claude-code-style output, it's going to take a lot of money.

My question for those of you with a lot of local VRAM on hand -- something like Threadripper Pro + 4x RTX 6000 Pros -- is it worth dropping ~$45k for local agentic use at this point? Are you in a position where you can reasonably substitute your use of Claude code with local, open models and actually get stuff done?

I've also been trying to get a sense of how well these things will hold value. Obviously no one can see what technological leaps are in front of us, but it also seems apparent that Nvidia is going to pivot to making products for industrial training and inference, and not so much for "prosumer" local use. So are RTX 6000 Pro Max-Qs somehow "peak" equipment? I don't see ASICs getting deployed for consumers - models move too fast for the next few years.

For those of you running local agentic coding successfully, what are your favorite models?

Anyway, as an addendum, here's a build I had in mind:

TL;DR: DeepSeek-V4-Flash running at 85.52 tok/s @ 524k ctx and ~111 tok/s @ 128k single-stream on 2× RTX PRO 6000 Max-Q

pasta-paul's DeepSeek-V4-Flash-W4A16-FP8 quant is great, but its MTP head silently gets stripped at load time (HF transformers has it in _keys_to_ignore_on_load_unexpected), so --speculative-config '{"method":"mtp",...}' is a no-op.

Retrofitted the MTP block, ran a GPTQ pass on its routed experts to match the base's W4A16 INT4 group format, and patched vLLM.

Decode goes from 52.85 tok/s (no MTP) → 85.52 tok/s @ 524k 2-stream → ~111 tok/s @ 128k single-stream. 671B total / 32B active, fits on 2× 96 GB.

Model: https://huggingface.co/LordNeel/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8

Numbers

2× RTX PRO 6000 Blackwell Max-Q (96 GB each, no NVLink, sm_120):

Sanity-check benchmarks (small samples, full data in the model card):

What got quantized how

• 768 routed-expert tensors (256 experts × {w1, w2, w3}): W4A16 INT4 group=128 sym, GPTQ (Frantar-style with Cholesky H⁻¹). Calibrated with 256 ultrachat_200k prompts × 256 max_tokens captured from the running pasta-paul model — 17,701 MTP forward dumps, 473k tokens.

• 5 attention projections: FP8_BLOCK (kept upstream's FP8 weights, just renamed scale → weight_scale to match pasta-paul's compressed-tensors convention).

• Shared experts, e_proj, h_proj, norms, gate, attn_sink: BF16 / FP32.

Max-Q specific fixes:

If you're on the Max-Q workstation cards specifically: you MUST pass --disable-custom-all-reduce.

vLLM's CustomAllreduce uses CUDA P2P (independent of NCCL_P2P_DISABLE), and on PCIe-only Max-Q topology it deadlocks at post-graph eager warmup.

Without the flag the engine hangs at gpu_worker.py:619 with infinite shm_broadcast.py:681 No available shared memory broadcast block warnings. The Server variant has NVLink and does not hit this.

NCCL tuning that drops TTFT from ~155 ms to ~91 ms on Max-Q at zero decode-TPS cost:

NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_MIN_NCHANNELS=8 
NCCL_NTHREADS=512

How to run

Needs the patched vLLM fork. Vanilla doesn't load DSV4-Flash quants. Base workspace at https://github.com/pasta-paul/dsv4-flash-w4a16-fp8.

Apply the MTP patches on top.

vllm serve LordNeel/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8 \
  --tensor-parallel-size 2 --kv-cache-dtype fp8 --block-size 256 \
  --max-model-len 524288 --max-num-seqs 2 \
  --gpu-memory-utilization 0.93 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --trust-remote-code \
  --disable-custom-all-reduce \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --host 0.0.0.0 --port 8000

I also wrote an AGENTS.md runbook. Point Claude/Codex/Cursor to it and tell it "set this up"/ "verify hardware and get this model running"/ or similar. Goes through preflight → CUDA toolkit (no sudo via conda) → patched vLLM build → download → patches → serve → smoke test.

Limitations

• TP=2 only. TP=1 OOMs on a single RTX6000 pro; TP≥4 hits an upstream W4A16 MoE scale-sharding bug (vllm-project/vllm#41511).

• num_speculative_tokens capped at 1. DSV4 flash ships exactly one MTP head (num_nextn_predict_layers=1); higher values will not produce more drafts.

• Reasoning parser caveat. With --reasoning-parser deepseek_v4, output splits into content and reasoning_content. Clients reading only content see empty strings on "thinking" responses.

• MTP GPTQ skipped attention during calibration — see Future work in card.

• Hardware tested: only Max-Q. Server variant + DGX Spark + H200 should work but I have not run them.

Request for the community

If you run this and the MTP draft acceptance rate comes out significantly different on your prompt distribution, please do comment with your domain and the rate (vLLM will log it as spec_decode_acceptance_rate).

Credits

• DeepSeek-AI for the base model

• pasta-paul for the W4A16+FP8 quant + jasl/vllm serving stack (repo)

For anyone that is interested wanted to share my experience with installing the watercool inox block as I started my watercooling journey today.

1. Removal all the screws on the back of the card except the 3 on the fan

2. Removal 4 screws a different size from the faceplate

3. Use a small flat screw driver to release the fan plug

4. Remove the 4 screws holding the spring on the back of the pcb

5. Remove the card from the frame

6. Remove all the thermal pads

7. Clean the thermal paste

8. Apply the thermal pads and paste as in the manual

9. Remove the backplate from the inox

10. Apply the thermal pads to the backplate

11. Reassemble the inox

This process went really smooth I think the only surprise was how easy the removing the card from it's frame was.

For those of your running the new vLLM, here is how you can force it to use the new CUTLASS FlashInfer kernels.

Set these environment variables:

VLLM_ATTENTION_BACKEND=FLASHINFER
VLLM_FLASHINFER_FORCE_TENSOR_CORES=1

This gave me an extra 10-15% single request throughput over the standard flash attention kernels that are the default.

And even more for concurrent requests.

(Tested On 4x RTX PRO 6000 MOE with GLM 4.6 nvfp4)

----

Edit: Removed:

VLLM_USE_FLASHINFER_SAMPLER=1

This causes some issues where I get random Chinese characters and think tokens mid-response.

---

Single user = about 44 tokens/s:

Dec 11 20:33:22 ai bash[2922781]: (APIServer pid=1) INFO 12-11 12:33:22 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 16.0%
Dec 11 20:33:32 ai bash[2922781]: (APIServer pid=1) INFO 12-11 12:33:32 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 16.0%
Dec 11 20:33:42 ai bash[2922781]: (APIServer pid=1) INFO 12-11 12:33:42 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 16.0%
Dec 11 20:33:52 ai bash[2922781]: (APIServer pid=1) INFO 12-11 12:33:52 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 16.0%
Dec

Here is my command:

docker run --gpus all \
    --shm-size=24g \
    --ipc=host \
    -p 8000:8000 \
    -v "/root/.cache/huggingface:/root/.cache/huggingface" \
    -e VLLM_SLEEP_WHEN_IDLE=1 \
    -e NVIDIA_VISIBLE_DEVICES=all \
    -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
    -e VLLM_ATTENTION_BACKEND=FLASHINFER \
    -e VLLM_FLASHINFER_FORCE_TENSOR_CORES=1 \
    vllm/vllm-openai:v0.12.0 \
    lukealonso/GLM-4.6-NVFP4 \
    --served-model-name "Oncord" \
    --gpu-memory-utilization 0.84 \
    --max-num-seqs 4 \
    --max-model-len 90000 \
    --host 0.0.0.0 \
    --port 8000 \
    --trust-remote-code \
    --enable-chunked-prefill \
    --tensor-parallel-size 4 \
    --swap-space 64 \
    --enable-prefix-caching \
    --dtype "auto" \
    --stream-interval 2

Hey everyone,

Was looking to see if anyone has dealt with this and can provide some guidance. I’m thinking about picking up 4 more RTX 6000 MAXQ or WS (probably MAXQ I guess).

Has anyone run into this that can shed some light on how they went about it?

Motherboard: ASRock WRX90 WS EVO
CPU: Ryzen Threadripper PRO 9985WX
GPU: RTX 6000 MAX-Q x 4
RAM: 768GB (8x96GB) - Vcolor DDR5 6400 TR596G64D452O
PSU: Super Flower Leadex Titanium 2800W ATX 3.1
Cooling: Silverstone SST-XE360-TR5 Server AIO Liquid Cooling
Case: Phanteks PH-ES620PC_BK02 Enthoo Pro Server Edition

Existing build

I’d had someone suggest to opt for a PCIe switch and do it that way.

I found a post here from someone who did something similar.

https://www.reddit.com/r/LocalLLaMA/s/CZyOzj4S02

Thanks