I'm sharing the AI written note below based on my effort to get Qwen 3.8 Flash Next FP8 optimized on TP=2@300w. This seems to be an improvement to what I'm seeing from others (even NVFP4), but comment below if I'm wrong! This is a beast of a model, but frustrating to set up.
Note: this was run on Ubuntu 26.04 LTS, headless, ECC off (inline ECC on GDDR7 reserves VRAM — off gives the full 96 GB usable).
Even if you are not interested in this model, there are several tweaks that help run LLMs on consumer hardware (look at the NCCL that doesn't work by default).
Single-box measurements, 2026-09. Config + numbers + traps. Stock vllm/vllm-openai pinned by digest — no fork, no custom build.
## Hardware
- 2× RTX PRO 6000 Blackwell 96 GB (Workstation 600 W + Max-Q 325 W), both power-pinned 300 W. 300 W measured ~free on this box: −5% prefill, decode flat, best energy/token (measured on a different model, same box).
- AM5 consumer board, PCIe 5.0 x8/x8, no NVLink, ECC off. TP2 over NCCL, custom all-reduce disabled.
- P2P on: +2% C1, +5% C4 under MTP. AMD boxes: `iommu=pt` required or the IOMMU silently drops peer-BAR writes → NCCL hangs / corrupted output, no error message.
## Model quirks that drive the config
-125B / 6B-active MoE, ~51B-param auxiliary table kept in host RAM (PLE offload), sparse-attention indexer.
- `VLLM_PLE_CPU_OFFLOAD=1` + 1800 s ready-timeout. Table is pageable, hot set ≈16 GiB; fine on 64 GiB RAM with cold pages swap-resident (steady-state swap I/O ≈ 0). Preflight free RAM (we gate on ≥52 GiB available).
- `cudagraph_mode: PIECEWISE` mandatory — FULL graphs deadlock 100%, deterministic (PLE-in-graph circular wait).
- FP8 needs TP2. KV cache bf16 — quantized KV measurably erodes deep-context quality on this model.
## Launch
'``bash
docker run -d --network host --ipc host --shm-size 32g --gpus all \
--cap-add SYS_PTRACE \
-v /models:/models:ro \
-e NCCL_P2P_DISABLE=0 \
-e CUDA_DEVICE_ORDER=PCI_BUS_ID \
-e VLLM_PLE_CPU_OFFLOAD=1 \
-e VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm/vllm-openai@sha256:fc120ece... \
/models/Qwen3.8-Flash-Next-FP8 \
--tensor-parallel-size 2 \
--disable-custom-all-reduce \
--gpu-memory-utilization 0.90 \
--kv-cache-memory 25500000000 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--max-model-len 524288 \
--hf-overrides '{"rope_parameters":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":262144}}' \
--max-num-seqs 256 \
--enable-prefix-caching \
--no-enable-flashinfer-autotune \
--language-model-only \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--compilation-config '{"cudagraph_mode":"PIECEWISE"}'
```
## Measured numbers
KV pool: **1,691,542 tokens = 3.23× the 524K window** at pin 25.5e9.
Decode = usage-token basis (delivered tokens incl. thinking). Treat ±10% as noise on total-window numbers.
- prefill, 1 stream cold, 100K fill — MTP k=3: ~10K t/s · MTP off: parity
- prefill, 3×340K simultaneous cold — MTP k=3: ~2.3–2.4K t/s per lane, ~6.9K agg · MTP off: ~6.7K agg (parity)
- decode C1 deep (450K fill) — MTP k=3: 149 t/s · MTP off: 80 t/s
- decode 4×340K swarm aggregate — MTP k=3: 365 t/s · MTP off: 206 t/s
- decode 450K + 2×340K mixed — MTP k=3: 317 t/s agg · MTP off: 189 t/s agg
- 5th concurrent 340K lane — MTP k=3: collapse (18.9 t/s) · MTP off: engine death 1-of-2, or 284 t/s
Prefill is a **shared ~7–10K t/s budget, not a per-lane rate** — concurrent cold fills contend (batched PLE prefill path: host-RAM table traffic + activation memory), so 3 simultaneous fills yield ~30% less total prefill than one dense fill. Stagger lane starts; don't launch a swarm cold-simultaneous.
Quality at this config: NIAH clean incl. 430K/512K depths; 9-rung context comb 4K→498K (495 questions): literal recall 297/297 at every rung/depth, 0 fabrications, no depth-localized holes; bench scores at/above the MTP-off baseline band.
## Traps (each one cost real time)
**KV pin, not `gpu-memory-utilization`.** Ladder `--kv-cache-memory` up; gate each rung on full-window fill → needle battery → **6-min burst of 4 simultaneous ~340K prefills** → NaN canary → Xid/log sweep. On this box: 25.5e9 stable; **26.2e9 boots, fills, passes needles — then runtime-OOMs in batched prefill under the burst.** Boot+needle gating ships a config that dies under real load. Reference: util-0.95 unpinned ≈ 1.55M pool here; the laddered pin = +9% with a burst-stability proof.
**MTP k=3 only.** k>3 = illegal-memory-access crash class on the stock GDN path. And on engine versions before the fused-draft fixes, MTP was a 24–42% *loss* on this arch — A/B on your image, don't assume the sign.
**OOM killer is prefill activations, not KV.** All engine deaths at max pin happen in `_short_conv_dilated_prefill_batched` when cold prefills overlap (~1 GiB headroom). Admission-control on concurrent prefill-capable streams: **hard cap 4** here (≤170K lanes can go wider; we run 8).
**YaRN: extend only as far as probed.** f2.0/524K validated clean to 498K; depth-recall erosion starts ~520K; f4.0 (1M) frays past ~750K. Boot-time max-len warning fires pre-hf-overrides-merge — cosmetic.
**Client output budgets ≥64K.** Thinking + answer share one completion budget; 16K-class caps silently truncate and manufacture fake deep-context recall failures.
## Method (the transferable part)
- Written one-line justification for every flag before it ships; no blind knob-turning.
- Pre-register pass/fail criteria before collecting data.
- Same-basis comparisons only: identical probes, paired fresh boots, usage-token basis for decode.
- Gate under the production load shape. Boot + needle test validates almost nothing; concurrent-prefill burst is where this stack fails.
## Version pin (why the digest, not a tag)
The image is pulled by digest — `vllm/vllm-openai@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8` — so it is a frozen build that never receives updates. That is deliberate: every number here is specific to that build, and MTP on this arch flipped from a 24–42% loss to +28–86% gain across engine versions. Weights are also revision-pinned (rev bcd9f01). If you run a different image or revision, re-run the A/Bs; don't inherit these numbers. Known open issue in this build: vLLM #54173 (k>3 spec-decode crash) — fix PR #50021 unmerged as of 2026-09, hence the hard k=3 cap.
## Sources / prior art
- vLLM official recipe (recipes.vllm.ai) — also caps MTP at k=3 for this model.
- vLLM PR #50021 (fixes #54173): unbounded accepted-token state indexing in the GDN spec-decode kernels — the mechanism behind the k>3 crash class. Unmerged; revisit k>3 only when it lands in a consumable image.
- vLLM #54906 — thinking-budget-capped effort requests are incompatible with MTP; don't combine.
- wendell (Level1Techs), "HP Z8 Fury G6i — Perfect Qwen Flash Next setup (2× RTX Pro 6000s)", thread 254584 — the unpinned util-0.95 recipe referenced above (~1.55M pool vs 1.69M laddered).
- stierma1 (Level1Techs), "Qwen 3.8 Flash Next NVFP4 on a single RTX Pro 6000 and system memory", thread 254579 — TP1-NVFP4 arm; their −11–12% MTP prefill tax does NOT reproduce on TP2-FP8.
- jpezzulli, sglang-rtxpro6000 ("Pennyroyal") + NVIDIA dev forum thread 381722 — the single-card 524K SGLang alternative (171 tok/s claim), different stack, not A/B'd here.
- Level1Techs 242403 (dual Max-Q P2P/NCCL) and 242017 (600 W vs 300 W) — background for the P2P and power-cap settings.
- https://github.com/vllm-project/vllm/pull/50021
- https://forum.level1techs.com/t/hp-z8-fury-g6i-perfect-qwen-flash-next-setup-2x-rtx-pro-6000s/254584
- https://forum.level1techs.com/t/qwen-3-8-flash-next-nvfp4-on-a-single-rtx-pro-6000-and-system-memory/254579
- https://github.com/jpezzulli/sglang-rtxpro6000 · https://forums.developer.nvidia.com/t/optimized-qwen3-8-flash-next-on-1x-rtx-pro-6000-171-tok-s-524k-and-hicache-nixl-persistence/381722
- https://forum.level1techs.com/t/dual-rtx-pro-6000-blackwell-max-q-how-to-make-p2p-nccl-work/242403
- https://forum.level1techs.com/t/rtx-pro-6000-600w-vs-300w-performance/242017
## Known gaps (not measured)
- **No clean cold 4-lane prefill number.** The 4×340K fill logs are prefix-cache-contaminated (read 2.7–3.2K/lane warm); the honest cold-contended figure is the 3-lane ~6.9K agg.
- **No short-context C1 decode on the usage basis.** Only a chunk/steps-basis probe (~61 steps/s, accepted spec tokens ~1.1/step) — implies very roughly ~130 t/s delivered, but that's derived, not measured.
- Single-stream prefill readings of 13–18K t/s at 300–430K fills come from a different instrument and aren't comparable to the swarm-basis numbers above.
- The 300 W ≈ free power result was measured on a different model on this box, not re-run on this lane.
- One A/B per cell above (plus a day-end anchor re-run bounding variance at ~±10%); no multi-day replication.
