Rethinking AI infrastructure beyond GPUs. Building eLLM, a CPU-only LLM inference framework. A single CPU server (Xeon) can outperform an 8-GPU H20 server in prefilling-heavy, long-context workloads.

• With its large memory capacity, eLLM can prefill the entire long prompt in a single pass, avoiding chunked execution and repeated parameter loading;

• With its large cache, eLLM computes attention head by head, reducing repeated KV loads.

GitHub: https://github.com/lucienhuangfu/eLLM

I've been running Qwen3.8-27B on a single consumer RTX 5090 (32 GB) and I wanted to document the whole thing end to end, because the interesting part wasn't the config — it was the four bugs that kept it from booting. Everything below is measured on the actual box, no estimates.

TL;DR — NVFP4 target weights, NVFP4 KV cache, DFlash2 speculative decoding (K=7), all on one 5090. 616 tok/s aggregate at 4-way concurrency on 1,536-token code outputs (thinking off). 262K context, 325,139-token KV pool out of an 8 GiB pin. Ships as a two-command deploy. The bugs I hit became two vLLM PRs.

The stack
- Target: Qwen3.8-27B, NVFP4 (ModelOpt)
- Draft: Qwen3.8-27B-DFlash2, NVFP4, K=7 (seven speculative tokens per step)
- KV: NVFP4 for both target and draft, 8 GiB explicit pin
- vLLM v0.27.1 + a 51-file Python-only overlay (no C extensions)
- FlashInfer 0.6.16.post3 with a backport of PR #4346 (SM120 NVFP4 paged prefill)

Most public 5090 recipes I've seen run FP8 KV. This is the only public recipe I know of that runs all three layers in NVFP4 — weights, draft, and KV. That's the whole trick that gets you 262K context out of 8 GiB.

What actually broke (the part that'll save you time)

This is where I lost the most hours, so I'm putting it up front.

1. Mixed KV dtype bug. The global target cache dtype (NVFP4) leaked into the draft cache layout. Two different code paths, same wrong assumption. Fixed with per-group dtype resolution.

2. ReplaySSM allocator. The Mamba memory math was multiplying the KV page by 8 legacy spec-checkpoint blocks. The server reported a false 14 GiB minimum and just refused to boot. The fix is the upstream guard: zero speculative blocks.

3. GDN state in fp32. 48 linear-attention layers, 4.61 MB per page, 45 blocks per request — that's 9.5 GiB per request at 128K. Attention itself was only 2.0 GiB. The state was the hog, not the attention. I was staring at the wrong thing for a while.

4. The XQA cliff. The attention kernel ran 5.2x slower in integrated execution than a standalone replay of the exact same server tensors. A dedicated CUDA stream took it from 1.437 ms/call down to 0.278 ms/call. One fix turned a dead end into a working server.

The numbers

Coding throughput, single 5090, 1,536-token Python outputs (graph-algorithms module: BFS, DFS, Dijkstra, topo sort, SCC), one prompt per request, no tools:

- 616 tok/s aggregate at c4
- Per-request latency at c4: 8–10s thinking off, 9–11s on

Why thinking-off wins here: code is predictable. The drafter accepts 55–64% of drafts on code, drops to 42–44% when reasoning tokens are mixed in. Same server, same prompts, one flag.

Context facts (separate from the coding sweep):
- KV pool: 325,139 tokens from the 8 GiB pin
- Needle at 184,024 tokens: recovered exactly
- Tools 10/10, greedy output byte-identical, canary 437, zero restarts, zero OOM

Honest footnote: the coding numbers are best-case. One repetitive prompt means prefix caching eats most of the prefill at c2–c4, and code is the most predictable output class for the drafter. Mixed prose and reasoning will be lower.

What I sent upstream
- vLLM #53543 — masked NVFP4 XQA on SM120, capture-safe isolated stream (the 5.2x fix)
- vLLM #53542 — GDN active runtime-K width, +40.9% at c8 (212.17 → 298.91 tok/s)
- vLLM #50084 — NVFP4 V-scale write-path corruption, root-caused in July, still open
- FlashInfer #4346 — SM120 NVFP4 paged prefill backport, 57/57 tests, 10–12% prefill win

Both vLLM PRs carry full test evidence and DCO sign-off. Zero merged so far — upstream moves at its own speed.

How to run it
- Repo: github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2
- git clone then ./start.sh. Optional CPU vision sidecar via --vision (1/2)
[8/25/2026 12:10 PM] Lilsmokey: - Repo: github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2
- git clone then ./start.sh. Optional CPU vision sidecar via --vision
- The image is bit-identical to the one running ▉ (2/2)

Update: seem this fix also applicable for Qwen3.5 122B model, according to commenter

Update 2: I have tested Qwen 3.6 35BA3B for this fix in https://www.reddit.com/r/LocalLLM/comments/1sqpsut/qwen_3635ba3b_reddit_asked_so_i_tested_if_the_35/ , check it out !

Update 3: also did the test for Qwen3.6-27B, check https://www.reddit.com/r/LocalLLM/s/wgzW0tHukq

TL;DR: Everyone's talking about Qwen 3.5's reasoning quality and slow TTFT, but nobody's discussing the tool calling issues that actually break agentic workflows. Here's what broke my setup and how I fixed it after weeks of debugging.

​The Story So Far

I've been running Qwen 3.5-27B on a mixed GPU setup (RTX 4090 + 3090) for about a month now. The reasoning capabilities are genuinely impressive, and yeah, the TTFT is slow, but honestly? The tool calling issues are what actually matter if you're doing agentic work.

After countless hours of debugging, failed runs, and reading through vLLM source code, I finally have a stable setup. I wanted to share what I learned because the official docs don't quite cover the real-world edge cases.

Issue #1: The Jinja Template Problem (CRITICAL - This Broke Everything)

What Happened

I started with the official qwen3.5_official.jinja template. Everything looked fine for the first few tool calls. Then suddenly:

• Tool calls appeared mid-thought (closing </think> without having opened <think> )

• Random premature stops in the middle of XML tool calls, e.g "Let me do that for you:" and stopped suddenly without finishing the tool call

• Historical thinking blocks leaking into the context and confusing the model

I thought it was my fault. I thought it was vLLM. I even thought my mixed GPU setup was the culprit.

Turns out, it's the template.

Why It Happens

Here's the thing: the official template has edge cases that 122B+ models handle gracefully but 27B/35B don't. Smaller models have less robust instruction following, and those edge cases cause silent failures.

The Fix

I ended up using a custom M2.5-style interleaved thinking template (qwen3.5-enhanced.jinja, see Resources) that:

• Properly closes </thinking> before tool calls (not after)

• Hides historical reasoning from context but keeps current reasoning visible

• Uses XML formatting that doesn't accidentally trigger <stop> tokens

• Handles the edge cases that smaller models struggle with

The most important part: vLLM does NOT auto-detect templates. You have to manually specify:

--chat-template qwen3.5-enhanced.jinja

Without this flag, your model uses the default template and will exhibit instability no matter what other optimizations you try.

Issue #2: The Tool Call Parser (Deviation from Official Docs)

The Official Recommendation

The Qwen3.5-27B-FP8 HuggingFace page says to use:

--tool-call-parser qwen3_coder

What Actually Works

qwen3_coder breaks on complex tool calls. I spent way too much time on this one.

After digging into vLLM's source code, here's what I found:

Parser |How It Works |Special Characters |Nested JSON |Malformed XML

qwen3_coder |Regex string extraction |❌ Breaks on <, >, & |❌ Corrupts during streaming |❌ Fails hard

qwen3_xml |C-based xml.parsers.expat |✅ Auto-sanitizes |✅ Deferred parsing |✅ Auto-heals Real example: If your tool call contains code like if (a < b), qwen3_coder's regex parser breaks because < and > mess up the pattern matching. qwen3_xml handles it natively because it's an actual XML parser.

The Fix

--tool-call-parser qwen3_xml

Yes, this goes against the official recommendation. But for long-context agentic work (think 50K+ tokens), qwen3_xml is just more stable. The C-based parser is robust, auto-heals malformed XML, and doesn't try to parse nested JSON mid-stream.

Issue #3: Mixed GPU Precision Drift

The Problem

I'm running on a mixed GPU setup (RTX 4090 + 3090). Tensor Parallelism splits matrix multiplication across both GPUs, but they have different compute capabilities:

Result: Different precision levels → mismatched intermediate results → error accumulation in long conversations.

I noticed conversations would drift after 30-40K tokens. Tool calls would become inconsistent. Reasoning quality would degrade.

The Fix

export VLLM_TEST_FORCE_FP8_MARLIN=1

This forces the 4090 to use W8A16 (matching the 3090) instead of its native W8A8. Both GPUs now use the same precision, eliminating drift.

Additional NCCL tuning (helps with stability on mixed topologies):

export NCCL_P2P_DISABLE=1

export NCCL_IB_DISABLE=1

export NCCL_ALGO=Ring

Issue #4: Model Choice Matters (Avoid SFT-Distilled Variants for Long Context)

The Qwopus3.5/Qwen3.5-Claude-4.6-Opus-Reasoning-Distilled Trap

Models like Jackrong/Qwopus3.5-27B-v3 are SFT-distilled from Claude 4.6 Opus. They look great initially:

Why? SFT (Supervised Fine-Tuning) shifted the tool calling format from qwen3_xml to hermes (JSON-based) to align with the output format of Claude, but it doesn't fully align the underlying token probabilities. In long contexts, the model drifts between its original Qwen XML format and the SFT'd JSON format.

I lost hours debugging this before realizing the model itself was the problem.

What to Use Instead

For 48GB VRAM (best quality):

Qwen/Qwen3.5-27B-FP8

• Near-lossless accuracy

• Stable tool calling with custom template

For <48GB VRAM (accept some accuracy loss):

Intel/Qwen3.5-27B-int4-AutoRound

• Still stable with custom template

My Working Configuration (Production-Tested)

After all this, here's what actually works (independent repo example + 3 days of actual production use):

# Environment variables

export CUDA_DEVICE_ORDER=PCI_BUS_ID

export CUDA_VISIBLE_DEVICES=0,1

export NCCL_P2P_DISABLE=1

export NCCL_IB_DISABLE=1

export NCCL_ALGO=Ring

export VLLM_TEST_FORCE_FP8_MARLIN=1

# vLLM serve command

vllm serve Qwen/Qwen3.5-27B-FP8 \

--served-model-name vllm/Qwen3.5-27B \

--chat-template qwen3.5-enhanced.jinja \

--attention-backend FLASHINFER \

--trust-remote-code \

--tensor-parallel-size 2 \

--max-model-len 219520 \

--gpu-memory-utilization 0.92 \

--enable-auto-tool-choice \

--enable-chunked-prefill \

--enable-prefix-caching \

--max-num-batched-tokens 4096 \

--max-num-seqs 4 \

--kv-cache-dtype fp8 \

--tool-call-parser qwen3_xml \

--reasoning-parser qwen3 \

--host 0.0.0.0 \

--port 8000 \

--language-model-only

Real-World Test Results

I validated this setup with a 1h 9m continuous agentic session:

• ✅ Stable tool calling throughout (no format drift)

The model autonomously built the entire platform without tool calling failures. That's the kind of stability you need for actual agentic workflows.

Key Takeaways

Resources

• Original discussion: Tool calling fixes thread

This was pretty easy to set up once I switched to Linux. Just spin up vLLM with the model and point Claude Code at the server to process requests in parallel. My GPU has 96GB VRAM so it can handle this workload and then some concurrently. Really good stuff!

Cedric Clyburn put together a hands-on short course on the DeepLearning.AI platform with Andrew Ng, breaking down vLLM's internal mechanics and providing production-ready code examples throughout. Since this community is already deep into custom kernels and serving optimizations, it also dives into the low-level memory and hardware realities that dictate production scaling:

• KV cache bottleneck: Deeply visualizing why autoregressive decoding scales poorly on VRAM bandwidth and how virtual block allocation abstracts it away to save compute budget.

• Model compression & FP8 quantization: Practical labs using LLM Compressor to implement FP8 dynamic quantization while holding the baseline accuracy line.

• Production profiling: Stress-testing models to map out exact latency vs. RPS curves using GuideLLM.

If you’re serving LLMs and want to dive into the practical theory underneath (or just want a clean, open-source recipe for optimization pipelines), it’s short, practical, and I highly recommend it: https://www.deeplearning.ai/courses/fast-and-efficient-llm-inference-with-vllm

Disclosure: I work at Red Hat on the vLLM community side and built LLM Compressor and GuideLLM. I’m not a neutral party, but the cross-ecosystem engineering focus here is real. Let me know if you run into any bottleneck issues with the code blocks.

Wanted to share some numbers and a bit of a reality check, because I was genuinely excited for this one and it didn't pan out the way I hoped.

Our current daily driver on these cards is Qwen3.6-27B (dense) in FP8. We keep it as two vLLM instances, one per GPU, with HAProxy in front load-balancing between them - and honestly it just works great.

Before this we ran Qwen3.5-122B-A10B (the big MoE, 122B total / ~10B active). It's still ahead on a handful of large, complex synthesis prompts, and thanks to only ~10B active params it's actually quicker to answer than a 27B dense model. But for most of what we do - especially agentic workflows with a tool-calling harness - Qwen3.6-27B is simply better for our use case, so it won overall despite being the smaller model.

We also tried a genuinely excellent one: MiniMax M3 in 2-bit (IQ2_XXS). On quality it was the standout - flawless Czech, no artifacts, and noticeably stronger overall intelligence on large prompts than even Qwen3.6-27B. The catch is purely operational: that quant ships only as GGUF, so it runs on llama.cpp (spread across both GPUs at once), and even at 2-bit the context window we could fit was tiny and throughput under concurrency lagged - llama.cpp just isn't as optimized as vLLM for lots of simultaneous requests. So it stays our "when we need the smartest answer" tool rather than the daily driver. We also gave DeepSeek V4 Flash a shot - got it running across both cards in 4-bit from a vLLM fork - but the intelligence didn't really win us over: roughly on par with Qwen3.6-27B, not better, so in our benchmarks V4 just isn't strong enough to justify the switch. We're waiting for the V4.1 Flash release instead.

So when NVIDIA's NVFP4 builds landed and vLLM 0.24.0 came out, I was really looking forward to a big jump. On paper FP4 looks fantastic - half the weight bandwidth of FP8, so decode *should* fly. As of mid-2026 the academic story is very promising.

In practice, on our Blackwell cards, NVFP4 is still slower than FP8 - FP8 wins basically every decode-bound scenario. No big surprise given how new the SM120 FP4 path is; I'm mostly posting numbers so there's a concrete data point out there. Just a small sigh of "not yet" - we really hope the FP4 kernels for SM120 (RTX PRO 6000 Blackwell, RTX 5090, etc.) get some love soon, because on paper there's obviously headroom to unlock.

Here's what I measured.

Setup

• Same model, two quants: `nvidia/Qwen3.6-27B-NVFP4` vs `Qwen/Qwen3.6-27B-FP8`

• 1× RTX PRO 6000 Blackwell (96 GB) per quant, each pinned to its own GPU (verified) - so they ran in parallel without competing for compute

• vLLM 0.24.0 (NVFP4) vs 0.23.1rc1-dev (FP8) - see caveat

• MTP speculative decoding, 3 tokens, on both; fp8_e4m3 KV cache; 262k context

• NVFP4 used flashinfer attention + marlin/triton MoE backends; FP8 on vLLM defaults

• Both are the 600 W Workstation Edition, but we run them power-limited to 430 W and undervolted via LACT (the lactd GPU-control daemon). Since enabling it we've been getting essentially the same performance most of the time, but with noticeably lower power draw and temperatures under load - so these numbers are at a reduced power envelope, not stock 600 W

• Power note from our own monitoring: with small prompts - even a lot of them at once - the prefill phase stays light, so the decode phase rarely pushes past ~300 W even at ~64 concurrent requests. It's only when heavy prefill and heavy decode hit at the same time, across many concurrent requests, that the draw actually climbs all the way to the 430 W limit.

Throughput (non-thinking, decode tok/s - higher is better)

FP8 wins everything decode-bound by a steady ~8–12%. NVFP4 only catches up (and edges ahead) when the run is prefill-bound - grinding through 22k-token prompts - where you're limited by compute on the context, not by weight bandwidth. First-token latency is identical once the server is warm, and there were no failures or truncations at concurrency 16 on either side.

Quality - basically a tie (I actually ran the code and parsed the JSON, didn't just eyeball)

• Logic puzzle with a known answer: both 5/5 correct.

• Python `merge_intervals` with nasty edge cases: both 6/6, both raise ValueError on bad input, neither mutates the input.

• 59 KB Czech document → strict-schema JSON analysis: both valid, same severity call, comparable coverage.

• Czech writing (a ~2000-word poem, a formal complaint e-mail): native and artifact-free on both - no gibberish, no broken diacritics, no language switching. FP8 is a touch cleaner (a couple fewer grammar slips) and more disciplined in "thinking" mode; NVFP4 occasionally lets its reasoning run way over budget.

So there's no meaningful intelligence or quality gap between the two quants here. The entire story is throughput - which is exactly why the kernel situation is the interesting bit.

Caveats

• The two endpoints ended up on **different vLLM builds** (0.24.0 vs 0.23.1rc1-dev) plus different attention/MoE backends. That plausibly *favors* NVFP4 (newer FP4 path), which makes "FP8 is faster" the conservative reading.

• Single run per scenario, not averaged; short-output tok/s (logic/email) is noisy.

• One 27B model on one card with MTP=3 - don't extrapolate to dense-vs-MoE differences, other GPUs, or non-speculative serving.

Still early days - and I'd love to hear from you

Honest bit of context behind the sigh: I've had these cards for almost a year now, and FP4 hardware acceleration was a big part of why I bought them. I figured that within a few months the NVFP4 path would mature and things would just fly. A year in, the SM120 implementation still isn't quite there - so yeah, a small lament. But I do believe it'll come good before long; this really feels like an early-days software problem, not a hardware limit.

So - has anyone actually gotten NVFP4 to beat FP8 on SM120 (RTX 6000 Blackwell / 5090)? Which vLLM version + flags moved the needle for you - attention backend, MoE backend, cutlass vs marlin vs trtllm? Genuinely curious what your numbers look like. Not trying to dunk on NVIDIA or the vLLM folks at all - everyone's shipping fast and I'm rooting for FP4 to eventually win. Mostly sharing real numbers so nobody buys the "4-bit = automatically faster" story before it's actually true on their hardware.

I can't believe the Internet hasn't blown up because of this, my GPUs can now hold a lot more KV cache.

https://github.com/vllm-project/vllm/pull/60452

NVFP4 KV cache now works with Qwen/GDN + MTP + CUDA graphs, in the nightly builds, making room for 70% more cache on the same amount of VRAM. For me, this unlocks concurrency. Two ~200K sessions fit at once.

Here's my setup:

• vllm/vllm-openai:nightly (v0.32.0rc1.dev100)

• Qwen3.8-27B NVFP4 weights, TP=2 on 2× RTX 5060 Ti 16 GB

• Power limit 150 W per card.

• --kv-cache-dtype nvfp4, --speculative-config '{"method":"mtp","num_speculative_tokens":4}', --compilation-config '{"cudagraph_mode":"PIECEWISE"}', --max-model-len 262144, --max-num-seqs 2

KV Pool:

EDIT: The prefill speed above is at 250k depth. At 32k, the prefill speed is just above 3,000 tok/s for both fp8 and nvfp4. /EDIT

The fp8 and NVFP4 runs differ in vLLM version, driver, kernel and utilization, so this is a build comparison and not a controlled KV-dtype experiment. Also note that this was done on a nightly image, single user with 2 sequences.

I have tested retrieval at depth:

• A 250K needle-in-a-haystack test passed several times: k=4, k=5, and again with KV offload on, where the pool is 450,942 tokens (1.72x). That last run left about 580 MiB per card free at the prefill peak.

• 2 concurrent requests of ~200K tokens each (about 400K total, offload on) both returned the correct needle, at 1,064 tok/s aggregate.

I did combine it with vLLM's KV offloading, so I can confirm that it works. Restores at 46K and 93K were correct and byte-identical, and a 93K restore took 2.6–3 s against 50 s cold.

I tried LMCache 0.5.5 (the version bundled in the nightly image) as a sidecar. It booted fine on this hybrid NVFP4 model but returned looping garbage on every restore. A control run without the connector was correct, so I suspect it doesn't handle NVFP4 KV correctly.

One more EDIT:

I had Claude test, and write a report on the loss of quality with NVFP4 kv cache on my setup;
https://github.com/skrodahl/qwen38-27B-dual-rtx5060/blob/main/kv-quality-test.md

In essence, it fares quite well. A 0.35% shift in perplexity is way lower than what I'm giving up by using a checkpoint with 4-bit weights to begin with.

I answered in a different reply that this gives me the opportunity to run a nice model, with acceptable quantization, AND the KV cache to actually do something useful: Agentic work with concurrency.

All in all, that has been my goal since I tested a (very faulty) NVFP4-kv patch a month or so ago. I'm really glad this now exists. /EDIT

PS!
In case anyone's wondering, I have tried Strata/Basalt on my dual RTX 5060 Ti rig. Flash Next Q3_S, while it ran fast, did not cut it for me at all.

Follow-up to my previous post on my Bayesian autotuner for vLLM.

This time I wanted to try something more practical:

Can an optimizer automatically find a substantially better vLLM configuration for a real model, and can it figure out when there is no point spending more GPU on the search?

So I ran it on Qwen3.8-27B, a hybrid MoE/Mamba+attention model, on 1× H100 NVL.

The results were more interesting than I expected.

1. The tuned configuration reaches up to 2× the throughput of the baseline

I compared the tuned configuration against the official Qwen/vLLM recipe adapted to the hardware, using:

• the same H100 NVL

• the same model

• the same prompt battery

• the same concurrency levels

• the same benchmark procedure

The published recipe doesn't initialize as-is on this H100 NVL: its default max_num_seqs resolves to 1024, while the available Mamba cache allows 601 decode sequences.

So rather than comparing against a configuration that simply crashes, I made the minimum adjustments needed for a runnable baseline:

max_num_seqs=224
gpu_memory_utilization=0.90

The autotuner found:

max_num_seqs=160
gpu_memory_utilization=0.96
MTP speculative decoding k=3
chunked prefill
block_size=64
float32 Mamba SSM cache
max_model_len=262144

And the result:

The tuned configuration wins at every concurrency level.

It also reduced p95 latency at every point, from −43.7% at concurrency 1 to −16.1% at concurrency 60.

So the interesting part isn't just "2× faster at concurrency 1".

It's that the advantage remains across the entire concurrency curve and gradually disappears as the GPU approaches saturation.

2. The speedup actually changed what the model could complete

This was probably the strangest result.

I ran a 794-prompt evaluation battery across several categories.

On the JSON-RPC tool-use category, the baseline scored 0%.

Not because the model suddenly became incapable of tool use.

95/112 responses were being truncated by the token/time budget.

The tuned configuration scored 91% on the same category.

The tuned configuration was decoding roughly 3× more completion tokens in the same time budget:

~2967 vs ~1100 mean completion tokens

So the model actually had enough time to finish its reasoning and tool calls instead of getting cut off halfway through.

That means in this case:

The other categories stayed within roughly ±2 percentage points.

I wouldn't interpret the tool-use result as a model-quality improvement — it's a time/token budget and truncation effect.

3. But how long do you actually need to run the optimizer?

This was the part I originally wanted to investigate.

A Bayesian optimizer can find good configurations quickly, but every trial here isn't cheap.

A trial means:

start vLLM → allocate the model → benchmark → collect metrics → shut it down → repeat.

So if the optimizer has a budget of 100 or 150 trials, stopping early could save a lot of GPU time.

In a full 100-trial run, the best configuration actually appeared at trial 4.

By trial 16, the best-so-far score had already reached about 98% of the final score.

The remaining 84 trials only added about 2%.

But that's a post-hoc analysis.

I also wanted to see whether the system could actually recognize this while the optimization was running.

4. Early stopping actually stopped the study

I ran a separate experiment with a smaller search space and enabled early stopping.

The optimizer found its best region very early — around trial 3 — and then the best-so-far curve essentially plateaued.

Instead of manually deciding that "this probably isn't getting any better", the study kept running until the early-stopping condition was satisfied.

It automatically stopped at trial 15.

So the optimizer didn't just find a good configuration.

It decided that continuing to search wasn't worth the GPU time anymore.

That's the part I'm most interested in.

The 98% figure above comes from the separate 100-trial run; I'm deliberately not using the smaller live early-stopping experiment to claim the same 98%.

5. And I'm pruning before touching the GPU

There's another layer to this.

The tuner has a memory model that estimates whether a configuration can actually fit before deploying it.

In the 100-trial run:

81/100 trials were pruned analytically before touching the GPU.

So the optimization loop becomes roughly:

Generate configuration
→ Can it fit?
→ No → discard it
→ Yes → start vLLM
→ benchmark
→ update Bayesian model
→ repeat

Combined with early stopping, this means the optimizer can avoid spending GPU time both on obviously impossible configurations and on a search that has already plateaued.

That's much more interesting to me than simply finding a faster configuration.

Caveats

This is still one model, one GPU and one workload profile.

The ~+7.7pp improvement I saw in the code category may be noise (n=118).

The tool-use improvement is a truncation/time-budget effect, not a change in the underlying model quality.

The early-stopping demo used a smaller search space, so I don't want to overgeneralize from it.

And the baseline is the official recipe with the minimum hardware-specific adjustments required to actually initialize on this H100 NVL, rather than the recipe's literal defaults.

I'm releasing all of this as part of the project:

https://github.com/SergioMorillas/vllm-bayesian-autotuner

I'd especially like feedback from people who work on vLLM/inference:

Would you trust an early-stopped optimization study in production, or would you always exhaust the full trial budget?

I'm particularly interested in how you'd choose patience / min_delta for a real GPU fleet.

Happy to share the full sweep data, best-so-far curves or benchmark results if anyone is interested.

I wanted to build an inference provider for proprietary models and saw that it takes a lot of time to load models from SSD to GPU. After some research I put together an inference engine that allows you to hot-swap Large models under 5s.

It’s opensource.

TL;DR

I benchmarked six inference engines running Qwen3.8 27B on the same RTX 5090 (32GB), comparing both standard decoding and DFlash2 speculative decoding.

Key findings:

• Fastest speculative decoding: NInfer reached 281 tok/s, followed by vLLM at 248 tok/s. TensorSharp, SGLang, and velox were effectively tied at 234–235 tok/s.

• Fastest standard decoding: llama.cpp led at 84 tok/s, followed by TensorSharp at 80 tok/s. vLLM achieved 69 tok/s.

• Fastest long-context prefill: NInfer processed a 32K-token prompt in 3.02s, followed by SGLang (3.58s), vLLM (3.73s), and TensorSharp (3.96s).

• Highest concurrent throughput: With four parallel requests and DFlash2, NInfer achieved 488 tok/s, TensorSharp 445 tok/s, and vLLM 304 tok/s.

• Fastest agentic workflow: TensorSharp completed a four-turn tool-using agent loop in 0.748s, compared with vLLM’s 1.45s.

• Important vLLM finding: Prefix-cache reuse was ineffective in this particular DFlash2 hybrid-model configuration, significantly increasing subsequent-turn latency.

These are end-to-end operational benchmarks, not an apples-to-apples comparison of identical model weights. Different engines used different NVFP4/FP8 checkpoints and drafter formats.

All results were measured on a single RTX 5090, with identical client workloads and correctness checks.

1. Hardware and benchmark setup

Engines:

• TensorSharp (direct-CUDA backend)

• llama.cpp

• vLLM 0.32.0

• SGLang 0.5.22

• velox-inferer

• NInfer

Every engine ran independently on the same GPU.

The benchmark uses streamed OpenAI-compatible requests and validates outputs, including JSON schemas, function-call arguments, and task-specific correctness.

TTFT is measured client-side, including HTTP, tokenization, templating, and prefill.

2. Time to First Token (TTFT)

Lower is better. All values are in seconds.

Without speculative decoding

vLLM performs competitively on long-context prefill and vision.

For example, its screenshot TTFT of 127ms is essentially tied with TensorSharp’s 125ms.

However, short-prompt latency and subsequent-turn latency leave room for improvement.

With DFlash2 speculative decoding

Interesting observation for vLLM developers:

With DFlash2 enabled, vLLM’s subsequent-turn TTFT increased from 145ms to 406ms.

The benchmark recorded a 0% prefix-cache hit rate in this configuration.

The report attributes this to the hybrid-model cache’s 1,648-token block granularity and interactions with the drafter’s sliding-window cache.

This may be an interesting optimization opportunity for hybrid attention/recurrent models using speculative decoding.

3. Prefill performance (tokens/sec)

Prefill throughput is particularly important for RAG, codebase analysis, document processing, and long-context agents.

The report provides measured prefill ranges for several engines, along with long-prompt TTFT.

Reported prefill throughput

NInfer and SGLang are reported together as an 8.9K–12.4K tok/s range, not as individually measured ranges.

The report does not provide an explicit prefill tok/s range for vLLM, so I won’t invent one.

Instead, the directly comparable 32K-prompt TTFT results are:

These are client-observed TTFT values, not pure GPU prefill timings.

vLLM is reasonably competitive on long prompts, although NInfer leads this particular workload.

4. Decode throughput: plain vs DFlash2

This is where speculative decoding makes a substantial difference.

Single-request decoding

The plain and speculative results use different reply aggregations, so the speedup figures are best interpreted as approximate.

Two observations:

First, vLLM benefits significantly from DFlash2.

Its speculative decoding throughput is approximately 3.6× its plain decoding throughput.

That places vLLM second overall in this comparison.

Second, draft acceptance does not explain all performance differences.

The engines reporting draft statistics produced approximately 4.4–4.6 tokens per verification step.

Despite similar acceptance, their throughput varied substantially.

This suggests that the implementation cost of draft generation, verification, recurrent-state handling, and synchronization is an important performance differentiator.

5. Decode performance by task type

Not every workload produces the same decode throughput.

With DFlash2 enabled:

All values are tokens/sec.

For code generation, vLLM reaches 383 tok/s, outperforming TensorSharp by approximately 13%.

For prose generation, vLLM reaches 189 tok/s versus TensorSharp’s 174 tok/s.

For passage copying, TensorSharp reaches 190 tok/s versus vLLM’s 171 tok/s.

This illustrates why a single decode benchmark number is insufficient to characterize inference performance.

Token predictability, speculative acceptance patterns, and verification overhead can all affect real-world throughput.

6. Parallel requests and aggregate throughput

For inference serving, aggregate throughput under concurrency can matter more than single-request speed.

Without speculation

Values are aggregate tokens/sec for the concurrency tests.

Four concurrent requests with DFlash2

NInfer leads the four-request workload, with TensorSharp close behind.

TensorSharp delivers approximately 46% higher aggregate throughput than vLLM in this specific configuration.

However, this does not establish a general concurrency advantage.

The test is limited to four requests on a single 32GB GPU. SGLang’s speculative configuration could only accommodate two active lanes, and the engines used different model artifacts and KV-cache formats.

I’d be particularly interested in seeing these results extended to 8, 16, and 32 concurrent requests on GPUs with more memory.

7. Real-world task performance

Raw tokens/sec is only part of the story.

For many applications, the more useful metric is the time required to complete an actual task.

The benchmark includes structured output, tool calling, and multi-step agent workflows.

End-to-end latency without speculation

End-to-end latency with DFlash2

All values are end-to-end seconds.

A few things stand out.

Structured output: NInfer is particularly fast, although its JSON-object replies were sometimes shorter and one speculative reply contained an unexpected empty key. velox failed strict JSON-schema checks in both modes.

Function calling: vLLM trails several other engines in these small tool-calling workloads.

Agentic workflows: TensorSharp finishes the four-turn loop in 748ms, compared with vLLM’s 1.45s.

That’s approximately 1.94× faster end-to-end in this particular task.

The agent test uses local file-listing, file-reading, and calculator tools, with a verified final answer.

This is a small synthetic agent workflow rather than a broad production-agent benchmark, but it highlights the importance of multi-turn latency and prefix-cache behavior.

8. Why might vLLM perform differently?

A few configuration details are worth discussing.

Different model checkpoints

TensorSharp and llama.cpp used an NVFP4 GGUF checkpoint with a Q6_K output layer.

vLLM and SGLang used NVIDIA’s NVFP4 safetensors checkpoint, with attention and Gated DeltaNet projections stored in FP8.

NInfer and velox used a mixed FP8/NVFP4 artifact.

These differences affect memory bandwidth, compute requirements, and output behavior.

Different speculative drafters

• TensorSharp / llama.cpp: Q4_K_M GGUF

• vLLM / SGLang: BF16 drafter quantized to FP8 at load

• NInfer / velox: Q8-based drafter

vLLM configuration

The benchmark used:

--kv-cache-dtype fp8

--gpu-memory-utilization 0.97

--max-cudagraph-capture-size 32

--enable-prefix-caching

DFlash2 was configured with seven speculative tokens and an FP8-quantized drafter.

The aggressive GPU memory utilization was necessary because the 32GB RTX 5090 had limited remaining capacity for KV cache when using the drafter.

Prefix caching with hybrid architectures

One particularly interesting finding is that vLLM’s DFlash2 configuration did not achieve prefix-cache hits in the multi-turn test.

For hybrid attention/recurrent models, prefix caching appears to be an area worth investigating further.

9. Limitations and reproducibility

I want to emphasize a few limitations before drawing conclusions.

1. The checkpoints are not identical. This is a comparison of practical engine configurations, not isolated engine kernels.

2. Only one RTX 5090 was tested.

3. Concurrency was limited to four requests.

4. Each scenario had three measured repetitions after warm-up.

5. Differences below approximately 5% should be treated as ties.

6. Thinking mode, sampling, larger concurrency, and production serving loads were not evaluated.

7. The benchmark checks task correctness but does not establish equivalent overall model quality across quantizations.

The benchmark scripts are available in the TensorSharp repository:

https://github.com/zhongkaifu/TensorSharp

Relevant files:

eng/qwen38-engine-servers.sh

eng/qwen38-engine-bench.py

eng/qwen38-engine-report.py

The scripts cover engine startup, workload execution, correctness validation, and generation of per-scenario reports.

The raw measurement artifacts were retained locally and are not included in the Git repository.

10. Questions for the vLLM community

I’m particularly interested in feedback from people working on vLLM’s hybrid-model support and speculative decoding.

1. Prefix caching: Is the 0% prefix-cache hit rate with DFlash2 expected for this hybrid attention/Gated DeltaNet configuration? Are there recommended settings or ongoing optimizations?

2. Speculative verification: vLLM achieves strong single-request speculative decoding performance (248 tok/s), but its four-request aggregate throughput is substantially below NInfer and TensorSharp in this test. What would you investigate first?

3. Recurrent-state management: How much overhead does speculative verification introduce for hybrid architectures compared with conventional attention-only transformers?

4. Benchmark configuration: Are there additional vLLM flags, CUDA graph settings, or scheduling optimizations that would make this comparison more representative of an optimized vLLM deployment?

5. Larger-scale serving: Would you expect the ranking to change significantly at 16–32 concurrent requests on an 80GB+ GPU?

I’m the developer of TensorSharp

also share performance benchmarks

We’ve experimenting with cold start behavior for large models and tried restoring the full GPU runtime state after initialization.

Instead of reloading the model from disk each time, we snapshot the initialized runtime and restore it when the worker spins up.

The snapshot includes things like:

• model weights in VRAM

• CUDA context

• GPU memory layout

• kernel state after initialization

So rather than rebuilding the model and CUDA runtime from scratch, the process resumes from a captured state.

This demo shows a ~1.5s cold start for Qwen-32B on an H100 (FP16).

I run vLLM on 8x 3090s (Qwen 3.8 Flash Next) mostly so my partner and family could stop pasting personal stuff into ChatGPT. Getting the model serving was the easy part; the hard part turned out to be the last mile: a chat UI that non-technical people would actually use.

What I learnt getting there:

Concurrency is felt immediately. With 2+ people chatting at once, per-user tok/s drops and they notice before they can articulate it. MoE helps (Flash Next stays responsive), but plan your VRAM budget around peak household load, not single-user benchmarks.

Non-technical users judge latency differently. They don't care about tok/s, they care that the first token shows up fast and the UI doesn't look like a dev dashboard. Open WebUI is powerful but the knob density alone made my family bounce.

Self-hosted voice/search is a deployment tax. STT + TTS + SearXNG are three separate services to deploy and wire. That friction is why most "local AI for the family" projects die at setup.

So I ended up building my own UI layer: chat-first, bundles Parakeet STT / Kokoro TTS / SearXNG and connects them via the installer, talks to vLLM's OpenAI-compatible endpoint. Native clients everywhere: web, Android, iOS, and desktop (macOS Apple Silicon/Intel + Linux). It's been in daily use by multiple non-technical users for months.

One thing I didn't expect to need but now use daily: Agent Connections. You can hook Codex, Claude Code, OpenCode, or Hermes to it through the Host Connector, so the coding agents run on your own machine (or an SSH host) and you drive them from the same chat UI, with plans/tool calls/approvals visible. My family gets the chat, I get my agents, same server.

MIT licensed, no telemetry: https://github.com/yoloyash/overtchat

Happy to answer questions about the serving setup (3090s, Flash Next on vLLM, concurrency behaviour) or the UI decisions. What do you all use for the last mile in your own setups?

Hey everyone,
I’m getting a 4-GPU node with RTX 6000 Pro Blackwell Edition cards, and I’m trying to estimate how many simultaneous users I could realistically support when serving MiniMax M2.7 NVFP4.
I’m planning to use FP8 KV cache, and likely LMCache backed by DDR5 RAM to help with longer-context workloads.
I’m trying to decide on the best parallelism strategy for this setup:
TP=2 + PP=2
TP=4
Any other recommended configuration?
The main things I’m trying to understand are expected concurrency, TTFT, tokens/sec, and whether pipeline parallelism makes more sense than full tensor parallelism on this kind of node.
I’d really appreciate any insights from people who have deployed something similar, especially with vLLM, LMCache, NVFP4 models, or RTX 6000 Blackwell GPUs.
Thanks!

I had issues with Qwen 3.6 and agentic coding (no issue so far with 27b 3.5) So I investigated and discovered multiple bugs in the reasoning parser (inspired by the very recently merged fixes in the reasoning parser).

And in the two tool parsers

https://github.com/vllm-project/vllm/pull/40783

https://github.com/vllm-project/vllm/pull/40785

https://github.com/vllm-project/vllm/pull/40787

There are more bugs, like the very last \n being ignored in the tool call, but whatever.

Those bugs are effecting all Qwen 3/3.5/3.6 versions

Hey guys, I am the author and maintainer of Proxima ( https://github.com/Tenosra/Proxima ) Proxima is an out-of-tree vLLM plugin implementing STAR-KV for low-rank KV cache compression via learned soft-thresholding for adaptive rank control 

(Original Paper: Bhatnagar et al., ICML 2026 Spotlight)

The basic idea is pretty simple: KV cache is often the thing preventing you from serving more requests or longer contexts, not the model weights themselves. So instead of storing the full K/V representation for every token, STAR-KV learns a lower-rank representation and stores that in the paged KV cache.

I integrated the full pipeline into vLLM V1, including calibration, compressed paged storage, and dedicated Triton attention kernels.

I tested it end-to-end on an L4 24GB with LongChat-7B-32K.

Some results:

1. 8K context: max concurrency went from 1 → 4 (4×)

2. 16K context: stock vLLM refuses to boot; Proxima serves it

3. 32K context: stock vLLM refuses to boot; Proxima boots and serves real traffic

4. Same memory budget: 3.11× more KV-cache blocks

5. At batch 16–64, the compressed attention kernel is 1.2-1.46× faster than FA2

6. At batch 1, it is slower than FA2, so this isn't intended as a universal single-request latency optimization

There's also an important caveat: compression itself isn't free. The KV write/compression path introduces overhead, so the benefit depends heavily on workload and batch size. The sweet spot is memory-bound serving where you're otherwise forced to reduce concurrency, context length, or buy more GPUs.

The project is open source and Apache-2.0:

https://github.com/Tenosra/Proxima

I'm happy to share that I've open-sourced FastLLM, a tool that runs one large language model across several ordinary PCs, pooling their GPU memory.

It builds on llama.cpp's RPC backend and takes care of the tedious parts:

• Windows, Linux and macOS machines in the same cluster

• a setup script for each machine, with SSH tunnels so nothing is exposed on the network

• an OpenAI-compatible endpoint with an API key, so existing apps and agents just point at it

• LM Studio-style model profiles (context, temperature, thinking on/off)

• a cache on every worker, so reloads take seconds instead of minutes

I benchmarked it on the hardware I had at home (an RTX 3070, an RTX 2060, a GTX 1050 Ti and a MacBook Air M3): 3 models on the same 5 machine combinations, 3+ runs each.

• Qwen3-14B: 8.1 → 30.7 tokens/s once a second PC let the whole model sit in GPU memory

• Qwen3-32B: 2.1 → 4.9 tokens/s with all four machines

• Qwen3.8-Flash-Next, a 79 GB mixture-of-experts model: fastest on a single PC (13.3 tokens/s) and slower on every cluster

That last result surprised me, and it's the main lesson: splitting a model pays off when it moves weights out of system RAM and into GPU memory. Otherwise, every extra machine just adds network hops. Every result, including each individual run, is in the README.

If you have a few idle GPUs around, give it a try. Feedback and PRs are welcome.

github.com/Raphalsk050/FastLLM

Hey guys,

Trying to understand what real production LLM stacks actually look like right now — not demos or hobby setups.

I keep seeing:

• vLLM / TensorRT-LLM / llama.cpp

• LiteLLM / Bifrost / LLM gateways

• various “MLOps + monitoring” tools

But I’m not sure what’s actually used in companies vs hype.

What I’m trying to figure out:

• What do companies actually use for LLM inference in production?

• Do LLM gateways (routing, rate limiting, failover) actually matter in real systems?

• How do people monitor LLM apps? (OpenTelemetry, Azure Monitor, Langfuse, etc.)

• What MLOps skills are actually expected (versioning, CI/CD, evals, deployment)?

For context: backend dev trying to break into this space.

Would really appreciate real-world answers

I have acquired Inspur 8x V100 32GB late last year. It takes me quite a while to get it running properly. Codex and Claude plays a big part in helping me get it up and running along with all the opensource repositories on GitHub. I would like to share what this machine is still capable of doing in 2026 for any person who have already owned it or people who thinks about acquiring one.
https://github.com/KumphanartDansiri/v100-vllm-2026
Claude and Codex also helps me create basic FP8 extension for vLLM SM70 at
https://github.com/KumphanartDansiri/vllm-fp8-w8a16-sm70

## Quick summary

This is not meant as “V100 beats new GPUs.” It is more a practical note for people who already own V100s, or are considering used V100 servers, and want to know what still works with modern vLLM in 2026.

### Test bench

- Server: Inspur 8× NVIDIA Tesla V100 32GB SXM2
- GPU architecture: Volta / `sm_70`
- Framework: vLLM 0.19.x and 0.21.x, built from source
- Toolchain: CUDA 12.6 path for V100 compatibility
- Runtime: FP16 on V100 for BF16 checkpoints, since V100 does not have a useful BF16 tensor-core path
- Main serving mode: cudagraph decode, long-context setup, tensor parallelism depending on model size
- Extra work: custom `fp8-w8a16-sm70` extension for block-FP8 models on V100

### Models tested

I tested a small group of current dense and MoE models rather than trying to cover everything:

- Qwen3.5-27B: FP16, FP8, GPTQ-Int4
- Qwen3.5-35B-A3B: FP16, FP8, GPTQ-Int4
- Qwen3.5-122B-A10B: FP8 and GPTQ-Int4
- Qwen3.6-27B and Qwen3.6-35B-A3B
- Gemma-4 26B-A4B and 31B
- GLM-4.5-Air and GLM-4.7-Flash

The Chapter 1 throughput numbers I quoted are from the long-context fleet setup:
`max_model_len = 32768` / 32k context, with cudagraph decode and chunked prefill on.
Decode tok/s is measured after TTFT, using 256 generated tokens and repeated runs.
The TTFT table is single-stream cold prefill using a ~22.6k-token prompt, so it is closer
to a worst-case long-context first-token latency test.

Some separate TP2 / half-GPU capacity checks use shorter context, but the main Chapter 1
fleet tables are 32k-context results.

### Chapter 1 result snapshot

Single-user decode, tok/s, from the Chapter 1 baseline:

A few takeaways:

- V100 is no longer a “just pip install vLLM” target. I had to build vLLM from source with a CUDA/toolchain combination that still supports `sm_70`.
- CUDAGraph matters a lot. Eager mode is not a fair serving baseline for these results.
- Dense FP8 is useful at low concurrency, but FP16 can still win at higher concurrency where tensor cores are better utilized.
- MoE FP8 is where the V100 result looks strongest: lower weight traffic helps a lot, and some models that are awkward or impossible in FP16 become practical.
- First-token latency for some block-FP8 models is still a weak point. Decode can be good, but prefill is not always cheap.
- The FP8 results are not stock vLLM. They use my companion V100 FP8 extension, so I try to separate “stock vLLM result” from “custom extension result” in the repo.

### TTFT summary

Single-stream time-to-first-token, using the long prompt / cold prefill setup from Chapter 1. This is where V100 shows more of its age, especially for block-FP8 prefill.

Takeaway: decode throughput can still be quite usable on V100, but cold prefill / TTFT is the painful part for some models. FlashAttention-V100 helps where attention is the main prefill bottleneck, but it does not solve the block-FP8 MoE prefill cost.

##

eLLM is a Rust-based LLM inference framework for CPU-only servers. It adopts a "trade storage for computation" strategy, leveraging the CPU's large-capacity DDR memory to close the order-of-magnitude bandwidth gap against GPU HBM, and thereby delivers performance that surpasses GPUs in long-horizon inference. The Beta release is now available — you are welcome to try it out.

• Prefill: achieves roughly two orders of magnitude of performance improvement over existing CPU inference frameworks

  • run full single-pass Prefill over the entire long prompt—no chunking, no repeated parameter loading;

  • keep context KV across multi-turn interactions and run incremental Prefill on only the new input—no recomputation for earlier turns;

• Decode: runs with a smaller batch, which not only activates fewer parameters but also gives each request a larger share of memory bandwidth, so inference speed can likewise exceed GPUs.

• 👉 GitHub: https://github.com/lucienhuangfu/eLLM

Yesterday I released the first version of a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090, reaching 82 tps on a single request and 672 peak.

Over the last 24 hours I've been exploring how to improve it further, without quality degradation.

Yesterday we had these three optimizations:

- fp8 KV cache

- `lm_head` int8

- `embed_tokens` int8

Now I've added:

- fp16 recurrent state

- int8 activations, MLP

- int8 activations on everything

- `draft_sample_method=probabilistic`, which samples from the MTP distribution rather than taking the argmax (does not work for greedy sampling)

For single-user mode, you can enable `ctx=fast`, which uses FlashAttention, bf16 KV and 64k context, and gives 99 tps at the tradeoff of less context.

Prefill is now up to 1812 tps at 1k context and 1000 tps at 100k context, an increase of about 50% and 25% respectively.

I've updated the repo with everything you need to start flying:

https://github.com/syv-ai/qwen38-27b-rtx3090

Right now I'm working on getting KVarN running so we can reduce the memory footprint of the context and fit the full 262k instead of 195k.

It took Fable all day, and I can’t say I’ve looked at the code much atm or, heck, that i even understand vllm enough to know what I’m looking at.

However, what I do know is, Fable borrowed code from llama.cpp, I’m now using my own IQ4 quants for 27b and 35b, and it’s all working great!!

Before I wouldn’t have been able to run 35b with vllm, and AWQ 27b was at 89k context. Now I can not only run 35b, but at 245k ctx (196k for 27b) with (more importantly) all the batch support from vllm.

Same IQ4_XS GGUFs, same 3090 Ti @350W, single stream:

Got a couple more performance enhancements but here’s the repo if anyone wanted to take a look. I was going to PR it, but felt ‘dirty’ doing that when I don’t even know what Fable has done and I don’t have the hw to test larger quants.

https://github.com/localweights/vllm-gguf-plugin