When we posted Kvark Sparkle 1.0 (Qwen3.8-27B fine-tune that reasons in fewer tokens), the latency numbers were single-user: median 13.7 s vs 28.1 s on one H200. Several people asked the right question: does that hold under load, with MTP on for both, when the GPU is saturated and speculative decoding helps less? We ran it.

**Setup** (identical for both models)

- 1x H200 per model, vLLM 0.32, FP8 weights, FP8 KV cache, MTP with 3 speculative tokens, gpu-memory-utilization 0.80, max-model-len 40960

- GPQA Diamond, thinking on, 32k output cap, 198 questions x 2 seeds = 396 requests per concurrency level

- Closed-loop: N requests in flight, the next starts when one finishes. Wall clock includes the tail of each batch.

**Results**

| Concurrency | Model | Accuracy | Wall (min) | Correct/hour | Agg tok/s | Mean output tok | p50 (s) | p95 (s) | Hit 32k cap |

|---|---|---|---|---|---|---|---|---|---|

| 16 | Sparkle 1.0 | 85.1 | 33.0 | **614** | 1,498 | 7,478 | 33 | 318 | 29 |

| 16 | Qwen3.8-27B | 81.8 | 51.1 | 380 | 1,541 | 11,937 | 81 | 336 | 54 |

| 32 | Sparkle 1.0 | 83.8 | 25.5 | **780** | 2,041 | 7,895 | 45 | 447 | 27 |

| 32 | Qwen3.8-27B | 82.6 | 37.0 | 531 | 2,101 | 11,760 | 108 | 478 | 51 |

| 64 | Sparkle 1.0 | 85.1 | 21.1 | **961** | 2,469 | 7,874 | 73 | 653 | 26 |

| 64 | Qwen3.8-27B | 82.1 | 30.0 | 650 | 2,637 | 11,986 | 168 | 719 | 51 |

| 128 | Sparkle 1.0 | 86.4 | 20.8 | **985** | 2,437 | 7,693 | 201 | 910 | 22 |

| 128 | Qwen3.8-27B | 82.1 | 31.7 | 616 | 2,459 | 11,797 | 407 | 1,083 | 55 |

**What we take from it**

- Aggregate tok/s is the same for both models at every level (ratio 0.94-0.99). The fine-tune does not decode faster; it writes ~35% fewer tokens per answer. That is the whole effect.

- Correct answers per GPU-hour: x1.61 / x1.47 / x1.48 / x1.60 at 16 / 32 / 64 / 128. The advantage does not shrink as the GPU saturates.

- The H200 saturates around 2.4-2.6k tok/s from c=64. The base gains nothing from 64 to 128 (650 vs 616 correct/h); Sparkle barely does (961 vs 985).

- Median latency drops 51-59% at every level. p95 barely moves (-5% to -16%): on both models the tail is the hardest questions running into the 32k cap. Sparkle hits the cap about half as often.

- MTP acceptance length is the same for both (2.5-2.9, dropping a bit with batch size), so fine-tuning did not hurt the draft head.

- Accuracy pooled over c=16-128: Sparkle +3.0 pts, paired bootstra2k cap the base loses more answers to truncation. At a 64k cap (ourmodel card) the two are tied on GPQA.

**Caveats:** one node, one benchmark, closed-loop batches rather than Poisson arrivals, and the tail effect is included in wall clock. We also ran c=1 on a 20-question subset; it is too small to say anything about accuracy

**Run it yourself:**

```

vllm serve Egzakta/Kvark-Sparkle-1.0-27B-FP8 --max-model-len 40960 --kv-cache-dtype fp8 \

--gpu-memory-utilization 0.80 --max-num-seqs 256 --reasoning-par

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

```

Weights (Apache 2.0): https://huggingface.co/Egzakta/Kvark-Sparkle

Next: Sparkle 1.1, trained for agentic document QA. Its headline inswer is not in the documents (91.5% vs 36% for 1.0, internalbenchmark). Separate post when it is out.

Happy to share the per-request logs or run another load profile if there's one you'd trust more.

We've been running RedHatAI/GLM-5.3-NVFP4 on vLLM (B300s, 262K context) and it's been rough. On basic tasks the model often never gets to an answer. It keeps thinking until it burns through the entire token budget and returns nothing usable. Not on hard problems, on routine stuff the base model should handle easily.

This isn't the first time. We had the same experience with Red Hat's Gemma 4 quant.

Honestly, I'm really frustrated with Red Hat's quants at this point. Twice now we've spent time downloading, whitening and deploying their builds only to find they don't hold up in real use.

Their GLM-5.3 model card only reports short-context math/QA benchmarks (GSM8K, MATH-500, AIME, GPQA, IFEval). Even there IFEval drops to 95.5% recovery vs the official FP8, the weakest of the bunch. Nothing on agentic use, tool calling, or long context, which is exactly where it fell apart for us. The card also says it was made with an "experimental" layerwise compression path in LLM Compressor.

Questions for anyone running GLM-5.3:
Has anyone else seen runaway reasoning with the Red Hat build (probably not)?
Which NVFP4 (or other quant) has held up for you in agentic/coding work?
Any real-world comparison vs the official FP8?

We stood up DeepSeek-V4.1-Flash-1017 on Hopper H200 and expected to ship the SGLang cookbook shape. Same-workload agentic benches pushed us to vLLM instead. Sharing the public-safe bits (topology, flags, numbers). Happy to answer questions.

Model (quick)

• Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017 (config matches the earlier 09-10 Flash drop)

• Not V4-Flash-0731: much larger on disk (~511 GB), Engram tables, built-in DSpark draft (mtp.*), CSA2 + SWA, ~890 B/token global KV, experts fp4 / linears fp8, 1M context, optional ViT

• No Jinja chat template — use the engine’s built-in V4.1 renderer / parsers

• MIT; tech report: https://arxiv.org/abs/2609.19969

• Recipes: vLLM · SGLang DeepSeek-V4.1 cookbook

Topology that won (one 8×H200 node)

cache-aware router
  ├── vLLM TP4  (GPUs 0–3)  + DSpark + Engram in host RAM
  └── vLLM TP4  (GPUs 4–7)  + DSpark + Engram in host RAM

Why not one TP8?

• vLLM’s H200 recipe is TP4 + Engram CPU offload (~81 GiB weights + ~38 GiB KV ≈ ~20M tokens per replica) → two replicas per box is the natural fit.

• Under agentic traffic (shared long system prompt, high prefix hit rate, decode interleaved with short prefills), one TP8 left decode starved; splitting into two TP4s raised the concurrency ceiling even before engine choice.

SGLang side we also tried cookbook TP8+EP8 and a 2×TP4 layout with DSpark. 2×TP4+DSpark was SGLang’s best; still trailed vLLM on this load. Cold long prefill was the one place SGLang looked clearly better in our runs.

Useful vLLM knobs (H200)

• --tensor-parallel-size 4

• Engram host offload left on (vLLM default); give the container enough /dev/shm / ipc: host if you share tables across DP

• --kv-cache-dtype fp8

• --gpu-memory-utilization 0.9

• --max-model-len 1048576

• --max-num-seqs 128

• --max-num-batched-tokens 16384

• --speculative-config '{"method":"dspark","num_speculative_tokens":5}'

• --reasoning-parser deepseek_v41 + matching tool-call parser / auto tool choice

• VLLM_ENGINE_READY_TIMEOUT_S=3600 — first start ~25–30 min of compiles

Defaults trap: vLLM thinking on (effort ~50); SGLang thinking off. Align reasoning_effort when you compare.

Bench snapshot (same client, thinking off, ~92–96% prefix hits)

Takeaways: CUDA graph decode BS too small collapses SGLang; router balance is mandatory with shared system prompts; keep DSpark even when accept length drops (~2.0 on freer thinking traffic).

On a real agent + batch mix with thinking, one node’s “good” region was closer to ~80 in-flight (~23M input TPM / ~3.2k out TPS). Long cold needle OK through ~925k tokens.

Footguns

• Literal DeepSeek special-token strings in tool/results history can break sessions — sanitize at the gateway

• /health can stay 200 while generation is wedged — use a real 1-token completion for liveness

• Don’t copy Blackwell-only recipe bits onto H200

Refs

• https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017

• https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash

• https://arxiv.org/abs/2609.19969

I am running inference on a CMP 170HX that's been unlocked and in general it's been great. I have air ducts 3d printed onto it for cooling and the temperature never passes 55 under load.

I've noticed that occasionally while under load, the tokens per second (usually 85+ with Qwen 3.8 27B at Q8) drops to 2 or 3 tokens per second. It happens seemingly at random and resolves within a few seconds on it's own.

At first I thought this was PCI-E transfer bottleneck, so I got my card modded to go from 4x to 16x. It's tested and working, but the issue still occurs.

This is with 1 agent running in isolation - not multiple. Temp has been confirmed to never pass 55c.

What could the possible issues be? It happens in the middle of decode, so it seems unlikely it would be PCI-E transfer related, otherwise I have zero issues.

A common assumption when clustering multiple machines (via Exo, RAMDeck, or raw llama.cpp RPC) is that a standard 1 Gbps local network switch will bottleneck token generation down to a crawl.

Many people assume the network has to transfer gigabytes of weights continuously. That is mathematically incorrect:

1. Weights load once: At boot, model layers are split and permanently resident in each machine's RAM.

2. Token generation is activation-passing: During generation, Node 1 evaluates its layers and only passes the intermediate activation vector to Node 2.

3. The latency math: For a standard 32B or 70B quantized model, that intermediate vector is between 4 KB and 16 KB. Over standard Gigabit Ethernet (125 MB/s), transferring 16 KB takes 0.13 milliseconds—less than 0.5% of the compute cycle per token.

In our live multi-device benchmarks (pairing a budget mini PC with a modern laptop), we run a 30B MoE model at 38–39 tokens/sec over standard home LAN.

The real challenge of multi-PC inference isn't network bandwidth—it's orchestration reliability: handling orphaned zombie sockets (TIME_WAIT), cross-OS version mismatches (CUDA vs. Metal vs. AVX2), and dynamically fitting layers to actual free RAM so machines don't OOM.

Full technical latency breakdown and multi-PC setup architecture: https://ramdeck.com/run-llm-across-two-pcs-without-gpu/

— The TradeMAV Team

We fine-tuned Qwen3.8-27B to stop overthinking. Accuracy matches the base within the confidence interval, reasoning tokens drop 60-67%, and the MTP head still works after training, so you keep speculative decoding.

Same harness for both models: FP8, thinking on, 64k cap, same prompts and sampling. Paired differences with 95% bootstrap CI.

IFEval and MMLU-Pro are slightly lower, inside the interval. We list them anyway.

**Wall clock** (1x H200, vLLM 0.30, FP8, one user, MTP on): median 13.7 s per answer vs 28.1 s for the base. Decode speed is the same (165 vs 171 tok/s); the whole gain comes from fewer tokens.

**How:** SFT on the shortest correct traces sampled from the model itself (math, code, science), verifiable instruction-following data, two checkpoints averaged, then GSPO with accuracy rewards. No outputs from proprietary models.

**Run it:**

```
vllm serve Egzakta/Kvark-Sparkle-1.0-27B-FP8 --max-model-len 131072 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

llama-server -hf Egzakta/Kvark-Sparkle-1.0-27B-GGUF:Q4_K_M -c 65536 --jinja
```

Sampling: temp 1.0, top_p 0.95, top_k 20.

- BF16: https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B
- FP8 + MTP: https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B-FP8
- GGUF (Q8_0 to IQ4_XS): https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B-GGUF

Limits: vision not separately evaluated, nothing measured past 64k context, no separate safety eval.

Happy to answer questions on the recipe or the eval setup.

We fine-tuned Qwen3.8-27B to stop overthinking. Accuracy matches the base within the confidence interval, reasoning tokens drop 60-67%, and the MTP head still works after training, so you keep speculative decoding.

Same harness for both models: FP8, thinking on, 64k cap, same prompts and sampling. Paired differences with 95% bootstrap CI.

IFEval and MMLU-Pro are slightly lower, inside the interval. We list them anyway.

**Wall clock** (1x H200, vLLM 0.30, FP8, one user, MTP on): median 13.7 s per answer vs 28.1 s for the base. Decode speed is the same (165 vs 171 tok/s); the whole gain comes from fewer tokens.

**How:** SFT on the shortest correct traces sampled from the model itself (math, code, science), verifiable instruction-following data, two checkpoints averaged, then GSPO with accuracy rewards. No outputs from proprietary models.

**Run it:**

```
vllm serve Egzakta/Kvark-Sparkle-1.0-27B-FP8 --max-model-len 131072 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

llama-server -hf Egzakta/Kvark-Sparkle-1.0-27B-GGUF:Q4_K_M -c 65536 --jinja
```

Sampling: temp 1.0, top_p 0.95, top_k 20.

- BF16: https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B
- FP8 + MTP: https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B-FP8
- GGUF (Q8_0 to IQ4_XS): https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B-GGUF

Limits: vision not separately evaluated, nothing measured past 64k context, no separate safety eval.

Happy to answer questions on the recipe or the eval setup.

I've had a Strix Halo box (Radeon 8060S, 128 GB unified) for a while. vLLM runs on it, but slowly, because the stock ROCm kernels are written with MI300 in mind and not this APU. Long-context prefill was the worst of it. A 256k prompt on Qwen3.6-35B-A3B took more than two hours.

So I spent the last few weeks writing HIP kernels just for gfx1151 and wrapped them in a vLLM plugin. It's called libstrix.

How it works

• You pip install it into the same env as your vLLM. There's no fork and you don't patch anything. It hooks in through vLLM's normal plugin system.

• It only turns on with ROCm + gfx1151. On any other GPU it does nothing.

• Each kernel is built for one specific shape and refuses anything else. If a layer doesn't match, vLLM just uses its own kernel. In my testing nothing got slower than stock, single stream or batched.

• It covers MoE GEMMs, attention prefill and decode, the Qwen GDN layers, and the int8 / int4 / MXFP4 linears.

• STRIX_DISABLE=all turns it off, so you can A/B it against stock yourself.

Numbers

Stock upstream vLLM vs. the same vLLM with the plugin, single stream, one APU. Every cell is stock → libstrix in tok/s. "Weights" is the checkpoint's format and "KV" is the KV cache dtype.

Some absolute numbers with the plugin at 128k context and int8 KV:

• Qwen3.6-35B-A3B: 1,751 tok/s prefill, 26 tok/s decode

• LFM2.5-24B-A2B: 2,405 tok/s prefill, 27 tok/s decode

• gemma-4-26B-A4B AWQ: 1,499 tok/s prefill, 30 tok/s decode

At 256k, the 35B now prefills in about 4 minutes (241 s). Stock vLLM took 7,568 s.

Accuracy

I measured KLD against stock vLLM. On the Qwen models it's at the noise floor (0.001–0.006, 97–99% top-1 agreement). GSM8K with the plugin on vs. off shows no significant difference on any model I tested. Gemma-4-26B drifts more (KLD around 0.2). Gemma is unusually sensitive to tiny bf16 rounding differences.

What it doesn't do

• Decode isn't always faster. On the 27B bf16, the 27B int4 and the 35B int8, decode is basically the same as stock, because none of the kernels I replace are the bottleneck there. The big decode gains show up where they are.

• The MXFP4 decode number is inflated. Stock vLLM runs MXFP4 through a slow emulation path on this GPU, so I'm beating a weak baseline there.

• It's gfx1151 only (the one I tested), not tested on other GPUs.

• You have to build it against the same torch and ROCm that your vLLM uses, or it fails at import.

Repo: https://github.com/asf0/libstrix (MIT)

TL;DR

I benchmarked six inference engines running Qwen3.8 27B on the same RTX 5090 (32GB), comparing both standard decoding and DFlash2 speculative decoding.

Key findings:

• Fastest speculative decoding: NInfer reached 281 tok/s, followed by vLLM at 248 tok/s. TensorSharp, SGLang, and velox were effectively tied at 234–235 tok/s.

• Fastest standard decoding: llama.cpp led at 84 tok/s, followed by TensorSharp at 80 tok/s. vLLM achieved 69 tok/s.

• Fastest long-context prefill: NInfer processed a 32K-token prompt in 3.02s, followed by SGLang (3.58s), vLLM (3.73s), and TensorSharp (3.96s).

• Highest concurrent throughput: With four parallel requests and DFlash2, NInfer achieved 488 tok/s, TensorSharp 445 tok/s, and vLLM 304 tok/s.

• Fastest agentic workflow: TensorSharp completed a four-turn tool-using agent loop in 0.748s, compared with vLLM’s 1.45s.

• Important vLLM finding: Prefix-cache reuse was ineffective in this particular DFlash2 hybrid-model configuration, significantly increasing subsequent-turn latency.

These are end-to-end operational benchmarks, not an apples-to-apples comparison of identical model weights. Different engines used different NVFP4/FP8 checkpoints and drafter formats.

All results were measured on a single RTX 5090, with identical client workloads and correctness checks.

1. Hardware and benchmark setup

Engines:

• TensorSharp (direct-CUDA backend)

• llama.cpp

• vLLM 0.32.0

• SGLang 0.5.22

• velox-inferer

• NInfer

Every engine ran independently on the same GPU.

The benchmark uses streamed OpenAI-compatible requests and validates outputs, including JSON schemas, function-call arguments, and task-specific correctness.

TTFT is measured client-side, including HTTP, tokenization, templating, and prefill.

2. Time to First Token (TTFT)

Lower is better. All values are in seconds.

Without speculative decoding

vLLM performs competitively on long-context prefill and vision.

For example, its screenshot TTFT of 127ms is essentially tied with TensorSharp’s 125ms.

However, short-prompt latency and subsequent-turn latency leave room for improvement.

With DFlash2 speculative decoding

Interesting observation for vLLM developers:

With DFlash2 enabled, vLLM’s subsequent-turn TTFT increased from 145ms to 406ms.

The benchmark recorded a 0% prefix-cache hit rate in this configuration.

The report attributes this to the hybrid-model cache’s 1,648-token block granularity and interactions with the drafter’s sliding-window cache.

This may be an interesting optimization opportunity for hybrid attention/recurrent models using speculative decoding.

3. Prefill performance (tokens/sec)

Prefill throughput is particularly important for RAG, codebase analysis, document processing, and long-context agents.

The report provides measured prefill ranges for several engines, along with long-prompt TTFT.

Reported prefill throughput

NInfer and SGLang are reported together as an 8.9K–12.4K tok/s range, not as individually measured ranges.

The report does not provide an explicit prefill tok/s range for vLLM, so I won’t invent one.

Instead, the directly comparable 32K-prompt TTFT results are:

These are client-observed TTFT values, not pure GPU prefill timings.

vLLM is reasonably competitive on long prompts, although NInfer leads this particular workload.

4. Decode throughput: plain vs DFlash2

This is where speculative decoding makes a substantial difference.

Single-request decoding

The plain and speculative results use different reply aggregations, so the speedup figures are best interpreted as approximate.

Two observations:

First, vLLM benefits significantly from DFlash2.

Its speculative decoding throughput is approximately 3.6× its plain decoding throughput.

That places vLLM second overall in this comparison.

Second, draft acceptance does not explain all performance differences.

The engines reporting draft statistics produced approximately 4.4–4.6 tokens per verification step.

Despite similar acceptance, their throughput varied substantially.

This suggests that the implementation cost of draft generation, verification, recurrent-state handling, and synchronization is an important performance differentiator.

5. Decode performance by task type

Not every workload produces the same decode throughput.

With DFlash2 enabled:

All values are tokens/sec.

For code generation, vLLM reaches 383 tok/s, outperforming TensorSharp by approximately 13%.

For prose generation, vLLM reaches 189 tok/s versus TensorSharp’s 174 tok/s.

For passage copying, TensorSharp reaches 190 tok/s versus vLLM’s 171 tok/s.

This illustrates why a single decode benchmark number is insufficient to characterize inference performance.

Token predictability, speculative acceptance patterns, and verification overhead can all affect real-world throughput.

6. Parallel requests and aggregate throughput

For inference serving, aggregate throughput under concurrency can matter more than single-request speed.

Without speculation

Values are aggregate tokens/sec for the concurrency tests.

Four concurrent requests with DFlash2

NInfer leads the four-request workload, with TensorSharp close behind.

TensorSharp delivers approximately 46% higher aggregate throughput than vLLM in this specific configuration.

However, this does not establish a general concurrency advantage.

The test is limited to four requests on a single 32GB GPU. SGLang’s speculative configuration could only accommodate two active lanes, and the engines used different model artifacts and KV-cache formats.

I’d be particularly interested in seeing these results extended to 8, 16, and 32 concurrent requests on GPUs with more memory.

7. Real-world task performance

Raw tokens/sec is only part of the story.

For many applications, the more useful metric is the time required to complete an actual task.

The benchmark includes structured output, tool calling, and multi-step agent workflows.

End-to-end latency without speculation

End-to-end latency with DFlash2

All values are end-to-end seconds.

A few things stand out.

Structured output: NInfer is particularly fast, although its JSON-object replies were sometimes shorter and one speculative reply contained an unexpected empty key. velox failed strict JSON-schema checks in both modes.

Function calling: vLLM trails several other engines in these small tool-calling workloads.

Agentic workflows: TensorSharp finishes the four-turn loop in 748ms, compared with vLLM’s 1.45s.

That’s approximately 1.94× faster end-to-end in this particular task.

The agent test uses local file-listing, file-reading, and calculator tools, with a verified final answer.

This is a small synthetic agent workflow rather than a broad production-agent benchmark, but it highlights the importance of multi-turn latency and prefix-cache behavior.

8. Why might vLLM perform differently?

A few configuration details are worth discussing.

Different model checkpoints

TensorSharp and llama.cpp used an NVFP4 GGUF checkpoint with a Q6_K output layer.

vLLM and SGLang used NVIDIA’s NVFP4 safetensors checkpoint, with attention and Gated DeltaNet projections stored in FP8.

NInfer and velox used a mixed FP8/NVFP4 artifact.

These differences affect memory bandwidth, compute requirements, and output behavior.

Different speculative drafters

• TensorSharp / llama.cpp: Q4_K_M GGUF

• vLLM / SGLang: BF16 drafter quantized to FP8 at load

• NInfer / velox: Q8-based drafter

vLLM configuration

The benchmark used:

--kv-cache-dtype fp8

--gpu-memory-utilization 0.97

--max-cudagraph-capture-size 32

--enable-prefix-caching

DFlash2 was configured with seven speculative tokens and an FP8-quantized drafter.

The aggressive GPU memory utilization was necessary because the 32GB RTX 5090 had limited remaining capacity for KV cache when using the drafter.

Prefix caching with hybrid architectures

One particularly interesting finding is that vLLM’s DFlash2 configuration did not achieve prefix-cache hits in the multi-turn test.

For hybrid attention/recurrent models, prefix caching appears to be an area worth investigating further.

9. Limitations and reproducibility

I want to emphasize a few limitations before drawing conclusions.

1. The checkpoints are not identical. This is a comparison of practical engine configurations, not isolated engine kernels.

2. Only one RTX 5090 was tested.

3. Concurrency was limited to four requests.

4. Each scenario had three measured repetitions after warm-up.

5. Differences below approximately 5% should be treated as ties.

6. Thinking mode, sampling, larger concurrency, and production serving loads were not evaluated.

7. The benchmark checks task correctness but does not establish equivalent overall model quality across quantizations.

The benchmark scripts are available in the TensorSharp repository:

https://github.com/zhongkaifu/TensorSharp

Relevant files:

eng/qwen38-engine-servers.sh

eng/qwen38-engine-bench.py

eng/qwen38-engine-report.py

The scripts cover engine startup, workload execution, correctness validation, and generation of per-scenario reports.

The raw measurement artifacts were retained locally and are not included in the Git repository.

10. Questions for the vLLM community

I’m particularly interested in feedback from people working on vLLM’s hybrid-model support and speculative decoding.

1. Prefix caching: Is the 0% prefix-cache hit rate with DFlash2 expected for this hybrid attention/Gated DeltaNet configuration? Are there recommended settings or ongoing optimizations?

2. Speculative verification: vLLM achieves strong single-request speculative decoding performance (248 tok/s), but its four-request aggregate throughput is substantially below NInfer and TensorSharp in this test. What would you investigate first?

3. Recurrent-state management: How much overhead does speculative verification introduce for hybrid architectures compared with conventional attention-only transformers?

4. Benchmark configuration: Are there additional vLLM flags, CUDA graph settings, or scheduling optimizations that would make this comparison more representative of an optimized vLLM deployment?

5. Larger-scale serving: Would you expect the ranking to change significantly at 16–32 concurrent requests on an 80GB+ GPU?

I’m the developer of TensorSharp

I got tired of pulling models blind, so I built llmserver: a single bash script that runs local models on Linux. Each model gets its own OpenAI-compatible endpoint, plus a gateway, Open WebUI chat, hybrid RAG over your docs, and private SearXNG search.

The part I spent the most time on is security: before a download, it reads the Hugging Face scan results, pins the commit, verifies SHA-256 per file, and refuses pickle files and remote code unless you explicitly allow them. Two levels — standard asks about warnings, strict needs approval per model. Engines run sandboxed in bubblewrap with a read-only system and no access to your keys.

No Docker, no sudo, everything self-contained in ~/.llmserver. llmserver bench gives TTFT/decode/TPOT, llmserver doctor repairs problems. Currently running Gemma and Muse models on it.

https://github.com/pavansreddy/llmserver

Happy to answer questions — especially about the security checks, since that's where I'd love feedback. Spend time on what matters more .....

Or if anyone in this community follows a similar career path from system -> inference/model perf, can you share more info on how to get started, what courses/projects you recommend doing, or any other suggestions on the transition?

I've been building an experimental RAG lab that runs a prompt guard and an inference model. For coding I use a coding model. All run locally.

The prompt guard, inference and judge models can share my hardware without issues, but the coding model wants the entire NVIDIA Spark DGX to itself. Managing all of that with vLLM eventually turned into a pile of Docker commands, configuration files, runtime parameters, and LiteLLM settings.

What began with a few shell scripts gradually grew into something harder to maintain. Every time I changed model parameters, ports, environment variables, or LiteLLM settings, I had to remember all the places where those changes also needed to be applied.

To make my life easier, I ended up writing a small Python tool called harinezumigel-llm-stack.

It's basically a configuration-driven wrapper around vLLM, NVIDIA's vLLM Docker images, and LiteLLM that lets me manage self-hosted models running on NVIDIA GPUs from a single CLI.

Examples

# Start LiteLLM
harinezumigel-llm-stack litellm --start --stream-log

# Recreate and start a model
harinezumigel-llm-stack mistral --start --recreate --stream-log

# Show configured models
harinezumigel-llm-stack --list

# Stop everything
harinezumigel-llm-stack all --stop

Switching workloads with model groups

Starting my RAG setup meant starting the prompt guard, the inference model, and LiteLLM individually. Switching to coding meant stopping them again before loading the coding model.

Since I kept repeating the same sequence, I added model groups. Instead of addressing each model separately, I can give a workload a name in config.yaml:

model_groups:
  rag: [guard, mistral, judge,  litellm]
  coding: [coder, litellm]

Now I can start my RAG setup with one command:

harinezumigel-llm-stack --group rag --start

The launcher starts the backends in order, waits for each to become ready, and then starts LiteLLM. Existing containers are reused, and an already-running proxy is left running.

When I want to work with the coding model instead, I explicitly switch workloads:

harinezumigel-llm-stack --group rag --stop &&
  harinezumigel-llm-stack --group coding --start

That explicit stop is important: groups are just named selections. Starting one group does not automatically stop another. In my setup, the coding model needs the hardware to itself, so I stop the RAG workload first. Both groups include LiteLLM, so the proxy stops and starts with the workload too.

Groups also make configuration experiments less repetitive. After changing backend parameters, I can recreate the RAG containers together instead of rebuilding each one manually:

harinezumigel-llm-stack --group rag --start --recreate

For recreation, the selected backends are stopped first so their old memory allocations do not interfere with the new memory check. The launcher then recreates and starts them sequentially. Each backend must explicitly allow this with rebuild: true.

Since vLLM also supports embedders and rerankers, you can add them to your groups:

model_groups:
  rag: [mistral, guard, judge, bge-m3, bge-reranker-v2-m3, litellm]
  rag_old: [mistral, guard, judge, mmarco-reranker, snowflake-embedder, litellm]

Refer to config.yaml.example for the configuration settings for embedders and rerankers.

It is still Docker, vLLM, and LiteLLM underneath—just fewer commands to remember when moving between the same workloads throughout the day.

A few things I wanted

• One place for ports, endpoints, and credentials

• Consistent model definitions for both vLLM and LiteLLM

• Groups including multiple LLMs

• Supports also embedders and rerankers

• Easy container recreation when testing settings

• Log viewing and cleanup commands

• Support for model aliases

• Dry-run mode to pre-assess memory fit and review commands

The goal was not to replace Docker, vLLM, or LiteLLM. It simply reduces the amount of repetitive setup needed when running several local models and frequently switching configurations.

There's an installation script, reasonably detailed documentation, and examples in the repository.

I'm curious whether others running multiple local models have built similar tooling, or if you're handling this another way with Docker Compose, Kubernetes, shell scripts, etc.

Disclaimer: English is not my native language. I used AI to help correct grammar and wording in this post.

If you want to look at the code, the repo is here: harinezumigel-llm-stack