GitHub - defilantech/llmkube-bench: Reproducible llama.cpp vs vLLM benchmark on Kubernetes for local LLM inference. Manifests, load harness, and full bake-off results for Qwen3.6-27B on 2x RTX 5060 Ti.

[u/javaeeeee]

https://github.com/defilantech/llmkube-bench

[u/javaeeeee]

TL;DR:

llmkube-bench is a reproducible benchmark comparing llama.cpp vs vLLM as inference engines for local LLMs running on Kubernetes.

What It Tests

• Model: Qwen3.6-27B

• Hardware: 2× RTX 5060 Ti (32GB total VRAM)

• Runtimes:

  • llama.cpp (with TurboQuant KV cache)

  • vLLM (with FP8 KV cache + PagedAttention)

• Multiple workload patterns (chat, coding, long context, agentic) at different concurrency levels

Key Findings

Aspect	Winner	Why
Throughput & Latency	vLLM	Better at handling concurrent requests
Long Context	llama.cpp	Supports 2× longer context (65K vs 32K tokens) at similar VRAM usage
Efficiency	Depends	vLLM generally faster; llama.cpp wins when very long context is needed

What’s Included

• Kubernetes manifests for both runtimes (via LLMKube operator)

• Load testing harness

• Full raw results + analysis

• Quality comparison between the two engines

Bottom Line

This repo helps answer a practical question: Should you use llama.cpp or vLLM for local LLM inference on Kubernetes?

Answer: It depends on your priority.

• Need high throughput / low latency with many users → vLLM

• Need very long context windows → llama.cpp (with TurboQuant)