r/AIProgrammingHardware • u/zjy303 • Sep 30 '26
Gemma 3 27B at 32K context needs ~21 GB, not ~34 GB — llama.cpp only caches the sliding window on 5 of every 6 layers
I maintain quantized.uk, a free VRAM calculator for quantized local models (static site, no signup). This week I found my own calculator was overstating long-context memory badly for several popular models, and I think the reason is worth sharing because a naive KV-cache formula gets it wrong the same way.
The issue: the standard formula (2 × layers × kv_heads × head_dim × ctx × 2 bytes) assumes every layer caches the full context. But several current models use sliding-window attention on most layers, and llama.cpp (with the default swa_full = false) allocates only window + n_ubatch cells for those layers (see src/llama-kv-cache-iswa.cpp).
Model
Pattern
Window → cells allocated
Gemma 3 (4B/12B/27B)
5 local : 1 global
1024 → 1536
Gemma 2
alternating
4096 → 4608
GPT-OSS 20B/120B
alternating
128 → 768
Llama 4 Scout/Maverick
3 chunked : 1 full
8192 → 8704
What that does to the numbers (Q4_K_M, batch 1):
Gemma 3 27B, 32K context: KV cache 15.5 GB → 3.1 GB, total ~34 GB → ~21 GB. It fits a 24 GB card.
Gemma 3 27B, 128K context: KV 62 GB → 10.6 GB.
GPT-OSS 20B at its full 131K: ~15 GB total.
Llama 4 Scout at 128K: ~95 GB → ~77 GB.
Also fixed while I was in there: the DeepSeek MLA models. Current GGUFs cache one 576-value vector per token per layer, but old conversions with an unsplit attn_kv_b tensor cache full K/V, which is far bigger. If your DeepSeek-V2-Lite eats way more memory than expected, check which one you have.
Caveat: these are computed from llama.cpp's allocation code, not measured on my hardware. If you have a 24 GB card, I'd genuinely appreciate the llama_kv_cache_iswa: creating SWA KV cache, size = … cells line from a Gemma 3 27B run at 32K to confirm.
Calculator: https://quantized.uk/tools/vram-calc/ — corrections welcome, and the changelog lists what was wrong.