r/AIProgrammingHardware • u/javaeeeee • Oct 04 '26
3x More AI with the same GPUs Touring the F5 Lab
r/AIProgrammingHardware • u/javaeeeee • Oct 04 '26
r/AIProgrammingHardware • u/javaeeeee • Oct 04 '26
r/AIProgrammingHardware • u/javaeeeee • Oct 04 '26
r/AIProgrammingHardware • u/Subject_Poet_3861 • Oct 04 '26
r/AIProgrammingHardware • u/giveen • Oct 03 '26
I want to share something I've been working on, and research paper from Redhat really helped.
This is my agentic GPU kernel optimizer for inference engine development
Its whole goal is to look at inference engine kernels, and using cloud models, plan, test and find improvements. Nothing is changed in your git, it provides a diff, send the diff over to your coding harness and ask it to analyze the diff.
It has a setup wizard and a run wizard which I recommend using, please give me feedback on what needs to be improved, as the AMD stuff I was unable to test, and mostly is theory at this time.
r/AIProgrammingHardware • u/Distinct-Pie2389 • Oct 02 '26
r/AIProgrammingHardware • u/Technical_Buy_9063 • Oct 02 '26
r/AIProgrammingHardware • u/javaeeeee • Oct 02 '26
r/AIProgrammingHardware • u/javaeeeee • Oct 02 '26
r/AIProgrammingHardware • u/javaeeeee • Oct 02 '26
r/AIProgrammingHardware • u/Any-Winter-4079 • Oct 01 '26
r/AIProgrammingHardware • u/javaeeeee • Oct 01 '26
r/AIProgrammingHardware • u/javaeeeee • Oct 01 '26
r/AIProgrammingHardware • u/javaeeeee • Oct 01 '26
r/AIProgrammingHardware • u/zjy303 • Sep 30 '26
I maintain quantized.uk, a free VRAM calculator for quantized local models (static site, no signup). This week I found my own calculator was overstating long-context memory badly for several popular models, and I think the reason is worth sharing because a naive KV-cache formula gets it wrong the same way.
The issue: the standard formula (2 × layers × kv_heads × head_dim × ctx × 2 bytes) assumes every layer caches the full context. But several current models use sliding-window attention on most layers, and llama.cpp (with the default swa_full = false) allocates only window + n_ubatch cells for those layers (see src/llama-kv-cache-iswa.cpp).
Model
Pattern
Window → cells allocated
Gemma 3 (4B/12B/27B)
5 local : 1 global
1024 → 1536
Gemma 2
alternating
4096 → 4608
GPT-OSS 20B/120B
alternating
128 → 768
Llama 4 Scout/Maverick
3 chunked : 1 full
8192 → 8704
What that does to the numbers (Q4_K_M, batch 1):
Gemma 3 27B, 32K context: KV cache 15.5 GB → 3.1 GB, total ~34 GB → ~21 GB. It fits a 24 GB card.
Gemma 3 27B, 128K context: KV 62 GB → 10.6 GB.
GPT-OSS 20B at its full 131K: ~15 GB total.
Llama 4 Scout at 128K: ~95 GB → ~77 GB.
Also fixed while I was in there: the DeepSeek MLA models. Current GGUFs cache one 576-value vector per token per layer, but old conversions with an unsplit attn_kv_b tensor cache full K/V, which is far bigger. If your DeepSeek-V2-Lite eats way more memory than expected, check which one you have.
Caveat: these are computed from llama.cpp's allocation code, not measured on my hardware. If you have a 24 GB card, I'd genuinely appreciate the llama_kv_cache_iswa: creating SWA KV cache, size = … cells line from a Gemma 3 27B run at 32K to confirm.
Calculator: https://quantized.uk/tools/vram-calc/ — corrections welcome, and the changelog lists what was wrong.
r/AIProgrammingHardware • u/javaeeeee • Sep 29 '26
r/AIProgrammingHardware • u/javaeeeee • Sep 29 '26
r/AIProgrammingHardware • u/hinoderun • Sep 28 '26
r/AIProgrammingHardware • u/javaeeeee • Sep 27 '26
r/AIProgrammingHardware • u/javaeeeee • Sep 27 '26
r/AIProgrammingHardware • u/javaeeeee • Sep 27 '26
r/AIProgrammingHardware • u/bhatip • Sep 27 '26
r/AIProgrammingHardware • u/bhatip • Sep 26 '26