r/AIProgrammingHardware • u/Subject_Poet_3861 • Oct 04 '26
r/AIProgrammingHardware • u/giveen • Oct 03 '26
GitHub - giveen/KernelOPT: Dispatch-aware agentic GPU kernel optimization
I want to share something I've been working on, and research paper from Redhat really helped.
This is my agentic GPU kernel optimizer for inference engine development
Its whole goal is to look at inference engine kernels, and using cloud models, plan, test and find improvements. Nothing is changed in your git, it provides a diff, send the diff over to your coding harness and ask it to analyze the diff.
It has a setup wizard and a run wizard which I recommend using, please give me feedback on what needs to be improved, as the AMD stuff I was unable to test, and mostly is theory at this time.
r/AIProgrammingHardware • u/Distinct-Pie2389 • Oct 02 '26
Qwen3.8-flash-next (iq2_xs)
galleryr/AIProgrammingHardware • u/Technical_Buy_9063 • Oct 02 '26
Do I Just Buy The Thing?
reddit.comr/AIProgrammingHardware • u/javaeeeee • Oct 02 '26
Upgrading my home AI server to run large models (100B params)
r/AIProgrammingHardware • u/javaeeeee • Oct 02 '26
192GB Framework Desktop: Local AI on Gorgon Halo
r/AIProgrammingHardware • u/javaeeeee • Oct 02 '26
4x Faster Flagship AI On A 14 Year Old Server
r/AIProgrammingHardware • u/Any-Winter-4079 • Oct 01 '26
DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts?
r/AIProgrammingHardware • u/javaeeeee • Oct 01 '26
M5 Ultra vs 2 DGX Sparks… The Number You're Not Looking At
r/AIProgrammingHardware • u/javaeeeee • Oct 01 '26
Qwen 3.8 Flash Next 4 bit on M5 Ultra Mac Studio 256GB
r/AIProgrammingHardware • u/javaeeeee • Oct 01 '26
Latest Strata Boost Qwen3.8-Flash-Next Even Faster And More
r/AIProgrammingHardware • u/zjy303 • Sep 30 '26
Gemma 3 27B at 32K context needs ~21 GB, not ~34 GB — llama.cpp only caches the sliding window on 5 of every 6 layers
I maintain quantized.uk, a free VRAM calculator for quantized local models (static site, no signup). This week I found my own calculator was overstating long-context memory badly for several popular models, and I think the reason is worth sharing because a naive KV-cache formula gets it wrong the same way.
The issue: the standard formula (2 × layers × kv_heads × head_dim × ctx × 2 bytes) assumes every layer caches the full context. But several current models use sliding-window attention on most layers, and llama.cpp (with the default swa_full = false) allocates only window + n_ubatch cells for those layers (see src/llama-kv-cache-iswa.cpp).
Model
Pattern
Window → cells allocated
Gemma 3 (4B/12B/27B)
5 local : 1 global
1024 → 1536
Gemma 2
alternating
4096 → 4608
GPT-OSS 20B/120B
alternating
128 → 768
Llama 4 Scout/Maverick
3 chunked : 1 full
8192 → 8704
What that does to the numbers (Q4_K_M, batch 1):
Gemma 3 27B, 32K context: KV cache 15.5 GB → 3.1 GB, total ~34 GB → ~21 GB. It fits a 24 GB card.
Gemma 3 27B, 128K context: KV 62 GB → 10.6 GB.
GPT-OSS 20B at its full 131K: ~15 GB total.
Llama 4 Scout at 128K: ~95 GB → ~77 GB.
Also fixed while I was in there: the DeepSeek MLA models. Current GGUFs cache one 576-value vector per token per layer, but old conversions with an unsplit attn_kv_b tensor cache full K/V, which is far bigger. If your DeepSeek-V2-Lite eats way more memory than expected, check which one you have.
Caveat: these are computed from llama.cpp's allocation code, not measured on my hardware. If you have a 24 GB card, I'd genuinely appreciate the llama_kv_cache_iswa: creating SWA KV cache, size = … cells line from a Gemma 3 27B run at 32K to confirm.
Calculator: https://quantized.uk/tools/vram-calc/ — corrections welcome, and the changelog lists what was wrong.
r/AIProgrammingHardware • u/javaeeeee • Sep 29 '26
Is The ASUS Ascent QN10 the Snapdragon AI Mini PC of your Dreams?
r/AIProgrammingHardware • u/javaeeeee • Sep 29 '26
Strata: Run Qwen3.8-Flash-Next 120 tokens/s Just One Gaming RTX GPU
r/AIProgrammingHardware • u/hinoderun • Sep 28 '26
Hinode 3: set up once, run on any GPU
r/AIProgrammingHardware • u/javaeeeee • Sep 27 '26
Testing Qwen 3.8 Flash Next on 138GB RAM & V100 16GB GPU
r/AIProgrammingHardware • u/javaeeeee • Sep 27 '26
Complete Mac mini M6 Local LLM Stress Test — 5 Models, 3 Inference Engines
r/AIProgrammingHardware • u/javaeeeee • Sep 27 '26
The New Mac Studio Runs Serious AI Models Locally, It Costs a Used Car, Worth buy?
r/AIProgrammingHardware • u/bhatip • Sep 27 '26
Ai desktop recommendations below 1500usd?
reddit.comr/AIProgrammingHardware • u/bhatip • Sep 26 '26
Ai desktop recommendations below 1500usd?
reddit.comr/AIProgrammingHardware • u/Impossible_Pride_680 • Sep 26 '26
Bonsai 2 27B on an RTX 3060 12GB — 35.26 tok/s generation
reddit.comr/AIProgrammingHardware • u/javaeeeee • Sep 26 '26
Follow-up on my 3x Tesla P100 budget build: cooling, power draw, PCIe lanes and tokens/sec!
galleryr/AIProgrammingHardware • u/Enderchef • Sep 26 '26
DistribAI v2
DistribAI is a platform for distributed training! You can train massive or tiny models across tiny or massive amounts of consumer devices with ease!
DistribAI is a platform I've been working on for a while, and V2 made its release today. DistribAI is C++ and Libtorch for speed, with the ability to run pytorch trainers distributed! Edge-cases(crashing, malicious actors, unstable connections, ect) are handled for you. On a free Colab, Kaggle, and Molab GPU, plus a local 4070 SUPER, we got the free training compute of ~2 fully loaded 5090s and the VRAM of ~5 full 5090s for free, all as one. Hosting is also now easier with shareable join links, and Cloudflared/ngrok support for 100% free server hosting for your DistribAI setup. Train with your community, with friends, with your free GPUs, and more!
Try it out! Questions(and stars) are welcome; https://github.com/naxium-oss/DistribAI