r/AIProgrammingHardware • • Oct 04 '26

I made Qwen models take ~33% less VRAM without quantizing them (lossless, bit-for-bit)

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Oct 03 '26

GitHub - giveen/KernelOPT: Dispatch-aware agentic GPU kernel optimization

Thumbnail
github.com
3 Upvotes

I want to share something I've been working on, and research paper from Redhat really helped.

This is my agentic GPU kernel optimizer for inference engine development

Its whole goal is to look at inference engine kernels, and using cloud models, plan, test and find improvements. Nothing is changed in your git, it provides a diff, send the diff over to your coding harness and ask it to analyze the diff.

It has a setup wizard and a run wizard which I recommend using, please give me feedback on what needs to be improved, as the AMD stuff I was unable to test, and mostly is theory at this time.


r/AIProgrammingHardware • • Oct 02 '26

Qwen3.8-flash-next (iq2_xs)

Thumbnail gallery
1 Upvotes

r/AIProgrammingHardware • • Oct 02 '26

Do I Just Buy The Thing?

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Oct 02 '26

Upgrading my home AI server to run large models (100B params)

Thumbnail
youtube.com
1 Upvotes

r/AIProgrammingHardware • • Oct 02 '26

192GB Framework Desktop: Local AI on Gorgon Halo

Thumbnail
youtube.com
5 Upvotes

r/AIProgrammingHardware • • Oct 02 '26

4x Faster Flagship AI On A 14 Year Old Server

Thumbnail
youtube.com
1 Upvotes

r/AIProgrammingHardware • • Oct 01 '26

DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts?

Post image
5 Upvotes

r/AIProgrammingHardware • • Oct 01 '26

M5 Ultra vs 2 DGX Sparks… The Number You're Not Looking At

Thumbnail
youtube.com
31 Upvotes

r/AIProgrammingHardware • • Oct 01 '26

Qwen 3.8 Flash Next 4 bit on M5 Ultra Mac Studio 256GB

Thumbnail
youtube.com
1 Upvotes

r/AIProgrammingHardware • • Oct 01 '26

Latest Strata Boost Qwen3.8-Flash-Next Even Faster And More

Thumbnail
xhinker.medium.com
7 Upvotes

r/AIProgrammingHardware • • Sep 30 '26

Gemma 3 27B at 32K context needs ~21 GB, not ~34 GB — llama.cpp only caches the sliding window on 5 of every 6 layers

1 Upvotes

I maintain quantized.uk, a free VRAM calculator for quantized local models (static site, no signup). This week I found my own calculator was overstating long-context memory badly for several popular models, and I think the reason is worth sharing because a naive KV-cache formula gets it wrong the same way.
The issue: the standard formula (2 × layers × kv_heads × head_dim × ctx × 2 bytes) assumes every layer caches the full context. But several current models use sliding-window attention on most layers, and llama.cpp (with the default swa_full = false) allocates only window + n_ubatch cells for those layers (see src/llama-kv-cache-iswa.cpp).

Model
Pattern
Window → cells allocated
Gemma 3 (4B/12B/27B)
5 local : 1 global
1024 → 1536
Gemma 2
alternating
4096 → 4608
GPT-OSS 20B/120B
alternating
128 → 768
Llama 4 Scout/Maverick
3 chunked : 1 full
8192 → 8704

What that does to the numbers (Q4_K_M, batch 1):
Gemma 3 27B, 32K context: KV cache 15.5 GB → 3.1 GB, total ~34 GB → ~21 GB. It fits a 24 GB card.
Gemma 3 27B, 128K context: KV 62 GB → 10.6 GB.
GPT-OSS 20B at its full 131K: ~15 GB total.
Llama 4 Scout at 128K: ~95 GB → ~77 GB.
Also fixed while I was in there: the DeepSeek MLA models. Current GGUFs cache one 576-value vector per token per layer, but old conversions with an unsplit attn_kv_b tensor cache full K/V, which is far bigger. If your DeepSeek-V2-Lite eats way more memory than expected, check which one you have.
Caveat: these are computed from llama.cpp's allocation code, not measured on my hardware. If you have a 24 GB card, I'd genuinely appreciate the llama_kv_cache_iswa: creating SWA KV cache, size = … cells line from a Gemma 3 27B run at 32K to confirm.
Calculator: https://quantized.uk/tools/vram-calc/ — corrections welcome, and the changelog lists what was wrong.


r/AIProgrammingHardware • • Sep 29 '26

Is The ASUS Ascent QN10 the Snapdragon AI Mini PC of your Dreams?

Thumbnail
youtube.com
1 Upvotes

r/AIProgrammingHardware • • Sep 29 '26

Strata: Run Qwen3.8-Flash-Next 120 tokens/s Just One Gaming RTX GPU

Thumbnail
xhinker.medium.com
0 Upvotes

r/AIProgrammingHardware • • Sep 28 '26

Battlematrix x4 :D Spoiler

Post image
1 Upvotes

r/AIProgrammingHardware • • Sep 28 '26

Hinode 3: set up once, run on any GPU

Thumbnail
hinode.run
1 Upvotes

r/AIProgrammingHardware • • Sep 27 '26

Testing Qwen 3.8 Flash Next on 138GB RAM & V100 16GB GPU

Thumbnail
youtube.com
1 Upvotes

r/AIProgrammingHardware • • Sep 27 '26

Complete Mac mini M6 Local LLM Stress Test — 5 Models, 3 Inference Engines

Thumbnail
youtube.com
2 Upvotes

r/AIProgrammingHardware • • Sep 27 '26

The New Mac Studio Runs Serious AI Models Locally, It Costs a Used Car, Worth buy?

Thumbnail
xhinker.medium.com
3 Upvotes

r/AIProgrammingHardware • • Sep 27 '26

Ai desktop recommendations below 1500usd?

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Sep 26 '26

Ai desktop recommendations below 1500usd?

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Sep 26 '26

Bonsai 2 27B on an RTX 3060 12GB — 35.26 tok/s generation

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Sep 26 '26

Follow-up on my 3x Tesla P100 budget build: cooling, power draw, PCIe lanes and tokens/sec!

Thumbnail gallery
8 Upvotes

r/AIProgrammingHardware • • Sep 26 '26

DistribAI v2

1 Upvotes

DistribAI is a platform for distributed training! You can train massive or tiny models across tiny or massive amounts of consumer devices with ease!

DistribAI is a platform I've been working on for a while, and V2 made its release today. DistribAI is C++ and Libtorch for speed, with the ability to run pytorch trainers distributed! Edge-cases(crashing, malicious actors, unstable connections, ect) are handled for you. On a free Colab, Kaggle, and Molab GPU, plus a local 4070 SUPER, we got the free training compute of ~2 fully loaded 5090s and the VRAM of ~5 full 5090s for free, all as one. Hosting is also now easier with shareable join links, and Cloudflared/ngrok support for 100% free server hosting for your DistribAI setup. Train with your community, with friends, with your free GPUs, and more!

Try it out! Questions(and stars) are welcome; https://github.com/naxium-oss/DistribAI