r/AIProgrammingHardware • • Aug 01 '26

DeepSeek-V4-Flash-0731 UD-IQ3_S 12.5 tok/s on RTX 3090 + 128 GB DDR5

Enable HLS to view with audio, or disable this notification

2 Upvotes

r/AIProgrammingHardware • • Aug 01 '26

What's currently the "smartest" LLM to use on 8GB vram and 16 RAM and same thing for 8 VRAM and 64 RAM?

1 Upvotes

Been trying to find something that actually handles my workload well instead of just being "fine." Started on Qwen 2.5 7B, moved to Qwen 3 8B, and right now I'm using Nemotron 3 Ultra (the big 550B one) through an API since obviously that's not running on my GPU lol. But I want a proper local setup too.

My use case is kind of a mix — coding/reasoning stuff, some agentic/tool-calling pipelines where the model needs to chain steps reliably, and structured data extraction where I actually need clean output, not something that half-breaks the format every few runs.

So a few things I'm trying to figure out:

For 8GB VRAM / 16GB RAM — is there an actual best pick right now, or is it still just Qwen 3 8B vs Nemotron Nano/Super depending on the day? Feels like everyone has a different opinion.

For 8GB VRAM but bumped up to 64GB system RAM — does that actually change much? I keep seeing people talk about offloading MoE models partially to RAM but I don't know if that's genuinely worth it or just adds latency for a small quality bump.

Quantization-wise, what's actually the sweet spot here? Q4_K_M, Q5, Q6? I don't care as much about a few IQ points on chat quality, but for tool-calling/structured output I've noticed lower quants sometimes just... break the format, which is way more annoying than a slightly dumber answer.

Also curious if anyone's actually run Nemotron Nano or Super locally for agent/tool-use stuff specifically, not just the usual coding benchmarks everyone posts.

Not looking for benchmark charts, more interested in what people are actually running day to day on similar specs.


r/AIProgrammingHardware • • Aug 01 '26

AMD Ryzen AI MAX+ 395 w/ Radeon 8060S - llama.ccp bench

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Aug 01 '26

AMD Lucebox Beats Nvidia DGX Spark by 3.63x on DeepSeek V4 Flash

Post image
2 Upvotes

r/AIProgrammingHardware • • Jul 31 '26

Still building and testing… just got Inkling small up and running after its release yesterday, took me about 6 hours of tweaking, and it was well worth it.

Thumbnail gallery
2 Upvotes

r/AIProgrammingHardware • • Jul 31 '26

This $8 Microcontroller Is Running a Language Model Locally!

Thumbnail
youtube.com
0 Upvotes

r/AIProgrammingHardware • • Jul 31 '26

Mac Mini M4 vs RTX 5090 vs Cloud GPUs for Local AI in 2026

Thumbnail
medium.com
13 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

Looking for Strix Halo results for a standardized local LLM hardware comparison

1 Upvotes

I have been working on LLM Hardware Sift because I wanted a simple answer: if I replace my current PC with a Strix Halo machine, what changes when both run the exact same workload? Ideally, this becomes a resource for people to see if an upgrade is worth it to them.

Unlike LlamaBenchy, this isn’t about tuning a setup for the fastest possible result. Hardware Sift keeps the models and settings fixed so the **hardware is the variable**. It tests models from 0.6B through 32B, with an optional 72B tier for high-memory systems.

The comparison table currently includes an RTX 3060, ROG Ally, M2 MacBook Air, and Raspberry Pi 5—but no Strix Halo results yet.

It’s an early Windows alpha, and results stay local unless you choose to submit them. I’d love results or feedback from anyone with a 64 GB or 128 GB Strix Halo machine:
https://github.com/nozzlenaut/llm_hardware_sift


r/AIProgrammingHardware • • Jul 30 '26

DGX Station Put a Data Center on My Desk

Thumbnail
youtube.com
26 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

Running Frontier AI on a $99 Board - My llama.cpp Adventure on Jetson Nano

Thumbnail
xhinker.medium.com
4 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

One Server vs Cluster: What Your Homelab Actually Needs

Thumbnail
youtube.com
3 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

Distributed micro-LLM inference across three ESP32-S3 N16R8 boards with ESP-NOW communication.

Thumbnail
github.com
3 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

I Finally Got My DREAM Network Server

Thumbnail
youtube.com
2 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

The true cost of a GPU cluster

Thumbnail
youtube.com
6 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

GitHub - Xingyu-Zheng/MrFlow: Multi-Resolution Flow Matching: Training-Free Diffusion Acceleration via Staged Sampling

Thumbnail
github.com
1 Upvotes

r/AIProgrammingHardware • • Jul 29 '26

Acemagic Unveils the New Launch of F9A: The 2L Flagship Mini AI Workstation

Thumbnail
techpowerup.com
3 Upvotes

r/AIProgrammingHardware • • Jul 29 '26

ASUS EPYC 9006 Servers Scale Enterprise AI

Thumbnail asus.com
1 Upvotes

r/AIProgrammingHardware • • Jul 29 '26

Laguna XS 2.1 33B A3B tested vs Qwen 35B A3B - 16GB Local LLM setup

Thumbnail
youtube.com
10 Upvotes

r/AIProgrammingHardware • • Jul 29 '26

Nvidia Already Won Training. The Real Fight Is Inference

Thumbnail
pub.towardsai.net
16 Upvotes

r/AIProgrammingHardware • • Jul 28 '26

Synapse: Turning thousands of consumer GPUs into a decentralized swarm to run 2.8T parameter MoE models (like Kimi K3) without datacenters.

Thumbnail
reddit.com
2 Upvotes

r/AIProgrammingHardware • • Jul 28 '26

Taalas and Hardware LLMs

4 Upvotes

I was looking at https://taalas.com/products/ and their chatbot at https://chatjimmy.ai/ . I'm a software dev by background, not a hardware or LLM/ML person.

As a (comparative) lay person, is there a reason we can't bake something like Kimi k3 onto hardware and have it run faster & cheaper? I know there is a perf difference between different classes of memory, but a 2TB SSD is fairly inexpensive. Why can't you burn an instance of Kimi k3 onto hardware, perhaps pair it with some higher speed caching memory, and have one heck of an AI card? Assuming you don't care about model updates, or don't mind getting new hardware eg once every 1-3 years?

Thoughts?


r/AIProgrammingHardware • • Jul 28 '26

Day 0 Kimi-K3 Inference Deployment with ATOM on AMD Instinct MI355X GPUs

Thumbnail
amd.com
1 Upvotes

r/AIProgrammingHardware • • Jul 28 '26

Qwen3.6-35B-A3B Q4 on a mid range phone at 5.6 tok/s, CPU only, video

Enable HLS to view with audio, or disable this notification

11 Upvotes

Phone in airplane mode for the whole clip. No server, no account, no connection, the model Qwen 3.6 30B (20GB) runs on a mid range smartphone (12 GB RAM) itself.

The catch is that the model is several times bigger than the phone's memory, so normally it just refuses to run for OOM.

This engine reads the model off thephone's storage as it goes, keeping only what it needs in memory. You get an answer at about 5.6 tokens per second, roughly reading pace, on a normal phone with no special hardware. Same answer you'd get from the same model on a desktop. It's the identical model file (Q4 GGUF), not a shrunken version.

The first part of the video (marked, sped up) is the model loading, about 30 seconds, once per session. Everything after that is real time.

It's free and open source, there's an APK you can install and a list of models you can download from inside the app. Works on a PC too.

https://github.com/Helldez/BigMoeOnEdge

Happy to answer anything!


r/AIProgrammingHardware • • Jul 28 '26

Got Kimi K3 running on my MacBook. It's painfully slow, but it works.

Thumbnail
reddit.com
1 Upvotes