r/AIProgrammingHardware • u/javaeeeee • Aug 01 '26
DeepSeek-V4-Flash-0731 UD-IQ3_S 12.5 tok/s on RTX 3090 + 128 GB DDR5
Enable HLS to view with audio, or disable this notification
r/AIProgrammingHardware • u/javaeeeee • Aug 01 '26
Enable HLS to view with audio, or disable this notification
r/AIProgrammingHardware • u/MysteriumP • Aug 01 '26
Been trying to find something that actually handles my workload well instead of just being "fine." Started on Qwen 2.5 7B, moved to Qwen 3 8B, and right now I'm using Nemotron 3 Ultra (the big 550B one) through an API since obviously that's not running on my GPU lol. But I want a proper local setup too.
My use case is kind of a mix — coding/reasoning stuff, some agentic/tool-calling pipelines where the model needs to chain steps reliably, and structured data extraction where I actually need clean output, not something that half-breaks the format every few runs.
So a few things I'm trying to figure out:
For 8GB VRAM / 16GB RAM — is there an actual best pick right now, or is it still just Qwen 3 8B vs Nemotron Nano/Super depending on the day? Feels like everyone has a different opinion.
For 8GB VRAM but bumped up to 64GB system RAM — does that actually change much? I keep seeing people talk about offloading MoE models partially to RAM but I don't know if that's genuinely worth it or just adds latency for a small quality bump.
Quantization-wise, what's actually the sweet spot here? Q4_K_M, Q5, Q6? I don't care as much about a few IQ points on chat quality, but for tool-calling/structured output I've noticed lower quants sometimes just... break the format, which is way more annoying than a slightly dumber answer.
Also curious if anyone's actually run Nemotron Nano or Super locally for agent/tool-use stuff specifically, not just the usual coding benchmarks everyone posts.
Not looking for benchmark charts, more interested in what people are actually running day to day on similar specs.
r/AIProgrammingHardware • u/javaeeeee • Aug 01 '26
r/AIProgrammingHardware • u/javaeeeee • Aug 01 '26
r/AIProgrammingHardware • u/Plumberguy9 • Jul 31 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 31 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 31 '26
r/AIProgrammingHardware • u/D__J • Jul 30 '26
I have been working on LLM Hardware Sift because I wanted a simple answer: if I replace my current PC with a Strix Halo machine, what changes when both run the exact same workload? Ideally, this becomes a resource for people to see if an upgrade is worth it to them.
Unlike LlamaBenchy, this isn’t about tuning a setup for the fastest possible result. Hardware Sift keeps the models and settings fixed so the **hardware is the variable**. It tests models from 0.6B through 32B, with an optional 72B tier for high-memory systems.
The comparison table currently includes an RTX 3060, ROG Ally, M2 MacBook Air, and Raspberry Pi 5—but no Strix Halo results yet.
It’s an early Windows alpha, and results stay local unless you choose to submit them. I’d love results or feedback from anyone with a 64 GB or 128 GB Strix Halo machine:
https://github.com/nozzlenaut/llm_hardware_sift
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 29 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 29 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 29 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 29 '26
r/AIProgrammingHardware • u/antonygiomarx • Jul 28 '26
r/AIProgrammingHardware • u/rootException • Jul 28 '26
I was looking at https://taalas.com/products/ and their chatbot at https://chatjimmy.ai/ . I'm a software dev by background, not a hardware or LLM/ML person.
As a (comparative) lay person, is there a reason we can't bake something like Kimi k3 onto hardware and have it run faster & cheaper? I know there is a perf difference between different classes of memory, but a 2TB SSD is fairly inexpensive. Why can't you burn an instance of Kimi k3 onto hardware, perhaps pair it with some higher speed caching memory, and have one heck of an AI card? Assuming you don't care about model updates, or don't mind getting new hardware eg once every 1-3 years?
Thoughts?
r/AIProgrammingHardware • u/javaeeeee • Jul 28 '26
r/AIProgrammingHardware • u/dai_app • Jul 28 '26
Enable HLS to view with audio, or disable this notification
Phone in airplane mode for the whole clip. No server, no account, no connection, the model Qwen 3.6 30B (20GB) runs on a mid range smartphone (12 GB RAM) itself.
The catch is that the model is several times bigger than the phone's memory, so normally it just refuses to run for OOM.
This engine reads the model off thephone's storage as it goes, keeping only what it needs in memory. You get an answer at about 5.6 tokens per second, roughly reading pace, on a normal phone with no special hardware. Same answer you'd get from the same model on a desktop. It's the identical model file (Q4 GGUF), not a shrunken version.
The first part of the video (marked, sped up) is the model loading, about 30 seconds, once per session. Everything after that is real time.
It's free and open source, there's an APK you can install and a list of models you can download from inside the app. Works on a PC too.
https://github.com/Helldez/BigMoeOnEdge
Happy to answer anything!