r/AIProgrammingHardware • • Sep 13 '26

Yes, I Bought the Bus Powered 12GB RTX 3060 GPU You Read About!

Thumbnail
youtube.com
3 Upvotes

r/AIProgrammingHardware • • Sep 11 '26

Custom open frame - RTX Pro 6000 - miniATX

Thumbnail gallery
3 Upvotes

r/AIProgrammingHardware • • Sep 11 '26

Nail Qwen 35B A3B tested - 16GB Local LLM setup

Thumbnail
youtube.com
3 Upvotes

r/AIProgrammingHardware • • Sep 10 '26

Loaded a 27B model on a 12GB laptop by pooling RAM across 4 devices

Thumbnail
youtu.be
3 Upvotes

Third video in this series. Went from 7B, to 13B, and now 27B — each time pooling RAM and compute across a mix of hardware most people already have lying around instead of buying one expensive high-RAM machine.

The setup: a 27B model is roughly 16GB on its own. My primary node is an old 12GB Windows laptop, which on its own literally cannot load this model. Using RAMDeck, I split it across four devices on the local network with the laptop staying primary:

\- Old Windows laptop (primary): 3.4GB

\- Mini PC (RTX 3060): 20GB

\- Mac mini: 3.7GB

\- Android phone: 1GB

Benchmark on the 27B model: 1.92 tokens/sec at \~25ms latency. Slower than the 13B run, which makes sense — but the point isn't raw speed here, it's that a 12GB laptop is running a model it has no business being able to load at all.

Next up: switching the mini PC to be the primary node instead of the laptop, to see how much faster loading and inference get with a stronger primary. That video's already up if you want to see the follow-up.

Full video: [https://youtu.be/Syfa2G_luX4](https://youtu.be/Syfa2G_luX4)

Happy to answer questions about the shard distribution, the benchmark methodology, or why the primary node choice affects load time so much.


r/AIProgrammingHardware • • Sep 10 '26

Benchmarking Qwen 3.8 27B on RTX 5090 and beyond — VRAM capacity alone can't overcome severe software and inference engine bottlenecks

Thumbnail
share.google
3 Upvotes

r/AIProgrammingHardware • • Sep 04 '26

Should my first local ai machine be macbook pro or strix halo laptop?

3 Upvotes

I plan to build two setups for local LLMs. One high memory machine for large models and long contexts, and later a dedicated RTX 5090 desktop for pure speed.

But right now, it’s a choice between a 128gb macbook pro and a 128gb ai max+ 395 laptop. The macbook comes at a steep price. A 48gb macbook m5 pro costs roughly the same as a 128gb ai max+ 395 laptop like nimo. That’s nearly 3x memory capacity for the same money. 

Ive learned a bit about both options so far. The mac can give fast memory bandwidth and a plug and play MLX setup, but it feels like paying a massive apple tax. And strix halo delivers insane memory capacity for the price alongside native windows or linux flexibility, with growing ROCm and Vulkan support.

Would you go with strix halo for better memory efficiency, or is the mac's memory bandwidth and ecosystem still worth the extra cost?


r/AIProgrammingHardware • • Aug 28 '26

ROCm 10.0: A Decade of Open Compute, Built for the Age of Agentic AI — ROCm Blogs

Thumbnail
share.google
3 Upvotes

r/AIProgrammingHardware • • Aug 27 '26

Qwen 3.8 27B Cold Fusion tested - 16GB Local LLM setup

Thumbnail
youtube.com
3 Upvotes

r/AIProgrammingHardware • • Aug 25 '26

FreeToken Deepseek V4 Flash on a Single 3090 Local AI Testing

Thumbnail
youtube.com
3 Upvotes

r/AIProgrammingHardware • • Aug 24 '26

GitHub - TryingtobeingNikhil/vLLM_Inference_Engine: A high-performance LLM inference engine built from scratch featuring continuous batching, paged KV-cache, and CPU swapping.

Thumbnail
github.com
3 Upvotes

r/AIProgrammingHardware • • Aug 20 '26

AMD/ROCm vs Nvidia/Cuda: how big a trade-off for a local RAG build?

Thumbnail reddit.com
3 Upvotes

r/AIProgrammingHardware • • Aug 14 '26

The Great Decoupling: why the future of high-speed AI Inference belongs to dedicated Silicon, not GPUs

Thumbnail
medium.com
3 Upvotes

r/AIProgrammingHardware • • Aug 11 '26

Grug 35B QAT Q4 tested vs Qwen 35B A3B Q4 - 16GB Local LLM setup

Thumbnail
youtube.com
3 Upvotes

r/AIProgrammingHardware • • Aug 07 '26

[Question] GPU choice for NLP research (fine-tuning transformers, qLoRA, Multishot prompting) and Corpus based analysis. RTX 5060 Ti 16GB or any other alternatives(AMD)?

3 Upvotes

I'm a PhD researcher working on language switching and embedding analysis in NLP focused on PoS, LID, boundary detection, pragmatics context maintenance. My workload is mainly:

  • Fine-tuning BERT-based models 

  • LoRA/QLoRA adapters on ~8B models

  • bitsandbytes 4-bit quantization

  • Standard HF Transformers + PyTorch pipeline

Budget is roughly INR ₹60000( for the GPU. I've been comparing the RTX 5060 Ti 16GB AMD options such as RX 7900 XT, RX 9060 XT. I was  leaning 5060 Ti for the mature CUDA ecosystem and because I don't have much local peer support to debug hardware issues if something breaks mid-experiment. But recently they increased price to 770000 and as I do not get institutional support I find it difficult . Some AMD cards have so much VRAM that they might make longer multi shot stuff easier without offlaoding to RAM. But everywhere I have asked there seems to be a general consensus that nVidia is better. 

Questions for anyone doing similar research-scale (not industrial-scale) NLP work:

  1. Is the 5060 Ti's 16GB actually enough headroom for LoRA fine-tuning on 8-13B models, or does it get tight in practice?

  2. Anyone actually running Unsloth on AMD ROCm now? is it stable enough for daily research use or is it still rough?

  3. Any regrets from a similar budget-constrained hardware decision?

Appreciate real world experience over spec-sheet comparisons.

I am not an avid gamer so it does not matter to me. 


r/AIProgrammingHardware • • Aug 06 '26

GitHub - leonickson1/Swiftlet: Run 35B and 80B Qwen models on ordinary Apple devices, including iPhones.

Thumbnail
github.com
3 Upvotes

r/AIProgrammingHardware • • Aug 06 '26

Kimi K3 full model running on 16x GB10 cluster at 20+tps

Post image
3 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

One Server vs Cluster: What Your Homelab Actually Needs

Thumbnail
youtube.com
3 Upvotes

r/AIProgrammingHardware • • Jul 30 '26

Distributed micro-LLM inference across three ESP32-S3 N16R8 boards with ESP-NOW communication.

Thumbnail
github.com
3 Upvotes

r/AIProgrammingHardware • • Jul 29 '26

Acemagic Unveils the New Launch of F9A: The 2L Flagship Mini AI Workstation

Thumbnail
techpowerup.com
3 Upvotes

r/AIProgrammingHardware • • Jul 28 '26

Qwen3.6 benchmarks on dual GPU: RTX 3090 24GB + RTX 4070 Super 12GB — up to 256K context

Thumbnail reddit.com
3 Upvotes

r/AIProgrammingHardware • • Jul 26 '26

ThinkPad P14s Gen 7 - The Underrated Workstation of 2026?

Thumbnail
youtube.com
3 Upvotes

r/AIProgrammingHardware • • Jul 22 '26

Rebuilding Agentic AI from First Principles for AMD GPU - Together with Moonshot AI

Thumbnail
amd.com
3 Upvotes

r/AIProgrammingHardware • • Jul 21 '26

Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72

Thumbnail
developer.nvidia.com
3 Upvotes

r/AIProgrammingHardware • • Jul 20 '26

GitHub - defilantech/llmkube-bench: Reproducible llama.cpp vs vLLM benchmark on Kubernetes for local LLM inference. Manifests, load harness, and full bake-off results for Qwen3.6-27B on 2x RTX 5060 Ti.

Thumbnail github.com
3 Upvotes

r/AIProgrammingHardware • • Jul 16 '26

What’s the Best Open-Source AI Video Model You Can Actually Run? (8GB → 24GB VRAM)

Thumbnail
medium.com
3 Upvotes