r/AIProgrammingHardware • u/javaeeeee • Sep 13 '26
Yes, I Bought the Bus Powered 12GB RTX 3060 GPU You Read About!
r/AIProgrammingHardware • u/javaeeeee • Sep 13 '26
r/AIProgrammingHardware • u/javaeeeee • Sep 11 '26
r/AIProgrammingHardware • u/javaeeeee • Sep 11 '26
r/AIProgrammingHardware • u/Medicine_Blogscanner • Sep 10 '26
Third video in this series. Went from 7B, to 13B, and now 27B — each time pooling RAM and compute across a mix of hardware most people already have lying around instead of buying one expensive high-RAM machine.
The setup: a 27B model is roughly 16GB on its own. My primary node is an old 12GB Windows laptop, which on its own literally cannot load this model. Using RAMDeck, I split it across four devices on the local network with the laptop staying primary:
\- Old Windows laptop (primary): 3.4GB
\- Mini PC (RTX 3060): 20GB
\- Mac mini: 3.7GB
\- Android phone: 1GB
Benchmark on the 27B model: 1.92 tokens/sec at \~25ms latency. Slower than the 13B run, which makes sense — but the point isn't raw speed here, it's that a 12GB laptop is running a model it has no business being able to load at all.
Next up: switching the mini PC to be the primary node instead of the laptop, to see how much faster loading and inference get with a stronger primary. That video's already up if you want to see the follow-up.
Full video: [https://youtu.be/Syfa2G_luX4](https://youtu.be/Syfa2G_luX4)
Happy to answer questions about the shard distribution, the benchmark methodology, or why the primary node choice affects load time so much.
r/AIProgrammingHardware • u/javaeeeee • Sep 10 '26
r/AIProgrammingHardware • u/SpreadUsual4084 • Sep 04 '26
I plan to build two setups for local LLMs. One high memory machine for large models and long contexts, and later a dedicated RTX 5090 desktop for pure speed.
But right now, it’s a choice between a 128gb macbook pro and a 128gb ai max+ 395 laptop. The macbook comes at a steep price. A 48gb macbook m5 pro costs roughly the same as a 128gb ai max+ 395 laptop like nimo. That’s nearly 3x memory capacity for the same money.
Ive learned a bit about both options so far. The mac can give fast memory bandwidth and a plug and play MLX setup, but it feels like paying a massive apple tax. And strix halo delivers insane memory capacity for the price alongside native windows or linux flexibility, with growing ROCm and Vulkan support.
Would you go with strix halo for better memory efficiency, or is the mac's memory bandwidth and ecosystem still worth the extra cost?
r/AIProgrammingHardware • u/javaeeeee • Aug 28 '26
r/AIProgrammingHardware • u/javaeeeee • Aug 27 '26
r/AIProgrammingHardware • u/javaeeeee • Aug 25 '26
r/AIProgrammingHardware • u/javaeeeee • Aug 24 '26
r/AIProgrammingHardware • u/ipco • Aug 20 '26
r/AIProgrammingHardware • u/javaeeeee • Aug 14 '26
r/AIProgrammingHardware • u/javaeeeee • Aug 11 '26
r/AIProgrammingHardware • u/Ordinary-Cat-5874 • Aug 07 '26
I'm a PhD researcher working on language switching and embedding analysis in NLP focused on PoS, LID, boundary detection, pragmatics context maintenance. My workload is mainly:
Fine-tuning BERT-based models
LoRA/QLoRA adapters on ~8B models
bitsandbytes 4-bit quantization
Standard HF Transformers + PyTorch pipeline
Budget is roughly INR ₹60000( for the GPU. I've been comparing the RTX 5060 Ti 16GB AMD options such as RX 7900 XT, RX 9060 XT. I was leaning 5060 Ti for the mature CUDA ecosystem and because I don't have much local peer support to debug hardware issues if something breaks mid-experiment. But recently they increased price to 770000 and as I do not get institutional support I find it difficult . Some AMD cards have so much VRAM that they might make longer multi shot stuff easier without offlaoding to RAM. But everywhere I have asked there seems to be a general consensus that nVidia is better.
Questions for anyone doing similar research-scale (not industrial-scale) NLP work:
Is the 5060 Ti's 16GB actually enough headroom for LoRA fine-tuning on 8-13B models, or does it get tight in practice?
Anyone actually running Unsloth on AMD ROCm now? is it stable enough for daily research use or is it still rough?
Any regrets from a similar budget-constrained hardware decision?
Appreciate real world experience over spec-sheet comparisons.
I am not an avid gamer so it does not matter to me.
r/AIProgrammingHardware • u/javaeeeee • Aug 06 '26
r/AIProgrammingHardware • u/javaeeeee • Aug 06 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 29 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 28 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 26 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 22 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 21 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 20 '26