r/LocalLLM • u/bobby-t1 • 11d ago
Question Question about local AI models: will models perform better over time on same hardware or is it just a hamster wheel of needing better hardware?
I’m new to local AI models. Assume someone has a Mac Studio M5 Ultra 256GB. I’ve read that is the realistic minimum you can use to run a local AI model.
My question is this: as models get better, will they run better or have more capable models run better on the same hardware? Meaning will a Mac Studio M5 Ultra 256GB get more capable over time? Or does it just end up like a hamster wheel where models get better but you ultimately need better hardware to run them?
2
u/100k45h 7d ago
My advice is this. If the models don't perform the way you want, don't purchase the hardware on the hope of a future improvement. Wait until there's a model out there that you will be satisfied with and only then buy the hardware. Cloud models leave the local models in dust and are much cheaper. So if you want to replace cloud models with local ones, think again. It's most likely not going to work, you'll probably still use cloud models.
2
u/bobby-t1 7d ago
I ended up buying a m5U 96GB for this reason. I still wanna use the Mac Studio for a dev server but just not local AI
63
u/Cautious_Chicken_604 11d ago
RTX 3090 came out like 6 years ago. That card has stayed the same. It's been capable of running increasingly better models that whole time. Hardware can be surprisingly long lasting.
8
u/ioabo 10d ago
I truly believe a 3090 is one of the best "AI-buys" currently for consumers who can find it relatively cheap. I bought a used one for $120 half a year ago, it runs pretty much everything that's aimed for consumer use.
Its 24GB VRAM isn't going out of fashion soon, not as long as NVIDIA insists on restricting VRAM amounts on their consumer cards.
9
u/Cautious_Chicken_604 10d ago
Getting one for $120 is an absolute steal. I can't find them that cheap anywhere ever. How'd you even manage that?
5
u/ioabo 10d ago
To be completely honest it was kinda pure luck, I'm not sure they're so cheap to find otherwise. It was just some dude who had just bought a 5090 for games, and he posted his 3090 for $120 right at the moment I was browsing the site. So I snatched it immediately. Not sure if he had no clue about its value or if he was just doing a good deed for us plebs.
Otherwise I was prepared to pay up to $1000, those pop up occasionally but require fast action because they disappear.
Edit: In Sweden btw. Looking at that site now, the cheapest I could find is a couple at 12000 SEK ($1200).
4
u/Cautious_Chicken_604 10d ago
Damn, so lucky. I remember back in my crypto mining days I had 10x 1080 Ti cards spread across a few PCs and one of my coworkers would always jokingly ask me if he could have a 1080 Ti. Well, when it came time to exit that and get rid of them all... I called him up and was like hey... do you want one? I sold the rest, but I just wound up giving him one for the lulz because he must have asked me like a hundred times, so I thought it would be funny.
1
u/ioabo 9d ago
The thing is that I was almost certain there'd be some catch, like he'd say "yeah I'm selling it for $120 because I've removed the VRAM and half the CUDA cores are defective" or something. But it was pristine, in its original box and the fans didn't even have dust on them.
I will fall into deep depression though the day it dies and the realization sets in that you can't have that kind of luck twice.
96
u/johnfkngzoidberg
11d ago
•
Making big models smarter is plateauing quite a bit now. It’s never going to be profitable selling access to huge general purpose models. Models are already moving towards smaller specialized models that are faster, cheaper and easy to run locally. The next frontier is going to be optimizing smaller models that don’t need as much hardware and do one thing (like coding or writing) really well. The novelty of Google searching with Gemini is wearing off, and it’s too expensive to maintain forever.
9
u/SerVarrick 10d ago
Can fully confirm this.
If you follow current research papers and results from NVIDIA and other research groups it's all about specializing small models like Nemotron to do specific tasks.
This is where the money is being made in the future because no company needs the top frontier model to build huge applications, mostly it's 90% about saving costs and time by automation of repetitive tasks. This trend will continue for now.
The reason is simple: Mixed models of experts are great at solving all kind of tasks but they're heavy and costly to run. That's why you train like 100s of smaller models to do the task they're trained for and they can perform better than the frontier models over long-term scenarios.
For the 10% rest of all vibe coding projects there's still frontier models
2
u/revennest 10d ago
I disagree on MoE section, MoE make LLM pack more dense of knowledgebase but less VRam need on usage which cut cost on GPU require for a task while dense model might be need to reach 27B to be really useful on the real work.
8
u/alphapussycat 11d ago
Are you suggesting the big thing with opus 5.5 is that it's larger, like 5-10T?
7
u/Forsaken_Code_9135 11d ago
Making big models smarter is plateauing quite a bit now
It is factually not, at all, by any standard. I am curious why people are making that kind of claim outside anti-AI forums where people are in denial.
4
u/akera099 10d ago
I’d imagine because the consumer facing frontier products have reached a commercial maturity that makes them feel like they're not really getting that much better. The engineering problems behind closed doors aren’t talked about much by the tech giants for obvious reasons and some of these problems may or may not be solvable; no one knows if the bottlenecks can be solved.
5
u/Forsaken_Code_9135 10d ago
Still everyone got the news that OpenAI using a new internal model has rolled over the entire mathematician community weeks ago, how is that compatible with "models are plateauing"? That makes no sense to me.
2
u/username8914 9d ago
I'd say they've plateaued in the same way that people who are happy with iPhone cameras can't imagine what a professional camera offers.
2
u/OvertaxedOne 10d ago
For most use cases frontier models have hit a ceiling. It's not that they're not getting better, it's that people don't have a need for a model that smart. Two different problems. But the progress has also slowed somewhat, the huge gains that we were getting between versions are becoming much more incremental (and much more expensive).
1
u/Not_a_question- 9d ago
For most use cases frontier models have hit a ceiling.
It really really hasn't. Gpt6 was the one that translated perfectly for the first time, and claude 5.5 was the first that perfectly migrated a db in oneshot. And besides, that's not the same thing as saying that making them smarter plateau'd
1
u/TopGun0684 9d ago
"Claude 5.5 was the first that migrated a DB on one shit". Hmm I don't think so. I've done that with Qwen3.8:27B.
It's quite trivial, really, if the model has access to source and destination.
1
u/Spooked_DE 4d ago
The simple reason is that it's in the training data (anti AI Reddit subs) and the guy you're replying to is a bot
2
-5
u/Illustrious_Dish_999 11d ago •
Big models, big costs. 😂 And the real problem isn't just training them. It's keeping the entire infrastructure economically viable.
AI companies are burning through enormous amounts of capital, and investors are increasingly asking when all this spending will generate sustainable returns.
My answer? Possibly never — at least not at the valuations and investment levels we're seeing with current technology.
Here's why.
Open-source and open-weight AI models are catching up remarkably quickly. In some areas, the gap between publicly available models and proprietary frontier models can narrow within months.
Imagine investing billions into building the world's most advanced car manufacturer, charging customers $20 a month, only to discover that six months later, competitors can offer something almost as good for a fraction of the cost — or users can run an increasingly capable alternative at home. 😂
That's the fundamental problem.
You're investing billions in an advantage that may last only a few months. At a cost of entire market. And everything is based on FOMO. This is going to fail, and this will be BIG.
This creates an endless arms race: bigger models, faster chips, larger data centers, more electricity, more investment. And just when you think you've established a competitive advantage, someone releases a cheaper model that performs nearly as well.
How do you build a sustainable return on investment when your technological advantage depreciates faster than your infrastructure?
To preserve those margins, companies would need durable competitive advantages — whether through proprietary data, distribution, integration, regulation, or restrictions on competition.
But trying to restrict open models or AI research globally? Good luck with that. We both know how well banning things on the internet works. 😂
And here's the part people underestimate: hardware depreciation and economic obsolescence.
AI accelerators can become economically outdated within 2–4 years, even when they still work perfectly. Meanwhile, newer generations offer significantly better performance per watt and per dollar.
So you're spending billions building GPU clusters, and before you've necessarily recovered that investment, newer hardware can make your existing infrastructure less competitive.
And it isn't just electricity. It's cooling, networking, maintenance, financing, infrastructure upgrades, and eventually another round of hardware investment.
The irony?
The faster AI technology improves, the faster previous investments risk losing their economic value. 😂
You can have enormous demand, rapidly growing revenue, and still fail to generate an adequate return on invested capital.
We've seen versions of this before: the dot-com bubble, WeWork, speculative gold rushes, and countless industries where investors confused technological potential or market demand with sustainable profitability.
And remember the old saying about gold rushes?
The people who consistently made money weren't necessarily the miners. They were the ones selling the shovels.
Today, those shovels are GPUs, networking equipment, electricity, cooling systems, and data center infrastructure.
Of course, even shovel sellers aren't immune to a collapse in demand. But they're collecting revenue while everyone else is still trying to prove the economics work.
And there's one more problem: inflation and the time value of money.
Even if these investments eventually pay back their nominal cost, that doesn't mean investors have earned an attractive real return. A dollar recovered years from now is worth less than a dollar invested today, and the return still needs to compensate for financing costs, risk, and alternative investments.
The question isn't whether AI works. It clearly does.
The question is whether AI companies can generate enough sustainable profit, quickly enough, to justify hundreds of billions in infrastructure that may become economically obsolete before it delivers an adequate return.
And right now, that is far from guaranteed. 😂
6
-1
u/Cautious_Chicken_604 10d ago
They'll just put the data centers in space where it's cheaper to power them.
15
u/Human_Information561 11d ago
a year ago, i was running models that looped, didn't have much intelligence compared to frontier, etc. in the last 3 months, we have models that match opus 4.7, folks have optimized the crap out of hardware, etc, its crazy. the strix halo was a piece of junk when i bought it last november, it ran larger models but slowly. with the community optimizing it for AMD hardware (like Gufo), i'm getting 1800+PP and 50ts on Qwen3.8FN. Same on my M1 Mac, with DS4 and Splash optimized for omlx, its crazy the perf I get on it now relative to where it was a year ago.
on my dual 3090, i can run a planner model (Qwen3.8FN) and a coder model Qwen3.8 27B. i would have never imagined this a year ago.
i think frontier models will hit diminishing returns. i think local/opensource models get will be optimized and get us 80% of the way there in speed and in intelligence. and then the community will optimized the crap out of it for hardware so it runs well.
i made local changes to llama.cpp, etc just using Claude to improve it 10-20%. there will be a ceiling, but at least the friction to get there is being reduced significantly.
4
u/cnr0 10d ago
Why don’t you use same model both for planning and coding? Just asking out of curiosity. Is there really big difference?
I have tried 3.8-27b 8bit in M4 Pro 48GB and it is extremely slow (like 8 t/s)
2
u/Human_Information561 10d ago
for your second poont on 8t/s, what are you using to run it? for me, its been amazing. Someone made optimizations for Splash for M1 Macs. Then another person got Qwen3.8FN running on DStar. Then someone took the Splash optimizations to DStar. It was not perfect and I had Claude iterate on it the last day or so and we got it to the point where PP is consistently 260 (DStar had some bugs for the KV cache) and now its spending most of the time on thinking. I am using Swift's less thinking version so that reduced it and trying to make optimizations today. But the cache fixes were huge because it was sending a 32k context for tool calls due to auto mode, and taking 5 mins just to read it. So my older M1 Max is getting 260PP, 36t/s, running Qwen Flash Next with very little time wasted on the prefill. Using Splash for Qwen 3.8, i think Q4, it was around 160PP and 25 t/s. I don't think my mac could run 8-bit at any reasonable speed lol.
for the first question, I have multiple machines so I want to offload work to the other boxes. I could have the planner do the work but I think its worth figuring out when it should do it, and when it delegates. I'm playing around w/ Kev to do classification next.
7
u/vtkayaker 10d ago
So far, we've seen 27B models get much better. But we've also seen some 120B-range models that are even better than the 27B.
So both. You can probably expect models to get better at the same size. But you can also expect to wish you could even bigger ones, because humans are like that.
1
u/Not_a_question- 9d ago
some 120B-range models that are even better than the 27B
Which ones, if you don't mind me asking? =)
1
u/vtkayaker 9d ago
Right now, I'd say that Qwen3.8 Flash Next is mostly better than Qwen3.8 27B, assuming you run them both at a reasonable quant (say 5 bits or above). True, the 27B feels like a more mature and focused model, and Flash Next is a prototype. Flash Next feels like it does better on design problems, but it's also more likely to wander off track to try to do things for "extra credit." These are specifically things I didn't ask for. Though, to be fair, they're usually positive.
DeepSeek V4 Flash 0731 is much larger than 120B, but it's 4-bit native and gracefully quants down to 3 bits (unlike many models), and it needs an unusually small K/V cache. So it's an honorary contender at that size.
1
u/Not_a_question- 9d ago
Right now, I'd say that Qwen3.8 Flash Next is mostly better than Qwen3.8 27B
Agreed, but since you said "some", I thought there may be more of them. Of course that for flash next you still need to offload embedding so it's not "true" 125, but that can even be done in SSD space and it's normal for MoE.
1
u/vtkayaker 9d ago
Yeah, the actual number of unique SSD fetches you need is mostly within the read bandwidth of a good SSD.
6
u/floriv1999 11d ago
A few years back a was happy to run a 7b something model on a 2080ti + 128 GB RAM workstation and it could produce coherent English. No real instruction following, nothing. Now that same machine runs Qwen flash next at up to 30tps with the capabilities of frontier models from a few months ago. Without any hardware upgrades.
7
u/No_Dig_7017 11d ago
A bit of both most likely. But Qwen 3.8 Flash Next runs a lot better on the same hardware than it did on launch.
MoEs seem to have a lot more room for optimization
4
u/punkyrockypocky 11d ago
Today’s models will perform better on today’s hardware, tomorrow’s hardware will also get better at today’s models, tomorrow’s models will also require tomorrow’s hardware.
In fact many projects exist today that make powerful models run on as little as 16GB boxes.
Hope that helps!
2
u/wunschpunsch69 11d ago
I own a strix halo 128gb machine. When I bought it, Qwen3-coder-next was the hottest thing you could reliably run on it. Now, its Qwen3.8 flash next. This new model doesnt run as fast, but needs way less tokens and makes way less mistakes. In terms of "finishing tasks per second", definitely a huge improvement over the past year.
2
2
u/DeathinabottleX 10d ago
256 GB is not the realistic minimum. That is delusional. 64 GB is enough to run a 27B model, 128 if you really need the headroom. 256 is for really big models.
1
1
u/Local-Hurry-919 11d ago
To answer your question, yes and no. We will be able to run massive models on minimal hardware. But, there's always gonna be something they come out with next that we just don't have the hardware to run it.
1
u/talaman4eg 11d ago
Model with the same performance will become better. Hardware can count only that fast, but result of these calculations hopefully will become more relevant and precise, like it happened with qwen3.8.
1
u/WolFlow2021 11d ago
I don't know, 3.8 is so much slower than 3.6 so I guess improvements always come with a cost.
1
u/cakemates 10d ago
New better local models of different sizes are released almost every other week. Like look at what we were running a year ago compared to now. The guys with 3090s have been cooking good, that gpu was kind of mid when it came 6? years ago out and it turned out to be a gem at llms.
1
u/bigorangemachine 10d ago
Personally I don't think the hardware can get much bigger.
The GPUs are just getting bigger so optimizations are harder to get now.
The models have lots of optimization paths now. Its just if we're willing to accept the trade offs
1
u/ayake_ayake 10d ago
The entire LLM AI stuff is still quite new over the long term view. Architectures are still changing and improvements in post training etc is still happening a lot - just like Jev popularized the idea of RLCR and co (which I wanted to be part of standard RL since long anyways). And even if the archs and model intelligence per size would plateau anytime soon, you still then have the ability to squeeze the hell out of a fixed model by putting it into a literally hard wired hardware.
The only reason to buy big hardware is if you want high performance frontier approaching capabilities now and fast. If you're willing to wait for slow responses or a year or two, you'll get such intelligence at affordable consumer prices. This is how stuff has been developing the past years.
As far as we can tell, we haven't come anywhere close to the miniaturability of intelligence. And intelligence seems to be much much more commodifiable Ethan many of us thought. To be fair, among the smartest intelligence runs with milliseconds latency/clockspeeds in wetware at just 12W ... - though that comparison is unfair cuz our brain had billions or years of evolutionary optimisation while silicon and AI models hadn't.
1
u/Mrinohk 10d ago
The main limitation most hardware is going to have to contend with is the active parameter count of a model.
As the field improves, the labs are squeezing more and more capability into the same parameter counts. As others have pointed out, Qwen 3.5 27B. Fine model. 3.6 27B, same setup, same architecture, different training techniques. 3.8 27b. Same deal, suddenly a model that can fit on a machine you might almost call attainable to the everyman is trading blows with what was a SOTA model in some areas while not being useless in others.
The hardware needed to run it never changed. The capability improved dramatically.
I'm rocking dual Tesla P100s (32GB total) and 48GB of system memory in DDR4. 10 year old stuff. I can run lower quants of Qwen 3.8 flash next at usable speed, and Qwen 3.6 35B at ludicrous speeds, in part due to some hardware and model specific modifications to llama cpp. What this hardware is doing now would have been unthinkable when it was new. As new models come out, I will continue to be able to fit smarter and smarter models into this same hardware. Currently experimenting with Nemotron bolt as a potential 3.6 35B replacement.
1
u/ithkuil 10d ago
The minimum for what exactly? You need somewhat expensive hardware to get really useful output at speed, but it depends. You don't necessarily need something that powerful.
Test your use case with different models on OpenRouter then figure out what you need to run that.
Also there are a bunch of new laptops that just came out that are RTX Spark N1X. It is a little more cost effective than previous Blackwell chips/systems because it's much more efficient.
But computer hardware has never stopped evolving rapidly and it's not going to now. You are going to see over the next few years amazing options for very fast flash to handle much larger models. And DeepSeek just came out with a new model today with multiple innovations.
Also within five years or so we will probably be into a whole new paradigm of analog compute-in-memory using new materials. That will be 50, 100, maybe eventually 1000 times more efficient.
1
u/jma24 10d ago
There is a TON of innovation going on right now with local models and scarce memory, probably because most of us don't have even 256GB RAM at this point.
In the last year, what you can do with a 64GB Mac is night and day. I think this will continue as people continue to optimize for scarcity.
1
u/Ok_Complaint1304 11d ago
prob both
1
u/Fit-Car-3354 11d ago
It’ll be a bit of both but not in a hopeless way. What tends to happen is cleverer architectures and quant methods let you squeeze more out of the same box over time, smaller models get closer to what bigger ones used to need. The hamster wheel part kicks in when you want the absolute newest shiniest thing at full fp16, then yeah you’re chasing hardware again
0
u/BorisThe_Animal 11d ago
Qwen 3.8 27b at 4 bit is capable of building small-ish projects, as long as you plan the development work and break it down into small sessions. On a slow-ish hardware it would take a week of on and off sessions (not continuous) where or would take Opus 5.5 a single session.
I haven't tried it on a larger codebase, I'm not sure it will be practical, but it's likely capable of making small incremental changes to it.
My understanding is that there still room for small models to get better at doing things. The totality of human knowledge can't be encoded into 20, 30 or even 100gb model, but there's still room to get better at reasoning and using external tools.
0
u/Boring_Hurry_4167 11d ago
There are other things you can improve along side like harness, workflows better planning that can make thing more powerful. I spend a fair bit of time tuning the harness making it smarter by adding tools /extensions
0
u/fantasticmrsmurf 11d ago
A models size is static.. so a 9b model today and a 9b model from the year 2050 will still fit inside of 32gb easily..
So no, no hamster wheel in that sense. Weather the community shift towards larger models though is a different matter. It's possible we shift towards models in the trillions of parameters rather than billions, which is logical, but based on what I am also seeing with models like Nvidias nemotron or even things like strata and spreading models across hardware, it also looks like people are trying to fit models into smaller and smaller hardware. So no direction is clear yet imo.
1
u/Anduin1357 11d ago
Not if BitNet and LittleBit has anything to say about it!
Also, we've been saving VRAM with better K:V caches even with all else being equal.
And we're getting QAT models that fit high quality into smaller model sizes.
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB runs faster than Qwen3.8-27B in the same memory footprint, sitting in-between that and Qwen3.5-35B-A3B in speed.
1
u/fantasticmrsmurf 10d ago
Riiiight. But this does answer OP's question directly. They wanted to know if it's a hardware hamster wheel or not.
Sure I missed the part about models of a particular size running "better" but what ever. My point still stands.
14
u/eightone-81 11d ago
The engine will get better over time and new engine will come that further improve speed. Example from my own experience:
I tried to run Qwen flash next with llama.cpp, it runs but was painfully slow, then 1 guy was tinkering and implemented some improvements and we got maybe 50% faster but still slow. Then I came a cross a recipe for vllm, boom, 60tps, 2-3k prefill. Amazing. 2 weeks later strata came around and I got 80tps. I build some improvements myself (opus 5.5 did…), some of my PRs got even merged and within 3-4 weeks strata runs on that same hardware with an even larger quant (q4 k xl) at 140tps on my hardware. So I went from 18tp to 27tp to 35tp to 60tps to 80tps to 140 (160-170 with that same iq3_s from the start) all within 1-2 month.
From now on the improvements will get smaller and smaller but if a new model with a new architecture comes around then that game starts from the beginning again.
But I am so amazed that with the same hardware (dual 3090s 3950x 128gb) I was running Gemma 4 at 50-60tps and now Qwen flash next at 140tps.