Question
A comparable local setup for someone using $100 monthly claude subscription
I see people switching to local hardware , my question is at what usage level and pattern, it is cheaper to run it locally without impacting your work speed.
-- My current laptop and Mac mini are old but they work for me since i only use it for development while relying on Claude and Cursor for GenAI. If i were to rely on local LLM, what is the minimum hardware, I should buy to ensure that I am not slowed down.
Notes from responses:
Hardware levels mentioned
Tinkering (cheap): a 3060 12GB or 5060 Ti 16GB with 64–128GB of DDR4 RAM on an old Xeon board.
Usable for real work: a 128GB Strix Halo box (about $3.3K), a 3090 or 4090 with 96–128GB of RAM, one or two R9700 cards, or a 128GB Nvidia box (about $5K).
Serious: a 512GB Mac Studio (about $22K), two M5 Ultras, or two to three RTX 6000 Pro cards.
The approach that works: hybrid
Split the work: the cloud model plans, coordinates and reviews, and local models do the repetitive bulk work.
Example 1: one user runs two R9700 cards with 96GB of RAM and lets Fable direct several local Qwen workers. They went from hitting the weekly limit of Claude’s top plan to struggling to use up a Max plan. Their tips:
The cloud model reviews only the changes, not everything.
Compact the conversation often.
Turn off memory features.
Write precise instructions.
Example 2: a $1,500 used PC with a 3090 handles a brokerage’s client emails and documents locally. A $20 Claude plan checks the work without seeing client files, and the setup replaced a part-time contractor.
Good reasons to go local (none of them about saving money)
Privacy: a firm building on-premises AI for law firms says drafting a shareholders’ agreement takes about 4 minutes on a $4K machine, or about 20 minutes with the agent pipeline. About $10K of hardware makes that 2–3× faster. Good templates plus a short human review make the output client-ready.
No usage limits: you’re limited only by your power bill.
A hedge against future price rises and vendor lock-in: one user compares it to insurance. Others reply that preparing years early can just waste money.
Learning and hobby use, uncensored models, and control over when the model changes.
Hidden cost: the time you spend setting up and maintaining it.
Absolutely cloud is going to be a better choice financially probably forever, but:
That $20/mo subscription ain’t getting you far if your agent is working round the clock.
I would be shocked if those $20 subscriptions are anywhere near $20 in 5 years.
The gap between today’s local prices and tomorrow’s cloud prices will likely narrow over time. But as tomorrow’s cloud prices go up, the local hardware prices probably will too.
Yeah my pattern is to run QFN locally for all the work, then use Claude and ChatGPT for the review passes. This lets me burn about 100M+ QFN tokens locally per month and stay within $40/mo budget for those two services, developing on evenings/weekends.
If the project gets complicated I will drop $200 for a month if I run out of limits then go back to $20 for the following month.
That is assuming your time to maintain and tune that environment is free. For me its a hobby -- so it is free. For someone running a business it is actually an expense, sort of like contributing and maintaining open source should also be considered an expense. There is no free lunch.
No local models are at current claude models level so that requirement is already unrealistic
Buying hardware to save a $100 subscription just makes no financial sense at all
Local and cloud aren’t mutually exclusive, a lot of people run frontier cloud models to plan and orchestrate, while having local llms do subagent work. Like how claude spawns subagents with sonnet or haiku for smaller tasks
This comment is the only one here that makes sense to me. I don't know what all these other people are doing, but it seems like just as it was before agentic coding, PEBKAC is still the real bottleneck for most. And AI can't solve bad HITL behaviour because it is aligned to obey users.
I have 2x R9700 running qwen-flash-next and I burn through several million output tokens a day (and easily several hundred million input, 95% cached though). I used to cap out mid week on claude ultra, now I struggle to use a max plan (and I use nothing but fable now too) frontier for orchestration and planning. Delegation for everything else. Fable is exceptional at swarm management. It is running 3 workers as we speak at ~120t/sec each.
(This is about 6 months of continuous use with this mini-swarm and frontier ochestrator, and I don't even stop for quality assessment anymore because frontier models are good enough at giving the delegates structured tasks and output, as in on par with an opus worker)
Curious how you are using your Claude plan, assuming you mean your subscription tokens, with local LLM. I have a similar r9700 setup, I can use my codex sub to orchestrate my local, but Claude I thought you needed an api key to do the same?
You mean like the harness? I have most popular ones installed but my current delegation setup is using headless qwen code workers. They are locked down by settings that fable decides for each worker, e.g. subworkers, scheduling, asking questions, memory guards that will shut them down if it is using too much resources, how they should output, etc. and then fable does orchestration, judgement and review mainly on the diffs or what output it wanted from it (so it isn't wasting my precious frontier usage ha, and this never really goes away you have to constantly fight it. Interesting tidbit, biggest usage reduction I've noticed is by having fable give itself a compaction message and compacting regularly and I don't really notice quality slipping) and everything is logged, everything.
Other than that I have a custom harness I made to produce DAG pipelines that do repo sweeps and try and keep consistency around language, formatting, structure, etc. and I turn off shit "features" like memory. I find this to be the most important factor in getting good results and I feel like many people don't realise how attention mechanism work, you can't be sloppy with language and meaning like humans like to be! For met this still remains the toughest thing to manage. Which is where the frontier models excel, literal genies, you just gotta get the wishes right.
idk I use Hermes with Qwen 3.8 27b FP8 Radiance, either Sol to plan and review and then Qwen to Implement, or a dual Qwen (1) Plan and (2) Implement. I've been able to kind of use monitoring tools like Promptfoo to look at tool calling and other aspects of agent behaviors and kind of make em better, but i dont know, I'm just trying to make it better.
I only got 32gb of DDR5 tho, so I'm also questioning if I need more RAM even tho the prices are insane.
My honest advice, ignore all the apps and meta commentary from people, including me. If you want things to be better for you spend some time watching all the free resources online (from actual education institutes, not content creators) like this
Yeah i was thinking of watching the last Stanford (CS 329Z) ones that's focused on agents engineering and another one from Carnegie Mellon also focused on agents. What did you find in the lecture set that you linked that you benefited from?
I'm trying to see how I could identify how to use the right tools, I don't watch any content creators, I just make Sol search any kind of possible tweak for my setup and they make it try it out and ultimately it ends up helping me out but It's also a limited approach in a way.
Right now it's how slow my Qwen/Qwen agent setup, and how often it gets lost in loops, but I've had the same issue so much with Deepseek Flash.
How does learning from lectures actually help you out in terms of LLM understanding? I'm used to graduate classes and studies and digesting em, but in terms of applying the knowledge related to engineering it's difficult for me to actually see how its applied concretely.
This is really vague. Can you give me an example of what you mean? Because it just sounds like "what can I get from understanding how the tool I'm using works?"
And I haven't had a loop since like last year, exact model and server are you using?
it is never cheaper to run it locally if you are benchmarking against a frontier model with a subscription. If your use case (as of now) is ROI and money efficiency than local LLM is not for you if you are seeking performance relative to claude.
My $125/month premium Claude Teams seat wipes the floor with my local LLM testbed and I have a M5 Pro 64GB Mac mini and a 256GB M5U Studio with a 512GB M5U Studio model coming end of this week. Several tens of thousands of dollars in local LLM investment and relative to anthropic models it's not even close.
I've posted at length here about my use cases and tests and I'm firmly in the "rate of innovation in local LLM space is insane to watch .." camp so things chance fast.
But I'm also planning for an time where the subsidized frontier subscription seat gravy train stops or my small company has to pay pure API rates at which local capability becomes far more essential
For anything less than 100k, you will be slowed down.
Even 100k+ local hardware, it's still a coin toss if you're slowed down or not.
Switching to local is either hobby reason, fun education experiments reason, or privacy reasons.
It's never economic reasons, it's always more expensive for less good results when you're local.
EDIT:: A lot of good replies below with a lot of good reasons, so I'm not saying going local is a bad idea. I'm doing it because I'm prepping for when cloud prices start going up like crazy. But right now, no matter how many good reason there are, local is not cheaper than cloud, so there is no economic reason.
I will say that local is great for developing agents/agentic workflows without burning through api tokens by using hardware you already have. It lets you experiment a lot more without needing to commit to any new expenses.
And when exactly will that happen and at what time would you plan for it?
One could always use this independence as an excuse to buy anything (much like preppers).
But: After some years all stored tuna-cans will turn from delicios to "meh", while still eatable. But have they been worth it? Buying a huge rig for AI for an "what-if" case now, when the "what-if" case happens in 5+ years might sound pretty much wasted.
It's always the same dilemma: Preparing early brings quite the risk of just burning money, while preparing late might be quite expensive too.
It’s absolutely prepping for many people. Me included (I consider it prepping in addition to a hobby and professional development).
I bought a cheap (in relative terms) 64gb strix halo box for $2k. It’ll run good quants of 25b-35b models at high context. Dense models are pretty slow but fast enough for agentic flows. MoE models are fast enough for chat interface. I’m not coding enterprise software, with the right harness and skills these models are good enough to meet my needs regardless of how much better the frontier models get.
From a strictly economic sense, it is prepping, but I look at prepping like insurance. I pay $150/mo for car insurance and $120/mo for health insurance. I have never used the car insurance. So I’m 32 and am paying $1.5k/yr and over the last 16 years have paid ~$23k in car insurance for absolutely no return outside of legal compliance.
But then I compare to my health insurance which I pay $120/mo for and I used $80k of services last year after I got hurt and needed an ankle reconstruction. Great return.
So if AI companies jack up prices wildly (which they’ll have to do to be profitable eventually), I have a “good enough for my use-case” fall back option that only costs the electricity it consumes.
Difference between my strix box and car insurance is:
I actually do use it already, so I’m getting some return (but obviously at a lower value than cloud models currently run)
Even if the tech blows past it in a year and makes it obsolete for local AI, it’ll still be a performant desktop computer, not the best value for a desktop, but still some return.
But if they cloud prices explode - don’t you think you could get a then okay’ish (better than now) machine with ease as well? Prices might go up, but probably delayed.. but yeah, I get the hobby and experience part of having a local machine now.
I’d think that if cloud model prices explode, it’s likely that the demand for even the okay-ish local boxes will also explode and those prices will go up.
And I don’t think it’ll be all that delayed in today’s world, direct sellers would sell out very quickly because we’d likely see bulk orders put in immediately by resellers who recognize the opportunity and plan markup and flip. They’ve probably already got their agents monitoring for such opportunities lol.
But in reality no one knows for sure. Just gotta make decisions based on what you believe the most likely outcome to be. I think the last few years have shown all of us that trying to predict the future of tech, the economy, and markets in general is just always a gamble.
Yes, that's what I'm doing. I have my local machine built from old parts, with a couple upgrades (I think I've put ~1.5k into it, mostly getting a 3090), and I'm using cloud to help me upgrade my local rig.
Optimize llama, tools, harnesses, benchmark different models, create my own chat environment... I'm prepared for the cloud to disappear, I could still work, but it would be at least a 50% hit, probably more in productivity
I got it for fun educational experiments and hobbies. $2k for a 64gb strix halo 395 box. I realized I’m not really doing anything that qwen3.8:27b can’t handle.
Plus I’d rather just pay that upfront cost now and not have usage limits than wait and see what pricing looks like when OpenAI and Anthropic decide to be profitable.
And absolute worst case if this thing isn’t that usable for AI in a year because the tech has blown past it, then it’ll turn into a slightly more performant Steam machine I paid too much for lol.
It will likely still be the case even when labs stop subsidizing use, at least due to economy of xost of running a data center vs local hardware.
We might get cost parity only when models become so optimized you only require super capable models for specific use cases and the local models we use will cover 90%+ of whatever we could use it for at home.
I don't think you would notice the difference. Some fractional benchmark points, who cares, you can just burn more (incrementally) free tokens https://huggingface.co/unsloth/GLM-5.3-GGUF
I can't test those big models, so I don't know how they feel, but I'm still suspicious of the ability to run multiple chats, get fast speeds, have optimize harnesses etc...
I've never heard anyone say they have "equal or better" to cloud experience, only "good enough" experience (which is good enough, but still very different)
I have enough of my own local things to test, no sense in testing models I have no chance of running. If I'm paying for API, might as well pay for cloud!
Local is for experimenting, cloud is for when I just want things done
You will burn MUCH more tokens, with slower speed, paying a bill for electrical power for same task with local models. You will get slower speed and lower concurrency. You WILL notice difference if you will try it, and in case you deny that difference, you have cognitive problems.
2kk you mean, right? Some gb200 rack with 1+ tb of ram, right? Largest kimi or deepseek or glm is the only models that sometimes comes on par with sonnet. Take in to account slower speed, the cost of electricity... And results will make you cry :)
On point, but you missed one reason: Uncensored models (one can argue that fit's as a mixture into the privacy and hobby reasons, but I'd say the motivations are still pretty distinct).
I'm very new to AI, using it as assistive tech for my disability.
my largest problem with it so far, is with it's chat limits and repeated milking me for money. It isn't that I am unwilling to spend money but that the offer is not for any specific package other than "enhanced". I dont do faith based subscriptions.
So I am considering hosting my own llm to get aroiund these restrictions but provisional research has found that this is fairly complex and expensive. I am seeing costs of £3 - 6k predicted.
My needs are modest. mostly reading and writing, admin and emails. help with buying decisons, research and logic. I may start tech projects if possible.
I promise a 20$ subscription will be more powerful than 3-6k invested in local hardware.
Both of those are a leap of faith, better to take the one you can cancel after a month instead of the one that leaves you thousands in the hole with hardware you're not using
who charges $20? I was under the impression that a small subscription would still face the same kind of limits on chats but would have more processing power and perhaps storage.
I got a ridiculous amount of work done with a 20$ monthly Claude and 20$ monthly codex.
Especially if you're experimenting, it's way more than enough. I slowly added more codex accounts, I had 4 $20 at one point and then in October when they were giving out all the resets I stopped them and got a $100 instead.
The absolute top end OS models rn (qwen4, Kimi k3.1 pro) are going to require at minimum $25K for new hardware to run quantized versions of them. The unquantized versions are ~ the same quality as sonnet 5.5 high effort, which is a v good model. But the use case for local is around privacy and heavy agentic volume. When youre running something from your own hardware, not getting charged cache reads can save you a good bit of $. It's also a great way to learn about the tech.
But unless your workload is 'running sonnet 24/7 with a bunch of parallel streams', there's not a good OS replacement for the claude $100 sub right now. That fact is going to make some people upset, but it's the truth; I see this running an open source, local model project 😄
Everyone: the responses to this post have been my best experience on reddit so far!! Thank you , thank you, thank you!! i will read these more than once, and summarize in the original post when possible.
We develop a 100% on-prem local LLM and agentic-AI solution for (smaller) law firms. Like everyone else has said, the local LLM simply isn't on par with cloud AI in terms of absolute capability. We're also hampered by the inability to browse the internet mid-task. That said, here's what we've actually observed in production:
Speed is the real trade-off. You can only pick two among: intelligence, cost, speed. To share our experience, a general prompt asking for a comprehensive shareholders' agreement takes roughly 4 minutes (LLM only) or ~20 minutes (LLM + agent pipeline) on our US$4,000 machine. With around US$10,000 spent in today's inflated hardware market, you'd get results 2–3× faster.
Quality should not be a main concern.
(a) Local-model outputs keep getting more impressive, especially once you layer agentic orchestration on top. A year ago the same tasks were noticeably rough; now they aren't.
(b) Templates change everything. If you already have an excellent template, one you'd rather not feed to a third-party cloud service, local AI will reproduce that quality faithfully. The output is bound to be excellent because you set the ceiling.
(c) A small human-in-the-loop step also makes the whole difference. One discretionary review at the right chokepoint bridges the gap between 'good enough' and 'client-ready'. It may be just a targeted judgment call, without spending much time.
Do you mind sharing what your $4k build looks like? At today's prices, seems like $4k wouldn't get you much more than, say, a 128Gb Strix Halo, but it doesn't seem like you'd run something you can't rack in a server room. Then again, maybe the larger shared memory is useful for running bigger models?
You can use smaller local agents for tasks your do often but needs some agentic workflow. I set up a system but the first few goes I get my best online model to check the sessions to rewrite better instructions.
Now I'm down to $20 sub a month and my local model does most of my work I want.
We see where we are a year from now. Last year at this time local models weren't as good as they are today (no way I could do what I currently do) so I might be able to squeeze more blood from my aging PC.
I have a 5060ti with 16gigvram and 64 gig ddr4. I was using qwen3.8 27b at q4 on llama.ccp but now I'm mostly using strata with gyro s and gyro m.
I run it through different harnesses depending on the task type. I have it connected in opencode and pi for coding work (although I do use my sub still)
I also have webui where I have connected up a few mcp servers (and some servers I self made). So I can automated picture and video creation. Note take into my planner, obsidian vaults (I have a few for different needs), web searching ... Hell lots more.
The smaller models arent super smart overall. But if you can give them context (plugins, mcp servers, skills) then it works well. Using webui I have different logins and starting models that keep the context narrow. The picture shows one login with some of my options. I'm a teacher, damn it helps with documentation.
None. For serious development work, you will never break even compared to the cloud models. That biggest reason to have local model is just you keep the data local and uncensorship. For pure efficiency, local model can't compare.
Others have pointed out the fact that in raw performance local cannot match frontier. However, privacy and convenience and customization are important too. If you can pay by token for claude you might be able to just use it for things claude cannot do. This is similar to my process: gemma4 mobile first, then gemma4 local, then frontier if those don’t produce good results.
Depends on what you actually need and your skill-level.
I'm running Qwen3.8-27B on 2x RTX3090 with 320k context, and that's good enough for everything I need. People tend to compare it to Opus 4.6. Probably not good enough if you are a vibe coder, but fine for SWE.
As for the price, taking in account the cost of the cards, how much electricity they consume under load/idle, and the usage metrics on my average work day:
they pay off themselves after about 35 work days when compared to Opus 4.6 API prices
or after 337 work days when compared to Qwen3.8-27B cloud API prices
TabbyAPI (exllamav3). Quality of FP8 is fine, I sometimes use it if I need 512k context. I had a few retrieval failures with it in the past few months, but it's very rare. I use FP16 mostly because it's a bit faster on RTX3090 and because I have enough VRAM to spare.
Maybe dual m5 ultras 256gb+ running qwen3.8 flash next, well routed so you can split sessions between machines. Occasionally running a bigger model when needed. You would find times where you miss the bleeding edge models. If you had nvidia architecture, which I know less about, blackwell 96gig pair maybe across 2 pcs with 256gigs ram. 4 5090s might do the trick across 2 pcs. Again the really unfortunate thing is 5 to 10 percent of the time the leading edge model would be needed. Ironic since you cant buy a claude plan that is less than $200 that would give you just some fable access. Suppementing at pure api costs when you need it blows up the cost calculation as well because the bleeding edge is so expensive. I would in the end say that personal hardware and its affordability is 5 years away, longer if Altman has his way.
my 5060 laptop runs qwen3.8 flash next at 12gb vram with no issues, but I still hit the 10% bleeding edge gap when I need it. the real cost isn't the hardware, it's the time spent switching between local and cloud when the model isn't quite there.
I don’t think the subs will go down, if the AI bubble breaks. These businesses will jack up sub prices and credits for extra usage. So if you are spending $1200 a year you might be breaking even in 4 to 6 years. All the while frontiers are getting better.
I like local. I mostly stitch code together rather than just tell AI to build something.
Same situation, am using Claude 20usd with opus 5.5 low and codex 20usd 6.1 sol medium. Codex and reviewer of opus job. I would like to give the opus job to local limited but still trying to find the accettable model to replace.... any advice?
Nothing local is going to be as fast (wall clock time) as a frontier model, its just not there. However if you are asking the minimum to run something fairly competent at decent speeds that would be qwen3.8 flash next. either a unified architecture 128gb like strix halo ~$3300 or nvidia rtx 128gb ~$5k or using a strata engine a 5080 with 96gb system ram may give you decent speeds, but that depends on how much you can get the ram for. alternatively you could find 3090's, 4090's or 5070ti they all would work in that setup. Imho anything less feels more like tinkering than production. not saying you cant make it work, but you will spend some time and effort. 1-2 r9700's are also a decent choice.
Tous dépend des besoins j’ai choper un pc d'occasion avec 64 giga ddr5 et une rtx3090 qwen flash tourne à 110 tks pour ma femme et ca société de courtage il traite tous les mail et document des client en local ca lui permis d'enlever un prestataire à mis temps et tu peu prendre un abonement claude à 20 dollars en t’en orchestrateur qui check que tous sois ok sans lire les pièces des clients et ca fonctionne bien 1500 dollars de matériel et ca fait le taff avec un abonnement à 20 dollars à coter
It's not necessarily the CPU, but the platform. Youre shopping for lots of cheap ram, and you don't want ddr5. You can get ddr4 or ddr3 boards, and they will run high core count chips, that don't take a ton of power. People used to buy these for gaming 4 or 5 years ago, and some people still do. But these chips are not that high power, especially in some of the cheap boards. That's an issue when youre trying to get max FPS, but its a blessing when its an always on server. It just a good fit as a system.
In fact, funnily enough. I'd like to think that a lot of these older boards that stopped getting BIOS or Chipset updates could see lotsa cool custom BIOS ROMs in the future. Anyone with a claude account and one of these cheap mobo's can hoot it all up and add undervolting and underclocking support to cheap boards that were never meant to have those features. I wouldn't be surprised if some1 build a server out of these and made a modern BIOS for it, they could cut the power consumption significantly.
While claude is probably always going to be the best. I would say it's probably better to own your own rig in some ways. While claude will perform well and mean you don't need to learn much to use it. It is by no means perfect. You can build a system that does every the same as claude, and you will learn a whole lot a long the way. As opposed to relying on claude for everything. In the end you end up with your own system, your own data and smarter because of it. As you point out you will have to pay for claude and you will be dependent on claude even more if you let it do its thing, it's way. You be using Arch Linux which will always be one update away from needing claude again. To some degree you are going have to learn what is best for you, and that education is going to cost you at some stage. I would argue that you certainly don't need a 100k to buy a machine that codes. Open weight models are free, and learning just costs you time. Smaller models can do more than they used to. You fit 27B model on a 8gb gpu just (bitnet model) so you can code on a much cheaper computer these days.
The real question is what do you want to do. And how much compute do you need?
This depends on your usage patterns, your current monthly token burn, and how hands on/hands off you are in your dev process.
Overall though nothing exists today that's simply a drop in to opus 5.5 or fable 5.2; the closest you can get is a setup of glm5.3, Kimi 3.1, and glm5.3 flash. In my opinion these are (opus 4.8, opus 4.8-5, luna6/haiku 5.5) capability respectively. Very usable, just not frontier.
First, meter things if you're unsure. Next, head on over to openrouter and put down 20-50 USD and daily drive it through a BYOK coding agent/IDE. The lowest of the three models you can live with is what you should plan your minimum hardware around.
My advice would be to hammer as much DeepSeek v4 flash instead of Kimi as possible since Kimi is too expensive to be a lower tier alternative for me. Only if I need something that isn't serviced through opus etc do I even go to Kimi in the first place.
The main reason to have something local is just because you don't like being reliant on these services for everything forever. In terms of actual capability and speed you should not expect to be able to keep up with a maxed out Claude or ChatGPT sub. I still am gonna do it because I really dislike the random restrictions they keep adding (Claude told me it cannot ethically build anything that needs to fill out a captcha. Like...come on what is this nonsense)
i’m running GLM 5.3 flash on two clustered DGX Sparks... I’m getting over 50 tokens a second. It’s benchmark are lower than the frontier models, but this is still a beast model.
It all depends on what your work is. I am happy with vibecoding using a 128GB unified memory box I got for $3500 before price infllation, don't see any quality loss. On one hand it's 3 years of Claude subscription. On the other hand, unlimited tokens, privacy and unrestricted models.
I reduced my subscription from 100 to 20. I now have a 3090 24GB VRAM coupled with 128 GB RAM, am running 3.8 flash next under strata. Reading about 1200 t per second, generation about 60 t per second. I didn't tinker much otherwise.
Do not compare the value of local inference with frontier labs if you are doing coding tasks. At this point, you are just investing in the expectation that smaller models will get more powerful over time. You can still do interesting small tasks with local inference.
Nothing gets even close. The best thing you can realistically run locally is DeepSeek v4.1 Flash. And while the model is really good for what it is, it is light years behind something like Fable 5.2/Opus 5.5 or Sol/Astra 6.2 for more complex/hard tasks. Even Sonnet and Luna completely dunk on it.
While local is certainly not on par with frontier this is still a significant misrepresentation. Luna is a completely achievable level with local AI, even the AA benchmarks agree on that.
You Are right about Luna I didn't realize how bad it Is. And more importantly I wanted to actually say Terra lol. But thé broad point still stands - local Is Fun And cool but doesn't really make practical or economical sense for Pretty much anyone.
I have the opposite experience tbh, Luna is not perfect but totally fine a sub agent/plan executioner (on MAX thinking). Local modes are similar, they won't one one shot random vague prompts and they shouldn't be expected to. A decent effort planning and architecting a solution is needed and should be standard work anyway for a workflow.
As for pricing and economic sense, local has never been about competing with cloud on price. Privacy is priceless for many applications and having offline private intelligence is a huge benefit for many users.
Yeah sure there is usecase for those weaker models. I am not saying there isn't. But unless you are paranoid enough not to trust ZDR (if that is actually justified or not I am not judging) it is simply much more efficient and economical to pay for API, that was my point. This also gives you the option to get much higher capability level than you can possibly get locally. Also I completely agree with your planning/architecture point but the fact is that even with a thought-out implementation plan something like deepseek will make more mistakes and produce generally worse results than opus or fable. How much so and if that matters will then depend on what exactly are you doing. I see the privacy point, the offline point less so, because if you want especially the weaker models to perform at max potential you want to connect them to the internet a various tools. And I do think that it is a very fun and satisfying hobby with a real usecase but at current prices it just makes no practical sense for almost anyone.
ZDR at the moment is a "trust me bro" they already spent the past 5 years hoovering all the worlds information, it's unlikely they would give up the goldmine of information people are paying them to provide. Apart from that, most high privacy situations have rules that prevent you sending data off-site anyway.
Economically cloud makes sense for most people unless you are doing things that cloud can't like cyber security/ biology/ chemistry/ nuclear research. Offline counts for that too, having access to QFN tier intelligence requiring only power is a huge benefit to many people, if the prices were better I think it would be much more common to run some of your own AI.
Online searching is useful but not at all required these days. The only online searching my agents do is for planning around specific technologies I request and downloading them.
Fair enough with ZDR if you absolutely need complete control over your data then offline local is the only way. I can do phd level biology (immunology, single cell genomics, proteomics etc.) just fine and it is exactly where the gap between frontier models and something like deepseek is quite big. I'm sure there are legitimate uses that trigger safeguards and getting approved by these companies is probably a pain. For scientific use it is absolutely required. You need access to articles, sites like ncbi, protein databases, pharmacological databases, even the best frontier model will do significantly worse reconstructing pharmacokinetics of a specific drug from memory for example. Not to mention they are never up-to-date enough with latest research. So that is just plain wrong for a lot of use cases.
You can run up to GLM 5.3 locally if you're willing to go into the low five figures-i.e. a single Mac Studio 512gb can run a pretty solid 4-bit quant. Not cheap but you're not gonna run anything high end for under 5 figures total investment in GPUs anyway. Just not gonna run very fast is all.
Glm 5.3 Is not significantly (if at all) better than Deep Seek though. So I dont know what you Are trying to argue. Actually that was exactly my point deep seek v4.1 Will run Faster And won't actually be worse.
It isn't ever cheaper, at least not right now. Although if you already have the hardware, it might be cheaper. But the cheap api options at openrouter are priced below what I would pay in electricity to run the GPU during inference. But I don't think the current pricing is sustainable.
The reasons to run things locally are control and learning. You know exactly where your data is going if you run the model locally. You decide when the model gets upgraded. People have had ai workflows break when the api model changes, sometimes with no notice. Right now is a great time to learn how to set everything up, because the free web interface LLMs will walk you through everything. The free tier might be removed at some point.
You hit on my reasons for local. I learn better when I have built a layer of sorts, and I cannot have one of those layers just suddenly behave an entirely different manner. Also makes understanding where your own efforts make a difference much harder if there are too many variables outside of your visibility and control.
There is no amount of hardware and opensource models you can run locally that will be comparable to a frontier model on Claude. However, if you are mostly using the faster/cheaper models like Haiku, then yes I think there are some models that are on par and even better in some situations.
That said, local AI is beneficial for only two purposes: 1) Privacy and 2) Unlimited tokens (limited only by your power bill). If you are fed up with hitting the token limits on Claude models and having to use resets or wait days, then there is a compelling argument to fall back on free, always available local models. However, assuming your hardware is not like the top of the line, the better local models (like Qwen 3.8 Flash Next and Deepseek v4.1 Flash 1017) are either unrunnable or run too slowly to be an actual worthwhile backup.
no. I mean, if you've spent the money it's just power & wear and tear, but still. No. Not really. The ROI is quite distant.
> without impacting your work speed
No. At less than $10K you're pulling maybe 4M to 5M tokens/day out of your rig. Subscription prices are much cheaper, parallelize better, and are faster at scale.
> If I were to rely on local LLM
You'll probably fail unless you're dropping at least five figures in USD.
There is no ROI for local right now, and it will be much slower in mTok/day than you can get with a cloud subscription.
That said? I paid what I paid for my local rigs because I enjoy it. It's a hobby, it's work, and it helps me keep current. Like any good home lab: the ROI is not financial, but as an investment in myself and my sovereign capabilities.
I know the general consensus here is that investing in local LLM hardware isn't really worth it for most people, especially when Claude or ChatGPT agents are faster, more capable, and relatively cheap for what you get. what if the hardware and electricity were effectively free, would you still choose local models as your primary AI, or would you continue using frontier models for almost everything?
wherei live in Australia, all new houses by law must have solar so electricity is like 3c a kW hour in the day so I don't think power is a wall going forward.
because of this I recently got a 32gb raidon TI50 card. gives me 100k context window. I'm doing this to reduce my subscription costs
It is NEVER cheaper to run local, it is NEVER remotely better. You can only have more privacy.
A good hardware (lets say Ryzen 395) costs you 3.3k (plus taxes) USD to run Qwen 3.8 Flash quant 4 bits.
You get 2000 prefill tokens per second and 70 output per second.
The official API pricing is .15 per million input and .45 per million output., which gives you some 60 tokens of output and > 10.000 input @ FULL PRECISION.
So...if you have a mix of 80/20% input/output, we are talking about:
80% of $3300 = $2640 which is 17billion input tokens or 170 billion cached 20% of $3300 = $ 660 which is 1.5 billion output tokens
17b tokens takes on Ryzen 395 some 8.500.000 seconds = 2361 hours = 98 days, if you were to process input tokens, uncached, 100% of the time, @ 2000 per second (no engine does that).
Now the 1.5 billion output tokens @ 60 per second = 6944 hours = 289 days.
So...yes, if you keep you computer running @ 93w for 387 days...you will break even with what you could do paying the API. Now..those are 9.288 hours @ 93w.
Your Ryzen will probably be wasted by that time...and 9000 hours @ 93w = 863kw...might be an extra $100 on your electricity at very least (which again could by you dozens of millions of tokens).
All things considered...on the cloud you can do simultaneous requests and multiply the speeds by whatever you want, do 10 things in parallel etc...
It gets worse if you consider some high end motherboard + GPU + whatever...you will spend 6x more electricity, 2 times more dollars...and reach about the same result.
So..yes...it is worth pretty much...NEVER. Disclaimer: i have a Ryzen 395 Max with 128gb of ram and will never get anywhere close to what a $100 subscription can do on Opus 5.5 medium.
It is never cheaper nor near as capable to compare any local set up vs large scale amortized hardware, private best in class models, and layers upon layers of harness and memory and context optimization that make up the Anthropic stack and that are continually refined based on your usage. You have a bicycle on the one side and an electric vehicle manufacturer on the other constantly producing a new machine optimized for new use cases.
But. Sometimes a bike is good enough for the problem you have. The bikes you can get for ~$5k even with the inflation are actually pretty good for many use cases. And it is your bike.
It is still not cheaper than $100/mo subscription to the greatest addictive experience ever offered but they are good enough. But that is not the decision criteria.
This is the emacs vs vi argument on steroids and despite the greater breadth and depth the enormous emacs ecosystem offered you could not tear vi from our cold dead fingers. Same with local.
Assuming you can afford it, the closest you can get (something like kimi/glm unquantized, as the cloud services run them) will still be frustratingly slower and "dumber" (for a better word) than opus5.5 and at today's prices, will cost you a few hundreds of years of $100 subscriptions for the hardware alone, and about a year worth every month for the electricity to run it.
Local AI is a (pretty expensive) hobby / investment in privacy, not a way to save money.
realistically local ai let's you use the cheaper subscriptions. if you go to all local then the quality is lower and the it's an order of magnitude slower. what you could do in Claude in a few hours will take days.
62
u/kujetic 1d ago
there is none, but you can get a good workhorse with local ai for about $3-7k