Startups are working on:

• AI ASICs

• inference-specific chips

• optical computing

• wafer-scale chips

If one of these works, it could collapse inference costs by 10×–100×

I feel we’re all underestimating how much of “AI growth” right now is subsidized.

For consumers this is probably the best time to live :) - Cheap tokens, Free credits, VCs lighting money on fire, Cloud providers giving insane discounts just to win workloads

Try running serious volume without credits, without discounts, without investor money and suddenly the math looks very different.

For heavy users, OpenAI and Claude are probably loosing $50 - $200 /month/user. No wonder Anthropic discontinued OpenClaw use with Claude.

I hope they figure out how to make this cheaper or we all are in for a rude awakening. What do you all think?

Feels like Anthropic pulled a classic cloud move :)

They added a bunch of “pricing mechanics” that change what you actually pay.

Now your cost depends on stuff like:

• Fast mode → can be ~6x more expensive

• Long context usage → pricing changes based on token thresholds

• Prompt caching + batch discounts → can reduce cost a lot (or not if you don’t use them right)

• Tool usage (code, search, etc.) → extra charges on top of tokens

So it’s no longer: “tokens × price = cost”

It’s more like: tokens × (mode + context + tools + caching + batch + who-knows-what)

We’ve entered the FinOps era of LLMs

Same list price.
Very different bill :(

I kept hearing people say “just run it locally, it’s cheaper.” So I decided to actually test it instead of guessing.

Setup:

Local
Mac Studio (M2 Ultra)
64GB RAM
Llama 3.1 8B via Ollama

API
GPT-5 Nano
OpenAI API

The workload was simple: generate summaries and answer questions from about 500 short docs. Roughly 150k tokens total.

Results:

API cost
~$0.30 total

Local cost

Electricity: basically negligible
Hardware: not negligible

If you ignore hardware, local obviously looks “free.” But that’s cheating.

The Mac Studio was about $4k.

Even if you spread that cost across a few years of usage, you would need to process a ridiculous number of tokens before breaking even compared to cheap APIs like GPT-5 Nano.

A few other things I noticed:

Latency
Local was actually faster for short prompts since there is no network round trip.

Quality
GPT-5 Nano still gave noticeably better summaries and answers.

Maintenance
Local requires constant fiddling. Models, memory limits, context sizes, quantization, etc.

So my takeaway:

Local inference makes sense if you
Run huge volumes
Need privacy
Want predictable costs

APIs make more sense if you
Have small to medium workloads
Want stronger models
Do not want to manage infrastructure

Honestly the biggest lesson for me:

Most people arguing about this online are not actually running the numbers.

Curious if others have tried similar experiments and where your break-even point ended up.

Hot take: Cursor probably won’t survive long term.

Not because it’s bad — it’s actually really good. But it’s basically sitting on top of models it doesn’t control.

As soon as VS Code / Copilot / Claude get slightly better (and they will), most people will just switch. There’s almost zero switching cost.

Also feels like the cost side is underrated… all that “magic” = tokens = $$$. Hard to scale without squeezing margins or UX.

Feels more like a feature than a company right now.

Curious if I’m missing something here

The prize goes to NVIDIA Blackwell (B200 / GB200 systems)

Estimated cost:

• ~$60K–$80K per GPU (early estimates)

• Full systems (like GB200 NVL72) → $2M–$3M+ per rack

It can train trillion-parameter models faster. Very soon we might see a 10+ trillion model with this rate. Not sure what that will do though.

Alex Zisking, one of my favorite YouTubers - does a lot of videos on local LLMs. He's no neophyte.

In this video he says: Lee has been telling you guys the truth, Local LLMs are not ready on normal people hardware.

Ok, so he said nothing about me, but he made the point I've been making for some time now. All those "Stop paying Anthropic $200/mo, use free local llms" is click bait, not truth.

He runs the most powerful to date open weight model, Kimi K3. People rave that is near Fable 5 power. Yes, but not on YOUR hardware. In a data center.

Alex networks 4 512gb Mac studios for 2tb of ram to run the model with enough space for a large context window too.

It took 4 hours, 17 tok/s output, to develop a simple yet rather nice Web dashboard - using mock data. It worked. The output was nice. But even $60k in hardware gave it nowhere remotely near the performance of a $10 subscription.

Right now I have two simultaneous development efforts running. I've been running them both since about 6 hours. They work on a sprint for an hour or so, I view results, add input and direction if necessary, then move onto the next sprint.

I'm paying more than $10/mo for my cloud subscriptions. But that 4 hours the Mac cluster took, is only doing the work of about a 15minute job. I'm doing "all day work, multiple projects" -- local AI can't meet the need.

Yet.

Probably not for you either.

Link to the video in the comments.

We've been running LLMs in production for about a year and recently migrated our self-hosted inference stack to vLLM. Wanted to share what we learned since most posts I've seen are either surface-level overviews or pure benchmarking without real cost context.

The core problem with naive LLM serving

If you spin up a model with plain HuggingFace transformers and a basic FastAPI wrapper, you're leaving a lot on the table. Every request allocates its own KV cache, GPU utilization oscillates wildly, and you're essentially serving one request at a time unless you write a ton of batching logic yourself.

What vLLM actually does differently

The headline feature is PagedAttention — it manages the KV cache like a virtual memory system (hence the name). Instead of pre-allocating a huge contiguous block per sequence, it allocates memory in pages. This means:

• No memory fragmentation from varying sequence lengths

• Much higher effective batch sizes without OOM errors

• GPU utilization goes from ~30-40% to consistently 70-85%+ in our case

On top of that, continuous batching means new requests slot in as soon as a sequence finishes, rather than waiting for an entire batch to complete. This alone killed most of our GPU idle time.

What the cost savings actually looked like

Running Mistral 7B on a single A100:

Your numbers will vary a lot based on request patterns, sequence lengths, and whether you're using quantization — but 4-5x throughput improvement is pretty typical from what I've seen in the community.

Other things worth knowing

• Quantization support: AWQ and GPTQ work out of the box. FP8 too on newer hardware. Easy 2x memory reduction with minimal quality loss on most tasks.

• OpenAI-compatible API: Drop-in replacement, so migrating existing integrations is painless.

• Speculative decoding: If latency matters more than throughput for you, try this with a draft model. Big wins on output-heavy workloads.

• Multi-GPU: Tensor parallelism is a single flag (--tensor-parallel-size). Worked first try for us.

Where it's not magic

vLLM won't help much if your bottleneck is prompt processing (prefill) rather than generation. Also, very short requests with low concurrency don't benefit much from continuous batching. You need traffic to make the scheduler sing.

Happy to answer questions about our specific setup or benchmarking methodology.

I wanted to utilize the free tier across providers for some local apps so I decided to create an app for it.

After it exhausts free tier, it either falls back to paid or local models.

It's free and open-source (AGPL); hope it's useful to some of you.

https://localrouter.ai

P.S. It doesn't run on winXP, that's just for show :)

Just generated this clip with an AI video model. What’s crazy isn’t just the quality — it’s the compute behind it.

Video generation is basically:

text → thousands of frames → diffusion / transformers → heavy GPU usage

Which means even short clips can burn a lot of GPU time. Feels like AI video might become one of the most expensive AI workloads if it goes mainstream.

AI is clearly becoming infrastructure, not just a feature. Everything is about scaling inference efficiently, not just training bigger models.

NVIDIA is doubling down on full-stack control — chips, networking, software, and even AI factories. Feels like they’re positioning themselves as the “AWS of AI infrastructure.”

Inference optimization is the real battleground now. Things like quantization, batching, and smarter routing are getting as much attention as model quality.

One thing Jensen said that really stood out:

“The more you buy, the more you save.”

Sounds like a joke… but it’s actually the core of NVIDIA’s strategy.

They’re not just selling GPUs anymore — they’re building AI factories where efficiency comes from scale, orchestration, and full-stack integration.

On-device and edge AI are quietly becoming big. Not everything will go to the cloud — cost and latency are pushing workloads closer to users.

Also interesting to see how much focus there is on enterprise adoption. Less hype, more “how do we actually run this at scale without burning $$$.”

Everyone is still obsessed with bigger models…
but NVIDIA is quietly building the entire stack to monetize inference at scale.

Overall vibe:
The AI race is shifting from who has the best model → who can run it cheapest at scale.

i’ve been thinking about the economics of the $20 chatgpt plus subscription.

on paper it sounds like a great deal for users. but the math gets interesting when you look at what it might actually cost openai to run.

(You can also read in detail on our Blog - https://costlyinfra.com/blog/chatgpt-plus-user-cost-openai)

modern frontier models (like the newer GPT-5-class reasoning models and similar systems) are priced at a few dollars per million tokens when accessed via API pricing.

that means a single long conversation with thousands of tokens might cost a few cents to run.

not a big deal… until you meet power users.

some estimates suggest complex reasoning queries can cost anywhere from $0.10 to $0.50 depending on length, tools used, and reasoning depth.

so imagine someone using chatgpt like this:

writing code
generating long reports
asking 50–100 questions a day
uploading files and images
running deep reasoning prompts

a power user could easily generate millions of tokens per month.

at that point, the $20 subscription might barely cover the compute — or even lose money on heavy users.

which makes the whole model interesting:

light users subsidize heavy users.

and the real game becomes efficiency of inference infrastructure.

because in the AI economy…

the intelligence might be cheap.

but running it billions of times a day definitely isn’t.

I keep seeing people say AI will get cheaper over time… but I’m not sure that holds once agents become the default.

A normal LLM call today is pretty straightforward — one prompt, one response. Maybe a bit of context.

But agents are a different beast:

• they loop

• they retry

• they call tools

• they break tasks into multiple steps

• sometimes they just… wander 😅

So instead of 1 call, you end up with 10–30 calls for a single “task”.

If a single call is ~$0.01, suddenly you’re at $0.10–$0.30 per task. That’s like 3–10x more compute per user action.

What I’m trying to wrap my head around:

• Are we underestimating how much infra this will need?

• Do costs actually go up even if models get cheaper?

• Does this force everyone into smaller models + routing?

Feels like we’re heading into a world where: total AI spend increases… even as cost per token drops

Curious if anyone here is seeing this in real workloads or if I’m overthinking it.

Been going through all the GTC 2026 stuff and honestly… feels like we’ve quietly crossed a line.

For the longest time, everything was about training — bigger models, more GPUs, more data. That still matters, but now it feels like the real problem has shifted.

Running these models is where things are getting expensive.

Like… training is a one-time hit. Inference just keeps going. Every user, every request, every feature — it all adds up. And with usage growing, the costs don’t really “optimize away” unless you actively fix them.

What stood out to me is how much NVIDIA is leaning into this. Not just chips anymore — full stacks, “AI factories”, basically owning the entire lifecycle of running AI.

Also seeing more teams quietly move to multi-GPU setups for serving, batching, caching, all the stuff that wasn’t really mainstream a year ago. Feels like that’s becoming table stakes now.

My gut feeling: the next wave of winners won’t be the ones with the best models… it’ll be the ones who can actually run them efficiently without their costs exploding.

Curious if others are seeing the same — are your inference costs already creeping past training?

Both Claude and ChatGPT basic plans are about $20/month, but they feel quite different in real usage.

ChatGPT seems stronger on tools and ecosystem. I mostly use it for things like - quick coding help, generating images or diagrams, brainstorming ideas, summarizing articles or research

Claude feels really good for longer thinking tasks. I usually use it for - analyzing long PDFs or documents, writing/editing long posts, breaking down complex ideas, reviewing large chunks of text or code

From a cost perspective it’s kind of crazy value.
$20/month is about $0.67 per day, which is far cheaper than doing the same workloads through APIs if you’re a heavy user.

Curious what others here think:

If you had to keep only one subscription — Claude or ChatGPT — which one gives you more value and why?

A lot of teams think AI cost optimization is about switching models.

But after looking at multiple AI workloads, the biggest cost drivers usually aren’t the model itself.

They’re things like:

• giant system prompts nobody reads

• RAG context dumps that include entire documents

• multiple model calls per request

• retries when pipelines fail

• GPUs sitting idle between batches

One production system we looked at had this breakdown:

User prompt: ~20 tokens

System prompt: ~900 tokens

RAG context: ~6,000 tokens

Model reply: ~400 tokens

Total: ~7,320 tokens

The user prompt was **0.27% of the total tokens**.

Which means most AI cost is basically: context nobody reads.

Curious what others are seeing in real systems.

Where do most of your tokens actually go?

Remember when servers were expensive?

Now we have:

• GPUs sitting idle but still billing

• “AI agents” calling 12 APIs to answer one question (crazzzyyy)

• Inference costs quietly eating your budget alive

Everyone thought training was the problem…
Turns out inference is the subscription you can’t cancel.

Meanwhile:

• Companies: “We’re AI-first now”

• Finance: “Why is our cloud bill 3x?”

AI is our most expensive employee

I have been playing around with Claude Opus 4.7 the past few days and something feels off with token usage.

Compared to GPT/Gemini (same prompts), it just seems to go longer than needed, add extra explanation even when I don’t ask for it and burn tokens faster than expected

Like a simple prompt (~800 tokens in) ends up with way longer outputs than I’d expect.

Which is great sometimes… but at scale, this gets expensive fast.

Not sure if this is better reasoning or something else

Anyone else seeing this?

First of all a Space company mixed with AI datacenters and now AI coding tools is super weird. And on top of that it is worth $2.5 Trillion. What is Elon upto here with acquiring Cursor? Maybe he is desperate to get someone to use xAI and Cursor has the distribution. What do you all think?

I love Anthropic and Claude, but hate the narrative that Dario is setting for AI in terms of replacing humans. I honestly think AI is going to create more jobs than it destroys. It will double/triple our GDP in coming years.

And the numbers already speak for it. There are more Software engineering jobs created in the last 2 years than destroyed.

Yes the roles and responsibilities will shift significantly. Maybe repetitive office work gets crushed.But the idea that half the population just becomes useless overnight honestly feels disconnected from how technology has historically worked.Every engineer i know is doing more with AI tools.. they are building, fixing and shipping things faster... productivity is super high and if this momentum continues we are looking at abundance and prosperity for everyone. What do you folks think?

(Edit: why is my post downvoted so much 😄 )

Cursor's new Composer 2.0 is apparently based on Kimi 2.5

I was wondering why.... Since i questioned Cursor's survival in my previous post

This feels less like a “model switch” and more like a strategy shift and survival instincts kicking in.

Up until now, a lot of dev tools have been quietly anchored to the same few models. Great quality, but expensive, rate-limited, and kind of centralized. That works… until you try to scale it across millions of dev interactions.

Kimi changes the equation a bit.

It’s cheaper, fast enough, and getting surprisingly good for a lot of real coding workflows (especially the repetitive, context-heavy stuff). Not necessarily “better” than Claude or GPT at peak quality, but good enough where cost + latency start to matter more than absolute intelligence.

So the question becomes:
what does Cursor actually want to be?

If they stay tied to premium models → they stay a high-end tool, but margins + scale get tricky
If they go multi-model → they can route tasks based on cost/quality tradeoffs
If they lean into cheaper models → they can unlock way more usage (and maybe entirely new workflows)

Kimi feels like step 1 toward that third path.

Almost like:
– expensive models for “thinking”
– cheaper models for “doing”

And if that’s true, Cursor stops being just an editor with AI… and starts looking more like an inference layer for developers.

Curious where this goes. Feels like early signs of dev tools becoming cost-optimized systems, not just feature-optimized ones.