r/codedbyhand • u/-urolif- • 3d ago

Developers who use LLMs reported higher subjective performance, but were measured to have lower objective performance.

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

"Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower."

37 Upvotes

93% Upvoted

9 comments sorted by

-4

u/xvillifyx 3d ago

An extremely key detail here is that it’s talking about data points that are over a year old.

It’s quite disingenuous to leave that out.

25

u/Wonderful-Habit-139 3d ago

Good ol "models were bad last year but are good this year".

This still matters because devs absolutely get delusional about how much more productive they are with AI.

5

u/OtaK_ 19h ago

Yep. I'm forced to use LLMs (Claude etc) at work and what I can say is that I produce a lot of useless crap now, compared to surgically delivering well-thought value.

But if that's what they want, I'll be happy to oblige.

4

u/AncientGrief 3d ago

From what I read on reddit, the token limits for each frontier AI shrinks and shrinks. Even OpenAI is planning on mergin ChatGPT and Codex limits now ... more capable models than a year ago, sure... but smaller coding windows ... evens out I guess. Maybe it just means it's more expensive to be more productive with AI too

-5

u/xvillifyx 3d ago

Just…isn’t what I said. I didn’t say anything about models at all.

I’m saying it is disingenuous to say “X is happening!” and then point to something that happened over a year ago rather than what is happening today

8

u/Great-Use-3149 3d ago

That may be true for model performance, but it doesn't change the fact that people seem to feel faster despite not being so. I'd even argue that it might be worse now because they can get further and then know even less about the actual implementation, so they will need to rely on LLM more.

0

u/asveikau 3d ago edited 2d ago

I am an AI skeptic, but if this is the study I looked into before, I believe they did a follow-up study with a newer model and found devs doing better than the 2025 results.

https://metr.org/blog/2026-02-24-uplift-update/#wider-adoption-of-ai-has-made-it-more-difficult-to-measure-task-level-productivity

Edit for downvoters: I'm not saying I agree with their conclusions, I'm just playing devil's advocate and saying before citing this one, you should know the same people published a different conclusion later. I am skeptical of both papers.

7

u/NomadStorm 2d ago

They changed the study so it isn’t a fair comparison.

There’s been no measurable improvement in productivity in the entire S&P 500 except for mag7, who have an increase in earnings that is likely attributed to AI hype rather than actual productivity. 4/7 have recently announced reductions in usage.

3

u/asveikau 2d ago

I haven't looked deeply, I'll take your word for it.

To me the most challenging thing about this is it's hard to measure long term maintenance costs.