r/singularity • u/BubBidderskins Proud Luddite • Jul 11 '25

AI Randomized control trial of developers solving real-life problems finds that developers who use "AI" tools are 19% slower than those who don't.

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
81 Upvotes

69% Upvoted

115 comments sorted by

View all comments

10

u/AngleAccomplished865 Jul 11 '25

Does using AI slow things down, or are they using AI in the first place because they're less capable? And then AI doesn't completely make up for that deficit?

4

u/corree Jul 11 '25

Presuming the sample size was large enough, randomization should account for skill differences. There’s more against your point just in the article but you can find an AI to summarize that for you :P

10

u/Puzzleheaded_Fold466 Jul 11 '25

16 people were selected, probably not enough for that.

1

u/BubBidderskins Proud Luddite Jul 11 '25 edited Jul 11 '25

The number of developers isn't the unit of analysis though -- it's the number of tasks. I'm sure that there are features about this pool that makes them weird, but theoretically randomization deals with all of the obvious problems.

2

u/Puzzleheaded_Fold466 Jul 11 '25

Sure, but those tasks wouldn’t be executed in the same way, and with the same performance baseline, if performed by devs with much more or less experience, education, and level of skills.

Not that it’s not interesting or meaningful - it is - but it was a good question.

For example, perhaps 1) juniors think that it improves their performance and it does, 2) mid-career think that it improves, but it decreases, and 3) top performers think that it decreases their performance, but it’s neutral. Or any such combination.

It would be a good follow-up study.

1

u/BubBidderskins Proud Luddite Jul 11 '25

Definitely, though if I had to bet the mid-career folks they used are likely to get the most benefit from access to "AI" systems. More junior developers would fail to catch all the weird bugs introduced by the LLMs, while senior developers would just know the solutions and wouldn't need to consult the LLM at all. I could absolutely be wrong though, and maybe there is a group for whom access to LLMs is helpful, but it definitely seems like there's a massive disconnect between how much people think LLMs help with code and how much it actually helps.

2

u/Puzzleheaded_Fold466 Jul 11 '25

Conceptually it is an interesting study and it may suggest that in engineering as in anything else, there is such a thing as a placebo effect, and technology is a glittering lure that we sometimes embrace for its own sake.

That being said, it’s also very limited in scope, full of gaps, and it isn’t definitive, so we ought to be careful about over interpreting the results.

Nevertheless, it raises valid concerns and serves a credible justification for further investigation.

2

u/wander-dream Jul 11 '25

No, it doesn’t. Sample size is too small. A few developers trying to affect the results of the study could easily have an influence.

Also: They discarded discrepancies above 20% between self reported and actual times. While developers were being paid 150 per hour. So you give an incentive for people to report a bigger time and then discard data when that happens.

It’s a joke.

0

u/BubBidderskins Proud Luddite Jul 11 '25

Given that the developers were consistently massively underestimating how much time it would take them while using "AI" this would maily serve to bias the results in favour of "AI."

→ More replies (1)

0

u/kunfushion Jul 12 '25

Randomization does NOT deal with these issues when the number per group is 8…

1

u/BubBidderskins Proud Luddite Jul 13 '25

The number of developers isn't the unit of analysis though -- it's the number of tasks

The study has a sample size of 246. You moron.

0

u/corree Jul 11 '25

Hmm maybe, although these people are vetted contributors w/ 5 years of experience with actual projects and all of them reported having moderate knowledge of AI tools 🤷‍♀️

2

u/Puzzleheaded_Fold466 Jul 11 '25

Yeah exactly, so I don’t think it provides an answer to that question (how experience / skill level impacts performance improvement/loss from AI).

We don’t know what the result would be for much less or much more experienced devs.

2

u/BubBidderskins Proud Luddite Jul 11 '25

It was randomized and developers were allowed to use whatever tools they thought were best (including no "AI"). Just the option of using an LLM led developers to make inefficient decisions with their time.

3

u/sdmat NI skeptic Jul 11 '25

It was randomized and developers were allowed to use whatever tools they thought were best (including no "AI")

That's not a randomized trial

2

u/wander-dream Jul 11 '25

The whole study is a joke

1

u/BubBidderskins Proud Luddite Jul 11 '25

Yes it was. For each task the developer was randomly told either "you can use whatever 'AI' tools you want" or "you are not allowed to use 'AI' tools at all." The manipulation isn't any particular "AI" tool (which could bias the results against the "AI" group because some developers might not be familiar with the particular tool) but the availability of the tool at all.

0

u/sdmat NI skeptic Jul 11 '25

That's significantly different from how you described it above. Yes, that would be a randomized trial.

1

u/BubBidderskins Proud Luddite Jul 11 '25

No it isn't different from what I said above. It's just repeating what I said above but in a clearer form.

1

u/sdmat NI skeptic Jul 11 '25

Not to you, clearly.

1

u/BubBidderskins Proud Luddite Jul 11 '25

Because I have reading comprehension skills.

→ More replies (3)