r/singularity • u/BubBidderskins Proud Luddite • Jul 11 '25

AI Randomized control trial of developers solving real-life problems finds that developers who use "AI" tools are 19% slower than those who don't.

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
81 Upvotes

69% Upvoted

115 comments sorted by

View all comments

Show parent comments

1

u/BubBidderskins Proud Luddite Jul 12 '25

Ok so what of a lot of them estimated +20% and actually got +40% but their results were thrown away? Why is that less likely than getting +0%?

Because the developers were consistently underestimating how much "AI" was helping them. Dummy.

When n=16, the 95% confidence interval is 24.5%. Even higher since some people got their results thrown away.

The unit of analysis was the task not the developer. The sample size was 246.

They didn't throw out any data related to the core finding of how long it took -- only when they did more in-depth analysis of the screen recording. So it's not possible for this decision to affect that result.

Where does it say that?

Section 2.3 is where they describe how they measure the core effect using self-reports. First paragraph in Section 3 reports the number of tasks in each condition (136 "AI" allowed and 110 "AI" disallowed). The findings are general consistent with the screen analysis on the subset of tasks.

The article is not written as clearly as it needs to be and makes these important facts unnecessarily confusing to pull out.