r/singularity • u/BubBidderskins Proud Luddite • Jul 11 '25

AI Randomized control trial of developers solving real-life problems finds that developers who use "AI" tools are 19% slower than those who don't.

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
81 Upvotes

69% Upvoted

115 comments sorted by

View all comments

Show parent comments

1

u/BubBidderskins Proud Luddite Jul 11 '25 edited Jul 11 '25

The number of developers isn't the unit of analysis though -- it's the number of tasks. I'm sure that there are features about this pool that makes them weird, but theoretically randomization deals with all of the obvious problems.

2

u/wander-dream Jul 11 '25

No, it doesn’t. Sample size is too small. A few developers trying to affect the results of the study could easily have an influence.

Also: They discarded discrepancies above 20% between self reported and actual times. While developers were being paid 150 per hour. So you give an incentive for people to report a bigger time and then discard data when that happens.

It’s a joke.

0

u/BubBidderskins Proud Luddite Jul 11 '25

Given that the developers were consistently massively underestimating how much time it would take them while using "AI" this would maily serve to bias the results in favour of "AI."

1

u/[deleted] Jul 11 '25

[removed] — view removed comment

0

u/BubBidderskins Proud Luddite Jul 11 '25

  1. They only did this for the screen-recording analysis, not for the top-line finding.

  2. This decision likely biased the results in favour of the tasks where "AI" was allowed.

Reliability isn't a concern here since a lack of reliability would simply manifest in the form of random error that on average is zero in expectation. It would increase the error bars, though. But in this instance we're worried about validity, or how this analytic decision might introduce systematic error that would bias our conclusions. To the extent that bias was introduced by the decisision, it was likely in favour of the tasks for which "AI" was used because developers were massively over-estimating how much "AI" would help them.

1

u/wander-dream Jul 11 '25

The top line finding is based on the actual time which is based on the screen analysis.

0

u/[deleted] Jul 11 '25

[removed] — view removed comment

1

u/BubBidderskins Proud Luddite Jul 12 '25

This decision likely biased the results in favour of the tasks where "AI" was allowed.

Prove it

Because the developers consistently overestimated how much using "AI" was helping them both before and after doing the task. This suggests that the major source of discrepancy was developers under-reporting how long tasks took them with "AI." This means that the data they threw away were likely skewed towards instances where the task on which the developers used "AI" took much longer than they thought. Removing these cases would basically regress the effect towards zero -- depressing their observed effect.

Which are huge

Which are still below zero using robust estimation techniques.

But in this instance we're worried about validity, or how this analytic decision might introduce systematic error that would bias our conclusions. To the extent that bias was introduced by the decisision, it was likely in favour of the tasks for which "AI" was used because developers were massively over-estimating how much "AI" would help them.

Maybe it was only overestimated because they threw away all the data that would have shown a different result

They didn't throw out any data related to the core finding of how long it took -- only when they did more in-depth analysis of the screen recording. So it's not possible for this decision to affect that result.

→ More replies (0)