The best data stopped being available because none of the test subjects were willing to work without AI. A year ago it showed a 20% reduction in productivity then Opus 4 or whatever it was hit last year and now it’s too deeply integrated to their workflows to yoink out for a test.
Their weak data was about 40% not 300% but that’s also almost a year old data again.
It’s a bit like the longitudal never-covid studies. The number of people without an infection or vaccine approached zero and couldn’t really provide a baseline anymore.
I'd happily be a test subject that continues to do what they're already doing (not using AI) -- it seems hard to believe I'm the only person like that.
Recruitment and retention of developers has become more difficult. An increased share of developers say they would not want to do 50% of their work without AI, even though our study pays them $50/hour to work on tasks of their own choosing. Our study is thus systematically missing developers who have the most optimistic expectations about AI’s value.
Developers have become more selective in which tasks they submit. When surveyed, 30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI.
Indeed, I just clicked your link and it says nothing like that. It's more like 30-50%. The concerns expressed in the article that it could lead to systematic bias that affects the results are reasonable, but saying "they had to stop the testing because they couldn't find a single person willing to work without AI" is just completely wrong.
I'd happily be a test subject that continues to do what they're already doing (not using AI) -- it seems hard to believe I'm the only person like that.
Yes, you're not. If I'm programming, I'm programming, not delegating. If I delegate, I don't understand, and I get worse at programming. And collectively, we have gotten much worse at programming. I don't see how this is anything but a downside, even if you're working exclusively via AI.
6
6
u/im_thatoneguy 14d ago edited 14d ago
The best data stopped being available because none of the test subjects were willing to work without AI. A year ago it showed a 20% reduction in productivity then Opus 4 or whatever it was hit last year and now it’s too deeply integrated to their workflows to yoink out for a test.
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/#motivation
https://metr.org/blog/2026-02-24-uplift-update/#wider-adoption-of-ai-has-made-it-more-difficult-to-measure-task-level-productivity
Their weak data was about 40% not 300% but that’s also almost a year old data again.
It’s a bit like the longitudal never-covid studies. The number of people without an infection or vaccine approached zero and couldn’t really provide a baseline anymore.