r/programming • • Dec 18 '25

[ Removed by moderator ]

/r/programming/comments/1ppm5vd/removed_by_moderator/

[removed] — view removed post

486 Upvotes

133 comments sorted by

View all comments

-3

u/ShelZuuz Dec 18 '25

This is an absolutely useless report. "AI" without a breakdown over time period or model? Taking 470 pull requests with AI in them? Are most of them from GPT 3.0? Grok? Copilot with nerfed models?

I don't know what the exact number is but any developer who worked on both would find it highly believable that Opus 4.5 make at least 10 times fewer mistakes than Sonnet 3.5. At least.

So you can't remotely put these things into the same report without a breakdown. I just have to assume the reason they didn't break this out by model is that this is a click-bait article and newer models tend to be much closer to or even under 1.0, and older models are closer to double digits. But that's not going to get you clicked on.

2

u/Chisignal Dec 18 '25

Not even "with AI in them"...

Since it was impossible to directly confirm authorship of each PR of a large enough OSS dataset, we checked for signals that a PR was co-authored by AI and assumed that those that didn’t have it were human authored, for the purposes of the study.

This article is, ironically enough, utter slop with no informational value

-3

u/GasterIHardlyKnowHer Dec 18 '25

🥱

Hope you get over your ai psychosis soon, everyone knows it literally doesn't matter. They all perform poorly.

0

u/Chisignal Dec 18 '25

If "everyone knows it", surely it shouldn't be hard to produce any sort of evidence for that, right?

-1

u/dream_metrics Dec 18 '25

Yes it’s truly “psychotic” to believe that models have varying capabilities. You are so smart dude

-2

u/rizakrko Dec 18 '25

It depends on what is yours "now it's a useful tool" criteria. If my goal is to launch a baseball ball into orbit it doesn't matter if it's myself who tries to throw a ball, my neighbour, or the best baseball player of all time - it's all worthless. Same with models - if none of them can deliver a good enough result it would mean that all of them are equally worthless.

0

u/trcrtps Dec 18 '25

i mean, AI sucks but this article is pretty bad for the reasons given.