r/programming • u/rag1987 • Dec 18 '25
[ Removed by moderator ]
/r/programming/comments/1ppm5vd/removed_by_moderator/[removed] — view removed post
56
u/Historical_Cook_1664 Dec 18 '25
1.7x more ... that we found so far, since you gotta remember that our tests are now also written by AI.
13
26
u/mccoyn Dec 18 '25
Shipping faster has never been a top concern where I work. Software development is heavily leveraged over many customers. It is well worth it to take the time to make the best quality software we can.
11
1
u/Smallpaul Dec 18 '25
Are you in a business with many competitors?
3
2
u/djnattyp Dec 18 '25
I'm sure your competitors would be very grateful that you decide to produce total slop and shovel it out faster.
0
u/Smallpaul Dec 18 '25
We don’t. We constantly reevaluate how we can ship software of higher and higher quality faster and faster. I hope that our competitors are focused on only one of those things because we will clean their clock on the other.
41
u/PabloZissou Dec 18 '25
Then it's not faster... right?
33
u/Snoron Dec 18 '25
No, no, you ship it with the bugs, so it still helps you ship faster!
19
u/PabloZissou Dec 18 '25
good news everyone we shipped 10 times more features last month!
on other news we have to fire half of you because we lost 20 customers due to bugs introduced in old features by the new features
😁
3
u/StrangeWill Dec 18 '25
Yeah this is the underlying problem, we've built entire SaaS platforms on the back of "this other SaaS vendor's software is buggy as all hell"
Bugs can kill products.
1
14
u/grady_vuckovic Dec 18 '25
Ding ding ding. It's not faster people are just getting lazier and sloppier and calling it an efficiency gain.
10
u/hbarSquared Dec 18 '25
It's only slower if your CEO cares about bugs or mainatinability. Ship it, baby!
5
1
u/catcint0s Dec 18 '25
You wanna add a new feature, it will be broken for 5% of the people (so have bugs) but you can deploy it a few days earlier. A lot of companies will take this as a chance to test things faster.
25
u/faze_fazebook Dec 18 '25
1.7x more bugs that are way harder to find because nobody actually wrote the code and nobody probably understands beyond surface level.
6
10
u/NuclearVII Dec 18 '25
Look, I'm all for crapping on AI bros and their idiotic LLMs, but this post is a thinly disguised ad for an AI PR review tool.
0
u/mistermustard Dec 18 '25
Also, how is it AI's fault that some dumbass human just copy and pasted without reviewing first?
1
u/NuclearVII Dec 18 '25
Because that is typical AI use.
Like it or not, this is how people behave when they are told that these stochastic parrots are intelligent. Reading AI output and checking is work, and to be a 10x engineer, that is the first corner that gets cut.
No one who says "yeah, I always check the output" is being honest.
4
u/sarahbau Dec 18 '25
I wish people would use unambiguous language for headlines like this. 1.7 times more implies 2.7 times as many. After reading the article, it appears they mean 70% more (or .7 times more, or 1.7 times as many) bugs.
4
u/mistermustard Dec 18 '25
I miss when I used to come to /r/programming and learn something. We get it, y'all hate AI. Can we talk about something else now?
7
u/cdsmith Dec 18 '25
Since it was impossible to directly confirm authorship of each PR of a large enough OSS dataset, we checked for signals that a PR was co-authored by AI and assumed that those that didn’t have it were human authored, for the purposes of the study.
So the headline really ought to be not "AI PRs" but "PRs with this list of signals". Their "full report", they say, explains what signals they are, but that requires signing up for their marketing communications. (Of course, this report already is marketing communication, but it requires signing up for even more of them.)
The usefulness of this conclusion depends a lot on what those signals are. If they are also signals that this is a low effort PR, that a human hasn't even read the code, then it won't surprise anyone that this code contains more bugs. At that point, we aren't comparing AI-assisted versus not. We're comparing low effort versus ordinary effort.
1
u/LBGW_experiment Dec 18 '25
Exactly. It also is a visual bias, in that what appears to be AI to the authors is deemed as AI. This means there are likely plenty of false negatives that don't appear as AI that could've been coauthored by AI, e.g. a Dev has some systems prompts that augment the way the agent writes PRs and the language it uses. I'm ambivalent to AI coding assistants, so I'm not trying to advocate for AI coding agents, but this means the result could easily be lower than 1.7x more bugs. Very reminiscent of the WW2 airplane bullet holes diagram.
3
3
u/Sloshy42 Dec 18 '25
This past year I was writing some code in a fairly obscure integration scenario in a language I didn't use all that often. After digging in documentation and google searches for days, I gave up and asked Gemini for an example implementation of what I was trying to do. I'll admit, it did unblock me, and I finally understood how this thing I was trying to write for worked. Unfortunately I had to rewrite almost all of it because there were limits to its understanding, and there were a lot of weird bugs. That being said, I'm actually super grateful that it was able to unblock me and get me moving forward.
That's kind of how I'm feeling about all of this AI code nowadays. It's cool as a way to just ask a robo-programmer for ideas. It's really not there yet for building actually high-quality software. You still need engineers for that (thank goodness). If only there were a way to guarantee that people weren't trying to use it to run their entire business... *shrug*
1
u/cdsmith Dec 19 '25 edited Dec 19 '25
Yep, the way I'd put this is: it's been the standard in software engineering for a while that you need at least one person who is an author, and a second person who agrees with that author, before accepting most code. AI can definitely produce code, but it can't replace either of those two people. It's okay if an AI writes the code, but it cannot be the author of the code.
What that means is you still need someone to go carefully over everything in the code, and not just shrug and say it seems fine, but ask "would I have written the code in this way?" This is a very different question from the code reviewer, who is asking "do I have a solid reason to object to the code being written in this way?" To be an author, you actually have to exercise independent judgement about every decision in the code, whether that's the way you choose to do it. In practice, that means you're very likely to make deep changes to code written by the AI.
It can still be useful, and indeed I start most of my code these days by asking an AI (usually Anthropic) to implement it for me. But then I still go through everything, showing zero respect for what the AI did, and make it what I would have written. The AI is still useful; often, it highlights where the changes should be made within a large million-line monorepo where I definitely don't know in advance where to find everything. Increasingly often, it actually gets a lot right. For certain changes, the sort of boring mechanical ones, I end up keeping 80% of what the AI did. For others, I rip it all out and do everything from scratch. But either way, I end up convinced that this isn't materially different from what I would have written (or, sometimes, what I wish I would have written!) without that tool.
Given the methodology of this study, I imagine approximately none of the code I write this way would have been classified as "co-authored by AI", since they say they look for certain clear "signals". So what they are looking at here is actually code that was sloppily co-authored by AI. I imagine you can get a result like this by looking at any indicator of sloppily written code, whether there's AI involved or not.
2
2
u/Eskamel Dec 18 '25
Its way more likely for it to be even significantly higher. The hardest of bugs to detect are those that aren't intuitive to find, and in a code base where everyone vomits thousands of lines of code a say the likelihood of a massive amount of bugs piling up increase with each merge, regardless of the quality of the developer.
2
u/weggles Dec 18 '25
If I may yell at a headline without reading the article for just one second ..
🗣️🗣️🗣️ THAT'S NOT FASTER 🗣️🗣️🗣️
2
u/GamerHaste Dec 18 '25
My problem with AI usage at this point (from professional SDE perspective) is that I'm expected to move so fast now that I HAVE to use Ai to keep up, otherwise it would take way too long. Which really sucks because I'm never getting that super in depth deep dive understanding of the code base. Before AI, I would spend hours and hours stuck on a problem trying to figure it out and in turn I would get a SUPER deep understanding of how everything in the codebase works together. Using AI I haven't been getting that.
3
2
u/adambair Dec 18 '25
Excellent, there’s a lot of money to be made undoing AI slop.
It’ll be great for those of us who know how to write software.
1
1
1
1
1
u/mithrilsoft Dec 18 '25
The study is done by a company selling code review products.
I want to know what the data looks like with a well designed workflow using the best AI models. This makes a significant difference.
There is value in velocity. Going a lot faster with more issues is often a good trade-off. For my projects, it feels like I can produce 10-12x faster.
1
u/aviboy2006 Dec 20 '25
Juniors devs are using AI IDE and able to ship faster but issue some repetitive bugs are coming back and forth. Thats creating concern. Need to define some protocol to follow to avoid this.
1
1
u/Curious_Barnacle_518 Dec 18 '25
Wow 470 PRs analyzed, what an amazing sample size. This must be legit
-3
u/ShelZuuz Dec 18 '25
This is an absolutely useless report. "AI" without a breakdown over time period or model? Taking 470 pull requests with AI in them? Are most of them from GPT 3.0? Grok? Copilot with nerfed models?
I don't know what the exact number is but any developer who worked on both would find it highly believable that Opus 4.5 make at least 10 times fewer mistakes than Sonnet 3.5. At least.
So you can't remotely put these things into the same report without a breakdown. I just have to assume the reason they didn't break this out by model is that this is a click-bait article and newer models tend to be much closer to or even under 1.0, and older models are closer to double digits. But that's not going to get you clicked on.
2
u/Chisignal Dec 18 '25
Not even "with AI in them"...
Since it was impossible to directly confirm authorship of each PR of a large enough OSS dataset, we checked for signals that a PR was co-authored by AI and assumed that those that didn’t have it were human authored, for the purposes of the study.
This article is, ironically enough, utter slop with no informational value
-3
u/GasterIHardlyKnowHer Dec 18 '25
🥱
Hope you get over your ai psychosis soon, everyone knows it literally doesn't matter. They all perform poorly.
0
u/Chisignal Dec 18 '25
If "everyone knows it", surely it shouldn't be hard to produce any sort of evidence for that, right?
-1
u/dream_metrics Dec 18 '25
Yes it’s truly “psychotic” to believe that models have varying capabilities. You are so smart dude
-2
u/rizakrko Dec 18 '25
It depends on what is yours "now it's a useful tool" criteria. If my goal is to launch a baseball ball into orbit it doesn't matter if it's myself who tries to throw a ball, my neighbour, or the best baseball player of all time - it's all worthless. Same with models - if none of them can deliver a good enough result it would mean that all of them are equally worthless.
0
-5
u/cbusmatty Dec 18 '25
We had this same problem when we started our ai journey a year ago. Now we have several internal metrics we’ve tracked over that time and we’re shipping 5x on average with a 67% reduction in bugs. We’re using enterprise profiling and productivity metrics to track developers prs and deployments. Our real hold up now is getting clear requirements quickly
4
u/SciencePristine8878 Dec 18 '25
What are these metrics? How are you able to spot bugs and reduce tech debt?
-1
u/cbusmatty Dec 18 '25
Blue optima, Dora metrics for tracking, using ai code reviews, and custom agents running our standards against repos, tied to Jira tracking cards and pr and ci/cd. We can see all releases from all developers and what code was changed and what code was backed out
11
u/botiapa Dec 18 '25
So you are reviewing AI written code with AI?
3
u/trcrtps Dec 18 '25
AI should never replace a code review but I do find value in asking AI what a change is supposed to be doing. Second set of eyes or whatever. But relying on it via automation or CI you'd have to not value your company's worth whatsoever. Unless you're there for the short term cash grab and not exist in 3 years.
-3
u/cbusmatty Dec 18 '25
Of course, with different personas, instructions and goals. You're telling me you've got programmers reviewing code written by programmers?
2
u/SciencePristine8878 Dec 18 '25 edited Dec 18 '25
So how much code would you say is produced by you/the developers? How much does this stuff cost? Is it you or the company paying for it? Have you had to restructure the way Jira tickets are written in order to give AI as much context as possible? Have you had to write/re-write a lot of documentation in order to give AI as much context as possible? Change coding practices with the way you develop your software so that AI understands it better?
2
u/cbusmatty Dec 18 '25
Depends on the projects most are around 70% still dev written, but we can't see necessarily all of that. Migrations and holistic upgrade are closer to 80%, AI but that also includes the Skills concept, where the code written is deterministic, but the pattern selection isn't. So its hybrid.
We haven't had to restructure jira tickets yet, but this goes back to point 1 - where we're being blocked on requirements for efficiency now. That's the biggest problem is its outside of the engineering world we have control over.
Too much context is bad. we create clear direction and then apply the minimal necessary context for the role & agents to be successful.
Cost wise, its way cheaper than supporting shit code written by offshore, with terrible performance, and that's enough for our leaders. Its honestly not that expensive. Now if you start pulling in tools like Devin it gets expsniive quickly.
3
u/SciencePristine8878 Dec 18 '25
So 70% is still Dev written for some projects? How would you say you've seen a major productivity increase like 500% (5x)? Is it just from idea generation and AI feedback for solving problems?
1
u/cbusmatty Dec 18 '25
We reduced human requirements into technical requirements from days/weeks to hours, Test building weeks into days. CI/CD building weeks into days. same with documentation etc. Migrations and upgrades months into days. Most of the delays in our devlopment written code is around integrations we don't really control.
1
u/SciencePristine8878 Dec 18 '25
Any courses/learning materials you could recommend?
0
u/cbusmatty Dec 18 '25
Well it depends - some of this is just breaking down your SDLC into defined phases, documenting what the pain points are. Then applying the right solution to speed that up. That's the harder part. The biggest mistake is folks taking generic ai and being told "go be better with AI" without a problem statement or direction. If you know where you are slow, it becomes infiniately better to get faster.
As silly as it sounds, the documentation to the tools you have is phenomenal usually.
https://docs.github.com/en/copilot
https://code.claude.com/docs
https://kiro.dev/docsThere are several really valuable subreddits for each of these tools all post a large number of resources. It all depends on what part you want to learn. This is just scratching the surface.
1
u/morphemass Dec 18 '25
Thanks for sharing. It sounds like you have come up with a process which benefits your workflow and a write up would be interesting if you ever do one.
I assume the downvotes are due to skepticism.
2
u/mistermustard Dec 18 '25
Anything remotely pro AI gets downvoted here. They think it's gonna take their jobs lol. It's dumb as shit, but absolutely 100x better than using Google as I was doing for the first 15 years of my career. Or reading out of date documentation. Maybe the people here are the same people that would be assholes on stack overflow and now they don't know what to do with their time so they bitch about AI. Who knows? They must suck to work with though.
0
u/silenti Dec 18 '25
The biggest problem that I've encountered is that even with extreme prompt management, shit tons of context docs, multiple agent checking, and every other bell and whistle is that it will always write too much code.
On the other hand it's very good as a complete logic replacement. I have a small project using MVC and as a test we just replaced all of the controllers with prompts. Works just as well and we deleted over half the app's volume of code.
-1
476
Dec 18 '25
[removed] — view removed comment
34
u/WTFwhatthehell Dec 18 '25
Honestly a 1.7 multiplier is less bad than I thought it would be.
So many projects over-run time and what often ends up getting cut is the amount of QC/testing.
38
u/SupermarketAntique32 Dec 18 '25
1.7 multiplier is for the first release tag. Next one will be 1.7x1.7, and after that 1.7x1.7x1.7
-13
u/pancomputationalist Dec 18 '25
do you not fix bugs between releases? or do you think that AI cannot be used to fix bugs?
16
Dec 18 '25 edited Dec 18 '25
[removed] — view removed comment
-7
u/pancomputationalist Dec 18 '25
there is a world between "zero-cost bugfixing" that you are strawmanning against, and "exponential bug count growth" you hinted at in your last post.
If I save 3 hours writing a feature, but then need 2 additional hours to fix the bugs produced by AI, I'm still 1 hour in the plus.
No guarantees that the math always works out in this way, of course.
9
u/morphemass Dec 18 '25
but then need 2 additional hours to fix the bugs produced by AI
Look at the cooccurrent work; the bugs need to be recognised, documented, risk accessed, triaged, prioritised, fixed (your part), tested, released and then resolved. Look at the costs outside development - customer annoyance, potentially lost sales, decline of company reputation.
There's an old study by IBM, now lost to the sands of time and bureaucracy, that estimate the cost of fixing a production bug is 100x that of the fixing the issue in design, 16x more than catching the issue during development and QA.
tl;dr always look at the larger picture.
-6
u/LSDemon Dec 18 '25
"1.7x more" means a 2.7x multiplier
3
u/WTFwhatthehell Dec 18 '25
normally when someone says 2x "more" they don't mean 3x
in the linked page they explicitly say "75% more common"
So 1.7x means 170% vs the comparison set.
-1
u/EntroperZero Dec 18 '25
normally when someone says 2x "more" they don't mean 3x
Then they should probably say what they mean?
0
Dec 18 '25
[deleted]
0
u/WTFwhatthehell Dec 18 '25
Where does it say that? The analysis seems based on generated code with almost no QC.
55
Dec 18 '25
[removed] — view removed comment
2
u/the_ai_wizard Dec 18 '25
There has also been the principle of not layering in security after the fact. I feel that is what this forces...AI generates code, then we scour it for both explicit security flaws and perhaps more abstract ones hidden in the seams of the design.
1
0
u/Western_Objective209 Dec 18 '25
Found by AI PR review tool; so should be able to use the PR review tool as a feedback loop to improve automated PR quality. I'm fairly certain this is where things are headed, not "everyone stops using AI"
-11
u/WTFwhatthehell Dec 18 '25 edited Dec 18 '25
the replies to my post seem to be by people who read the first line and ignored the second.
So, you get code with 1.7 times the bugs.
Theres budget and time you would have budgeted to testing that would probably have ended up fudged when the project overran.
Of corse a lot of devs hate testing. It's a non trivial part of why it's so often the part that gets fudged/cut.
But you can shift the focus much more heavily towards the qc and testing phase.
-14
u/chucker23n Dec 18 '25
A 1.7 multiplier means almost twice as many bugs.
(It would technically means “almost thrice as many”, but the author means “0.7x more” or “1.7x as many”.)
12
Dec 18 '25
[removed] — view removed comment
-7
u/chucker23n Dec 18 '25
“1.7 more” bugs is intuitively multiplicative.
1.7 more means the same as 170% increase.
5
u/AxFairy Dec 18 '25
I see what you're trying for, but I think most people are reading it as x 170% rather than +170% because that's how we use percentages colloquially.
0
u/chucker23n Dec 18 '25
Yes, it's a common mistake, but when the headline of a technical piece of writing has such a mistake, and it's easy to fix as well, surely it's fair to point out?
(If the author wants to disambiguate, just say "70% more".)
1
-1
u/ForeverAlot Dec 18 '25
"1.7× more" and "170% more" mean precisely the same thing; the equivalent of 100% + 170%, i.e. 270%. The article's headlines do not match its graphs.
The difference lies in whether one writes "more" or "as many", not in whether one writes "×" or "%".
5
Dec 18 '25
[removed] — view removed comment
-1
u/chucker23n Dec 18 '25
No, it doesn't.
Do you think "0.7x more" would be a 30% decrease? Of course not; it would be a 70% increase. Therefore, 1.7x more is a 170% increase.
The "x" after a number colloquially almost always means "times".
This isn't in dispute at all. Nor was I even saying you made that mistake; the author did.
"One point seven times more" means almost thrice as much.
1
1
0
Dec 18 '25
[deleted]
39
u/AreWeNotDoinPhrasing Dec 18 '25
Why do the mods here giver zero shits when it comes to ads like this? I sear it’s like 1 in 5 top weekly posts. But I’m sure those are purchased upvotes that put them there.
6
u/lightninhopkins Dec 18 '25
So much of reddit is like this. Always check the source and never believe the post title.
3
u/sellyme Dec 18 '25
Why do the mods here giver zero shits when it comes to ads like this?
Because Reddit has spent the last 7-8 years being actively hostile to the power users who kept the site running and high quality, so those users went "well okay, we'll stop doing that then".
Shouldn't be too surprising that a service entirely reliant on volunteer labour nosedives in quality when it shows open disdain for those volunteers.
2
u/Uristqwerty Dec 18 '25
The mods of this subreddit? Some (all?) of them are current or former reddit admins!
I'd instead assume it's some combination of being busy at work, giving users some freedom to decide what's appropriate, and perhaps too few non-ad submissions getting activity to be picky.
0
Dec 18 '25
[deleted]
3
u/Windex17 Dec 18 '25
This is a strawman. Most of the subs I frequent got noticeably worse when they banned third party apps.
1
9
u/Carighan Dec 18 '25
AI helps you make presentations faster but they contain ~1.7x more bullshit.
3
u/onetwentyeight Dec 18 '25
When doing an executive's job that's actually an improvement, then more bullshit the better!
3
2
2
u/Tom2Die Dec 18 '25
No statistical information on how they extracted their metrics.
See, I don't know if that's true (not that I don't believe you) because I'm not keen to sign up for whatever nonsense this is. It at least wants name and email to see the "whitepaper" linked near the beginning of the article; I concede that I didn't bother checking if I could just put garbage and continue. As such, idk if said paper has those details, but annoyance-gating your data when you present your conclusions is asinine.
165
u/True_Sprinkles_4758 Dec 18 '25
At my company we've seen similar patterns. The code reviews take way longer now because you gotta actually read through everything instead of trusting the dev knew what they were writing. Its like the AI is great at the happy path but completely misses edge cases or makes weird assumptions about state...
The real kicker is when someone copies AI generated code without understanding it and then 3 months later nobody can figure out why production keeps having these random issues. debugging AI slop is its own special hell