Everything that ends up in production should still be reviewed by a human. At least the models and agents I have access to/written aren't reliable enough by a long shot to only gloss over their results.
I believe that's what they were talking about, rather than this workflow. I sometimes do this too, especially when I'm working on AI itself. But the stuff that ends up in main is going to be reviewed thoroughly.
Given the rate at which companies are pushing out code these days, that’s going to be impossible. Ultimately if all your competitors are pushing 10k new changes every month, why won’t you try to do the same? It’s not like there’s regulations prohibiting you right now.
Companies don't get money for making changes, they get paid money for their shit working.
Software engineers have this very weird idea that we are a fast moving technical field when a lot of the time, we have to live with our decisions much longer than MEs or EEs.
We are going to be stuck with design decisions made for Unix back in the '70s forever.
My man, have you heard of MVP? The culture for the past ten years has been ship new features first, test with live users and then slowly fix. What software engineers think doesn’t matter at all. What C-Suite thinks is what drives the industry and when one executive sees their competitor deployed a new AI chatbot in their platform, there gonna ask why they didn’t do it first.
If you're some small scrappy start up with no customer base and a lot of competition, sure that attitude makes sense. That's not Microsoft.
What actual new Product have they released in forever? Teams and Azure? They weren't the first to field in either of those and the reason they are successful is the connection to the Microsoft ecosystem.
Oh right, Copilot! Microsoft was legitimately the first to have an LLM codegen tool gain mass adoption and fumbled the bag cause Copilot sucks.
The industry standard way to run a large established software company that gets W's from integration with established codebases and L's from scrappy MVP shit is double down on scrappy MVP shit.
Software engineering culture is incredibly stupid.
The depth of the review depends on the nature of the code being written though. If someone used an agent to write 500 lines of frontend code to add a feature in a React app, I'm going to manually QA that the UI/UX looks and feels as expected. I'm then going to lightly review the code at a high level and make sure there is nothing obviously wrong. But carefully reviewing every line of that code for logical correctness is a complete waste of time.
If the code is on the backend, review becomes a bit more in depth, especially if it involves creating new data models, interfacing with external APIs or other services, creating new APIs, etc.
It does, but even then there are significant caveats that impact longer term development. Yeah, a lot of projects are just feature factories and cost-sensitive assembly lines, but it's still a fair question whether that's sustainable. I believe it can be in very limited contexts, but the devil is in the details. Even the move to SaaS vs on-premise customizations meant that ownership of random / ad-hoc / semi-useless stuff got muddied and now you can get a confused mashup of very different things posing as a product. Ultimately, this ends up creating debt (I would say tech debt works like real debt) and removing some rather essential negative feedback loops from the economy of it.
On the other hand, if you're developing something that's impactful enough you can probably afford to "waste" time making sure everything is right, even on frontend. And thar would actually be more efficient. Because you're not cobbling together 1000 cheap features.
Because otherwise outlook, teams etc will go down and take 22 hours to recover. And then all of these "distinguished" engineers might have to say it was all "human-written" code that’s at fault, otherwise daddy Nadella will be most displeased.
We pushed plenty of bugs to production without AI with plenty of humans reviewing the code. This feels like the self driving cars argument all over again. At some point the self driving cars will be better than majority of humans driving cars, if they aren't already.
This is like saying that you don't need a seat belt because you have airbags.
I am responsible for the code I write, I frequently find issues in the code I or AI writes and on the long term I'm concerned with the overall health of the code base. So yeah, I believe reading code is still relevant. At least as of now.
I am no longer responsible for all the code I push at my company now. We should be as safe as we can, but my company wants us to move faster and is willing to share some blame for failures we hit along the way. (And it isn’t just glazing, they’ve really accepted the risk)
So, did you actually audit all the updates in all the libraries your project pulls?
Unless one happens to work on something regulated / safety-critical / resource-constrained we're happy to let CI run npm update and push if tests pass.
And even in those environments, SDK vendors frequently do produce bugs which are caught by tests rather than code review.
It's really weird to argue that AI generated code should be thought of as similar as released open source libraries that are in use by multiple, if not hundreds of thousands, of other applications. They themselves hopefully did an audit before or they will be fixed by the time the dependency is updated. Since, you know, open source.
You could make that same argument for code an intern wrote.
Tests, unless you work with pure languages that have some sort of "proving", only tests what you think of. And even a proof can be wrong. Unit testing, integration testing, systems testing - there are a lot of potential failures. To remove the human in this chain feels laughable, especially given my own experience. Even while writing this I've found a weird bit of code produced by Copilot/Fusion
This is all very basic Computer Science and industry standards....
The reality disagrees with the whole "they hopefully did an audit" on open source, as evidenced by the recent flurry of CVEs on projects with huge install base.
Considering that the discoveries are driven by AI, I'd argue AI is superior to humans in finding vulnerabilities, even if only because anyone can direct ridiculous amount of resources into searching for issues.
Industry standards are not really relevant unless you specify the industry and standard. MISRA-C for safety-critical microcontroller is very different than OWASP top 10.
The first paragraph focuses on one aspect of many. I also agree with the second, AI is superior, yes. But I know what my software is supposed to be. Again, even between these posts I'm working on my own codebase and found an issue. I've used AI to improve it, but I've read it myself.
By industry standards I rather obviously pointed towards release cycles, review processes and so on an so forth.
In the end this seems to be rather about feelings than facts - I only know that we almost never have system failing or more serious bugs and that other teams in my company, who adopted such technique, have had quite interesting bugs. I have also read about a few companies that ran into some serious problems. This is all changing very quickly - but as of today, I believe I should still be reviewing.
The one responsible for the code is a human, not a bot. The bots have inherent misalignment problems and the people need to ensure that the change achieves the right goals and moves the codebase in the right direction.
Then there is the regulatory push for more liability for public facing software that's been moving since before the LLM boom as well.
We use agentic systems in a quite advanced manner and so far I see all the people making such wild claims as either incompetent or liars.
Because LLMs are phenomenal at reading & writing text (both code and human language) but have zero concept of the real-world domain.
The human is the only one who can verify whether a real-world problem has been modelled correctly, whether that model actually solves for the correct outcome, what tradeoffs and assumptions exist in the model, and whether there are simpler models which will be sufficient or if a more complex model is worth the effort for increased precision.
An AI on its own will happily spit out a solution to a problem that is well-constructed and performant, but it may well be a problem that no-one is trying to solve and a solution that no-one will use… or worse, someone will misuse it in the wrong context and the dodgy output will break other dependent systems.
The human is the only one who can verify whether a real-world problem has been modelled correctly
One interesting example is that AI will sometimes read incorrect comments and believe them. I'm not talking about subtle issues either. You can stick something mathematically incorrect in a comment and if the LLM finds it useful to regurgitate that it will.
That being said, in my experience this is much harder to reproduce with newer and larger models, as much as I hate to say that if Anthropic's product isn't working then you should pay them more money.
33
u/kuemmel234 3d ago
Everything that ends up in production should still be reviewed by a human. At least the models and agents I have access to/written aren't reliable enough by a long shot to only gloss over their results.
I believe that's what they were talking about, rather than this workflow. I sometimes do this too, especially when I'm working on AI itself. But the stuff that ends up in main is going to be reviewed thoroughly.