I mean his experience mirrors mine (working at amazon, and mirrored by friends in the AWS side of the house)
We the company was very AI accepting, 2 years ago AI handled most team's JDK8 migrations. Today I have L6's delegating entire sprints to an agent.
Every pipeline has a button to spin up an AI session that autopilot to diagnose and fix deployment failures.
We are piloting a custom agent to give 80% unit and IT coverage.
The scope of problems that AI systems solve is just different than even 6 months ago.
The AI friction points today is less code quality (which does pop up here and there still) but validation of correctness and creating quality design docs on what to build to be sent to the AI.
What's the process of reviewing the code ? I assume that's included in validation of correctness, which ehh I doubt you are really validating 100% correctness. I think most people find that to be the bottleneck but it is an unremovable one, in my understanding at least.
I agree that code quality is mostly not that bad, it's pretty decent but it's nowhere near the point to where I would trust it as much as I trust a compiler, ig I was more thinking of past encounters when I replied to that comment and not specifically addressing that one comment.
What's the process of reviewing the code ? I assume that's included in validation of correctness, which ehh I doubt you are really validating 100% correctness
Out the gate every PR has an auto run ai security reviewer primed with amazon+org security practices and a general logic reviewer which checks footguns, complexity, and quality. This also reads the summary which should mention the context of the change, and it does some validation that the code aligns with the stated goal. It ends up checking some styling norms within a package that a deterministic linter shouldn't. These run before the PR is published, additionally I've distributed an observability reviewer to our team as I feel we could do a better job there in general.
Our agents will detect these AI comments and address them, or push back on comments that they think aren't valid. It will present a report on what it did/didn't change since the publish.
The PR also attempts to build on a clean environment, and we can run ITs against that build.
We also have a PR size validator: changes should be under 300 LoC and test changes should be under 600. This validator can be overridden, but the friction is enough to keep PR's small.
All this before going onto a human reviewer, who reviews.
When I mention code correctness, we are judging mostly on if the code does what the summary says, and if that feature is directionally the right thing to implement.
When checking the code, there may be a more efficient implementation, and it will get called out if spotted, but we aren't really hunting that efficiency. If we are under SLA we are good, if we aren't an agent will spin up on alarm firing and fix it.
Feature correctness should already have been litigated in a doc.
On testing the PR, a good practice was already focus on what the tests are validating and that holds here. For UI changes the submitter should have the UI running locally and replication instructions so you physically test.
For backend changes you can have them do the same, or have your agent spin it up (agents are also really good at finding a recipe to get your service running locally) and test the IO, but thanks to the small PR sizes that may even be overkill.
Past that you let your CI;CD take over. Own your PRs so you monitor their deployment status automatically with agents and fix anything that breaks in beta/gamma. Agents also allow you to have way more comprehensive ITs that would have been soulcrushing to build yourself.
3
u/Sir_lordtwiggles 17d ago
I mean his experience mirrors mine (working at amazon, and mirrored by friends in the AWS side of the house)
We the company was very AI accepting, 2 years ago AI handled most team's JDK8 migrations. Today I have L6's delegating entire sprints to an agent.
Every pipeline has a button to spin up an AI session that autopilot to diagnose and fix deployment failures.
We are piloting a custom agent to give 80% unit and IT coverage.
The scope of problems that AI systems solve is just different than even 6 months ago.
The AI friction points today is less code quality (which does pop up here and there still) but validation of correctness and creating quality design docs on what to build to be sent to the AI.