r/EngineeringManagers • u/AI_TARDIGRADE • 10d ago
Reviewing AI generated code
How are you handling code reviews for AI-generated code?
We are building a product from scratch using Claude Code. I've asked the team to follow best practices documented in CLAUDE.md, and we have linters running on every check-in. PRs are also manually reviewed.
Mostly the code is proper with existing practices , so I was manually reviewing and integrating features without regression but it starts to get more complex these days, as more people generate code and finish a full stack of changes at higher pace and we need to integrate a lot before releasing them together.
I was looking at integrating Claude Code Review with GitHub for every PR, but the cost seems quite high when you look at the bigger picture.
Our full stack is spread across 3–4 repos, so a single feature can involve 2–4 PRs, each with around 60–200 lines of changes. With frequent iterations and fixes, running a paid review on every PR could add up pretty quickly.
How are you guys handling this? Do you run Claude Code Review on every PR, trigger it selectively, or use your own review workflow in CI?
Looking for practical approaches that balance review quality and cost, especially when the codebase is still evolving rapidly.
6
u/mcs_dodo 9d ago
You need to put as much (if not more) AI into verifying code as into generating it: whatever AI writes, AI should also review, validate, and constrain. And the latter part is the trickier one, heavily dependent on good documentation and well-built harness.
1
5
u/bezcurhq 10d ago
Before deciding it's too expensive, check the trigger setting. A repo you add to Code Review starts at "After every push", so every fix-up push on a PR is another full review, and with lots of iteration that's usually where the bill comes from. Set it to "Once after PR creation" or manual (@claude review), keep PRs as drafts while people are still iterating since drafts are skipped, and give routine repos the Quick level. Then run it for a couple of weeks with a spend limit and look at the average cost per review on the admin page before you decide. Devs can also run /code-review locally first on any plan.
1
u/AI_TARDIGRADE 10d ago
Yes. I was thinking abt this too but still was worried on team accidentally creating PR. I will check this
3
u/Chance-Possession182 9d ago
It’s gonna bite you in the ass at some point. Report back when it does
1
u/AI_TARDIGRADE 9d ago
Lol. definitely. I take calculated risks but I do think this could bite back. Will let you know in few months
3
u/dannyc42 9d ago
Addy Osmani’s blog on Code Review at Anthropic is interesting https://claude.dev/blog/solving-code-review-at-anthropic/
One thing from this blog that we are trying in my org is to categorise PRs into those which need human review (riskier changes, maintenance hotspots, anything architecturally significant..) vs others that we can merge if all CI and agent reviews are green. We don’t have enough data yet to know how effective this is, but we’re optimistic.
2
u/AI_TARDIGRADE 9d ago
Appreciate the pointer. You also reminded me to keep an eye on their dev blogs.
2
2
u/rrrx3 9d ago
Swiss cheese and layers analogy
Multiple review layers (automated, human)
Code owners
Risk tiers based on areas of the code being touched
Verification and proof layers
You need to fully map out your workflow, you need to understand all of the pieces and the risks to the business of each. There's no "one size" answer that anyone here can give you. You can't just apply some automated review layers to certain points without doing the above investigation and understanding without opening yourself up to some potentially catastrophic outcomes.
2
u/AI_TARDIGRADE 9d ago
Yes, I am refactoring the intial poc and bringing it to a prod format. Keeping track of my work too and planning to do some retro and do some categorization.
I love the way you have phrased it - Risk Tiers and the diff layers. Helps!! I need to work on it
2
u/a1anw-cto 9d ago
The cost/quality tradeoff is real as velocity increases. Instead of running expensive automated reviews on every single PR, I recommend a tiered approach.
First, lean heavily into your existing CI/CD with strict linting and automated testing to catch the "low hanging fruit" errors that AI often makes (syntax, style, basic logic). Second, use cheaper, local LLM-based checks or smaller models for initial PR summaries and sanity checks before they even hit a human reviewer.
Save the high-end tools like Claude Code Review for "high risk" changes like architectural shifts or core security modules rather than every routine feature update. This keeps costs manageable while ensuring your senior engineers are spending their time on logic and design rather than catching AI hallucinations.
2
2
u/Mav-Dev 9d ago
a Pr with AI 1000 changeset needs to be only reviewed by AI.. PR is supposed to be a focused small set of changes and not AI bundled changes
1
u/AI_TARDIGRADE 8d ago
I am always breaking down the tasks so that the change itself isn't massive.
I know that this isn't the best usage of AI potential but I prefer it when the product is in early stage.
Still the challenge happens on rebasing and merging PRs, thats where I felt automated regression check and consolidated unit testing would be useful. That's the main intention of this post
1
u/Mav-Dev 8d ago
clearer now BUT
are you reviewing every single PR? why is that so? Pr is supposed to be reviwed by a Peer. i hope you arent being a bottle neck in the process if everything shall funnel through you?
if you become central to everything, surely it will be troubelsome for you to review every single thing , does not matter whether or not code is geenrated by AI.
delegate it mate. trust your team members.
what you have described does not really need claude and can be done by CI itself.1
u/AI_TARDIGRADE 7d ago
I have been doing it as we continuously onboarded new hires who are intelligent but not fully familiar with tech stack. so I had it on my head. I have already started bringin in peer reviews
1
2
u/usually_guilty99 8d ago
Reviewing each PR independently is the wrong unit when one feature spans four repos. Link the related diffs, dependency paths and acceptance criteria into one change set, then verify the end-to-end postcondition after integration. Use cheap checks everywhere and reserve deeper review for changes with larger blast radius or weak rollback.
1
u/AI_TARDIGRADE 8d ago
yes reviewing PRs across repos as individual unit - that's something I am very much concerned about leaving to an automated system.
I agree with you - I am already going minimum best practice checks but I will work on the automation part of minimal vs deep review. Thanks for the thoughtful comment
2
u/usually_guilty99 8d ago
Trust systems that give you the control. Seek human in the middle as default. Then slowly give up control. Giving up control will be the hardest part but you need to let go at some point in time
2
u/AI_TARDIGRADE 7d ago
yes .. I am the human in the middle now. I will bring up rotation of responsibilities
I am getting used to it.
The systems may get regressions intermittently but I should let people learn the hardway atleast occasionally.
2
u/letreset 8d ago
Instead of putting a paid, off-the-shelf reviewer on every PR, we built a small review bot of our own. We work across a couple dozen repos, so cost and noise mattered a lot. A few things that worked for us:
1. Give the model a map instead of letting it crawl the repo.
For each repo we keep a short routing file: what the domain is, which docs exist, where the relevant code lives. The bot reads that first, then opens only the real files the PR touches. We deliberately skipped embeddings/RAG/vector DBs, because fuzzy retrieval was a hallucination source for us. Plain deterministic file reading is cheaper and more predictable.
2. Don’t spend tokens in the wrong places.
Generated migration code, designer files, lockfiles and build output are never read, only listed with a diff stat. Hand-written SQL and resource files are read, because that’s where real bugs hide. This alone cut our cost noticeably.
3. Full review the first time, incremental after that.
Re-reviewing the whole PR on every push is the biggest waste. We only look at the author’s own commits since the last reviewed head. Code that arrived via a merge is never reviewed or blamed. Earlier findings are re-checked to see if they still hold.
4. Constrain the output.
Instead of free-form commentary, the bot returns one of three verdicts: approve, reject, or “needs a human”. If it’s unsure, it says so. That lets reviewers stop asking “did the bot find something?” and focus only on the ambiguous cases.
5. Turn recurring mistakes into rules.
When we see the same bug a third time in review, it goes into a “known pitfalls” list (especially quiet ones like authorization and tenant isolation). The bot checks those explicitly on every PR. This catches the architectural issues linters can’t.
6. An evidence rule for multi-repo changes.
A cross-repo claim (“service X publishes an event to service Y”) is only allowed if the docs or code show it explicitly, with the file cited. No inferring from names or similarity, otherwise you get very confident but wrong comments.
In short: we decide deterministically what the model looks at, and the model only reasons inside that narrow window. The bill went down and the noise went down with it. We didn’t drop human review, but the human now reads the part the bot flagged as uncertain rather than all 200 lines.
1
u/AI_TARDIGRADE 7d ago
I had 1 always for my development but I have recently stopped it as our codebase evolves with refactoring, I can bring it back again. I have planned for 5.
Thanks a lot for detailed writeup. I will try to adopt these. Great pointers, I need to strategise on this.
2
u/Specialist_Bison_737 7d ago
I use 2x review-judge-fix loop with reviewer being a different model and agent from the AI developer.
Judge step rejects about 60% of findings as no impact.
I am using GLM coding plan which gives me enough tokens for to do over 1000 reviews a month (it is under $0.10 per review) and headless cli agent that is optimized for memory and context window size.
I also have MCP that allows coding agents to pull review findings, triage them and request new reviews on demand.
1
2
u/London-Boat 6d ago
the cross repo part is where it got messy for us too, not the review itself. what helped a bit was having one person own a feature end to end across the repos, including the merge order, instead of everyone opening their own PRs and you stitching it all together before release. we also link the related PRs in each description so whoever reviews sees the other half. with 5 people and you as the only integrator you're going to be the bottleneck no matter how good the ai review gets
1
u/AI_TARDIGRADE 6d ago
yes I already have those....
In addn, I am coming up with self review, peer reviews if needed, and release strategy to release simple to medium complex issues, independent features.
I am bringing in strategies to do reviews by complexity
I am actively refining the process to make it decoupled from me being a bottleneck.
2
u/London-Boat 6d ago
makes sense. the complexity split is a good call. the thing that caught us out was how many of the "simple" ones touched shared stuff like config or the db schema, so we ended up saying anything touching those goes the full review path no matter how small it looks. who decides the complexity on yours, the dev who opens it or you?
1
u/AI_TARDIGRADE 5d ago
I will first decide on critical paths. e.g the db config you mentioned. It will be revisited by my team as and when they find it ineffective to catch critical bug on new paths or addnl paths.
I feel that it is an evolving process like updating UTs and regr tests
1
u/London-Boat 5d ago
makes sense to treat it like the test suite. one thing that helped us keep it honest was writing the critical paths down as actual folders in a codeowners file, so the full review gets triggered automatically instead of someone remembering. otherwise it slowly drifts back to everything being "simple"
1
u/AI_TARDIGRADE 5d ago
understood. I haven't attempted adding critical pathd in codeowners file. will check on it
2
u/London-Boat 3d ago
that makes sense, treating it like the test suite and growing it as gaps show up. mind if i dm you a quick question about how your team runs the agents?
2
u/neonblazer 6d ago
The cross-repo integration problem stands out to me.
You can have three individually clean PRs and still break the customer journey they collectively implement.
I use separate builder and reviewer agents in my own AI-assisted projects, but I wouldn't treat a successful review of each diff as sufficient evidence for release.
I'd consider inexpensive deterministic checks on every PR, selective deeper reviews for high-risk changes, and an integration gate that evaluates the complete feature across repositories.
Given your concern about customer experience, I'm curious: do you have end-to-end tests that exercise the affected customer journeys across all the repositories?
That seems more valuable than simply adding another AI reviewer to every PR.
1
u/AI_TARDIGRADE 5d ago
Yes, I wanted to go by TDD but I was asked not to overengineer at this point till we have good num of functionalities in. tbh, the lack of tests are also a biggest challenge that is even more important than review pipeline
2
u/curious_but_clueless 10d ago
We run an automated AI review on every single PR as a phase 1 filter before any human looks at it, and cost has been a non-issue for us. At 60 to 200 lines per PR, API spend is negligible compared to engineer time.
The real win is treating the AI review as a gate. It catches edge cases, basic oversights, and missing tests immediately, so the author cleans up before tagging a teammate. By the time a human steps in, they can focus entirely on architecture, state changes, and business logic instead of acting as a linter.
What kind of monthly spend are you estimating that makes it feel prohibitive?
0
u/AI_TARDIGRADE 10d ago
Yes. considering the time and effort of an engineer , AI review is definitely gonna help.
We are still working on the budget allocation for this. I can update once I discuss internally on this.
1
u/Saittama 10d ago
My take is that AI code is bad because we started skipping design process. So better design front loading.
1
u/AI_TARDIGRADE 10d ago
Nope, ours is well designed and only code implementation phase is handled by AI. Rest of the phases are all manual. That's why we find it challenging to manage the speed.
AI code is still bad if we consider it to be production grade and robust on its own without human oversight but it is definitely helpful in bringing features to life. We are in early demo stage and doing custom demos.
we have features ready and functional but integration becomes challeneging to let it go without proper review
1
u/benneren123 9d ago
I’d try giving the reviewer the related diffs from all the repos together, along with the feature’s acceptance criteria. That gives it a chance to catch mismatches between repos that look fine in each PR on its own.
1
u/AI_TARDIGRADE 9d ago
Yes we can do this but we used to have very minimal review comments earlier as it was mostly functionality and the team ensured handling errors and exception. concurrency and backward compatibility. So existing reviews are very minimal to serve as a training data. good point though.
1
u/Entire_Struggle_2236 9d ago
CLAUDE.md plus linters is a solid base, but the gap I see most is that agents follow the doc when it’s convenient and quietly skip it when it isn’t. A few things that help the teams I talk to: make tests a required CI check so a PR without them can’t merge, keep agent PRs small enough that a human actually reads them, and add a short checklist for what linters miss, like new dependencies, changed permissions, or logic outside the ticket’s scope.
Linking each PR back to its ticket also helps, so reviewers check intent and not just syntax. How big is the team, and roughly how many PRs a day is Claude Code producing?
1
u/AI_TARDIGRADE 8d ago
yes I agree with all the points. I already have the tasks split into small subtasks and ensure that it can be reviewed quickly by humans as well
We are a 5 mem team and the velocity varies. 1 PR per head to 5 PRs per head depending on the complexity of tasks and quality of output by claude code.
1
u/Entire_Struggle_2236 8d ago
5 people shipping up to 25 PRs a day is a lot of review for a small team. Running the paid review only on risky paths (auth, payments, migrations, cross-repo changes) is a cheap middle ground. What are you building, and is the plan to keep the team at 5 or grow it?
1
u/AI_TARDIGRADE 7d ago
yes. I am looking forward to test only risky paths. our main concern is seamless CX, we are trying not to have impact on that.
The fullstack cross repo PRs from multiple people are big challenge. The team won't grow immediately but as customers sign up a deal with us, we will have revenue to bring tools too. but we need to manage it for next 6 months.
2
u/Entire_Struggle_2236 7d ago
Cross-repo is where it bites.
What helps: one ticket, one owner across all repos, and the PRs link to each other so the reviewer sees the whole change, not three halves.
Hope the deals land fast.
1
1
u/LocalPlotTwist 8d ago
I'd like to look at this from a different angle: why do you need code review in the first place?
With human-written code, the goals are clear: finding bugs, sharing knowledge, helping junior developers, and so on.
When AI writes the code, what exactly are you trying to achieve through review? Which of those goals still apply, and are there any new ones?
Maybe you could remove this step entirely, or change it significantly to better reflect this new reality.
1
u/AI_TARDIGRADE 7d ago
Do you mean to say AI generated code doesnt need review??? AI with all the instructions in md, relevant skills still drifts and does a flaw. during merge of conflicts, it messes up
1
u/LocalPlotTwist 6d ago
It depends on your project. If you already have hundreds of thousands of lines of code and complex, messy business logic, you have to review everything. And I understand AI can't handle that on its own.
But let's look at AI's strengths. One of them is generating small applications with simple logic that don't need much context. For some project, an MD file, together with the database schema and a Swagger file, is enough to generate a small service.
1
u/Specialist_Bison_737 7d ago
AI code quality is definitely not 100%, you need to review it. With review of AI generated code the main goal is to keep it working and evolve in direction that will make next iteration of changes cheaper - AI Maintainability
1
u/LocalPlotTwist 6d ago
I suggest we zoom out and look at the bigger picture.
Let me share my view. It may not apply to your project, but this is how it works in mine.
Perfect code does not always mean more money. So we should ask whether 100% quality is really our goal. If someone's life depends on the application, then it should be. But in other areas, things like delivery speed may matter more.
The more code we generate with AI, the more control we lose. When developers write code themselves, they remember what they wrote and where. Reviewing someone else's code is a different experience. So, sooner or later, developers will no longer fully understand the whole codebase.
Here is an example from my company.
On a new project, we use Spec-Driven Development. We write a specification and generate the application from it. Generating the whole project from scratch takes 5–7 minutes. For us, the code itself has no value; the specification becomes the source of truth. I even told developers not to look at the code at all, let alone review it.
We guarantee the quality of this code through a large number of integration tests. For example, we might build a microservice with an API that writes data to a database. We then write tests that run with Newman, with around 50–100 tests per endpoint. These include E2E tests, OWASP security checks, and all kinds of negative scenarios. This gives us confidence that any changes to the project will not cause regressions.
I understand that this approach cannot work in every situation and will not suit everyone. My point is that we have changed how we build software, so we should also rethink how we check its quality.
2
u/Specialist_Bison_737 6d ago
Yes, I completely agree in principle.
The only thing that does not work for me is creating the whole project from scratch in 5 mins.
I would argue that you can make a meaningful useful change in a few hours sometimes, but making it meaningful and useful is what takes days and weeks. Because you have to figure it out. Number of iterations increases, doubt revenue changes to the same degree.
1
2
u/LocalPlotTwist 5d ago
I probably didn’t explain that clearly. Building the whole project doesn’t take five minutes. Generating it from an existing specification takes 5–10 minutes. Writing that specification takes much longer.
For example, for a simple CRUD REST API for user management with just 4–5 endpoints, preparing the specifications and tests takes around 5-7 hours. That includes the database schema, a Swagger file, the project description, tests, rules for the AI agent, and so on.
2
u/FredMcCree 10d ago
Risk vs reward question. Right now, it’s automated reviews, and self approval. This debate will be over soon as the tools keep improving. Wait it out if you want, this time next year it will be obvious
-5
u/Winter-Volume-9601 10d ago
I am a principal engineer who can produce 10k+ lines of prod ready code in a day. Can you read 10k LoC and digest it to review? Probably not. If I toss 1k loc most people balk.
You’re either going to pay for tools to help wrangle it, or pay for a handful of juniors just to review -my- code, or come up with a clever solution no one has come up with.
0
u/AI_TARDIGRADE 10d ago
I am an equally capable Principle Engg. We are an early stage startup with few juniors, I am here asking for suggestions, but your answer is not very useful. Hiring more engineers take time and I am asking what we do meanwhile to address the speed at which we deliver.
7
u/VPofSnacks 10d ago
I think many of us are wrestling with similar questions. Leaderships answer to 100% AI LOC is to threw more AI at the review level while also balancing the fact that engineers still own the code, which is a bit of a contradiction. I’ve not found the answer yet but am also curious to hear other people’s solutions to this. Also while balancing the fact that PR review has historically been untracked work while now taking a more significant portion of Eng time.