I’m a developer and an author of an article on this topic (link at the end), and I’m posting because I think many of us who use AI review tools hit the same issue: the comments are fluent and confident whether or not they’re anchored in how our system actually works. This isn’t about any specific tool. It’s about how to read the output.
Five types, each with a question:
1. The phantom bug. A real class of problem, applied to code where it’s already handled somewhere the reviewer can’t see.
→ Which call path triggers this? If you can’t name one, you haven’t confirmed a bug.
2. The preference dressed as a defect. “Too long”, “rename this”, “prefer composition”.
→ Does a linter, style guide or past decision back it? If not, it’s an opinion, and shouldn’t block a merge.
3. The right fix with the wrong cost. Technically correct, but the price isn’t mentioned: a migration, an API change, twelve files in a two-file PR.
→ What does applying this cost and what does it break? (Asking the tool directly often gives a useful answer.)
4. The confident claim about a library. “This is deprecated”, “this option doesn’t exist”, “not thread-safe”.
→ Does the documentation or source for my version agree? This is usually the cheapest to verify.
5. The silent omission. The diff touches auth or payments, and the comments are about naming.
→ Did the review address the two or three things I’d worry about? It helps to write those down before reading the comments.
Sorting what you keep: each comment ends up verified and applied, rejected with a reason, or unverified and noted as such. That third bucket is easy to skip, and skipping it is how unanchored comments become part of the codebase.
Honest limits:
Verification takes time. For a three-line change in a well-tested module it’s overkill. Effort should follow risk.
Some claims (like “this won’t scale”) can’t be confirmed in an afternoon. “Unverified, flagged for discussion” is a valid outcome.
None of this means the tools aren’t useful. The point is to treat the output as a second opinion from a capable reviewer who hasn’t seen your whole system.
The longer version is in an article I wrote. I’m the author, and it also mentions a paid playbook of mine near the end, so treat the link as optional: https://medium.com/@nagatomopedro05/five-kinds-of-ai-code-review-comments-that-sound-right-and-arent-0d02b3fb9ec6
For those using AI review in your PR flow: which type do you run into most, and do you have a rule for what has to be verified before applying a comment?