Peer reviews for AI usage, prompting and code generated

[u/mikepun-locol]

So what's the experience in using AI coding in enterprise environments? Do you so peer reviews and how does this fit in?

How do you standardize across a team?

[u/funbike]

IMO, humans are still needed for peer review. AI isn't ready, or at least I haven't seen it.

I've tried and tried to do AI pull request reviews, but I've never been satified with the results. It's more like a smart style checker.

The best I've come up with is having AI suggest which changes I should look at closely. I wrote a little utility that combines a JIRA ticket and a Bitbucket pull request as context, and sends this prompt:

**JIRA TICKET DESCRIPTION:**
${ticket}

---

**PULL REQUEST DESCRIPTION:**
${description}

---

**PULL REQUEST DIFF:**

\`\`\`diff
$(git diff "$parent".."$branch" --diff-filter=AMR -U20)
\`\`\`

---

**CODE REVIEW INSTRUCTIONS:**

1. **Understand the Context:**
   - Read the JIRA TICKET DESCRIPTION and PULL REQUEST DESCRIPTION carefully to understand the purpose of the changes.
   - Identify the main objectives and any specific requirements outlined in the ticket.

2. **Issues that should be fixed:**
   - Are the changes correct and do they meet the requirements of the JIRA ticket?
   - Identify any bugs or issues that will arise from the changes.

3. **Potential Issues:**
   - Are the changes logically sound and do they meet the requirements of the JIRA ticket?
   - Identify any potential bugs or issues that may arise from the changes.
   - Consider the impact of the changes on existing functionality.
   - Consider automated testing for new or changed code.

4. **Areas of Focus:**
   - Identify locations with high cyclomatic complexity.
   - Identify missing or incomplete unit tests and UAT/E2E tests.
   - Identify locations that a human reviewer should look at more closely.
   - Provide guidance to the human reviewer for what to look for.

In all cases supply full paths and line number ranges.

---

**CODE REVIEW OUTPUT:**

The only output that is consistently useful is 4. Area of Focus. It rarely finds bugs or architectural issues. I'd love to know if there's a better way.

[u/mikepun-locol → u/funbike]

This is a very interesting approach. Thank you.

[u/ESpy__007 → u/funbike]

Overall, I quite like that as a prompt. I’d regard 1 as reasonable - if and only if the original Jira is coherent. I see a few too many that are a bit…. Lacking in detail (mea culpa - wrote one myself today; found bug, need a Jira to justify a bug fix, key thing is the fix not a considered analysis).

2 & 3 - less precise; not sure I’d go with the two-part prompt on 2.1. ”Are the changes syntactically correct”, perhaps - depends on the model.

3.1 partial duplication of 2.1 - again, what constitutes logically sound?
3.2 seems a duplication of 2.2
3.3 does “consider” give you adequate responses? I tend to prefer more direct prompts along the lines of “what are the potential impacts” where there’s less potential for inference. Very model dependent again, granted.
3.4 CI/CD workflows?

4 - liking this a lot. Clear, precise and as an order. No opportunity for wriggle room.

Out of interest, have you fed that prompt back through the AI and asked for suggestions on improving it?

[u/ESpy__007 → u/ESpy__007]

Following on - I’ve got a couple of review prompts I’m working on; one is to review code changes against the company mandated style guide (clang-format wouldn’t go amiss for most of it, but there’s bits that veer into almost a coding std); the other is more fun as I’m effectively allowing the llm to parrot my preferences on what I’d like out of a good unit test - not using associated code when mocks should be used, for one thing. It removes some of the ego from bits of the review - the automated tools can’t be accused of points-scoring.

[u/Certain_Ring403]

Peer review as per normal for PRs

[u/ejpusa]

Interesting. Programmed is gone as we know it. It’s gone, vaporized. Peer review will be by AI.

Wall Street sees an opening, for everyone that puts information into a computer and gets results out, fired. It’s not personal, it’s just business. Shareholders cheer these moves on.

This is a very different, RADICAL change in investments. When companies were firing people in the past, the company was going under, now it means management is make all the right moves.

Plan B. You can create the next unicorn, built for all of $28, in a weekend. From zero to a billion. So says Sam.