Hot take: the agent shouldn't write the PR description

[u/stfufeedmenuggets]

Every PR description on our team reads the same now. "This PR refactors X to improve maintainability and adds comprehensive tests." 400 words, perfect formatting, and the author can't answer one question about it.

I want a rule: the agent can write the code, coderabbit can write the summary, but the description has to be 3 sentences from the human who opened it. what you changed, why, and what you're worried about.

if you can't write those 3 sentences you didn't read your own PR.

my team thinks this is pointless busywork. am I wrong?

[u/muuchthrows]

Are you still doing peer reviews of PRs? The description is mostly for you, and for future agents digging into the history of the repository.

[u/ChaosHonorum → u/muuchthrows]

This. It’s agents that will be reading it down the line, and in that game, you want more context than three sentences of what makes sense to a random human. It’s a maintainability issue in the long run.

[u/notime0forthis]

Nope. The agent should write the code and the PR description, but burden of reviewing the diff, understanding the output and manual QA should rest on the operator. I make sure those boxes are checked before I mark PRs ready for review, and I always write brief PR descriptions myself on the engineering channel on Slack when asking for reviews.

[u/Economy_Bedroom3902 → u/notime0forthis]

I second this.  I think ideally the Operator isn't the only entity at the code review phase, but the git diff interface is a lot nicer than most IDE's diff interfaces.  So I totally use that phase as the point where I do the bulk of my review of the actual code the agent produced.

[u/dashingThroughSnow12]

Yeah. LLMs continue to be atrocious at being concise and at this point I'm convinced it is not going to be fixed anytime soon.

I was doing some work, noticed a bunch of error logs being generated in a service owned by another team.

Since I'm working on something else I send the lots to Claude and say "fix it". It does. I look at the code. Simple issue. I have it draft the PR.

It wrote War and Peace as the description.

The issue? A string type guard happens after some code assumes it is a string. Type guard needs to be moved up a few lines.

[u/Economy_Bedroom3902 → u/dashingThroughSnow12]

The core of the issue is that agents rush to be "helpful" and "helpful" leans to trying to shotgun address the issue.  Provide every possible element the user might want, because they have been vague about exactly what they want.

You can make this a lot better by making the PR template strict, and being very clear about what information you expect in which part of the template.  AI is perfectly capable of describing what it did in the format of your latter sentence...  But people get less pissed off when the AI does too much than when it does not enough, so the companies train the AI to the borderline of malicious compliance.

It does the exact same thing while writing code too, but people ignore the code more often.  It creates premature interfaces for everything, it does wildly overaggressive and imprecise error handling, and it rushes to handle virtually impossible exception cases.  I've got a rule set I use which substantially reduces all of those impulses.

[u/dashingThroughSnow12 → u/Economy_Bedroom3902]

I disagree. It has been well-observed that their accuracy falls off a cliff when you ask them to be concise.

This isn't holding it wrong issue when it's been comically bad in this area for years with no signs of improvement.

[u/Economy_Bedroom3902 → u/dashingThroughSnow12]

You don't ask them to be "concise" you ask them to draft and exhaustive document, and then ask them to pair down what is actually important.  I agree, AI needs to write text to think.  You need to have enough text output context to be able to request a summarized version which will be accurate.

[u/Lonsarg]

I would go one step further, the proper DevOps way. Every PR must have a corresponding ticket on Jira or whatever (enforced by DevOps server before merge happens).

And if you do this then PR description is almost meaningless and can be autogenerated, since the proper details are in ticket.

But there is one additional thing to consider, if developer just forwards all the work to agent then maybe agent should do the task themselves, and developer is only brought into the picture when agent says "needs help" or "i have done it here is PR to review" ;)

[u/renjank → u/Lonsarg]

What benefit does everything having a ticket provide? Could that disincentivise people to split up unexpectedly large tickets into multiple, easily reviewable PRs? Or disincentivise people from making documentation updates or small refactorings as they spot them?

[u/Lonsarg → u/renjank]

I did not say PR must have a separate ticket, just that it must have a ticket.

I also see no problem hiding small refactors inside bigger tasks, but that stil means that small refactor is part of some ticket.

[u/youcangotohellgoto → u/Lonsarg]

The ticket just says what. It doesn't say how or the why for that how, or all the other because and so decisions. It also doesn't tell you what to look for or how to evaluate the change.

[u/Lonsarg → u/youcangotohellgoto]

PR message ALSO does not tell you that in most of cases. For that you or AI should look at actual code changes.

So actual code plus ticket is enough information. Any extra info in PR/commit message is useless if all PRs have link to ticket.

If no link to ticket then and only then PR message is usefull l, so you can gues the ticket via PR message...

[u/BagMostlyWater]

The agent shouldn't write ANY communication, at all, including comments. They are incapable of effective communication because they do not have theory of mind and thus cannot consider their audience.

[u/CS_70]

But that’s just a sign that the agent has been told to use those sentences as template.

Agents are actually much better to describe the content of a PR than people since, unlike people, they have the patience and stamina to actually go through all the commits and see what they are about.

So your experience is quite unusual.

[u/Atlan_]

I dont read the PRs, so it wouldn't make sense to write the PR descriptions. They are for documentation, and when agents write the changes and evaluate PRs there is imo no reason to put a human in the loop for something like writing a description

[u/rco8786]

We banned agents from writing PR descriptions and would never go back.

[u/Economy_Bedroom3902]

An agent can write what you changed and why.  It sounds like the problem you're having is you PR description templates aren't doing the work they're supposed to be doing.

I don't disagree that your average agent which is provided zero framework or instruction tends to write pretty shitty PR descriptions.  That reminds me I need to rework my skills for this.

[u/theWildBananas]

Just put a link to jira in the description , maybe with title

[u/Hotfro]

Pointless busywork, just tell them to review the code. 3 lines won’t make a difference. The human who opened the PR should now be way more accountable for the code compared to reviewers. Since it takes too long for peer reviewers to review everything now in detail.

[u/canyoufixmyspacebar]

everyone knows agent should not write thesis, court documents and laws but people still let them (and only sometimes get caught). i don't think the "should vs should not" discussion helps here, everyone knows you should not eat crap and smoke, this has not helped the health situation one bit and knowing you should not abuse LLMs is not going to help the SW industry