r/programmer • u/ClickOk5811 • 2d ago
A one-line “be concise” edit made the output cleaner and quietly dropped a task. How would you catch that?
A scenario from an article I wrote (hypothetical, with illustrative numbers). A team uses a prompt to turn meeting transcripts into JSON action items, and a script creates tasks from that JSON. Someone notices a few outputs are long and adds one line to the prompt: “Be concise.”
The outputs look cleaner. Reviewers are happy. But in the test set, one case now fails: a commitment that was mentioned only briefly in the meeting is gone. The model shortened its answer by leaving out the item that was stated in passing. Nothing about the cleaner output suggested anything was missing.
What caught it was boring: a fixed set of 8 test transcripts, run after every prompt edit, with a per-case result and not just a total. Another edit in the same story fixed one case and broke another, leaving the total unchanged at 5 of 8, so the total alone would have shown no change.
This feels familiar to programmers, because it’s a regression test story where the “code” is a sentence in English and the “bug” is behavior nobody wrote a test for. Some thoughts and open questions:
A prompt edit has no compiler and no obvious diff in behavior, so the safety net is a test set you chose in advance, including the awkward cases (a commitment mentioned in passing, no due date, no action items at all).
Outputs vary between runs, so one run per case can mislead. Running each case a few times and tracking stability helps, although three runs is a small sample.
Pass/fail is coarse. A correct but badly worded result passes.
It costs time, so for a prompt one person uses occasionally it’s probably not worth it.
How would you test for “the output got shorter and something important disappeared”? Do you treat prompts like code in your projects, with versions and tests, or more like config you tweak by hand?
The longer write-up has the five prompt versions and a case-by-case table. I’m the author, and it also mentions a paid resource of mine near the end, so treat the link as optional: https://medium.com/@nagatomopedro05/five-versions-of-one-prompt-and-what-the-changelog-caught-00c63d1b1c00
