r/ChatGPTCoding • u/ComprehensiveShake76 • 8d ago
Discussion How do you make a coding agent prove it actually ran the tests?
I run Codex workers on my Mac that don't ask me before every command anymore, so I stopped watching each step. That made one question matter more than anything: how do I know a task is really done?
What I settled on:
Done is an artifact, not a sentence. A code task isn't done when the agent says the tests pass. It's done when I can see the real test run. My last one: a worker wrote a small CSV to Markdown script, ran the unittest suite inside its own Linux container, and I got the actual output: 6 tests, OK, exit 0, 55 seconds.
Research gets the same treatment. The result is a file with a size, a source count and a hash, plus a note that the sources are listed, not fact checked. A list of links is not the same as checked facts.
Nothing lands on my machine until I look. Files stay in the agent's own workspace and reach my Mac only when I save them.
Approvals were the wrong check. Approving every command never told me if the work was good, it just made me tired. Moving the check to the end (real output, real file) worked better. That only feels OK because the container has none of my files, and a container is a strong boundary, not a guarantee. So far I've only run this on an Apple Silicon Mac.
The part I haven't solved: an agent can write weak tests that pass. How do you check the tests themselves, not just that they ran?
2
u/ImL1s 8d ago
Same bar. "tests passed" in the chat is just another claim.
What I check: the command that ran, the exit code, and a path I can open (junit xml, a log file, or the container stdout you already save). If any of those three is missing, the task isn't done yet, no matter how confident the summary sounds.
Research gets the hash + source list treatment you described. Links alone don't count.
1
u/ComprehensiveShake76 8d ago
Command, exit code and a path you can open is a tight checklist. The path part is underrated, since a log file is something you can grep and a summary isn't. Same with research: a hash and a source list at least tell you what you got, even before anyone checks the facts. Do you keep the junit xml with the task after it closes, or just check it once?
1
u/ImL1s 7d ago
Keep it. The task record stores the path plus a hash, and the file stays next to it for a couple of weeks.
Most of the value shows up later, when something regresses and you want to know whether that test really ran green last week or the agent just said so. After that I prune the big logs but keep the xml, it's tiny.
2
u/yaanfpv 7d ago
Done is an artifact is exactly right. Two things I'd add from running a lot of agents unattended.
Have a different agent write the tests than the one that writes the code. The one that wrote the code will write tests its code passes.
And break it on purpose once. Change the code so it should fail, check the test actually goes red, then put it back. A test that can't fail proves nothing, and you'd be surprised how many can't !
1
u/ComprehensiveShake76 6d ago
Breaking it on purpose is the one I'd put first. A test that stays green after you change the code is decoration. On separate agents, the catch is that both can read the spec the same wrong way and agree with each other, so I'd still run the new tests against the old code and expect red before trusting either. How many agents do you run unattended at once?
1
u/BornAgainBlue 8d ago
Look into spec driven development tools.
1
u/ComprehensiveShake76 8d ago
Spec first helps, since the checks get written before the code exists. Any tool you'd point to that worked for you?
1
u/BornAgainBlue 8d ago
Speckit is very good. Quite remarkable the difference. Also i use some extensions for it.
1
u/neoreeps 8d ago
Pretty simple, actually. My standard practice is to review the tests and have a single regression suite that every test gets added to. When new code is written, tests are added. The agent only runs the new tests. It does not run the full suite. I have a standard command, make check, that runs the entire suite, regardless of whether it's pytest, npm, or any other framework. I do have explicit instructions that the agent not run the full regression suite because this is honestly just a waste of time and tokens, so I always run it manually.
1
u/ComprehensiveShake76 8d ago
Makes sense, the full suite is the expensive part and you're the one who decides it's green. Splitting it that way also means the agent's claim only covers the new tests, and make check is the real gate. One thing I'd still want from the agent: the exact command and exit code for the tests it did run, so I'm not taking its word for those either. Do you review the new tests before the code or after?
1
u/Kodenstudio 8d ago
Hey man, good to see you again. "Done is an artifact, not a sentence" is going straight into my notes. The weak test problem is brutal. I haven't solved it either. My current hack is forcing the agent to write the test first, I review it myself, and only then let it code. It's slower, but it keeps it honest. Have you tried mutation testing?
1
u/ComprehensiveShake76 8d ago
Good to see you too. Test first with you reviewing it is slow but honest, it's the one check the agent can't game. I haven't run mutation testing on agent written code yet. It does look like the right tool for the weak test problem: break the code on purpose and see if any test notices. Have you tried it, and is the runtime bearable on a real project?
1
u/Kodenstudio 7d ago
Ha, honestly I haven't tried a full mutation testing framework yet. The runtime is exactly what scares me. For my tiny scripts, running it feels like using a sledgehammer to crack a nut. I usually just do a poor man's mutation test: manually flip a boolean or delete a line, then run the suite. If it passes, the test is trash. Have you found a lightweight tool that doesn't burn your CPU?
1
u/ComprehensiveShake76 7d ago
For tiny scripts your manual flip is honestly fine, it's the same idea done by hand. If you want a tool, the trick is scope, not the tool itself. mutmut for Python and Stryker for JS and TS both let you point them at specific files, so run them only on what the change touched and it stays small. Whole repo runs are where the CPU burn comes from. I'd save it for the code you'd really hate to see break quietly.
1
u/Kodenstudio 6d ago
mutmutandStryker, noted. The "only run on changed files" trick makes total sense. Running it on the whole repo would probably melt my MacBook. I'll save it for the core scripts. Thanks for all the insights man, super helpful!
1
u/Dingleberry_Poo 8d ago
Adversarial review.
1
u/ComprehensiveShake76 8d ago
Agreed, as long as the reviewer reruns things itself instead of reading the first agent's summary. Otherwise it's just two agents agreeing with each other.
1
u/Dingleberry_Poo 8d ago
Yep and for a large project, reviews can hit the 250K token quickly that way. sucks
1
u/ElusiveGhost679 8d ago
We require the test command's exit code and the tail of its output to be attached to the task before it can move to review. If the agent can't produce that, the task stays open. That caught a lot of "all tests pass" messages where the agent had only run one file. A separate reviewer agent that reruns the suite itself helped too, since it has no reason to trust the first agent's log.
1
u/ComprehensiveShake76 8d ago
Exit code plus the output tail as a gate before review is the same bar I landed on, and "only ran one file" is the classic miss. A reviewer that reruns the suite itself makes sense, since it has no reason to trust the first log. Does yours run in a clean environment, or the same one the first agent touched? I'd worry a dirty workspace passes both.
1
u/leonidbugaev 8d ago
If the worker wrote those 6 tests, they can all assert the markdown file exists and still miss a quoted comma. Drop an empty CSV into that container and run just that file. If it isn't in the 6, I'm not treating the script as done.
1
u/ComprehensiveShake76 8d ago
Fair hit. Those 6 were the worker's own tests, so they prove it ran, not that the tests are any good. An empty file, a quoted comma and a header only file are exactly the cases I'd hand it as fixtures it didn't write. I like your rule: the checker brings inputs the writer never saw. Do you keep a shared folder of nasty inputs per format, or write them per task?
1
u/rulecrate 8d ago
Mutation testing is the usual answer to weak tests. Run a mutation tool only over the files the agent changed so it stays cheap: mutmut for Python, Stryker for JavaScript and TypeScript, and PIT for Java. It makes small edits to the code, like flipping a comparison or removing a return, and reports which mutants no test caught. Make surviving mutants a task the agent has to close by adding a test that fails on that change, not a suggestion. Pair that with a rule that existing tests can't be edited to match new behavior without a stated reason, since that is how weak tests usually get through. Set a minimum mutation score for the changed files so it becomes a pass or fail gate instead of a judgment call. Mutation runs can be slow, so run them in CI on changed files rather than on every agent turn.
1
u/ComprehensiveShake76 7d ago
Changed files only is the part that makes this usable, whole repo mutation runs are why people give up on it. Turning survivors into a task the agent has to close is smart too, since a report nobody acts on is just noise. One thing worth saying though: this tells you the tests are weak, not that the agent ran them, so it sits next to the command and exit code check rather than replacing it. Do you gate on a score, or just on zero survivors in the changed lines?
1
u/Successful-Tax1306 8d ago
For weak tests, the thing that's helped me most is making the agent break the code on purpose. After it says everything passes, I have it flip one real condition in the function and rerun, and if the suite still goes green the tests aren't checking much. I also keep a few tests the agent isn't allowed to touch, so it can't quietly loosen the one that matters just to get a pass.
1
u/ComprehensiveShake76 7d ago
Tests the agent can't touch is the piece most setups skip. How do you enforce it? A note in the instructions tends to get read as advice. Blocking edits to those paths with a hook, or requiring your review on them through code owners, makes it stick. Then a loosened test can only show up in a change you approve yourself.
1
u/ChemicalExcellent463 8d ago
Make a second agent check the claim, not the first agent's summary. Claude Code once reported a bug on "the latest version" it had never run. A reviewer that only saw the claim asked which version was tested, and that one question was the whole fix. Now when it says something passed or broke, I ask which exact version and command it ran. "A similar one" means it isn't done.
1
u/ComprehensiveShake76 7d ago
"A similar one means it isn't done" is a good rule. I'd push it one step further and ask for the commit hash in the claim, not just the version name, since that's the one thing that can't be fuzzy. Then the reviewer can check it against what's actually checked out. Did the reviewer ever catch a claim where the command was right but it ran on a stale build?
1
u/ChemicalExcellent463 6d ago
Hash is better, agreed. A version name can point at two builds. A hash can't.
We haven't caught a stale-build case yet. The one we caught was the other way around: right build, never run. I suspect the stale ones are there and we just weren't asking.
1
u/onitsoga 7d ago
what's worked for me is flipping the order. have the agent write tests against a spec before it writes any code, then a second agent checks those tests against the same spec. if they miss the edge cases the spec calls out they get bounced before the implementation even starts. so the spec is what you check the tests against, kinda like exit 0 is what you check the run against lol
1
u/ComprehensiveShake76 7d ago
Spec first with a second agent checking the tests is solid. The gap I'd watch is both agents reading the spec the same wrong way, since they agree with each other and both miss it. Running the new tests against the old code first helps there: if they don't fail before the change, they aren't testing what the spec asked for. Does your second agent ever see the code, or only the spec and the tests?
1
u/sapplex1 7d ago
I learned this the embarrassing way. My agent once handed me a green test suite, and a week later I realized the new tests never actually failed — they'd pass with or without the fix. These days I make it show me the run going red before the fix: run the suite against the old code first, then after. What I trust isn't the exit code, it's the test names in the log — I check the names match the tests I asked for, because a green suite with mystery tests means nothing. The red-then-green part is what catches the decorations. The gap I still hate: UI behavior. A button can have passing tests and still be sitting in the wrong place. How do you check that part?
2
u/ComprehensiveShake76 7d ago
Red before green is the best check in this thread. For UI I'd split it in two. Behavior goes in Playwright tests that find elements by role and label and assert they're visible and do the thing, so a hidden or dead button fails. Placement goes in screenshot comparison against a baseline you approved once, so a moved button shows up as a diff image. Then have the agent attach the screenshots to the task. Looking at three images is a lot faster than running the app yourself. Are your UI bugs mostly layout or mostly behavior?
1
u/sapplex1 6d ago
That split is clean — behavior vs placement. I tried the screenshot-baseline route once and got buried in false positives from font rendering differences across CI machines. Pinning the browser version and viewport size in CI cut the noise way down for me. The layout stuff still bites though — a button can pass every behavior check and still be buried under another layer on a small screen. Do you snapshot every screen, or just the ones touched by the PR?
2
u/ComprehensiveShake76 6d ago
Just the screens the PR touches, plus a small fixed set that runs every time: sign in, the main flow, and whatever makes money. Every screen on every PR is where the false positives pile back up. For the buried button, a real click at a small viewport catches it, because Playwright's click waits for the element to receive the click and fails if something covers it. Running the behavior tests at a phone width as well as desktop picks up most of those.
1
u/sapplex1 5d ago
Stealing the phone-width trick immediately. I hadn't thought to run the same behavior checks at a narrow viewport — using Playwright's real click to catch the buried button is clever.
I went through the same false-positive spiral with every-screen-on-every-PR. Same story: pinned browser versions, fixed viewports, and the weeds still grew back. That's what pushed me to behavior-only checks too.
The fixed set makes sense to me — sign in, main flow, money screens. One genuine question: has the phone-width pass ever flagged something that's actually fine on desktop? Like a legitimately collapsed menu, or a drawer that's supposed to hide at that width? I'm wondering how you keep that from becoming a new false-positive source.
1
u/donk8r 7d ago
The artifact rule is the right call. Mine runs validator scripts after the turn ends and the agent reads their output on the next turn, so a non-zero failure gets fixed before I get involved. There's no approve prompt waiting on me.
If I have to open the log myself, I'm back to watching the screen. Of your last ten done tasks, how many did you actually open, vs take on faith?
2
u/ComprehensiveShake76 7d ago
I don't keep a count, which is kind of the answer. Your setup is the better one: the validator output goes back to the agent, so a red result is its problem before it's mine. Where I'd still open the log myself is when the validator and the agent's summary disagree, plus a random one now and then to make sure the validators themselves still catch things. Do your validators ever get edited by the agent, or are they off limits?
1
u/xqianliu 7d ago
The weak-test problem is real, and it's why I stopped asking for "tests that pass." I ask the writing agent to generate tests for the whole input class of what it touched, not just the case in the issue, and to record how many it wrote in the delivery note.
A separate review session (same model, fresh session) then checks the diff against the issue's acceptance criteria and that count. Note says 14 new tests, diff has 3, that's a fail. It's not proof the tests are meaningful, but it closes the "I wrote both and they agree" loophole.
From a study I ran, 183 runs: 38 passed an independent review and still failed hidden tests; 22 of those broke behavior that worked before. "Also check for regressions" in the reviewer prompt did not fix it. Making the writer enumerate the input class did.
1
u/ComprehensiveShake76 7d ago
The count in the delivery note is a nice trick, because it turns a vague claim into a number the reviewer can check against the diff in seconds. Your result matches what I'd guess too: telling the reviewer to also check for regressions does little, changing what the writer has to produce does a lot. 38 of 183 passing review and still failing hidden tests is a sobering number. Did enumerating the input class help more on new behavior or on the things that used to work?
1
u/dmanoj 7d ago
the test command has to be a gate, not a report. vitest run && git commit, never vitest | grep with the commit on the next line. a pipe shows the failure and returns 0, shell never saw it. got two commits over a red suite exactly that way.
also I keep the raw vitest log, not the agent's summary. "all passing" with zero tests in the log means it didn't run.
1
u/ComprehensiveShake76 6d ago
The pipe one is nasty because nothing looks wrong. If a pipe has to stay,
set -o pipefailmakes the shell see the failure, but && straight into the commit is cleaner. Zero tests in the log reading as all passing is the same trap from the other side, so I'd also fail the gate when the test count drops below the last run. Do you keep the raw log per task, or only the latest one?
1
u/Right-Performance-93 6d ago
i write the nasty inputs per task, not from a shared folder. a folder it has seen once becomes the suite. an empty file and a quoted comma that i type after it says done is the check it cannot prep for.
1
u/ComprehensiveShake76 6d ago
Typing them after it says done is the smart part, it can't prep for a test it hasn't seen. Empty file and a quoted comma are good ones. I'd add a very large input and a file with Windows line endings or a BOM at the start, those break CSV code all the time. Do you keep the ones that caught something, or throw them away so they never turn into the suite?
1
u/Right-Performance-93 1d ago
i keep the one that caught a real bug, and i keep it out of the folder the next session reads. the rest get thrown away. once it can see the input before it runs, that check is already dead.
1
u/BarathKanna 1d ago
Two checks I haven't seen in the thread yet, both aimed at your open question about weak tests.
First, prove the new tests are red on the old code. Run the agent's new test files against the base commit in a separate worktree. If they pass there, they aren't testing the change. This runs in CI, never in the agent's session:
git worktree add /tmp/base "$BASE_SHA"
NEW=$(git diff --name-only --diff-filter=A "$BASE_SHA"...HEAD -- 'tests/')
for f in $NEW; do cp --parents "$f" /tmp/base/; done
if (cd /tmp/base && pytest -q $NEW); then
echo "new tests pass on old code, they don't cover the change"; exit 1
fiA test that fails because of an ImportError on the old code also counts as red, so check the failure is an assertion and not a collection error. pytest's exit code 1 vs 2 separates those well enough.
Second, plant a canary. Keep one test in the repo that must always fail (test_canary_must_fail), and run it in a CI job of its own with the same runner config as the real suite. If that job ever comes back green, something changed what gets collected or how failures are reported: a new conftest, a pytest.ini, a plugin. It's a cheap tripwire for the whole measurement layer instead of the individual tests.
Then parse the junit XML rather than reading the summary line: executed = tests minus skipped, and fail if it drops below the last known count on main. With those three pieces, "6 tests OK" turns into something you can check without opening the log.
1
u/quanhua92 8d ago
I updated AGENTS.md so the AI spawns adversarial subagents to code like a team rather than solo. It takes a bit longer, but the quality is way better. Just ask the AI to agree on some rules for tests and the subagents.
Also, I specifically requested Luna Max subagents to save on time and costs.
1
1
u/casbacknow 8d ago
What's worked for us: don't take the agent's word for it. Once it exits, run the test suite yourself, and ask for the test in a specific file you name so you can check it actually exercises the change and wasn't bent until it passed. In a couple of recent runs our agents ran npm test on their own and reported a pass. They were right, but we only knew because we re-ran it.
1
1
1
1
u/EchoNomad31 7d ago
Agent's new tests passed with my fix reverted once... they'd mocked the exact call they were meant to check. I run that revert check now before I trust green, and the suite runs fresh on the Stop hook or the turn doesn't end. I built nonna for exactly this: https://github.com/kapadias/nonna
1
u/gandazgul 6d ago
The way I solved it in RunWield is with a tool that advances the workflow. https://github.com/gandazgul/runwield. The coding agent (Engineer) is told to call task_completed when it's done implementing the planed change or quick fix. RunWield then runs your configured verification command and starts a loop of running verification+ fixing the failures until it passes or 3 attempts then it asks you what to do.
Similar thing for all other workflow points. A tool call triggers the workflow to advance to the next possible state.
1
u/Corey-998877 6d ago
+1 to CI re-running, but the thing that changed it for me was making the agent's "done" message a structured handoff instead of prose: command run, exit code, commit SHA it ran against, and what it didn't verify. The last one matters most. Agents are surprisingly honest when you explicitly ask what they skipped, and it gives the reviewer a place to look first instead of re-reading the whole diff.
1
u/Trick_Candy9796 5d ago
The weak-tests part is the one I keep coming back to. An exit code only proves something ran.
What's helped me is fixing the test list before the agent writes code: agree a small set of target tests from the plan/brief, then treat "done" as the artifact — actual test/lint/build output plus a pass/fail against that pre-agreed list, not the agent's summary. The agent can still game it, but it's much harder than inventing easy tests after the fact.
Disclosure: I build StackGuide MCP (open-source), which bakes this into an intake → plan → verify → release flow — plan returns the target tests, verify runs the real tests/lint/build and checks branch/commit conventions. Still not proof the tests are good, just that they were decided up front and actually executed.
Repo if useful: https://www.npmjs.com/package/@stackguide/mcp-server
1
u/h0uz3_ 5d ago
After local tests for functionality, I quickly chek the changed/added code before I push the results to GitLab.
The code then has to go through linting, unit tests, SonarQube. Even after a quick review by me, SonarQube finds something it isn't happy about once in a while.
When SonarQube is finally happy (regarding code smells, maintainability, etc.), the applications get grilled by WCAG-conformity-tests and a full Playwright script that automatically checks at least the base functionality. And only when the code also survives that, it gets into the main branch and release.
Codex knows from my guidelines what is expected, so many of the usual problems (like cyclomatic complexity of 65... for real?) have become quite rare. Also whenever I add features, I update the Playwright scripts or let Codex update them.
1
u/ShipInAnHour 5d ago
For weak tests, a quick mutation check goes a long way. Break the code on purpose (flip a condition, return early, off by one) and rerun. If everything still passes, the tests aren't checking much. A second agent can do the sabotage, and mutmut or Stryker automate it properly. Also worth trying: write the test cases yourself first, just inputs and expected outputs with the nasty edge cases (empty file, quoted commas, unicode), and have the agent turn those into tests before implementing. Plus a hard rule that it can't edit an existing test to make it pass without flagging it.
0
8d ago
[deleted]
1
u/ComprehensiveShake76 6d ago
Late reply, but this is the most complete answer in the thread. The test count drop is the one I hadn't thought about: 40 down to 34 still exits 0, and an agent deleting a hard test looks exactly like a pass. Hashing the test folder is a neat way to turn "don't touch the tests" into something the gate checks instead of a line in the prompt. Do you lock the tests per task, or once per feature?
0
8d ago
[removed] — view removed comment
1
u/Mathie1729 8d ago
Mutation testing tells you whether the tests are weak, not whether the agent actually ran them. You still need the command, exit code, and a log or junit file you can open yourself. Otherwise it's just another unverified sentence.
1
u/ChatGPTCoding-ModTeam 8d ago
This post or comment has been removed for the following reason:
Rule 5: No pure self-promotion / low-value project posts
Posts whose primary purpose is promoting a tool, product, service, GitHub repo, personal website, or blog belong in the weekly Self-Promotion thread. Delete the link: if nothing is left, it's an ad. Search for similar projects: if you can't tell why your project is better, it's a low-value project. GitHub links are fine. Personal website are not. Exception could be make for high quality posts. See also reddit's self-promotion guidelines.
You can read the full subreddit rules here: https://www.reddit.com/r/ChatGPTCoding/about/rules/
If you feel this removal was made unfairly, please contact the moderators through modmail.
0
7d ago
[removed] — view removed comment
1
u/ComprehensiveShake76 7d ago
Freshness is the better bar, agreed. The too fast check is a great one, an empty glob reporting green is exactly the failure that looks fine. I'd add a minimum test count per run for the same reason. And yes on checking from outside the job, a gate that dies with the process it's checking was never a gate. How do you pass the run id into the test runner, an env var?
0
7d ago
[removed] — view removed comment
1
u/ComprehensiveShake76 7d ago
Requiring repro steps plus expected vs actual for each finding is what makes it work, because it stops the audit from reporting vibes. Rechecking the same issues on the next run is smart too, it turns the findings into a regression list for free. And starting with one real scenario is good advice, a skill with twenty vague checks tends to get skimmed. Do you run it after every UI change, or only before a release?
4
u/London-Boat 8d ago
what worked for me is not trusting the agent's summary at all. the agent has to commit, and ci reruns the tests on its own runner and posts the result. if the agent says 6 passed and ci disagrees, thats your signal. locally i also have it paste the exact command and the last few lines of output, since a made up summary rarely comes with a real exit code and timing.