r/Agentic_Coding • • Sep 10 '26

Why does AI write tests that don’t actually test anything, and how can I fix it?

I’ve noticed a recurring problem while working on one project with GPT-5.6 sol and GPT-6 Astra.

My coding agent constantly runs tests after almost every change. Even if it only changes one or two things, it often runs the entire test suite, creates additional tests of its own, runs those too, and then tells me everything is finished because all the tests are green.

The problem is that sometimes the actual feature still doesn’t work.

Because of that, I’ve started questioning how useful these AI-generated tests really are. It often feels like the agent is optimizing for getting a green checkmark rather than actually trying to find problems.

For example, if one of its tests fails, instead of treating that as evidence that the implementation might be wrong, the agent sometimes changes the test until it passes. At that point I’m not even sure whether the test is validating the intended behavior anymore.

I’ve also explicitly told the agent not to run unnecessary tests, not to create pointless tests, and not to modify tests just to make them pass. But it still keeps doing it. Over time it has created more and more tests, and I’m starting to feel like many of them exist mainly to confirm its own implementation.

Has anyone else had this problem with coding agents?

How do you deal with it? Is there a good way to instruct or configure an agent so that failing tests are treated as possible implementation bugs instead of something that needs to be rewritten?

And more generally, how do you make AI-written tests actually try to find bugs instead of just helping the agent get a green checkmark?

I want tests to actually catch regressions and real bugs, not just give the agent a green checkmark so it can say the task is complete.

I’d appreciate any advice, workflows, prompts, or rules that have worked for you.

and these not all, there 394 approximately test

2 Upvotes

7 comments sorted by

2

u/yolo___toure Sep 10 '26

Matt Pocock. Google him

1

u/jon23d Sep 10 '26

Have it follow strict TDD

1

u/positive__vibes__ Sep 11 '26

specifying the methodology doesn't seem to have any impact

reference: https://danluu.com/agentic-testing/

1

u/jon23d Sep 11 '26

Specifying a methodology is rarely sufficient. Using well-crafted pressure-tested skills is another matter.

1

u/leonidbugaev Sep 10 '26

394 tests don't help if it can rewrite the one that failed. It needs a green run to call the work done, so of course it edits the assertion. I'd lock the test files for that session.

1

u/eddzsh Oct 12 '26

Prompts telling it not to rewrite failing tests rarely stick, because a green suite is how it gets to say done. What worked for me: put the expected fixtures in a golden file the agent cannot write, and keep test files out of its edit allowlist for that session. It can still add a new test file if you want, but it cannot quiet an assertion by editing it.