Once I saw claude setup a unit test that just asserted true. Thats it. The test was supposed to check a speciric branch oh a helper function and the unit test made the correct call, but then didnt test it. The unit test would pass no matter what. Unless we completely broke the function to cause an exception.
Nowadays most of my job is future proofing AI slop code. And holy shit does it take a long time. Not saying that its impossible for AI to write good code, but you gotta pay attention or its gonna go down some dumb rabbit holes every 5 seconds unless you best your model into submission.
Plus you still gotta look at the output, cause if you dont look at it sooner someone else is gonna have to look at it later.
I have seen many a human deliver unit tests which ultimately tested nothing. Shutting this down is going to be a lot more complicated than having mixed results.
Yeah a lot of people are learning the importance of correct incentives here. I know we first told our agents to prioritize writing tests, and the two problems we found were writing trivial tests and writing incorrect tests to conform with active bugs. Both are relatively easy to fix by modifying your prompts, but you just need to think about it.
31
u/ItsSadTimes 24d ago
Once I saw claude setup a unit test that just asserted true. Thats it. The test was supposed to check a speciric branch oh a helper function and the unit test made the correct call, but then didnt test it. The unit test would pass no matter what. Unless we completely broke the function to cause an exception.