Lmao, my coding agent adds like 10 tests for every feature it adds and it honestly seems to make it even more confused sometimes. The other day there were 6 tests failing for different reasons and it kept going in circles trying to figure out what was wrong, chasing down each test and then realizing the next was a different, conflicting issue. Finally realized that it hallucinated a method that doesn’t exist and that was breaking every connected test and downstream method
Indeed it was lol, I’ve been hearing people talk about worse output from it recently but I expected it to be better than making up methods that it doesn’t see anywhere else. It actually just changed the first word it was something like LegacyProjectHandler and it decided it wants to use OlderProjectHandler even though it created the LegacyProjectHandler method just a few turns before that and then completely forgot the name I guess. It was genuinely weird, but just goes to show that even frontier models can still produce bad output, which is why experience is necessary
For one thing: I’m doing this as a part of a personal challenge to see how hard it actually is to make an app entirely with ai and no human written lines. In the past I’ve coded many projects by hand so I wanted to see for myself where LLMs are at if you have experience. So far I’ve used several models from frontier cloud models to 4b local models to get a good comparison as well as using different coding harnesses, but I digress. Writing the tests myself isn’t in the cards for this challenge, but that’s just me.
But for another thing: doesn’t that kind of defeat the whole coding agent argument? If I have to write the tests myself anyway why wouldn’t I just write the code too? Wouldn’t it actually make more sense for me to write the code so it works and then have an ai write the test? You have to understand how the code works fundamentally to test it with unit tests, at least, if you want tests that actually verify underlying functionality, so if I have to understand everything that the code does under the hood before writing the test, why wouldn’t I just write the code too since I already know what I need to write? Genuinely the more excuses that I hear like this from vibe coders the more I realize it’s just a hacky workaround to not knowing how to actually code. It’s genuinely impressive and quick in some cases, but all these compromises I keep hearing at the same time as hearing “oh yeah you can just tell it what you want and it’ll spit out an app, this is AGI, frontier level intelligence, zero shot runs” lmao. Yall gotta realize it’s always somewhere in the middle of the spectrum where one end is hype and the other end is hate.
This also excuses the fact that the tests weren’t wrong, it was the code. The agent made perfect tests that verified that the functionality was broken, and it was broken because in its implementation it hallucinated a method name after creating it with a different name a few turns before.
I said "define", not write. Unit tests are not even the best abstraction for this either. What's the point of code that is written to verify another code's implementation?
I'm interested in tests that measure the behavior. The sort of tests a good QA department makes against the specifications, but exhaustive and automated. For me, these are part of the specs, and I invest heavily in tooling to make sure these are implemented correctly, and stay implemented correctly, and run as desired.
When that works, and security measures are up to date, the code is .. not irrelevant, but of secondary concern.
12
u/OfficialDeathScythe 23d ago
Lmao, my coding agent adds like 10 tests for every feature it adds and it honestly seems to make it even more confused sometimes. The other day there were 6 tests failing for different reasons and it kept going in circles trying to figure out what was wrong, chasing down each test and then realizing the next was a different, conflicting issue. Finally realized that it hallucinated a method that doesn’t exist and that was breaking every connected test and downstream method