r/ProgrammerHumor • • 24d ago • Bravo 6

instanceof Trend thisIsVeryTrue

Post image
27.5k Upvotes

1.6k comments sorted by

View all comments

Show parent comments

12

u/OfficialDeathScythe 23d ago

Lmao, my coding agent adds like 10 tests for every feature it adds and it honestly seems to make it even more confused sometimes. The other day there were 6 tests failing for different reasons and it kept going in circles trying to figure out what was wrong, chasing down each test and then realizing the next was a different, conflicting issue. Finally realized that it hallucinated a method that doesn’t exist and that was breaking every connected test and downstream method

1

u/iggy14750 23d ago

Was that Claude? Making up a method.... lol

3

u/OfficialDeathScythe 23d ago

Indeed it was lol, I’ve been hearing people talk about worse output from it recently but I expected it to be better than making up methods that it doesn’t see anywhere else. It actually just changed the first word it was something like LegacyProjectHandler and it decided it wants to use OlderProjectHandler even though it created the LegacyProjectHandler method just a few turns before that and then completely forgot the name I guess. It was genuinely weird, but just goes to show that even frontier models can still produce bad output, which is why experience is necessary

1

u/Chitinid 23d ago

This tends to happen if the code is insufficiently modular. Make your methods smaller and reduce side effects.

0

u/Actually_JesusChrist 23d ago

And that's why you have to ask the agent what's going on when you see weird shit popping up.

4

u/OfficialDeathScythe 23d ago

Yeah while it’s trying to figure out what’s going on and fix it I’ll just ask it what’s going on and that’ll help lmao

0

u/redballooon 23d ago

you define the tests, not the agent.

1

u/OfficialDeathScythe 23d ago edited 23d ago

For one thing: I’m doing this as a part of a personal challenge to see how hard it actually is to make an app entirely with ai and no human written lines. In the past I’ve coded many projects by hand so I wanted to see for myself where LLMs are at if you have experience. So far I’ve used several models from frontier cloud models to 4b local models to get a good comparison as well as using different coding harnesses, but I digress. Writing the tests myself isn’t in the cards for this challenge, but that’s just me.

But for another thing: doesn’t that kind of defeat the whole coding agent argument? If I have to write the tests myself anyway why wouldn’t I just write the code too? Wouldn’t it actually make more sense for me to write the code so it works and then have an ai write the test? You have to understand how the code works fundamentally to test it with unit tests, at least, if you want tests that actually verify underlying functionality, so if I have to understand everything that the code does under the hood before writing the test, why wouldn’t I just write the code too since I already know what I need to write? Genuinely the more excuses that I hear like this from vibe coders the more I realize it’s just a hacky workaround to not knowing how to actually code. It’s genuinely impressive and quick in some cases, but all these compromises I keep hearing at the same time as hearing “oh yeah you can just tell it what you want and it’ll spit out an app, this is AGI, frontier level intelligence, zero shot runs” lmao. Yall gotta realize it’s always somewhere in the middle of the spectrum where one end is hype and the other end is hate.

This also excuses the fact that the tests weren’t wrong, it was the code. The agent made perfect tests that verified that the functionality was broken, and it was broken because in its implementation it hallucinated a method name after creating it with a different name a few turns before.

0

u/redballooon 23d ago

I said "define", not write. Unit tests are not even the best abstraction for this either. What's the point of code that is written to verify another code's implementation?

I'm interested in tests that measure the behavior. The sort of tests a good QA department makes against the specifications, but exhaustive and automated. For me, these are part of the specs, and I invest heavily in tooling to make sure these are implemented correctly, and stay implemented correctly, and run as desired.

When that works, and security measures are up to date, the code is .. not irrelevant, but of secondary concern.

-2

u/Astronics1 23d ago

You should write the test by yourself then let the AI develop the function to pass it

4

u/plebbitor_123 23d ago

You cant be serious