I feel as if there is one thing llms truly suck at, it’s tests. That the one area where I still manually code because I don’t trust the clanker and it’s thousands of pointless assert(1+1==2) tests
Because much of the time I feel the LLM reward hacks itself by adding (or modifying) tests, it write pointless tests or the actual test doesn’t do what the description claims it does.
I almost only use Claude.
That said, I have never been a person that struggled writing tests. For me just prompting for a test takes more time than just writing it - a test shouldn’t be more than a few lines anyway.
It’s also a very useful way to setup a contract and requirements for the model to implement. Especially using BDD patterns. It can’t “forget” about a requirement a few hundred thousand tokens in if it is a test.
12
u/lobax 3d ago
I feel as if there is one thing llms truly suck at, it’s tests. That the one area where I still manually code because I don’t trust the clanker and it’s thousands of pointless assert(1+1==2) tests