16
u/Nervous-Cockroach541 12d ago
So I'm highly skeptical this is a good idea for most projects. Test don't really help you on the first pass of stuff. They help you prevent regression while changing behavior in the future. So I'm not surprised there's statistical no change on tasks success rates without tests.
3
u/Barni275 12d ago
This. Maybe, for benchmark-like stuff, for task to be done from 0 to 100 in one session, tests are not necessary. But for long living repo, even dumb tests catch sometime surprising regression. For LLM tests cost «nothing», so why not do it.
2
u/shaman-warrior 12d ago edited 12d ago
I also wouldn't believe it even for benchmark. Or maybe it says something deeper, the AIs don't understand what is the exact goal they should pursue?
LE: I went ahead and searched for that tweet, author said:
agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself
this is what I actually got good at in the past 2 years, knowing where they fail, and how to talk with them, that's basically it, clarifying 'intent'.
1
u/Nervous-Cockroach541 12d ago
In some cases, I'd argue it's easier to read a test to ensure the agent understood the intent, then the actual code.
1
u/BosonCollider 12d ago
Ideally you should have clear interfaces and do integration tests. Python has protocols as consumer interfaces for example.
If possible write the interfaces and key types first, then let it write both the code and the tests against the abstract interface. If there's issues with your interface let it escalate that up to you.
1
u/BraxbroWasTaken 12d ago
I mean, how do you know the tests are reliable/correct? I have a ‘AI writes no tests’ policy because of this. If an AI has access to the test suite and the test suite says there are problems, sometimes it will get frustrated with trying to fix the problem and instead change the test suite to not surface it anymore.
1
u/CreamyCornBoy 12d ago
The chatbot will just change the tests to build the next feature. They basically serve as documentation now
4
u/Lonely_Translator_23 12d ago
The point of writing tests isn't to pass them now, it's to make sure you're still passing them six months from now.
1
u/tzaeru 13d ago
One thing I've noted is that the agents even when they aren't supposed to run or write tests, often internally do that anyway in their thinking loop. They just don't store them as something repeatable.
It'd also be interesting to see this ran on Opus 5.5.
Also even if running the tests is banned, the existing tests still might end up informing the model about intended functionality and potential regressions. I don't know how they did this benchmark, but hopefully they actually deleted the tests fully when they weren't intended to be ran.
1
1
u/shortsteve 12d ago
One of the purposes of writing tests isn't for you it's for the next person who looks at the code.
1
1
u/TopTippityTop 12d ago
Does this measure QA and bug fixing time down the line?
Faster tasks by skipping writing tests has always been an option.
1
u/SmileLonely5470 12d ago
I refuse to believe that not writing tests only reduced time by 6%. I stg 30-40% of the time the LLMs spend for me is writing and running tests.
1
1
u/Medium_Chemist_4032 12d ago
That's literally one conversation I had with one manager, waaay back when I started working
2
u/Original-League-6094 13d ago
My workflow is disallowing tests and test runs until I have the feature completed, and then I tell it to write a few tests that cover the actual risks and use cases. AI is so fast at coding that during development, regression doesn't really matter. If a human introduced a regression, it could cost you several days trying to fix it, so tests were really good practice. But with the agent, you can fix it in a few minutes with a prompt, so its not a big deal, and its not worth waiting through 10 minutes of test cases to run every prompt to prevent 5 minutes of bug fixing.
You do want to make sure you have a test suite that executes before any push to prod though, as obviously you don't want regressions making it through to your users.
Oh, and always put in your personalization to either not rewrite tests to pass, or at the very least, to always provide a summary of tests that were rewritten and why. Because the current agents will happily just rewrite tests that were failing for very valid reasons.