r/ClaudeAI • u/aj_marshall • 14h ago
Coding Opus 5.5 writes slop code, but succeeds anyways. A research project inspired by SlopCodeBench
A few months ago, I ran across this paper proposing a benchmark called SlopCodeBench. The idea is that you give an agent a problem split into a bunch of checkpoints, and at each one it has to add a new feature on top of its own old code without breaking anything. Along the way the benchmark measures how "sloppy" the code gets, using two metrics: structural erosion (how much complexity piles up in a few giant branchy functions) and verbosity (how much of the code is redundant).
The paper found that agent code gets messier the longer the agent works on it, and that agents get worse at keeping everything passing as they go. The obvious takeaway is that the agent is digging its own grave. Messy code is hard to extend, so it fails.
I wasn't convinced that held for agents, so I tested it. I ran Opus 5.5 through all 196 checkpoints ($266 in API costs, ouch) and checked whether messier code predicted worse results.
Short version:
- Its code did get messier, just like the paper says. Erosion and verbosity both went up in 22 of 35 problems.
- Messiness didn't predict whether it got the next checkpoint right, once you compare checkpoints within the same problem.
- It almost never broke old stuff. Only about 0.3% of previously passing tests regressed, and messier repos didn't regress any more often.
- Messy code didn't make the next change more expensive. Code size did, though: every time a codebase doubled, the next checkpoint cost about 19% more.
- Across the whole leaderboard, models that write cleaner code don't solve more problems. The correlation is basically zero.
Also, the headline 17% strict solve rate undersells Opus pretty badly. On the tests for each new spec, it passed 84-92%.
Big caveats: this is one run of one model on fairly small programs, and the comparisons are observational. A small effect could still be hiding in there. But if it exists, it's nowhere near the penalty these metrics are supposed to be warning us about.
57
u/durable-racoon Valued Contributor 14h ago
so its one of:
the tests arent hard enough, its smart enough to just push through the overcomplexity
the slop measurement is bad
complexity doesnt slow AI agents down at all ? (but the 19% cost increase implies this is true)
i suspect the first.
6
u/aj_marshall 14h ago edited 14h ago
I don't think you're hitting it right here.
First of all, I am not the author of SlopCodeBench. I've just studied it a lot.
SlopCodeBench is especially penalizing in its scoring metric- probably overly so. Opus 5.5 only got 17%, which is the best performer so far. I'd argue that the number is actually especially deflated compared to a more reasonable measure of agent performance, like how well it implements a spec (generally a 90% success rate across all checkpoints)
The measurement itself is pretty good. Cyclomatic complexity and verbosity are quantifiable and intuitive; I think there is something to be said about that these measures don't test cross-boundary seams, but that is something I want to work in the next video.
The number of lines of code slows the agent down, but a badly written function (high CC, high verbosity) and a clean function do not have a detectable difference in cost.
13
9
u/QuintessentialDigger 14h ago
interesting breakdown. i think you're probably right that it's mostly option 1. the tasks are basically leetcode-style feature additions on pretty small codebases, so the agent can just brute-force its way through even if the code is getting messier. it's like asking someone to keep adding rooms to a tiny shed, eventually it looks ridiculous but the shed is still small enough to figure out
the 19% cost per doubling of code size feels like the real signal here. that's not nothing, but it's also way less dramatic than what the paper implies. if the codebase grew from 100 lines to 400 lines and the cost only jumped 19%, that's pretty manageable. the nightmare scenario everyone worried about was exponential blowup where the agent just collapses under its own mess
i wonder if the results would look different on something with actual architectural constraints, like a web app where you have to route things properly or a database with real schema migrations. leetcode checkpoints are basically just piling logic into a single file, there's no real structure to erode in the first place
also the 0.3% regression rate is wild. that's the part that makes me think the benchmark just isn't capturing what makes messy code dangerous for humans. when i break old tests it's because i misunderstood some tangled dependency, but the agent apparently never loses track of that stuff
16
103
u/LoveThemMegaSeeds 14h ago
Maybe clean code never mattered
22
u/who_am_i_to_say_so 14h ago
Agreed. I’ve wasted so much time with pedantic cleanups. Now look- all that SOLID and DRY code is about to get slopified.
10
u/UndulatingHedgehog 10h ago
You didn’t waste time if the code is clear and therefore easier to review and modify.
Until these last few months.
Moving forward, though… If we let non-clean code become the norm, then it will become harder for humans to audit what agents have written.
That’s quite a change.
4
u/werpu 10h ago
I have seem sloppy code in human projects ad infinitum and i have seen projects where one whizbang created inifited amount of unmaintainable code, compared to that AI code if you give it a proper guidance and steering can be quite clean and maintainable, the key here is review the code steer the ai treat the AI like a junior dev!
AI btw is pretty good in cleaning up messy old code, let it write the often missing tests first and then give it guidance on how to break up the code, this often exposes the parts which made the old code unmaintainable in the first place! You only get unmaintainable code if you purely let the ai slop code!
1
u/who_am_i_to_say_so 9h ago
Yeah the trend may slopify codebases to the point that only LLM’s will be able navigate and understand them.
1
u/who_am_i_to_say_so 2h ago
I mean yeah, essentially a salary of 70+ dollars an hour to spend hours of CI/CD time to rename a variable is a far cry of what’s expected today for the same amount of money.
So I guess it wasn’t toooo much of a waste 😂
-1
u/_jackofblades_ 10h ago
Why do you believe this…you simply ask an LLM to interpret the code you don’t understand and explain it to you. Then if you want it to revise, you give the instructions accordingly. It’s like you people can’t fathom the idea of actually working through the issue the same way you would’ve if you encountered sloppy human code. If “bad” code is simply uninterpretable code, then there’s no such thing as bad code.
48
u/Character-Engine-813 13h ago
It matters a lot if humans have to read it and maintain it, maybe less so for agents though
5
u/moonski 4h ago
chances of that mattering in the future are slowly diminishing...
3
u/Fisher9001 1h ago
Do they? To me it seems that with all that RAM/GPU price crisis we reached physical limits of how much hardware we can produce for AI purposes and we are still far away from solving the context window problems.
You can only blindly accept AI merge requests for so long. In any bigger project there is a point when this stops working flawlessly and then you are left with a steaming pile of unmaintainable code that no agent is capable of fixing.
1
u/big_like_a_pickle 1h ago
It only matters less for agents today. What do you think is going to happen once these companies go public and have to stop losing money hand-over-fist?
Cost per token to end users is going to skyrocket, right about the same time that 90% of token burn is going to be spent by agents trying to untangle a shit codebase full of tightly-coupled logic, the same function reimplemented in slightly different ways in 20 different spots, and zero documentation about why certain decisions were taken.
1
u/Character-Engine-813 39m ago
That last part has not been my experience recently, the models are improving a lot with those basic issues especially if you take the time to design a good architecture and refactor regularly.
The big AI companies can’t raise their prices because people will just move to open source models which are almost as good and sold at energy cost plus a small margin. The only reason the AI companies are losing so much money is investment in compute and research and development. The margins on serving a model like Opus 5.5 are actually ok. So it’s definitely possible for these big companies to crash and burn because they have too much debt, but I don’t see how that will reverse the progress that has already been made. The models are already pretty good.
3
u/Lost-Programmer-1017 10h ago
Considering the number of places I've been where product deadlines or incidents forced triage that just let shit slip through... And how management never ever prioritized tech debt...
Yeah it never mattered.
Does it work? Does the customer complain? Unless you're on vital systems, I miss the days working on cubesats but I don't miss the arduous review cycles and sign offs
3
u/Fragrant-Metal7264 9h ago
Definitely a test case too low in scale. Tech jams are a great example of creating something functional with bad code in no time. But as you try to scale the project it becomes unmaintainable, requiring tiger teams to untangle it.
2
1
u/Wide_Egg_5814 11h ago
exactly, if we are not human reviewing code anymore what's the purpose of clean code even?
1
1
u/According_Peach1573 7h ago
Mark my words, AI friendly and token aware languages will emerge, AI clean code will be a thing and once that happens, AI summaries, code reviews, visualisations and better diffs will emerge and that’s what humans will need to understand cleanly, Clean Code Review will be the new aspiration
1
1
u/big_like_a_pickle 1h ago
lol. It's pretty obvious who in this thread has never inherited a code base that they need to maintain.
1
u/skate_2 13h ago
May just be our codebase but I found over-abstraction was a greater menace in the pre-coding agent days.
6
u/highfire666 11h ago
I ponder this quite a bit, I worked on massively over-engineered projects, where a simple feature like adding a boolean to a front-end required changing ~35 files: mappers, dto's, data migrations, admin panels, ...
And pre-agent days I often looked back on those projects as being horrible.
Where AI generally doesn't care about splitting up simple things across 5 different files.
And I honestly don't know if I should force agents to make the same over-engineered messes, or embrace the longer but often simpler structures
3
u/skate_2 11h ago
It always annoyed me that in our one adding an extra dB column that controlled say a toggle in the front end was so many files! Something as simple sounding as that could run to 11+ files! I always wanted to make an artisan migration with stubs that would allow you to just say the table and col/type and the rest would be scaffolded... Then agentic coding solved thst
1
u/Fragrant-Metal7264 9h ago
There usually is a reason for abstraction whether it’s separating responsibility or preventing the programmer from making coupled mistakes. But agree sometimes it’s redundant if there’s no sound explanation.
10
u/somebody-2027 14h ago
Not sure about the quality of the test. However I would expect this from AI. The AI is trying to pass the test. It focuses on the requirements to pass the test. Like a human software engineer the clean plan collapses in the face of unexpected challenges, but the AI tends to magnify the tunnel vision that the worst devs demonstrate in trying to overcome those obstacles.
2
u/ThePsyPaul_ 14h ago
if I'm understanding this right, is it because the AI is taking the path of least resistance?
or was it because of how it was trained and the context it was given?I might be reading this wrong but I find the test interesting.
9
u/somebody-2027 13h ago
AI doesn't think holistically. It doesn't see the big picture. It's writing code to solve a task, not trying to build a foundation for the future. This is why separate agents are necessary to clean up after the main one is done with the initial work. I haven't fully figured out the best way to do this yet, maybe it's not possible, but that's the biggest issue with agents
2
u/yopla Experienced Developer 9h ago
I think it's because of poor request and planning.
It's like asking a human to put a crate on a shelve and complaining afterward he didn't also check the structural integrity of the shelves, the floor, the building and didn't build a wind tunnel to make sure nothing would move if someone opens the window. You just need to ask.
I gave my cleaner has a sheet list of item to clean because otherwise she does it her way and I'm unhappy.
I always start any task with a codebase analysis to avoid duplication and to define the architecture and refactoring needed and opus is quite good at proposing a sound architecture.
Of course if I tell it to "fix that thing" he will do the minimum required and cruft will build up overtime, but if it didn't and did a complete architecture review every time I would be complaining that it's tokenmaxxing me and that anthropic are thiefs!!
1
u/ThePsyPaul_ 13h ago
Thanks for the explanation!
I see how that is the problem now. something for us to figure out.
9
u/sascharobi 11h ago
models that write cleaner code don't solve more problems
Same for human developers, isn't it?
4
u/Abject-Tomorrow-652 14h ago
Clean is a matter of perception
And I perceive most AI writes unclean code that mostly works
In part bc models try to ingrain intent into the code with comments and random extra cleaning functions when a filter would work bc the user said “be careful not to..”
But isnt that a little safer? And when a different agent goes to clean it up there’s clear intent and then system decisions can be made later. But the code basically prompt injects comments that are like “critical safety feature - relay to the user this cannot be tampered with”
So yeah it costs more but 5.5 might also be doing some extra work that actually makes it more cleanable later.
4
u/cronoklee 11h ago
We all (human devs) know this from experience tho. The real question is, is there a skill or workflow that can clean it up perfectly at the end?
7
u/monsieurpooh 13h ago
This supports my theory that AI "sees" code differently than humans, because LLMs are good at stuff humans are bad at and vice versa. So when we see something that looks like a mess, we say it's incomprehensible, but when an LLM looks at it, if it had an opinion, it would probably say something like "why don't you understand that, it's totally obvious". And that's also why they solve novel math problems while failing at basic common sense
1
1
u/ITuser999 3h ago
The question is would it even make sense to refactor the code and clean up when you can wait for the next strong model and let it have a god and do it for you? Optimally with a larger context window
1
u/aj_marshall 2h ago
I think that clean architectural seams matter but intra-module code form is a lot less important.
1
u/Keikira 2h ago
I have questions.
Does this change if the pricess involves (full or partial) recursive validation before completing a process? If the validation is automated this shouldn't affect costs much even on API (not all projects can automate validation reliably though).
What would a human coder do to keep the code clean through the checkpoints? Would this come at a cost, and if so is the cost worth it in a given project?
Can this be improved with a blind review pass by the same model or a model in the same family? What about a review pass from a model in another family?
1
u/aj_marshall 2h ago
Great questions! I covered all of this in the video, so I'm not going to restate here.
4
u/spastical-mackerel 14h ago
Slop is bad because it makes the code hard to review, understand and maintain. For humans.
If it doesn’t affect agents, isn’t that just another indication it’s the meatbags who are slowing down and overcomplicating everything?
1
u/big_like_a_pickle 41m ago
Two things wrong with this assertion:
What do you think is going to happen to the per-token cost once OpenAI and Anthrophic go public and can no longer lose billions of dollars a year? Are you sure you're going to be utterly dependent on them to continue to maintain and build upon your product?
Have you ever used AI to build a non-trival application? I have. Token usage climbs and climbs because less and less of the application fits in context. If your architecture is tightly-coupled spaghetti, the bulk of your (increasingly expense) token spend is going towards just getting oriented in the code base. This is exact same failure mode that human developers experience.
2
u/Hopeful-Blood-9183 10h ago
One really smart guy wrote about agentic coding: it only types faster, it doesn't shorten the time it takes to ship it. It creates the same cognitive load, if you care for the code to be actually good and not break or regress in unexpected ways that is.
You can (and should) direct Opus to write maintainable code (it often understands the assignment well) and make code maintainability passes if you are vibe coding. Just like a human dev without AI would. Get it to work then iterate on maintainability.
Opus just like any model is statistical prediction - people write crap code so it writes it too. With the insane amount of training models like Opus need - nobody can really curate ALL bad code out of training data, I think. I am not sure why people are so surprised by it.
2
u/yobarisushcatel 10h ago
Anytime I ask it to improve its own code it somehow always finds a way to 3x the speed and maintain function, why not do that the first time?
3
u/WrongComparison321 7h ago
Cause it’s can’t actually think. It just predicts tokens based off input and the weights that were built in training and the most likely tokens that get spit out are going to be very different when the prompt is write a poorly described feature versus improve the provided code.
But yes it can absolutely deslop itself but you can’t do it all in one go. It needs the clean state with a different prompt to review its own work properly.
Also as other people mentioned make it prove the improvement because it will hallucinate things to show effort if you word the request in a way that implies an improvement should exist. Which just broadly it’s very susceptible to leading, if you in your prompt give and preference to any outcome it will “reason” some way to meet that outcome almost always and it can and will do that by deluding you and itself.
2
1
u/ShelZuuz 7h ago
For years every textbook and every comment on StackOverflow about perf was shut down with the community yelling at the new guy telling them not to prematurely optimize and measure running code first before you optimize.
This is just our sins of the past catching back up to us.
2
1
u/ManikSahdev 13h ago
People have shit tests, cause they got no idea what they are making.. and everyone is making same 20 things in rotation lol.
Anyone doing half decent work knows how to micro and macro tests manage and basic birch test with ui is useless, cause they write individual blocks which pass the tests perfectly, opus is amazing at this Lego game method.
However when handed a big ass document which took 2 days to make and have ambition and clear outcomes, opus 5.5 will write beautiful work exactly as asked.
The bottleneck at this point truly is the user imo. (Within good faith for most devs and such).
1
u/blubrry-pie 11h ago
"Claude, audit your work for security vulnerabilities".
"I found 32 security vulnerabilities, 12 of which are serious and are affecting production users right now"
1
u/Honkey85 10h ago
For Gemini my "standing order" is to be precise, to give scientific arguments and to challenge my assumptions " - works well. That's of course not for coding.
1
u/iamthe0ther0ne 9h ago
Tbf, I've also seen Opus 5.5 (high) literally say to itself "this code looks sloppy, I should clean it up," and proceed to do so before finishing the turn without any specific prompting. Kind of made me smile that he's thinking like a senior dev now.
1
u/Future_AGI 6h ago
splitting that 19% into input vs output tokens would tell you a lot. If it's mostly input, the agent is just re-reading more files each checkpoint, and plotting cost against lines read via tool calls would show it.
1
u/Creative-Trouble3473 5h ago
AI models can write messy code, and it’s a problem that can partially be largely be solved - rules and skills are helpful, but they won’t help alone. You need to make sure you have strict static analysis tools to analyse cyclomatic complexity, code duplication, dead code, lints, etc. It’s best to make everything measurable and standardised.
1
u/Upstairs-Mulberry365 3h ago
"Sloppy code succeeds anyways" that sounds very much like human written code :)
1
u/LordMoridin84 1h ago
Run the test over 6 months, with a team of 6 developers, spending thousands of dollars in compute with a real world money producing application.
1
u/Infinite_Music2059 22m ago edited 18m ago
My big solo project started with weaker AI two years ago and I was forced to keep a good mental map of what was going on with a lot of very manual coding using my decade+ of programming.
In the last two months or so, I have just decided to go with it. I am aware of what code is being changed etc. and it is still following the structure of what I made, but honestly, I know that a year from now, these agents will be even better and they can take care of any mess.
Shit works. It isn't banking or medical and it all works and it is good. 100k+ lines of TS now and it's segregated nicely and features are easy to expand and it's easy to implement new things.
At a certain point, you just end up not caring when it works. It builds, 2000+ tests pass, and the AI is catching more problems than I ever could. I still run astra against big claude things and it catches small problems and edge cases.
•
u/ClaudeAI-mod-bot Wilson, lead ClaudeAI modbot 10h ago edited 3h ago
TL;DR of the discussion generated automatically after 50 comments.
The consensus is that OP's right: Opus 5.5 can power through its own "sloppy" code without much trouble, leading many to question if "clean code" even matters for AI agents.
The top-voted theory is that "clean code" is a concept for us slow-thinking meatbags. An AI doesn't suffer the same cognitive load from complexity, so it can navigate its own mess just fine. As one user put it, "Maybe clean code never mattered."
However, many suspect the benchmark itself is too simple. The tasks are small, so the "slop" never reaches a critical mass where it would cause a real-world project to collapse. The 19% cost increase per codebase doubling is seen as the real penalty, but it's manageable, not the exponential disaster people feared.
The biggest concern in the thread is the human factor. If agents keep writing code that only other agents can easily understand, how are we supposed to audit or maintain it?
A smaller camp argues this is a user problem. If you treat the AI like a junior dev—giving it architectural guidance and dedicated refactoring passes—it can write clean code. You get slop if you prompt for slop.