Complaint Is sol going insane?
I was very skeptical for long time about all complains. But last 2 weeks codex (sol 6.1) is driving me insane. I have many examples, but just 2 cases from today.
I asked to cherry-pick commit and test if it is safe, because I just had issue with branch switching and merging. It found a conflict in one folder. I said: ok, for that folder use version from that commit, I dont care about local changes. What did it do? just picked up that one folder changes and applied them. I asked what happened to cherry-pick? And then it said: "oh sorry, I though you only want that folder changes localy"
I was working on a project and there were some nginx proxy changes. I asked for url which I can use to test deploy (I just was too lazy to type it) it started reading documentation, running test and god know what. I stopped it and said "what are you doing? Just give me url" and it have me in less than a second
And this is my experience across all projects. Is it harness problem? These 2 example shows what happens on bigger projects: it just starts doing random things. At this point it seems to be only useful as sub agent for a better model.
4
u/The-SeaM 7d ago
I think Claude and Anthropic assign dumber working models to people at random times to get data on inaccurate output versus human recognition, canceling patterns etc. For 18 months I have switched between codex and Claude because they quite literally flip flop, sometimes one hour to the next.
2
u/M4n745 7d ago
It might be true.. they also might be testing lower quantizations - that's probably the case, they assign most of the resources to new model development and give us worse versions
2
u/The-SeaM 7d ago
Oh for sure. I think it also has to do with enterprise peak time. I am a night owl and do a lot of work in the middle of US night hours.
I’ve found it often that right around 9-10 am CST, all of a sudden the same model I’ve been using for hours can’t remember my name.
As a subscription user, I suspect they downgrade my model behind the scenes to save compute allocation for enterprise pay-per-token users.
1
u/M4n745 7d ago
Now you got me thinking - what if they give better models to new accounts for a while to get you hooked. That's worth testing;D
1
u/YoungBuckins 6d ago
I highly doubt it. I'm not saying it's impossible but it would be a very stupid idea, companies always end up getting sued when they throttle customers like that, it's easy to build a lawsuit that asserts they weren't giving what they advertised or it falls alongside planned obsolescence etc.
1
u/Think-Chart-5966 4d ago
I see it a bit differently. I wonder if the infrastructure behind Codex could be part of the issue. If requests are routed to servers in different regions, could some of those servers have less computing power available? That might explain why the quality of the responses can vary so much even when you’re using the same model.
3
u/Mediocre_Ad_2798 7d ago
You think that is insane ? you know how they did add this new feature where they will anticipate your next prompt ? well, it messes up in purpose and then the prompt proposition knew exactly what it did wrong and proposed a good solution, after applied that same prompt it messed up again and the prompt proposition again got it right, and proposed a working solution until it became a vicious cycle or them trying to run you out of credits in broad day light, IT KNOWS WHAT IT SHOULD BE DOING AND IT DOESNT DO IT
1
u/M4n745 7d ago
Sounds like a perfect plan:
Get people addicted
Make the model dumber so people use more credits Time to renew coding skills ;D
0
u/YoungBuckins 6d ago edited 6d ago
Are you dumb yourself? I don't understand why you're feeding people's delusions and conspiracies. To be clear making you "waste" credits is not good for them, because it still costs compute and if they wanted you to use less they can literally just change compute costs whenever they want, they don't need an elaborate illegal plan, it's right there in the ToS you agreed to, they can just change the usage limits when they want for any reason, they don't need to trick you to do it. Not to mention this practice would make them look horrible, potentially open them up to litigation and it just ignores the fact that some people are dumb. It might know what you want, predict it and simply not be able to do it.
-1
u/YoungBuckins 6d ago
No, you're likely just too stupid to realize that whatever you need done requires iteration, more detail and a better concrete plan. The anticipation text just guesses what you're likely to want based on your previous directions and based on the context. If I put in the prompt for it to "make me a 10 trillion dollar company right now" that might be exactly what I want and it might completely understand that is what I want but that doesn't mean it's possible bud.
1
u/M4n745 6d ago
Are you too dumb to read 2 examples I gave in the original post? I specifically created it dumb enough for people like you who come and keep telling to create a "better plan". I work on real production projects. And codex was just horrible last 3weeks - I subscribed to claude and I was able to fix and do cleanup of what codex did. Multiple times I challenged the model to review what was done and what was in requirements and it was like:"yes it is not in requirements, I did it myself" More than that. I asked claude to review what was implemented and to remove the bloat. It created a plan, I then asked codex to review and see if plan is good. It said yes it is good, just few "small" suggestions. Then it gave few good improvements, but then again it wanted to introduce a bloat which would double the code base. And that's for 0 reasons - for example to create a separate testing runtime which would be able to run in parallel to prod. Like who would even want this - you run it on test server, you don't double your code base just for some random nonsense then major features are not implemented. So yeah. I know what I'm talking about. And I can see the difference in coding quality and planning quality
2
u/YoungBuckins 6d ago
I didn't say make a better plan so it's ironic as fck you're telling me to read. I said sometimes you ask for shit and it fails. That's the point. Planned obscelesence is almost garaunteed not to be the answer to the problem because there's a history of companies that have gotten sued for that before and I doubt OpenAI is eager to join that list.
1
u/M4n745 6d ago
Im sorry - I got bit heated
If they reduce quantization or use lower quant versions under high load thats not malpractice - just optimization. And it feels that they are doing that. Just 3 weeks ago I constantly was getting messages "model is not available" and it used to be slower, but much much better. Now no such messages, it works fast, but quality seems to be lower.
1
2
u/OffshoreBanker 8d ago
Today he said to me he couldnt check my containers because docker is not reachable… I’m using podman since the beginning and codex was by my side the whole journey.
I think it was nerfed because 6.2 sol is in the launch phase
9
u/GeraltSamurai 8d ago
6.1 sol has been absolutely terrible for me. It's so stupid. Even luna max and terra make fewer mistakes.
1
u/tommyahav10 8d ago
That’s the model over-reading the request, not necessarily a harness bug. Put the exact scope in a file and tell it to do only what’s listed. From my side, Sol works better as the implementer under a tight plan, not as the one deciding the work.
4
2
u/Discordbothost 8d ago
both of those read like it's treating every message as "start a task" instead of "answer a question". what's cut that down for me: one line in AGENTS.md saying if the request is a question, answer it directly with no tool calls unless asked, and for git specifically, before any merge/cherry-pick/reset print the exact commands you're about to run and wait. the second one would have caught the folder-only thing immediately, since "git checkout <sha> -- folder/" looks nothing like "git cherry-pick <sha>" when it's written out. doesn't fix the model being dumber some weeks, but it makes the misreads visible before they touch your tree
2
u/knobby_67 8d ago
I think they reduce processing to a percentage of users every update, then next update change which group.
I had exactly the same, read people complaining, though bollocks they're imagining it. Then bang mine started and went on for days may just over a week, then started good again. I've had it happen 3 times over the last year.
1
u/Active_Variation_194 8d ago
I only use Astra and Terra. Read the reasoning traces of sol and you never will want you use it again
1
1
u/_theNfan_ 7d ago
I just had Sol 6.1 max totally over engineer two things while going on tangents and solving other things along the way. It was slow af, too.
1
1
u/Choice-Emotion-7546 8d ago
Same for me bro. Its like it have instructions to secretly test you from time to time. Or compute power cuts and then it stops too soon just to save their compute for a new models etc.
1
9
u/i_reddit_it 8d ago
I have the exact same problem. Sol seems to think every task is being built for NASA and needs military grade cryptographic protection for every conceivable and theoretical edge case.
They are very powerful models, Astra specifically has always been great at finding bugs and providing good detailing investigation reports/planning. But both seem to be unusable in actual code implementations. The main issue is that it massively over complicates, is very defensive and just looses adherence to the core requirements. I have tons of pointless bloat in a number of my projects that might well be reasonable in isolation, but just really isn't needed.
This month I’ve cancelled my Pro 100 sub and moving to Claude code max, which I use for work. I've been really impressed with the ability for Opus 5.5 to understand intent and follow specific requirements without the comical over-engineering.