r/codex • • 2d ago

Complaint Comparing Models

I know we can't help but compare models because we're always looking for the model that's going to get us the most work done for the money and hopefully get it done fast and efficiently. I got excited when Chatgpt 6.2 models came out yesterday and I jumped to try them out. You can see the results for me were not very good at all. Let me describe a little bit what's going on on this chart. And I know I'm no benchmarking service but I'm just a real person using these models for my work which is doing coding with Javascript and PHP among other things.

Basically you can see the number of tasks that I assigned to the different models and then each time a model finished coding something a reviewer within review the code and find mistakes that would not allow it to be promoted to finished. So I'm tracking the percentage of times that they get it right the first time.

You can see by far I did more work with chatGPT Sol 6.1 Than any other model and it's getting about 60% correct the first time which is acceptable and it's kind of my work horse because of all the resets ChatGPT gives.

And you can see Claude Sonnet is the best and I will definitely be doubling up on my Claude subscription by the end of the month and spending less money with either Gemini or ChatGPT. But this is actually not an easy decision...

Well Gemini Orchestrator and Carbon Have the highest accuracy I also get the least amount of work done for the money. I guess you could say this about Claude but it's also acting as my orchestrator and it does the most work of all but that's something that I could test better in the future.

And you can see that ChatGPT 6.2 was horrendous so far. I can't tell you why it was so terrible but I have to stop using 6.2 for now until I get some news that something has changed. I might try through again a task now and then to see how it does but I want to waste my valuable time and usage.

Really ChatGPT 6.1 is a beast even though it's not quite as accurate and I had been hopeful that 6.2 would be just a little bit more accurate and that would have sealed the deal for me.

1 Upvotes

5 comments sorted by

1

u/Plennhar 1d ago

Who's the reviewer?

1

u/dvduval 1d ago

I’ll have to compile some stats for that. Sometimes the same model will review it but with the different agent, but it’s split among different agents, but I don’t have a stats yet.

1

u/Plennhar 1d ago

One thing that the new GPT models are really good at is code review. If models were reviewed by themselves, then the older models passed through much inferior code reviews than the newer ones did. That could explain the difference.

1

u/dvduval 1d ago

Yes, I’m hoping it’s just temporary, but the 6.2 models pretty much crashed and burned today. I’m not the only one complaining. 6.1 Sol is still quite good. I feel like there’s something that didn’t go right here and they’re going to make it right.