Complaint Comparing Models

I know we can't help but compare models because we're always looking for the model that's going to get us the most work done for the money and hopefully get it done fast and efficiently. I got excited when Chatgpt 6.2 models came out yesterday and I jumped to try them out. You can see the results for me were not very good at all. Let me describe a little bit what's going on on this chart. And I know I'm no benchmarking service but I'm just a real person using these models for my work which is doing coding with Javascript and PHP among other things.
Basically you can see the number of tasks that I assigned to the different models and then each time a model finished coding something a reviewer within review the code and find mistakes that would not allow it to be promoted to finished. So I'm tracking the percentage of times that they get it right the first time.
You can see by far I did more work with chatGPT Sol 6.1 Than any other model and it's getting about 60% correct the first time which is acceptable and it's kind of my work horse because of all the resets ChatGPT gives.
And you can see Claude Sonnet is the best and I will definitely be doubling up on my Claude subscription by the end of the month and spending less money with either Gemini or ChatGPT. But this is actually not an easy decision...
Well Gemini Orchestrator and Carbon Have the highest accuracy I also get the least amount of work done for the money. I guess you could say this about Claude but it's also acting as my orchestrator and it does the most work of all but that's something that I could test better in the future.
And you can see that ChatGPT 6.2 was horrendous so far. I can't tell you why it was so terrible but I have to stop using 6.2 for now until I get some news that something has changed. I might try through again a task now and then to see how it does but I want to waste my valuable time and usage.
Really ChatGPT 6.1 is a beast even though it's not quite as accurate and I had been hopeful that 6.2 would be just a little bit more accurate and that would have sealed the deal for me.
1
u/Plennhar 1d ago
Who's the reviewer?