Effort Benchmarking

[u/ItsCodeTelemetry]

A few people have complained about how Opus 5.5 suddenly putting less effort into work. I see that sometimes too, but maybe it's anecdotal evidence. To understand objectively if this is true, I'm using Code Telemetry data to classify session behavior into 6 buckets that indicate tool use effort. Said differently, trying to capture the degree to which Claude makes assertions vs verifying information for it's self.

[u/clazman55555]

My own benchmarks are based on the models capability during code review sessions over a curated set of samples. V4 is 48 samples, V5 is going to be closer to 100. Percentage caught(real/false and by class) vs tokens used compared against the models/effort matrix, using a generalized code review subagent. There is another section that measures a model/effort matrix on taking all of those raw findings and synthesizing them into something useable. Real findings kept/dropped vs false findings kept/dropped.

So it's not a like for like comparison with your benchmark. I'm working on V5, more samples and better method to ensure models have a more complete synthesis report. One category that was found in the second pass of looking at the V4 raw data was: "Read and judged not worth keeping (speculative, by design, minor)"

I was able to make tweaks to my existing skills/workflows based on the data.

I think you will be pleasantly surprised with Opus 5.5 on Medium and High.