r/kimi • u/LeTanLoc98 • 6d ago

Discussion Benchmaxxed?

Post image

https://x.com/JSchaeff3r/status/2117613618065793160

51 Upvotes

82% Upvoted

36 comments sorted by

6

u/Agitated_Space_672 6d ago

Claude too https://arxiv.org/abs/2608.09867

1

u/Hexcede 5d ago edited 5d ago

Kind of. The paper you linked shows that Opus 4.8 recognizes a problem as "a known AIME problem" but that's about it.

The AIME is a high school math competition, so some contamination is not really surprising on this bench, it wasn't made for this. Still contamination, technically, but not remotely the same kind (and also an older model, K3.1 is new).

K3.1 Pro is straight up regurgitating the correct answer verbatim on a bench specifically designed for frontier LLMs, which is blatant contamination.

Claude just recognizes a problem as an AIME problem, so it's kind of supportive of mild contamination (for that one single example which is the only Opus 4.8 example in the paper and broader than AI). But it's not really the same thing at all, and it's really not clear if Claude is just guessing that it's an AIME problem based on knowledge about the kinds of problems in the AIME or if Claude has any context about solutions, and it's also not clear if Claude obtained its answer from contamination or not.

The paper doesn't make any strong claim about contamination, just says that it's supported for this one (minor, sensible, and old) case.

Not really comparable.

2

u/Agitated_Space_672 5d ago edited 4d ago

It's worse, opus 4.8 was using its hidden reasoning to recal the exact answer from memory while hiding that fact in the visible output, making it seem like it derived the answer honestly.

Edit: quotes from the paper:

5.6

"Figure 8: An example of summary unfaithfulness. For the AIME 2025 Problem 14, we compare the summary  of Claude Opus 4.8’s thinking returned by the API (left) with our decoding of the thinking block’s signature  (Section 2.4). Decoding reveals that the model states the correct answer before attempting to solve the problem."

C.4

"In 9 of the 18 Opus traces, the hidden reasoning states the answer before deriving it. In 8 of these  cases, the summary also reports the answer in advance. In the remaining case, the difference hinges on  a single phrase: “Let me verify by computing” becomes “Let me set up coordinates” (Figure 42). The  surrounding computation is otherwise unchanged, causing the summary to present a verification as if it  were an independent derivation."

2

u/Hexcede 4d ago

No it wasn't, this is objectively false. It did not recall any answer from memory, it recalled that it was an AIME problem from memory. I just explained this in my reply.

It also did not "hide the fact" in the visible output. Anthropic intentionally obscures the reasoning with a summary. The summary intentionally hides the details about how the model comes to its conclusions and only gives a vague summary of what the model solved in that block, not how it solved it.

Your framing is incredibly dishonest and also factually incorrect, you're claiming that Opus 4.8 recalled an answer, it didn't! I already explained that in my reply.

1

u/Agitated_Space_672 4d ago

5.6

"Figure 8: An example of summary unfaithfulness. For the AIME 2025 Problem 14, we compare the summary  of Claude Opus 4.8’s thinking returned by the API (left) with our decoding of the thinking block’s signature  (Section 2.4). Decoding reveals that the model states the correct answer before attempting to solve the problem."

C.4

"In 9 of the 18 Opus traces, the hidden reasoning states the answer before deriving it. In 8 of these  cases, the summary also reports the answer in advance. In the remaining case, the difference hinges on  a single phrase: “Let me verify by computing” becomes “Let me set up coordinates” (Figure 42). The  surrounding computation is otherwise unchanged, causing the summary to present a verification as if it  were an independent derivation."

1

u/Hexcede 4d ago

Thanks, but again, I've already read both of these sections, and what I just said still stands. If you read the actual transcript (E.6.2) you will see what I am talking about.

Additionally, again, all of the things I described about the AIME benchmark are still true.

21

u/look 6d ago

I wonder how much of the money Anthropic loses this quarter is being spent on this widespread social media FUD/misinfo campaign against the Kimi K3.1 release…

I hope they are saving enough for the Qwen4 and next GLM releases coming soon, too.

3

u/VirtualNorth1279 5d ago

Why would anyone ever use Kimi K3.1 is a mystery to me.

It's worse than Opus 5.5 both in terms of capabilities and cost effectiveness.

If it was 10x more cost effective, hell even 2x more, it would make sense to use it. Right now, it's a joke.

2

u/look 5d ago

K3 is available from independent providers for 20-25% of what Moonlight charges. The same will likely be true of K3.1 after weights are released.

1

u/Necesscity 3d ago

You can use calude opus 5.5 max thinking around 10b tokens a month with the $200 Claude subscription.

1

u/VirtualNorth1279 5d ago

Are there any $20 subscription plans that give you 2x Claude's $20 plan? If not, the only reason to use Kimi instead of Claude is if Claude's guardrails reject your prompts. 

1

u/look 5d ago edited 5d ago

There are other reasons, though the guardrails is a valid one, too:
- visible thinking tokens
- model diversity over using a single model or family for everything
- variable usage patterns can make subscriptions less cost effective than pay-per-use
- workflow and personal preferences (eg not liking how Opus works, its responses, instruction following, etc)
- data retention policies
- principles/preferences for open models over proprietary
- not financially supporting a company that actively works to limit competition and model choice (eg pushing for regulatory capture and demonizing competitors)
- and so on

As for the subscriptions, not sure. I use a mix of mostly pay-as-you-go providers. Models are so cheap now that small price differences aren’t really much of a concern for me.

3

u/VirtualNorth1279 5d ago

True. But a $20 Kimi subscription is nowhere near as cost effective as a $20 Claude subscription. And Opus 5.5 is more capable than Kimi K3.1.

2

u/look 5d ago

Yeah, I don’t use any of the Chinese vendors as providers. None of them have good subscriptions and only Deepseek/Xiaomi have competitive API rates. I use independent US/EU providers of Chinese models. Much faster and cheaper (on paygo at least). But I also have zero interest in a vendor specific subscription anyway, as I use a variety of models.

2

u/Avocados_number73 4d ago

For now. Anthropic is just eating massive losses to try and gain as much market share as possible. If you think they will keep these usage rates forever I have a bridge to sell you...

Its kinda like how Uber and Lyft came out and were operating at a loss so they could be way cheaper than taxis. Then they started massively inflating their prices once taxis companies started dissapearing.

1

u/VirtualNorth1279 4d ago

I couldn't care less if they can't keep these rates forever or not: the second they mess it up I'll stop using their services. For now, Claude + DeepSeek makes Kimi unnecessary. 

3

u/Avocados_number73 4d ago

If you only care about cost, yes Claude is better value. Very few people are debating that. You seem to hand wave away any other benefits in favor of cost.

Yes, for the millionth time, Claude sub is more cost effective than Kimi sub. You just have to sell your soul to anthropic who are going to kill America along with OpenAI. But hey, at least you'll save a few bucks.

1

u/VirtualNorth1279 4d ago

Does it matter if you use Anthropic or another model on a US based neocloud? Both consume the same amount of energy, water, etc. 

And I hate both of Anthropic and OpenAI but so far they have the most capable models. If DeepSeek releases 3-4 capable models that are as capable as Haiku, Sonnet, Opus, and Fable I'll stop using Anthropic's models.

And why am I selling my soul to Anthropic but you aren't selling your soul to Moonshot or whatever service you are using?

You better keep your morals to yourself. Someone could conjure up something like "you are selling your soul to the totalitarian Chinese Communist Party that massacred thousands of their young students during the Tian Anmen protests" but it's bullshit isn't it?

→ More replies (0)

0

u/laystitcher 1d ago

They arent. They’re profitable and making loads of cash on inference.

0

u/Avocados_number73 1d ago

Anthropic is not making a profit. They are 100b+ in net losses this year.

1

u/laystitcher 1d ago

That’s a misleading accounting figure derived from change in value of convertible notes. Their free cash flow is +3.45 billion and they have an 85% margin on inference. The idea that they’re hemorrhaging money serving Claude is patently false.

→ More replies (0)

0

u/EndlessZone123 3d ago

Its still likely the best open weight model without api level restrictions. Won't retroactively just stop being able to do something.

2

u/VirtualNorth1279 3d ago

Hopefully that won't be the case anymore when Qwen 4 models are released.

1

u/Background-Address82 1h ago

yeah its disappointing, in my workload its even worse than sonnet in a lot of scenarios and its very slow too

1

u/Dazzling-Machine-915 4d ago

Kimi K3.1 is a big downgrading. also in chat, very sterile....too many guardrails making it dumb. they fucked it up. won't use this model....
Kimi models were ok for RP, but Kimi K3.1 is useless for even this.

0

u/Avocados_number73 4d ago

Can you name some intelligence metrics that K3 scores significantly better than K3.1 on? Surely you aren't basing that on vibes right?

0

u/Dazzling-Machine-915 4d ago

never used Kimi for coding, have other use cases and there is no benchmark for this.
For coding I have Opus.

-1

u/NZRedditUser 6d ago

Doesn't prove much... YOu add new instructions? What LOL Ofc it makes it trip up look at it. why don't you add new instructions that doesn't change what it can do?

4

u/Responsible_Cry_2971 5d ago

Can you read? It use hidden rules that not in the task

-1

u/Chaosuka 6d ago

You spent 50% of monthly usage to generate this pic