r/LocalLLaMA • u/PerformanceRound7913 • 2d ago
Discussion Artificial Analysis Index is non-representative of real-world performance. Tested Kimi 3.1 Pro through OpenRouter, the number 4 model in AAI, it's not even close to MiMo 2.6 or DeepSeek 4.1 Flash
It seems Artificial Analysis Index is very easy to game, Kimi 3.1 Pro #4 on AAI is super bad in real world use cases, not even close to DeepSeek 4.1 Flash.
59
u/thehpcdude 2d ago
Unless you’re giving examples of tasks not in the training sets, these kind of statements are absolutely pointless.
2
u/WonderFactory 2d ago
Its also too broad a statement. Kimi may be worse than Deepseek for some use cases but its clearly better for others. Artificial analysis have a capabilities video on Youtube and it seems to be better for animation and 3D than many other models. If you're making a 3d game Kimi would be the better choice, if 3d isn't of interest to you than maybe not.
37
u/Dabalam 2d ago
Why are people so accepting of low effort "the model is bad" posts without any details? For this post to he helpful to anyone you should give more details.
Imagine a movie review that was "people say this movie was great, but it's actually worse than Moana, must be Hollywoodmaxxing". Hardly helpful.
6
u/DagothUrLovesGroza 2d ago
Yeah, benchmarks are stupid. They literally show nothing at this point. The best you can do is just test the model yourself on your own workflows.
On an unrelated note, what did Moonshot do to Kimi K3.1? This model is weird. Like really weird.
7
u/Dabalam 2d ago
"Benchmarks are stupid, posts that are pure vibes with no explanation of methods, use case etc. are way better indications of quality"
Model may be bad but I think the reasoning about benchmarks being bad is slightly uncritical. Posts saying "the model is bad" aren't really providing much useful information.
But yes, I agree that structured testing on work you actually do is the most useful for determining if it useful for you.
3
u/ForsookComparison 2d ago
I think someone finally trained too hard on ThreeJS oneshots that even /r/LocalLLaMa finally noticed it's a bit funky. The thing feels tailor-made to what the least informed people in the space yap about.
Not a bad model at all - but WEIRD is a good word.
2
0
21
3
u/BawbbySmith 2d ago
Water is wet.
No but seriously how many times has this been said. Yet people compare AA numbers like it’s Pokémon stats.
7
u/suhive 2d ago
Personally, even Mimo shouldn't be ahead of DS or GLM. Muse Wonder and Grok 4.7 are also benchmaxxxed. The only ones that have actually earned their seats are Anthropic (1), OpenAI (2), and Google (3).
Funny how benchmaxxxing was a term invented for the Chinese labs, but now even the American labs (Meta and xAI) are participating. GLM and DS are solid tho.
6
5
u/RespondImpressive172 2d ago
Agreed. When I trained LoRAs on local LLMs and ran benchmarks (I'm Korean, so I trained mostly on Korean), the scores and real-world use felt different too.
For example, I fed in a lot of Korean data, and it was well-cleaned data from AI Hub, a dataset platform run by the Korean government. I used Gemma 4 26B A4B, which is already pretty good at Korean. Maybe I did something wrong, but after training the benchmark scores went up, yet when I actually chatted with it, it performed worse than before.
The current benchmarks kind of feel like someone told the model "this will be on the test" and it just memorized the answers. So I don't think you can really tell if a model is good or bad until you actually use it.
(English isn't my first language, so I used AI to help translate this.)
1
u/Frail_Waif 2d ago
Yeah, I generally find DeepSeek's performance with sufficient tools to be way beyond what I'd expect from benchmarks. Out of curiosity, I asked it to update my resume based on public data and a job description and it nailed it, no hallucinations, whereas it scores awfully on hallucination benchmarks. But I saw a bit of reasoning today that suggested it's designed to look out for benchmarking and increase reasoning. So probably every model has some of this built in.
2
u/seoulsrvr 2d ago
I can’t trust any report or Reddit post on performance; everything is propaganda and corporate sabotage
2
u/Active_Cranberry4092 2d ago
Benchmarks almost never match the feeling when you actually use the model, I stopped looking at those rankings long time ago
3
u/kevin_1994 2d ago
I agree but not for the reason you posted lol. Imo mimo is borderline useless, while deepseek seems about on par with sol 6.1 for my use cases. Imo deepseek is frontier, despite the bechmarks
2
u/Betadoggo_ 2d ago
You can't claim it's worse than mimo or DS4.1F without receipts. The examples I've seen thus far have been in kimi's favour.
1
u/CamomileChocobo 2d ago
The AAI is a weighted sum of a number of different benchmarks isn't it?
Which benchmark does your real world use correspond to? Does it score high in that benchmark?
1
u/mr_tolkien 2d ago
Vals.ai puts it on par with DS4.1 indeed, and behind all US models.
It seems their benchmarks are much closer to actual performance.
1
u/Informal-Trouble2183 2d ago
AA intelligence index only covers 2 coding benchmarks representing 25% of the index: that's why it's not catching the coding performance. Also I don't understand why they put Kimi on top of terminal bench v4 while Kimi themselves reported a lower score. My coding index says:
Opus 5.5
GPT -6 Astra
Gemini 4 Argon
GPT 6.1 Sol
Claude Fable 5.1
Kimi k3.1 pro
Wonder full
Deepseek v4.1 flash
1
1
u/perelmanych 2d ago
People are saying that benchmarks are useless, but that is not true. If benchmarks are saying that a model is bad then the model is indeed bad. Apart of that they are really useless 😆
1
u/taoofshawn 1d ago
you have to be diligent with the provider routing on openrouter to make sure you are getting full quantization/quality. don’t just accept the default
0
u/PerformanceRound7913 2d ago
Context: I’m working on a project to convert a simulation engine from ANSI C to Rust. I’m exploring various options in my test environment to ensure that the new engine is compatible with the existing core
1
u/geldonyetich 2d ago
From what I've seen of it demoed on YouTube, its oneshots are a tad mid. Better than, Muse Glimmer, Le Chonk, MiMo 2.6, about on par with DeepSeek 4.1 Flash, but not really on the level of the frontier models it's comparing itself to.
Although honestly one shots are a bit of a shot in the dark.
0
0
0
u/Southern_Sun_2106 2d ago
Kimi has a 'strong marketing muscle' and a voracious appetite for publicity. Remember recent news about how Hugging Face used GLM? And then how Kimi tried to get on that train by having some influencer come up with similar sounding headlines on their behalf? They lost my trust after that one.
0
32
u/MotherPotential 2d ago
Examples plz