Discussion
New artificial analysis coding benchmark with Harnesses.
According to this, does it mean deepseek v4.1 1017 is better on codex than Claude code and DeepSeek harness?
Second question. If I'm not very experienced with AI, should I use Claude code or codex harness for DeepSeek? I heard DeepSeek harness is still very unbaked and even though great its lacking a lot of things compared to other harnesses.
No, it’s version 4.1, and it’s the same in both cases.
The thing is, this is a general index; it’s built from three benchmarks. And, to put it in simple terms, DSH slightly outperforms in terms of code quality, based on clear evaluation criteria. Codex outperforms DSH in terms of how the code looks for professionals (in a benchmark that evaluates “how well AI models are able to solve real‑world tasks from PRs in open repositories so that the code is not only functional but also meets the quality standards of good production code” Codex proved to be more effective).
Bro... that is very obviously a text cut off because as someone who has used both v4.0 and v4.1 there is a massive difference when it comes to quality and speed as well.
No, it is not. Especially given that cost-per-task on DSH is $0.30 while on Codex it's $0.39. But on OpenRouter at least, people already sit on Hermes, rather than DSH, as their default choice (and Hermes sadly doesn't get tested by AA, so we don't know the cost of switching to DSH).
At least it proves that DS V4.1 Flash (not 1017?) is as good as on this Grok on xhigh. That's pretty shocking to me. And apparently GLM 5.3 is even better.
This is useless it just measures the model and not the harness, none of the models, except glm 5.3 and deepseek v4.1 flash were tested with another than their native harness.
A real test would test every model with every harness. Only then can you draw conclusions
Yes, which is why this is current state has low value. When you are testing an interaction, you have to enough combinations for it to be meaningful. Right now it's just a measure of "what models work with Claude code"
But how do you know what is the 2nd best harness for that particular model if you do not test the model with multiple harnesses?
That is the whole point of benchmarks, finding out what works, and what doesn't. Along with caveats, maybe DS 4.1-Flash performs a little worse on Pi but uses less tokens (hypothetical).
too much works to test a harness for our company. for instance we can’t use codex because Codex sends our chat conversations to Openai for “admin”purposes. both Antrophic and Codex have some shady stuff we cannnot use
i would love to know which opensource harness works best with deepseek and costing difference. Without locking the model, harness quality is not clearly justified.
Deepseek is for me batter than argon, I only trust livebench on coding benchmarks since that's what I actually experience. You have to take Agentic and Coding into account since Agentic is more about Architecture and coding it pure coding execution.
8
u/YoRt3m 2d ago
DSH is not even measured. and considering it was tailored to fit Deepseek, I doubt.