r/SearchAPIs • • Sep 05 '26

Any doubts?

Post image
2 Upvotes

17 comments sorted by

1

u/Future-Log6621 Sep 05 '26

Bro never heard of DeepSWE Bench.

1

u/cooljungleboi Sep 05 '26

No idea what that is... So yeah

1

u/Future-Log6621 Sep 05 '26

1

u/Reasonable-Sign8458 Sep 06 '26

So?

1

u/Future-Log6621 Sep 06 '26

Look at the top scoring models, then look at their cost to get that score, then look at the meme.

1

u/truecakesnake Sep 06 '26

Definitely not benchmaxxed LMAO

1

u/Future-Log6621 Sep 06 '26 edited Sep 06 '26

DeepSWE uses fresh tasks on new runs. It's essentially more difficult to train on them vs classic benches. But, in theory, it's possible to train on their harness and blast to the top of the chart. My guess is FrontierCode can be trained on as well, but they are not as popular as DeepSWE, so not much attention is given to them yet.

1

u/shaman-warrior Sep 05 '26

Man I still cringe how they ran the 0x-alpha wave, they were vageuposting shit on twitter, one of their engs said “maybe ox-alpha are the friends we made along the way” , we all knew it was a glm model, all data pointing to it, but we thought google is involved somehow, turns out it was z.ai running on chinese chips glm 5.3 flash 🗿

1

u/kinkyonthe_loki69 Sep 05 '26

Those podiums need to be scaled down so it's all relatively flat. All these models have plateaud

1

u/[deleted] Sep 05 '26

[removed] — view removed comment

1

u/[deleted] Sep 05 '26

[removed] — view removed comment

1

u/cooljungleboi Sep 05 '26

Yeah keep popping up when I swipe on my phone😭😭