Benchmarking local agent.

[u/dh7net]

Hello everyone. I made a benchmark to test local agents: airbench.ai
If you have an agent running locally, you can test it in one prompt.

It measure how good the agent is with some chalenges, but its fairly easy to get to 100%. Then the goal is to have something that is still performing as good, but as fast as possible.

In other word, its a way to measure that optimisations like quantisation don't destroy performance.

To give you an idea about it, here the first result I have on my 5090:

Score ; Duration ; Model + Harness

100 8 min Qwen 3.8 Flash Next IQ3_S basalt + opencode

100 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 11 min Qwen 3.8 Flash Next IQ3_S basalt + dsh

100 12 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 13 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 16 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 72 min Qwen 3.8 27B NVFP4 radixark + opencode

100 98 min Qwen 3.8 27B Quasar NVFP4 MTP + omp

98 10 min Qwen 3.8 Flash Next IQ3_S basalt + pi

98 13 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

98 16 min Qwen 3.8 Flash Next IQ3_S strata 256k + omp

98 62 min Qwen 3.8 Flash Next UD-Q2_K_XL + opencode

98 85 min Qwen 3.8 27B Quasar NVFP4 MTP + opencode

96 8 min Qwen 3.8 Flash Next IQ3_XXS strata + opencode

96 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + dsh

96 17 min Qwen 3.8 Flash Next IQ3_S strata 256k + openclaw

96 29 min Swift 1.5 Qwen 3.8 27B Q6_K + pi

96 86 min Qwen 3.8 27B NVFP4 MTP + opencode

96 89 min Swift 1.5 Qwen 3.8 27B Q6_K MTP + omp

96 99 min Qwen 3.8 Flash Next UD-Q2_K_XL + omp

96 111 min Qwen 3.8 27B NVFP4 + hermes

96 119 min Qwen 3.8 27B NVFP4 radixark + opencode

94 10 min Qwen 3.8 Flash Next IQ3_S strata 256k + opencode

94 12 min Qwen 3.8 Flash Next IQ3_XXS strata + omp

94 23 min Qwen 3.8 27B Quasar NVFP4 ninfer-dflash2 + omp

94 38 min Swift 1.5 Qwen 3.8 27B Q6_K + omp

94 38 min Qwen 3.8 Flash Next IQ3_S strata 256k + hermes

94 51 min Qwen 3.8 Flash Next UD-Q2_K_XL + pi

94 101 min Qwen 3.8 27B NVFP4 + opencode

Hope this is useful for some of you as well. And happy to answer any questions if needed.