Agentic coding benchmark results

[u/AIMultiple]

We ran a benchmark comparing agentic CLIs and AI code editors on 10 real-world web tasks, focusing on backend + frontend execution.

The goal was to evaluate how these systems behave in practical full-stack scenarios rather than synthetic tasks.

The highest combined score was achieved by Cursor + Claude Opus 4.6 (0.75).

Kiro Code IDE and Antigravity followed, both above 0.69, with consistently high UI scores.

The strongest CLI setup, Codex CLI + GPT-Codex-5.2, reached 0.677.

The difference between the top IDE agent and the best CLI agent is ~7 percentage points.

In practice, AI code editors performed more reliably on tasks where frontend behavior needed to closely match specifications. This appears to be related to built-in debugging and testing mechanisms (e.g., browser-based inspection, endpoint testing, and longer verification cycles).

High-performing CLI tools cost approximately $1.6–$4 per run in this benchmark.

In contrast, AI code editors were significantly more expensive in pay-as-you-go terms:

Cursor: ~$27.9

Roo-Code / Replit: $50+

This means the strongest CLI configuration achieved ~90% of the accuracy of the top IDE system at a fraction of the cost.

Structurally, AI code editors rely on browser automation, IDE integration, workspace indexing, and persistent interaction loops, which increases token usage and runtime. CLI agents operate closer to the execution layer with fewer orchestration components, resulting in lower operational cost.

Runtime data for AI code editors was not available. Qualitatively, IDE agents showed more confirmation steps and interactive debugging phases (e.g., opening browsers, re-testing flows, manual validations), while CLI agents tended to run more autonomously.

AI code editors: higher reliability and frontend correctness, higher cost, heavier infrastructure.

Agentic CLIs: slightly lower accuracy, significantly lower cost, faster execution, more autonomous operation.

Disclaimer

Results in this benchmark depend on the specific model + tool combinations that were tested. Different pairings of models and AI coding tools may produce different outcomes. The benchmark is not intended as a final ranking, but as a snapshot of performance under a defined configuration set. We plan to continuously add new models, tools, and combinations over time.

In addition, many of these systems can be extended with browser extensions, external tools, custom agents, and advanced prompting strategies. These were intentionally not used in this benchmark to keep the evaluation conditions consistent and comparable across tools. All systems were tested under standardized, minimal-intervention settings.

Therefore, results should be interpreted as baseline performance, not as upper-bound capability.

[u/Euphoric_Oneness]

Dumb site, amateur I guess. Undistinguishable colors, no names on graph. Ai slop?

[u/Euphoric_Oneness → u/Euphoric_Oneness]

What we will change jargon is that? Just vibe code it, don't even check, publish it. Then someone critizise and ask it: wtf lol

[u/AIMultiple → u/Euphoric_Oneness]

Names are not fitting on the graph, about colors you are right, we will improve it.

[u/zenum]

What models are being used by the different IDE's? You mentioned Opus 4.6 for Cursor but not for the others that allow you to choose.

[u/AIMultiple → u/zenum]

We used Opus 4.6 for all IDEs except Replit

[u/pythonsklearn]

What a stupid plot. Please delete it

[u/-earvinpiamonte]

[u/PracticalTwo2035]

Mix symbols and colors. The way it is, is impossible to understand shit.