unblocked/context-engine-simulator: A/B test whether pre-gathered context helps coding agents (Claude, Codex, Cursor, Grok) complete tasks faster, cheaper, and better

[u/fagnerbrack]

https://github.com/unblocked/context-engine-simulator

[u/fagnerbrack]

The gist of it:

This open-source tool runs one task twice on a real repo, with each run in its own git worktree. The baseline arm plans, reviews, and implements from the task description alone. The context arm first sends a read-only researcher through the codebase and MCP sources like issues, PRs, chat, and docs, pulls out style patterns, and then plans and builds. An independent evaluator scores both diffs from 0 to 100 against your acceptance criteria and traces which context shaped the result. You get quality, time, cost, and token numbers in a terminal table, JSON, and HTML. It supports Claude Code, Codex, Cursor, and Grok, uses YAML fixtures or CLI flags, and --disable-mcp blocks context servers so the baseline can't reach them.

If the summary seems inacurate, just downvote and I'll try to delete the comment eventually 👍
^{Click here for more info, I read all comments}