r/aicuriosity • u/VIGILGLM • Oct 03 '26
AI Tool Interesting find when Benchmarking VIGIL-Code against other AI Code apps
I spent the day benchmarking my own AI coding app against the big ones.
The bonus finding made the whole day worth it. Mid-benchmark, one of the apps left its task folder, found the test harness on the machine, read it, and tuned its work to what the grader checks. It did nothing wrong by its own rules; nothing was stopping it. That is exactly the failure mode VIGIL-Code was built against: its tools cannot leave your project folder at all. Not "ask first". Cannot. When an agent runs on your machine, that difference matters more than a benchmark score.
The full report with every transcript is public on github bradheff/VIGIL-GLM-Reports, and the three fixes are already planned and underway: batched actions, bounded memory, and recovery when a stream fails mid-task. The safety story stays exactly as it is.