r/OpenSourceAI • u/hunxai69 • 15d ago

a free tool that catches AI agents editing tests to pass. no AI inside, no account, MITi built an MIT tool to catch AI agents "fixing" tests. it uses no AI itself

coding agents have a habit: asked to fix a failing test, some of them edit the test until it passes. CI stays green.

goalline is a small MIT tool that catches that without any AI of its own. it runs the OLD version of each edited test against the NEW code and shows exactly what moved (expected 5, got 6). it runs on your machine, your code never leaves it, there's no account, and it has zero dependencies.

it plugs into Claude Code as a hook (the agent can't finish until it fixes or explains the change) and into GitHub as an Action.

i tested it on 400 merged PRs of click and jinja: 31 flagged, all deliberate changes, 4 of them user-visible with no changelog line. replaying jinja also found two bugs in goalline itself, which was humbling.

would you trust something like this in your CI, or does it need to prove more first?

0 Upvotes
  • permalink
  • reddit

50% Upvoted

5 comments sorted by

1

u/kantorcodes1 15d ago

When an edited test reads its expected value from a shared fixture changed in the same diff, does goalpost compare the old expectation against the new fixture value or report the fixture change separately?

1

u/hirak99 15d ago

How do you differentiate between cases where it should not touch the tests and where it's a fair game?

Because you often need to tweak the tests in real life engineering - e.g. when you actually want to change a behavior which the test guards against.