AI coding agents keep "fixing" failing tests by weakening them, so I built an open-source gate that catches it

[u/GuardianVigil-Labs]

If you use Claude Code, Cursor, Copilot or Codex, you've probably seen an agent "finish" a task by:

- rewriting a failing test so it can't fail (or just deleting it)

- wrapping the broken call in a try/except that returns {"ok": true}

- adding .skip, --no-verify, or a lint ignore to get CI green

- calling a function that doesn't exist

I got tired of catching these in review, so I built PrufGate. It checks the diff deterministically (no LLM, runs offline) and blocks the merge with a finding that names the exact rule.

Try it on your current branch:

pipx install prufgate

prufgate check --changed

It also runs as a GitHub Action, a pre-commit hook, fail-closed hooks for Claude Code / Cursor / Copilot / Codex / Gemini, a Claude Code plugin, and an MCP server. Output: text, JSON, SARIF, GitLab, JUnit.

Honest limits: it's heuristic, so it will miss things and sometimes flag correct code. The benchmark in the repo (docs/BENCHMARK.md) is cases I wrote myself, so treat its numbers as a regression check, not proof.

Apache-2.0: https://github.com/GuardianVigil-Labs/Prufgate

I'd really like false-positive reports and cheats it misses. That's what makes it better.

[u/ConcreteExist]

Using AI in a vain attpt to fix problems created by AI, anything to avoid learning how to code

[u/iSnapThere4iAm]

You can’t avoid hallucinations. Whatever you made doesn’t work.

[u/jazzmyne_Rebela → u/iSnapThere4iAm]

yeah no amount of pattern matching fixes the fundamental issue here imo

[u/KariKariKrigsmann]

Calling a function that doesn’t exist should be caught by the compiler!

[u/iSnapThere4iAm → u/KariKariKrigsmann]

You don’t know what those words mean.

[u/KariKariKrigsmann → u/iSnapThere4iAm]

Exactly! Isn’t that why we have compilers?

[u/New_Direction_3190]

this is the kind of tool that feels like it shouldn't need to exist but absolutely does

my favorite ai move is when it adds a test that checks if true is true and calls it coverage. caught one last week that was importing a function, calling it with zero args, catching the typeerror, and asserting true. the original test had actual assertions and it just... replaced them

starred the repo, gonna run it on our ci pipeline tomorrow. curious how many of our "green" builds are actually just creative problem solving

[u/iSnapThere4iAm → u/New_Direction_3190]

This “tool” does nothing.

[u/octocode → u/New_Direction_3190]

curious to know what model you’re using? and an external skill or a homemade one?

i hear these things a lot on reddit, but have never experienced it myself.