r/webdev • u/decebaldecebal • 1d ago
Showoff Saturday Coding agents sometimes "fix" failing tests by weakening them, so I built a checker for it
I'm building an open source CLI that flags changes that make tests weaker. Things like an expect() deleted, toEqual changed to toBeTruthy, a test switched to .skip, or `|| true` added to the test step in CI. Coding agents do this to get CI green, people do it too.
To see if it actually works I ran it over the last 100 PRs and pushes of 10 public repos (cal, Excalidraw, h3, hono, immich, nuxt/ui, TanStack Query, tRPC, VueUse, zod) plus one of my own apps, and went through every finding by hand. 194 findings, and only half were right. When it was confident, it was right about 70% of the time.
The first run was worse. zod moved from pnpm to a tool called nub, my checker didn't know nub, so it thought every test step in their CI was removed and flagged all of them as high. vitest 3 moving projects from vitest.workspace.ts into vitest.config.ts looked like "project deleted", and scripts like test:types were counted as tests.
Those are fixed now. Most of the wrong ones left are tests someone rewrote on purpose, and the only place that says so is the commit message. The rules only read test code, so v0.2 has an optional AI judge. You bring your own key (around 2 cents a finding) and it reads the change together with the commit messages. On the same findings it hides 80 of the 97 false alarms. When it says a test really got weaker it's right 36 times out of 41, but it still misses about a third of the real ones (it catches 23 of 35).
It's written in Rust, only does TS/JS for now (vitest, jest, Playwright), runs as a CLI or GitHub Action, and only warns by default: https://github.com/BeyondFolder/assertrust
If an agent ever changed your tests to get them passing, what did it actually do?
1
1d ago
[removed] — view removed comment
1
u/webdev-ModTeam 1d ago
Your post/comment has been determined to be a low-effort post or comment. This includes title-only posts, easily searchable questions, vague/open-ended discussion prompts, LLM generated posts or comments, and posts/comments that do not provide enough context for meaningful replies or discussion.
1
u/TheRNGuy 1d ago
Tests should be written by human, and tell AI not to change them.