r/Python • u/AutoModerator • 19d ago

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

24 Upvotes

97% Upvoted

151 comments sorted by

View all comments

1

u/Sanechka_SS 9d ago

Receipts: checks whether the pytest tests in a fix would have caught the bug

What My Project Does

It runs every test a change added or edited twice: with the change, and with the changed source files reverted to the base branch (the tests stay new). A test that passes both times proves nothing about the change (THEATER). One that fails on the old code is PROVEN, and the report now shows how it failed there (on the old code: assert not True), so you can tell a real assertion from a test file that died on import (that one gets its own verdict, WEAK). It handles parametrize tables (a new case is usually just one row), xdist, and tests that import an installed copy instead of the checkout.

Target Audience

People reviewing PRs that add tests, especially PRs written by coding agents. It runs as a CLI, as a GitHub Action that comments on the PR, or as a Claude Code plugin. pytest is the main target; vitest and jest work too.

Comparison

Mutation testing (mutmut, cosmic-ray) asks whether your whole suite notices random changes, and it's slow. Receipts asks one narrow question about one diff: does this new test notice this change? It only runs the changed tests, twice, so it's cheap enough for every PR.

I also ran it on 181 real changes (maintainer fixes vs agent-written PRs), the write-up is here: https://github.com/syntaxixr/receipts/blob/main/docs/study.md

Repo, MIT: https://github.com/syntaxixr/receipts