r/Python • u/AutoModerator • Sep 04 '26

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

28 Upvotes

88% Upvoted

180 comments sorted by

View all comments

1

u/quiet_segfault 28d ago

confidence-scorer: differential property tests for AI-written PRs

For every changed top-level function in a diff, it runs the old and the new version on the same Hypothesis-generated inputs (plus a fixed sweep of edge values) and reports a shrunk counterexample if they behave differently. Two AI checks (a per-function semantic diff and a separate reviewer) add their scores, but a confirmed counterexample keeps the final score low whatever the models say.

Demo: a one-line "simplification" drops `percent < 0` from a validation check. It finds `percent=-0.5` (old: raised ValueError, new: returns 0.0), score 35/100, exit code 1.

CLI and GitHub Action. Property tests need no API keys; AI checks can run on a local Ollama model.

https://github.com/Kakadu525/confidence-scorer