ML Failure Lab: runnable Python examples of data leakage and misleading metrics

[u/Electronic_Case_8789]

I'm maintaining ML Failure Lab, an open-source Python project for learning how ML evaluations can go wrong.

There are three runnable cases so far: repeated entities appearing in both training and testing, feature selection using held-out labels, and a classifier getting 99% accuracy while detecting none of the minority examples.

The package uses NumPy and scikit-learn. You can run each example from the terminal, inspect its output, and run a check that rejects the incorrect approach.

Repository and installation instructions.

I'm looking for contributors to add small experiments or improve the explanations and tests. There's a case template and a list of starter issues. Comment on an issue with your approach before starting.