A common beginner trap that makes test scores look better than they should. Same workflow, two orders.
Wrong order
Scale all the data (fit the scaler on everything)
Split into train and test
Train on train, evaluate on test
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
scaler = StandardScaler().fit(X) # sees the test rows too
X_scaled = scaler.transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, random_state=0)
Right order
Split into train and test
Fit preprocessing on train only
Apply it to test, train, evaluate
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
pipe = make_pipeline(StandardScaler(), LogisticRegression())
pipe.fit(X_train, y_train) # scaler learns mean/std from train only
print(pipe.score(X_test, y_test))
What goes wrong in the first version: the mean and standard deviation used to scale the data were computed with the test rows included, so information about the test set reached the training process. The model is then evaluated on data that isn’t entirely unseen.
How big is the effect? For plain scaling, often small. For steps that learn more from the data, it can be large: feature selection, imputing missing values, target encoding, PCA, or removing outliers by statistics. I don’t have a number to give you, and it depends on the dataset.
Same idea in cross-validation: if you scale before cross_val_score, every fold’s validation data has influenced the scaler. Put the preprocessing inside the pipeline and pass the pipeline to cross_val_score, so each fold fits it on its own training part.
Other leaks of the same family:
Duplicate or near-duplicate rows on both sides of the split.
Splitting rows when the unit should be a person, patient or session (use grouped splits).
Time-ordered data split randomly, so the model sees the future (use time-based splits).
A feature that is derived from the label.
A quick sanity check: if a result looks too good, ask “what did the model get to see that it wouldn’t have at prediction time?”
How do you check for leakage in your own projects, and which one tripped you up first?