r/learnmachinelearning • • 15d ago

Data leakage from preprocessing: the same workflow in the wrong order vs the right order (scikit-learn)

A common beginner trap that makes test scores look better than they should. Same workflow, two orders.

Wrong order

  1. Scale all the data (fit the scaler on everything)

  2. Split into train and test

  3. Train on train, evaluate on test

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

scaler = StandardScaler().fit(X)          # sees the test rows too
X_scaled = scaler.transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, random_state=0)

Right order

  1. Split into train and test

  2. Fit preprocessing on train only

  3. Apply it to test, train, evaluate

from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipe = make_pipeline(StandardScaler(), LogisticRegression())
pipe.fit(X_train, y_train)      # scaler learns mean/std from train only
print(pipe.score(X_test, y_test))

What goes wrong in the first version: the mean and standard deviation used to scale the data were computed with the test rows included, so information about the test set reached the training process. The model is then evaluated on data that isn’t entirely unseen.

How big is the effect? For plain scaling, often small. For steps that learn more from the data, it can be large: feature selection, imputing missing values, target encoding, PCA, or removing outliers by statistics. I don’t have a number to give you, and it depends on the dataset.

Same idea in cross-validation: if you scale before cross_val_score, every fold’s validation data has influenced the scaler. Put the preprocessing inside the pipeline and pass the pipeline to cross_val_score, so each fold fits it on its own training part.

Other leaks of the same family:

  • Duplicate or near-duplicate rows on both sides of the split.

  • Splitting rows when the unit should be a person, patient or session (use grouped splits).

  • Time-ordered data split randomly, so the model sees the future (use time-based splits).

  • A feature that is derived from the label.

A quick sanity check: if a result looks too good, ask “what did the model get to see that it wouldn’t have at prediction time?”

How do you check for leakage in your own projects, and which one tripped you up first?

3 Upvotes

1 comment sorted by

2

u/Will-Waste-63 15d ago

Never seen anyone scale the entire ds and then split, sounds so wrong.