learn deep. ship safe.Practice testsFollow ByteMunSubscribe

Concept

Data leakage

Information from the test set or the future reaching training, which makes results look better than they are.

Data leakage explained

Data leakage is when information that will not be available at prediction time gets into training, so the model looks far better in evaluation than it will be in real use. The usual causes are test data influencing training (for example scaling before splitting) and features that secretly contain the answer. Leakage is one of the most common reasons a model that scored well collapses in production.

The two kinds of leakage

  • Train–test contamination: a preprocessing step learned from the full dataset, such as a scaler's mean, an imputer's fill value or selected features, before the split. The test set has then shaped the model.
  • Target leakage: a feature that is only known after the outcome, such as "account_closed_date" when predicting churn, or "treatment_given" when predicting a diagnosis.

How to prevent data leakage

  1. Split first, then fit every preprocessing step on the training data only and apply it to validation and test.
  2. Use a pipeline so cross-validation refits the scaler and encoder inside each fold.
  3. Split by time for anything time-ordered: train on the past, evaluate on the future.
  4. Group related rows: put all rows from the same user, patient or document on one side of the split.
  5. Ask of every feature: would I really have this value at the moment I make the prediction?
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)      # the scaler only ever sees training data
print(model.score(X_test, y_test))

Warning signs

  • A validation score that is suspiciously high, or far above similar published work.
  • One feature with overwhelming importance that would not exist at prediction time.
  • A big drop between offline evaluation and the first week in production.
  • Duplicate or near-duplicate rows shared by train and test.

What interviewers ask about data leakage

  • "Where is the leak in this pipeline?" Look for any fit on data that includes the test set.
  • "How would you split time-series data?" Chronologically, with the test period after the training period.
  • "Your model was great offline and poor in production. Why?" Leakage and data drift are the first two suspects.
Concept map
Data leakage

What should you learn before Data leakage?

  1. Feature engineeringTurning raw data into inputs a model can learn from: scaling, encoding and new features.
  2. Machine learningBuilding systems that learn patterns from data instead of following hand-written rules.
  3. Supervised learningLearning from labelled examples to predict a label or value.
  4. Model evaluation and metricsLeakage is about the train/test split going wrong.

Read and watch: Data leakage explained

Practise Data leakage

All topics