← Apps ← My Digital Space
Model Security · Lesson 03

Data Leakage

Your model appears to work with 99% accuracy, but in reality it is simply peeking at the future. The most subtle and dangerous mistake in machine learning: Data Leakage.

3
Leakage Types
8
Feature Tests
6
Real Scenarios
MODEL TRAIN DATA TEST DATA ⚠ LEAKAGE DETECTED Train: 99% Real: 42%
target leakage · train-test contamination · future leakage · preprocessing leakage · data snooping · duplicated rows · target leakage · train-test contamination · future leakage · preprocessing leakage · data snooping · duplicated rows
01 · Concept

What exactly is
Data Leakage?

It happens when information that would not be available at prediction time leaks into the model during training. The model ends up predicting the future by already knowing it — like a student who peeked at the exam questions beforehand.

TYPE 01

Target Leakage

One of the features is directly tied to the target variable and is only measured after the outcome occurs. The model already knows the answer.

"Medication dosage" feature → determined after the diagnosis is made
TYPE 02

Train-Test Contamination

Test data bleeds into the training process. Normalization, feature selection, or cleaning is performed on the entire dataset before splitting.

Use scaler.fit(X_train) instead of scaler.fit(X_all)
TYPE 03

Future Leakage

In time-series problems, data from the future is fed into the model at training time. Train/test split must follow chronological order, not random shuffling.

Like using 2024 data to predict 2023 outcomes
ANALOGY

Peeking at the exam questions

Imagine a teacher who hands out the exam questions in advance and then administers the very same exam. Students score 100%. But in the real world, they fail miserably. That is exactly what Data Leakage is — the model is trained on information it will never have access to in production.

02 · Interactive Lab

Leakage
Hunt

You are building a bank loan default prediction model. Select any of the 8 features below. Accuracy metrics and real-world performance will update in real time. Which features cause leakage?

DATASET

Bank Loan Default Dataset

How it works: When you select a feature, the model is trained instantly. Leaky features will make metrics look suspiciously high, but the "Production" accuracy will collapse. Try to identify which features leak.

Model Performance

IDLE
Training Accuracy --
Test Accuracy --
Production (Real World) --
DIAGNOSIS
Select features on the left to begin...
Selected: 0 / 8 Leaky: 0
03 · Scenario Detective

Real-World
Cases

The 6 cases below are inspired by real industry mistakes. Read each scenario, form your own answer, then flip the card to reveal the truth.

Click on any card to reveal the answer.
04 · Prevention

How to Prevent
Leakage?

STRATEGY 01

Use Pipelines

Wrap every preprocessing step inside a Pipeline. fit() only sees train data, transform() applies to both train and test.

from sklearn.pipeline import Pipeline pipe = Pipeline([ ('scaler', StandardScaler()), ('model', RandomForest()) ]) pipe.fit(X_train, y_train)
STRATEGY 02

Split First, Process Later

Split the data, then explore. Perform EDA only on the train set. Don't even look at the test set.

# 1. Split first X_tr, X_te, y_tr, y_te = \ train_test_split(X, y) # 2. Fit only on train scaler.fit(X_tr) X_tr_s = scaler.transform(X_tr) X_te_s = scaler.transform(X_te)
STRATEGY 03

Split by Time

Never randomly split time-series data. Train on the past, test on the future. Use TimeSeriesSplit.

from sklearn.model_selection \ import TimeSeriesSplit tscv = TimeSeriesSplit(n_splits=5) for tr_idx, te_idx in tscv.split(X): # Past → Future ...
STRATEGY 04

Feature Audit

For every feature, ask: "Will I have this information at prediction time?" If the answer is "no", that feature leaks. Drop it.

for col in df.columns: available_at_pred = ask( "Will this feature be" " available at inference?" ) if not available_at_pred: df.drop(col, axis=1)
Wrong Approach
# Scale the entire dataset (LEAKAGE!) scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # Then split — it's already too late! X_train, X_test = train_test_split(X_scaled) # The mean/std of the test set # leaked into the scaler. The model # already "knows" the test data → # misleadingly high scores.
Correct Approach
# Split first X_train, X_test, y_train, y_test = \ train_test_split(X, y, test_size=0.2) # Fit only on train scaler = StandardScaler() X_train_s = scaler.fit_transform(X_train) X_test_s = scaler.transform(X_test) # transform only! # Test data stays fully isolated.
Key Takeaway

If your model's results look "too good" in testing
but "terrible" in production...

Chances are you have data leakage.

Always ask yourself: "Will this feature actually be available at the moment of prediction?"
If the answer is "no", that feature is fooling your model.

"All models are wrong, but some of them leak."