The Path to the Champion Model: Machine Learning Workflow Lab

To build robust ML models without optimistic bias, raw data is split into three strictly isolated subsets before any feature engineering or model training begins.

Dataset Partition Distribution 70% Train | 15% Val | 15% Test

TRAIN (70%)

VAL (15%)

TEST (15%)

Training Split %: 70%

Validation Split % (Remainder is Test): 15%

Training Set

Used by algorithms to learn underlying features, weights, and parameters.

Validation Set

Used during HPO to tune hyperparameters and pick candidate architectures without touching Test.

Test Set

Locked in a "vault". Touched ONLY ONCE at the end to evaluate real-world readiness.

Each model candidate undergoes individual Hyperparameter Optimization (HPO) using Grid Search or Bayesian Optimization to maximize its raw learning potential.

CNN AUC: --

Learning Rate:

Filter Depth:

Conv2D + MaxPool Architecture

Transformer AUC: --

Attention Heads:

Dropout Rate:

Self-Attention Mechanism

SVM AUC: --

C (Penalty):

Gamma (RBF):

Kernel Support Vector Machine

Model performance is evaluated on the Validation Set using ROC-AUC. Because ROC-AUC measures discrimination power across all thresholds simultaneously, it provides an unbiased benchmark to select the Champion Model.

Crowned Champion

Transformer

ROC-AUC: 0.942

Selected strictly based on highest raw discrimination ability before applying any specific business decision threshold.

Now that the Champion Model is fixed, we optimize the Decision Threshold t (from 0 to 1) on the Validation Set based on business problem costs (e.g., False Negatives in Medical Diagnosis are far more costly than False Positives).

Decision Threshold Control (t)

Default probability cutoff = 0.50

0.50

0.05 (High Recall / Catch All) 0.50 (Balanced) 0.95 (High Precision / Strict)

Precision

88.5%

TP / (TP + FP)

Recall (Sensitivity)

91.2%

TP / (TP + FN)

F1-Score

0.898

Harmonic Mean

Estimated Misclassification Cost

$1,450

Weighted FN + FP Cost

Test Set Unlocked

Final evaluation on unseen holdout test dataset using the tuned Champion Model & threshold.

Final Confusion Matrix

True Positive (TP) 142

False Positive (FP) 18

False Negative (FN) 12

True Negative (TN) 328

Unbiased real-world test assessment.

Deployment Readiness Report

Test Accuracy: 94.0%

Test F1-Score: 0.904

Generalization Gap: 1.2% (Minimal Overfit)

Status: APPROVED FOR PROD

Note: Because Test set was never used during model selection or threshold tuning, this performance estimate is clean of data leakage.

The Path to the Champion Model ML Pipeline Lab

3-Way Data Split

HPO Tuning

ROC-AUC Champion

Threshold Opt.

Final Test Determination

The 3-Way Data Split

Training Set

Validation Set

Test Set

Transformer

Test Set Unlocked

Stage Deep Dive Key Concept

Why 3-Way Splitting Matters

Dataset Environment

Stage Concept Breakdown