All modulesS11 · Evaluation and validation▾
This is where the contest is won or lost. A model that looks good on the public leaderboard can be weak on the hidden one, which is what counts at the end. This module is about having justified trust in your score. It's the least flashy part and the one that makes the difference between a top 10 and a mid-table finish.
Why accuracy lies to you
Accuracy is the share of correct predictions. It sounds reasonable, but on imbalanced classes it's misleading. If 98% of the examples are the no class, a model that always says no gets 98% accuracy and zero value. That's why you need metrics that look at each class separately.
The confusion matrix: the foundation
The confusion matrix counts the four kinds of outcome for a binary classification. It's the basis every other metric is computed from, so it's worth understanding well.
- TP (true positive): it was positive, you said positive. Correct.
- TN (true negative): it was negative, you said negative. Correct.
- FP (false positive): it was negative, you said positive. A false alarm.
- FN (false negative): it was positive, you said negative. You missed it.
Which one hurts more depends on the problem. For a disease test, a false negative (a sick person sent home) is worse than a false positive. For a spam filter, it's the other way around. The right metric reflects the real cost of each kind of mistake.
Precision, recall, F1
Precision and recall pull in opposite directions. If you say positive only when you're very sure, precision goes up but recall goes down. If you say positive often, it's the reverse. F1 reconciles them into a single number, their harmonic mean, which is small if either of them is small.
Validation without leaks
Data leakage is when, without meaning to, information from the test set makes it into training. The result: a great local score that collapses on the real leaderboard. It's the sneakiest way to lose points, because everything looks fine.
You defend against it by splitting the data correctly. Keep a validation set the model doesn't see during training and use it to estimate the real score. Better, use k-fold: you split the data into k parts, train on k-1 and test on one, rotating, then average. That way you use all the data and get a more stable estimate.
When the classes are imbalanced, use stratified k-fold, which keeps the class proportions in each fold. Otherwise a fold might not contain the rare class at all. When the data has time or groups (the same patient in several rows), the split has to respect them, otherwise the model peeks.
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=cv, scoring="f1_macro")
print(scores.mean(), scores.std())Bias, variance and the learning curve
Two opposite diseases. High bias (underfitting): the model is too simple, it does badly on both training and validation. High variance (overfitting): the model is too complex, it does great on training but badly on validation, because it memorized.
The learning curve diagnoses them: you plot the training score and the validation score as you add more data. If both are low and close together, you have bias, you need a stronger model. If there's a big gap between them, you have variance, you need more data or regularization.
The two final submissions
- Accuracy lies on imbalanced classes; use precision, recall, F1.
- The confusion matrix (TP, TN, FP, FN) is the basis of all metrics.
- Precision and recall fight each other; F1 reconciles them.
- Stratified k-fold estimates the score stably, with no leaks.
- The learning curve tells bias (underfitting) from variance (overfitting).
- Choose your two final submissions: one on local, one on public.