Developing ML solutions
Machine Learning Models Performance Evaluation
Datasets for evaluation
- Training set trains the model.
- Validation set checks generalization while you're still improving the model.
- Test set is the final check before production.
Model fit
Compare errors on the training data and the evaluation data:
- Underfitting: poor on training data (high bias).
- Overfitting: good on training data, poor on evaluation data. The model memorized instead of generalizing (high variance).
- Balanced: low bias and low variance.
On the bullseye analogy, bias is how far the shots land from the center and variance is how spread out they are.
Classification metrics
Built from the confusion matrix: true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN).
| Metric | Formula | Use it when |
|---|---|---|
| Accuracy | (TP + TN) / all predictions | Classes are balanced. Misleading when there are many true negatives |
| Precision | TP / (TP + FP) | False positives are costly, for example a spam filter hiding real email |
| Recall (sensitivity) | TP / (TP + FN) | False negatives are costly, for example missing a serious illness |
| F1 score | Harmonic mean of precision and recall | You need a balance of precision and recall |
| AUC-ROC | Area under the true positive rate versus false positive rate curve across thresholds | Comparing models and choosing a threshold |
Regression metrics
- Mean squared error (MSE): the average of squared prediction errors. Lower is better.
- R squared: the share of variance the model explains, from 0 to 1. Closer to 1 is better.
Business metrics
- Tie model metrics to the KPIs set during business goal identification, such as more sales, lower costs, or less churn.
- Check that the metrics reflect the business's tolerance for errors, and consider a cost function for the economic impact of each kind of error.
- Compare model variants in production with A/B testing or canary deployments.