learn deep. ship safe.Practice testsFollow ByteMunSubscribe

Concept

Model evaluation and metrics

Measuring how well a model works: train/test splits, accuracy, precision, recall, F1 and more.

Model evaluation and metrics explained

Model evaluation is measuring how well a machine learning model will work on data it has not seen. You hold back data the model never trains on, make predictions on it, and score them with metrics that match the real goal: accuracy, precision, recall and F1 for classification, and error measures such as MAE or RMSE for regression. A model is only as good as the evaluation that judged it.

Train, validation and test sets

  • Training set: the model learns from it.
  • Validation set: you compare models and tune settings on it.
  • Test set: touched once, at the end, for the number you report.

With little data, k-fold cross-validation rotates which part is held out and averages the scores, so one lucky split does not decide the result.

Classification metrics from the confusion matrix

  • Accuracy = correct predictions ÷ all predictions. Misleading when one class is rare: predicting "not fraud" every time can score 99%.
  • Precision = TP ÷ (TP + FP): of everything flagged positive, how much really was.
  • Recall = TP ÷ (TP + FN): of all real positives, how many were found.
  • F1: the harmonic mean of precision and recall, high only when both are.
  • ROC-AUC: how well the model ranks positives above negatives across every threshold.

Precision and recall trade off through the decision threshold. A spam filter wants high precision (don't bin real mail); a disease screen wants high recall (don't miss cases).

Regression metrics

MAE is the average absolute error, easy to explain in the target's units. RMSE squares errors before averaging, so it punishes large misses more. R² says how much of the variation the model explains compared with always predicting the mean.

Common mistakes

  • Reporting accuracy on imbalanced data instead of precision, recall or PR-AUC.
  • Tuning on the test set, which turns it into a second validation set and inflates the final score.
  • Data leakage: information from the test set or the future reaching training.
  • Random splits on time-ordered data; evaluate on a later period instead.

What interviewers ask about model evaluation

  • "Your model is 98% accurate. Is it good?" Ask about class balance and the cost of each kind of error.
  • "Precision or recall for this use case?" Tie it to which mistake is more expensive.
  • "Why keep a separate test set?" To get one unbiased estimate after all choices are made.
Concept map

What should you learn before Model evaluation and metrics?

  1. Feature engineeringTurning raw data into inputs a model can learn from: scaling, encoding and new features.
  2. Machine learningBuilding systems that learn patterns from data instead of following hand-written rules.
  3. Supervised learningLearning from labelled examples to predict a label or value.

What to learn after Model evaluation and metrics

  • Data leakageLeakage is about the train/test split going wrong.
  • Imbalanced dataAccuracy misleads on imbalanced data; precision and recall do not.
  • LLM evaluationTesting language-model output with test sets, rubrics and model-graded checks.
  • MLOpsPractices for deploying, monitoring and retraining models in production.
  • OverfittingYou spot overfitting by comparing training and test scores.

Read and watch: Model evaluation and metrics explained

Practise Model evaluation and metrics

All topics