🕯️LORE

Model Evaluation & Metrics

Properties2
tagssklearn, python, metrics, evaluation, confusion-matrix
aliasesaccuracy_score, confusion_matrix, r2_score

3 min read

Model Evaluation & Metrics

Definition

sklearn.metrics provides scoring functions for classification, regression, and clustering — used to quantify how well a model performs, both during evaluation and as the scoring= argument in cross-validation/tuning.

Classification Metrics

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, classification_report, roc_auc_score
)
 
accuracy_score(y_test, y_pred)          # (correct predictions) / (total predictions)
precision_score(y_test, y_pred)           # of predicted positives, how many were actually positive?
recall_score(y_test, y_pred)                # of actual positives, how many did we catch?
f1_score(y_test, y_pred)                      # harmonic mean of precision and recall
 
classification_report(y_test, y_pred)           # precision/recall/f1/support per class, in one table

Confusion Matrix

cm = confusion_matrix(y_test, y_pred)
#              Predicted 0   Predicted 1
# Actual 0    [    TN     ,      FP    ]
# Actual 1    [    FN     ,      TP    ]
 
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_predictions(y_test, y_pred)
graph TD
    A[Confusion Matrix] --> B["True Positive (TP)
correctly predicted positive"]
    A --> C["True Negative (TN)
correctly predicted negative"]
    A --> D["False Positive (FP)
predicted positive, actually negative
(Type I error)"]
    A --> E["False Negative (FN)
predicted negative, actually positive
(Type II error)"]
MetricFormulaAnswers
Accuracy(TP+TN) / TotalOverall, how often is the model right?
PrecisionTP / (TP+FP)Of positive predictions, how many were correct?
Recall (Sensitivity)TP / (TP+FN)Of actual positives, how many were found?
F1 Score2 * (Precision*Recall)/(Precision+Recall)Balance of precision and recall

Accuracy is misleading on imbalanced datasets

A model predicting “no fraud” for every transaction can score 99% accuracy if fraud is rare — always check precision/recall/F1 (or classification_report) on imbalanced data, not just accuracy.

ROC-AUC & Precision-Recall Curves

from sklearn.metrics import roc_auc_score, roc_curve, RocCurveDisplay
 
roc_auc_score(y_test, y_proba[:, 1])       # requires predicted PROBABILITIES, not labels
RocCurveDisplay.from_predictions(y_test, y_proba[:, 1])
 
from sklearn.metrics import PrecisionRecallDisplay
PrecisionRecallDisplay.from_predictions(y_test, y_proba[:, 1])

ROC-AUC vs Precision-Recall curves

ROC-AUC is a good general summary; Precision-Recall curves are more informative for heavily imbalanced datasets (fraud, disease detection) where the negative class dominates.

Multi-Class Averaging

f1_score(y_test, y_pred, average="macro")        # unweighted mean across classes — treats all classes equally
f1_score(y_test, y_pred, average="weighted")        # weighted by class frequency (support)
f1_score(y_test, y_pred, average="micro")             # aggregate TP/FP/FN globally, then compute

Regression Metrics

from sklearn.metrics import (
    mean_absolute_error, mean_squared_error, root_mean_squared_error,
    r2_score, mean_absolute_percentage_error
)
 
mean_absolute_error(y_test, y_pred)             # average absolute error, same units as target
mean_squared_error(y_test, y_pred)                # penalizes large errors more heavily
root_mean_squared_error(y_test, y_pred)             # RMSE — same units as target, still penalizes large errors
r2_score(y_test, y_pred)                              # proportion of variance explained (1.0 = perfect)
mean_absolute_percentage_error(y_test, y_pred)          # error as a % — scale-independent
MetricSensitive to outliers?Interpretability
MAELessSame units as target, easy to explain
MSE / RMSEMore (squares errors)RMSE is same units, MSE is squared units
R²N/A0-1 scale, ”% variance explained”
MAPELessPercentage — good for comparing across different scales

Custom Scorers

from sklearn.metrics import make_scorer, fbeta_score
 
f2_scorer = make_scorer(fbeta_score, beta=2)   # weight recall higher than precision
cross_val_score(model, X, y, cv=5, scoring=f2_scorer)

Class Imbalance Handling

model = LogisticRegression(class_weight="balanced")   # auto-adjusts weights inversely to class frequency
model = RandomForestClassifier(class_weight={0: 1, 1: 10})  # manual weights

For deeper imbalance handling (SMOTE, undersampling), see the external imbalanced-learn package (pip install imbalanced-learn), which extends the sklearn API with resampling transformers.

Graph View

Backlinks