03. Mastering Logistic Regression & Evaluation Metrics

Sigmoid, Cross-Entropy, Confusion Matrix, ROC/PR Curve

Learning Objectives

After completing this tutorial, you will be able to:

1. Understand Sigmoid function and Log-Loss (Binary Cross-Entropy) formulas
2. Understand logistic regression principles and Gradient Descent learning process
3. Fully understand Confusion Matrix components: TP, FP, TN, FN
4. Calculate and analyze Precision, Recall, F1-Score trade-offs
5. Interpret ROC Curve, AUC, PR Curve
6. Meet business requirements through Decision Threshold optimization
7. Understand Imbalanced Data problems and class_weight solutions

Key Concepts

1. What is Logistic Regression?

An algorithm that solves Binary Classification problems.

Passes linear combination through Sigmoid function to convert to probability:

P(y=1|x) = σ(z) = 1 / (1 + e^(-z))

Where z = wᵀx + b (linear combination)

Sigmoid Function Properties

2. Log-Loss (Binary Cross-Entropy)

L(w) = -(1/n) Σ[yᵢlog(p̂ᵢ) + (1-yᵢ)log(1-p̂ᵢ)]

Intuitive Understanding of Loss Function

3. Understanding Confusion Matrix

                  Predicted
                 Neg    Pos
Actual   Neg     TN     FP
         Pos     FN     TP

4. Evaluation Metrics Summary

5. ROC Curve & AUC

• ROC Curve: Plots TPR (True Positive Rate) vs FPR (False Positive Rate) at all thresholds
• AUC: Area under the ROC curve

from sklearn.metrics import roc_curve, auc, roc_auc_score
 
fpr, tpr, thresholds = roc_curve(y_test, y_proba)
roc_auc = auc(fpr, tpr)

6. Precision-Recall Curve

• More useful than ROC for imbalanced data
• AP (Average Precision): Area under the PR curve

from sklearn.metrics import precision_recall_curve, average_precision_score
 
precision, recall, thresholds = precision_recall_curve(y_test, y_proba)
ap = average_precision_score(y_test, y_proba)

7. Precision-Recall Trade-off

Adjusting threshold changes Precision and Recall inversely:

Threshold Selection by Business Scenario

# Find threshold that maximizes F1
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-10)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]

Code Summary

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, classification_report,
    roc_auc_score, average_precision_score
)
 
# Training
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
 
# Prediction
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]
 
# Evaluation
print(f"Accuracy:  {accuracy_score(y_test, y_pred):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall:    {recall_score(y_test, y_pred):.4f}")
print(f"F1:        {f1_score(y_test, y_pred):.4f}")
print(f"ROC-AUC:   {roc_auc_score(y_test, y_proba):.4f}")
 
# Detailed report
print(classification_report(y_test, y_pred))

Handling Imbalanced Data

# Using class_weight (higher weight for minority class)
model = LogisticRegression(class_weight='balanced')
 
# Or specify directly
model = LogisticRegression(class_weight={0: 1, 1: 10})

Evaluation Metric Selection Guide

Interview Questions Preview

1. Explain the trade-off between Precision and Recall
2. When do you use ROC-AUC vs PR-AUC?
3. How does class_weight='balanced' work?
4. Why is Log-Loss more suitable than MSE for classification?

Check out more interview questions at Premium Interviews (opens in a new tab).

Practice Notebook

Practice the above concepts with the Breast Cancer dataset:

The notebook additionally covers:

• Visualization of Sigmoid function and Log-Loss
• Logistic regression from Scratch implementation (Gradient Descent)
• Performance comparison experiments at various thresholds
• Decision Boundary visualization using PCA
• Imbalanced data generation and class_weight effect comparison
• Practice problems (Cost-Sensitive Learning, algorithm comparison, etc.)

Previous: 02. Linear Regression | Next: 04. Decision Tree