Classification & Regression Mastery

Loss Functions, Metrics, and Probability Calibration

In medical diagnostics and autonomous driving, getting the right answer isn't enough—you need to know how certain the model is. A self-driving car must distinguish between a '99% confident pedestrian' and a '51% confident shadow.'

In this chapter, we explore probability calibration for medical trust and Huber Loss for robust sensor processing, ensuring your models are reliable in high-stakes environments.

Classification Beyond Benchmarks

Accuracy is just the surface. In clinical settings, we focus on Precision-Recall. A model that predicts 'Cancer' incorrectly is a problem (Precision), but a model that misses Cancer is a catastrophe (Recall). We use Probability Calibration to ensure that if a model says 80% confidence, it actually means 80/100 cases are correct.

PythonRuns entirely in your browser — nothing is sent to a server.

# Medical - Confidence Calibration Simulation

class ProbabilityCalibrator:
    def evaluate(self, confidence_scores, actual_labels):
        print("Analyzing relationship between confidence and accuracy...")
        # In a real tool, this would adjust the threshold for high-stakes recall
        calibrated = [1 if s > 0.25 else 0 for s in confidence_scores]
        return calibrated

# Scenario: Oncology screening (Priority: High Recall)
scores = [0.12, 0.45, 0.82, 0.31]
calibrator = ProbabilityCalibrator()
print("Calibrated Decisions (Threshold 0.25): " + str(calibrator.evaluate(scores, None)))

# Medical - Confidence Calibration Simulation

class ProbabilityCalibrator:
    def evaluate(self, confidence_scores, actual_labels):
        print("Analyzing relationship between confidence and accuracy...")
        # In a real tool, this would adjust the threshold for high-stakes recall
        calibrated = [1 if s > 0.25 else 0 for s in confidence_scores]
        return calibrated

# Scenario: Oncology screening (Priority: High Recall)
scores = [0.12, 0.45, 0.82, 0.31]
calibrator = ProbabilityCalibrator()
print("Calibrated Decisions (Threshold 0.25): " + str(calibrator.evaluate(scores, None)))

Regression for Control Systems

Autonomous vehicles use regression to predict steering angles and braking torque. However, LIDAR and Radar sensors are noisy. Standard Mean Squared Error (MSE) punishes large errors (outliers) so heavily that a single 'ghost' reading can make the car jerk. Huber Loss solves this by ignoring the most extreme outliers.

PythonRuns entirely in your browser — nothing is sent to a server.

# AV - Robust Huber Loss (Anti-Outlier) Simulation

class RobustController:
    def calculate_loss(self, error, mode="MSE"):
        if mode == "Huber":
            # Linear penalty for large errors, quadratic for small
            return abs(error) if abs(error) > 1.0 else 0.5 * (error**2)
        return error**2

# Scenario: A 'Ghost' sensor reading of 100m error vs a real 2m error
controller = RobustController()
print("MSE Loss (Ghost Reading): " + str(controller.calculate_loss(100, "MSE")))
print("Huber Loss (Ghost Reading): " + str(controller.calculate_loss(100, "Huber")))

# AV - Robust Huber Loss (Anti-Outlier) Simulation

class RobustController:
    def calculate_loss(self, error, mode="MSE"):
        if mode == "Huber":
            # Linear penalty for large errors, quadratic for small
            return abs(error) if abs(error) > 1.0 else 0.5 * (error**2)
        return error**2

# Scenario: A 'Ghost' sensor reading of 100m error vs a real 2m error
controller = RobustController()
print("MSE Loss (Ghost Reading): " + str(controller.calculate_loss(100, "MSE")))
print("Huber Loss (Ghost Reading): " + str(controller.calculate_loss(100, "Huber")))

Practice Questions

Question 1

Why is Huber Loss preferred over MSE for noisy sensor data in AVs?

It makes the weights larger
It is less sensitive to extreme outliers than MSE
It only works with images
It never penalizes small errors