Model Evaluation & Optimization

Beyond Accuracy: The Metrics That Matter

A model with "95% accuracy" sounds impressive—until you learn that 95% of transactions are legitimate anyway. For **Visa**, the only metric that matters is: "Did we catch the fraud?" This chapter teaches you to think beyond simple accuracy.

We explore the **Confusion Matrix**, **Precision vs. Recall**, and the **F1 Score**. Then we tackle **Hyperparameter Tuning** with Grid Search to find the optimal settings for **Walmart's** demand forecasting model.

The Confusion Matrix: Truth vs. Prediction

The **Confusion Matrix** breaks down predictions into four categories: True Positives (correctly caught fraud), True Negatives (correctly cleared legit), False Positives (falsely blocked a user), and False Negatives (missed a fraud). For **Visa**, a False Negative is catastrophic—it means a fraudster got through.

PythonRuns entirely in your browser — nothing is sent to a server.

# Visa Fraud Model: Confusion Matrix Breakdown
class ConfusionMatrix:
    def __init__(self, predictions, actuals):
        self.preds = predictions
        self.actuals = actuals

    def calculate(self):
        tp = sum(1 for p, a in zip(self.preds, self.actuals) if p == 1 and a == 1)
        tn = sum(1 for p, a in zip(self.preds, self.actuals) if p == 0 and a == 0)
        fp = sum(1 for p, a in zip(self.preds, self.actuals) if p == 1 and a == 0)
        fn = sum(1 for p, a in zip(self.preds, self.actuals) if p == 0 and a == 1)
        
        print("CONFUSION MATRIX:")
        print("                Predicted FRAUD  Predicted LEGIT")
        print("Actual FRAUD         TP={0}              FN={1}".format(tp, fn))
        print("Actual LEGIT         FP={0}              TN={1}".format(fp, tn))
        print("\nTP={0} (Caught!) | FN={1} (MISSED!) | FP={2} (False Alarm) | TN={3} (Correct Clear)".format(tp, fn, fp, tn))
        return tp, tn, fp, fn

# Scenario: Evaluating Visa's fraud model on 10 transactions
predictions = [1, 0, 1, 1, 0, 0, 1, 0, 0, 1] # Model's predictions
actuals     = [1, 0, 0, 1, 0, 1, 1, 0, 0, 1] # Ground truth
matrix = ConfusionMatrix(predictions, actuals)
matrix.calculate()

# Visa Fraud Model: Confusion Matrix Breakdown
class ConfusionMatrix:
    def __init__(self, predictions, actuals):
        self.preds = predictions
        self.actuals = actuals

    def calculate(self):
        tp = sum(1 for p, a in zip(self.preds, self.actuals) if p == 1 and a == 1)
        tn = sum(1 for p, a in zip(self.preds, self.actuals) if p == 0 and a == 0)
        fp = sum(1 for p, a in zip(self.preds, self.actuals) if p == 1 and a == 0)
        fn = sum(1 for p, a in zip(self.preds, self.actuals) if p == 0 and a == 1)
        
        print("CONFUSION MATRIX:")
        print("                Predicted FRAUD  Predicted LEGIT")
        print("Actual FRAUD         TP={0}              FN={1}".format(tp, fn))
        print("Actual LEGIT         FP={0}              TN={1}".format(fp, tn))
        print("\nTP={0} (Caught!) | FN={1} (MISSED!) | FP={2} (False Alarm) | TN={3} (Correct Clear)".format(tp, fn, fp, tn))
        return tp, tn, fp, fn

# Scenario: Evaluating Visa's fraud model on 10 transactions
predictions = [1, 0, 1, 1, 0, 0, 1, 0, 0, 1] # Model's predictions
actuals     = [1, 0, 0, 1, 0, 1, 1, 0, 0, 1] # Ground truth
matrix = ConfusionMatrix(predictions, actuals)
matrix.calculate()

Precision, Recall & the F1 Score

**Precision** answers: "Of all transactions I flagged as fraud, how many were actually fraud?" **Recall** answers: "Of all actual fraud cases, how many did I catch?" For **Visa**, high Recall is more important—missing a fraud is worse than a false alarm. The **F1 Score** balances both.

PythonRuns entirely in your browser — nothing is sent to a server.

# Visa Fraud Model: Precision, Recall, F1
class MetricsCalculator:
    def __init__(self, tp, fp, fn):
        self.tp = tp
        self.fp = fp
        self.fn = fn

    def compute(self):
        precision = self.tp / (self.tp + self.fp) if (self.tp + self.fp) else 0
        recall = self.tp / (self.tp + self.fn) if (self.tp + self.fn) else 0
        f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) else 0
        
        print("PRECISION: {0:.2f} (Of flagged, how many were actually fraud?)".format(precision))
        print("RECALL   : {0:.2f} (Of actual fraud, how many did we catch?)".format(recall))
        print("F1 SCORE : {0:.2f} (Harmonic mean of Precision and Recall)".format(f1))
        
        if recall < 0.9:
            print("\nWARNING: Recall below 90%. Visa requires catching nearly ALL fraud.")

# Scenario: Model caught 4 fraud, 1 false alarm, missed 1
calc = MetricsCalculator(tp=4, fp=1, fn=1)
calc.compute()

# Visa Fraud Model: Precision, Recall, F1
class MetricsCalculator:
    def __init__(self, tp, fp, fn):
        self.tp = tp
        self.fp = fp
        self.fn = fn

    def compute(self):
        precision = self.tp / (self.tp + self.fp) if (self.tp + self.fp) else 0
        recall = self.tp / (self.tp + self.fn) if (self.tp + self.fn) else 0
        f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) else 0
        
        print("PRECISION: {0:.2f} (Of flagged, how many were actually fraud?)".format(precision))
        print("RECALL   : {0:.2f} (Of actual fraud, how many did we catch?)".format(recall))
        print("F1 SCORE : {0:.2f} (Harmonic mean of Precision and Recall)".format(f1))
        
        if recall < 0.9:
            print("\nWARNING: Recall below 90%. Visa requires catching nearly ALL fraud.")

# Scenario: Model caught 4 fraud, 1 false alarm, missed 1
calc = MetricsCalculator(tp=4, fp=1, fn=1)
calc.compute()

Hyperparameter Tuning: Grid Search

Every ML model has "knobs" (hyperparameters) that control its behavior. For **Walmart's** demand forecaster, this might be the learning rate or tree depth. **Grid Search** systematically tries every combination to find the best one.

PythonRuns entirely in your browser — nothing is sent to a server.

# Walmart Demand Model: Grid Search Tuning
class GridSearchTuner:
    def __init__(self, param_grid):
        self.grid = param_grid

    def search(self):
        print("GRID SEARCH: Testing all hyperparameter combinations...")
        best_score = 0
        best_params = {}
        
        for lr in self.grid['learning_rate']:
            for depth in self.grid['max_depth']:
                # Simulating model score (higher is better)
                score = round(0.7 + lr * 0.2 - (depth - 3) ** 2 * 0.01, 4)
                print(" - lr={0}, depth={1} -> Score: {2}".format(lr, depth, score))
                
                if score > best_score:
                    best_score = score
                    best_params = {"lr": lr, "depth": depth}
        
        print("\nBEST CONFIGURATION: {0}".format(best_params))
        print("BEST SCORE: {0}".format(best_score))

# Scenario: Tuning Walmart's XGBoost demand model
grid = {
    "learning_rate": [0.01, 0.1, 0.5],
    "max_depth": [2, 3, 5]
}
tuner = GridSearchTuner(grid)
tuner.search()

# Walmart Demand Model: Grid Search Tuning
class GridSearchTuner:
    def __init__(self, param_grid):
        self.grid = param_grid

    def search(self):
        print("GRID SEARCH: Testing all hyperparameter combinations...")
        best_score = 0
        best_params = {}
        
        for lr in self.grid['learning_rate']:
            for depth in self.grid['max_depth']:
                # Simulating model score (higher is better)
                score = round(0.7 + lr * 0.2 - (depth - 3) ** 2 * 0.01, 4)
                print(" - lr={0}, depth={1} -> Score: {2}".format(lr, depth, score))
                
                if score > best_score:
                    best_score = score
                    best_params = {"lr": lr, "depth": depth}
        
        print("\nBEST CONFIGURATION: {0}".format(best_params))
        print("BEST SCORE: {0}".format(best_score))

# Scenario: Tuning Walmart's XGBoost demand model
grid = {
    "learning_rate": [0.01, 0.1, 0.5],
    "max_depth": [2, 3, 5]
}
tuner = GridSearchTuner(grid)
tuner.search()

Practice Questions

Question 1

For Visa's fraud detection, why is Recall more important than Precision?

Because Recall sounds more professional
Because missing actual fraud (False Negative) is far more costly than occasionally blocking a legitimate transaction (False Positive)
Because Precision only works with numbers
Because Recall is faster to compute

Question 2

What is the purpose of Grid Search in hyperparameter tuning?

To search for files on the computer
To systematically try all combinations of hyperparameters and find the configuration that produces the best model performance
To delete unused parameters
To reduce the size of the dataset