Regression & Classification Algorithms

Decision Trees, Forests, and the Power of Ensembles

At the heart of Machine Learning lie two families of problems: **Regression** (predicting a number, like Walmart's next-week demand) and **Classification** (predicting a category, like Visa's "Fraud" vs. "Legitimate"). In this chapter, we explore the most powerful algorithms for these tasks.

We start with **Linear Regression** for demand forecasting, then build up to **Decision Trees** and their ensemble counterparts: **Random Forests** and **XGBoost**, which power Visa's real-time fraud detection pipeline.

Linear Regression: Predicting Demand

**Linear Regression** finds the "best-fit line" through your data. For **Walmart**, this means finding the relationship between historical sales volume and upcoming demand. If January sales have grown by ~500 units/week for the last 10 weeks, the model predicts next week's demand by extending that line.

PythonRuns entirely in your browser — nothing is sent to a server.

# Walmart Demand Forecasting: Linear Regression
class DemandForecaster:
    def __init__(self, weekly_sales):
        self.sales = weekly_sales # List of (week_number, units_sold)

    def fit_and_predict(self, target_week):
        # Simple linear regression: y = mx + b
        n = len(self.sales)
        sum_x = sum(w for w, _ in self.sales)
        sum_y = sum(u for _, u in self.sales)
        sum_xy = sum(w * u for w, u in self.sales)
        sum_x2 = sum(w * w for w, _ in self.sales)
        
        m = (n * sum_xy - sum_x * sum_y) / (n * sum_x2 - sum_x ** 2)
        b = (sum_y - m * sum_x) / n
        
        prediction = m * target_week + b
        print("MODEL: y = {0:.1f}x + {1:.1f}".format(m, b))
        print("Week {0} Predicted Demand: {1:.0f} units".format(target_week, prediction))
        return prediction

# Scenario: Forecasting Week 6 demand
history = [(1, 1000), (2, 1450), (3, 2100), (4, 2500), (5, 3050)]
forecaster = DemandForecaster(history)
forecaster.fit_and_predict(target_week=6)

# Walmart Demand Forecasting: Linear Regression
class DemandForecaster:
    def __init__(self, weekly_sales):
        self.sales = weekly_sales # List of (week_number, units_sold)

    def fit_and_predict(self, target_week):
        # Simple linear regression: y = mx + b
        n = len(self.sales)
        sum_x = sum(w for w, _ in self.sales)
        sum_y = sum(u for _, u in self.sales)
        sum_xy = sum(w * u for w, u in self.sales)
        sum_x2 = sum(w * w for w, _ in self.sales)
        
        m = (n * sum_xy - sum_x * sum_y) / (n * sum_x2 - sum_x ** 2)
        b = (sum_y - m * sum_x) / n
        
        prediction = m * target_week + b
        print("MODEL: y = {0:.1f}x + {1:.1f}".format(m, b))
        print("Week {0} Predicted Demand: {1:.0f} units".format(target_week, prediction))
        return prediction

# Scenario: Forecasting Week 6 demand
history = [(1, 1000), (2, 1450), (3, 2100), (4, 2500), (5, 3050)]
forecaster = DemandForecaster(history)
forecaster.fit_and_predict(target_week=6)

Decision Trees & Random Forests

A **Decision Tree** makes predictions by asking a series of yes/no questions: "Is the transaction amount > $500? Is the location different from the last transaction?" For **Visa**, a single tree is powerful but fragile. A **Random Forest** builds hundreds of independent trees and lets them vote, drastically reducing errors.

PythonRuns entirely in your browser — nothing is sent to a server.

# Visa Fraud Detection: Random Forest Voting
class DecisionTree:
    def __init__(self, tree_id, bias):
        self.tree_id = tree_id
        self.bias = bias # Simulated threshold

    def predict(self, amount):
        return "FRAUD" if amount > self.bias else "LEGIT"

class RandomForest:
    def __init__(self, n_trees):
        import random
        # Each tree has a slightly different decision boundary
        self.trees = [DecisionTree(i, 400 + i * 50) for i in range(n_trees)]

    def ensemble_predict(self, amount):
        votes = [t.predict(amount) for t in self.trees]
        fraud_count = votes.count("FRAUD")
        legit_count = votes.count("LEGIT")
        
        print("RANDOM FOREST ({0} trees):".format(len(self.trees)))
        print(" - Transaction Amount: 0".format(amount))
        print(" - Votes: FRAUD={0}, LEGIT={1}".format(fraud_count, legit_count))
        result = "FRAUD" if fraud_count > legit_count else "LEGIT"
        print(" - FINAL VERDICT: {0}".format(result))

# Scenario: $550 purchase from unusual location
forest = RandomForest(n_trees=5)
forest.ensemble_predict(amount=550)

# Visa Fraud Detection: Random Forest Voting
class DecisionTree:
    def __init__(self, tree_id, bias):
        self.tree_id = tree_id
        self.bias = bias # Simulated threshold

    def predict(self, amount):
        return "FRAUD" if amount > self.bias else "LEGIT"

class RandomForest:
    def __init__(self, n_trees):
        import random
        # Each tree has a slightly different decision boundary
        self.trees = [DecisionTree(i, 400 + i * 50) for i in range(n_trees)]

    def ensemble_predict(self, amount):
        votes = [t.predict(amount) for t in self.trees]
        fraud_count = votes.count("FRAUD")
        legit_count = votes.count("LEGIT")
        
        print("RANDOM FOREST ({0} trees):".format(len(self.trees)))
        print(" - Transaction Amount: 0".format(amount))
        print(" - Votes: FRAUD={0}, LEGIT={1}".format(fraud_count, legit_count))
        result = "FRAUD" if fraud_count > legit_count else "LEGIT"
        print(" - FINAL VERDICT: {0}".format(result))

# Scenario: $550 purchase from unusual location
forest = RandomForest(n_trees=5)
forest.ensemble_predict(amount=550)

XGBoost: The Grandmaster of Tabular Data

**XGBoost** (Extreme Gradient Boosting) takes ensembles further. Instead of building independent trees, each new tree focuses on the *mistakes* of the previous one. This iterative correction makes XGBoost the dominant algorithm for structured/tabular data—the backbone of **Visa's** production fraud pipeline.

PythonRuns entirely in your browser — nothing is sent to a server.

# Visa Fraud Detection: XGBoost Boosting Simulation
class XGBoostSimulator:
    def __init__(self, n_rounds):
        self.n_rounds = n_rounds

    def train_and_predict(self, features):
        print("XGBOOST TRAINING ({0} boosting rounds):".format(self.n_rounds))
        error = 0.45 # Initial error rate
        
        for i in range(self.n_rounds):
            # Each round reduces the error by focusing on mistakes
            correction = error * 0.4
            error -= correction
            print(" - Round {0}: Error={1:.4f} (Corrected {2:.4f})".format(i+1, error, correction))
        
        print("\nFINAL MODEL ACCURACY: {0:.2f}%".format((1 - error) * 100))
        print("Input Features: {0}".format(features))
        result = "FRAUD" if features[0] > 500 else "LEGIT"
        print("PREDICTION: {0}".format(result))

# Scenario: High-value suspicious transaction
xgb = XGBoostSimulator(n_rounds=5)
xgb.train_and_predict(features=[750, 1, 0])  # [$750, foreign_location=1, night_tx=0]

# Visa Fraud Detection: XGBoost Boosting Simulation
class XGBoostSimulator:
    def __init__(self, n_rounds):
        self.n_rounds = n_rounds

    def train_and_predict(self, features):
        print("XGBOOST TRAINING ({0} boosting rounds):".format(self.n_rounds))
        error = 0.45 # Initial error rate
        
        for i in range(self.n_rounds):
            # Each round reduces the error by focusing on mistakes
            correction = error * 0.4
            error -= correction
            print(" - Round {0}: Error={1:.4f} (Corrected {2:.4f})".format(i+1, error, correction))
        
        print("\nFINAL MODEL ACCURACY: {0:.2f}%".format((1 - error) * 100))
        print("Input Features: {0}".format(features))
        result = "FRAUD" if features[0] > 500 else "LEGIT"
        print("PREDICTION: {0}".format(result))

# Scenario: High-value suspicious transaction
xgb = XGBoostSimulator(n_rounds=5)
xgb.train_and_predict(features=[750, 1, 0])  # [$750, foreign_location=1, night_tx=0]

The 7 Most-Used ML Algorithms

Not every algorithm fits every problem. This reference covers the **7 most-used algorithms in industry**, their types (Regression vs. Classification vs. Clustering), and which enterprise scenarios they dominate. From **Linear Regression** at Walmart to **XGBoost** at Visa, this is the toolkit every data scientist needs.

PythonRuns entirely in your browser — nothing is sent to a server.

# The 7 Most-Used Machine Learning Algorithms
class AlgorithmReference:
    def demonstrate(self):
        algorithms = [
            ("1. Linear Regression",
             "Regression (predicts a number)",
             "Finds the best-fit line through data",
             "Walmart: Forecast next week's demand based on historical sales trends"),
            ("2. Logistic Regression",
             "Classification (predicts a category)",
             "Estimates probability of a binary outcome (0 or 1)",
             "Visa: What is the probability this transaction is fraudulent?"),
            ("3. Decision Tree",
             "Both (Classification & Regression)",
             "Splits data using yes/no questions at each node",
             "Netflix: If user age>30 AND genre=drama THEN recommend 'The Crown'"),
            ("4. Random Forest",
             "Both (Ensemble of Decision Trees)",
             "Builds many independent trees and aggregates their votes",
             "Visa: 500 trees vote on whether a transaction is fraud or legit"),
            ("5. XGBoost (Gradient Boosting)",
             "Both (Sequential Ensemble)",
             "Each new tree corrects the mistakes of the previous ones",
             "Visa: Production fraud scoring (dominates Kaggle competitions)"),
            ("6. K-Means Clustering",
             "Unsupervised (finds groups)",
             "Groups similar data points into K clusters by distance",
             "Netflix: Discover taste communities without user labels"),
            ("7. K-Nearest Neighbors (KNN)",
             "Both (Instance-based)",
             "Classifies a point based on its K closest neighbors",
             "Walmart: Predict spoilage risk by finding similar past shipments")
        ]
        
        print("THE 7 MOST-USED ML ALGORITHMS:")
        print("=" * 60)
        for name, algo_type, how, scenario in algorithms:
            print("{0}".format(name))
            print("  Type : {0}".format(algo_type))
            print("  How  : {0}".format(how))
            print("  Use  : {0}".format(scenario))
            print("")

ref = AlgorithmReference()
ref.demonstrate()

# The 7 Most-Used Machine Learning Algorithms
class AlgorithmReference:
    def demonstrate(self):
        algorithms = [
            ("1. Linear Regression",
             "Regression (predicts a number)",
             "Finds the best-fit line through data",
             "Walmart: Forecast next week's demand based on historical sales trends"),
            ("2. Logistic Regression",
             "Classification (predicts a category)",
             "Estimates probability of a binary outcome (0 or 1)",
             "Visa: What is the probability this transaction is fraudulent?"),
            ("3. Decision Tree",
             "Both (Classification & Regression)",
             "Splits data using yes/no questions at each node",
             "Netflix: If user age>30 AND genre=drama THEN recommend 'The Crown'"),
            ("4. Random Forest",
             "Both (Ensemble of Decision Trees)",
             "Builds many independent trees and aggregates their votes",
             "Visa: 500 trees vote on whether a transaction is fraud or legit"),
            ("5. XGBoost (Gradient Boosting)",
             "Both (Sequential Ensemble)",
             "Each new tree corrects the mistakes of the previous ones",
             "Visa: Production fraud scoring (dominates Kaggle competitions)"),
            ("6. K-Means Clustering",
             "Unsupervised (finds groups)",
             "Groups similar data points into K clusters by distance",
             "Netflix: Discover taste communities without user labels"),
            ("7. K-Nearest Neighbors (KNN)",
             "Both (Instance-based)",
             "Classifies a point based on its K closest neighbors",
             "Walmart: Predict spoilage risk by finding similar past shipments")
        ]
        
        print("THE 7 MOST-USED ML ALGORITHMS:")
        print("=" * 60)
        for name, algo_type, how, scenario in algorithms:
            print("{0}".format(name))
            print("  Type : {0}".format(algo_type))
            print("  How  : {0}".format(how))
            print("  Use  : {0}".format(scenario))
            print("")

ref = AlgorithmReference()
ref.demonstrate()

Practice Questions

Question 1

Why does a Random Forest outperform a single Decision Tree?

It uses more colors in its output
It aggregates votes from many independent trees, reducing the variance and overfitting of any single tree
It requires less data to train
It only works with Visa data

Question 2

How does XGBoost differ from a Random Forest?

XGBoost uses colors; Random Forest uses shapes
In XGBoost, each new tree specifically corrects the errors of the previous trees (sequential boosting), whereas Random Forest trees are independent
Random Forest is always more accurate
XGBoost can only be used for regression