SVM, Decision Trees & Random Forest

A Support Vector Machine (SVM) is a powerful classifier that doesn't just find a line—it finds the widest possible road (the margin) between two classes. In HR, we use this to find a "Hard Margin" for candidate selection where there is no ambiguity between a 'Hirable' and 'Non-Hirable' profile.

The "Support Vectors" are the individual employees who sit exactly on the edge of the road. These are your most critical data points because if they moved, the decision boundary would move too.

# SVM: Linear Decision Roaddef svm_decision(features, weights, bias):    # Sum of (Feature * Weight) + Bias    score = sum(f * w for f, w in zip(features, weights)) + bias    return 1 if score > 0 else 0 # Weights: [Experience=2.0, Skill=1.5] | Bias: -10weights, bias = [2.0, 1.5], -10print(f"Candidate (5yr Exp, 8 Skill): {svm_decision([5, 8], weights, bias)}") # 1

Sometimes, talent data isn't separable by a straight line. Imagine "High Potential" employees are clustered in two different spots (e.g., highly technical seniors AND highly empathetic juniors). A straight line would miss half of them.

The Kernel Trick: SVMs use "Kernels" (like RBF or Polynomial) to mathematically project your 2D data into a 3D space. Suddenly, a flat plane can slice through the cloud of data, separating the groups perfectly. It's like adding a new dimension of insight (e.g., combining "Introversion" with "Leadership Style").

# SVM Kernels: Projecting into Higher Dimensionsdef rbf_kernel_sim(x, landmark):    import math    # Measures 'Closeness' to a specific profile type    distance = sum((a - b)**2 for a, b in zip(x, landmark))    return math.exp(-0.1 * distance) senior_profile = [10, 5] # 10yr Exp, Level 5junior_prodigy = [1, 2]  # 1yr Exp, Level 2candidate       = [2, 1]  print(f"Closeness to Junior Prodigy: {rbf_kernel_sim(candidate, junior_prodigy):.2f}")

While we usually use SVM for categories, it's also excellent for Regression (predicting numbers). Unlike Linear Regression which tries to hit every point, SVM Regression (SVR) tries to fit as many points as possible inside the road, while limiting "Marginal Violations".

In HR, we use this for Fair Pay Modeling. We want to ensure 95% of our employees' salaries fall within a specific "Predictable Band" based on their skills, flagging only the outliers who are paid too much or too little.

# SVR: Predicting Salary Banddef predict_salary_band(exp, base=40000, margin=5000):    midpoint = base + (exp * 4000)    return (midpoint - margin, midpoint + margin) band = predict_salary_band(5)print(f"Expected 5yr Salary Band: ${band[0]:,} to ${band[1]:,}")

How does the machine actually calculate the widest road? It solves a Quadratic Programming optimization problem. It's looking for the minimum "Weights" that still satisfy the constraint that every "Yes" point is on one side of the margin and every "No" point is on the other.

The Decision Function is simply: ŷ = sign(wᵀx + b). If the result is positive, the candidate is a fit; if negative, they aren't. Training the model is just the process of finding the optimal 'w' and 'b'.

# The Signum Function (The Final Decision)def sign(x):    return 1 if x >= 0 else -1 # w*x + b = 2.5print(f"Decision for score 2.5: {sign(2.5)} (HIRE)")# w*x + b = -1.2print(f"Decision for score -1.2: {sign(-1.2)} (REJECT)")

Decision Trees are the most "Human" models. They work by asking a series of binary questions: "Is their tenure > 2 years?" -> "Is their Python score > 8?" -> "Hired!".

The tree decides which question to ask first by calculating Gini Impurity. It wants to find the question that best "purifies" the groups. If asking about "Department" splits your data into perfectly separate groups of leavers and stayers, that question becomes the Root Node of your tree.

# Decision Tree: Simple Conditional Logicdef hire_tree(python_score, has_portfolio):    if python_score > 7:        if has_portfolio:            return "Hire (High Potential)"        else:            return "Interview (Verify Skills)"    else:        return "Reject" print(f"Score: 9, Portfolio: True  -> {hire_tree(9, True)}")print(f"Score: 8, Portfolio: False -> {hire_tree(8, False)}")

Decision Trees have a major weakness: they are "greedy". If left alone, a tree will keep growing until it has a leaf for every single employee in your database. This is Overfitting—the tree has memorized your data rather than learning general patterns.

To fix this, we use Regularization. We set a max_depth (capping the number of questions) or min_samples_leaf (ensuring every group has at least, say, 10 employees). This forces the tree to generalize, making it much more accurate on NEW candidates.

# Regularization: Capping Tree Growthdef regularized_tree(score, depth, max_depth=2):    if depth >= max_depth:        return "Generic Category" # Stop growing!    return "Specific Category" print(f"Depth 1: {regularized_tree(0.8, 1)}")print(f"Depth 3: {regularized_tree(0.8, 3)}") # Forced to be generic

One HR manager might be biased or make a mistake. But if you ask 100 managers and take their average, the bias usually cancels out. This is Bootstrap Aggregating (Bagging).

Bagging: We train 100 different trees, each on a slightly different random subset of our employees (sampling with replacement). Pasting is the same thing, but without replacement. By averaging the results, we get a much more stable and reliable prediction of employee turnover.

# Bagging Simulation: Voting Ensemblepredictions = [1, 0, 1, 1, 1, 0, 1] # 7 different trees' opinions def aggregate_vote(votes):    return 1 if sum(votes)/len(votes) > 0.5 else 0 print(f"Ensemble Decision: {aggregate_vote(predictions)} (Majority Wins!)")

A Random Forest is just a collection (ensemble) of Decision Trees, but with a twist. Not only is each tree trained on a random subset of employees, but each "node" in the tree is only allowed to look at a random subset of features (e.g., only "Salary" and "Tenure", ignoring "Department").

This "Feature Randomness" ensures the trees are very different from each other. If one tree focuses too much on a noisy variable, the other 99 trees will correct it. This makes Random Forests one of the most accurate and popular models in HR Analytics today.

# Random Forest: Feature Sub-samplingfeatures = ["Salary", "Tenure", "Skills", "Commute", "Morale"]import random def get_random_features(all_feats, count=2):    # Each 'tree' only sees part of the truth    return random.sample(all_feats, count) print(f"Tree 1 sees: {get_random_features(features)}")print(f"Tree 2 sees: {get_random_features(features)}")

When dealing with massive datasets (e.g., global LinkedIn profiles), we can't look at everything at once.

• Random Subspaces: We keep all employees but only look at a random subset of their skills.
• Random Patches: we take a random subset of BOTH employees and their skills.

This drastically speeds up training and allows us to build powerful models even on limited hardware, perfect for internal HR tools running on standard company servers.

Boosting is a strategy where models learn from each other's mistakes. Instead of training trees independently, we train them sequentially.

• AdaBoost: The first tree tries to predict turnover. It fails on 5 specific employees. The SECOND tree is then forced to focus much harder on those 5 employees. We repeat this until the "Weak" patterns become "Strong" predictions.
• Gradient Boosting: Instead of focusing on failed employees, the new tree tries to predict the Residual Error of the previous tree. It's like a scientific experiment that keeps getting more and more precise over time.

# Boosting: Iterative Error Correctionactual_turnover = 1.0prediction_1 = 0.6 # Tree 1 missed by 0.4error = actual_turnover - prediction_1 # Tree 2 focuses ONLY on that 0.4 errorprediction_2 = prediction_1 + (0.5 * error) print(f"Step 1 Prediction: {prediction_1}")print(f"Step 2 (Boosted) : {prediction_2}")

Stacking (Stacked Generalization) is the ultimate ensemble technique. Instead of a simple majority vote, we train a "Meta-Model" (a Blender) to learn which of our primary models is most trustworthy.

For example, if our SVM is great at predicting Senior hires, but our Random Forest is better at Junior hires, the Blender will learn to listen to the SVM when it sees a senior profile and the Forest when it sees a junior one. This creates a highly nuanced and robust talent engine.

# Stacking: The Blenderdef model_blender(svm_pred, rf_pred, weight_svm=0.7):    # Blender 'learned' that SVM is 70% more reliable here    return (svm_pred * weight_svm) + (rf_pred * (1 - weight_svm)) print(f"Final Intelligence Score: {model_blender(0.9, 0.4):.2f}")