Machine Learning & Web Scraping

The pinnacle of quantitative analysis. Learn to build the complete 'AI Data Pipeline': Scraping alternative data from the web, visualizing market correlations, and training predictive Machine Learning models using Scikit-Learn.

1. Phase 1: Data Ingestion (Web Scraping)

In modern finance, structured API data is not enough. Alpha (a trading edge) is found in Alternative Data: scraping Federal Reserve press releases, parsing HTML tables of corporate SEC filings, or scraping news sentiment from financial websites.

BeautifulSoup (bs4) is the industry-standard python library for extracting data from messy HTML web pages.

  • soup.find_all('table'): Extracts all HTML tables from a scraped webpage.
  • soup.find('div', class_='sentiment').text: Extracts the raw text from specific HTML elements.
Python
from bs4 import BeautifulSoupimport pandas as pd # EXPLANATION: We downloaded the raw HTML code of a financial news websiteraw_html = """<html>  <body>    <h1>Latest Corporate Actions</h1>    <table id="earnings">      <tr><th>Ticker</th><th>EPS</th><th>Sentiment</th></tr>      <tr><td>AAPL</td><td>$1.50</td><td class="pos">Bullish</td></tr>      <tr><td>TSLA</td><td>$0.60</td><td class="neg">Bearish</td></tr>    </table>  </body></html>""" # 1. Parse the raw HTML text into a structured BeautifulSoup objectsoup = BeautifulSoup(raw_html, 'html.parser') print("--- 🔍 Initiating Alternative Data Scrape ---")# 2. Extract specific elements (Scrape the <td> tags)for row in soup.find_all('tr')[1:]: # Skip the header row    cols = row.find_all('td')    ticker = cols[0].text    eps = cols[1].text    sentiment = cols[2].text    print(f"Scraped -> {ticker} reported {eps}. Market reaction: {sentiment}")

2. Phase 2: EDA & Visualization

Before writing a single line of Artificial Intelligence code, Data Scientists must perform Exploratory Data Analysis (EDA). You have to visually understand your data to know what model to build.

  • Matplotlib (plt): The foundation of all Python plots. Good for charting line graphs (Stock Prices) and scatter plots (Asset Distributions).
  • Seaborn (sns): Built on top of Matplotlib, designed specifically for statistical data. Incredible for building Heatmaps to measure the mathematical correlations between dozens of assets at once.
Python
import matplotlib.pyplot as pltimport numpy as np # EXPLANATION: Visualizing the performance of two trading algorithmsdays = np.array([1, 2, 3, 4, 5])algo_a_returns = np.array([2.5, 3.1, 4.0, 3.8, 5.2])algo_b_returns = np.array([1.0, 1.5, -2.0, 8.5, 9.0]) # We use Matplotlib to plot this data# (Note: In standard Python this opens a UI window. In our REPL, we render the graph virtually!)fig, ax = plt.subplots(figsize=(6, 4))ax.plot(days, algo_a_returns, label='Strategy A (Mean Reversion)', marker='o', color='blue')ax.plot(days, algo_b_returns, label='Strategy B (Momentum)', marker='x', color='orange') ax.set_title("Trading Strategy PnL Over Time")ax.set_xlabel("Days Administered")ax.set_ylabel("Cumulative Return (%)")ax.legend()ax.grid(True) print("--- 📊 Virtual Chart Generated ---")

3. Phase 3: Machine Learning (Scikit-Learn)

< p > After cleaning and visualizing data, we trainMachine Learning Algorithms to find patterns invisible to humans.

< h4 > 1. Linear Regression(Predicting Numbers) < /strong>

Used to predict a continuous specific number. < em > "Based on historical interest rates, what will the exact price of this bond be tomorrow?" < /em>

2. Logistic Regression(Binary Classification) < /strong>

Despite the name, this is used for Classification < /strong>. It predicts a binary Yes/No outcome. < em > "Based on this user's transaction history, is this new $5,000 transfer Fraudulent? (True/False)" < /em>

Python
from sklearn.linear_model import LinearRegressionimport numpy as np # EXPLANATION: Predicting Future Stock Prices based purely on Time(Days passed)# X must be a 2D array for Scikit - Learn(hence the reshape)X_days = np.array([1, 2, 3, 4, 5]).reshape(-1, 1) y_historical_price = np.array([50, 52, 53, 56, 58]) print("--- 🧠 Training Machine Learning Model ---")# 1. Initialize the bare mathematical model    model = LinearRegression() # 2. 'Fit'(Train) the model on the historical data    model.fit(X_days, y_historical_price)    print("Model mathematically fitted successfully!") # 3. Predict the future!(What will the price be on Day 6 and Day 7 ?)    future_days = np.array([6, 7]).reshape(-1, 1)    predictions = model.predict(future_days)     print(f"\n--- 🔮 AI Predictions ---")    print(f"Day 6 Predicted Price: ${predictions[0]:.2f}")    print(f"Day 7 Predicted Price: ${predictions[1]:.2f}")

4. Phase 4: Advanced Modeling (Decision Trees)

< p > Linear Regression is mathematically simple(drawing a straight line).But financial markets are non - linear and incredibly complex.

< p > Decision Trees < /strong> are algorithms that build a complex flowchart of rules to classify data. They are highly popular in Banking for Credit Risk Assessment because regulators require loans to be interpretable(you must explain < em > why < /em> an algorithm denied a loan).

  • clf = DecisionTreeClassifier(max_depth = 3) < /code>: Creates an AI that will branch out based on features like Income, Debt Ratio, and Credit Score.
  • clf.predict(new_applicant) < /code>: Runs a new customer through the generated flowchart to output Default or < code > Repay < /code>.
Python
from sklearn.tree import DecisionTreeClassifierimport numpy as np # FEATURES (X): [Annual Income ($k), Debt-to-Income Ratio]X_applicants = np.array([    [120, 0.1], # Rich, low debt    [45, 0.6],  # Low income, high debt    [80, 0.4],  # Medium income, med debt    [35, 0.8]   # Low income, extreme debt]) # LABELS (y): Did they historically Default (1) or Repay (0)?y_defaults = np.array([0, 1, 0, 1]) print("--- 🌳 Training Decision Tree Classifier ---")clf = DecisionTreeClassifier(max_depth=2)clf.fit(X_applicants, y_defaults)print("Tree mapped!") # Testing a brand new applicant: $50k income, 0.7 Debt Rationew_applicant = np.array([[50, 0.7]])prediction = clf.predict(new_applicant) print("\n--- 🏦 Bank Auto-Approval System ---")result = "DEFAULT RISK (Denied)" if prediction[0] == 1 else "REPAY (Approved)"print(f"Applicant [50k Income, 0.7 Debt]: {result}")
PythonRuns entirely in your browser — nothing is sent to a server.

Practice Questions

Question 1

Why do quantitative analysts use BeautifulSoup (Web Scraping) instead of just relying on APIs?

  • Because APIs are faster than Web Scraping.
  • To extract exclusive 'Alternative Data' (like news sentiment, messy SEC filings, or press releases) that haven't been structured into an expensive API yet.
  • Because BeautifulSoup is a database management system.
  • APIs are illegal to use in financial trading.

Question 2

In Scikit-Learn, which algorithm is mathematically designed to predict a continuous numerical value (e.g. 'This asset will be exactly $140.50 tomorrow')?

  • Logistic Regression
  • Decision Trees
  • Linear Regression
  • BeautifulSoup

Question 3

Why are Decision Trees overwhelmingly preferred by retail Banks when building AI models to automatically Approve or Deny credit loans?

  • They are 100% accurate and never make mistakes.
  • They are 'interpretable'. By law, banks must explain exactly why a loan was denied. Decision Trees generate a literal human-readable flowchart of mathematical rules.
  • Decision Trees are the fastest algorithms in existence.
  • They cannot predict False positives.