Cloud Data Platforms & Lakehouses

Databricks, Delta Lake, and the Modern Data Stack

The traditional data warehouse is evolving. Companies like **Walmart** now use **Lakehouses**—a hybrid of data lakes (cheap, scalable storage) and data warehouses (structured, fast queries). **Databricks** and **Delta Lake** provide this unified platform, letting analysts run SQL and ML on the same data without moving it.

In this chapter, we explore **Star Schemas** for dimensional modeling, compare **Snowflake** vs. **BigQuery** vs. **Databricks**, and understand how **Walmart** architects its global data infrastructure across regions.

The Lakehouse Architecture

A **Lakehouse** combines the best of both worlds: the cheap, scalable storage of a Data Lake (like S3/ADLS) with the transactional guarantees and schema enforcement of a Data Warehouse. **Databricks** with **Delta Lake** is the pioneering platform for this pattern. For **Walmart**, this means storing raw IoT sensor data AND cleaned analytics tables in one system.

PythonRuns entirely in your browser — nothing is sent to a server.

# Walmart Lakehouse: Architecture Simulation
class LakehouseArchitect:
    def __init__(self):
        self.bronze = [] # Raw, unprocessed data
        self.silver = [] # Cleaned, validated data
        self.gold = []   # Business-ready aggregations

    def ingest_to_bronze(self, raw_data):
        self.bronze = raw_data
        print("BRONZE LAYER: {0} raw records ingested (IoT sensor dumps)".format(len(raw_data)))

    def clean_to_silver(self):
        self.silver = [r for r in self.bronze if r.get("valid")]
        print("SILVER LAYER: {0} validated records (removed {1} invalid)".format(
            len(self.silver), len(self.bronze) - len(self.silver)))

    def aggregate_to_gold(self):
        regions = {}
        for r in self.silver:
            region = r.get("region", "Unknown")
            regions[region] = regions.get(region, 0) + r.get("units", 0)
        self.gold = regions
        print("GOLD LAYER (Business Metrics):")
        for region, total in sorted(self.gold.items()):
            print(" - {0}: {1} units".format(region, total))

# Scenario: Walmart's multi-region inventory pipeline
raw = [
    {"region": "APAC", "units": 5000, "valid": True},
    {"region": "EU", "units": 3000, "valid": True},
    {"region": "APAC", "units": -1, "valid": False},
    {"region": "NA", "units": 7000, "valid": True}
]
lake = LakehouseArchitect()
lake.ingest_to_bronze(raw)
lake.clean_to_silver()
lake.aggregate_to_gold()

# Walmart Lakehouse: Architecture Simulation
class LakehouseArchitect:
    def __init__(self):
        self.bronze = [] # Raw, unprocessed data
        self.silver = [] # Cleaned, validated data
        self.gold = []   # Business-ready aggregations

    def ingest_to_bronze(self, raw_data):
        self.bronze = raw_data
        print("BRONZE LAYER: {0} raw records ingested (IoT sensor dumps)".format(len(raw_data)))

    def clean_to_silver(self):
        self.silver = [r for r in self.bronze if r.get("valid")]
        print("SILVER LAYER: {0} validated records (removed {1} invalid)".format(
            len(self.silver), len(self.bronze) - len(self.silver)))

    def aggregate_to_gold(self):
        regions = {}
        for r in self.silver:
            region = r.get("region", "Unknown")
            regions[region] = regions.get(region, 0) + r.get("units", 0)
        self.gold = regions
        print("GOLD LAYER (Business Metrics):")
        for region, total in sorted(self.gold.items()):
            print(" - {0}: {1} units".format(region, total))

# Scenario: Walmart's multi-region inventory pipeline
raw = [
    {"region": "APAC", "units": 5000, "valid": True},
    {"region": "EU", "units": 3000, "valid": True},
    {"region": "APAC", "units": -1, "valid": False},
    {"region": "NA", "units": 7000, "valid": True}
]
lake = LakehouseArchitect()
lake.ingest_to_bronze(raw)
lake.clean_to_silver()
lake.aggregate_to_gold()

Star Schema: Dimensional Modeling

A **Star Schema** organizes data into a central **Fact Table** (transactions, events) surrounded by **Dimension Tables** (products, customers, time). For **Visa**, the fact table contains every transaction, while dimension tables store merchant details, cardholder profiles, and geographic metadata. This structure makes analytical queries lightning-fast.

PythonRuns entirely in your browser — nothing is sent to a server.

# Visa Analytics: Star Schema Design
class StarSchema:
    def __init__(self):
        # Dimension Tables
        self.dim_merchant = {1: "Amazon", 2: "Walmart", 3: "Netflix"}
        self.dim_card = {101: "VISA-Gold", 102: "MC-Platinum"}
        # Fact Table (transaction_id, merchant_id, card_id, amount)
        self.fact_transactions = [
            (1, 1, 101, 250), (2, 2, 102, 50), (3, 3, 101, 15), (4, 1, 102, 1200)
        ]

    def query_merchant_revenue(self):
        print("STAR SCHEMA QUERY: Revenue by Merchant")
        revenue = {}
        for _, m_id, _, amt in self.fact_transactions:
            name = self.dim_merchant.get(m_id, "Unknown")
            revenue[name] = revenue.get(name, 0) + amt
        
        for merchant, total in sorted(revenue.items(), key=lambda x: -x[1]):
            print(" - {0}: 1".format(merchant, total))

# Scenario: Visa quarterly merchant analytics
schema = StarSchema()
schema.query_merchant_revenue()

# Visa Analytics: Star Schema Design
class StarSchema:
    def __init__(self):
        # Dimension Tables
        self.dim_merchant = {1: "Amazon", 2: "Walmart", 3: "Netflix"}
        self.dim_card = {101: "VISA-Gold", 102: "MC-Platinum"}
        # Fact Table (transaction_id, merchant_id, card_id, amount)
        self.fact_transactions = [
            (1, 1, 101, 250), (2, 2, 102, 50), (3, 3, 101, 15), (4, 1, 102, 1200)
        ]

    def query_merchant_revenue(self):
        print("STAR SCHEMA QUERY: Revenue by Merchant")
        revenue = {}
        for _, m_id, _, amt in self.fact_transactions:
            name = self.dim_merchant.get(m_id, "Unknown")
            revenue[name] = revenue.get(name, 0) + amt
        
        for merchant, total in sorted(revenue.items(), key=lambda x: -x[1]):
            print(" - {0}: 1".format(merchant, total))

# Scenario: Visa quarterly merchant analytics
schema = StarSchema()
schema.query_merchant_revenue()

Platform Comparison: Snowflake vs. BigQuery vs. Databricks

Choosing the right platform depends on your workload. **Snowflake** excels at separated compute/storage for variable workloads. **BigQuery** offers serverless simplicity for Google-native teams. **Databricks** is the king of unified analytics (SQL + ML + Streaming). For **Walmart**, Databricks is ideal because it handles both their Kafka streams and their ML pipelines.

PythonRuns entirely in your browser — nothing is sent to a server.

# Cloud Platform Comparison Engine
class PlatformAdvisor:
    def __init__(self):
        self.platforms = {
            "Snowflake": {"sql": 9, "ml": 5, "streaming": 4, "cost_flex": 9},
            "BigQuery": {"sql": 8, "ml": 7, "streaming": 6, "cost_flex": 7},
            "Databricks": {"sql": 7, "ml": 10, "streaming": 9, "cost_flex": 6}
        }

    def recommend(self, priorities):
        print("PLATFORM COMPARISON (Priorities: {0}):".format(priorities))
        scores = {}
        for name, caps in self.platforms.items():
            score = sum(caps.get(p, 0) for p in priorities)
            scores[name] = score
            print(" - {0}: Score={1} {2}".format(name, score, caps))
        
        best = max(scores, key=scores.get)
        print("\nRECOMMENDATION: {0} (Score: {1})".format(best, scores[best]))

# Scenario: Walmart needs SQL + ML + Streaming
advisor = PlatformAdvisor()
advisor.recommend(priorities=["sql", "ml", "streaming"])

# Cloud Platform Comparison Engine
class PlatformAdvisor:
    def __init__(self):
        self.platforms = {
            "Snowflake": {"sql": 9, "ml": 5, "streaming": 4, "cost_flex": 9},
            "BigQuery": {"sql": 8, "ml": 7, "streaming": 6, "cost_flex": 7},
            "Databricks": {"sql": 7, "ml": 10, "streaming": 9, "cost_flex": 6}
        }

    def recommend(self, priorities):
        print("PLATFORM COMPARISON (Priorities: {0}):".format(priorities))
        scores = {}
        for name, caps in self.platforms.items():
            score = sum(caps.get(p, 0) for p in priorities)
            scores[name] = score
            print(" - {0}: Score={1} {2}".format(name, score, caps))
        
        best = max(scores, key=scores.get)
        print("\nRECOMMENDATION: {0} (Score: {1})".format(best, scores[best]))

# Scenario: Walmart needs SQL + ML + Streaming
advisor = PlatformAdvisor()
advisor.recommend(priorities=["sql", "ml", "streaming"])

Practice Questions

Question 1

What is the key advantage of a Lakehouse over a traditional Data Warehouse?

It uses more storage
It combines cheap scalable storage with structured query capabilities, supporting both SQL analytics and ML on the same platform
It only works with Databricks
It is always faster than a warehouse

Question 2

In a Star Schema, what is the role of the Fact Table?

It stores static reference data like product names
It contains the core measurable events (transactions, clicks, sales) with foreign keys to dimension tables
It is used to store backups
It is only used for visualization