NLP & Transformers

From Tokens to Self-Attention and Generative AI

Understanding human language is one of the most complex tasks in AI. Whether it's an **Autonomous Vehicle** interpreting a passenger's voice command or a **Medical AI** parsing a messy clinical report, Natural Language Processing (NLP) is the bridge between human intent and machine action.

In this chapter, we master the evolution of NLP: from basic **Vectorization** to the revolutionary **Transformer** architecture. We explore how **Self-Attention** allows models to understand the context of every word in a sentence simultaneously, powering the modern age of Large Language Models (LLMs).

Vectorization: Turning Text into Tensors

Computers don't see words; they see numbers. **Vectorization** is the process of mapping words to numerical tensors. We move from simple **One-Hot Encoding** (which treats words as isolated entities) to **Embeddings**, where words with similar meanings (like "Tachycardia" and "Arrythmia") are placed near each other in a multi-dimensional space.

PythonRuns entirely in your browser — nothing is sent to a server.

# Simulation: Clinical Text Vectorizer (NLP)

class MedicalEmbedder:
    def __init__(self):
        # A simplified 'Vocabulary' mapping
        self.vocab = {"patient": 1, "heart": 2, "stable": 3, "critical": 4}
        
    def vectorize(self, text):
        tokens = text.lower().split()
        # Mapping tokens to their indices (simplified embedding lookup)
        vector = [self.vocab.get(t, 0) for t in tokens]
        print("Input Text: {0}".format(text))
        print("Vectorized Tensors: {0}".format(vector))
        return vector

# Scenario: Parsing a clinical status update
embedder = MedicalEmbedder()
embedder.vectorize("Patient HEART remains STABLE")
print("\nInsight: Embeddings allow the model to learn that 'Stable' and 'Healthy' share semantic space.")

# Simulation: Clinical Text Vectorizer (NLP)

class MedicalEmbedder:
    def __init__(self):
        # A simplified 'Vocabulary' mapping
        self.vocab = {"patient": 1, "heart": 2, "stable": 3, "critical": 4}
        
    def vectorize(self, text):
        tokens = text.lower().split()
        # Mapping tokens to their indices (simplified embedding lookup)
        vector = [self.vocab.get(t, 0) for t in tokens]
        print("Input Text: {0}".format(text))
        print("Vectorized Tensors: {0}".format(vector))
        return vector

# Scenario: Parsing a clinical status update
embedder = MedicalEmbedder()
embedder.vectorize("Patient HEART remains STABLE")
print("\nInsight: Embeddings allow the model to learn that 'Stable' and 'Healthy' share semantic space.")

Intent Classification & Language Models

A **Language Model** predicts the probability of the next word in a sequence. In an AV context, this powers **Intent Classification**: determining if a passenger wants the car to "Stop now!" versus "Stop at the next signal." We use Softmax layers to turn raw model outputs into a probability distribution over possible intents.

PythonRuns entirely in your browser — nothing is sent to a server.

# AV - Intent Classification (Voice Command) Simulation

class IntentClassifier:
    def predict_intent(self, command):
        # Output: Probabilities for [Emergency_Stop, Lane_Change, Navigation]
        if "now" in command.lower() or "emergency" in command.lower():
            probs = [0.92, 0.05, 0.03]
        else:
            probs = [0.10, 0.20, 0.70]
            
        print("Command Received: '{0}'".format(command))
        print("Intent Probs [Stop, Lane, Nav]: {0}".format(probs))
        
        # Taking the 'Argmax' (highest probability)
        labels = ["EMERGENCY_STOP", "LANE_CHANGE", "NAVIGATION"]
        return labels[probs.index(max(probs))]

# Scenario: Passenger gives a sudden command
ai_pilot = IntentClassifier()
action = ai_pilot.predict_intent("Stop the car immediately!")
print("\nACTION TRIGGERED: {0}".format(action))

# AV - Intent Classification (Voice Command) Simulation

class IntentClassifier:
    def predict_intent(self, command):
        # Output: Probabilities for [Emergency_Stop, Lane_Change, Navigation]
        if "now" in command.lower() or "emergency" in command.lower():
            probs = [0.92, 0.05, 0.03]
        else:
            probs = [0.10, 0.20, 0.70]
            
        print("Command Received: '{0}'".format(command))
        print("Intent Probs [Stop, Lane, Nav]: {0}".format(probs))
        
        # Taking the 'Argmax' (highest probability)
        labels = ["EMERGENCY_STOP", "LANE_CHANGE", "NAVIGATION"]
        return labels[probs.index(max(probs))]

# Scenario: Passenger gives a sudden command
ai_pilot = IntentClassifier()
action = ai_pilot.predict_intent("Stop the car immediately!")
print("\nACTION TRIGGERED: {0}".format(action))

The Transformer: Attention is All You Need

Traditional RNNs process text word-by-word, which is slow and forgets long-term context. The **Transformer** architecture solves this using **Self-Attention**. It allows every word in a sentence to "attend" to every other word, regardless of distance. This is how a model knows that in the sentence *"The patient was prescribed aspirin because his fever was high"*, the word *"his"* refers back to *"patient."*

PythonRuns entirely in your browser — nothing is sent to a server.

# Transformer - Self-Attention (Heuristic Simulation)

class AttentionHead:
    def calculate_attention(self, tokens):
        # In a real Transformer, this involves Q, K, V matrix multiplications
        print("Input Sequence: {0}".format(tokens))
        print("Calculating Scaled Dot-Product Attention...")
        
        # Simulating attention weights for 'The patient'
        weights = {"Patient": 0.85, "Aspirin": 0.10, "Fever": 0.05}
        for token, score in weights.items():
            print(" - Attention for '{0}': {1:.2f}".format(token, score))

# Scenario: Understanding a Medical Discharge Summary
transformer = AttentionHead()
transformer.calculate_attention(["Patient", "Aspirin", "Fever"])
print("\nInsight: Multi-head attention allows the model to track multiple relationships simultaneously.")

# Transformer - Self-Attention (Heuristic Simulation)

class AttentionHead:
    def calculate_attention(self, tokens):
        # In a real Transformer, this involves Q, K, V matrix multiplications
        print("Input Sequence: {0}".format(tokens))
        print("Calculating Scaled Dot-Product Attention...")
        
        # Simulating attention weights for 'The patient'
        weights = {"Patient": 0.85, "Aspirin": 0.10, "Fever": 0.05}
        for token, score in weights.items():
            print(" - Attention for '{0}': {1:.2f}".format(token, score))

# Scenario: Understanding a Medical Discharge Summary
transformer = AttentionHead()
transformer.calculate_attention(["Patient", "Aspirin", "Fever"])
print("\nInsight: Multi-head attention allows the model to track multiple relationships simultaneously.")

Practice Questions

Question 1

What is the primary advantage of Embeddings over One-Hot Encoding?

Embeddings are easier to spell
Embeddings capture semantic relationships by placing similar words close together in vector space
One-hot encoding is too fast for modern GPUs
Embeddings only work with pictures

Question 2

Why is Self-Attention superior to RNNs for long-range dependencies in text?

It processes words one-by-one very slowly
It allows every word to interact with every other word in a single step, regardless of distance
It requires a special keyboard to type
It only works on the first word of a sentence