Understanding AI Language Models: Architecture and Core Mechanisms

Detailed Description

Artificial Intelligence language models represent one of the most transformative technological breakthroughs of the 21st century. At their core, these models are sophisticated neural networks designed to understand, process, and generate human language with remarkable fluency and contextual awareness. Modern language models, particularly those built on the transformer architecture, have revolutionized how machines interact with text, enabling applications ranging from conversational AI and content generation to code completion and complex reasoning tasks. Understanding the architecture and core mechanisms of these models is fundamental to mastering prompt engineering, as it provides the foundation for crafting effective prompts that leverage the model's capabilities while working within its constraints.

The importance of understanding AI language models extends far beyond academic curiosity. In today's rapidly evolving technological landscape, these models power countless real-world applications that impact millions of users daily. Virtual assistants like ChatGPT, Claude, and Bard rely on sophisticated language models to engage in natural conversations. Content creation platforms use them to assist writers, marketers, and journalists in generating ideas and drafting text. Software development tools like GitHub Copilot leverage language models trained on code to accelerate programming workflows. Customer service chatbots employ these models to provide intelligent, context-aware support. Healthcare organizations use them to analyze medical literature and assist in diagnostic processes. Legal professionals utilize them for document analysis and research. The breadth of applications underscores why understanding these models is not just valuable for AI researchers but essential for anyone working with modern technology.

The transformer architecture, introduced in the seminal 2017 paper "Attention Is All You Need," forms the backbone of modern language models. This architecture replaces the sequential processing of earlier recurrent neural networks with a parallel processing approach centered on attention mechanisms. At its heart, the attention mechanism allows the model to weigh the relevance of different parts of the input when processing each word or token. This enables the model to capture long-range dependencies in text and understand context in ways previous architectures could not. Key concepts include tokenization (breaking text into processable units), embeddings (representing tokens as high-dimensional vectors), positional encodings (preserving word order information), multi-head attention (allowing the model to focus on different aspects simultaneously), and feed-forward networks (transforming representations at each layer). The model learns through exposure to vast amounts of text data, developing statistical understanding of language patterns, semantic relationships, and world knowledge encoded in the training corpus.

Training methodologies for language models typically involve two main phases: pre-training and fine-tuning. During pre-training, models learn from enormous datasets comprising billions or trillions of tokens, developing broad language understanding through objectives like next-token prediction or masked language modeling. This phase requires substantial computational resources and can take weeks or months on specialized hardware. Fine-tuning adapts the pre-trained model for specific tasks or to follow instructions more reliably. Instruction-tuned models, often refined through techniques like Reinforcement Learning from Human Feedback (RLHF), are explicitly trained to follow user instructions, provide helpful responses, and exhibit desired behaviors. Understanding the differences between base models (which simply predict the next token) and instruction-tuned variants (which are optimized for helpful, harmless, and honest responses) is crucial for prompt engineering, as it influences how you should structure your prompts and what behaviors to expect.

Throughout this lesson, you will master the fundamental concepts that govern how language models process and generate text. You'll learn how tokenization affects prompt design, why context window limitations matter, and how probability distributions determine model outputs. You'll understand the role of temperature and other sampling parameters in controlling generation diversity. You'll explore how embeddings capture semantic meaning and enable the model to reason about relationships between concepts. By examining the attention mechanism in detail, you'll gain insight into how models maintain context across long passages. You'll also learn to distinguish between the capabilities of base models versus instruction-tuned variants, helping you set appropriate expectations and design more effective prompts. This foundational knowledge will enable you to think like the model, anticipating its behavior and crafting prompts that align with its processing mechanisms.

By the end of this lesson, you will possess a comprehensive understanding of language model architecture that directly informs your prompt engineering practice. You'll be able to explain how transformers process sequences, describe the role of attention in capturing context, understand how training objectives shape model behavior, and recognize the implications of architectural choices for practical applications. This knowledge transforms prompt engineering from trial-and-error experimentation into an informed practice grounded in understanding of the underlying technology. Whether you're designing prompts for content generation, building AI-powered applications, or simply seeking to use language models more effectively, this foundational understanding provides the essential context for all advanced prompt engineering techniques.

Practical Examples

Example 1: Exploring Tokenization and Its Impact on Prompts

Tokenization is the process of breaking text into units (tokens) that the language model can process. Understanding tokenization is crucial because it affects prompt length, cost, and even model behavior. Different tokenization schemes produce different results, and being aware of token boundaries helps you craft more efficient prompts.


# Example 1: Understanding Tokenization
# This example demonstrates how text is tokenized and why it matters

import tiktoken  # OpenAI's tokenizer library

def analyze_tokenization(text, model="gpt-4"):
    """
    Analyze how text is tokenized and provide insights.
    
    Args:
        text: The input text to tokenize
        model: The model name to use for tokenization
    
    Returns:
        Dictionary containing tokenization analysis
    """
    # Get the encoding for the specified model
    encoding = tiktoken.encoding_for_model(model)
    
    # Encode the text to get token IDs
    tokens = encoding.encode(text)
    
    # Decode each token individually to see what they represent
    decoded_tokens = [encoding.decode([token]) for token in tokens]
    
    # Calculate token count
    token_count = len(tokens)
    
    # Estimate cost (example rates for GPT-4)
    cost_per_1k_tokens = 0.03  # Input cost in USD
    estimated_cost = (token_count / 1000) * cost_per_1k_tokens
    
    return {
        'text': text,
        'token_count': token_count,
        'tokens': decoded_tokens,
        'token_ids': tokens,
        'estimated_cost': estimated_cost
    }

# Example texts demonstrating different tokenization scenarios
examples = [
    "Hello, world!",  # Simple English
    "Hello,world!",   # No space after comma
    "GPT-4 is amazing",  # Hyphenated term
    "人工智能很强大",  # Chinese characters
    "print('Hello')",  # Code
    "supercalifragilisticexpialidocious"  # Long uncommon word
]

print("=" * 80)
print("TOKENIZATION ANALYSIS")
print("=" * 80)

for text in examples:
    result = analyze_tokenization(text)
    print(f"\nOriginal Text: {result['text']}")
    print(f"Token Count: {result['token_count']}")
    print(f"Tokens: {result['tokens']}")
    print(f"Token IDs: {result['token_ids']}")
    print(f"Estimated Cost: ${result['estimated_cost']:.6f}")
    print("-" * 80)

# Demonstrate the impact of tokenization on prompt design
def compare_prompt_efficiency(prompt1, prompt2):
    """
    Compare two prompts to see which is more token-efficient.
    """
    result1 = analyze_tokenization(prompt1)
    result2 = analyze_tokenization(prompt2)
    
    print("\n" + "=" * 80)
    print("PROMPT EFFICIENCY COMPARISON")
    print("=" * 80)
    print(f"\nPrompt 1: {prompt1}")
    print(f"Tokens: {result1['token_count']}")
    print(f"\nPrompt 2: {prompt2}")
    print(f"Tokens: {result2['token_count']}")
    print(f"\nToken Difference: {abs(result1['token_count'] - result2['token_count'])}")
    
    if result1['token_count'] < result2['token_count']:
        print(f"✓ Prompt 1 is more efficient ({result1['token_count']} vs {result2['token_count']} tokens)")
    elif result2['token_count'] < result1['token_count']:
        print(f"✓ Prompt 2 is more efficient ({result2['token_count']} vs {result1['token_count']} tokens)")
    else:
        print("Both prompts use the same number of tokens")

# Compare verbose vs. concise prompts
verbose_prompt = "I would like you to please write a summary of the following text"
concise_prompt = "Summarize the following text"

compare_prompt_efficiency(verbose_prompt, concise_prompt)

Output Explanation:

This example demonstrates several key insights about tokenization. You'll notice that punctuation often forms separate tokens, spaces are included with words, uncommon words may be split into multiple tokens, and non-English text may require more tokens per character. The comparison function shows that concise prompts can significantly reduce token usage without losing clarity, which saves costs and preserves context window space.

Best Practices:

Use concise language in prompts to minimize token usage
Be aware that uncommon words and non-English text consume more tokens
Test prompt variations to find the most token-efficient phrasing
Remember that both input and output tokens count toward costs and limits

Common Mistakes to Avoid:

Assuming character count equals token count (they differ significantly)
Using overly verbose language that wastes tokens
Ignoring tokenization when working with multilingual content
Forgetting that code and special characters have unique tokenization patterns

Example 2: Understanding Embeddings and Semantic Similarity

Embeddings are vector representations of text that capture semantic meaning. Understanding embeddings helps you grasp how models understand relationships between concepts and why certain prompts work better than others.


# Example 2: Exploring Text Embeddings and Semantic Similarity
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
import openai
import os

# Note: This example requires an OpenAI API key
# Set your API key: openai.api_key = "your-api-key"

def get_embedding(text, model="text-embedding-ada-002"):
    """
    Get the embedding vector for a given text.
    
    Args:
        text: Input text to embed
        model: Embedding model to use
    
    Returns:
        Numpy array containing the embedding vector
    """
    text = text.replace("\n", " ")
    response = openai.Embedding.create(input=[text], model=model)
    return np.array(response['data'][0]['embedding'])

def calculate_similarity(text1, text2):
    """
    Calculate semantic similarity between two texts using embeddings.
    
    Args:
        text1: First text
        text2: Second text
    
    Returns:
        Similarity score between 0 and 1
    """
    emb1 = get_embedding(text1).reshape(1, -1)
    emb2 = get_embedding(text2).reshape(1, -1)
    
    similarity = cosine_similarity(emb1, emb2)[0][0]
    return similarity

def analyze_semantic_relationships(query, candidates):
    """
    Analyze how similar a query is to multiple candidate texts.
    
    Args:
        query: The query text
        candidates: List of candidate texts to compare
    
    Returns:
        Sorted list of (candidate, similarity_score) tuples
    """
    query_embedding = get_embedding(query)
    
    similarities = []
    for candidate in candidates:
        candidate_embedding = get_embedding(candidate)
        
        # Calculate cosine similarity
        similarity = cosine_similarity(
            query_embedding.reshape(1, -1),
            candidate_embedding.reshape(1, -1)
        )[0][0]
        
        similarities.append((candidate, similarity))
    
    # Sort by similarity (highest first)
    similarities.sort(key=lambda x: x[1], reverse=True)
    
    return similarities

# Example: Understanding semantic similarity
print("=" * 80)
print("SEMANTIC SIMILARITY ANALYSIS")
print("=" * 80)

# Define a query and related/unrelated texts
query = "machine learning algorithms"

candidates = [
    "artificial intelligence and neural networks",  # Highly related
    "deep learning models",  # Related
    "cooking recipes for dinner",  # Unrelated
    "statistical models for prediction",  # Somewhat related
    "computer programming",  # Tangentially related
    "supervised and unsupervised learning"  # Highly related
]

# Analyze relationships (Note: This requires API access)
# Uncomment when you have API credentials
"""
results = analyze_semantic_relationships(query, candidates)

print(f"\nQuery: '{query}'\n")
print("Similarity Rankings:")
print("-" * 80)

for i, (candidate, score) in enumerate(results, 1):
    # Create visual bar for similarity
    bar_length = int(score * 50)
    bar = "█" * bar_length + "░" * (50 - bar_length)
    
    print(f"{i}. [{bar}] {score:.4f}")
    print(f"   {candidate}\n")
"""

# Demonstrate why this matters for prompt engineering
def demonstrate_prompt_variations():
    """
    Show how different prompt phrasings can be semantically similar.
    """
    base_task = "Classify this email as spam or not spam"
    
    prompt_variations = [
        "Determine whether this email is spam or legitimate",
        "Is this email spam? Yes or no",
        "Categorize this email as either spam or ham",
        "What is the weather today?",  # Unrelated for comparison
    ]
    
    print("\n" + "=" * 80)
    print("PROMPT VARIATION ANALYSIS")
    print("=" * 80)
    print(f"\nBase Task: '{base_task}'\n")
    print("Comparing semantic similarity of variations:")
    print("-" * 80)
    
    # This demonstrates the concept even without API access
    # Actual similarity scores would require API calls
    
    explanations = [
        ("High similarity - same task, different words", 0.92),
        ("High similarity - simpler phrasing, same intent", 0.89),
        ("High similarity - different terminology (ham/spam), same task", 0.87),
        ("Low similarity - completely different task", 0.15)
    ]
    
    for variation, (explanation, estimated_score) in zip(prompt_variations, explanations):
        print(f"Variation: '{variation}'")
        print(f"Estimated Similarity: {estimated_score:.2f}")
        print(f"Explanation: {explanation}\n")

demonstrate_prompt_variations()

# Key insights about embeddings
print("\n" + "=" * 80)
print("KEY INSIGHTS ABOUT EMBEDDINGS")
print("=" * 80)
print("""
1. Embeddings capture semantic meaning, not just keyword matching
   - "ML algorithms" and "machine learning techniques" are highly similar
   - Even without shared words, semantically related concepts cluster together

2. Similar prompts produce similar model behavior
   - Multiple ways to phrase the same request will trigger similar responses
   - The model recognizes intent beyond exact wording

3. Embeddings enable semantic search
   - Find related documents without exact keyword matches
   - Power retrieval-augmented generation (RAG) systems

4. Vector space properties
   - Closer vectors = more similar meanings
   - Relationships can be captured (king - man + woman ≈ queen)

5. Implications for prompt engineering
   - You can rephrase prompts while maintaining similar results
   - Understanding semantic similarity helps predict model behavior
   - Context matters: embeddings capture surrounding information
""")

Output Explanation:

This example illustrates how language models represent text as high-dimensional vectors where semantic similarity corresponds to vector proximity. The semantic similarity scores reveal that models can recognize conceptual relationships even when exact words differ. This explains why various phrasings of the same prompt often produce similar results.

Best Practices:

Leverage semantic similarity to test prompt variations efficiently
Use embeddings to find relevant examples for few-shot prompting
Understand that models recognize intent beyond exact wording
Consider semantic relationships when designing prompt templates

Example 3: Attention Mechanisms and Context Window Management

The attention mechanism is fundamental to how transformers process sequences. Understanding attention helps you manage context windows effectively and structure prompts for optimal information flow.


# Example 3: Understanding Attention and Context Windows
import math

class SimpleAttentionVisualizer:
    """
    A simplified visualization of how attention mechanisms work.
    This helps understand how models weigh different parts of input.
    """
    
    def __init__(self, context_window_size=4096):
        self.context_window_size = context_window_size
    
    def calculate_simple_attention(self, query, keys):
        """
        Calculate simplified attention scores (dot-product attention).
        
        Args:
            query: Query vector (what we're looking for)
            keys: Key vectors (what we're looking at)
        
        Returns:
            Attention weights showing which keys are most relevant
        """
        # Calculate dot product between query and each key
        scores = [sum(q * k for q, k in zip(query, key)) for key in keys]
        
        # Apply softmax to get probabilities
        exp_scores = [math.exp(score) for score in scores]
        sum_exp = sum(exp_scores)
        attention_weights = [exp_score / sum_exp for exp_score in exp_scores]
        
        return attention_weights
    
    def demonstrate_attention_pattern(self, sentence, focus_word_index):
        """
        Demonstrate how attention focuses on relevant words.
        
        Args:
            sentence: Input sentence as list of words
            focus_word_index: Which word we're focusing on (query)
        """
        print(f"\nFocusing on word: '{sentence[focus_word_index]}'")
        print("Attention distribution across sentence:")
        print("-" * 60)
        
        # Simulate attention weights (in reality, these come from learned parameters)
        # For demonstration, we create plausible attention patterns
        num_words = len(sentence)
        
        # Generate simulated attention scores based on proximity and relevance
        attention_scores = []
        for i, word in enumerate(sentence):
            # Simulate attention based on distance and mock "relevance"
            distance = abs(i - focus_word_index)
            # More attention to closer words, with some randomness
            score = 1.0 / (1.0 + distance * 0.5)
            attention_scores.append(score)
        
        # Normalize to sum to 1
        total = sum(attention_scores)
        attention_weights = [score / total for score in attention_scores]
        
        # Display attention visualization
        for word, weight in zip(sentence, attention_weights):
            bar_length = int(weight * 50)
            bar = "█" * bar_length
            print(f"{word:15} [{bar:50}] {weight:.3f}")
    
    def analyze_context_window_usage(self, prompt, max_tokens):
        """
        Analyze how a prompt uses the context window.
        
        Args:
            prompt: The input prompt
            max_tokens: Maximum tokens for completion
        """
        # Simplified token counting (assume ~4 chars per token)
        estimated_tokens = len(prompt) // 4
        available_for_completion = self.context_window_size - estimated_tokens
        
        print("\n" + "=" * 60)
        print("CONTEXT WINDOW ANALYSIS")
        print("=" * 60)
        print(f"Total Context Window: {self.context_window_size} tokens")
        print(f"Prompt Length: ~{estimated_tokens} tokens")
        print(f"Available for Completion: ~{available_for_completion} tokens")
        print(f"Requested Max Tokens: {max_tokens} tokens")
        
        if max_tokens > available_for_completion:
            print("\n⚠️  WARNING: Requested tokens exceed available space!")
            print(f"    Reduce prompt by ~{max_tokens - available_for_completion} tokens")
        else:
            print("\n✓ Context window usage is optimal")
            
        # Calculate utilization percentage
        total_usage = estimated_tokens + max_tokens
        utilization = (total_usage / self.context_window_size) * 100
        
        print(f"\nContext Window Utilization: {utilization:.1f}%")
        
        # Visual representation
        prompt_portion = int((estimated_tokens / self.context_window_size) * 50)
        completion_portion = int((max_tokens / self.context_window_size) * 50)
        empty_portion = 50 - prompt_portion - completion_portion
        
        visual = "█" * prompt_portion + "▓" * completion_portion + "░" * empty_portion
        print(f"\n[{visual}]")
        print(f" █ Prompt ({estimated_tokens}) | ▓ Completion ({max_tokens}) | ░ Unused")

# Example usage: Understanding attention patterns
visualizer = SimpleAttentionVisualizer()

print("=" * 80)
print("ATTENTION MECHANISM DEMONSTRATION")
print("=" * 80)

# Example sentence
sentence = ["The", "cat", "sat", "on", "the", "mat", "and", "purred"]

# Show attention for different focus words
for focus_index in [1, 5]:  # "cat" and "mat"
    visualizer.demonstrate_attention_pattern(sentence, focus_index)

# Demonstrate context window management
print("\n\n" + "=" * 80)
print("CONTEXT WINDOW MANAGEMENT")
print("=" * 80)

# Example 1: Efficient prompt
short_prompt = "Summarize: " + "AI models process text. " * 10
visualizer.analyze_context_window_usage(short_prompt, max_tokens=500)

# Example 2: Inefficient prompt
long_prompt = "I would like you to please carefully read and thoroughly analyze " * 50
visualizer.analyze_context_window_usage(long_prompt, max_tokens=2000)

# Best practices for context window management
print("\n\n" + "=" * 80)
print("BEST PRACTICES FOR CONTEXT WINDOW MANAGEMENT")
print("=" * 80)
print("""
1. FRONT-LOAD CRITICAL INFORMATION
   ✓ Place most important context at the beginning
   ✓ Models attend more strongly to recent information
   
2. STRUCTURE LONG PROMPTS HIERARCHICALLY
   ✓ Use clear sections and headings
   ✓ Make key points stand out
   
3. OPTIMIZE TOKEN USAGE
   ✓ Remove unnecessary verbose language
   ✓ Use bullet points instead of prose when appropriate
   
4. CONSIDER ATTENTION DECAY
   ✓ Important information buried in long prompts may be overlooked
   ✓ Repeat crucial instructions if context is long
   
5. MONITOR CONTEXT LIMITS
   ✓ Know your model's context window (4K, 8K, 32K, 100K tokens)
   ✓ Leave adequate space for the completion
   
6. USE TRUNCATION STRATEGICALLY
   ✓ If input is too long, truncate less important middle sections
   ✓ Keep beginning (context) and end (recent focus) intact
""")

# Demonstrate attention decay simulation
def simulate_attention_decay():
    """
    Simulate how attention decays over long contexts.
    """
    print("\n" + "=" * 80)
    print("ATTENTION DECAY OVER LONG CONTEXTS")
    print("=" * 80)
    
    positions = [0, 25, 50, 75, 90, 95, 99]
    context_length = 100
    
    print("\nRelative attention strength by position in context:")
    print("(Position 0 = start, Position 99 = end)\n")
    
    for pos in positions:
        # Simulate attention: stronger at beginning and end
        if pos < 20:
            attention = 0.9 - (pos * 0.02)  # Strong at start
        elif pos > 80:
            attention = 0.5 + ((pos - 80) * 0.02)  # Increases toward end
        else:
            attention = 0.3  # Weaker in middle
        
        bar_length = int(attention * 50)
        bar = "█" * bar_length + "░" * (50 - bar_length)
        print(f"Position {pos:3d}: [{bar}] {attention:.2f}")
    
    print("\n⚡ Key Insight: Information at the beginning and end of prompts")
    print("   receives stronger attention than information in the middle.")

simulate_attention_decay()

Output Explanation:

This example demonstrates how attention mechanisms allow models to focus on relevant parts of the input. The visualizations show that attention is not uniform across the context—models pay more attention to recent tokens and important information at the beginning. The context window analysis helps you understand token budget constraints and optimize prompt design.

Best Practices:

Place critical information at the beginning or end of long prompts
Structure prompts to guide attention to important elements
Monitor context window usage to avoid truncation
Use clear section markers in long prompts to help maintain focus

Example 4: Temperature and Sampling Parameters

Understanding how probability distributions and sampling parameters affect generation is crucial for controlling model outputs and achieving desired behaviors.


# Example 4: Understanding Temperature and Sampling Parameters
import random
import math
from collections import Counter

class SamplingSimulator:
    """
    Simulate how different sampling parameters affect text generation.
    """
    
    def __init__(self):
        # Simulated probability distribution over next tokens
        # In reality, this comes from the model's output layer
        self.vocabulary = [
            "great", "good", "excellent", "wonderful", "nice",
            "okay", "fine", "decent", "acceptable", "adequate",
            "bad", "poor", "terrible", "awful", "horrible"
        ]
        
        # Base probabilities (before temperature scaling)
        self.base_probs = [
            0.30, 0.25, 0.15, 0.10, 0.08,  # Positive words
            0.04, 0.03, 0.02, 0.01, 0.01,  # Neutral words
            0.005, 0.003, 0.002, 0.001, 0.001  # Negative words
        ]
    
    def apply_temperature(self, logits, temperature):
        """
        Apply temperature scaling to logits.
        
        Temperature controls randomness:
        - temperature → 0: More deterministic (peaks at highest probability)
        - temperature = 1: No change
        - temperature > 1: More random (flattens distribution)
        
        Args:
            logits: Log probabilities before softmax
            temperature: Temperature parameter
        
        Returns:
            Probabilities after temperature scaling
        """
        # Convert probabilities to logits
        logits = [math.log(p + 1e-10) for p in logits]
        
        # Apply temperature
        scaled_logits = [logit / temperature for logit in logits]
        
        # Apply softmax
        exp_logits = [math.exp(logit) for logit in scaled_logits]
        sum_exp = sum(exp_logits)
        probs = [exp_logit / sum_exp for exp_logit in exp_logits]
        
        return probs
    
    def sample_with_top_p(self, probs, top_p=0.9):
        """
        Apply nucleus (top-p) sampling.
        
        Only sample from the smallest set of tokens whose cumulative
        probability exceeds top_p.
        
        Args:
            probs: Probability distribution
            top_p: Cumulative probability threshold
        
        Returns:
            Filtered probability distribution
        """
        # Sort probabilities in descending order
        sorted_probs = sorted(enumerate(probs), key=lambda x: x[1], reverse=True)
        
        # Find cutoff point
        cumsum = 0
        cutoff_index = 0
        for i, (idx, prob) in enumerate(sorted_probs):
            cumsum += prob
            if cumsum >= top_p:
                cutoff_index = i + 1
                break
        
        # Keep only top-p tokens
        kept_indices = [idx for idx, _ in sorted_probs[:cutoff_index]]
        
        # Renormalize
        new_probs = [0] * len(probs)
        total = sum(probs[idx] for idx in kept_indices)
        for idx in kept_indices:
            new_probs[idx] = probs[idx] / total
        
        return new_probs
    
    def sample_with_top_k(self, probs, top_k=5):
        """
        Apply top-k sampling.
        
        Only sample from the k most likely tokens.
        
        Args:
            probs: Probability distribution
            top_k: Number of top tokens to keep
        
        Returns:
            Filtered probability distribution
        """
        # Get indices of top-k probabilities
        top_k_indices = sorted(range(len(probs)), key=lambda i: probs[i], reverse=True)[:top_k]
        
        # Create new distribution
        new_probs = [0] * len(probs)
        total = sum(probs[idx] for idx in top_k_indices)
        for idx in top_k_indices:
            new_probs[idx] = probs[idx] / total
        
        return new_probs
    
    def generate_samples(self, temperature=1.0, top_p=None, top_k=None, num_samples=100):
        """
        Generate samples to demonstrate parameter effects.
        
        Args:
            temperature: Temperature for scaling
            top_p: Nucleus sampling parameter (None to disable)
            top_k: Top-k sampling parameter (None to disable)
            num_samples: Number of samples to generate
        
        Returns:
            List of sampled tokens
        """
        # Apply temperature
        probs = self.apply_temperature(self.base_probs, temperature)
        
        # Apply top-p sampling if specified
        if top_p is not None:
            probs = self.sample_with_top_p(probs, top_p)
        
        # Apply top-k sampling if specified
        if top_k is not None:
            probs = self.sample_with_top_k(probs, top_k)
        
        # Generate samples
        samples = random.choices(self.vocabulary, weights=probs, k=num_samples)
        
        return samples, probs

# Demonstrate different temperature settings
simulator = SamplingSimulator()

print("=" * 80)
print("TEMPERATURE EFFECTS ON SAMPLING")
print("=" * 80)

temperatures = [0.1, 0.7, 1.0, 1.5, 2.0]

for temp in temperatures:
    print(f"\n{'='*80}")
    print(f"Temperature: {temp}")
    print('='*80)
    
    samples, probs = simulator.generate_samples(temperature=temp, num_samples=100)
    
    # Show probability distribution
    print("\nProbability Distribution:")
    for word, prob in zip(simulator.vocabulary[:5], probs[:5]):
        bar_length = int(prob * 50)
        bar = "█" * bar_length + "░" * (50 - bar_length)
        print(f"{word:12} [{bar}] {prob:.4f}")
    
    # Show sample diversity
    counter = Counter(samples)
    most_common = counter.most_common(5)
    
    print(f"\nMost