Detailed Description

Natural Language Processing (NLP) represents one of the most fascinating and challenging frontiers in artificial intelligence, sitting at the intersection of linguistics, computer science, and cognitive psychology. At its core, NLP is concerned with enabling computers to understand, interpret, and generate human language in ways that are both meaningful and useful. This journey from raw text to computational understanding has evolved dramatically over the past several decades, transitioning from rigid rule-based systems that required extensive manual linguistic knowledge to modern neural approaches that learn patterns directly from vast amounts of text data. Understanding this evolution is crucial for anyone entering the field, as it provides context for current methodologies and hints at future directions.

The importance of NLP in today's digital landscape cannot be overstated. Every time you ask Siri or Alexa a question, search for information on Google, receive autocomplete suggestions while typing an email, or see a post automatically translated on social media, you're experiencing NLP in action. Businesses leverage NLP for sentiment analysis to understand customer feedback, use chatbots to provide 24/7 customer service, extract insights from massive document repositories, and automate content moderation. Healthcare organizations employ NLP to parse medical records and identify treatment patterns, while financial institutions use it to analyze market sentiment and detect fraud. The ability to process and understand human language at scale has become a fundamental competitive advantage across virtually every industry.

To truly grasp NLP, you must understand several foundational linguistic concepts that inform how we build computational models. Morphology deals with the structure of words—how prefixes, suffixes, and roots combine to create meaning (consider "unhappiness" = "un-" + "happy" + "-ness"). Syntax concerns the rules that govern how words combine to form grammatically correct sentences, helping us understand that "The cat chased the mouse" and "The mouse chased the cat" have identical structures but very different meanings. Semantics addresses meaning itself—both at the word level (lexical semantics) and sentence level (compositional semantics). Beyond these, pragmatics deals with how context influences interpretation, explaining why "Can you pass the salt?" is understood as a request rather than a yes/no question. Modern NLP systems must grapple with all these levels, either explicitly through hand-crafted rules or implicitly through learned representations.

Common use cases for NLP span a remarkable range of applications. Text classification tasks include spam detection, sentiment analysis, topic categorization, and intent recognition in conversational systems. Information extraction encompasses named entity recognition (identifying people, places, organizations), relationship extraction, and event detection from unstructured text. Language generation applications include machine translation, text summarization, dialogue systems, and creative content generation. Question answering systems range from simple FAQ bots to sophisticated reading comprehension models that can extract answers from documents. More advanced applications include semantic search, document similarity, knowledge graph construction, and multimodal understanding that combines text with images or speech.

By the end of this lesson, you will have mastered several critical competencies that form the foundation of your NLP journey. You'll understand the historical progression of NLP techniques and why modern approaches have largely superseded older methods while still incorporating linguistic insights. You'll be able to identify and explain key linguistic concepts and recognize how they manifest in computational challenges. You'll have a working Python development environment with essential NLP libraries installed and verified, ready for hands-on experimentation. Most importantly, you'll have processed your first text data programmatically, gaining practical experience with tokenization, part-of-speech tagging, named entity recognition, and other fundamental NLP operations. This foundation will enable you to confidently progress to more advanced topics in subsequent lessons, whether your interest lies in building conversational AI systems, developing text analytics pipelines, or conducting NLP research.

Practical Examples

Example 1: Setting Up Your NLP Development Environment

Before diving into NLP tasks, we need a properly configured environment with essential libraries. This example demonstrates installing and verifying key NLP libraries, along with downloading necessary linguistic resources.


"""
Example 1: NLP Environment Setup and Verification
This script sets up essential NLP libraries and verifies their installation
"""

import sys
import subprocess

def install_libraries():
    """
    Install essential NLP libraries using pip
    In production, you'd typically use a requirements.txt file
    """
    libraries = [
        'nltk',           # Natural Language Toolkit - classical NLP
        'spacy',          # Industrial-strength NLP
        'transformers',   # State-of-the-art transformer models
        'torch',          # PyTorch for deep learning
        'scikit-learn',   # Machine learning utilities
        'pandas',         # Data manipulation
        'numpy'           # Numerical operations
    ]
    
    print("Installing NLP libraries...")
    for lib in libraries:
        try:
            subprocess.check_call([sys.executable, '-m', 'pip', 'install', lib])
            print(f"✓ {lib} installed successfully")
        except subprocess.CalledProcessError:
            print(f"✗ Failed to install {lib}")

def download_nltk_resources():
    """
    Download essential NLTK data packages
    These include corpora, tokenizers, and trained models
    """
    import nltk
    
    # Essential NLTK resources for basic NLP tasks
    resources = [
        'punkt',          # Sentence tokenizer
        'averaged_perceptron_tagger',  # POS tagger
        'maxent_ne_chunker',  # Named entity chunker
        'words',          # Word list
        'stopwords',      # Common stop words
        'wordnet'         # Lexical database
    ]
    
    print("\nDownloading NLTK resources...")
    for resource in resources:
        try:
            nltk.download(resource, quiet=True)
            print(f"✓ {resource} downloaded successfully")
        except Exception as e:
            print(f"✗ Failed to download {resource}: {str(e)}")

def download_spacy_model():
    """
    Download a spaCy language model
    We'll use the small English model for quick loading
    """
    print("\nDownloading spaCy English model...")
    try:
        subprocess.check_call([sys.executable, '-m', 'spacy', 'download', 'en_core_web_sm'])
        print("✓ spaCy model downloaded successfully")
    except subprocess.CalledProcessError:
        print("✗ Failed to download spaCy model")

def verify_installation():
    """
    Verify that all libraries are properly installed and functional
    """
    print("\n" + "="*50)
    print("VERIFICATION REPORT")
    print("="*50)
    
    # Test NLTK
    try:
        import nltk
        print(f"✓ NLTK version: {nltk.__version__}")
    except ImportError:
        print("✗ NLTK not available")
    
    # Test spaCy
    try:
        import spacy
        print(f"✓ spaCy version: {spacy.__version__}")
        nlp = spacy.load('en_core_web_sm')
        print(f"✓ spaCy model loaded successfully")
    except Exception as e:
        print(f"✗ spaCy error: {str(e)}")
    
    # Test Transformers
    try:
        import transformers
        print(f"✓ Transformers version: {transformers.__version__}")
    except ImportError:
        print("✗ Transformers not available")
    
    # Test PyTorch
    try:
        import torch
        print(f"✓ PyTorch version: {torch.__version__}")
    except ImportError:
        print("✗ PyTorch not available")

if __name__ == "__main__":
    # Uncomment these lines to perform installation
    # install_libraries()
    # download_nltk_resources()
    # download_spacy_model()
    
    # Always run verification
    verify_installation()

Output: The script will display installation progress and a verification report showing which libraries are available and their versions. You should see all checkmarks (✓) indicating successful installation.

Best Practices:

Use virtual environments (venv or conda) to isolate project dependencies
Pin specific versions in requirements.txt for reproducibility
Download only the models you need to save disk space
Keep libraries updated to benefit from bug fixes and improvements

Common Mistakes to Avoid:

Installing libraries globally instead of in virtual environments
Forgetting to download language models (NLTK data, spaCy models)
Not verifying installation before starting projects
Using incompatible versions of interdependent libraries

Example 2: Basic Text Processing with NLTK

This example demonstrates fundamental NLP operations using NLTK, including tokenization, part-of-speech tagging, and basic linguistic analysis. These operations form the building blocks of most NLP pipelines.


"""
Example 2: Fundamental Text Processing Operations with NLTK
Demonstrates tokenization, POS tagging, stemming, and lemmatization
"""

import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk import pos_tag
from collections import Counter

# Sample text for analysis
sample_text = """
Natural Language Processing is revolutionizing how computers understand human 
communication. The field has evolved dramatically over the past decades, 
transitioning from rule-based systems to neural networks. Modern NLP systems 
can translate languages, answer questions, and even generate creative content. 
Companies are investing heavily in NLP technologies to improve customer 
experience and extract insights from textual data.
"""

def tokenize_text(text):
    """
    Tokenization: Breaking text into sentences and words
    This is the first step in most NLP pipelines
    """
    print("="*60)
    print("TOKENIZATION")
    print("="*60)
    
    # Sentence tokenization
    sentences = sent_tokenize(text)
    print(f"\nNumber of sentences: {len(sentences)}")
    print("\nFirst two sentences:")
    for i, sent in enumerate(sentences[:2], 1):
        print(f"{i}. {sent.strip()}")
    
    # Word tokenization
    words = word_tokenize(text)
    print(f"\nTotal tokens: {len(words)}")
    print(f"First 15 tokens: {words[:15]}")
    
    # Filter out punctuation and get actual words
    actual_words = [word for word in words if word.isalpha()]
    print(f"Actual words (excluding punctuation): {len(actual_words)}")
    
    return sentences, words, actual_words

def analyze_pos_tags(words):
    """
    Part-of-Speech Tagging: Identifying grammatical roles of words
    Understanding POS is crucial for many NLP tasks
    """
    print("\n" + "="*60)
    print("PART-OF-SPEECH TAGGING")
    print("="*60)
    
    # Perform POS tagging
    pos_tags = pos_tag(words)
    
    # Display first 10 tagged words
    print("\nFirst 10 tagged words:")
    for word, tag in pos_tags[:10]:
        print(f"  {word:20s} -> {tag:5s} ({explain_pos_tag(tag)})")
    
    # Count POS tag frequencies
    tag_freq = Counter([tag for word, tag in pos_tags])
    print("\nMost common POS tags:")
    for tag, count in tag_freq.most_common(5):
        print(f"  {tag:5s}: {count:2d} occurrences ({explain_pos_tag(tag)})")
    
    return pos_tags

def explain_pos_tag(tag):
    """Helper function to explain POS tags"""
    pos_explanations = {
        'NN': 'Noun, singular',
        'NNS': 'Noun, plural',
        'NNP': 'Proper noun, singular',
        'VB': 'Verb, base form',
        'VBG': 'Verb, gerund/present participle',
        'VBD': 'Verb, past tense',
        'JJ': 'Adjective',
        'RB': 'Adverb',
        'DT': 'Determiner',
        'IN': 'Preposition',
        'CC': 'Coordinating conjunction'
    }
    return pos_explanations.get(tag, 'Other')

def remove_stopwords(words):
    """
    Stopword Removal: Filtering out common words with little semantic value
    Important for focusing on meaningful content
    """
    print("\n" + "="*60)
    print("STOPWORD REMOVAL")
    print("="*60)
    
    # Get English stopwords
    stop_words = set(stopwords.words('english'))
    
    # Filter words (keep only alphabetic words for clarity)
    alpha_words = [word.lower() for word in words if word.isalpha()]
    filtered_words = [word for word in alpha_words if word not in stop_words]
    
    print(f"\nOriginal word count: {len(alpha_words)}")
    print(f"After stopword removal: {len(filtered_words)}")
    print(f"Removed: {len(alpha_words) - len(filtered_words)} stopwords")
    
    print(f"\nExample stopwords removed: {list(stop_words)[:10]}")
    print(f"\nFiltered words (first 15): {filtered_words[:15]}")
    
    return filtered_words

def stem_and_lemmatize(words):
    """
    Stemming and Lemmatization: Reducing words to their base forms
    Stemming is crude but fast; lemmatization is linguistic but slower
    """
    print("\n" + "="*60)
    print("STEMMING vs LEMMATIZATION")
    print("="*60)
    
    stemmer = PorterStemmer()
    lemmatizer = WordNetLemmatizer()
    
    # Sample words to demonstrate differences
    test_words = ['running', 'runs', 'ran', 'easily', 'fairly', 
                  'processing', 'processed', 'revolutionary']
    
    print("\n{:20s} {:20s} {:20s}".format("Original", "Stemmed", "Lemmatized"))
    print("-" * 60)
    
    for word in test_words:
        stemmed = stemmer.stem(word)
        lemmatized = lemmatizer.lemmatize(word, pos='v')  # v for verb
        print(f"{word:20s} {stemmed:20s} {lemmatized:20s}")
    
    # Apply to full text
    stemmed_words = [stemmer.stem(word) for word in words[:20]]
    lemmatized_words = [lemmatizer.lemmatize(word) for word in words[:20]]
    
    print(f"\nFirst 20 stemmed words: {stemmed_words}")

def main():
    """Execute all text processing operations"""
    print("NATURAL LANGUAGE PROCESSING: BASIC TEXT OPERATIONS\n")
    
    # Tokenization
    sentences, words, actual_words = tokenize_text(sample_text)
    
    # POS Tagging
    pos_tags = analyze_pos_tags(actual_words[:30])  # Analyze first 30 words
    
    # Stopword Removal
    filtered_words = remove_stopwords(words)
    
    # Stemming and Lemmatization
    stem_and_lemmatize(filtered_words)
    
    print("\n" + "="*60)
    print("Processing complete!")
    print("="*60)

if __name__ == "__main__":
    main()

Output Explanation: This script produces a comprehensive analysis showing sentence and word counts, POS tags with explanations, the effect of stopword removal, and comparisons between stemming and lemmatization. Each section provides insights into different levels of linguistic processing.

Key Learning Points:

Tokenization is rarely perfect—punctuation, contractions, and special cases require special handling
POS tagging accuracy depends on context; the same word can have different tags
Stopword lists are language-specific and sometimes domain-specific
Stemming is faster but produces non-words; lemmatization preserves real words

Example 3: Industrial-Strength NLP with spaCy

spaCy is designed for production use with excellent performance and a clean API. This example demonstrates how spaCy provides a more integrated approach to NLP tasks compared to NLTK.


"""
Example 3: Advanced Text Processing with spaCy
Demonstrates spaCy's unified pipeline for multiple NLP tasks
"""

import spacy
from spacy import displacy
from collections import Counter

# Load spaCy's English model
# This model includes tokenizer, POS tagger, parser, NER, and word vectors
nlp = spacy.load('en_core_web_sm')

# Sample text with rich linguistic features
text = """
Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayne 
in April 1976. The company, headquartered in Cupertino, California, 
is worth over $2 trillion. In 2023, Apple released the Vision Pro, 
a revolutionary mixed-reality headset. CEO Tim Cook stated that this 
technology represents the future of spatial computing.
"""

def basic_processing(text):
    """
    Demonstrate spaCy's basic processing pipeline
    One call processes the entire text through all components
    """
    print("="*70)
    print("SPACY BASIC PROCESSING")
    print("="*70)
    
    # Process text through pipeline
    doc = nlp(text)
    
    # Display basic statistics
    print(f"\nDocument statistics:")
    print(f"  Characters: {len(doc.text)}")
    print(f"  Tokens: {len(doc)}")
    print(f"  Sentences: {len(list(doc.sents))}")
    
    # Show first sentence with linguistic annotations
    first_sent = list(doc.sents)[0]
    print(f"\nFirst sentence: {first_sent.text}")
    print("\nToken-level analysis:")
    print(f"{'Token':<15} {'Lemma':<15} {'POS':<8} {'Tag':<8} {'Dep':<10} {'Shape':<10}")
    print("-" * 75)
    
    for token in first_sent[:10]:  # First 10 tokens
        print(f"{token.text:<15} {token.lemma_:<15} {token.pos_:<8} "
              f"{token.tag_:<8} {token.dep_:<10} {token.shape_:<10}")
    
    return doc

def named_entity_recognition(doc):
    """
    Extract and categorize named entities
    NER is crucial for information extraction tasks
    """
    print("\n" + "="*70)
    print("NAMED ENTITY RECOGNITION")
    print("="*70)
    
    # Extract all entities
    if doc.ents:
        print(f"\nFound {len(doc.ents)} named entities:\n")
        print(f"{'Entity':<25} {'Label':<15} {'Explanation':<30}")
        print("-" * 70)
        
        for ent in doc.ents:
            print(f"{ent.text:<25} {ent.label_:<15} {spacy.explain(ent.label_):<30}")
    
    # Group entities by type
    entity_types = Counter([ent.label_ for ent in doc.ents])
    print(f"\nEntity distribution:")
    for ent_type, count in entity_types.most_common():
        print(f"  {ent_type}: {count}")
    
    return doc.ents

def dependency_parsing(doc):
    """
    Analyze syntactic dependencies between words
    Dependency parsing reveals sentence structure
    """
    print("\n" + "="*70)
    print("DEPENDENCY PARSING")
    print("="*70)
    
    # Analyze first sentence
    first_sent = list(doc.sents)[0]
    print(f"\nSentence: {first_sent.text}\n")
    print(f"{'Token':<15} {'Dependency':<15} {'Head':<15} {'Children':<30}")
    print("-" * 75)
    
    for token in first_sent:
        children = [child.text for child in token.children]
        print(f"{token.text:<15} {token.dep_:<15} {token.head.text:<15} "
              f"{', '.join(children):<30}")
    
    # Find root verb and its subjects/objects
    root = [token for token in first_sent if token.dep_ == 'ROOT'][0]
    subjects = [token for token in root.children if 'subj' in token.dep_]
    objects = [token for token in root.children if 'obj' in token.dep_]
    
    print(f"\nSentence structure:")
    print(f"  Root verb: {root.text}")
    if subjects:
        print(f"  Subject(s): {', '.join([s.text for s in subjects])}")
    if objects:
        print(f"  Object(s): {', '.join([o.text for o in objects])}")

def noun_phrase_extraction(doc):
    """
    Extract noun phrases (noun chunks)
    Useful for identifying key concepts in text
    """
    print("\n" + "="*70)
    print("NOUN PHRASE EXTRACTION")
    print("="*70)
    
    noun_phrases = [chunk.text for chunk in doc.noun_chunks]
    print(f"\nFound {len(noun_phrases)} noun phrases:\n")
    
    for i, chunk in enumerate(doc.noun_chunks[:10], 1):
        print(f"{i}. '{chunk.text}' - Root: '{chunk.root.text}', "
              f"Dependency: '{chunk.root.dep_}'")
    
    return noun_phrases

def word_similarity(doc):
    """
    Demonstrate word vectors and similarity
    Requires a model with word vectors (medium or large models)
    """
    print("\n" + "="*70)
    print("WORD SIMILARITY (if vectors available)")
    print("="*70)
    
    # Check if model has vectors
    if nlp.meta['vectors']['width'] > 0:
        # Compare some words
        word_pairs = [
            ('apple', 'fruit'),
            ('apple', 'company'),
            ('technology', 'computer'),
            ('CEO', 'president')
        ]
        
        print("\nWord similarities:")
        for word1, word2 in word_pairs:
            token1 = nlp(word1)[0]
            token2 = nlp(word2)[0]
            similarity = token1.similarity(token2)
            print(f"  {word1} <-> {word2}: {similarity:.3f}")
    else:
        print("\nWord vectors not available in 'sm' model.")
        print("Use 'en_core_web_md' or 'en_core_web_lg' for similarity features.")

def custom_pipeline_component(doc):
    """
    Demonstrate how to add custom processing
    This is a simple example that counts tokens by type
    """
    print("\n" + "="*70)
    print("CUSTOM ANALYSIS")
    print("="*70)
    
    token_analysis = {
        'alphabetic': len([t for t in doc if t.is_alpha]),
        'numeric': len([t for t in doc if t.like_num]),
        'punctuation': len([t for t in doc if t.is_punct]),
        'stopwords': len([t for t in doc if t.is_stop]),
        'out_of_vocabulary': len([t for t in doc if t.is_oov])
    }
    
    print("\nToken analysis:")
    for category, count in token_analysis.items():
        percentage = (count / len(doc)) * 100
        print(f"  {category.capitalize():<20}: {count:3d} ({percentage:5.1f}%)")

def main():
    """Execute all spaCy demonstrations"""
    print("SPACY: INDUSTRIAL-STRENGTH NLP\n")
    
    # Basic processing
    doc = basic_processing(text)
    
    # Named entity recognition
    named_entity_recognition(doc)
    
    # Dependency parsing
    dependency_parsing(doc)
    
    # Noun phrase extraction
    noun_phrase_extraction(doc)
    
    # Word similarity
    word_similarity(doc)
    
    # Custom analysis
    custom_pipeline_component(doc)
    
    print("\n" + "="*70)
    print("Analysis complete!")
    print("="*70)

if __name__ == "__main__":
    main()

Why spaCy? spaCy processes text in a single pipeline, making it faster and more memory-efficient than processing with multiple separate tools. It's designed for production use with excellent performance characteristics.

Best Practices:

Choose the right model size: 'sm' for speed, 'lg' for accuracy
Process texts in batches using nlp.pipe() for better performance
Disable unused pipeline components to speed up processing
Cache processed documents when analyzing the same text multiple times

Example 4: Introduction to Transformers and Modern NLP

Transformers represent the current state-of-the-art in NLP. This example introduces the Hugging Face Transformers library and demonstrates how to use pre-trained models for common tasks.


"""
Example 4: Modern NLP with Transformers
Demonstrates using pre-trained transformer models for various NLP tasks
"""

from transformers import pipeline
import warnings
warnings.filterwarnings('ignore')

def sentiment_analysis():
    """
    Perform sentiment analysis using a pre-trained transformer model
    This is one of the most common NLP applications
    """
    print("="*70)
    print("SENTIMENT ANALYSIS WITH TRANSFORMERS")
    print("="*70)
    
    # Initialize sentiment analysis pipeline
    # This automatically downloads and caches the model on first use
    sentiment_analyzer = pipeline(
        "sentiment-analysis",
        model="distilbert-base-uncased-finetuned-sst-2-english"
    )
    
    # Sample texts with varying sentiments
    texts = [
        "This product exceeded my expectations! Absolutely wonderful.",
        "The service was terrible and the staff was rude.",
        "It's okay, nothing special but not bad either.",
        "I'm absolutely thrilled with my purchase!",
        "Disappointed and frustrated with the quality."
    ]
    
    print("\nAnalyzing sentiments:\n")
    print(f"{'Text':<55} {'Sentiment':<10} {'Confidence':<10}")
    print("-" * 75)
    
    for text in texts:
        # Get prediction
        result = sentiment_analyzer(text)[0]
        label = result['label']
        score = result['score']
        
        # Truncate text for display
        display_text = text[:52] + "..." if len(text) > 52 else text
        print(f"{display_text:<55} {label:<10} {score:<10.3f}")
    
    return sentiment_analyzer

def named_entity_recognition_transformer():
    """
    Use transformers for NER - often more accurate than traditional methods
    Especially good at recognizing entities in context
    """
    print("\n" + "="*70)
    print("NAMED ENTITY RECOGNITION WITH TRANSFORMERS")
    print("="*70)
    
    # Initialize NER pipeline
    ner_pipeline = pipeline(
        "ner",
        model="dslim/bert-base-NER",
        aggregation_strategy="simple"  # Combines sub-word tokens
    )
    
    text = """
    Elon Musk, CEO of Tesla and SpaceX, announced on Twitter that the company 
    will open a new manufacturing facility in Austin, Texas. The $1.1 billion 
    investment is expected to create 5,000 jobs by 2025. Microsoft and Amazon 
    have also expressed interest in expanding their presence in the region.
    """
    
    # Perform NER
    entities = ner_pipeline(text)
    
    print("\nExtracted entities:\n")
    print(f"{'Entity':<25} {'Type':<15} {'Confidence':<12} {'Position':<15}")
    print("-" * 70)
    
    for entity in entities:
        entity_text = entity['word']
        entity_type = entity['entity_group']
        score = entity['score']
        start = entity['start']
        end = entity['end']
        
        print(f"{entity_text:<25} {entity_type:<15} {score:<12.3f} "
              f"{start:>4}-{end:<4}")
    
    return entities

def question_answering():
    """
    Demonstrate extractive question answering
    The model finds answers within provided context
    """
    print("\n" + "="*70)
    print("QUESTION ANSWERING WITH TRANSFORMERS")
    print("="*70)
    
    # Initialize QA pipeline
    qa_pipeline = pipeline(
        "question-answering",
        model="distilbert-base-cased-distilled-squad"
    )
    
    context = """
    Natural Language Processing (NLP) is a subfield of artificial intelligence 
    that focuses on the interaction between computers and human language. Modern 
    NLP systems use deep learning techniques, particularly transformer 
    architectures, which were introduced by Vaswani et al. in 2017. These models 
    have achieved remarkable results on tasks like translation, summarization, 
    and question answering. BERT, introduced by Google in 2018, was one of the 
    first transformer models to achieve state-of-the-art results across multiple 
    NLP benchmarks by using bidirectional training.
    """
    
    questions = [
        "What is NLP?",
        "When were transformers introduced?",
        "Who introduced BERT?",
        "What year was BERT introduced?",
        "What technique does modern NLP use?"
    ]
    
    print("\nAsking questions about the context:\n")
    print(f"{'Question':<45} {'Answer':<25} {'Confidence':<10}")
    print("-" * 80)
    
    for question in questions:
        result = qa_pipeline(question=question, context=context)
        answer = result['answer']
        score = result['score']
        
        # Truncate question for display
        display_q = question[:42] + "..." if len(question) > 42 else question
        print(f"{display_q:<45} {answer:<25} {score:<10.3f}")

def text_summarization():
    """
    Automatic text summarization using transformers
    Useful for condensing long documents
    """
    print("\n" + "="*70)
    print("TEXT SUMMARIZATION WITH TRANSFORMERS")
    print("="*70)
    
    # Initialize summarization pipeline
    summarizer = pipeline(
        "summarization",
        model="facebook/bart-large-cnn"
    )
    
    long_text = """
    Artificial intelligence and machine learning have transformed numerous 
    industries over the past decade. From healthcare to finance, from 
    transportation to entertainment, AI systems are now ubiquitous. In 
    healthcare, AI assists doctors in diagnosing diseases by analyzing 
    medical images with superhuman accuracy. In finance, machine learning 
    algorithms detect fraudulent transactions in real-time, protecting 
    millions of customers. Self-driving cars, powered by computer vision 
    and deep learning, promise to revolutionize transportation. Meanwhile, 
    recommendation systems used by Netflix, Spotify, and Amazon have changed 
    how we discover content and products. Natural Language Processing enables 
    virtual