Processing Big Data

How do you count the frequency of every word in 1 petabyte of text files? A single computer would take years. MapReduce (published by Google in 2004) solves this by splitting the task across thousands of machines that process data in parallel.

The core insight is simple: if you can express your problem as a series of independent transformations (Map) followed by an aggregation (Reduce), you can parallelize it trivially.

The Data Flow Model

MapReduce is a 3-step pipeline that transforms a massive dataset into a summary:

flowchart LR
    Input[Splits] --> Map
    Map --> Shuffle
    Shuffle --> Reduce
    Reduce --> Output
    
    subgraph Mappers
    Map1[Map: Count Words]
    Map2[Map: Count Words]
    end
    
    subgraph Reducers
    Red1[Reduce: Sum 'Apple']
    Red2[Reduce: Sum 'Banana']
    end

1. The Map Step

The master node splits the input (terabytes of text) into disjoint chunks, typically 64MB-128MB each. Each chunk is assigned to a worker node ("Mapper").

• Logic: Each Mapper parses its chunk independently and emits Key-Value pairs.
• Example: map(text) -> list(word, 1)
• Parallelism: If you have 10,000 chunks and 1,000 Mapper nodes, each node processes ~10 chunks sequentially. The work is embarrassingly parallel.

# Map function: Emit (word, 1) for each word
def map_function(document_id: str, text: str):
    results = []
    for word in text.lower().split():
        # Remove punctuation
        clean_word = word.strip(".,!?;:'\"")
        if clean_word:
            results.append((clean_word, 1))
    return results

# Input: ("doc1", "Hello World Hello")
# Output: [("hello", 1), ("world", 1), ("hello", 1)]

2. The Shuffle Step (The Magic)

This is the most network-intensive part. The framework sorts and groups the output of all mappers by Key, sending all values for the same key to the same Reducer.

• Goal: Ensure all values for "apple" end up on the same Reducer, regardless of which Mapper produced them.
• Mechanism: hash(key) % num_reducers determines which Reducer receives each key.
• Result: { "apple": [1, 1, 1], "banana": [1], "cherry": [1, 1] }

3. The Reduce Step

The grouped data is passed to Reducers, which aggregate the values for each unique key.

• Logic: reduce(key, values) -> aggregated_result
• Output: { "apple": 3, "banana": 1, "cherry": 2 }

# Reduce function: Sum all counts for a word
def reduce_function(word: str, counts: list[int]) -> tuple:
    return (word, sum(counts))

# Input: ("apple", [1, 1, 1])
# Output: ("apple", 3)

Optimization: The Combiner (Mini-Reducer)

Sending millions of ("the", 1) pairs over the network kills performance. A Combiner is a local "mini-reducer" that runs on the Mapper node before the Shuffle step.

• Without Combiner: Mapper sends ("the", 1) 1,000 times → 1,000 network transfers.
• With Combiner: Mapper aggregates locally and sends ("the", 1000) once → 1 network transfer.

The Combiner function is typically identical to the Reduce function, but it only processes local data. This works when the reduce operation is commutative and associative (e.g., sum, max, min).

Fault Tolerance

MapReduce was designed to run on commodity hardware where failures are common:

1. Mapper failure: The master detects the failure (via heartbeats) and re-assigns the chunk to another Mapper. Since the input is on distributed storage (GFS/HDFS), any node can read the chunk.
2. Reducer failure: The master re-schedules the Reduce task. The Shuffle data is re-fetched from Mappers (or their local disks).
3. Stragglers: If one Mapper is abnormally slow (due to a dying disk or noisy neighbor), the master launches a speculative copy on another node. Whichever finishes first wins.

Real-World Use Cases

1. Inverted Index (Google Search)

How Google Search originally worked:

• Map Input: (DocID, "the cat sat on the mat")
• Map Output: ("the", DocID), ("cat", DocID), ("sat", DocID)
• Reduce Input: ("cat", [Doc1, Doc5, Doc9, Doc42])
• Reduce Output: A mapping from every word to every document containing it.
• Result: When you search "cat", Google instantly retrieves [Doc1, Doc5, Doc9, Doc42].

2. Log Analysis

Companies with thousands of servers generate terabytes of logs daily:

• Map: Parse each log line → emit (error_type, 1) or (url, response_time).
• Reduce: Count errors by type, compute average response time per URL.

3. Training Machine Learning Models

• Map: Compute gradients for a batch of training examples.
• Reduce: Average the gradients and update model weights.
• Used in distributed training frameworks before modern SGD implementations.

4. ETL (Extract, Transform, Load) Pipelines

• Moving data from operational databases to data warehouses.
• Map: Transform raw rows (clean, normalize, enrich).
• Reduce: Aggregate into summary tables.

Hadoop: The Open-Source Implementation

Apache Hadoop brought Google's MapReduce to the open-source world:

• HDFS (Hadoop Distributed File System): Stores input/output data across the cluster.
• YARN: Manages cluster resources (CPU, RAM allocation for Map/Reduce tasks).
• MapReduce Engine: Executes the Map and Reduce tasks.

While powerful, Hadoop MapReduce writes intermediate data to disk between steps, making iterative algorithms (like machine learning) extremely slow.

Apache Spark: The Modern Successor

Modern systems often use Apache Spark, which performs MapReduce-style operations in memory, avoiding expensive disk I/O.

Spark introduced Resilient Distributed Datasets (RDDs) — immutable, partitioned collections that can be cached in RAM across the cluster:

# Spark: Word Count in 3 lines
from pyspark import SparkContext

sc = SparkContext("local", "WordCount")
text = sc.textFile("hdfs:///data/wikipedia/")

# MapReduce in Spark
counts = (text
    .flatMap(lambda line: line.split())     # Map
    .map(lambda word: (word, 1))            # Map
    .reduceByKey(lambda a, b: a + b))       # Reduce

counts.saveAsTextFile("hdfs:///output/word_counts")

Spark vs. Hadoop MapReduce:

• Spark is 10-100x faster for iterative workloads (in-memory).
• Hadoop MapReduce is better for massive batch jobs where memory is limited.

Interview Tips 💡

1. Start with the problem: "Processing petabytes of data on a single machine is infeasible. MapReduce parallelizes this across thousands of nodes."
2. Explain the three phases clearly: "Map transforms data into key-value pairs. Shuffle groups by key across the network. Reduce aggregates."
3. Mention the Combiner: "A Combiner reduces network traffic by aggregating locally on each Mapper before the Shuffle."
4. Discuss limitations: "MapReduce writes to disk between stages, making it slow for iterative algorithms. Spark solves this with in-memory computing."
5. Give a concrete example: Inverted index for search, or word count.

Related Concepts

• Distributed File Systems — HDFS stores the data MapReduce processes
• Apache Kafka Architecture — Event streaming as input to MapReduce
• Top-K Heavy Hitters — A MapReduce-compatible algorithm
• Bloom Filters — Used in MapReduce to skip irrelevant data