How Apache Spark Works: Driver, Executors, Partitions and Shuffles

Apache Spark is a distributed engine for processing large datasets in parallel. A driver program turns your code into a plan of stages and tasks, and executors on a cluster run those tasks over partitions of the data. Understanding how Spark splits work, and especially when it must shuffle data across the network, explains almost every Spark performance problem you'll be asked about.

Why MapReduce became limiting

MapReduce made distributed batch processing possible, but writing it felt like assembly code. A 10-stage pipeline is 10 separate jobs, with 10 writes and 10 reads of intermediate results to HDFS, even when the whole dataset would fit in the cluster's memory.

• Rigid structure: everything must be map → shuffle → reduce. A graph algorithm that updates node values until they converge needs a whole job per iteration; "iterate until converged" can't be one job.
• Disk overhead between stages: map output goes to local disk, is shuffled, and reduce output goes to HDFS. Multi-stage pipelines pay this tax repeatedly, and for small intermediate data it dominates runtime.
• Poor iteration support: gradient descent may need hundreds of passes over training data; reading and writing HDFS every pass makes iterative ML impractical.

How Spark changed the model

• Flexible DAG execution: a job is a directed acyclic graph of operations (filter, map, join, aggregate) that the optimizer arranges: combining steps into stages, pushing filters early, choosing join strategies.
• In-memory caching: intermediate data can stay in memory across stages, so a 10-stage pipeline reads input once and writes output once. When data doesn't fit, Spark spills to disk instead of failing.
• A richer vocabulary: joins, aggregations, window functions and SQL out of the box. One engine covers batch ETL, interactive SQL, machine learning and graph processing.

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("top-urls").getOrCreate()

logs = spark.read.text("hdfs:///data/logs/access.log")            # or s3a://bucket/logs/
urls = logs.select(F.split("value", " ").getItem(6).alias("url"))  # 7th field, like awk '{print $7}'

top = (urls.groupBy("url").count()                                 # one shuffle for groupBy
           .orderBy(F.desc("count"))
           .limit(10))

top.write.mode("overwrite").parquet("hdfs:///reports/top_urls")    # read once, write once

Iteration is where the gap is dramatic. PageRank with 20 iterations in MapReduce is 20 jobs and 40 HDFS reads and writes. Spark loads the graph once, caches it, and updates scores in memory each iteration (it still writes shuffle files to local disk each round, and recomputes cached partitions if an executor is lost). Spark's original paper focused on iterative machine learning, where speedups of roughly 10–100× are possible, depending on data size, memory and iteration count.

• Use Spark for: multi-stage ETL pipelines, machine learning on historical data, interactive queries, and SQL analytics via Spark SQL.
• It still uses distributed storage: Spark reads and writes HDFS or object storage; what changes is how computation runs between those reads and writes.
• Trade-offs: caching needs enough cluster memory (otherwise data spills and the advantage shrinks), the optimizer sometimes needs manual tuning, and for a simple one-pass transformation the gain over MapReduce can be negligible.

Driver, executors and the cluster manager

• The driver holds the logical plan, asks the cluster manager for executors, and assigns tasks. If it fails, the job fails, so never collect() a huge result to it.
• Executors are processes with a fixed number of cores and memory. Each core runs one task at a time, and each task processes one partition.
• A partition is a chunk of the dataset. Parallelism equals the number of partitions in a stage, capped by total executor cores.

Lazy evaluation, stages and shuffles

Transformations (filter, select, join, groupBy) build a plan without running anything. Only an action (writing a table, count, collect) triggers execution. Spark's optimizer then rewrites the plan: it pushes filters down to the scan, prunes unused columns, and picks join strategies.

A shuffle writes intermediate data to disk on every executor, sends it over the network grouped by key, and reads it back. It's usually the most expensive part of a job, so most tuning is about having fewer, smaller and evenly balanced shuffles.

Join strategies

Data skew

If one join or grouping key has far more rows than others (a default "unknown" customer, one giant merchant), one task gets most of the data. The stage then waits for that single straggler while every other executor sits idle, or the task runs out of memory. Fixes:

• Adaptive query execution (AQE): Spark 3 can detect skewed partitions at runtime and split them, and can also coalesce many tiny partitions after a shuffle.
• Broadcast the smaller side if it fits, removing the shuffle entirely.
• Salting: append a random suffix to the hot key on the big side, replicate the matching rows on the small side for each suffix, join, then aggregate back.
• Handle the hot key separately: filter it out, process it on its own path, and union the results.
• Fix it upstream: nulls or placeholder keys are often data-quality bugs that shouldn't be joined at all.

Skew is the batch version of hot partitions in databases, and the fixes are similar.

Tuning checklist

1. Read less: store data in columnar formats, partition by common filters, select only needed columns.
2. Shuffle less: filter and aggregate early, broadcast small dimensions, avoid unnecessary repartition.
3. Size partitions sensibly, typically around 100-200 MB each after a shuffle, rather than thousands of tiny ones or a few huge ones.
4. Watch the Spark UI: look for stages with one long task (skew), heavy spill to disk (too little memory per task), or many tiny tasks.
5. Avoid Python UDFs on hot paths when built-in functions exist; they force data out of Spark's optimised format.
6. Write output with a sensible number of files, and compact small files in table formats.

Spark Structured Streaming runs the same engine in micro-batches over a stream. The same rules about shuffles and skew apply, plus the time semantics in windows and watermarks.