Distributed File Systems: How HDFS Works

MapReduce spreads computation over a cluster, but the data has to live somewhere, and no single machine holds petabytes. A distributed file system stores huge files across many ordinary servers and survives their failures by replication. The classic example is HDFS (Hadoop Distributed File System), modeled on Google's GFS; today cloud object stores such as S3 often play the same role.

Blocks and replicas

HDFS splits each file into large fixed-size blocks (typically 128 MB) and stores each block on several machines (3 replicas by default), so a dead disk loses nothing.

Why such big blocks? Less metadata: a petabyte is only about 8 million 128 MB blocks to track, not trillions of tiny entries. Amortized seeks: reading 128 MB sequentially is efficient, while thousands of tiny files mean thousands of seeks. (The flip side: HDFS handles huge numbers of small files badly.)

Two kinds of nodes

• DataNodes store the blocks. Every commodity server runs a DataNode that manages its local disks and serves block reads and writes over the network; thousands of them add up to petabytes.
• The NameNode keeps the metadata: which files exist, which blocks make up each file, and which DataNodes hold each block's replicas. It keeps all of it in memory, so it can handle tens of thousands of metadata operations per second.

Separating metadata from data is what makes HDFS scale. The NameNode only answers "where are the blocks?"; the bytes flow directly between clients and DataNodes. Adding DataNodes adds storage and throughput without loading the NameNode.

Reading a file

The NameNode is never on the data path: one lightweight RPC at the start, then terabytes can flow between clients and DataNodes.

Writing a file: pipeline replication

1. The client asks the NameNode to create the file; the NameNode allocates block IDs and picks DataNodes for the replicas.
2. The client streams data to the first DataNode only.
3. That DataNode forwards to the second, which forwards to the third: a pipeline.
4. Each DataNode acknowledges after writing to its disk, and acks flow back along the pipeline.
5. When all blocks are done, the client tells the NameNode the file is complete.

Failures and recovery

• DataNode failures are routine (disks die, machines crash, networks partition). DataNodes send periodic heartbeats; when they stop, the NameNode marks the node dead and tells other DataNodes to copy its blocks elsewhere until every block is back to its replication factor.
• NameNode failure is serious: on its own it's a single point of failure. Production clusters run a standby NameNode that shares the edit log (usually through a quorum of JournalNodes) and takes over within seconds to minutes. See health checks and failover.
• Rack awareness: replicas are placed on different racks, so losing a whole rack (switch or power failure) never loses every copy of a block.

Why it matters for MapReduce: data locality

Because the NameNode knows where every block lives, the scheduler runs each map task on a node that already holds its input block: the computation moves to the data instead of terabytes moving across the network. Large blocks support this too: 128 MB keeps a mapper busy long enough to make scheduling worthwhile. See MapReduce.