While I work on traditional SAN technology at Cisco, I’m fascinated by distributed storage systems being pioneered by companies like Google, Amazon, and Facebook. These systems take radically different approaches to storage, and there are valuable lessons for all of us.
The Scale Problem
Google’s scale is mind-boggling. They’re indexing billions of web pages, serving billions of search queries, and storing exabytes of data. Traditional storage approaches simply don’t work at this scale.
You can’t build a SAN with millions of disks. You can’t afford enterprise-grade storage arrays for web crawl data. You need fundamentally different architecture, which led Google to create the Google File System (GFS).
GFS: A Different Philosophy
GFS was designed with several assumptions that differ from traditional storage:
Failure is Normal: With thousands of servers, something is always failing. The system must handle failures gracefully without human intervention.
Large Files: GFS is optimized for multi-gigabyte files, not millions of small files.
Append-Heavy Workload: Most writes are appends, not random overwrites.
Co-design with Applications: Applications are designed to work with GFS characteristics, not the other way around.
These assumptions led to a design quite different from traditional file systems.
GFS Architecture
GFS consists of:
Master Server: Maintains metadata—which files exist, where chunks are located. The master is a single point of coordination but not a data bottleneck.
Chunk Servers: Store actual data in fixed-size chunks (64 MB). Data is replicated across multiple chunk servers.
Clients: GFS library linked into applications. Clients communicate directly with chunk servers for data, only contacting the master for metadata.
This separation of metadata and data paths is key to scalability. The master handles metadata operations, while data flows directly between clients and chunk servers.
Replication for Reliability
In GFS, every chunk is replicated to multiple chunk servers (typically 3). When a chunk server fails, the master detects it and re-replicates affected chunks to maintain the replication count.
This provides reliability through redundancy without expensive RAID arrays. You use commodity servers with commodity disks and accept that they fail frequently. The software handles failures.
This is very different from enterprise storage where we use RAID, redundant controllers, and high-quality components to prevent failures. GFS assumes failures and works around them.
Consistency Model
GFS provides relaxed consistency guarantees:
Concurrent Writes: Multiple clients can write to the same file concurrently. GFS guarantees atomicity of individual append operations but not ordering across clients.
Stale Reads: Clients might read stale data if they’re talking to a replica that hasn’t been updated yet.
This relaxed consistency enables better performance and scalability. Applications that can tolerate it benefit from the trade-off.
Compare this to FC SANs where we provide strong consistency guarantees. Every read sees the most recent write. This requires coordination that limits scalability.
BigTable: Structured Storage
For structured data, Google built BigTable on top of GFS. BigTable provides:
Sparse, Distributed, Persistent Multi-dimensional Map: Data is organized by row key, column key, and timestamp.
Scalability: Tables can be massive—billions of rows, millions of columns.
Compression: Data is compressed transparently.
Locality: Related data is stored together for efficient access.
BigTable powers many Google services: Gmail, Google Maps, Google Earth. It demonstrates that you can build sophisticated storage abstractions on top of simple distributed file systems.
Hadoop: Open Source Inspiration
The Hadoop project brings Google’s ideas to the open source world. Hadoop Distributed File System (HDFS) is inspired by GFS, and HBase is inspired by BigTable.
This makes distributed storage accessible to organizations beyond Google-scale companies. Hadoop is being adopted for big data analytics across many industries.
The Hadoop ecosystem is growing rapidly. Technologies like Hive and Pig provide higher-level interfaces to Hadoop data. This is democratizing big data processing.
MapReduce and Storage
Google’s MapReduce framework is designed to work with GFS. MapReduce jobs process massive datasets stored in GFS by:
Map Phase: Breaking the problem into parallel tasks that process chunks of data.
Reduce Phase: Aggregating results from map tasks.
The key insight is moving computation to where data is stored rather than moving data to computation. With massive datasets, data movement is the bottleneck.
Traditional storage architectures assume computation happens elsewhere and data moves over the network. Distributed storage systems co-locate storage and compute.
Lessons for Enterprise Storage
What can we learn from distributed storage systems?
Embrace Failure: Design for failure from the start. Automatic recovery is more scalable than manual intervention.
Separate Metadata and Data: Don’t put metadata operations on the data path. This enables independent scaling.
Application Co-design: Storage systems work best when applications understand and work with their characteristics.
Commodity Hardware: Software intelligence can make unreliable hardware reliable. You don’t always need expensive specialized hardware.
Scale-Out, Not Scale-Up: Add more nodes rather than bigger nodes. This provides better economics and fault tolerance.
These principles are gradually influencing enterprise storage. We’re seeing more scale-out NAS systems, more software-defined storage, and more use of commodity components.
CAP Theorem Implications
The CAP theorem states you can have at most two of: Consistency, Availability, and Partition tolerance.
Traditional enterprise storage chooses Consistency and Availability but can’t handle Partitions well. Distributed storage systems choose Availability and Partition tolerance but relax Consistency.
Neither is wrong—they’re optimizing for different requirements. Understanding these trade-offs helps choose the right storage for each use case.
Amazon Dynamo
Amazon’s Dynamo is another influential distributed storage system. It provides:
Always Writable: Writes always succeed, even during failures. This maximizes availability.
Eventual Consistency: Replicas eventually converge but might temporarily diverge.
Partitioning: Data is automatically distributed across nodes based on consistent hashing.
Tunable Consistency: Applications can choose consistency vs. availability trade-offs per operation.
Dynamo powers critical Amazon services like the shopping cart. Its design influenced many NoSQL databases like Cassandra.
Facebook’s Haystack
Facebook faced a unique problem: storing and serving billions of photos efficiently. Traditional file systems don’t work well with billions of small files.
Haystack’s approach:
Log-Structured Store: Photos are appended to large files (haystack store files). This reduces metadata overhead.
In-Memory Index: Locations of photos are kept in memory for fast lookup.
Immutable Photos: Photos are never modified, simplifying many aspects of the design.
This demonstrates domain-specific storage optimization. Understanding your specific workload enables specialized designs that outperform general-purpose storage.
Object Storage Evolution
Distributed storage systems have influenced the object storage market. Products like OpenStack Swift and Ceph provide:
Scale-Out Architecture: Add nodes to increase capacity and performance.
No Single Point of Failure: Fully distributed design with no master node.
Self-Healing: Automatic detection and recovery from failures.
Multi-Protocol Support: Object, block, and file access to the same underlying storage.
These systems bring distributed storage concepts to enterprise environments.
Performance Characteristics
Distributed storage systems have different performance characteristics than traditional storage:
Higher Latency: Network hops and coordination overhead add latency. Operations might take milliseconds rather than microseconds.
Massive Throughput: Parallelism across many nodes provides enormous aggregate throughput.
Variable Performance: Shared infrastructure and failure recovery can cause performance variability.
For workloads that can batch operations and tolerate latency, distributed storage provides excellent throughput. For latency-sensitive workloads, traditional storage may be better.
Operational Considerations
Operating distributed storage requires different skills than traditional storage:
Automation is Essential: Manual intervention doesn’t scale. Everything must be automated.
Monitoring and Analytics: With hundreds or thousands of nodes, you need sophisticated monitoring to understand system behavior.
Capacity Planning: Distributed systems can scale incrementally, but you still need to plan ahead.
Failure Handling: Understand normal failure modes and how the system recovers. Know when to intervene.
The operational model is more like managing a fleet of servers than managing a single storage array.
When Distributed Storage Makes Sense
Distributed storage excels for:
Massive Scale: When you need petabytes or exabytes of storage.
Unstructured Data: Object storage for photos, videos, documents, logs.
Batch Analytics: Processing large datasets with tools like Hadoop.
High Availability: When availability is more important than consistency.
Cost Sensitivity: When using commodity hardware is important.
For structured transactional data, low-latency requirements, or strong consistency needs, traditional storage often makes more sense.
The Hybrid Future
I believe the future is hybrid—using distributed storage where it excels and traditional storage where it excels, with tools to move data between them as needed.
We’re already seeing this. Enterprises use traditional SANs for databases and distributed storage for archives and analytics. Cloud providers offer both block storage (like EBS) and object storage (like S3).
The key is understanding the characteristics and trade-offs of each approach and matching them to workload requirements.
Conclusion
Distributed storage systems represent a fundamental rethinking of storage architecture. By embracing failure, relaxing consistency, and leveraging commodity hardware, they achieve scale and economics impossible with traditional approaches.
While I work on traditional FC storage at Cisco, studying distributed systems has deepened my understanding of storage architecture. The principles—separation of concerns, handling failure gracefully, understanding trade-offs—apply broadly.
Not every organization needs Google-scale storage, but the ideas pioneered by these systems are influencing all storage technology. Understanding distributed storage principles makes you a better storage architect, regardless of which technologies you work with daily.
The storage world is big enough for multiple paradigms. The future belongs to those who understand when to use each approach and how to integrate them effectively.