As 2012 comes to a close, I want to reflect on what I’ve learned over the past two years working on FC-Redirect at Cisco. It’s been an incredible learning experience, and I want to share the key lessons that have shaped my understanding of storage networking.
Technical Lessons
Lesson 1: Simplicity is Powerful
The most elegant solutions are often the simplest. I’ve seen complex designs that solved problems but were unmaintainable. I’ve also seen simple designs that solved 80% of the problem with 20% of the complexity.
In FC-Redirect, we constantly balance sophistication with simplicity. Yes, we can implement incredibly complex migration algorithms, but do they provide enough value to justify the complexity? Often, a simpler approach that’s easier to understand, test, and troubleshoot is better.
Takeaway: Before adding complexity, ask whether simpler approaches might be sufficient.
Lesson 2: Protocol Details Matter Enormously
Working on storage virtualization has taught me that protocol details matter far more than I initially appreciated. The difference between a frame being 64 bytes vs. 68 bytes seems trivial, but at line rate, those 4 bytes compound into significant overhead.
Understanding protocols at the packet/frame level—not just conceptually—is essential for building high-performance systems. You need to know not just what the protocol does, but exactly how it does it.
Takeaway: Deep protocol knowledge separates adequate implementations from excellent ones.
Lesson 3: Performance is About Eliminating Bottlenecks
Performance optimization is less about making fast things faster and more about identifying and eliminating bottlenecks. You can optimize the hot path all day, but if the bottleneck is elsewhere, performance won’t improve.
Profiling and measurement are essential. Assumptions about what’s slow are often wrong. Measure first, optimize second.
Takeaway: Measure to find bottlenecks before optimizing. Optimize the bottleneck, then measure again.
Lesson 4: Reliability Requires Paranoia
Building reliable systems requires assuming everything will fail. Hardware fails. Software has bugs. Networks partition. Operators make mistakes.
The systems that survive are those designed with this paranoia. Redundancy, error checking, graceful degradation, and comprehensive logging aren’t optional—they’re essential.
Takeaway: Design for failure from the beginning. It’s much harder to add reliability later.
Lesson 5: Abstractions Enable Innovation
Storage virtualization creates an abstraction layer between hosts and physical storage. This abstraction is powerful because it enables innovation at either layer independently.
We can add features in the virtualization layer without changing hosts or arrays. We can migrate between different array types transparently. Abstractions provide flexibility and longevity.
Takeaway: Well-designed abstractions are force multipliers for future innovation.
Operational Lessons
Lesson 6: Operations Matter as Much as Technology
The best technology poorly operated performs worse than mediocre technology operated well. Documentation, procedures, training, monitoring—these operational aspects determine success or failure.
I’ve seen customers with expensive, capable storage struggling because operations were neglected. I’ve seen customers with modest storage succeeding because they operated it excellently.
Takeaway: Invest in operations as much as technology. The return is substantial.
Lesson 7: Monitoring Prevents Problems
Comprehensive monitoring catches problems before they become critical. Without monitoring, you discover problems when users complain—too late.
Good monitoring provides early warning. Disk failures, performance degradation, capacity approaching limits—all detectable with proper monitoring.
Takeaway: Monitor everything. Set thresholds. Respond to alerts. Proactive beats reactive.
Lesson 8: Testing Isn’t Optional
Untested code doesn’t work. Untested configurations don’t work. Untested failover procedures don’t work. Testing seems expensive until you compare it to production failures.
Test extensively before production. Test failure scenarios. Test at scale. Test under load. Testing finds problems when they’re cheap to fix rather than expensive.
Takeaway: Budget time for testing. It’s not optional for production systems.
Lesson 9: Documentation is Future You’s Best Friend
When I look at code or configurations I wrote months ago, good comments and documentation are invaluable. What seemed obvious when writing is mysterious when reading later.
Document why, not just what. The code shows what; comments should explain why you made particular choices.
Takeaway: Document for your future self and others. You’ll be grateful later.
Lesson 10: Automation Reduces Errors
Humans make mistakes, especially under pressure or fatigue. Automation executes consistently. Manual processes don’t scale; automated processes do.
The time invested in automation pays back many times over. Scripts, tools, and automated testing improve reliability and efficiency.
Takeaway: Automate repetitive tasks. Free humans for tasks requiring judgment.
Customer Lessons
Lesson 11: Requirements Drive Architecture
Different customers have vastly different requirements. Some need absolute maximum performance. Others prioritize cost. Some need five-nines availability. Others can tolerate occasional downtime.
There’s no one-size-fits-all architecture. Understanding specific requirements is essential for designing appropriate solutions.
Takeaway: Start with requirements. Design follows requirements, not the other way around.
Lesson 12: Migration is Critical
Most customers aren’t building greenfield. They’re migrating from existing infrastructure. Migration capabilities—moving data without downtime—are critical.
Non-disruptive migration is often more valuable than any single feature. It enables evolution without disruption.
Takeaway: Design for migration from the start. Customers need evolution paths.
Lesson 13: TCO Matters More Than Purchase Price
Customers increasingly focus on Total Cost of Ownership over upfront purchase price. Power, cooling, management, software licensing—all factor into TCO.
A cheaper array that consumes more power and requires more management may cost more over its lifetime than a more expensive but efficient array.
Takeaway: Optimize for TCO, not just initial cost. The long-term view matters.
Lesson 14: Compatibility is Valued
Customers have heterogeneous environments. They value solutions that work with what they have, not just the latest and greatest.
Backward compatibility, support for legacy protocols, and interoperability with competitors’ gear are important. Customers can’t forklift their entire infrastructure.
Takeaway: Design for heterogeneous environments. Customers have existing investments.
Industry Lessons
Lesson 15: The Industry Moves Faster Than You Think
Two years ago, flash was exotic. Now it’s mainstream. Two years ago, FCoE was the future. Now it’s one option among many.
The pace of change is accelerating. Technologies and approaches become mainstream faster than traditional adoption curves suggested.
Takeaway: Continuous learning is essential. What’s cutting-edge today is standard tomorrow.
Lesson 16: Multiple Paradigms Coexist
FC isn’t dead despite predictions. iSCSI is thriving. FCoE has its niche. NAS remains important. Object storage is growing.
Multiple protocols and architectures coexist because they solve different problems well. Diversity is strength, not weakness.
Takeaway: Learn multiple technologies. The future is heterogeneous, not homogeneous.
Lesson 17: Software Matters More Than Hardware
The industry trend is clear: more innovation in software, less differentiation in hardware. Commodity hardware with sophisticated software beats proprietary hardware with basic software.
Software-defined everything reflects this trend. Hardware becomes fungible; software is the differentiator.
Takeaway: Software skills become increasingly valuable. Understand software deeply.
Lesson 18: Cloud Changes Everything
Cloud isn’t just about technology—it’s about business model, operations, and expectations. Self-service, automation, elasticity, pay-per-use—these expectations are reshaping even on-premises infrastructure.
Customers want on-premises infrastructure that works like cloud. This requires rethinking traditional approaches.
Takeaway: Understand cloud concepts even if you work on-premises. They’re influencing everything.
Personal Lessons
Lesson 19: Deep Expertise is Valuable
In an age of generalists, deep expertise in specific areas remains valuable. Understanding storage networking at a deep level—protocol details, performance characteristics, failure modes—enables solving problems others can’t.
Breadth is valuable, but depth in critical areas is perhaps more valuable.
Takeaway: Develop deep expertise in chosen areas. Surface knowledge isn’t sufficient.
Lesson 20: Teaching Solidifies Learning
Writing these blog posts has deepened my own understanding. Explaining concepts forces clarifying thinking. Questions from readers reveal gaps in my knowledge.
Teaching is one of the best ways to learn. When you can explain something clearly, you truly understand it.
Takeaway: Share knowledge. Teaching benefits the teacher as much as the student.
Lesson 21: Fundamentals Endure
Amid all the change—flash, cloud, software-defined—fundamentals endure. Physics doesn’t change. Latency, bandwidth, and capacity trade-offs remain. CAP theorem applies regardless of technology.
Understanding fundamentals provides a stable foundation as technologies change.
Takeaway: Invest in understanding fundamentals. They outlast specific technologies.
Lesson 22: Collaboration Amplifies Results
The best work happens through collaboration. Different perspectives, skills, and experiences combined produce better results than any individual could achieve alone.
Working with talented colleagues at Cisco has taught me far more than I could have learned alone.
Takeaway: Seek out collaboration. Diverse teams produce better solutions.
Looking Forward
As I look to 2013 and beyond, these lessons guide my approach:
Technically: Focus on fundamentals, measure before optimizing, design for failure, embrace simplicity.
Operationally: Automate, monitor comprehensively, test thoroughly, document well.
Professionally: Develop deep expertise, learn continuously, share knowledge, collaborate.
The storage networking field is evolving rapidly, but these principles remain valid.
Gratitude
I’m grateful for the opportunity to work on FC-Redirect at Cisco. It’s been an incredible learning experience. Working with talented engineers, engaging with customers, and tackling complex technical challenges has been professionally fulfilling.
I’m grateful for readers who have engaged with these blog posts. Your questions and feedback have helped me think more clearly about these topics.
What’s Next
2013 will bring new challenges and opportunities. Flash adoption will accelerate. Cloud integration will deepen. Automation will become more critical. The industry will continue evolving rapidly.
I’m excited to continue learning and sharing. Storage networking is at an inflection point, and the next few years will be transformative.
Thank you for following along on this journey. I hope these posts have been valuable. I’ll continue sharing insights and lessons as I learn them.
Conclusion
Two years at Cisco working on storage networking has taught me enormous amounts—technically, operationally, and professionally. The lessons I’ve shared reflect hard-won understanding from both successes and failures.
Storage networking is a fascinating field sitting at the intersection of hardware, software, networking, and systems architecture. It’s challenging but rewarding. The problems are hard, but solving them makes a real difference.
As we close 2012, I’m optimistic about the future of storage networking. The technologies we’re building today will enable applications and capabilities we can barely imagine.
Here’s to continuous learning, thoughtful engineering, and building systems that reliably store and serve the world’s data.
Thank you for reading, and I wish you success in your own storage networking endeavors.