Containers vs VMs: Storage Networking Implications

Docker is barely 18 months old, but it’s already changing how we think about infrastructure. As containers gain traction, I’ve been analyzing what this means for storage networking. The differences between container and VM storage models are profound.

Virtual Machines: The Traditional Model

VMs virtualize entire machines:

┌───────────────────────────────────┐
│     Virtual Machine 1             │
│  ┌─────────────────────────────┐ │
│  │    Guest OS (Full Linux)    │ │
│  ├─────────────────────────────┤ │
│  │  Application + Dependencies │ │
│  └─────────────────────────────┘ │
│  ┌─────────────────────────────┐ │
│  │  Virtual Disk (100GB+)      │ │ ← Entire OS disk
│  └─────────────────────────────┘ │
└───────────────────────────────────┘
        ↓ (via FC, iSCSI, etc.)
┌───────────────────────────────────┐
│      Storage Network (FC)         │
└───────────────────────────────────┘

VM storage characteristics:

• Large (50-100GB+ per VM)
• Long-lived (VMs run for weeks/months)
• Persistent across reboots
• Pre-provisioned
• Predictable I/O patterns

FC-Redirect handles VM storage well: relatively static, large volumes, predictable patterns.

Containers: The New Model

Containers share the host OS kernel:

┌───────────────────────────────────┐
│          Host OS (Linux)          │
├───────────────────────────────────┤
│ Container 1  Container 2  Container 3
│  ┌─────┐      ┌─────┐      ┌─────┐
│  │ App │      │ App │      │ App │
│  └──┬──┘      └──┬──┘      └──┬──┘
│     │            │            │
│  ┌──▼─────────────▼────────────▼──┐
│  │     Union FS (Layers)         │ ← Shared layers
│  └──────────────┬────────────────┘
│                 │
│  ┌──────────────▼────────────────┐
│  │  Volume Mounts (Data only)    │ ← Small, data-only
│  └───────────────────────────────┘
└───────────────────────────────────┘

Container storage characteristics:

• Small (MB, not GB)
• Short-lived (containers are ephemeral)
• Layered (shared base images)
• Dynamic (created/destroyed rapidly)
• Unpredictable I/O

This is fundamentally different from VMs.

Challenge 1: Storage Lifecycle

VM Workflow

# VM storage: pre-provision, attach, long lifetime
volume = create_volume(size_gb=100)  # Takes 5-10 seconds
attach_volume_to_vm(vm_id, volume_id)  # Takes 2-3 seconds

# VM runs for weeks/months
run_vm(vm_id)

# Eventually shutdown
detach_volume(vm_id, volume_id)  # Takes 2-3 seconds
delete_volume(volume_id)  # Takes 3-5 seconds

5-10 second provisioning is fine for long-lived VMs.

Container Workflow

# Container storage: ephemeral, created on-demand
docker run -v /data postgres  # Container starts in <1 second!

# But if volume provisioning takes 5 seconds...
# Container start latency: 1s + 5s = 6s (unacceptable!)

Containers start in milliseconds. Storage must keep up.

Solution: Pre-Provisioned Storage Pools

I implemented a storage pool system for FC-Redirect:

typedef struct storage_pool {
    char name[64];
    size_t volume_size_gb;

    // Pre-provisioned volumes
    volume_t *free_volumes;
    volume_t *allocated_volumes;

    atomic_uint32_t free_count;
    atomic_uint32_t allocated_count;

    uint32_t min_free;  // Maintain this many free volumes
} storage_pool_t;

// Background thread maintains pool
void* storage_pool_maintenance_thread(void *arg) {
    storage_pool_t *pool = (storage_pool_t*)arg;

    while (running) {
        uint32_t free = atomic_load(&pool->free_count);

        if (free < pool->min_free) {
            // Pre-provision more volumes
            int to_provision = pool->min_free - free;

            for (int i = 0; i < to_provision; i++) {
                volume_t *vol = provision_volume(pool->volume_size_gb);
                add_to_free_list(pool, vol);
            }
        }

        sleep(60);  // Check every minute
    }

    return NULL;
}

// Allocate from pool (fast!)
volume_t* allocate_volume_from_pool(storage_pool_t *pool) {
    volume_t *vol = remove_from_free_list(pool);

    if (vol == NULL) {
        // Pool empty, provision on-demand (slow path)
        vol = provision_volume(pool->volume_size_gb);
    }

    add_to_allocated_list(pool, vol);
    return vol;
}

// Return to pool (fast!)
void return_volume_to_pool(storage_pool_t *pool, volume_t *vol) {
    remove_from_allocated_list(pool, vol);

    // Wipe volume
    zero_volume(vol);

    add_to_free_list(pool, vol);
}

This reduces volume allocation from 5 seconds to <100ms.

Challenge 2: Copy-on-Write Layers

Container images use layered filesystems:

Container Image Layers:
┌────────────────────────────────┐
│  Application Layer (100MB)     │ ← Container-specific
├────────────────────────────────┤
│  Runtime Layer (200MB)         │ ← Shared (Node.js, etc.)
├────────────────────────────────┤
│  Base OS Layer (500MB)         │ ← Shared (Ubuntu, etc.)
└────────────────────────────────┘

Storage: Only store unique layers, not entire images per container

Traditional block storage doesn’t support layers well.

Solution: Integration with Docker Volume Plugins

I built a Docker volume plugin for FC storage:

package main

import (
    "github.com/docker/go-plugins-helpers/volume"
)

type fcVolumeDriver struct {
    fcClient *FCRedirectClient
    volumes  map[string]*FCVolume
}

func (d *fcVolumeDriver) Create(req *volume.CreateRequest) error {
    // Allocate FC volume from pool
    fcVol, err := d.fcClient.AllocateVolume(req.Name, req.Options)
    if err != nil {
        return err
    }

    d.volumes[req.Name] = fcVol
    return nil
}

func (d *fcVolumeDriver) Mount(req *volume.MountRequest) (*volume.MountResponse, error) {
    fcVol := d.volumes[req.Name]

    // Discover and mount FC LUN
    devicePath, err := d.fcClient.DiscoverAndMount(fcVol.WWN)
    if err != nil {
        return nil, err
    }

    // Format if needed
    if !isFormatted(devicePath) {
        formatDevice(devicePath, "ext4")
    }

    // Mount
    mountPath := filepath.Join("/mnt/fc_volumes", req.Name)
    os.MkdirAll(mountPath, 0755)

    err = syscall.Mount(devicePath, mountPath, "ext4", 0, "")
    if err != nil {
        return nil, err
    }

    return &volume.MountResponse{Mountpoint: mountPath}, nil
}

func (d *fcVolumeDriver) Unmount(req *volume.UnmountRequest) error {
    mountPath := filepath.Join("/mnt/fc_volumes", req.Name)

    // Unmount
    err := syscall.Unmount(mountPath, 0)
    if err != nil {
        return err
    }

    fcVol := d.volumes[req.Name]

    // Remove FC mapping
    return d.fcClient.UnmountAndRemove(fcVol.WWN)
}

func main() {
    driver := &fcVolumeDriver{
        fcClient: NewFCRedirectClient("localhost:8080"),
        volumes:  make(map[string]*FCVolume),
    }

    handler := volume.NewHandler(driver)
    handler.ServeUnix("fc", 0)
}

Usage:

# Create volume backed by FC storage
docker volume create --driver=fc --opt size=10GB mydata

# Use in container
docker run -v mydata:/data postgres

# Volume persists across container restarts
docker stop postgres
docker start postgres  # Same data

Challenge 3: I/O Patterns

VM I/O Patterns

VM I/O characteristics:
- Large sequential reads/writes (OS boot, application startup)
- Random access for databases
- Predictable patterns (same workload runs continuously)
- High per-volume throughput (single VM uses volume exclusively)

Container I/O Patterns

Container I/O characteristics:
- Many small files (stateless applications, config files)
- Bursty (containers start/stop frequently)
- Shared volumes (multiple containers access same data)
- Low per-container throughput (many containers share resources)

FC-Redirect QoS policies designed for VMs don’t work well for containers.

Solution: Container-Aware QoS

typedef struct container_qos_policy {
    // Per-container limits (small)
    uint32_t max_iops_per_container;
    uint32_t max_bw_mbps_per_container;

    // Aggregate limits (for all containers on host)
    uint32_t max_total_iops;
    uint32_t max_total_bw_mbps;

    // Burst handling
    bool allow_bursting;
    uint32_t burst_duration_sec;
} container_qos_policy_t;

void apply_container_qos(flow_entry_t *flow,
                        container_qos_policy_t *policy) {
    // Identify container from flow
    container_id_t container = identify_container(flow);

    // Per-container limit
    flow->max_iops = policy->max_iops_per_container;
    flow->max_bw_mbps = policy->max_bw_mbps_per_container;

    // Check aggregate limit
    uint32_t total_iops = calculate_host_total_iops(flow->host_id);

    if (total_iops > policy->max_total_iops) {
        // Throttle proportionally
        float reduction = (float)policy->max_total_iops / total_iops;
        flow->max_iops *= reduction;
        flow->max_bw_mbps *= reduction;
    }

    // Allow bursting for container startup
    if (policy->allow_bursting && is_container_starting(container)) {
        flow->burst_credits = policy->max_iops_per_container *
                             policy->burst_duration_sec;
    }
}

This provides fairness across many containers while allowing startup bursts.

Challenge 4: Volume Density

VM Density

Typical server:
- 10-20 VMs per host
- 1-2 volumes per VM
- Total: 20-40 volumes per host

FC-Redirect handles 40 volumes per host easily.

Container Density

Typical server:
- 100-200 containers per host
- 0.2 volumes per container (many are stateless)
- Total: 20-40 volumes per host (similar to VMs!)

Surprisingly, container volume density is similar to VMs because most containers are stateless.

But container churn is much higher:

VM churn: 1-2 VMs created/destroyed per day
Container churn: 100+ containers created/destroyed per hour

Volume operations:
VMs: ~2 volume ops/day
Containers: ~20 volume ops/hour (240/day) = 120x more operations

FC-Redirect’s volume operation rate needed significant scaling.

Solution: Optimized Volume Operations

// Batch volume operations
typedef struct volume_operation_batch {
    enum {
        OP_CREATE,
        OP_DELETE,
        OP_ATTACH,
        OP_DETACH
    } operations[BATCH_SIZE];

    volume_id_t volume_ids[BATCH_SIZE];
    int count;
} volume_operation_batch_t;

// Process batch in single transaction
void process_volume_operations_batch(volume_operation_batch_t *batch) {
    // Begin transaction
    begin_storage_transaction();

    for (int i = 0; i < batch->count; i++) {
        switch (batch->operations[i]) {
        case OP_CREATE:
            create_volume_in_transaction(batch->volume_ids[i]);
            break;
        case OP_DELETE:
            delete_volume_in_transaction(batch->volume_ids[i]);
            break;
        // ... other operations
        }
    }

    // Commit once (amortized cost)
    commit_storage_transaction();
}

Batching improved volume operation throughput by 10x.

Challenge 5: Networking

Containers use overlay networks (e.g., Flannel, Weave) that complicate storage access.

Traditional Model

VM → Physical NIC → FC HBA → FC Fabric → Storage
(Direct path, simple)

Container Model

Container → veth → bridge → physical NIC → FC HBA → FC Fabric → Storage
           (Container network encapsulation)

FC-Redirect must track containers across overlay networks.

Solution: CNI Integration

// Container Network Interface plugin integration
func setupNetwork(containerID string, netns string) error {
    // Get container's network namespace
    ns, err := netns.GetFromPath(netns)
    if err != nil {
        return err
    }

    // Get container's FC initiator WWPN
    containerWWPN := getContainerWWPN(containerID)

    // Set up FC-Redirect policy
    err = fcClient.CreatePolicy(&FCPolicy{
        InitiatorWWPN: containerWWPN,
        TargetWWPN:    storageWWPN,
        QoS:           "container",
        Namespace:     containerID,
    })

    return err
}

Real-World Results

Testing with container workloads:

Volume Operations:

• Create: <100ms (was 5s) - 50x faster
• Attach: <50ms (was 3s) - 60x faster
• Detach: <30ms (was 2s) - 66x faster

Scalability:

• Container starts/sec: 50 (was 5) - 10x improvement
• Concurrent volumes: 1000 per host (was 100)
• Operation queue depth: 10000 ops/sec sustained

Compatibility:

• Docker: Full support
• Kubernetes: Via FlexVolume driver
• Mesos: Via DVdi plugin
• Swarm: Native support

Container vs VM Storage: Summary

Aspect	VMs	Containers
Size	50-100GB	10-100MB
Lifetime	Weeks/months	Minutes/hours
Provisioning	Pre-provision	On-demand
I/O Pattern	Predictable	Bursty
Density	10-20/host	100-200/host
Operations	2/day	240/day
Storage	Block volumes	Volumes + layers

Lessons Learned

Container storage taught me:

1. Speed matters: Container startup latency is measured in milliseconds. Storage can’t take seconds.

2. Ephemerality changes everything: Design for create/destroy, not long-lived allocation.

3. Density requires efficiency: High container density means efficient resource usage is critical.

4. Layers are powerful: Copy-on-write layers dramatically reduce storage consumption.

5. Stateless is the goal: Most containers should be stateless. Design storage for the stateful minority.

Looking Forward

Container adoption is accelerating. Storage networking must adapt:

• Faster provisioning (milliseconds, not seconds)
• Higher operation rates (1000s of ops/sec)
• Better integration (Docker, Kubernetes, etc.)
• Smarter QoS (container-aware policies)

The future is hybrid: VMs for stateful workloads, containers for stateless. FC-Redirect supports both.

Containers aren’t replacing VMs; they’re complementing them. Smart infrastructure supports both seamlessly.

The storage industry is evolving from “VMs only” to “VMs and containers.” Those who adapt will thrive.