One of the most impactful optimizations I implemented while scaling FC-Redirect was message batching. It’s a technique that sounds simple in theory but requires careful consideration of trade-offs between latency, throughput, and consistency. Let me share what I learned building a production-grade batching system.

The Network Overhead Problem

When I first analyzed our FC-Redirect network traffic patterns, I was shocked. For every flow state change, we were sending an individual message to each peer node in the cluster. With 12,000 flows and changes happening hundreds of times per second, we were generating over 50,000 messages per second across our fabric.

The overhead wasn’t just the message payloads themselves. Each message required:

  • TCP/IP packet headers (40+ bytes)
  • Application protocol framing (32 bytes)
  • Acknowledgment handling
  • Context switching overhead
  • Network stack processing on both ends

For a typical 64-byte flow state update, we were paying nearly 100 bytes of overhead. We were essentially spending more resources on packaging than on the actual content.

Designing the Batching System

The goal was to aggregate multiple updates into single messages without introducing unacceptable latency or complexity. Here’s the architecture I designed:

Time-Based Batching Windows

I implemented a sliding window approach where updates are collected for a configurable time period (typically 50ms) before being sent. The key is choosing the right window size:

  • Too small: Minimal batching benefit, high message rate
  • Too large: Unacceptable latency for time-sensitive updates
  • Just right: Maximum efficiency while meeting latency SLAs

After extensive testing across different workload patterns, 50ms emerged as the sweet spot for our use case. It was short enough that applications never noticed the delay, but long enough to capture significant batching opportunities.

Priority Lanes

Not all messages are created equal. Some flow state changes are critical and need immediate propagation, while others can tolerate batching. I implemented a priority system with three lanes:

  1. Immediate: Critical updates sent instantly (flow failures, security events)
  2. High: Batched with 10ms maximum delay (active flow state changes)
  3. Normal: Batched with 50ms maximum delay (statistics, metadata)
typedef enum {
    MSG_PRIORITY_IMMEDIATE = 0,
    MSG_PRIORITY_HIGH = 1,
    MSG_PRIORITY_NORMAL = 2
} msg_priority_t;

void enqueue_update(flow_update_t *update, msg_priority_t priority) {
    batch_queue_t *queue = &batch_queues[priority];

    pthread_mutex_lock(&queue->lock);
    append_to_batch(queue, update);

    if (priority == MSG_PRIORITY_IMMEDIATE ||
        queue->batch_size >= MAX_BATCH_SIZE) {
        flush_batch(queue);
    }
    pthread_mutex_unlock(&queue->lock);
}

Intelligent Compression

Within each batch, I apply compression to remove redundancy:

  • Deduplication: If the same flow has multiple updates, only send the latest
  • Delta encoding: Send only what changed, not the entire state
  • Run-length encoding: Compress repeated values in bulk updates

This compression is incredibly effective. In typical workloads, we see 3-4x compression ratios before even applying general-purpose compression algorithms.

Implementation Challenges

Building this system revealed several subtle challenges:

Ordering Guarantees

When batching updates from multiple threads, maintaining causality is tricky. If thread A updates flow X, then thread B updates flow Y which depends on X’s new state, the batch must preserve this ordering.

I solved this with vector clocks and dependency tracking. Each update carries a logical timestamp, and the batching system ensures causal ordering is maintained even when updates are reordered for compression efficiency.

Memory Management

Batching requires buffering data, which means careful memory management. I implemented a pool-based allocator specifically for batch buffers:

typedef struct batch_buffer_pool {
    batch_buffer_t *free_list;
    size_t buffer_size;
    size_t num_buffers;
    pthread_mutex_t lock;
} batch_buffer_pool_t;

batch_buffer_t* allocate_batch_buffer(batch_buffer_pool_t *pool) {
    pthread_mutex_lock(&pool->lock);

    batch_buffer_t *buffer = pool->free_list;
    if (buffer) {
        pool->free_list = buffer->next;
    } else {
        buffer = malloc(sizeof(batch_buffer_t) + pool->buffer_size);
    }

    pthread_mutex_unlock(&pool->lock);
    return buffer;
}

This eliminates allocation overhead and fragmentation, keeping memory usage predictable even under high load.

Adaptive Window Sizing

Static window sizes work well for steady-state workloads, but real-world traffic is bursty. I added adaptive window sizing that adjusts based on observed traffic patterns:

  • High traffic: Reduce window size to 25ms for better responsiveness
  • Low traffic: Increase to 100ms for maximum batching efficiency
  • Burst detection: Temporarily disable batching when detecting coordinated updates

Results and Impact

The batching system delivered impressive improvements:

  • Network messages: Reduced from 50,000/sec to 8,000/sec (84% reduction)
  • Bandwidth: Decreased by 78% due to batching and compression
  • CPU utilization: 25% reduction in network stack overhead
  • Latency: P99 latency improved by 15ms (fewer packets = less queuing)
  • Scalability: Enabled smooth scaling to 12K flows without network saturation

Perhaps most importantly, the system has been rock-solid in production. The adaptive mechanisms handle workload variations gracefully, and we’ve had zero consistency issues related to batching.

Design Principles

Several principles emerged from this work:

  1. Measure the overhead: In distributed systems, packaging and transport often cost more than the payload. Optimize accordingly.

  2. Not everything should batch: Have an escape hatch for truly latency-sensitive operations.

  3. Adaptivity beats static tuning: Real-world workloads vary too much for one-size-fits-all approaches.

  4. Correctness first: Batching must never compromise consistency or ordering guarantees. Performance is meaningless if the system produces wrong results.

Broader Applications

While I developed this for FC-Redirect, the patterns apply broadly to any distributed system with high message rates:

  • Database replication
  • Distributed caching systems
  • Event streaming platforms
  • Microservice communication
  • Telemetry and monitoring systems

If you’re sending thousands of small messages between nodes, batching should be one of your first optimization targets. The implementation complexity is modest, but the performance gains can be transformational.

As we continue pushing FC-Redirect to higher scales throughout 2013, intelligent message batching remains one of our most valuable optimizations. It’s a perfect example of how stepping back and questioning basic assumptions can lead to breakthrough improvements.