After years of writing systems-level code in C and C++, our team embarked on a journey to rewrite critical infrastructure components in Rust. This wasn’t a decision made lightly—it came after careful evaluation of the performance, safety, and maintainability tradeoffs. This post shares our experience, patterns, and lessons learned from this transition.
Why Rust?
The decision to adopt Rust was driven by three primary factors:
1. Memory Safety Without Garbage Collection
// This won't compile - Rust prevents use-after-free at compile time
fn dangling_reference() -> &String {
let s = String::from("hello");
&s // Error: `s` does not live long enough
}
// Correct: return owned value
fn valid_string() -> String {
let s = String::from("hello");
s // Ownership transferred to caller
}In C/C++, this category of bugs has caused countless security vulnerabilities and production incidents. Rust’s ownership system eliminates them at compile time, with zero runtime overhead.
2. Concurrency Without Data Races
use std::sync::Arc;
use std::thread;
fn safe_concurrency() {
let data = Arc::new(vec![1, 2, 3, 4, 5]);
let mut handles = vec![];
for i in 0..3 {
let data_clone = Arc::clone(&data);
let handle = thread::spawn(move || {
let sum: i32 = data_clone.iter().sum();
println!("Thread {}: sum = {}", i, sum);
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
}
// This won't compile - can't have mutable aliasing
fn data_race_prevented() {
let mut data = vec![1, 2, 3];
let reference1 = &mut data;
let reference2 = &mut data; // Error: cannot borrow as mutable more than once
reference1.push(4);
reference2.push(5);
}The compiler enforces that shared data is immutable, and mutable data is exclusive. This eliminates entire classes of concurrency bugs.
3. Performance Comparable to C/C++
Rust’s zero-cost abstractions mean high-level code compiles to the same assembly as hand-optimized C:
// High-level iterator code
fn sum_even(numbers: &[i32]) -> i32 {
numbers.iter()
.filter(|&&n| n % 2 == 0)
.sum()
}
// Compiles to tight loop, equivalent to:
// int sum = 0;
// for (int i = 0; i < len; i++) {
// if (numbers[i] % 2 == 0) sum += numbers[i];
// }Real-World Use Case: High-Performance Packet Processing
Our first major Rust project was rewriting a packet processing pipeline that needed to handle millions of packets per second with sub-microsecond latency.
The C++ Baseline
// Original C++ code (simplified)
class PacketProcessor {
std::vector<uint8_t*> packet_buffers;
std::mutex buffer_lock;
public:
void process_packet(const uint8_t* data, size_t len) {
std::lock_guard<std::mutex> lock(buffer_lock);
// Parse packet
auto* header = reinterpret_cast<const PacketHeader*>(data);
// Process based on protocol
if (header->protocol == PROTOCOL_TCP) {
process_tcp(data + sizeof(PacketHeader), len - sizeof(PacketHeader));
}
// Potential issues:
// - No bounds checking on reinterpret_cast
// - Mutex contention under high load
// - Manual memory management prone to leaks
}
};The Rust Rewrite
use std::sync::Arc;
use crossbeam::channel::{bounded, Sender, Receiver};
use bytes::Bytes;
#[repr(C, packed)]
struct PacketHeader {
protocol: u8,
flags: u8,
length: u16,
}
enum Protocol {
Tcp,
Udp,
Unknown(u8),
}
impl From<u8> for Protocol {
fn from(value: u8) -> Self {
match value {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => Protocol::Unknown(value),
}
}
}
struct PacketProcessor {
workers: Vec<std::thread::JoinHandle<()>>,
sender: Sender<Bytes>,
}
impl PacketProcessor {
fn new(num_workers: usize) -> Self {
let (sender, receiver) = bounded(10000);
let receiver = Arc::new(receiver);
let workers: Vec<_> = (0..num_workers)
.map(|id| {
let receiver = Arc::clone(&receiver);
std::thread::spawn(move || {
Self::worker_loop(id, receiver);
})
})
.collect();
PacketProcessor { workers, sender }
}
fn process_packet(&self, data: Bytes) -> Result<(), String> {
self.sender.send(data)
.map_err(|_| "Channel full".to_string())
}
fn worker_loop(id: usize, receiver: Arc<Receiver<Bytes>>) {
loop {
match receiver.recv() {
Ok(packet) => {
if let Err(e) = Self::handle_packet(&packet) {
eprintln!("Worker {}: Error processing packet: {}", id, e);
}
}
Err(_) => break, // Channel closed
}
}
}
fn handle_packet(data: &[u8]) -> Result<(), String> {
// Safe parsing with bounds checking
if data.len() < std::mem::size_of::<PacketHeader>() {
return Err("Packet too small".to_string());
}
// Safe transmutation
let header = unsafe {
&*(data.as_ptr() as *const PacketHeader)
};
let protocol = Protocol::from(header.protocol);
let payload_offset = std::mem::size_of::<PacketHeader>();
if data.len() < payload_offset {
return Err("Invalid packet length".to_string());
}
let payload = &data[payload_offset..];
match protocol {
Protocol::Tcp => Self::process_tcp(payload),
Protocol::Udp => Self::process_udp(payload),
Protocol::Unknown(p) => {
Err(format!("Unknown protocol: {}", p))
}
}
}
fn process_tcp(payload: &[u8]) -> Result<(), String> {
// TCP-specific processing
// All bounds are checked, no buffer overflows possible
Ok(())
}
fn process_udp(payload: &[u8]) -> Result<(), String> {
// UDP-specific processing
Ok(())
}
}Performance Improvements
The Rust version delivered:
- 30% higher throughput: Lock-free channels instead of mutexes
- 50% lower latency variance: Predictable performance without GC pauses
- Zero memory safety issues: Caught 7 potential buffer overflows during compilation
Pattern: Zero-Copy Parsing
One of Rust’s strengths is zero-copy parsing of binary protocols:
use bytes::{Buf, BytesMut};
struct HttpRequest<'a> {
method: &'a str,
path: &'a str,
headers: Vec<(&'a str, &'a str)>,
body: &'a [u8],
}
impl<'a> HttpRequest<'a> {
fn parse(input: &'a [u8]) -> Result<Self, String> {
let mut lines = input.split(|&b| b == b'\n');
// Parse request line
let request_line = lines.next()
.ok_or("Empty request")?;
let mut parts = request_line.split(|&b| b == b' ');
let method = std::str::from_utf8(
parts.next().ok_or("Missing method")?
).map_err(|_| "Invalid UTF-8")?;
let path = std::str::from_utf8(
parts.next().ok_or("Missing path")?
).map_err(|_| "Invalid UTF-8")?;
// Parse headers (zero-copy)
let mut headers = Vec::new();
for line in lines.by_ref() {
if line.is_empty() || line == b"\r" {
break;
}
if let Some(pos) = line.iter().position(|&b| b == b':') {
let name = std::str::from_utf8(&line[..pos])
.map_err(|_| "Invalid header name")?;
let value = std::str::from_utf8(&line[pos+1..].trim_ascii())
.map_err(|_| "Invalid header value")?;
headers.push((name, value.trim()));
}
}
// Body is everything remaining
let body_start = input.len() - lines.as_slice().len();
let body = &input[body_start..];
Ok(HttpRequest {
method,
path,
headers,
body,
})
}
}
// No allocations, all references into original buffer
let request_data = b"GET /api/users HTTP/1.1\r\nHost: example.com\r\n\r\n";
let request = HttpRequest::parse(request_data).unwrap();Async Rust for High-Concurrency Services
For I/O-bound services, Rust’s async/await enables high concurrency without thread overhead:
use tokio::net::{TcpListener, TcpStream};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use std::sync::Arc;
struct Server {
listener: TcpListener,
handler: Arc<dyn RequestHandler + Send + Sync>,
}
#[async_trait::async_trait]
trait RequestHandler {
async fn handle(&self, request: &[u8]) -> Vec<u8>;
}
impl Server {
async fn new(addr: &str, handler: Arc<dyn RequestHandler + Send + Sync>) -> tokio::io::Result<Self> {
let listener = TcpListener::bind(addr).await?;
Ok(Server { listener, handler })
}
async fn run(&self) -> tokio::io::Result<()> {
loop {
let (socket, addr) = self.listener.accept().await?;
let handler = Arc::clone(&self.handler);
// Spawn task for each connection (cheap: ~2KB stack)
tokio::spawn(async move {
if let Err(e) = Self::handle_connection(socket, handler).await {
eprintln!("Error handling connection from {}: {}", addr, e);
}
});
}
}
async fn handle_connection(
mut socket: TcpStream,
handler: Arc<dyn RequestHandler + Send + Sync>,
) -> tokio::io::Result<()> {
let mut buffer = vec![0u8; 8192];
loop {
let n = socket.read(&mut buffer).await?;
if n == 0 {
break; // Connection closed
}
let response = handler.handle(&buffer[..n]).await;
socket.write_all(&response).await?;
}
Ok(())
}
}
// Example handler
struct EchoHandler;
#[async_trait::async_trait]
impl RequestHandler for EchoHandler {
async fn handle(&self, request: &[u8]) -> Vec<u8> {
request.to_vec()
}
}
// Usage: handles 100K+ concurrent connections on single thread
#[tokio::main]
async fn main() -> tokio::io::Result<()> {
let handler = Arc::new(EchoHandler);
let server = Server::new("0.0.0.0:8080", handler).await?;
server.run().await
}Interoperability with Existing C/C++ Code
Migration doesn’t require a big bang rewrite. Rust has excellent C FFI:
// Rust code exposed to C
#[no_mangle]
pub extern "C" fn process_data(
data: *const u8,
len: usize,
output: *mut u8,
output_len: usize,
) -> i32 {
// Convert raw pointers to safe Rust slices
let input = unsafe {
if data.is_null() {
return -1;
}
std::slice::from_raw_parts(data, len)
};
let output_slice = unsafe {
if output.is_null() {
return -1;
}
std::slice::from_raw_parts_mut(output, output_len)
};
// Safe Rust code
match process_safely(input, output_slice) {
Ok(bytes_written) => bytes_written as i32,
Err(_) => -1,
}
}
fn process_safely(input: &[u8], output: &mut [u8]) -> Result<usize, String> {
// Actual processing logic in safe Rust
if output.len() < input.len() {
return Err("Output buffer too small".to_string());
}
output[..input.len()].copy_from_slice(input);
Ok(input.len())
}// C code calling Rust
#include <stdint.h>
#include <stdio.h>
extern int32_t process_data(
const uint8_t* data,
size_t len,
uint8_t* output,
size_t output_len
);
int main() {
uint8_t input[] = {1, 2, 3, 4, 5};
uint8_t output[10];
int32_t result = process_data(input, 5, output, 10);
if (result >= 0) {
printf("Processed %d bytes\n", result);
}
return 0;
}Error Handling: Result Type vs Exceptions
Rust’s Result type forces explicit error handling:
use std::fs::File;
use std::io::{self, Read};
// Errors must be handled explicitly
fn read_config(path: &str) -> Result<String, io::Error> {
let mut file = File::open(path)?; // ? propagates errors
let mut contents = String::new();
file.read_to_string(&mut contents)?;
Ok(contents)
}
// Caller must handle errors
fn main() {
match read_config("config.toml") {
Ok(config) => println!("Config: {}", config),
Err(e) => eprintln!("Failed to read config: {}", e),
}
}
// Can't forget error handling - won't compile
fn bad_example() {
let file = File::open("config.toml"); // Error: unused Result
}Compare to C++ exceptions which can be silently ignored:
// C++: Easy to forget error handling
std::string read_config(const std::string& path) {
std::ifstream file(path); // Exception might be thrown, might not be caught
std::stringstream buffer;
buffer << file.rdbuf();
return buffer.str();
}Testing and Benchmarking
Rust’s built-in testing framework makes it easy to maintain quality:
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_packet_parsing() {
let data = vec![
6, 0, 0, 20, // TCP protocol, length 20
/* payload */
];
let result = PacketProcessor::handle_packet(&data);
assert!(result.is_ok());
}
#[test]
fn test_invalid_packet() {
let data = vec![6, 0]; // Too short
let result = PacketProcessor::handle_packet(&data);
assert!(result.is_err());
}
}
// Benchmarking with criterion
#[cfg(test)]
mod benches {
use criterion::{black_box, criterion_group, criterion_main, Criterion};
use super::*;
fn benchmark_packet_processing(c: &mut Criterion) {
let packet = vec![6, 0, 0, 100, /* 100 bytes payload */];
c.bench_function("process_packet", |b| {
b.iter(|| {
PacketProcessor::handle_packet(black_box(&packet))
})
});
}
criterion_group!(benches, benchmark_packet_processing);
criterion_main!(benches);
}Lessons Learned
What Worked Well
- Gradual Migration: Started with new components, then rewrote hot paths
- Team Training: Invested 2 weeks in Rust workshops before production code
- Cargo Ecosystem: Excellent package management and build system
- Compile-Time Guarantees: Caught bugs that would have been production incidents
Challenges
- Learning Curve: Borrow checker takes time to internalize (2-4 weeks)
- Compile Times: Larger projects can have long compile times
- Async Ecosystem: Still maturing, some rough edges
- Library Gaps: Some niche libraries not yet available in Rust
Best Practices
// 1. Use type system to encode invariants
struct NonEmptyVec<T> {
head: T,
tail: Vec<T>,
}
impl<T> NonEmptyVec<T> {
fn new(head: T) -> Self {
NonEmptyVec { head, tail: Vec::new() }
}
fn first(&self) -> &T {
&self.head // Always safe, guaranteed non-empty
}
}
// 2. Prefer iterators over manual indexing
fn process_items(items: &[Item]) {
// Instead of:
// for i in 0..items.len() { items[i].process(); }
// Use iterators:
items.iter().for_each(|item| item.process());
}
// 3. Use Cow for flexible ownership
use std::borrow::Cow;
fn process_string(input: Cow<str>) -> Cow<str> {
if input.contains("special") {
Cow::Owned(input.replace("special", "SPECIAL"))
} else {
input // No allocation if no changes needed
}
}Conclusion
Migrating to Rust has been one of the best technical decisions we’ve made. The combination of memory safety, performance, and excellent tooling has significantly improved our code quality and developer productivity.
When to consider Rust:
- Systems programming requiring high performance
- Safety-critical code where bugs are expensive
- High-concurrency services
- Long-lived infrastructure that needs maintainability
When to stick with other languages:
- Rapid prototyping where safety isn’t critical
- Teams without time for learning curve
- Domains with mature ecosystems in other languages
Rust isn’t a silver bullet, but for systems programming, it’s the best tool we’ve found for building reliable, performant software.