We’ve been transitioning our CipherTrust platform from a monolithic architecture to microservices. This isn’t just about following trends - there are real benefits for a key management system that needs to scale globally, deploy updates frequently, and maintain high availability. But microservices also introduce complexity, especially for security-critical systems. Let me share what we’re learning.
Why Microservices for Key Management?
Our monolithic key management system worked well initially, but we hit several limitations:
Scaling bottlenecks: Different components had different scaling requirements. Policy evaluation needed to scale horizontally for high request volumes, while key lifecycle operations needed stronger security guarantees but lower throughput. A monolith forces everything to scale together.
Deployment friction: Updating any component required deploying the entire application. This made us conservative about releases, batching changes into infrequent large deployments. Each deployment was risky because so much changed at once.
Technology lock-in: The monolith was built in Java. When we wanted to use Go for high-performance components, we couldn’t easily integrate it.
Team coordination: As the team grew, multiple engineers working on the same codebase caused merge conflicts and coordination overhead.
Microservices address these challenges. Each service can be scaled independently, deployed independently, and built with the best technology for that specific problem.
Service Decomposition Strategy
Decomposing a monolith isn’t straightforward. How do you decide where service boundaries should be? We followed several principles:
Business capability alignment: Each service should represent a distinct business capability. “Key lifecycle management” is a service. “Policy evaluation” is a service. “Audit logging” is a service.
Data ownership: Each service owns its data. The policy service owns policy data. The audit service owns audit logs. Services don’t directly access each other’s databases.
Independent scalability: Service boundaries should align with scaling requirements. Components that need to scale differently should be separate services.
Team ownership: Each service should be small enough for one team to own completely.
We identified these core services:
Key Lifecycle Service: Manages key generation, rotation, and destruction. Communicates with HSMs to perform cryptographic operations.
Policy Service: Evaluates access policies to determine if operations should be allowed. Needs to scale for high request volumes.
Audit Service: Collects, stores, and queries audit logs from all services.
Authentication Service: Handles user and service authentication, issues tokens.
HSM Proxy Service: Abstracts HSM integration, providing a consistent API regardless of HSM vendor.
Monitoring Service: Collects health and performance metrics from all services and HSMs.
Inter-Service Communication
Microservices need to communicate. We evaluated several patterns:
REST over HTTP: Simple and widely understood. Each service exposes REST APIs that others can call. This is what we started with and still use for most synchronous communication.
Message queues: For asynchronous communication where immediate response isn’t needed. We use RabbitMQ for audit log shipping and background tasks.
gRPC: Google’s RPC framework. Better performance than REST, with strong typing through protocol buffers. We’re evaluating this for high-throughput service-to-service communication.
The challenge with inter-service communication is handling failures. When the policy service calls the audit service to log a decision, what happens if the audit service is down?
We implement several patterns:
Circuit breakers: If a service is consistently failing, stop calling it temporarily and fail fast rather than waiting for timeouts.
Retries with exponential backoff: Transient failures can be retried, but with increasing delays to avoid overwhelming a recovering service.
Graceful degradation: Some operations can proceed even if dependent services are unavailable. Policy evaluation can continue if audit logging temporarily fails (we queue the logs and send them when audit service recovers).
Service Discovery
In a microservices architecture, services need to find each other. When the key lifecycle service needs to call the policy service, what address does it use?
Hardcoding addresses doesn’t work - services move between hosts, scale up and down, get redeployed. We need dynamic service discovery.
We’re using Kubernetes services for this (as I wrote about last month). Each microservice is fronted by a Kubernetes Service that provides a stable DNS name and load balances across pod instances.
Services simply use DNS names like “policy-service.production.svc.cluster.local” and Kubernetes handles routing to healthy instances.
API Gateway Pattern
We don’t expose individual microservices directly to external clients. Instead, we use an API gateway that:
Routes requests to appropriate backend services Authenticates requests and validates tokens Rate limits to prevent abuse Aggregates responses from multiple services when needed Provides a stable API contract even as backend services evolve
This gives us flexibility to refactor backend services without breaking client applications. The API gateway presents a logical API, and we can change how it’s implemented by backend services.
Data Management in Microservices
In a monolith, you have one database and can use ACID transactions. Microservices change this - each service has its own database, and you can’t easily perform transactions across services.
This is one of the hardest aspects of microservices. Consider this scenario: A user requests key deletion. The key lifecycle service needs to delete the key from the HSM, the policy service needs to delete policies associated with that key, and the audit service needs to log the deletion.
In a monolith, this would be one transaction. With microservices, it’s multiple operations across services. What happens if the key is deleted but policy deletion fails?
We use several patterns:
Saga pattern: Break distributed transactions into a sequence of local transactions, each with a compensating transaction that can undo it if later steps fail.
Event sourcing: Store events (key created, key deleted) rather than current state. Services can rebuild state from event logs and stay synchronized.
Eventual consistency: Accept that different services might have slightly different views of data, as long as they eventually converge.
For critical operations like key deletion, we use the saga pattern with explicit compensation. If any step fails, we execute compensating transactions to rollback the operation.
Security in Microservices
Microservices increase the attack surface. Instead of one application to secure, you have many services, each with network endpoints.
Service-to-service authentication: We use mutual TLS for service-to-service communication. Each service has a certificate that identifies it, and services validate each other’s certificates before accepting requests.
Token-based authorization: Requests include JWT tokens that specify the caller’s identity and permissions. Each service validates tokens before processing requests.
Defense in depth: Even though services are behind an API gateway, each service implements its own authentication and authorization. We don’t trust the network.
Secrets management: Each service needs credentials for its database, HSM, etc. We use Kubernetes secrets with encryption at rest and strict RBAC for who can access secrets.
Minimal trust: Services should trust each other minimally. Even if one service is compromised, it shouldn’t grant access to sensitive operations in other services.
Monitoring and Observability
With a monolith, monitoring is straightforward - watch one application. With microservices, you need to monitor many services and understand their interactions.
Distributed tracing: When a request flows through multiple services, how do you trace it? We’re using correlation IDs that are passed with each request, allowing us to correlate logs across services.
Centralized logging: All services ship logs to Elasticsearch. We can query across all services to understand what happened during a request.
Metrics aggregation: Each service exposes metrics (request count, latency, error rate). We aggregate these in Prometheus for visualization and alerting.
Health checks: Each service exposes a health endpoint that reports its status and dependency health. This feeds into Kubernetes liveness/readiness probes.
Deployment Strategy
Microservices enable independent deployment, but coordination is still needed. We use:
CI/CD pipelines: Each service has its own pipeline that builds, tests, and deploys independently.
Blue-green deployment: We deploy new versions alongside old ones, gradually shifting traffic to the new version. If issues arise, we can quickly roll back.
Canary releases: New versions are deployed to a small percentage of traffic first. If metrics look good, we gradually increase traffic to the new version.
Backward compatibility: Services must maintain API compatibility. Breaking changes require versioning and migration strategies.
Performance Considerations
Microservices introduce latency. A request that used to be one function call in the monolith might now be several network calls between services.
We optimize by:
Caching: Aggressively cache data that doesn’t change frequently. Policy decisions, for example, can be cached for short periods.
Batching: Instead of making multiple individual calls, batch requests when possible.
Async communication: Use message queues for operations that don’t need immediate responses.
Service colocation: For services that communicate frequently, consider deploying them on the same hosts to reduce network latency.
Versioning and Compatibility
With independent deployment comes the challenge of version compatibility. Service A might be deployed at version 2 while Service B still depends on version 1 APIs.
We handle this through:
Semantic versioning: Clear versioning that indicates whether changes are backward compatible.
API versioning: We include version in API URLs (/v1/keys, /v2/keys) to support multiple versions simultaneously.
Contract testing: Automated tests verify that services still satisfy the contracts their clients expect.
Graceful degradation: Services should handle missing or unexpected fields gracefully rather than failing.
Testing Microservices
Testing is more complex with microservices:
Unit tests: Test individual service logic in isolation.
Integration tests: Test how services interact, using test doubles for dependencies.
Contract tests: Verify that services satisfy the contracts their clients expect.
End-to-end tests: Test complete workflows across multiple services in a staging environment.
Chaos testing: Randomly kill services to verify the system degrades gracefully under failures.
We maintain all these test levels, with the testing pyramid guiding us: many unit tests, fewer integration tests, even fewer end-to-end tests.
Lessons Learned
After six months of microservices in production:
Start small: Don’t decompose everything at once. Start with a few services and learn before going all-in.
Invest in platform tooling: Microservices require good logging, monitoring, deployment automation. Build these before you have many services.
Conway’s Law is real: Service boundaries should align with team boundaries. Coordinating across teams for a single service is painful.
Distributed systems are hard: You’ll encounter new failure modes. Plan for partial failures, network issues, and eventual consistency.
Documentation matters: With many services, documentation about what each service does and how to use it is essential.
Looking Forward
We’re continuing to refine our microservices architecture. Next steps include:
- Service mesh for better traffic management and observability
- More sophisticated deployment strategies
- Better tooling for local development with many services
- Event-driven patterns for better service decoupling
Microservices aren’t a silver bullet. They introduce complexity that isn’t warranted for all systems. But for our global, multi-tenant key management platform, the benefits outweigh the costs.
Key Takeaways
For teams considering microservices:
- Have a clear reason for microservices beyond “everyone else is doing it”
- Invest in observability before you have many services
- Design for failure - network calls will fail, services will be unavailable
- Start with a few services and learn before going all-in
- Maintain backward compatibility to enable independent deployment
- Security is harder with many services - plan for service-to-service auth
- Accept eventual consistency where appropriate - not everything needs ACID transactions
Microservices change how you build, deploy, and operate software. The learning curve is real, but for complex systems that need to scale and evolve rapidly, they provide compelling benefits. Just make sure you’re ready for the operational complexity that comes with distributed systems.