Securing Secrets in Cloud-Native Applications: Beyond Environment Variables

Secrets management remains one of the most critical security challenges in cloud-native applications. Despite the importance, I’ve seen countless systems with secrets hardcoded, committed to Git, or stored in plain text environment variables. In this post, I’ll cover the spectrum of secrets management approaches, from basic practices to sophisticated solutions.

The Problem with Traditional Approaches

Let’s start with what NOT to do:

Hardcoded Secrets (Never do this):

// DON'T DO THIS
const (
    DatabasePassword = "super_secret_password"
    APIKey          = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
)

Secrets in Configuration Files (Also bad):

# config.yaml - DON'T COMMIT THIS
database:
  host: db.example.com
  username: admin
  password: my_password_123

Plain Environment Variables (Better, but insufficient):

export DATABASE_PASSWORD=my_password_123

These approaches fail because:

• Secrets are visible in code repositories and version control
• They’re exposed in process listings and logs
• No audit trail of access
• Difficult to rotate
• No encryption at rest

Kubernetes Secrets: The Baseline

Kubernetes provides a built-in Secret resource:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: production
type: Opaque
data:
  username: YWRtaW4=  # base64 encoded
  password: cGFzc3dvcmQxMjM=

Create from literal values:

kubectl create secret generic database-credentials \
  --from-literal=username=admin \
  --from-literal=password=password123 \
  --namespace=production

Consume in a pod:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
    - name: app
      image: myapp:latest
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: database-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: database-credentials
              key: password

Or mount as files:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: secrets
          mountPath: "/etc/secrets"
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: database-credentials

Limitations:

• Secrets are only base64 encoded, not encrypted in etcd by default
• Anyone with cluster access can read all secrets
• No audit trail
• Managing secrets outside of Git is inconvenient

Enabling Encryption at Rest

Kubernetes supports encrypting secrets in etcd:

# encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

Configure the API server:

# Add to kube-apiserver flags
--encryption-provider-config=/path/to/encryption-config.yaml

Re-encrypt existing secrets:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

This protects secrets at rest in etcd but doesn’t solve the management problem.

Sealed Secrets: GitOps-Friendly Encryption

Sealed Secrets allows you to commit encrypted secrets to Git:

# Install controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

# Install kubeseal CLI
brew install kubeseal

Create a sealed secret:

kubectl create secret generic mysecret \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml | \
  kubeseal -o yaml > sealed-secret.yaml

The sealed secret is safe to commit:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: production
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.../encrypted data.../

The controller decrypts it in the cluster:

git add sealed-secret.yaml
git commit -m "Add sealed secret"
git push

# Controller automatically creates the Secret
kubectl get secret mysecret -n production

Advantages:

• GitOps compatible
• Secrets encrypted with cluster-specific keys
• Audit trail through Git

Limitations:

• Key management for the sealing key
• Rotation requires re-sealing
• Secrets still in plain text in the cluster

External Secrets Operator

For integration with external secret stores:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace

Define a SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Create an ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/database/username
    - secretKey: password
      remoteRef:
        key: prod/database/password

The operator syncs secrets from AWS Secrets Manager into Kubernetes.

HashiCorp Vault Integration

Vault provides comprehensive secrets management:

# Install Vault
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault

Enable Kubernetes authentication:

vault auth enable kubernetes

vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Create a policy:

# database-policy.hcl
path "secret/data/database/*" {
  capabilities = ["read"]
}

Apply policy:

vault policy write database database-policy.hcl

Create a role:

vault write auth/kubernetes/role/myapp \
    bound_service_account_names=myapp \
    bound_service_account_namespaces=production \
    policies=database \
    ttl=24h

Using Vault Agent Injector:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "myapp"
    vault.hashicorp.com/agent-inject-secret-database: "secret/data/database/config"
    vault.hashicorp.com/agent-inject-template-database: |
      {{- with secret "secret/data/database/config" -}}
      export DB_USERNAME="{{ .Data.data.username }}"
      export DB_PASSWORD="{{ .Data.data.password }}"
      {{- end }}
spec:
  serviceAccountName: myapp
  containers:
    - name: app
      image: myapp:latest
      command: ["/bin/sh"]
      args: ["-c", "source /vault/secrets/database && /app/start"]

Vault injects secrets as files automatically.

Using Vault CSI Provider:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.vault:8200"
    roleName: "myapp"
    objects: |
      - objectName: "username"
        secretPath: "secret/data/database/config"
        secretKey: "username"
      - objectName: "password"
        secretPath: "secret/data/database/config"
        secretKey: "password"
---
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  serviceAccountName: myapp
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: secrets
          mountPath: "/mnt/secrets"
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "vault-database"

Application-Level Integration

For programmatic access:

package main

import (
    "context"
    "fmt"
    "log"

    vault "github.com/hashicorp/vault/api"
    auth "github.com/hashicorp/vault/api/auth/kubernetes"
)

func getSecrets() (map[string]interface{}, error) {
    config := vault.DefaultConfig()
    config.Address = "http://vault.vault:8200"

    client, err := vault.NewClient(config)
    if err != nil {
        return nil, fmt.Errorf("unable to initialize Vault client: %w", err)
    }

    // Kubernetes authentication
    k8sAuth, err := auth.NewKubernetesAuth(
        "myapp",
        auth.WithServiceAccountTokenPath("/var/run/secrets/kubernetes.io/serviceaccount/token"),
    )
    if err != nil {
        return nil, fmt.Errorf("unable to initialize k8s auth: %w", err)
    }

    authInfo, err := client.Auth().Login(context.Background(), k8sAuth)
    if err != nil {
        return nil, fmt.Errorf("unable to login: %w", err)
    }

    if authInfo == nil {
        return nil, fmt.Errorf("no auth info returned")
    }

    // Read secret
    secret, err := client.KVv2("secret").Get(context.Background(), "database/config")
    if err != nil {
        return nil, fmt.Errorf("unable to read secret: %w", err)
    }

    return secret.Data, nil
}

func main() {
    secrets, err := getSecrets()
    if err != nil {
        log.Fatal(err)
    }

    username := secrets["username"].(string)
    password := secrets["password"].(string)

    fmt.Printf("Connecting with user: %s\n", username)
}

Cloud Provider Secret Managers

AWS Secrets Manager:

import (
    "context"
    "encoding/json"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
)

func getAWSSecret(secretName string) (map[string]string, error) {
    cfg, err := config.LoadDefaultConfig(context.TODO())
    if err != nil {
        return nil, err
    }

    svc := secretsmanager.NewFromConfig(cfg)

    result, err := svc.GetSecretValue(context.TODO(), &secretsmanager.GetSecretValueInput{
        SecretId: aws.String(secretName),
    })
    if err != nil {
        return nil, err
    }

    var secrets map[string]string
    err = json.Unmarshal([]byte(*result.SecretString), &secrets)
    return secrets, err
}

Google Secret Manager:

import (
    secretmanager "cloud.google.com/go/secretmanager/apiv1"
    "cloud.google.com/go/secretmanager/apiv1/secretmanagerpb"
)

func getGCPSecret(projectID, secretID string) (string, error) {
    ctx := context.Background()
    client, err := secretmanager.NewClient(ctx)
    if err != nil {
        return "", err
    }
    defer client.Close()

    req := &secretmanagerpb.AccessSecretVersionRequest{
        Name: fmt.Sprintf("projects/%s/secrets/%s/versions/latest", projectID, secretID),
    }

    result, err := client.AccessSecretVersion(ctx, req)
    if err != nil {
        return "", err
    }

    return string(result.Payload.Data), nil
}

Secret Rotation

Implement automatic rotation:

# rotate-secrets.py
import boto3
import datetime

def rotate_secret(secret_name):
    client = boto3.client('secretsmanager')

    # Generate new secret
    new_password = generate_secure_password()

    # Update database
    update_database_password(new_password)

    # Update secret
    client.update_secret(
        SecretId=secret_name,
        SecretString=json.dumps({'password': new_password})
    )

    print(f"Rotated secret {secret_name}")

def lambda_handler(event, context):
    # Called by CloudWatch Events daily
    rotate_secret('prod/database/password')

Schedule rotation:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: rotate-secrets
spec:
  schedule: "0 2 * * 0"  # Weekly at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: rotate
              image: secret-rotator:latest
              env:
                - name: VAULT_ADDR
                  value: "http://vault.vault:8200"
          restartPolicy: OnFailure

Best Practices

1. Principle of Least Privilege: Grant minimal necessary access
2. Encryption Everywhere: At rest, in transit, and in use
3. Audit Everything: Log all secret access
4. Regular Rotation: Automate secret rotation
5. No Secrets in Logs: Sanitize before logging
6. Separate Secrets by Environment: Different secrets for dev/staging/prod
7. Use Short-Lived Credentials: Prefer dynamic secrets over static

Secret Scanning

Prevent secrets from entering Git:

# Install git-secrets
brew install git-secrets

# Configure
git secrets --install
git secrets --register-aws

Pre-commit hook:

#!/bin/bash
# .git/hooks/pre-commit

git secrets --pre_commit_hook -- "$@"

CI/CD scanning:

# GitHub Actions
name: Secret Scan
on: [push, pull_request]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: TruffleHog Scan
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD

Monitoring and Alerting

Monitor secret access:

apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-audit
data:
  audit.hcl: |
    audit "file" {
      file_path = "/vault/logs/audit.log"
    }

Alert on suspicious activity:

# vault-monitor.py
def analyze_audit_logs():
    suspicious_patterns = [
        'failed authentication',
        'permission denied',
        'secret not found'
    ]

    with open('/vault/logs/audit.log') as f:
        for line in f:
            entry = json.loads(line)
            if any(pattern in entry.get('error', '') for pattern in suspicious_patterns):
                send_alert(entry)

Compliance Considerations

For regulated industries:

• Audit Trail: Maintain immutable logs of all secret access
• Encryption Standards: Use FIPS 140-2 validated encryption
• Key Management: Implement proper key lifecycle management
• Access Controls: Enforce multi-factor authentication
• Data Residency: Ensure secrets stay in required regions

Conclusion

Secrets management in cloud-native applications requires a layered approach:

1. Never hardcode or commit secrets
2. Use Kubernetes Secrets as a minimum baseline
3. Enable encryption at rest in etcd
4. For GitOps, adopt Sealed Secrets or External Secrets
5. For comprehensive management, integrate Vault or cloud provider solutions
6. Implement rotation, monitoring, and auditing
7. Scan for leaked secrets continuously

The right solution depends on your requirements:

• Small teams: Sealed Secrets + External Secrets Operator
• Enterprise: HashiCorp Vault or cloud provider solutions
• Regulated industries: Vault with extensive auditing

Start with basic practices and evolve as your security requirements mature. The investment in proper secrets management pays dividends in security posture and compliance.