Skip to main content

Lab 5.4 - Secret Rotation

Objectives​

By the end of this lab, you will be able to:

  • Understand the secret rotation process
  • Create new secrets
  • Update deployments with new secrets
  • Delete old secrets
  • Implement rotation without downtime

Prerequisites​

  • Functional Kubernetes cluster
  • kubectl configured
  • Understanding of Secrets and Deployments

Exercise 1: Basic Rotation​

Step 1: Create the Initial Secret​

kubectl create secret generic db-secret-v1 \
  --from-literal=password=old-password-123

Step 2: Create a Deployment​

Create deployment-v1.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: app
        image: busybox:1.35
        command: ["sh", "-c", "echo 'Password: $DB_PASSWORD' && sleep 3600"]
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret-v1
              key: password

Step 3: Deploy​

kubectl apply -f deployment-v1.yaml
kubectl get pods -l app=myapp

Exercise 2: Create a New Secret​

Create Secret v2​

kubectl create secret generic db-secret-v2 \
  --from-literal=password=new-password-456

Verify​

kubectl get secrets
kubectl describe secret db-secret-v2

Exercise 3: Update the Deployment​

Update the Reference​

kubectl set env deployment/app-deployment \
  DB_PASSWORD=$(kubectl get secret db-secret-v2 -o jsonpath='{.data.password}' | base64 -d)

OR modify the YAML file:

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-secret-v2  # Changed from v1 to v2
      key: password

Then:

kubectl apply -f deployment-v1.yaml

Verify the Restart​

kubectl get pods -l app=myapp -w
kubectl rollout status deployment/app-deployment

Exercise 4: Rotation with Rolling Update​

Create a Deployment with Rolling Update​

Modify the deployment to use the new secret and force a rolling update:

# Method 1: Modify the referenced secret
kubectl patch deployment app-deployment -p '{"spec":{"template":{"spec":{"containers":[{"name":"app","env":[{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"db-secret-v2","key":"password"}}}]}]}}}}'

# Method 2: Annotation to force restart
kubectl patch deployment app-deployment -p '{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'$(date +%Y-%m-%dT%H:%M:%S%z)'"}}}}}'

Verify​

kubectl get pods -l app=myapp
kubectl logs <pod-name> | grep Password

Exercise 5: Automated Rotation with Script​

Create a Rotation Script​

Create rotate-secret.sh:

#!/bin/bash

SECRET_NAME=$1
NEW_SECRET_NAME=$2
DEPLOYMENT_NAME=$3
ENV_VAR_NAME=$4

# Create the new secret (must be created manually before)
# kubectl create secret generic $NEW_SECRET_NAME --from-literal=password=<new-password>

# Update the deployment
kubectl set env deployment/$DEPLOYMENT_NAME \
  $ENV_VAR_NAME=$(kubectl get secret $NEW_SECRET_NAME -o jsonpath='{.data.password}' | base64 -d)

# Wait for rollout
kubectl rollout status deployment/$DEPLOYMENT_NAME

echo "Rotation complete. Delete the old secret: kubectl delete secret $SECRET_NAME"

Use the Script​

chmod +x rotate-secret.sh
./rotate-secret.sh db-secret-v1 db-secret-v2 app-deployment DB_PASSWORD

Exercise 6: Rotation with Init Container​

Create a Deployment with Init Container​

Create deployment-init.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment-init
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp-init
  template:
    metadata:
      labels:
        app: myapp-init
    spec:
      initContainers:
      - name: secret-validator
        image: busybox:1.35
        command: ["sh", "-c", "test -n \"$DB_PASSWORD\" && echo 'Secret valid'"]
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret-v2
              key: password
      containers:
      - name: app
        image: busybox:1.35
        command: ["sh", "-c", "echo 'Password: $DB_PASSWORD' && sleep 3600"]
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret-v2
              key: password

Exercise 7: Post-Rotation Verification​

Verify Pods​

# View all Pods
kubectl get pods -l app=myapp

# Check logs
kubectl logs <pod-name> | grep Password

# Check environment variables
kubectl exec <pod-name> -- env | grep DB_PASSWORD

Verify Secrets​

# View secrets
kubectl get secrets | grep db-secret

# Verify the old secret is no longer used
kubectl get pods -o jsonpath='{.items[*].spec.containers[*].env[*].valueFrom.secretKeyRef.name}' | grep -o db-secret-v1

Cleanup​

# Delete deployments
kubectl delete deployment app-deployment app-deployment-init

# Delete old secrets (after verification)
kubectl delete secret db-secret-v1

# Keep db-secret-v2 for reference

Reflection Questions​

  1. Why is it important to verify that all Pods are using the new secret before deleting the old one?
  2. How can you automate secret rotation?
  3. What is the best strategy for rotation without downtime?
  4. How do you handle secret rotation for StatefulSets?

Bonus Challenge​

Create a CronJob that automatically checks the age of secrets and alerts if a secret has not been rotated in more than 90 days.

Lab created: December 2024