Skip to main content

Chapter 7.1 - In-Depth StatefulSets

Learning Objectives​

By the end of this chapter, you will be able to:

  • Understand StatefulSets in depth
  • Deploy databases with StatefulSets
  • Manage scaling and updates
  • Understand init containers in StatefulSets
  • Implement master-slave replication
  • Manage quorums and leader election

Introduction​

StatefulSets are essential for stateful applications. This chapter dives deeper into the concepts and presents advanced use cases.

StatefulSet Architecture​

Components​

Example: PostgreSQL Database​

Step 1: Headless Service​

apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
  - port: 5432
    name: postgres

Step 2: ConfigMap for Configuration​

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
data:
  postgresql.conf: |
    max_connections = 200
    shared_buffers = 256MB
    effective_cache_size = 1GB

Step 3: Secret for Credentials​

apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
type: Opaque
data:
  postgres-password: cG9zdGdyZXM=  # base64
  replication-password: cmVwbGljYXRpb24=  # base64

Step 4: StatefulSet​

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      initContainers:
      - name: init-postgres
        image: postgres:14
        command:
        - /bin/bash
        - -c
        - |
          set -e
          if [[ $HOSTNAME == "postgres-0" ]]; then
            echo "Initializing primary database..."
          else
            echo "Waiting for primary to be ready..."
            until pg_isready -h postgres-0.postgres; do
              sleep 1
            done
          fi
        env:
        - name: PGHOST
          value: postgres-0.postgres
      containers:
      - name: postgres
        image: postgres:14
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: postgres-password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        ports:
        - containerPort: 5432
          name: postgres
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
        - name: postgres-config
          mountPath: /etc/postgresql
      volumes:
      - name: postgres-config
        configMap:
          name: postgres-config
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "fast-ssd"
      resources:
        requests:
          storage: 50Gi

Init Containers​

Init containers run before the main containers and can:

  • Initialize the database
  • Wait for other services to be ready
  • Configure the environment
  • Check dependencies

Example:

spec:
  template:
    spec:
      initContainers:
      - name: wait-for-db
        image: busybox:1.35
        command:
        - sh
        - -c
        - |
          until nc -z postgres-0.postgres 5432; do
            echo "Waiting for database..."
            sleep 2
          done
      containers:
      - name: app
        image: my-app:1.0

Master-Slave Replication​

Architecture​

Configuration:

spec:
  template:
    spec:
      containers:
      - name: postgres
        env:
        - name: POSTGRES_REPLICATION_MODE
          value: "master"  # For postgres-0
        - name: POSTGRES_MASTER_HOST
          value: "postgres-0.postgres"  # For postgres-1, postgres-2

Quorum and Leader Election​

For applications requiring a quorum (e.g., etcd, Consul):

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd
spec:
  serviceName: etcd
  replicas: 3  # Quorum: (3/2) + 1 = 2
  template:
    spec:
      containers:
      - name: etcd
        image: quay.io/coreos/etcd:v3.5.0
        env:
        - name: ETCD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ETCD_INITIAL_CLUSTER
          value: "etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380"

Quorum: (N/2) + 1, where N = number of replicas

Scaling​

Scaling Up​

kubectl scale statefulset postgres --replicas=5

Process:

  1. Creates postgres-3
  2. Waits for postgres-3 to be Ready
  3. Creates postgres-4
  4. Waits for postgres-4 to be Ready

Scaling Down​

kubectl scale statefulset postgres --replicas=2

Process:

  1. Deletes postgres-4 (last)
  2. Waits for postgres-4 to terminate
  3. Deletes postgres-3
  4. Waits for postgres-3 to terminate

Important: For databases, ensure data is backed up before scaling down.

Updates​

Rolling Update​

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0  # Update all Pods

Process:

  1. Updates postgres-2 (last)
  2. Updates postgres-1
  3. Updates postgres-0 (first)

Partial Update (Canary)​

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2  # Keeps postgres-0 and postgres-1 on the old version

Result: Only postgres-2 is updated.

Useful Commands​

Management​

# View StatefulSets
kubectl get statefulset
kubectl get sts

# Details
kubectl describe statefulset postgres

# View Pods
kubectl get pods -l app=postgres

# Logs of a specific Pod
kubectl logs postgres-0

Scaling​

# Scaling up
kubectl scale statefulset postgres --replicas=5

# Scaling down
kubectl scale statefulset postgres --replicas=2

Updates​

# Update the image
kubectl set image statefulset/postgres postgres=postgres:15

# View update status
kubectl rollout status statefulset/postgres

# Rollback
kubectl rollout undo statefulset/postgres

Best Practices​

1. Regular Backups​

Set up automatic volume backups.

2. Monitoring​

Monitor the status of each Pod and performance metrics.

3. Appropriate Quorum​

Use an odd number of replicas to avoid ties.

4. Init Containers​

Use init containers for initialization and checks.

5. Headless Service​

Always create a headless service for DNS discovery.

Summary​

In this chapter, you learned:

In-depth StatefulSets: Stateful applications with stable identities
Databases: Deploying PostgreSQL with replication
Init containers: Initialization before main containers
Replication: Master-slave configuration
Quorum: Calculation and configuration for distributed applications
Scaling: Guaranteed order (ascending/descending)
Updates: RollingUpdate and partial updates
Best practices: Backups, monitoring, quorum

Next Steps​

Chapter 7.2: DaemonSets
Chapter 7.3: Jobs
Chapter 7.4: CronJobs

Chapter created: December 2024