Skip to main content

Lab 7.2 - DaemonSet for Monitoring

Lab Objectives​

By the end of this lab, you will be able to:

  • Understand the concept of DaemonSet.
  • Create a DaemonSet to deploy a monitoring agent.
  • Verify that the DaemonSet runs on all nodes.
  • Understand DaemonSet use cases.
  • Manage DaemonSets (update, delete).

Estimated Duration​

45-60 minutes

Prerequisites​

  • kubectl installed and configured.
  • Functional local Kubernetes cluster (minikube or kind).
  • Knowledge of DaemonSets (Chapter 7.2).

Part 1: Understanding DaemonSets​

A DaemonSet guarantees that a copy of a Pod runs on each node (or a subset of nodes). It is ideal for:

  • Monitoring agents (e.g., node-exporter)
  • Logging agents (e.g., Fluentd, Fluent Bit)
  • Security agents
  • Metrics collectors

Step 1.1: Check the Nodes​

Check the nodes in your cluster:

kubectl get nodes

For a local cluster (minikube/kind), you will typically have a single node.

Part 2: Creating a Simple DaemonSet​

We will create a DaemonSet that simulates a simple monitoring agent.

Step 2.1: Create the DaemonSet​

Create a file daemonset-monitoring.yaml:

# daemonset-monitoring.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
  labels:
    app: node-monitor
spec:
  selector:
    matchLabels:
      app: node-monitor
  template:
    metadata:
      labels:
        app: node-monitor
    spec:
      containers:
      - name: monitor-agent
        image: busybox:1.35
        command: ["/bin/sh", "-c"]
        args:
        - |
          while true; do
            echo "$(date): Node $(hostname) - CPU: $(cat /proc/loadavg | awk '{print $1}') - Memory: $(free -m | grep Mem | awk '{print $3}')MB used"
            sleep 30
          done
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi
        volumeMounts:
        - name: proc
          mountPath: /proc
          readOnly: true
        - name: sys
          mountPath: /sys
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys

Explanation:

  • The DaemonSet creates a Pod on each node.
  • The container mounts /proc and /sys from the node to access system metrics.
  • The container periodically displays information about the node.

Apply the DaemonSet:

kubectl apply -f daemonset-monitoring.yaml

Part 3: Verifying the DaemonSet​

Step 3.1: Check the Created Pods​

Check that the Pods are created:

kubectl get pods -l app=node-monitor

You should see one Pod per node. For a single-node cluster, you will see one Pod.

Check on which nodes the Pods are running:

kubectl get pods -l app=node-monitor -o wide

Step 3.2: Check the DaemonSet Status​

Check the DaemonSet status:

kubectl get daemonset node-monitor
# or
kubectl get ds node-monitor

You should see:

  • DESIRED: Number of nodes where the Pod should run
  • CURRENT: Number of currently created Pods
  • READY: Number of ready Pods
  • UP-TO-DATE: Number of Pods up to date with the template
  • AVAILABLE: Number of available Pods

Step 3.3: Check the Logs​

Check the logs of a Pod to see the collected metrics:

kubectl logs -l app=node-monitor --tail=10

You should see periodic messages with node information.

Part 4: Testing Node Addition (Simulation)​

If you had multiple nodes, the DaemonSet would automatically create a Pod on each new node. For a local cluster, we will simulate this by verifying the behavior.

Step 4.1: Check the Behavior​

The DaemonSet automatically monitors nodes. If a new node is added to the cluster, a Pod will be automatically created on that node.

Check the DaemonSet description for details:

kubectl describe daemonset node-monitor

Part 5: Updating the DaemonSet​

Step 5.1: Update the Image​

Modify the DaemonSet to change the collection frequency:

# daemonset-monitoring-updated.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
  labels:
    app: node-monitor
spec:
  selector:
    matchLabels:
      app: node-monitor
  template:
    metadata:
      labels:
        app: node-monitor
    spec:
      containers:
      - name: monitor-agent
        image: busybox:1.35
        command: ["/bin/sh", "-c"]
        args:
        - |
          while true; do
            echo "$(date): [UPDATED] Node $(hostname) - Monitoring active"
            sleep 10 # Changed from 30 to 10 seconds
          done
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi

Apply the update:

kubectl apply -f daemonset-monitoring-updated.yaml

Step 5.2: Observe the Update​

DaemonSet Pods are updated in a rolling fashion (one by one):

kubectl get pods -l app=node-monitor -w

You will see old Pods being deleted and new ones being created.

Check the new logs:

kubectl logs -l app=node-monitor --tail=5

You should see messages with "[UPDATED]" and a higher frequency.

Part 6: Node Selectors and Taints​

Step 6.1: Using a Node Selector (Optional)​

You can limit which nodes the DaemonSet runs on with a nodeSelector:

spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/os: linux # Runs only on Linux nodes
      containers:
      # ...

Step 6.2: Tolerating Taints (Optional)​

If some nodes have taints, you can add tolerations to the DaemonSet:

spec:
  template:
    spec:
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      # ...

Part 7: Cleanup​

Delete the DaemonSet:

kubectl delete daemonset node-monitor
# or
kubectl delete -f daemonset-monitoring.yaml

All Pods created by the DaemonSet will be automatically deleted.

Lab Summary​

In this lab, you explored DaemonSets for monitoring. You learned how to create a DaemonSet that runs on all nodes, verify its behavior, update it, and understand typical use cases (monitoring, logging, security).

Next Steps​

The next lab will show you how to use Jobs and CronJobs to run one-time and scheduled tasks.

Lab 7.3: Jobs and CronJobs

Lab created: December 2024