Skip to main content

Chapter 11.2 - Log Management

Learning Objectives​

By the end of this chapter, you will be able to:

  • Understand log management in Kubernetes
  • Use kubectl to view logs
  • Centralize logs with EFK/Loki
  • Configure Fluentd/Fluent Bit
  • Analyze logs with Kibana/Grafana
  • Implement log rotation

Introduction​

Logs are essential for debugging and monitoring. In Kubernetes, logs are managed at the container level and must be centralized for efficient analysis.

Logs with kubectl​

Viewing Pod Logs​

# Logs of a Pod
kubectl logs my-pod

# Logs of a specific container
kubectl logs my-pod -c my-container

# Logs of all containers
kubectl logs my-pod --all-containers=true

# Logs with follow (streaming)
kubectl logs -f my-pod

# Previous logs (if Pod restarted)
kubectl logs my-pod --previous

# Logs with timestamps
kubectl logs my-pod --timestamps

# Limit the number of lines
kubectl logs my-pod --tail=100

# Logs since a specific time
kubectl logs my-pod --since=1h

Logs from Multiple Pods​

# Logs of all Pods with a label
kubectl logs -l app=my-app

# Logs of a Deployment
kubectl logs deployment/my-app

# Logs of a StatefulSet
kubectl logs statefulset/my-app

Log Centralization​

EFK Stack (Elasticsearch, Fluentd, Kibana)​

Architecture:

  • Fluentd: Collects and sends logs
  • Elasticsearch: Stores logs
  • Kibana: Visualizes and searches logs

Loki Stack (Loki, Grafana)​

Architecture:

  • Fluent Bit: Collects logs
  • Loki: Stores logs (like Prometheus for logs)
  • Grafana: Visualizes logs

Installing the EFK Stack​

Via Helm​

# Add the repository
helm repo add elastic https://helm.elastic.co
helm repo update

# Install Elasticsearch
helm install elasticsearch elastic/elasticsearch \
  --namespace logging \
  --create-namespace

# Install Fluentd
helm install fluentd fluent/fluentd \
  --namespace logging

# Install Kibana
helm install kibana elastic/kibana \
  --namespace logging

Fluentd Configuration​

Fluentd DaemonSet​

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch.logging.svc.cluster.local"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Installing the Loki Stack​

Via Helm​

# Add the repository
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Loki
helm install loki grafana/loki-stack \
  --namespace logging \
  --create-namespace

Fluent Bit Configuration​

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
    
    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            docker
        Tag               kube.*
        Refresh_Interval  5
    
    [OUTPUT]
        Name        loki
        Match       kube.*
        Host        loki.logging.svc.cluster.local
        Port        3100
        Labels      job=fluentbit

Analysis with Kibana​

Access​

# Port-forward
kubectl port-forward -n logging svc/kibana-kibana 5601:5601

Kibana Query Language (KQL):

kubernetes.namespace: "production" AND level: "error"

Lucene:

kubernetes.namespace:production AND level:error

Analysis with Grafana​

Access​

# Port-forward
kubectl port-forward -n logging svc/loki-grafana 3000:80

LogQL (Loki Query Language)​

# Logs from a namespace
{namespace="production"}

# Logs with errors
{namespace="production"} |= "error"

# Log rate per second
rate({namespace="production"}[5m])

Log Lifecycle​

Best Practices​

1. Centralization​

Centralize all logs in a single system.

2. Rotation​

Configure log rotation to prevent storage exhaustion.

3. Indexing​

Index logs for fast searching.

4. Retention​

Define an appropriate retention policy.

5. Parsing​

Parse structured logs (JSON) for better searching.

Summary​

In this chapter, you learned:

kubectl logs: View Pod logs
Centralization: EFK Stack or Loki Stack
Fluentd/Fluent Bit: Log collection
Elasticsearch: Storage and search
Loki: Lightweight log storage
Kibana/Grafana: Visualization and analysis
Best practices: Centralization, rotation, indexing, retention

Next Steps​

Chapter 11.3: Alerting
Chapter 11.4: Distributed Tracing
Lab 11.3: Logging Configuration

Chapter created on: December 2024