Skip to main content

Chapter 11.1 - Kubernetes Monitoring

Learning Objectives​

By the end of this chapter, you will be able to:

  • Understand the importance of monitoring in Kubernetes
  • Install Prometheus for metrics collection
  • Configure Grafana for visualization
  • Understand essential Kubernetes metrics
  • Configure ServiceMonitors
  • Create custom dashboards

Introduction​

Monitoring is essential for understanding the health and performance of a Kubernetes cluster and its applications.

Monitoring Tools​

Prometheus​

Open-source monitoring and alerting system that collects and stores metrics as time series.

Features:

  • Pull-based metrics collection
  • Efficient storage
  • PromQL query language
  • Native integration with Kubernetes

Grafana​

Visualization platform that allows creating interactive dashboards from Prometheus metrics.

Features:

  • Customizable dashboards
  • Visual alerts
  • Support for multiple data sources
  • Sharing and collaboration

Installing Prometheus​

Via Helm​

# Add the repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install Prometheus
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Verification​

# View Pods
kubectl get pods -n monitoring

# View Services
kubectl get svc -n monitoring

# Access Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

Important Metrics​

Cluster Metrics​

  • CPU: CPU usage of nodes and Pods
  • Memory: Memory usage of nodes and Pods
  • Network: Incoming/outgoing network traffic
  • Storage: Storage usage

Application Metrics​

  • HTTP Requests: Request rate, latency, errors
  • DB Connections: Connection pool, response time
  • Queue: Queue size, processing time

Kubernetes Metrics​

  • Pods: State, restarts, ready/unready
  • Deployments: Desired vs actual replicas
  • Services: Available endpoints

ServiceMonitors​

ServiceMonitors allow Prometheus to discover and scrape metrics from services.

Example: ServiceMonitor​

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-metrics
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Service with Metrics​

apiVersion: v1
kind: Service
metadata:
  name: my-app-service
  labels:
    app: my-app
spec:
  selector:
    app: my-app
  ports:
  - name: http
    port: 8080
  - name: metrics
    port: 9090

Installing Grafana​

Via Helm (Included with kube-prometheus-stack)​

Grafana is included in kube-prometheus-stack.

Access​

# Port-forward
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

# Default credentials
# Username: admin
# Password: prom-operator

Pre-built Dashboards​

  • Kubernetes Cluster Monitoring
  • Node Exporter
  • Pod Monitoring
  • Etc.

Creating Dashboards​

Basic Dashboard​

{
  "dashboard": {
    "title": "My App Dashboard",
    "panels": [
      {
        "title": "Request Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{method}}"
          }
        ]
      },
      {
        "title": "Error Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total{status=~\"5..\"}[5m])"
          }
        ]
      }
    ]
  }
}

PromQL (Prometheus Query Language)​

Basic Queries​

# Request rate
rate(http_requests_total[5m])

# Sum by label
sum(rate(http_requests_total[5m])) by (method)

# Error percentage
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

# CPU usage
100 - (avg(rate(container_cpu_usage_seconds_total[5m])) * 100)

Useful Commands​

Prometheus​

# Access Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

# View targets
# In the Prometheus UI: Status > Targets

Grafana​

# Access Grafana
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

# Import a dashboard
# In Grafana: + > Import > Dashboard ID

Best Practices​

1. Essential Metrics​

Monitor critical metrics: CPU, memory, latency, errors.

2. Retention​

Configure appropriate retention for metrics.

3. Dashboards​

Create dashboards per application/service.

4. Alerts​

Configure alerts for critical metrics.

5. Performance​

Avoid scraping too frequently to prevent overloading.

Summary​

In this chapter, you learned:

Monitoring: Essential for cluster health
Prometheus: Metrics collection and storage
Grafana: Visualization with dashboards
Metrics: Cluster, application, Kubernetes
ServiceMonitors: Automatic metrics discovery
PromQL: Query language for metrics
Dashboards: Customizable visualization
Best practices: Essential metrics, retention, alerts

Next Steps​

Chapter 11.2: Logs
Chapter 11.3: Alerting
Lab 11.1: Installing Prometheus and Grafana

Chapter created on: December 2024