Skip to main content

Chapter 7.3 - Jobs

Learning Objectives​

By the end of this chapter, you will be able to:

  • Understand what a Job is
  • Create and manage Jobs
  • Configure completions and parallelism
  • Handle failures and retries
  • Understand different types of Jobs
  • Use Jobs for batch processing

Introduction​

Jobs create one or more Pods and guarantee that a certain number of them terminate successfully. Unlike Deployments that keep Pods running, Jobs execute one-time tasks.

What is a Job?​

A Job creates one or more Pods and guarantees that a certain number of them terminate successfully. Pods created by a Job are not automatically deleted, allowing you to view logs and debug.

Characteristics​

  • One-time task: Executes a task and terminates
  • Completions: Number of required successes
  • Parallelism: Number of parallel Pods
  • Retries: New Pods on failure

Types of Jobs​

1. Simple Job (Non-parallel)​

A single Pod that must succeed:

apiVersion: batch/v1
kind: Job
metadata:
  name: simple-job
spec:
  template:
    spec:
      containers:
      - name: task
        image: busybox:1.35
        command: ["sh", "-c", "echo 'Job completed' && sleep 30"]
      restartPolicy: Never

2. Job with Completions​

Multiple Pods must succeed:

apiVersion: batch/v1
kind: Job
metadata:
  name: completion-job
spec:
  completions: 5  # 5 Pods must succeed
  template:
    spec:
      containers:
      - name: task
        image: busybox:1.35
        command: ["sh", "-c", "echo Processing item $JOB_COMPLETION_INDEX"]
      restartPolicy: Never

3. Parallel Job​

Multiple Pods in parallel:

apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-job
spec:
  completions: 10  # 10 Pods must succeed
  parallelism: 3   # 3 Pods in parallel
  template:
    spec:
      containers:
      - name: task
        image: busybox:1.35
        command: ["sh", "-c", "echo Processing && sleep 10"]
      restartPolicy: Never

Process:

Example: Data Processing​

Batch Processing Job​

apiVersion: batch/v1
kind: Job
metadata:
  name: data-processing
spec:
  completions: 100  # Process 100 items
  parallelism: 10  # 10 workers in parallel
  backoffLimit: 3  # 3 max retries
  template:
    spec:
      containers:
      - name: processor
        image: data-processor:1.0
        env:
        - name: JOB_COMPLETION_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['batch.kubernetes.io/job-completion-index']
        - name: TOTAL_ITEMS
          value: "100"
        command:
        - /app/process.sh
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1Gi
      restartPolicy: Never

Failure Handling​

Backoff Limit​

Maximum number of retries:

apiVersion: batch/v1
kind: Job
metadata:
  name: retry-job
spec:
  backoffLimit: 5  # 5 max retries
  template:
    spec:
      containers:
      - name: task
        image: my-app:1.0
        command: ["/app/run.sh"]
      restartPolicy: Never

Behavior:

  • If a Pod fails, a new Pod is created
  • Exponential backoff delay between retries
  • After backoffLimit failures, the Job is marked as failed

Active Deadline Seconds​

Timeout for the Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: timeout-job
spec:
  activeDeadlineSeconds: 300  # 5 minutes max
  template:
    spec:
      containers:
      - name: task
        image: my-app:1.0
      restartPolicy: Never

Result: The Job is cancelled after 5 minutes.

Example: Backup​

Backup Job​

apiVersion: batch/v1
kind: Job
metadata:
  name: database-backup
spec:
  template:
    spec:
      containers:
      - name: backup
        image: postgres:14
        command:
        - /bin/bash
        - -c
        - |
          pg_dump -h postgres-service -U postgres mydb > /backup/backup-$(date +%Y%m%d).sql
          gzip /backup/backup-$(date +%Y%m%d).sql
          aws s3 cp /backup/backup-$(date +%Y%m%d).sql.gz s3://backups/
        env:
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        volumeMounts:
        - name: backup-storage
          mountPath: /backup
      volumes:
      - name: backup-storage
        emptyDir: {}
      restartPolicy: Never

Useful Commands​

Create and Manage​

# Create a Job
kubectl apply -f job.yaml

# View Jobs
kubectl get jobs

# Details
kubectl describe job data-processing

# View Pods
kubectl get pods -l job-name=data-processing

# Logs
kubectl logs -l job-name=data-processing

Delete​

# Delete a Job (also deletes Pods)
kubectl delete job data-processing

# Delete with Pods
kubectl delete job data-processing --cascade=foreground

View Results​

# View status
kubectl get job data-processing -o yaml

# View completed Pods
kubectl get pods --field-selector=status.phase=Succeeded

Best Practices​

1. RestartPolicy​

Use Never or OnFailure, never Always.

2. Resources​

Define resource limits to prevent exhaustion.

3. Timeouts​

Use activeDeadlineSeconds to prevent Jobs from running indefinitely.

4. Cleanup​

Delete completed Jobs to free resources.

5. Monitoring​

Monitor Jobs to detect failures.

Summary​

In this chapter, you learned:

Job: One-time task that creates Pods until success
Types: Simple, with completions, parallel
Completions: Number of required successes
Parallelism: Number of parallel Pods
Failure handling: backoffLimit, activeDeadlineSeconds
Use cases: Batch processing, backups, migrations
RestartPolicy: Never or OnFailure
Best practices: Resources, timeouts, cleanup

Next Steps​

Chapter 7.4: CronJobs
Lab 7.3: Jobs and CronJobs

Chapter created: December 2024