Automate etcd Snapshot Backups Using a CronJob

Scenario

You are a Kubernetes administrator responsible for the cluster's operational health. A periodic audit has flagged a critical gap: there is no automated etcd backup in place. You must configure a CronJob that automatically takes etcd snapshot backups every 6 hours and stores them on the control plane node.

The environment setup taints the control plane node cplane-01 with NoSchedule to prevent regular workloads from being scheduled on it. Configure the etcd-backup CronJob with the appropriate toleration so that its pods can be scheduled on cplane-01, where the etcd service is running.

Task

Create a CronJob named etcd-backup in the kube-system namespace that automatically creates an etcd snapshot every 6 hours.

Use the container image registry.k8s.io/etcd:3.6.8-0. Set the environment variable ETCDCTL_API=3. Use the Downward API to expose the Pod's metadata.uid as an environment variable named POD_UID — this ensures each snapshot file has a unique name.

Set the pod restart policy to OnFailure. Use nodeSelector with kubernetes.io/hostname: cplane-01 to ensure the pod runs on the control plane node. Configure appropriate tolerations to allow the pod to schedule on the tainted node cplane-01.

Create a HostPath volume named backup-volume that mounts /home/laborant/etcd-backup on the host to /backup inside the container.

Create a HostPath volume named etcd-certs that mounts /etc/kubernetes/pki/etcd on the host to /etc/kubernetes/pki/etcd inside the container as readOnly.

Configure the container to run:

command:
- etcdctl
args:
- snapshot
- save
- /backup/etcd-snapshot-$(POD_UID).db
- --endpoints=https://172.16.0.2:2379
- --cacert=/etc/kubernetes/pki/etcd/ca.crt
- --cert=/etc/kubernetes/pki/etcd/server.crt
- --key=/etc/kubernetes/pki/etcd/server.key

Once the CronJob is created, manually trigger a Job named etcd-backup-manual-test in the kube-system namespace from the CronJob and verify it completes successfully with a valid snapshot written to /home/laborant/etcd-backup/ on the host.

Important

Wait one minute for the environment setup to complete before starting the task.

Hint 1 — Inspect the etcd Static Pod

Find the exact certificate paths and endpoint used by the running etcd pod:

sudo cat /etc/kubernetes/manifests/etcd.yaml

Look for --cert-file, --key-file, --trusted-ca-file, and --advertise-client-urls. These values go directly into the CronJob container args.

Documentation

Hint 2 — Writing the CronJob YAML

Use nodeSelector to pin the pod to cplane-01 and add a toleration for the NoSchedule taint:

nodeSelector:
  kubernetes.io/hostname: cplane-01
tolerations:
- key: node-role.kubernetes.io/control-plane
  operator: Exists
  effect: NoSchedule

Use the Downward API to expose the Pod UID as an environment variable:

env:
- name: POD_UID
  valueFrom:
    fieldRef:
      fieldPath: metadata.uid

Documentation

Hint 3 — Manually Trigger and Verify the Job

After creating the CronJob, trigger a test run without waiting for the schedule:

kubectl create job --from=cronjob/etcd-backup etcd-backup-manual-test -n kube-system

Check the job and logs:

kubectl get job etcd-backup-manual-test -n kube-system
kubectl logs -l job-name=etcd-backup-manual-test -n kube-system
ls -lh /home/laborant/etcd-backup/

Documentation

Automate etcd Snapshot Backups Using a CronJob

Scenario

Task

⚒ Test Cases