Menu Close

CKA Exam Task: Backup and Restore etcd

Johannes Malsam

Backing up and restoring etcd data is a task you might have to perform in the Certified Kubernetes Administrator (CKA) exam. This post shows how to do it.

Example Exam Task: Backup and Restore etcd

Create a backup of the etcd data in your Kubernetes cluster. After you create the backup delete the namespace backup-ns.

Restore your etcd data from the backup you just created. Verify the namespace moon was restored.

Prerequisites

  • A Kubernetes cluster (I’m using a local minikube cluster for this example)
  • Kubectl installed and configured to access the Kubernetes cluster with administrator rights
  • Run the following script to prepare your Kubernetes cluster for this exercise:
#!/bin/bash
kubectl create ns backup-ns
kubectl create deploy test-app -n backup-ns --image nginx:1.25.1
kubectl create cm test-cm1 -n backup-ns
kubectl create cm test-cm2 -n backup-ns
kubectl create secret generic test-secret1 -n backup-ns
kubectl create secret generic test-secret2 -n backup-ns

Backup etcd

To interact with etcd we need to have the appropriate authentication certificates first. We can find the path where certs are stored on the control plane by looking at the spec of the etcd-minikube pod.

# We need to find the server.crt server.ky and ca.crt file, the manifests shows
# there is a mount /var/lib/minikube/certs/etcd where all of them are stored
kubectl get pod -n kube-system etcd-minikube | grep -i cert

# we also need the endpoints to reach etcd with a client
# - --listen-client-urls=https://127.0.0.1:2379,https://192.168.49.2:2379
# we'll use the second ip URL later because the first is a loopback only reachable from the pod itself
kubectl get pod -n kube-system etcd-minikube | grep listen-client

The best way to take the backup is by using ssh to connect to your control plane. Then install the etcd tools etcdctl and etcdutl.

🤓 In the exam, the tools you need are usually installed in your exam environment or on the control plane node. Because we're using minikube, we have to install them ourselves.
minikube ssh
sudo -s # switch to root user to have access to etcd paths

# Download etcd binaries
curl -LO https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz 

# Unpack etcd binaries and put them on your PATH
tar xvzf etcd-v3.5.6-linux-amd64.tar.gz 
export PATH=$PATH:$(pwd)/etcd-v3.5.6-linux-amd64

# Configure ETCD environment with the date we found above

cert_path=/var/lib/minikube/certs/etcd

export ETCDCTL_API=3 
export ETCDCTL_CACERT=$cert_path/ca.crt
export ETCDCTL_CERT=$cert_path/server.crt
export ETCDCTL_KEY=$cert_path/server.key
export ETCDCTL_ENDPOINTS="https://192.168.49.2:2379"

# Run a healthcheck to check configuration is valid
etcdctl endpoint health
https://192.168.49.2:2379 is healthy: successfully committed proposal: took = 6.7207ms

Now that we’ve configured etcdctl we can go ahead and do the backup of etcd data:

etcdctl snapshot save snapshot.db

This command creates a backup file on your control plane node in your current path called snapshot.db.

Delete the namespace

Open another terminal with kubectl access to your Kubernetes cluster. Use kubectl to delete the namespace moon:

kubectl delete ns backup-ns

Restore etcd data

Before running a restore operation for etcd, one should always stop all kube-apiserver and etcd pods. In our particular example, we need to stop the pods running on our minikube control plane node.

To do that, we ssh into the control plane node and temporarily move the static pod files to another directory:

# in the shell connected to your control plane
mv /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/
mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/

After the kube-apiserver stopped we can start the restore operation:

# get the data directory of etcd: --data-dir=/var/lib/minikube/etcd
cat /etc/kubernetes/etcd.yaml | grep data-dir

# clear the data dir from old data
rm -r /var/lib/minikube/etcd/*

etcdctl snapshot restore snapshot.db --data-dir=/var/lib/minikube/etcd

Afterward, we restart the etcd and kube-apiserver pods by moving the static pod manifest back to the old location.

mv /etc/kubernetes/etcd.yaml /etc/kubernetes/manifests/
mv /etc/kubernetes/kube-apiserver.yaml /etc/kubernetes/manifests/

Wait for the kube-apiserver to come up again. Once it is running verify the restore worked by looking for the backup-ns namespace:

kubectl get ns
NAME              STATUS   AGE
backup-ns         Active   74m
default           Active   74m
kube-node-lease   Active   75m
kube-public       Active   75m
kube-system       Active   75m

Conclusion

This post shows how to:

  • Connect to etcd on a minikube cluster
  • Backup the etcd data using etcdctl
  • Restore the etcd data from a backup file using etcutl

Don’t want to miss the next post in the Certified Kubernetes Administrator (CKA) series? Follow me on LinkedIn!

To support my efforts use my affiliate link to buy your courses and exams from the Linux Foundation.

Previous post in the CKA series: CKA Exam Task: Create a Persistent Volume.

Tagged ,