Backing up and restoring etcd data is a task you might have to perform in the Certified Kubernetes Administrator (CKA) exam. This post shows how to do it.
Example Exam Task: Backup and Restore etcd
Create a backup of the etcd data in your Kubernetes cluster. After you create the backup delete the namespace
Restore your etcd data from the backup you just created. Verify the namespace
moon was restored.
- A Kubernetes cluster (I’m using a local minikube cluster for this example)
- Kubectl installed and configured to access the Kubernetes cluster with administrator rights
- Run the following script to prepare your Kubernetes cluster for this exercise:
#!/bin/bash kubectl create ns backup-ns kubectl create deploy test-app -n backup-ns --image nginx:1.25.1 kubectl create cm test-cm1 -n backup-ns kubectl create cm test-cm2 -n backup-ns kubectl create secret generic test-secret1 -n backup-ns kubectl create secret generic test-secret2 -n backup-ns
To interact with etcd we need to have the appropriate authentication certificates first. We can find the path where certs are stored on the control plane by looking at the spec of the etcd-minikube pod.
# We need to find the server.crt server.ky and ca.crt file, the manifests shows # there is a mount /var/lib/minikube/certs/etcd where all of them are stored kubectl get pod -n kube-system etcd-minikube | grep -i cert # we also need the endpoints to reach etcd with a client # - --listen-client-urls=https://127.0.0.1:2379,https://192.168.49.2:2379 # we'll use the second ip URL later because the first is a loopback only reachable from the pod itself kubectl get pod -n kube-system etcd-minikube | grep listen-client
The best way to take the backup is by using ssh to connect to your control plane. Then install the etcd tools
🤓 In the exam, the tools you need are usually installed in your exam environment or on the control plane node. Because we're using minikube, we have to install them ourselves.
minikube ssh sudo -s # switch to root user to have access to etcd paths # Download etcd binaries curl -LO https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz # Unpack etcd binaries and put them on your PATH tar xvzf etcd-v3.5.6-linux-amd64.tar.gz export PATH=$PATH:$(pwd)/etcd-v3.5.6-linux-amd64 # Configure ETCD environment with the date we found above cert_path=/var/lib/minikube/certs/etcd export ETCDCTL_API=3 export ETCDCTL_CACERT=$cert_path/ca.crt export ETCDCTL_CERT=$cert_path/server.crt export ETCDCTL_KEY=$cert_path/server.key export ETCDCTL_ENDPOINTS="https://192.168.49.2:2379" # Run a healthcheck to check configuration is valid etcdctl endpoint health https://192.168.49.2:2379 is healthy: successfully committed proposal: took = 6.7207ms
Now that we’ve configured
etcdctl we can go ahead and do the backup of etcd data:
etcdctl snapshot save snapshot.db
This command creates a backup file on your control plane node in your current path called snapshot.db.
Delete the namespace
Open another terminal with kubectl access to your Kubernetes cluster. Use kubectl to delete the namespace moon:
kubectl delete ns backup-ns
Restore etcd data
Before running a restore operation for etcd, one should always stop all kube-apiserver and etcd pods. In our particular example, we need to stop the pods running on our minikube control plane node.
To do that, we ssh into the control plane node and temporarily move the static pod files to another directory:
# in the shell connected to your control plane mv /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/ mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/
After the kube-apiserver stopped we can start the restore operation:
# get the data directory of etcd: --data-dir=/var/lib/minikube/etcd cat /etc/kubernetes/etcd.yaml | grep data-dir # clear the data dir from old data rm -r /var/lib/minikube/etcd/* etcdctl snapshot restore snapshot.db --data-dir=/var/lib/minikube/etcd
Afterward, we restart the etcd and kube-apiserver pods by moving the static pod manifest back to the old location.
mv /etc/kubernetes/etcd.yaml /etc/kubernetes/manifests/ mv /etc/kubernetes/kube-apiserver.yaml /etc/kubernetes/manifests/
Wait for the kube-apiserver to come up again. Once it is running verify the restore worked by looking for the backup-ns namespace:
kubectl get ns NAME STATUS AGE backup-ns Active 74m default Active 74m kube-node-lease Active 75m kube-public Active 75m kube-system Active 75m
This post shows how to:
- Connect to etcd on a minikube cluster
- Backup the etcd data using etcdctl
- Restore the etcd data from a backup file using etcutl
Don’t want to miss the next post in the Certified Kubernetes Administrator (CKA) series? Follow me on LinkedIn!
To support my efforts use my affiliate link to buy your courses and exams from the Linux Foundation.
Previous post in the CKA series: CKA Exam Task: Create a Persistent Volume.