Backup a cluster with Velero

Velero is a popular open source backup solution for Kubernetes. Its core implementation is a controller running in the cluster that oversees the backup and restore operations. The administrator is given a CLI tool to schedule operations and/or perform on-demand backup and restores. This CLI tool creates Kubernetes resources that the in-cluster Velero controller acts upon. During installation the controller needs to be configured with a repository (called a ‘provider’), where the backup files are stored.

This document describes how to setup Velero with the MinIO provider acting as an S3 compatible object store.

Prerequisites

Enabling required components

DNS and helm are needed for this setup:

sudo microk8s enable dns
sudo microk8s enable helm3

Install MinIO

MinIO provides an S3 compatible interface over storage provisioned by Kubernetes. For the purposes of this guide, the hostpath storage add-on is used to satisfy the persistent volume claims:

sudo microk8s enable hostpath-storage

Helm is used to setup MinIO under the velero namespace:

sudo microk8s kubectl create namespace velero
sudo microk8s helm3 repo add minio https://helm.min.io
sudo microk8s helm3 install -n velero --set buckets[0].name=velero,buckets[0].policy=none,buckets[0].purge=false minio minio/minio

Create a demo workload

The workload we will demonstrate the backup with is an NGINX deployment and a corresponding service under the workloads namespace. Create this setup with:

sudo microk8s kubectl create namespace workloads
sudo microk8s kubectl create deployment nginx -n workloads --image nginx
sudo microk8s.kubectl expose deployment nginx -n workloads --port 80

Installing Velero

To install Velero we get the a binary from the releases page on github and place it in our PATH. In this case we install the v1.7.1 Linux binary for AMD64 under /usr/local/bin:

wget https://github.com/vmware-tanzu/velero/releases/download/v1.7.1/velero-v1.7.1-linux-amd64.tar.gz 
tar -xzf velero-v1.7.1-linux-amd64.tar.gz
chmod +x velero-v1.7.1-linux-amd64/velero
sudo chown root:root velero-v1.7.1-linux-amd64/velero
sudo mv velero-v1.7.1-linux-amd64/velero /usr/local/bin/velero

Before installing Velero, we export the kubeconfig file from MicroK8s.

mkdir -p $HOME/.kube
sudo microk8s config > $HOME/.kube/config

We also export the MinIO credentials so we can feed them to Velero.

ACCESS_KEY=$(sudo microk8s kubectl -n velero get secret minio -o jsonpath="{.data.accesskey}" | base64 --decode)
SECRET_KEY=$(sudo microk8s kubectl -n velero get secret minio -o jsonpath="{.data.secretkey}" | base64 --decode)
cat <<EOF > credentials-velero
[default]
    aws_access_key_id=${ACCESS_KEY}
    aws_secret_access_key=${SECRET_KEY}
EOF

We are now ready to install Velero:

velero install \
--use-restic \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.3.0 \
--bucket velero \
--secret-file ./credentials-velero \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000 \
--snapshot-location-config region=minio

Velero uses Restic for backing up Kubernetes volumes. To let Restic know of the kubelet directory in the MicroK8s context we need to patch its daemonset manifest:

sudo microk8s kubectl -n velero patch daemonset.apps/restic --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/volumes/0/hostPath/path", "value":"/var/snap/microk8s/common/var/lib/kubelet/pods"}]'

Backup workloads

To backup the workloads namespace we use the --include-namespaces argument:

 velero backup create workloads-backup --include-namespaces=workloads

To check the progress of a backup operation we use describe, providing the backup name:

 velero backup describe workloads-backup 

In the output you should see this operation completed:

Name:         workloads-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.23.3-2+3cea96839f0d64
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=23+

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  workloads
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2022-02-08 10:44:08 +0200 EET
Completed:  2022-02-08 10:44:10 +0200 EET

Expiration:  2022-03-10 10:44:08 +0200 EET

Total items to be backed up:  17
Items backed up:              17

Velero-Native Snapshots: <none included>

Restore workloads

Before restoring the workloads namespace, let’s delete it first:

 sudo microk8s.kubectl delete namespace workloads

We can now create a restore operation specifying the backup we want to use:

velero restore create --from-backup workloads-backup

A restore operation which we can monitor using the describe command is then created:

velero restore describe workloads-backup-20220208105156

The describe output should eventually report a “Completed” phase:

Name:         workloads-backup-20220208105156
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       Completed
Total items to be restored:  10
Items restored:              10

Started:    2022-02-08 10:51:56 +0200 EET
Completed:  2022-02-08 10:51:57 +0200 EET

Backup:  workloads-backup

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Preserve Service NodePorts:  auto

Listing the resources of the workloads namespaces confirms that the restoration process was successful:

sudo microk8s kubectl get all -n workloads

Summing up

Although Velero is a really powerful tool with a large set of configuration options it is also very easy to use. You are required to set up a backup strategy based on the backend that will hold the backups and the scheduling of the backups. The rest is taken care of by the tool itself.