search by tags

for the user

adventures into the land of the command line

heptio ark k8s cluster backups

Heptio Ark is this groovy tool that backs up entire k8s clusters by way of the etcd state and/or persistent volume snapshots to some kind of remote cloud storage location that you specify. It’s interesting.

How do we use it? (in this example, i am using microsoft azure’s cloud)

Prepare some cloud storage

Create a storage account and a blob container in the same subscription and resource group as the k8s cluster you want to be running backups on.

$ az storage account create \
    --name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $RESOURCE_GROUP \
    --sku Standard_LRS \
    --encryption-services blob \
    --https-only true \
    --kind BlobStorage \
    --access-tier Cool \
    --subscription $SUBSCRIPTION

$ az storage container create \
    -n $STORAGE_RESOURCE_NAME \
    --public-access off \
    --account-name $AZURE_STORAGE_ACCOUNT_ID \
    --subscription $SUBSCRIPTION

Get the storage account access key.

$ AZURE_STORAGE_KEY=`az storage account keys list \
    --account-name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $RESOURCE_GROUP \
    --query '[0].value' \
    --subscription $SUBSCRIPTION \
    -o tsv`

Create a service principle with appropriate permissions for heptio ark to use to read and write to the storage account.

$ az ad sp create-for-rbac \
    --name "heptio-ark" \
    --role "Contributor" \
    --password $AZURE_CLIENT_SECRET \
    --subscription $SUBSCRIPTION

Finally get the service principle’s id called a client id.

$ AZURE_CLIENT_ID=`az ad sp list \
    --display-name "heptio-ark" \
    --query '[0].appId' \
    --subscription $SUBSCRIPTION \
    -o tsv`

Provision ark

Next we provision an ark instance to our kubernetes cluster with a custom namespace. First clone the ark repo

$ git clone https://github.com/heptio/ark.git

You will need to edit 3 files.

ark/examples/common/00-prereqs.yaml
ark/examples/azure/00-ark-deployment.yaml
ark/examples/azure/10.ark-config.yaml

In these yamls, it tries to create a namespace called “heptio-ark” and then put things into that namespace. Change all of these references to a namespace you prefer. I called it “my-groovy-system”.

In the 10.ark-config.yaml, you also need to replace the placeholders YOUR_TIMEOUT & YOUR_BUCKET with some actual values. in our case, we use: 15m and the value of $STORAGE_RESOURCE_NAME, which in this case is ark-backups.

Create the pre-requisites.

$ kubectl apply -f examples/common/00-prereqs.yaml

    customresourcedefinition "backups.ark.heptio.com" created
    customresourcedefinition "schedules.ark.heptio.com" created
    customresourcedefinition "restores.ark.heptio.com" created
    customresourcedefinition "configs.ark.heptio.com" created
    customresourcedefinition "downloadrequests.ark.heptio.com" created
    customresourcedefinition "deletebackuprequests.ark.heptio.com" created
    customresourcedefinition "podvolumebackups.ark.heptio.com" created
    customresourcedefinition "podvolumerestores.ark.heptio.com" created
    customresourcedefinition "resticrepositories.ark.heptio.com" created
    namespace "my-groovy-system" created
    serviceaccount "ark" created
    clusterrolebinding "ark" created

Create a secret object, which contains all of the azure ids we gathered in part 1.

$ kubectl create secret generic cloud-credentials \
    --namespace my-groovy-system \
    --from-literal AZURE_SUBSCRIPTION_ID=$SUBSCRIPTION \
    --from-literal AZURE_TENANT_ID=$TENANT_ID \
    --from-literal AZURE_RESOURCE_GROUP=$RESOURCE_GROUP \
    --from-literal AZURE_CLIENT_ID=$AZURE_CLIENT_ID \
    --from-literal AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET \
    --from-literal AZURE_STORAGE_ACCOUNT_ID=$AZURE_STORAGE_ACCOUNT_ID \
    --from-literal AZURE_STORAGE_KEY=$AZURE_STORAGE_KEY
secret "cloud-credentials" created

Provision everything.

$ kubectl apply -f examples/azure/

$ kubectl get deployments -n my-groovy-system
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
ark       1         1         1            1           1h
$ kubectl get pods -n my-groovy-system
NAME                   READY     STATUS    RESTARTS   AGE
ark-7b86b4d5bd-2w5x7   1/1       Running   0          1h
$ kubectl get rs -n my-groovy-system
NAME             DESIRED   CURRENT   READY     AGE
ark-7b86b4d5bd   1         1         1         1h
$ kubectl get secrets -n my-groovy-system
NAME                  TYPE                                  DATA      AGE
ark-token-b5nm8       kubernetes.io/service-account-token   3         1h
cloud-credentials     Opaque                                7         1h
default-token-xg6x4   kubernetes.io/service-account-token   3         1h

At this point the ark server is running. To interact with it, we need to use a client.

Install the Ark client locally

Download one from here and unzip it and add it to your path. Here’s a mac example:

$ wget https://github.com/heptio/ark/releases/download/v0.9.3/ark-v0.9.3-darwin-amd64.tar.gz
$ tar -xzvf ark-v0.9.3-darwin-amd64.tar.gz
$ mv ark /Users/mygroovyuser/bin/ark
$ ark --help

Take this baby for a test drive

Deploy an example thing. Ark provides something to try with.

$ kubectl apply -f examples/nginx-app/base.yaml

This creates a namespace called nginx-example and creates a deployment and service inside with a couple of nginx pods.

Take a backup.

$ ark backup create nginx-backup --include-namespaces nginx-example --namespace my-groovy-system
Backup request "nginx-backup" submitted successfully.
Run `ark backup describe nginx-backup` for more details.

$ ark backup get nginx-backup --namespace my-groovy-system
NAME           STATUS      CREATED                          EXPIRES   SELECTOR
nginx-backup   Completed   2018-08-21 15:57:59 +0200 CEST   29d       

We can see in our Azure storage account container a backup has been created by heptio ark.

If we look inside the folder, we see some json and some gzipped stuff

Let’s simulate a disaster.

$ kubectl delete namespace nginx-example
namespace "nginx-example" deleted

And try to restore from the Ark backup.

$ ark restore create --from-backup nginx-backup --namespace my-groovy-system
Restore request "nginx-backup-20180821160537" submitted successfully.
Run `ark restore describe nginx-backup-20180821160537` for more details.

$ ark restore get --namespace my-groovy-system
NAME                          BACKUP         STATUS      WARNINGS   ERRORS    CREATED                          SELECTOR
nginx-backup-20180821160537   nginx-backup   Completed   0          0         2018-08-21 16:05:38 +0200 CEST   

Nice.

And to delete backups…

$ ark backup delete nginx-backup --namespace my-groovy-system
Are you sure you want to continue (Y/N)? Y
Request to delete backup "nginx-backup" submitted successfully.
The backup will be fully deleted after all associated data (disk snapshots, backup files, restores) are removed.

$ ark backup get nginx-backup --namespace my-groovy-system
An error occurred: backups.ark.heptio.com "nginx-backup" not found

And its gone.