Backup and Restore of MongoDB Deployment on Kubernetes
Every database environment should require a robust backup strategy as its a fundamental requirement to running a successful and robust database. No matter the size of the database, the function of the application, or how technologically advanced a company is, backups are now a requirement for everyone.
As a Solutions Engineers, we speak to database users from all types of companies, ranging from startups to the most complex database environments being used today. Interestingly enough, while talking about backups, we hear several concerning statements such as, “We never used backups in the past, so we don’t need them in this new environment”, “cloud services never fail” (hint – they do), and “this cluster is too big to fail.” It’s never an issue until something happens to your environment and you are unable to recover data and put your entire company at risk.
The adoption of the databases on Kubernetes (K8s) and other cloud-native platforms is definitely on the rise. There are multiple tools and approaches to deploying MongoDB on K8s. Assuming that you already have your MongoDB up and running on K8s, how do you implement a backup strategy? It could be a healthy amount of error-prone, manual work. You’d need to configure K8s Jobs from scratch, configure mongodump and all its parameters, setup a PersistentVolume, accompanying Claims. Additionally, you would need to take care about your PersistentVolume durability. What about streaming backup to some remote storage or, dealing with the backup of a complex, sharded cluster?
There are free tools that help you to streamline the entire MongoDB deployment process on K8s, including backups. The critical nature of backups doesn’t have to cost you any money to implement neither. Percona offers the following free enterprise-grade solutions for MongoDB:
Let’s explore how to take and restore backups using PSMDB Operator deployed on AWS Elastic Kubernetes Service (AWS EKS).
Architecture
When deployed outside of K8s, PBM requires a running pbm-agent process on each node (next to the mongod instance) in the cluster/replica set. PBM uses it’s own ‘control collections’ in the admin database to store config and relay commands from the user (who uses the “pbm” CLI) to the pbm-agent processes. PBM’s backups are ‘logical’ style, the same as mongodump. This means the data is copied using a database driver connection rather than copying the underlying data files on disk.
When you deploy PSMDB cluster using PSMDB Operator, pbm-agent is automatically deployed in each pod as a sidecar container next to the mongod container. PSMDB Operator writes commands to the PBM control collections directly (in a way, replacing what outside of the K8s deployments pbm CLI does) controlling the entire backup process. PSMDB Operator supports two types of backups: on-demand and scheduled backups, both being controlled entirely by PSMDB Operator.
Backups taken by the PSMDB Operator can be stored in any S3 compatible storage, be it AWS S3, Google Cloud Storage, or locally deployed cloud-native MinIO storage. The backup contains a metadata file, a dump of all collections from your database, and an oplog dump covering the timespan of the backup.
Configuration
We have a running PSMDB replica set deployed with PSMDB Operator in AWS EKS K8s cluster. Please note that we use PSMDB Operator v1.4.0 (the newest release at the moment of writing of this article). To deploy it, we followed these instructions and used all default settings.
Check if your PSMDB cluster is running correctly:
$ kubectl get pods NAME READY STATUS RESTARTS AGE my-cluster-name-rs0-0 2/2 Running 0 113s my-cluster-name-rs0-1 2/2 Running 2 72s my-cluster-name-rs0-2 2/2 Running 1 42s percona-server-mongodb-operator-568f85969c-5hqbw 1/1 Running 0 22m
Access secrets: deploy/backup-s3.yaml
Let’s start by adding our AWS access and secret access keys. The operator will use these keys to access your S3 bucket (all cloud providers have different methods of distributing these keys). The keys that you put into K8s secrets must be base64 encoded. You can encode your keys by running
echo -n ‘YOUR_KEY’ | base64
in bash CLI:
$ cat deploy/backup-s3.yaml apiVersion: v1 kind: Secret metadata: name: my-cluster-name-backup-s3 type: Opaque data: AWS_ACCESS_KEY_ID: ############# AWS_SECRET_ACCESS_KEY: ##############
Create a secret in K8s cluster with the following command:
$ kubectl apply -f deploy/backup-s3.yaml secret/my-cluster-name-backup-s3 created
S3 bucket: deploy/cr.yaml
Next, we need to edit
storages
section in the deploy/cr.yaml file so we can send our backups to an S3 bucket.
$ cat deploy/cr.yaml
storages:
s3-us-east: #(backup name)
type: s3
s3:
bucket: psmdb-operator-backup #(bucket name)
credentialsSecret: my-cluster-name-backup-s3 #(reference to credentials set in backup-s3.yaml)
region: us-east-1
Apply changes:
$ kubectl apply -f deploy/cr.yaml perconaservermongodb.psmdb.percona.com/my-cluster-name configured
We are ready to take backups now!
On-Demand Backup
On-demand backup can be taken at any point in time. Pbm-control tool is distributed together with the operator code. Based on the requested details it will use pre-configured storage to store the on-demand backup.
Let’s start with editing deploy/backup/backup.yaml to ensure we are all set. psmdbCluster should match our cluster name and
storageName
the storage defined in previous steps.
$ cat deploy/backup/backup.yaml apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDBBackup metadata: name: backup1 spec: psmdbCluster: my-cluster-name storageName: s3-us-east
Run backup by the following command:
$ kubectl apply -f deploy/backup/backup.yaml perconaservermongodbbackup.psmdb.percona.com/backup1 created
If we set up everything correctly, our backup should be uploaded to the S3 bucket. To check its status run:
$ kubectl describe perconaservermongodbbackup.psmdb.percona.com/backup1
If you look at your S3 AWS console, you should see backup files there:
Automated Backup
The second type of backup is a scheduled backup. The backup:tasks section of the deploy/cr.yaml file can be edited to schedule fully automatically executed backups. We can configure backups using the UNIX cron string format. Let’s say we want to take backups every day at midnight. Let’s edit the deploy/cr.yaml file:
$ cat deploy/cr.yaml
tasks:
- name: daily-backup
enabled: true
schedule: "0 0 * * *"
storageName: s3-us-west
Apply changes:
$ kubectl apply -f deploy/cr.yaml perconaservermongodb.psmdb.percona.com/my-cluster-name configured
Restoring Backups
To restore a backup, we need to find the backup name. We can obtain a list of all backups by using the following command:
$ kubectl get psmdb-backup NAME CLUSTER STORAGE DESTINATION STATUS COMPLETED AGE backup1 my-cluster-name s3-us-east 2020-06-11T07:52:27Z ready 28m 28m
Backup restoration configuration is in deploy/backup/restore.yaml. Let’s ensure there’s the appropriate backup name specified:
$ cat deploy/backup/restore.yaml apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDBRestore metadata: name: restore1 spec: clusterName: my-cluster-name backupName: backup1
To restore the backup, we execute the following command:
$ kubectl apply -f deploy/backup/restore.yaml perconaservermongodbrestore.psmdb.percona.com/restore1 created
To check the backup status use:
$ kubectl describe perconaservermongodbrestore.psmdb.percona.com/restore1
Percona Kubernetes Operator for MongoDB utilizing Percona Backup for MongoDB is a Kubernetes-idiomatic way to run, backup, and restore a MongoDB replica set. If you’d like to learn more about these tools, check out our documentation.
by Michal Nosek via Percona Database Performance Blog
Comments
Post a Comment