Percona Monitoring and Management 2 Docker Created Without pmm-data? No Problem!

By
PMM Docker Container Pmm-data

Disclaimer: This blog post is about migrating Percona Monitoring and Management 2 (PMM) data between PMM2 versions, and not for migrating data from PMM1 to PMM2. Restoring data from PMM1 to PMM2 is NOT supported since there were many architectural changes.

 

PMM Docker Container Pmm-dataI recently worked on a customer case where he was not using a pmm-data container mounted in /srv as instructed in the official doc. PMM2 data is stored under /srv (changed from PMM1, which stored data in other directories and had different mounts for pmm-data), which means that if the content of /srv is not mounted on a separate container, upgrade or recreation of pmm-server will lose the existing data.

The procedure of backing up data stored in pmm-server and recreating pmm-server + pmm-data containers with the correct mount in /srv and no data loss is described below, but first, you should check if pmm-data is correctly mounted:

Checking if pmm-data is Holding /srv:

If you are using pmm-data container mounted on “Destination: /srv/”, you should see the following outputs while checking pmm-data:

[root@centos vagrant]# docker inspect pmm-data | egrep "Source|Destination"
                "Source": "/var/lib/docker/volumes/6df58f0ad81199277892c2285e870f3edf563cc23ad79277633e26a593171d47/_data",
                "Destination": "/srv",

And if pmm-server is using pmm-data, then the volume filename “Source” should match the one from pmm-data:

[root@centos vagrant]# docker inspect pmm-server | egrep "Source|Destination"
                "Source": "/var/lib/docker/volumes/6df58f0ad81199277892c2285e870f3edf563cc23ad79277633e26a593171d47/_data",
                "Destination": "/srv",

 

If you are not using pmm-data mounted in /srv, then the output will be showing a different mount (or no mount at all) such as:

docker inspect pmm-server | egrep "Source|Destination"
                "Source": "/var/lib/docker/volumes/e0b4cd1c5c65737a24ee10d9e84a6d17a0fc5beff028410c80599840d2c89e67/_data",
                "Destination": "/var/lib/grafana",

 

Backup Existing Data

# create directory for backing up current PMM2 data
mkdir -p /root/pmm-data-backup/srv 

# stop pmm-server
docker stop pmm-server 

# copy PMM2 data outside the container
docker cp pmm-server:/srv /root/pmm-data-backup

# rename container as a backup
docker rename pmm-server pmm-server-backup

# if having a pmm-data container without mount in /srv, then you should backup it up 
docker rename pmm-data pmm-data-backup

 

Restore Into a New pmm-data Mounted in /srv:

# create pmm-data container mounted in /srv
docker create \
   -v /srv \
   --name pmm-data \
   percona/pmm-server:2 /bin/true

# restore data into pmm-data
docker cp /root/pmm-data-backup/srv pmm-data:/

# start new pmm-container using pmm-data container
sudo docker run -d \
   -p 443:443 \
   --volumes-from pmm-data \
   --name pmm-server \
   --restart always \
   percona/pmm-server:2

# restore permissions
docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown root:pmm /srv/clickhouse
docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana
docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana/grafana.db
docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana/png
docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown pmm:pmm /srv/logs
docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown postgres:postgres /srv/logs/postgresql.log
docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres
docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/prometheus

 

Results should be similar to this:

docker exec -it pmm-server ls -l /srv
total 12
drwxr-xr-x.  3 root     root       47 Jun  9 12:49 alertmanager
drwxr-xr-x.  9 root     pmm       140 Jun 10 22:37 clickhouse
drwxrwxr-x.  3 grafana  grafana    69 Jun 10 22:27 grafana
drwxrwxr-x.  2 pmm      pmm      4096 Jun 10 22:26 logs
drwxr-xr-x.  2 root     root      115 Jun  9 12:48 nginx
-rw-r--r--.  1 root     root        6 Jun  9 12:50 pmm-distribution
drwx------. 19 postgres postgres 4096 Jun 10 22:37 postgres
drwxr-xr-x.  4 pmm      pmm        31 Jun  9 12:50 prometheus
drwxr-xr-x.  2 root     root        6 Jun  9 12:49 update

Finally, pmm-server needs to be restarted so that it reloads files with correct permissions:

# restart pmm-server
docker restart pmm-server

Docker should now be working again with no data loss! And the backup of the containers  can be safely removed with:

docker rm pmm-server-backup
docker rm pmm-data-backup

 

Conclusion

It’s strongly suggested to use a separate pmm-data container mounted in /srv for PMM2 to avoid loss of historical data when upgrading/recreating pmm-server. Since the data path for PMM changed in PMM2 compared to PMM1, you might be using a pmm-data container with the wrong mounts (or no pmm-data at all!) which can be easily fixed by using the above procedure.


by Carlos Tutte via Percona Database Performance Blog

Comments

Post a Comment