How to Manually Remove Client Instances From Percona Monitoring and Management 2

In this blog post, we will take a look at how we can manually remove a client instance from Percona Monitoring and Management (PMM) version 2. Under normal operations, this shouldn’t be necessary, since a graceful PMM client uninstall shouldn’t leave any lingering instances around. However, it may happen that an instance is destroyed (for instance, an EC2 server that is abruptly terminated; or a vagrant machine or container removed) without executing proper uninstall steps.

Removing the instance won’t delete its associated data, but it will prevent the server from continuing to try to poll data from a non-existent client.

Nomenclature

Some PMM-related terms used below are:

• exporter/agent – process(es) in charge of making metrics (or data) available to the PMM server. Some examples are:
• • • mysqld_exporter – exports metrics for MySQL
    • mysql_slowlog_agent – exports query-related data for MySQL
• service – a group of agents, working together to provide data from one database. Some examples are:
• • • MySQL
    • MongoDB
    • PostgreSQL
• node – a group of services, working together to provide data from one instance (which may have many different database processes running)

Graceful Termination

First, let’s go over the steps we should take to avoid running into issues that will require manual intervention. There are two ways in which we can let the PMM server know we no longer want an instance or process to be monitored. These cases are for instances that are still working correctly (i.e.: we can access the client nodes, and use the pmm-admin client tool).

1- The first way is to remove the pmm2-client package altogether, meaning that you don’t want any agent at all to continue working in that instance.

For RHEL/CentOS:

shell> yum remove pmm2-client

For Debian/Ubuntu:

shell> apt-get remove pmm2-client

2- The second way is to remove a service in particular. First, use pmm-admin list to get the list of services running. An example output is:

shell> pmm-admin list
Service type  Service name         Address and port  Service ID
MySQL         node1-mysql          127.0.0.1:3306    /service_id/f5c1add5-e0f4-4858-a718-18c426490f67

Agent type                  Status     Agent ID                                        Service ID
pmm_agent                   Connected  /agent_id/f30fd2ba-63d1-4fbe-bf8f-309e9bea5a7b
node_exporter               Running    /agent_id/ea8ce2ab-414d-4351-b882-7554b361543a
mysqld_exporter             Running    /agent_id/03f85d08-8093-4e8e-8735-e51e902144f0  /service_id/f5c1add5-e0f4-4858-a718-18c426490f67
mysql_slowlog_agent         Waiting    /agent_id/17568192-33aa-4675-a0dc-6b111e29ca21  /service_id/f5c1add5-e0f4-4858-a718-18c426490f67

Let’s say we want to remove the MySQL service (note that it is not possible to remove one agent, only, since the remove subcommand operates at service level). We need to issue the following command:

shell> pmm-admin remove mysql node1-mysql
Service removed.

In the following section, we’ll explore how to remove them when things go awry.

Manual Removal of Nodes

If the instance was not cleanly deregistered from PMM, it can indefinitely linger on in the Home Dashboard and node/service drop-down lists, while graphs will show no data being populated. After the node is deleted, it may continue to show in the Home Dashboard if there was recent data still collected. If this is the case, you will have to wait until the data retention period purges the data (but at least you won’t see it after that :)).

If we no longer have access to the instance, we should resort to the pmm-admin inventory “hidden” feature. In this case, we won’t have direct access to the old client node, but we can use the pmm-admin client tool from any other currently-working node.

It is not really a hidden feature, though, since we can consult the tool about it:

shell> pmm-admin inventory --help

However, it will not show listed as a subcommand if we check pmm-admin –help (hence, the quotes around hidden). This functionality is not intended for day-to-day operations, and should only be used while there is no other way to remove lingering phantom nodes. The Dev team is working on fixing this, but in the meantime, we thought we should share these steps.

Removing the Node

The only thing we need for this is a working pmm-admin command from any node. We can list all nodes subscribed to PMM with the following command:

shell> pmm-admin inventory list nodes
Nodes list.

Node type     Node name            Address           Node ID
GENERIC_NODE  node2                192.168.54.33     /node_id/854eff2d-ce11-4e82-82e7-dfa500584aae
GENERIC_NODE  node1                192.168.54.32     /node_id/c0e5414c-67e9-41c2-ba2b-02d251d3de2c
GENERIC_NODE  pmm-server           127.0.0.1         pmm-server

And consult the help outputs on how to proceed:

shell> pmm-admin inventory remove node --help
usage: pmm-admin inventory remove node [<flags>] [<node-id>]

Remove node from inventory
[...]
      --force                  Remove node with all dependencies
[...]

Let’s suppose we want to remove node2, so we’ll need to execute the following command:

shell> pmm-admin inventory remove node --force \
         /node_id/854eff2d-ce11-4e82-82e7-dfa500584aae
Node removed.

Note how we are using its “Node ID” string as the last argument, which we previously got from the list nodes output above.

At this point, agents related to that node will no longer be scraped by PMM server, and should not appear as unhealthy services under the Prometheus targets page (located at http://<your_PMM_URL>/prometheus/targets).

Manual Removal of Nodes (Another Way)

Another thing we can do to overcome this is to use the –force argument while using the same node name to re-add the client. In this way, we will be effectively overriding the orphaned client with a new instance of it. Of course, this means that we will need to have a new client to add, which won’t always be the case. The command would look like:

shell> pmm-admin config --force [other_args] \
       IP_ADDRESS generic NODE_NAME

In this case, choose NODE_NAME to be the same as the orphaned node, and it will be overwritten.

Conclusion

If you want to avoid these kinds of issues, it’s always best to ensure a clean uninstall procedure is run. But if you find out it’s too late for that, you can use the inventory commands, while we provide a cleaner way to work around this situation.

Download Percona Monitoring and Management today!

by Agustín via Percona Database Performance Blog