Percona XtraDB Cluster 8.0 Behavior Change for pxc-encrypt-cluster-traffic
Percona has enforced stronger security in Percona XtraDB Cluster (PXC) 8, but this requires some attention during the rollout of the new server version, so let see the why and what.
In PXC there are two different kinds of traffic: client-server exchange (ie: application traffic), and replication traffic. The latter refers to any SST/IST, write-set, and other service messages the nodes exchange.
In PXC 5.7 it is possible to activate SSL encryption by enabling the variable pxc-encrypt-cluster-traffic by following the instructions.
In PXC 8, we choose to enable encryption by default on all replication traffic, to have the highest out-of-box security enforcement.
While this is an obvious improvement, PXC 8 needs to behave differently from its predecessor PXC 5.7. This article is meant to help you in becoming more familiar with the new version of PXC.
Let’s Start With The Basics:
On a PXC cluster, in order to have the nodes connecting and to be able to communicate when SSL is enabled, the certificates must be the same on all of the nodes that are supposed to be part of the cluster, no matter if you are using PXC 8 or PXC 5.7.
PXC can generate the certificates by itself if you let it do so (by default). The files are generated in the data directory.
The problem here is that each node will generate a different set of certificates, which means each set of *.pem files on a node is different from the others. Given that, we need to replace the *.pem files by choosing a single set from one of the nodes and copy them over to each other node, otherwise, the nodes will not be able to communicate.
A best practice is to define a different location for your files, such that you can easily keep in sync between nodes, and that they will not be removed or overwrite during SST, something like:
[mysqld] ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem OR ssl_capath = /etc/mysql/certs/ [client] ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/client-cert.pem ssl-key=/etc/mysql/certs/client-key.pem
In order to generate the *.pem file at initialization, your my.cnf should not have any reference to the above, or PXC will expect to find the *.pem files at the given path.
An easy solution is to comment the settings, initialize the database, generate the *.pem, and copy them to the location you have decided.
Let us now consider the four different scenarios:
- Upgrade from 5.7 no pxc-encrypt-cluster-traffic
- Upgrade from 5.7 with pxc-encrypt-cluster-traffic
- New install PXC8 without pxc-encrypt-cluster-traffic
- New install with pxc-encrypt-cluster-traffic
Note: When upgrading a PXC 5.7 online via the addition of PXC 8 nodes, ensure you set PXC 8 to READ_ONLY mode. Application writes cannot be sent to PXC 8 while there are PXC 5.7 nodes in the same cluster. For example, include the following entry in your server’s /etc/my.cnf: PXC8 node(s) cannot receive writes until a PXC 5.7 node is part of the cluster.
read_only=ON
Case 1 – Upgrade From PXC 5.7 No SSL
As mentioned previously, the difference between PXC 5.7 and 8 here is that the encryption is enabled by default.
If you try to have a PXC8 node join with defaults, you will get this error:
2020-05-01T13:49:21.864627Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://10.0.0.5:4567 failed: asio.ssl:336031996: 'unknown protocol' ( 336031996: 'error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol')
Indeed, an SSL issue because our 5.7 cluster IS NOT using SSL. We need to change the defaults for our PXC8, modifying the pxc-encrypt-cluster-traffic settings to:
pxc-encrypt-cluster-traffic = OFF
Doing so, our new PXC8 node will join and perform SST:
2020-05-01T13:52:34.489148Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST......... 2020-05-01T13:52:41.571224Z 0 [Note] [MY-000000] [WSREP-SST] ............Waiting for SST streaming to complete! 2020-05-01T13:52:45.399411Z 1 [Note] [MY-000000] [Galera] Flushing memory map to disk... 2020-05-01T13:53:17.771422Z 0 [Note] [MY-000000] [Galera] 3.1 (node2): State transfer to 0.2 (node4) complete. 2020-05-01T13:53:17.774292Z 0 [Note] [MY-000000] [Galera] Member 3.1 (node2) synced with group. 2020-05-01T13:53:17.785772Z 0 [Note] [MY-000000] [WSREP-SST] Preparing the backup at /opt/mysql_instances/gal8/data//sst-xb-tmpdir 2020-05-01T13:53:24.424885Z 0 [Note] [MY-000000] [WSREP-SST] Moving the backup to /opt/mysql_instances/gal8/data/ 2020-05-01T13:53:24.550030Z 0 [Note] [MY-000000] [WSREP-SST] Running post-processing........... 2020-05-01T13:53:24.563099Z 0 [Note] [MY-000000] [WSREP-SST] Opting for mysql_upgrade (sst): local version (8.0.18) != donor version (5.7.28)
It will also identify that the source is a 5.7 node and will perform an internal upgrade.
Case 2 – Upgrade From 5.7 With pxc-encrypt-cluster-traffic
So, you have SSL already active in your 5.7 cluster, and you want to upgrade to 8.
If your *.pem files are in the data directory, given you are migrating, this is the right moment to do the right thing and place them out of it in the new nodes.
In the new nodes, move them for best practice and define the path in the MySQL config file.
Steps:
- Create a safe place for the file like /etc/mysql/cert
- Copy file over from the donor
- Add the following to the my.cnf
[mysqld] ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem OR ssl_capath = /etc/mysql/certs/ [client] ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/client-cert.pem ssl-key=/etc/mysql/certs/client-key.pem
- grant mysql user only read/write access
- Start your PXC8 node
At this point, the node will do SST, then the internal upgrade, and will join the 5.7 cluster.
Case 3 – New Install PXC8 Without pxc-encrypt-cluster-traffic
If you want to start a new PXC8 cluster without encryption, you must explicitly disable pxc_encrypt_cluster_traffic.
Given so, on all nodes add to your my.cnf:
pxc_encrypt_cluster_traffic = OFF
And start the cluster as usual.
Case 4 – New Install With pxc-encrypt-cluster-traffic
As mentioned earlier, by default, PXC8 has pxc-encrypt-cluster-traffic enabled.
When booting the first node, you have two options:
- Let the server create the certificate for you
- Follow the instructions here (https://www.percona.com/doc/percona-xtradb-cluster/LATEST/security/encrypt-traffic.html#generate-keys-certs) to manually create them, and place them in the location you have chosen (not data directory) in all nodes.
If you do not have already the *.pem files and you like to have a new set automatically created, just:
- Comment the ssl entry (as above) in your my.cnf. PXC will create a new set for you.
- After the boot, you can stop the server
- Move the files in a safe place outside the data directory (like /etc/mysql/cert)
- Copy over them on any other node.
- Restart your first node
- Let the other nodes to join
Remember to grant permission to mysql user only:
chown -R mysql:mysql /etc/mysql/certs
Your new cluster will be up in a few seconds and all the replication/messaging traffic between nodes will be encrypted.
Conclusion
Enabling encrypted traffic between PXC 8 servers is expected to improve the security profile of your PXC environment but comes with additional complexity when upgrading or bootstrapping a new cluster. Operations, that if not done in the correct order, in case of an upgrade, may affect cluster stability.
Here my focus was to give simple instructions and understanding of this minor behavior change, which may cause a waste of time and frustration. PXC8 has many other additional features that we are going to illustrate in our blog, so follow us.
Please keep in mind that PXC8 is GA since April 2020 and PXC 8.0.19 is the first GA release, and as such we advise you to use it for heavy testing, but do not put it in production unless you have covered all the possible use cases and load scenarios.
Reference
Upgrading Percona XtraDB Cluster
by Marco Tusa via Percona Database Performance Blog
Comments
Post a Comment