Permissions Issue When Upgrading MongoDB with Custom dbPath or logPath
In Linux distributions, installing MongoDB for the first time using rpm/dpkg will create a conf file, dbPath, logPath and the init scripts on default paths. This includes the creation of a mongod user/group and provides mongod permissions to both the default dbPath (/var/lib/mongo) and logPath (/var/log/mongodb/mongod.log). Then you edit the settings in the config file at /etc/mongod.conf for custom settings like dbPath in different locations and start mongod service as you need.
But this is a little different when you are doing an upgrade or changing between Percona Server for MongoDB (PSMDB) and upstream MongoDB as they include the removal of the existing package and install of the new version. This article explains the problem in some cases with restored user mongod permissions when you are using custom dbPath or logPath locations when reinstalling the packages, and how to overcome it.
Behavior When Upgrading MongoDB Package
We know that uninstalling the MongoDB package would remove the related packages. But especially when you are uninstalling PSMDB, the user mongod is also dropped along with the packages which cause the existing directories to lose permissions and it is left with the uid/gid of the dropped mongod user/group as shown below.
Before Uninstall:
[root@app ~]# ls -l /var/lib/mongo/ total 16092 -rw-------. 1 mongod mongod 16384 Jan 1 13:46 collection-0-1422095431970586989.wt -rw-------. 1 mongod mongod 32768 Jan 1 13:46 collection-0-3711456140137791318.wt -rw-------. 1 mongod mongod 16384 Jan 1 13:46 collection-0--6887184941376867266.wt -rw-------. 1 mongod mongod 16384 Jan 1 13:46 collection-10--2978246447481368990.wt -rw-------. 1 mongod mongod 4096 Jan 1 13:46 collection-11-1422095431970586989.wt ... ... [root@app ~]# systemctl stop mongod.service [root@app ~]#
Now remove the MongoDB Package:
[root@app ~]# yum remove percona-server-mongodb* Loaded plugins: fastestmirror Resolving Dependencies --> Running transaction check ---> Package percona-server-mongodb.x86_64 0:4.0.13-7.el7 will be erased ---> Package percona-server-mongodb-mongos.x86_64 0:4.0.13-7.el7 will be erased ---> Package percona-server-mongodb-server.x86_64 0:4.0.13-7.el7 will be erased … …
After removing the packages, check the user permissions for the mongodb files. They are now assigned with 996:993 which was uid:gid of the dropped mongod user/group:
[root@app ~]# ls -ld /var/log/mongo/ drwxr-x---. 2 999 997 4096 Jan 1 13:57 /var/log/mongo/ [root@app ~]# ls -ltrh /var/lib/mongo/ total 16M -rw-------. 1 999 997 21 Jul 6 23:12 WiredTiger.lock -rw-------. 1 999 997 45 Jul 6 23:12 WiredTiger -rw-------. 1 999 997 114 Jul 6 23:12 storage.bson -rw-------. 1 999 997 16K Sep 20 12:44 index-9--2978246447481368990.wt -rw-------. 1 999 997 16K Sep 20 12:44 index-5--2978246447481368990.wt -rw-------. 1 999 997 16K Sep 20 12:44 index-3--2978246447481368990.wt -rw-------. 1 999 997 16K Sep 20 12:44 index-15--2978246447481368990.wt ... ...
And when you are reinstalling the PSMDB or upstream MongoDB packages, it would install the packages and create the default directories and mongod user again if not available:
[root@app ~]# percona-release enable psmdb-42 release * Enabling the Percona Server for MongoDB 4.2 repository <*> All done! [root@app ~]# [root@app ~]# yum install percona-server-mongodb Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: repos-va.psychz.net * epel: iad.mirror.rackspace.com * extras: repos-va.psychz.net * updates: repos-va.psychz.net percona-release-noarch ... ...
Now check the permissions of the db files on default dbpath:
[root@app ~]# ls -lad /var/lib/mongo/ drwxr-x---. 4 mongod mongod 4096 Dec 20 14:53 /var/lib/mongo/
Default Behavior of MongoDB Installation
If we look into the source code, the installation of the MongoDB package through rpm or through yum/apt-get repository creates the mongod user/group as system user/group as mentioned with SYS_UID_MIN/SYS_UID_MAX and SYS_GID_MIN/SYS_GID_MAX variables (for CentOS, the variables could be found in /etc/login.defs file). The MongoDB code related to it is shown below:
if ! /usr/bin/id -g mongod &>/dev/null; then /usr/sbin/groupadd -r mongod fi if ! /usr/bin/id mongod &>/dev/null; then /usr/sbin/useradd -M -r -g mongod -d /var/lib/mongo -s /bin/false -c mongod mongod > /dev/null 2>&1 fi
After that, the MongoDB-related directories and files are created and given with the mongod user permissions, as mentioned in the below code snippet, to the default ones. This applies to both PSMDB and upstream MongoDB installations.
Upstream MongoDB Code:
https://github.com/mongodb/mongo/blob/master/rpm/mongodb-org.spec
mkdir -p $RPM_BUILD_ROOT/var/lib/mongo mkdir -p $RPM_BUILD_ROOT/var/log/mongodb mkdir -p $RPM_BUILD_ROOT/var/run/mongodb touch $RPM_BUILD_ROOT/var/log/mongodb/mongod.log %attr(0755,mongod,mongod) %dir /var/lib/mongo %attr(0755,mongod,mongod) %dir /var/log/mongodb %attr(0755,mongod,mongod) %dir /var/run/mongodb
PSMDB Code:
install -m 750 -d %{buildroot}/var/log/mongo
touch %{buildroot}/var/log/mongo/mongod.log
install -m 750 -d %{buildroot}/%{mongo_home}
%attr(0750,mongod,mongod) %dir %{mongo_home}
%attr(0750,mongod,mongod) %dir /var/log/mongo
%attr(0640,mongod,mongod) %ghost /var/log/mongo/mongod.log
Issue Occurs When Using Custom db/log file Paths:
When you are using custom dbPath or logPath files after dropping the existing PSMDB package, the mongod user is dropped as mentioned above. So after the uninstall, and before reinstalling the MongoDB package again, there are some rare cases listed below that can cause permission issues to MongoDB files:
- The mongod user is created manually as ordinary user (not using -r option with useradd command which creates system user).
- drop mongodb packages
- create mongod user again as ordinary user
- install mongodb packages
- Creating some other system users before installing the MongoDB package. i.e. the newly-created system user might use the same uid/gid of the previously dropped mongod user. This will cause the new mongod user to have different uid/gid while installing the packages.
-
- drop mongodb packages
- create some other system users (which overwrites the pid/gid of dropped mongod user while removing MongoDB package)
- install mongodb package
Also, the third scenario is when you have created a mongod user manually as an ordinary user even before installing MongoDB for the first time, which might cause the said issue with the permission for the custom db/log files.
An Example of the Issue
See the example below for the same. The custom dbPath is /data/mongodb:
[root@app 7]# mkdir /data/mongodb [root@app 7]# groupadd -r mongod [root@app 7]# useradd -M -r -g mongod -d /var/lib/mongo -s /bin/false -c mongod mongod > /dev/null 2>&1 [root@app 7]# chown -R mongod.mongod /data/mongodb [root@app 7]# [root@app 7]# ls -ld /data/mongodb/ drwxr-xr-x. 2 mongod mongod 4096 Jan 1 14:44 /data/mongodb/
Install PSMDB 4.0 for the first time and check the directory permission again:
[root@app 7]# yum install percona-server-mongodb Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.fibergrid.in * epel: sg.fedora.ipserverone.com ..... ..... Complete! [root@app 7]# ls -lad /var/lib/mongo/ drwxr-x---. 4 mongod mongod 4096 Dec 20 14:53 /var/lib/mongo [root@app 7]# [root@app 7]# ls -ld /data/mongodb drwxr-xr-x. 2 mongod mongod 4096 Jan 1 14:44 /data/mongodb/ [root@app 7]# [root@app 7]# grep mongo /etc/passwd mongod:x:999:997:mongod:/var/lib/mongo:/bin/false [root@app 7]#
Edit the conf file with custom dbPath and start mongod service:
[root@app 7]# vi /etc/mongod.conf [root@app 7]# [root@app 7]# grep dbPath /etc/mongod.conf dbPath: /data/mongodb
[root@app 7]# service mongod start Redirecting to /bin/systemctl start mongod.service [root@app 7]# service mongod stop Redirecting to /bin/systemctl stop mongod.service
Now remove the MongoDB Package:
[root@app 7]# yum remove percona-server-mongodb* Loaded plugins: fastestmirror Resolving Dependencies --> Running transaction check ---> Package percona-server-mongodb.x86_64 0:4.0.13-7.el7 will be erased ---> Package percona-server-mongodb-mongos.x86_64 0:4.0.13-7.el7 will be erased ---> Package percona-server-mongodb-server.x86_64 0:4.0.13-7.el7 will be erased ---> Package percona-server-mongodb-shell.x86_64 0:4.0.13-7.el7 will be erased .... .... Complete!
Then check the permission for the data directory and as expected, they look like below – 999:997 which was uid/gid of dropped mongod user/group as part of “yum remove “.
[root@app 7]# ls -ld /data/mongodb/ drwxr-xr-x. 4 999 997 4096 Jan 1 15:13 /data/mongodb/ [root@app 7]#
As we have seen in the scenarios above and as per scenario 2, now create a system user/group called testusr as follows and you will be surprised to see the permission of /data/mongodb. This is because the user was created with the same uid/gid of dropped mongod user:
[root@app 7]# useradd -M -r testusr [root@app 7]# grep test /etc/passwd testusr:x:999:997::/home/testusr:/bin/bash [root@app 7]# ls -ld /data/mongodb/ drwxr-xr-x. 4 testusr testusr 4096 Jan 1 15:13 /data/mongodb/
Upgrade to PSMDB 4.2 package now:
[root@app 7]# percona-release enable psmdb-42 release * Enabling the Percona Server for MongoDB 4.2 repository <*> All done! [root@app 7]# [root@app 7]# yum install percona-server-mongodb Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: repos-va.psychz.net * epel: iad.mirror.rackspace.com * extras: repos-va.psychz.net * updates: repos-va.psychz.net percona-release-noarch | 2.9 kB 00:00:00 percona-release-x86_64 | 2.9 kB 00:00:00 psmdb-42-release-noarch | 2.9 kB 00:00:00 psmdb-42-release-x86_64 | 2.9 kB 00:00:00 (1/2): psmdb-42-release-noarch/7/primary_db | 1.1 kB 00:00:00 (2/2): psmdb-42-release-x86_64/7/primary_db | 16 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package percona-server-mongodb.x86_64 0:4.2.2-3.el7 will be installed --> Processing Dependency: percona-server-mongodb-tools = 4.2.2-3.el7 for package: percona-server-mongodb-4.2.2-3.el7.x86_64 .... .... .... Complete!
Check the permission for the custom data directory path (/data/mongodb) and to the default path(/var/lib/mongo). We notice that the mongod permission for the custom path /data/mongodb was not restored on its own. Also the /etc/mongod.conf is moved to /etc/mongod.conf.rpmsave while installing the PSMDB package, and a new file /etc/mongod.conf is created from the package installation. So you need to edit the conf file as needed or restore the old conf file:
[root@app 7]# ls -ld /data/mongodb/ drwxr-xr-x. 4 testusr testusr 4096 Jan 1 15:13 /data/mongodb/ [root@app 7]# ls /etc/mongod* /etc/mongod.conf /etc/mongod.conf.rpmsave [root@app 7]# vi /etc/mongod.conf
The following error will be received when starting with the lost permission:
[root@app 7]# grep dbPath /etc/mongod.conf
dbPath: /data/mongodb
[root@app 7]# systemctl start mongod
Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
[root@app 7]# tail /var/log/mongo/mongod.log
2020-01-01T15:23:31.929+0000 I CONTROL [initandlisten] build environment:
2020-01-01T15:23:31.929+0000 I CONTROL [initandlisten] distarch: x86_64
2020-01-01T15:23:31.929+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-01-01T15:23:31.929+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongod.pid" }, storage: { dbPath: "/data/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongo/mongod.log" } }
2020-01-01T15:23:31.930+0000 I STORAGE [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating
2020-01-01T15:23:31.930+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2020-01-01T15:23:31.930+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2020-01-01T15:23:31.930+0000 I - [initandlisten] Stopping further Flow Control ticket acquisitions.
2020-01-01T15:23:31.930+0000 I CONTROL [initandlisten] now exiting
2020-01-01T15:23:31.930+0000 I CONTROL [initandlisten] shutting down with code:100
Solution:
You have to manually provide mongod user permission and the old setting in config file as before to start with the new version of MongoDB package.
[root@app 7]# chown -R mongod.mongod /data/mongodb [root@app 7]# service mongod start
Update the Package Instead of Remove & Install
The other good news is you don’t need to follow these steps if you are updating the package instead of removing and installing the next version. This path doesn’t remove the mongod user and also doesn’t move the /etc/mongod.conf to /etc/mongodb.conf.rpmsave. So you can use this method to upgrade.
[root@app 7]# percona-release enable psmdb-42 release * Enabling the Percona Server for MongoDB 4.2 repository <*> All done! [root@app 7]# yum update percona-server-mongodb Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: repos-va.psychz.net * epel: iad.mirror.rackspace.com * extras: repos-va.psychz.net * updates: repos-va.psychz.net psmdb-42-release-noarch | 2.9 kB 00:00:00 psmdb-42-release-x86_64 | 2.9 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package percona-server-mongodb.x86_64 0:4.0.13-7.el7 will be updated ---> Package percona-server-mongodb.x86_64 0:4.2.2-3.el7 will be an update --> Processing Dependency: percona-server-mongodb-tools = 4.2.2-3.el7 for package: percona-server-mongodb-4.2.2-3.el7.x86_64 --> Processing Dependency: percona-server-mongodb-mongos = 4.2.2-3.el7 for package: percona-server-mongodb-4.2.2-3.el7.x86_64 --> Processing Dependency: percona-server-mongodb-shell = 4.2.2-3.el7 for package: percona-server-mongodb-4.2.2-3.el7.x86_64 --> Processing Dependency: percona-server-mongodb-server = 4.2.2-3.el7 for package: percona-server-mongodb-4.2.2-3.el7.x86_64 --> Running transaction check ---> Package percona-server-mongodb-mongos.x86_64 0:4.0.13-7.el7 will be updated ---> Package percona-server-mongodb-mongos.x86_64 0:4.2.2-3.el7 will be an update ---> Package percona-server-mongodb-server.x86_64 0:4.0.13-7.el7 will be updated ---> Package percona-server-mongodb-server.x86_64 0:4.2.2-3.el7 will be an update ---> Package percona-server-mongodb-shell.x86_64 0:4.0.13-7.el7 will be updated .. .. .. Complete!
While writing this blog post, there were a lot of discussions internally and per them, I have raised this JIRA https://jira.percona.com/browse/PSMDB-546 to update the documentation and reflect this update path instead of removing and installing the newer version.
CONCLUSION
When you are doing upgrade/downgrade, check the permissions of the data directory and the log files before starting the MongoDB service. If needed, provide the permissions of mongod user to the db files to start the service without issue. Hope this helps you if you are facing such issues.
by Vinodh Krishnaswamy via Percona Database Performance Blog
Comments
Post a Comment