One of our developers has encountered with an issue deriving from the usual process of system upgrade, ocurring in case of rpm-based systems, while configuring one of our clients’ software. It’s reasons and solution are pretty understandable and easy, but still may affect more of our customers without their awareness to it.
One of our clients asked us to configure BitNinja for them. After our developer almost finished with the task, he realized that the load on the server is unusually high. While he was digging deeper into this, he found the causes of the problem that may occur on the servers of more owners.
The high load has been caused by the SenseLog module of BitNinja, though this is not a module that is capable of doing such things in case of normal configuration, so our developer reviewed the logs generated by it, hoping to find anything that answers the increased load.
BitNinja logs have increased to 8-10 GBs by than, as they were not rotated since the software’s installation. Usually, logs are rotated because the file management is quite I(nput)/O(utput) costly. ( It does matter if we need to read a comic book or the Star Wars+ Harry Potter series on our journey by bus ) Furthermore, during the rotation a new empty file is created for the upcoming logs, while the former ones are being renamed and compressed. After a couple of days we can delete the previous logfiles without hesitation, if needed.
Firstly, he thought that the BitNinja logrotate rules somehow did not get into the /etc/logrotate.d/Bitninja directory, but opposing his predictions they were there, which made the whole issue more interesting and complex. After this, he reviewed the logfiles in the rotation rules, and found that the rotation did not take place.
The rotation should happen on a daily basis conducted by a cron, which can be found in the /etc/cron.daily/ folder under the name “logrotate”. In case of our customer, this cron could not be run, neither the logrotate.rpmnew executable that can be found next to it.
When starting BitNinja, the SenseLog searches the file and collects all the IPs that performed malicious activities and sends them to the BitNinja Center. This way, the incidents we already recorded get processed more than once.
Due to Cpanel or rpm upgrades, the contents of the logrotate file have been altered, and the system could not unlock the changes, so it left the decision-making on the admin/owner of the server. Unfortunately, this did not happen. The contents of the two files are relatively the same, except one line in the older file, which is a reference to another application that became unnecessary in the meantime.
The owner/admin needs to decide which version they are about to use ( The new rpmnew. is suggested )
So, if they assign the following commands the issue will be solved.:
# cd /etc/cron.daily
# mv logrotate.rpmnew logrotate
# logrotate -f /etc/logrotate.conf
