Fix your log rotate rules and eliminate high load

Nikoletta Szabo

One of our developers has encountered with an issue deriving from the usual process of system upgrade, ocurring in case of rpm-based systems, while configuring one of our clients’ software. It’s reasons and solution are pretty understandable and easy, but still may affect more of our customers without their awareness to it.

The Story

One of our clients asked us to configure BitNinja for them. After our developer almost finished with the task, he realized that the load on the server is unusually high. While he was digging deeper into this, he found the causes of the problem that may occur on the servers of more owners.

The high load has been caused by the SenseLog module of BitNinja, though this is not a module that is capable of doing such things in case of normal configuration, so our developer reviewed the logs generated by it, hoping to find anything that answers the increased load.

The Problem Itself

BitNinja logs have increased to 8-10 GBs by than, as they were not rotated since the software’s installation. Usually, logs are rotated because the file management is quite I(nput)/O(utput) costly. ( It does matter if we need to read a comic book or the Star Wars+ Harry Potter series on our journey by bus ) Furthermore, during the rotation a new empty file is created for the upcoming logs, while the former ones are being renamed and compressed. After a couple of days we can delete the previous logfiles without hesitation, if needed.

Firstly, he thought that the BitNinja logrotate rules somehow did not get into the /etc/logrotate.d/Bitninja directory, but opposing his predictions they were there, which made the whole issue more interesting and complex. After this, he reviewed the logfiles in the rotation rules, and found that the rotation did not take place.

The rotation should happen on a daily basis conducted by a cron, which can be found in the /etc/cron.daily/ folder under the name “logrotate”. In case of our customer, this cron could not be run, neither the logrotate.rpmnew executable that can be found next to it.

Effects

  • you can run out of empty space where the /var directory is
  • whatever that writes or reads the logs can cause high load because it takes a huge effort to browse more GB logs

When starting BitNinja, the SenseLog searches the file and collects all the IPs that performed malicious activities and sends them to the BitNinja Center. This way, the incidents we already recorded get processed more than once.

What caused the problem?

Due to Cpanel or rpm upgrades, the contents of the logrotate file have been altered, and the system could not unlock the changes, so it left the decision-making on the admin/owner of the server. Unfortunately, this did not happen. The contents of the two files are relatively the same, except one line in the older file, which is a reference to another application that became unnecessary in the meantime.

The Solution

The owner/admin needs to decide which version they are about to use ( The new rpmnew. is suggested )

So, if they assign the following commands the issue will be solved.:

# cd /etc/cron.daily
# mv logrotate.rpmnew logrotate
# logrotate -f /etc/logrotate.conf

Share your ideas with us about this article

Previous posts

Malware Museum
Today’s malwares are designed to be silent, unnoticeable and effective without drawing the attention of users to their maleficent activities. They steal our credit card and personal details without us realizing it. Although, it was not always the popular way to infect servers/computers. A couple of decades ago viruses usually featured colorful pictures, scrambled codes or even a statement on the corruption of your computer.       Mikko Hermanni Hyppönen, a Finnish chief resource officer has recently collected the viruses from the 1980’s and 1990’s and put them to...
How strong is a „strong” password? 1/2
The idea   Authentication methods have gone through huge development in the past few years and I believe that the spread of IoT devices will bring further advancement. Currently, however, the most widely used authentication method on the Internet is password-based authentication and we know that this method is far from being ideal – at least in the way many people use it. Let’s imagine a website on which people can create accounts. The web application stores a non-decryptable hash (or more precisely the output of a hash function) of the password given by the user. A hash funct...