Suspect BitNinja behind increased server load?
Anita Batari

Suspect BitNinja behind increased server load?

As you know, providing all-in-one server security, BitNinja protects 3000+ Linux web servers worldwide, capturing 100 million incidents a month and keeping 1.7 million suspicious IP addresses in its blocklists to protect you and your customers.

Mixing that up with custom infrastructures, configurations, and software in each leads to high load problems sometimes. Despite the best intentions - we always endeavor for the best solutions and we test our novelties constantly before and after each and every release -, these things happen. Most of them are temporary and everything falls back to normal very soon, but there are a few cases when it’s not.

We’re not letting you down this time either. So we’d like to give you a short guide about how to handle and prevent these – because in some cases you can really eliminate this side-effect!

Of course, the very first step always should be to make sure, what the root of the problem is. After excluding every other possible reason, you can use this guide to handle any BitNinja related issues.

And of course, we’re always open to answer your questions.

Symptoms

We can distinguish the reason of the problem based on the symptoms and our logging system. When we receive a complaint, first of all, we have to know what kind of high load can we’re talking about.

There are huge differences between high CPU, memory, disk or network usage.

Main reasons

In lots of cases, we can group these cases by the ‘guilty module’.

IPset

In the past, this was the most common cause of high load. While a spike in the system load is normal right after setup – it generally drops back to normal – within a few minutes. The reason for the higher load is because BitNinja stores the Greylist – and other IP lists in kernel space – and the server has to rearrange its memory for storing the IP sets. Normally, this is a relatively quick procedure.

Using BitNinja simultaneously with other programs that aggressively modify the IPset, is known to cause issues as well. For example, we know that Dome9 drops every iptables rules – other than its own – and as a result, Dome9 is incompatible with BitNinja, and this may cause high load in some cases.

Log Analysis

This module tends to be a bit CPU-hungry –but don’t be afraid.

Usually you won’t notice it at all – however, when your server generates hundreds of log entries in one second, the SenseLog module (Log Analysis) has to - in turn - work harder which can raise the CPU usage by a few percentages. Improper log rotation may cause similar symptoms. In addition, the size of the log file also factors in.

The reason for this is because the module that is currently utilizing regular expression, matches on the analyzed log lines to protect the server against Wordpress, Joomla and cPanel brute-force attacks, Shellshock code injection and directory traversal attacks, as well as many other potential vulnerabilities.

If you notice you are having this issue and you’re not hosting any Joomla or Wordpress websites on your server – you try disabling some of the supervisors that are responsible for preventing attacks against the above-mentioned services. You can configure the module by modifying the config.ini file located in the /etc/bitninja/SenseLog folder, in the section called Enable/Disable Supervisors. If you would like to disable the Joomla-login filter you should add the following line:

disabled[] = ‘ApacheJoomlaLogin’

Note 1: You can find the description of every supervisor on our documentation site.

After you have finished editing the configuration file, you should simply reload the SenseLog module using BitNinja CLI:

bitninjacli --module=SenseLog –-reload

Note 2: We are planning to make this workflow more user-friendly soon.

Malware Detection

We suggest manual scanning right after you turn on this module. When you start the Malware Detection module the first time, the system has to reinitialize inotify processes. This is a CPU and disc intensive task and can take hours to complete.

Malware Detection should traverse all directories and set a flag on it to be monitored. We advise enabling this module during an idle period when a slightly increased load or I/O performance drop is not causing any problems. You can avoid some potential, future inconveniences, by checking smaller directories instead of the whole /home folder.

Are you hosting a lot of dynamic and changing files on your server, and notice the CPU consumption is high

It’s because of the Malware detection too. The inotify process has to communicate with the kernel super intensively, while the scanning runs—using only one thread of the CPU.

Note 3: We’re currently working on the implementation of multi-threading of the processes, which will accelerate the entire process.

Other cases – High network traffic

During an incoming attack - due to the nature of BitNinja protection a drop in the network throughout may occur.

What should you do?

We are always happy to help you and will be able to accelerate the process of our investigation into your concern if you make sure to send us all of the following details regarding your particular case.

  • All the files from the server’s /var/log/bitninja folder in the problematic period contain useful information for our experts.
  • By sending the result of the command below executed during a high-load period we can see the incoming connections which will help us find out more about the nature of the problem:
    netstat -n --inet --inet6 
  • With the output of htop command we can see how many BitNinja related processes are running – and how many resources are they using.

Upcoming solutions: Maldet and LogAnalysis refactoring

Our development team is working on both of Malware Detection and SenseLog modules to help reduce resource consumptions.

We began with custom memory allocation of Malware Detection, but unfortunately, it was not enough to obtain a definitive result so far. But… we won’t give up!

Both of the above-mentioned modules will be refactored and accelerated by the end of this year with multi-thread running.

Share your ideas with us about this article

Previous posts

Test your security knowledge
Start the New Year with this security quiz to test your knowledge and get a chance to receive the latest edition of ‘The Art of Server Protection’ e-book, or -- win our BitNinja Server Security for 1 month, absolutely free.  What a great way to start the New Year! Our ninjas created a short quiz to test your knowledge; it contains 9 questions which are - more or less - related to security. At the end, don’t forget to enter your email address, because, without that, we’re unable to send you your prize. Every participant who scores 9/9 on the quiz will receive our CEO’s book, ‘The Art...
Describe your infrastructure as code
As we promised before, our article series inspired by V-day is continuing. Those who are provisioning servers day by day, certainly have some doubts about their process: being time-consuming, non-repeatable, hard to test or simply just something is going wrong in the existing infrastructure during the provisioning. There are opportunities to test failover or rollback processes, but… Through the years, a lot of providers have elaborated their own processes and solutions in order to accelerate deployment cycles. So-called provisioning scripts are everywhere, and helped with the well-k...