New Malware Monitoring Tool – Auditd is Replacing Inotify

It takes 99.9% less memory usage and 99.9% less time to set up… Unbelievable, but it’s possible with the new malware monitoring tool.

While BitNinja believes in the power of prevention and our proactive modules are very robust, we also want to provide an all-in-one server security service. We know that malware infections are a pain point for every server owner because they have very visible signs.

“There is always room for improvement.” So, we won’t lie to you. Many old and new users ask our help to clean malware from their servers. That’s the reason why we dedicated several projects to improve our Malware Detection system.

One of these projects is implementing the auditd in the Malware Detection module, which will completely take inotify’s place in the near future.

What is auditd?

Auditd is the user space component to the Linux Auditing System. It is an access monitoring and accounting for Linux, developed and maintained by RedHat. It was designed to integrate pretty tightly with the kernel and watch for interesting system calls, for example, file changes. It can start within 1 second and requires only a few kbytes to run.

The audisp process not only signs up for the Audit daemon events but is also able to complete actions, so there are a lot of opportunities in auditd for future developments too.

From now on, BitNinja users can use auditd instead of inotify in the Malware Detection module to monitor the file changes and find the malware.

What was the problem with inotify?

Here comes a real story: Some of our customers stopped using our Malware Detection feature because inotify consumed a lot of memory (even more than 3 GB!) and CPU. Also, it could take 10-20 hours while the inotify set up its watches (which are responsible for monitoring file changes) or it sometimes couldn’t even start for some mysterious reason.

In order to solve these problems, a PoC project was started 6 months ago and the integrated version became ready 2 months ago. Since then, we tested auditd with the help of our VIP partners. As the feedback was quite promising, we are now sharing this new tool with the public. 🙂

Why is auditd better than inotify?

This table sums up clearly why we are moving to auditd:

inotify auditd
memory usage high (even 3 GB) very low (only a few kbytes)
CPU usage moderately high low
time to start hours (even 10-20) instant (within 1 sec)

What’s more, auditd allows us to log the UID and PID, so it’s an extra bonus for auditd. 😉

Compatibility

Right now, the auditd can be used on the following distros:

  • CentOS 6-7
  • Debian 9-10
  • Ubuntu 16+

Unfortunately, there is a kernel bug in Debian 7-8 where the file changes are logged in a different format, so auditd can’t process it.

How to enable the auditd

Auditd is not enabled by default yet, but users can use it with some very simple commands.

Temporarily:

1. You can change inotify to auditd temporarily with the command

bitninjacli --module=MalwareDetection --use-auditd

2. After enabling the auditd, please check that the inotify process has been killed. If this disappears from the output of the htop, then it’s OK:

3. Also, a new process (auditisp) should appear:

4. Run the following command:

auditctl -l

The output should be like this:

5. Check the PID in /var/log/bitninja/mod.audisp_plugin.log

6. If all of these were good, then let’s do a test:

Create a file and make a modification (example: touch testfile.txt then echo testmodification >> textfile.txt)

If it appears in the inotify log, the auditd is working well. Both actions (file creation and change) will appear as MODIFY. (Note: auditd doesn’t have its own log file, it writes into the inotify logs.)

You can check the inotify log with this command:

tail -f /var/log/bitninja/inotify/inotify.log

Note: This CLI command enables the auditd temporarily. After a module restart, the inotify will start automatically.

Permanently:

1. If you are satisfied with the results and want to use auditd as the default file system monitor tool, you need to change the config (/etc/bitninja/MalwareDetection/config.ini). Please insert these lines to the config file:

[FileSystemMonitor]
monitor_type = 'auditd'
monitor_order[] = 'auditd'
monitor_order[] = 'inotify'
monitor_order[] = 'nullMonitor'

Note: Make sure that the monitor type contains only auditd and that auditd is the first one in the monitor order and there is no ; at the beginning of the rows.

2. After you changed the config, you need to restart the agent in order to apply the new settings.

service bitninja restart

Future plans with auditd

After a little fine-tuning, we’d like to make auditd the default file system monitor. Also, we love this tool so much because there are a lot of opportunities in it. We already have plans about which other parts of BitNinja could use auditd too (for example: protect SSH, virtual patch Linux vulnerabilities, implement it into the Log Analysis module, etc).

But, it’s for the future. What you can do now is try out auditd and enjoy the benefits of this feature.

And a little spoiler: This is not the only development that upgrades the performance of our Malware Detection system. 😉 Stay tuned, a lot of mind-blowing projects are on their way.