Troubleshooting - Manually or Automated?
Boglarka Angalet

Troubleshooting - Manually or Automated?

Time is a limited resource for all IT teams. They must be quick, effective and focused on the right goals. However, when the house is burning - let’s say hackers infected your servers, often you have to throw your projects away and start fire fighting to prevent further damage. But are you?

Shouldn’t a server security software do all this instead of long hours of manual work?

Hacked servers = Losing business

87% of unhappy customers won’t complain to you about any problem with their websites or servers. They just stop doing business with you and move on to your competitors. Confrontation is against human behaviour, you can’t blame them for that.

However, the more customers you lose, the less your business grows. So if you want to address customer churn, you must take care of their security - one of the leading issues.

Your challenge here is to recognize any security issue before your users do and fix them asap.

To tell the truth, the three best ways to do so are: prevention, prevention and prevention.

Attacks getting harder to spot

According to IBM, six out of seven cyberattacks go undetected. That’s 85,7%!

And what’s even more shocking is: 77% of organizations still lack a plan on how to respond to security incidents.


But why is that? Isn’t there enough top-notch sysadmins to cover their servers from head to toe?

The thing is, sysadmins fight on the front lines but server security is such a special area of IT that no one expects them to have master skills about it.

And boy, we know how that feels... 5 years ago, we were running a shared hosting company when we realized this. That’s when BitNinja was born, and that’s how we became an expert team of IT security.

Along the years, we’ve met hundreds of companies in the same shoes. And let me tell you, recognizing that server security is beyond their reach has always been an incredibly important step towards a well-secured server infrastructure.

The hard stuff: identifying infections

Let’s say you have a hacked Linux server. As we've already written about it, compromised servers have many unerring signs to hunt for.

However, what’s next? Because if you’re not the provider switching off the host/server and forgetting about the mess, you have a lot to do. Naturally, this means long hours of troubleshooting, tracing back the sign and cleaning up websites or restoring backups.

Yet, you don’t know who did it, what the server is doing now or what’s still hiding on it. Responding is not always easy - even with an outstanding team behind your back. And that’s why it is so important to protect your servers with an all-in-one tool, taking over the responsibility server-wide, so your team can spend 70% extra time focusing on business.

If you’re new here:

BitNinja is the perfect place to start, as it helps you to get rid of the pain of hacked websites in your daily routine. This software saves you at least 12 hours of troubleshooting per day by reducing the number of server alerts and customer complaints by 98%.

Sign up for a free trial

Server attacks: 9+1 shades of struggle

You’ll find loads of useful articles on the Internet helping you how to identify malicious processes on a hacked Linux box or how to deal with an infected Wordpress site.

We collected you some roadblocks to show where a good security solution can save you time and money by taking care of all these:

1. High load is always suspicious

Constant high server load can be a good sign of your server being part of a botnet. You may tell yourself “uh, that’s a busy server with thousands of websites, that is totally normal”... but believe us, it’s not. With BitNinja, the load can be decreased by over 90%, by eliminating all malicious traffic and cutting down the risk of being compromised.

‘Cus you know: at least 50% of your incoming traffic is generated by bots (20% by bad bots), seizing your server’s capacity:
(Credit)

2. Never believe your eyes

You can never trust information hackers show you. They’re good at hiding their steps, overwriting time stamps and faking legit services and codes. Recognizing these easily takes years of experience, and we all know you have better things to do -- especially when you can get software to do that instead of you.

You might think we’re overreacting about all this, but trust us: there are many obstacles even at the first step of investigations. Most hackers are not idiots: if they infect a file, they tend to restore the file modification date (create_time). If you don’t capture their activity real-time (which manually is technically impossible), you can only work with the ‘access_time’ info later on, which is not as good of a lead.

You’d better leave malware root cause investigations to Sherlock … or BitNinja. ;)

3. Manual investigation sucks

Hallelujah! Found the infected file/malicious process causing problems, but you must prevent it from penetrating the system again. Moreover, you have to find the backdoor and the vulnerability to patch it. Sounds like hours of work again. That’s the process every sysadmin wants to automate - for example, with BitNinja’s Defense Robot.

Even if you have many fractured solutions (like spam filters, WAF, fail2ban, or malware scanners), it’s hard to find correlations and fix the root cause proactively. That’s why keeping up with the hackers is always a rat race for sysadmins.

4. Sleeping malware

And there’s the problem of rarely activated malware and backdoors. With our BitNinja analysis team, we regularly capture botnets, where the CnC server only commands a robot server once a year, just for a few hours. Those are hard to catch manually; investigations are almost impossible!

5. Hiding malware

Anyway, most malware is prepared to be hard to detect. It’s designed to be inconspicuous in server traffic and to hide among legit files easily. Hackers will never take the silver bullet in your hands.
Moreover, if you miss blocking the route of infection, you can bet the same thing will pop up in different locations again and again.

6. Quick actions are essential

In cases like this, log rotation just keeps you under high pressure, as sysadmins usually have a limited time to track back the logs. If your team doesn’t have enough capacity for an instant reaction, you’re lost.

7. Monitoring gives a false sense of security

You may have pricy and complex monitoring systems on board. However, they are not made to identify anomalies hackers induce cautiously. They might alert at high server load, but it’s still you who have to wake up at 2 pm to discover what’s wrong.

8. Practice makes perfect

As we mentioned, server security requires special skills and knowledge. Without that, it can be hard to decide if a log is malicious or not, even if you’re a Linux expert. Plus, you must examine every sign of malicious activity in context to avoid false positives or uncovered attacks.

9. Blurred lines of responsibility

Another thing is, sysadmins usually rather concentrate on the infections of the servers, than the websites hosted on them. However, often the first step of a server infection is a compromised user account. If you don’t shoulder the responsibility and take the account as the user’s scope only, technically it’s you who let the hackers sneak into your servers.
Sounds bad. But it sounds true, doesn’t it?

10. Traffic analysis

Tracking down anomalies in outbound traffic can be resource-intensive and may last for months. Also, it can generate too much data for investigation, mainly if you’ve overlooking signs and the attack did not even happen in the scanned period.

One of the most common malicious outbound traffic is spam. Traffic spikes and users complaining about slow email delivery are inevitable signs of that. However, what if we’re talking about a mail server? How do you know that the increased email traffic is not one of your customers starting a campaign? Takes time to analyze without a good security tool ...

The line of roadblocks is endless and your team’s time is limited. So start making your pro/con list now - if still needed - and introduce the best server security tool to your infrastructure you can.

And as every R&D project contains testing, here’s your chance to sign up for a 1-week BitNinja trial, if you haven’t tried it yet.

Sign up for a free trial

The pioneers of web hosting know that only a layered, pro-active system solution protects their customers from 98% of all web attacks. By eliminating the pain of hacked websites out of your daily routine, BitNinja allows you to transform your security game server-wide in 24 hours and convert your team’s overall capacity to valuable business features.

Share your ideas with us about this article

Previous posts

The Most Famous Vulnerabilities - Cross-Site Scripting (XSS)
It’s been a while since I wrote the previous episode of my blog series. If you are interested in Remote Code Execution, then I definitely recommend reading the previous part. So, here we are again, the moment has come for my final article about cross-site scripting (XSS). What is Cross-Site Scripting (XSS)? Usually, XSS vulnerability occurs when there are untreated inputs and bad cookie usage. So, please let me tell you about a case that happened in 2005 on Myspace. A MySpace user found an XSS vulnerability on the site, and he wrote a payload called „Samy Worm.” This payload was a...
Manual Malware Scan – It's now available on the Dashboard
As a server owner, have you ever had an experience where one or more of the websites hosted by you were reported as a phishing website? Another common issue to get blacklisted on different lists or getting abuse reports which inform you that your servers are attacking other nodes. Perhaps this fact that we’re telling you isn’t new, which is the source of the outgoing attacks is an infection. A lot of hosting providers are frightened when they hear the word „malware,” and they have the reason for getting afraid. Malware is in the spotlight ELK Cloner was the very first malware (previou...