Expensive Data Loss Be Damned – Backups Saved the Day

The importance of data backup is one of those things that we all agree with, but don’t always act on.

After all, backups are one of the best ways to protect your business from expensive data loss and unplanned downtime - a phenomenon that costs businesses over $1.7 trillion dollars annually.

However, not every business has a complete backup and recovery plan in place - either due to time or cost required to set one up. At Liquid Web, we firmly believe in the importance of data backup, but we are constantly encountering customers with little to no backups for their business.

With that in mind, I have for you today a story of how backups saved one of our customers (we’ll call them Hardware-R-US) from losing everything. They counted themselves lucky that they had backups ready to go after finding themselves the victims of an attack.

Hopefully by the end of this cautionary tale, you feel inspired to create a solid backup and recovery plan for your business.

A Late Night Phone Call

Hardware-R-Us used their managed dedicated server hosting to run their entire website and store quite a bit of their business information - all of which amounted to about 200 GB of data.

Their server was core managed, which is our level of support for customers who want us to manage their infrastructure and hardware, but don’t need us to provide a cPanel or software support. Because their server was core managed, much of the setup was custom and differed from how we as a company normally build servers.

For example, Hardware-R-Us used special hard drives and handled any changes to their software and code on their own.

We soon dived into their setup, however, after receiving a panicked, late night phone call.

One of our third shift support technicians answered the phone and learned that Hardware-R-Us’ website was completely gone.

Needless to say, they were incredibly upset. Our technician quickly investigated and discovered that a hacker had gained root access, likely via a PHP exploit, and deleted their entire site.

While looking in their logs, our technician could see “rm filename” repeated over and over again, thousands of times. This command proved his suspicions that a hacker had gone through every file of their site and systematically deleted each one. There was no recovering their data.

“It was tragic”, our technician said, remembering what he found when he searched their logs. “All of their data was just gone.”

The Attack

Using the combined powers of our Support and Security teams, we were able to narrow down the attack vectors to three possibilities. Sadly, the vulnerabilities we discovered were wholly preventable - and likely would have been easily prevented had the server been fully managed by our team or had the customer’s technicians known to do a few simple things:

1. Outdated Version of PHP

Hardware-R-Us' server was running an old version of PHP, version 5.3.8 specifically. This version, at the time, had a known critical vulnerability that could allow an attacker to inject and execute code remotely. Since Hardware-R-Us had not updated their PHP version, this was a highly likely attack vector, and we recommended updating their software immediately to protect against future attacks.

2. No ModSecurity

We also discovered the server was not running ModSecurity, which is an Apache module that provides intrusion detection and prevention for web apps. ModSecurity shields web applications from known and unknown attacks, such as SQL injection attacks, cross-site scripting, path traversal attacks, and more. ModSecurity is something we include on all fully managed Liquid Web servers, so we not only recommended installing it immediately, but we also offered to help fine tune the module until it worked perfectly in their system.

3. Lack of Additional PHP Protection

All of Liquid Web’s fully managed servers also come with additional protection in the form of Suhosin, an advanced protection system for PHP installations designed to protect servers and users from known and unknown flaws in PHP applications and the PHP core. However, Hardware-R-Us' server did not include this additional protection and we recommended implementing it as soon as possible.

Saved by Offsite Backups

Fortunately for Hardware-R-Us, they were using our offsite backup solution. We were able to restore their entire site using the offsite backups.

Because everything had been deleted from their server, we restored their entire configuration - all 200 GB of data from backups. It was a lengthy process due to the amount of data, but at the end of a few hours, their site was returned to them in full working order and we could begin the process of protecting them from future attacks. 

Protect Your Business With Offsite Backups Through Acronis Cyber Backup