This site was hacked. While it’s still unclear exactly how it happened, or precisely when, sometime in the past 6 weeks my blog, at least 2 other websites and possibly my DreamHost shell account were all hacked. I’m generally a pretty security conscious person, but even I get lazy from time to time. It wasn’t clear to me just how dangerous that laziness could be until this week. I’m going to outline a bit below some of the issues which may have led to my problems, and talk about the steps that have now been taken to help prevent them from occurring again in the future.
The Problem
In retrospect I can see five things I did wrong, and all of them can be traced back to laziness or perhaps, to be less forbidding, they can be traced back to actions taken (or not taken) for the sake of convenience.
Error #1 – Out-of-date Software
Many of us take the time to make sure our operating systems and browsers are up-to-date and fully patched; but do we take the necessary time to make sure that all of our software is patched? Particularly things which don’t reside on our home computers? If you run your own blog, forum or other website and are responsible for your own updates can you say unequivocally that you are currently running the latest and greatest version? Software that is out of date by as little as one revision may have critical vulnerabilities which could allow for disruption of your site, or even execution of commands on your web server.
([aside: If you don't use Secunia's PSI product on your home PC at least once a month, you should.])
Error #2 – Abandoned Web Properties
This goes hand-in-hand with the out-of-date software but is, in some ways, a bit trickier to prevent. It is far easier to remember to update software on sites which you update and monitor on a regular basis. It’s far more difficult to monitor sites which have been, for lack of a better term, abandoned. In my case there were three separate sites on my account which were running versions of their software which were more than 12 months out-of-date. The reason was that I was no longer maintaining these sites and had, in essence, forgotten they were still there. I had hidden a couple of them by renaming the homepage which made it look (to the casual observer) like the sites weren’t there but of course all of the other pages were still in their normal locations and were full of holes.
Error #3 – Shared User Accounts
Sharing is good, right? Not in this case… I have a several different domains hosted under a single hosting account. DreamHost is really generous allowing customers to register any number of domains and attach them to the account. I host sites for myself, for family and for a couple of organizations I’m affiliated with. This in and of itself does not cause a problem. The security hole in my plan was that most of these domains were hosted on a single user account. This means that if that shared user account gets compromised, all of the domains which are run on that user account are potentially at risk.
Error #4 – Lack of Backups
The websites had no viable backups. Because no regular backups were being run of the account, it was virtually impossible to trace when the hack initially occurred. If there had been regular full or differential backups being made of the various websites it may have been possible to determine when the initial attack took place and roll all of the sites back to the way they were before they were compromised. In addition, if there had been any data loss (there does not appear to have been) the lack of backups could have meant the loss of many hours of work.
Error #5 – Reused Credentials
We hear it all the time – do not reuse usernames and passwords on your various accounts, particularly accounts you care about or are important. Account reuse increases the chances that a hack on one site can do more wide-spread damage than the initial compromised password should really allow. My main SSH credentials were a username and password that I had used on over 100 different sites and services. I know for sure that one of the web properties I use had these particular credentials released into the wild. Why didn’t I change the password? I don’t know. If that was the entry vector, it is quite possible that a number of other accounts of mine have also been compromised.
Overall Impact
The impact was (thankfully) minimal. Only two sites of value were compromised, and it appears that all of the data for those sites is undamaged. A number of other obsolete sites were compromised as well but as they are no longer actively used they are of no great loss.
It also appears that some sort of mass-mailing script was being run from the account as well. My server-side user account had received over 27K “Message Undeliverable” replies from various web servers. I hate to think how many it was able to send successfully.
The Cleanup
The cleanup had to be done in phases, addressing each of the five defects individually. Some of them were very easy to change, others required quite a bit more effort to implement and verify. However before any of the remediation could begin, the site needed to be cleansed.
The very first step was to ensure that my local machine had not been infected or compromised. I was pretty sure that it was clean as scans are run every night, but it would be like trying to wash a car with mud. No amount of scrubbing with the muddy sponge would get it clean. The machine checked out.
The second step was to change the passwords for all of the users on my hosting account, and change the main password for the account itself.
Next, data from the websites that needed to be saved was exported. None of the code for the software running those sites was saved, only the data. There was no way to tell if the software was clean or compromised so I decided to take no chances. The application software is not that difficult to install, and I was willing to take the hit on setting up modules, components and themes anew.
Once the data was backed up I wiped out all of the data on the user accounts which were being preserved. This meant a full wipe from the file-system from the operating-system shell on the server. All files and directories including “hidden” and “special” folders were wiped out. Some of these operations required the assistance of a DreamHost technician.
Step #1 – Remove all unused or obsolete websites
This was taken care of as part of the cleanup activities mentioned above. Simply removing the affected websites greatly decreased the attack surface of the account and reduced the number of attack vectors which could be used to attack the websites and/or the account.
Step #2 – Remove all un-needed user accounts
In the case of any obsolete sites, test accounts or test databases, these were removed directly from the hosting provider’s control panel as they would no longer be needed. Much like response #1, there is no sense in keeping any old files or data hanging around where they might later become a liability.
Step #3 – Change the passwords again
Once all of the files, scripts, data, databases, directories, logs and anything else I could think of were removed from the sites, the passwords were rotated again. This was done in the off-chance that there were cached credentials or some other form of persistent authentication lurking somewhere in the ether.
Step #4 – Create new per-domain user accounts
For each of the domains that would be remaining active, a new user account was created specifically for that user. These accounts would be used to connect to and install the necessary software on the websites, as well as to run backup and maintenance scripts. Passwords for these accounts were set to extremely long strings of random characters as they would not be required for day-to-day access and maintenance.
Step #5 – Set up public key authentication
For regular access to these sites, I decided to go with public key authentication. By requiring a private key (stored in an encrypted volume on my main desktop) and a lengthy but easy-to-remember passphrase I could fairly safely rely on the same public/private key pair to secure access to all of the websites. I found out during this step that both PuTTY’s puttygen application and my hosting provider’s implementation of OpenSSH have an upper-limit on the length of the passphrase. It is still a very long upper limit, but I was surprised to find it. If you share access to a website keys can be installed for each trusted user using the same method.
Step #6 – Change passwords again (optional)
Once the public-key authentication is in place the account passwords can be changed at will without affecting the state of the affected keys. This means that I have effectively made the public keys the only viable way of accessing the site over SSH short of having access to the main hosting provider account to do a password reset. Admittedly this step is for the very security conscious (read: paranoid) as I was quite certain at this point that the passwords on the system at this time had not been compromised. This however is to be the first step in a regularly scheduled series of password rotations that the system will handle on my behalf as a part of standard system maintenance.
Step #7 – Reinstall all server-side software
Once all of the base security measures was in place and tested, I set up the application software I wanted to run on the web server. The key here is to do the set up using copies of the software obtained only from trusted sources. What a trusted source is will vary from software package to software package, but typically the main project site for an open-source project (not a mirror) or the vendor website are good places to start. In this case downloading the latest stable WordPress release from the main website <link>.
I made sure not to rely on previously downloaded installation packages, getting the newest most up-to-date version I could lay my hands on.
Step #8 – Configure server-side software
Each software package is different, but going through all the configuration steps for your software package is important: don’t try to short-cut the process. In the case of WordPress we have to set up a MySQL database, set a number of hey/hash values which are used for authorization and cookies and finally set up the user accounts. I wanted to make sure that any passwords, keys or salt values were set using long randomly-generated strings. In my case I used the password generation function in LastPass. Other options would include tools like 1Password, RoboForm or Perfect Paper Passwords <Links>. The longer and more random the string is, the more difficult it will be to crack. I have been using values from 24 to 64 characters in length depending on the purpose.
If you have a system that assigns default passwords for new user accounts, be sure to change those default system-generated passwords and replace them with your own strong credentials at this stage.
Step #9 – Set up extensions and themes for server-side software
Once I got the base configuration is in place it was time to add in the additional features I required for these sites. In my case it was a collection of WordPress plugins and themes. It is easy to forget that each extension, plugin or theme that you add to your website’s software package is in fact additional software that will be executed when the website is used. Just as with the base software package it is important to trust the source of your plugins and themes. If you are suspicious as to the origins of the software, choose something else.
I also added the plugins and themes one at a time confirming after each step that there were no immediately visible defects.
Step 10 – Automated backup
The next step was to add a backup script for both the website and the associated database. By building this as a shell script it was possible to schedule full backups of the various sites and have them run on a set schedule. For now the script is very simple:
- Extract the contents of the database
- Zip the website and extracted database into a single archive
- Send that file over SFTP to a location off-site from the server
There are other ideas for automation as well, but this post is long enough as it is. I will save those for later.
Lessons Learned
This could have been much worse. In many ways I count myself very lucky. I could have had all of my data wiped out, I could potentially have seen malware/scripts injected into my websites to capture login credentials or other sensitive information. This attack served as a warning and though I have had to spend a number of hours rethinking the way my websites are set up and managed, at the end of the day I will have better control over the sites I manage, better practices in place for dealing with security, and with any luck, better personal habits for dealing with information security.
Last, but certainly not least, a big thank you to the folks at DreamHost for confirming my initial diagnosis, helping to find the possible entry vectors, providing guidance on cleanup and purging, and just generally doing that great customer service thing that they do.
