The Problem

In retrospect I can see five things I did wrong, and all of them can be traced back to laziness or perhaps, to be less forbidding, they can be traced back to actions taken (or not taken) for the sake of convenience.

Error #1 – Out-of-date Software

Many of us take the time to make sure our operating systems and browsers are up-to-date and fully patched; but do we take the necessary time to make sure that all of our software is patched? Particularly things which don’t reside on our home computers? If you run your own blog, forum or other website and are responsible for your own updates can you say unequivocally that you are currently running the latest and greatest version? Software that is out of date by as little as one revision may have critical vulnerabilities which could allow for disruption of your site, or even execution of commands on your web server.

([aside: If you don't use Secunia's PSI product on your home PC at least once a month, you should.])

Error #2 – Abandoned Web Properties

This goes hand-in-hand with the out-of-date software but is, in some ways, a bit trickier to prevent. It is far easier to remember to update software on sites which you update and monitor on a regular basis. It’s far more difficult to monitor sites which have been, for lack of a better term, abandoned. In my case there were three separate sites on my account which were running versions of their software which were more than 12 months out-of-date. The reason was that I was no longer maintaining these sites and had, in essence, forgotten they were still there. I had hidden a couple of them by renaming the homepage which made it look (to the casual observer) like the sites weren’t there but of course all of the other pages were still in their normal locations and were full of holes.

Error #3 – Shared User Accounts

Sharing is good, right? Not in this case… I have a several different domains hosted under a single hosting account. DreamHost is really generous allowing customers to register any number of domains and attach them to the account. I host sites for myself, for family and for a couple of organizations I’m affiliated with. This in and of itself does not cause a problem. The security hole in my plan was that most of these domains were hosted on a single user account. This means that if that shared user account gets compromised, all of the domains which are run on that user account are potentially at risk.

Error #4 – Lack of Backups

The websites had no viable backups. Because no regular backups were being run of the account, it was virtually impossible to trace when the hack initially occurred. If there had been regular full or differential backups being made of the various websites it may have been possible to determine when the initial attack took place and roll all of the sites back to the way they were before they were compromised. In addition, if there had been any data loss (there does not appear to have been) the lack of backups could have meant the loss of many hours of work.

Error #5 – Reused Credentials

We hear it all the time – do not reuse usernames and passwords on your various accounts, particularly accounts you care about or are important. Account reuse increases the chances that a hack on one site can do more wide-spread damage than the initial compromised password should really allow. My main SSH credentials were a username and password that I had used on over 100 different sites and services. I know for sure that one of the web properties I use had these particular credentials released into the wild. Why didn’t I change the password? I don’t know. If that was the entry vector, it is quite possible that a number of other accounts of mine have also been compromised.

Overall Impact

The impact was (thankfully) minimal. Only two sites of value were compromised, and it appears that all of the data for those sites is undamaged. A number of other obsolete sites were compromised as well but as they are no longer actively used they are of no great loss.
It also appears that some sort of mass-mailing script was being run from the account as well. My server-side user account had received over 27K “Message Undeliverable” replies from various web servers. I hate to think how many it was able to send successfully.

The Cleanup

The cleanup had to be done in phases, addressing each of the five defects individually. Some of them were very easy to change, others required quite a bit more effort to implement and verify. However before any of the remediation could begin, the site needed to be cleansed.

The very first step was to ensure that my local machine had not been infected or compromised. I was pretty sure that it was clean as scans are run every night, but it would be like trying to wash a car with mud. No amount of scrubbing with the muddy sponge would get it clean. The machine checked out.

The second step was to change the passwords for all of the users on my hosting account, and change the main password for the account itself.

Next, data from the websites that needed to be saved was exported. None of the code for the software running those sites was saved, only the data. There was no way to tell if the software was clean or compromised so I decided to take no chances. The application software is not that difficult to install, and I was willing to take the hit on setting up modules, components and themes anew.

Once the data was backed up I wiped out all of the data on the user accounts which were being preserved. This meant a full wipe from the file-system from the operating-system shell on the server. All files and directories including “hidden” and “special” folders were wiped out. Some of these operations required the assistance of a DreamHost technician.

Step #1 – Remove all unused or obsolete websites

This was taken care of as part of the cleanup activities mentioned above. Simply removing the affected websites greatly decreased the attack surface of the account and reduced the number of attack vectors which could be used to attack the websites and/or the account.

Step #2 – Remove all un-needed user accounts

In the case of any obsolete sites, test accounts or test databases, these were removed directly from the hosting provider’s control panel as they would no longer be needed. Much like response #1, there is no sense in keeping any old files or data hanging around where they might later become a liability.

Step #3 – Change the passwords again

Once all of the files, scripts, data, databases, directories, logs and anything else I could think of were removed from the sites, the passwords were rotated again. This was done in the off-chance that there were cached credentials or some other form of persistent authentication lurking somewhere in the ether.

Step #4 – Create new per-domain user accounts

For each of the domains that would be remaining active, a new user account was created specifically for that user. These accounts would be used to connect to and install the necessary software on the websites, as well as to run backup and maintenance scripts. Passwords for these accounts were set to extremely long strings of random characters as they would not be required for day-to-day access and maintenance.

Step #5 – Set up public key authentication

For regular access to these sites, I decided to go with public key authentication. By requiring a private key (stored in an encrypted volume on my main desktop) and a lengthy but easy-to-remember passphrase I could fairly safely rely on the same public/private key pair to secure access to all of the websites. I found out during this step that both PuTTY’s puttygen application and my hosting provider’s implementation of OpenSSH have an upper-limit on the length of the passphrase. It is still a very long upper limit, but I was surprised to find it. If you share access to a website keys can be installed for each trusted user using the same method.

Step #6 – Change passwords again (optional)

Once the public-key authentication is in place the account passwords can be changed at will without affecting the state of the affected keys. This means that I have effectively made the public keys the only viable way of accessing the site over SSH short of having access to the main hosting provider account to do a password reset. Admittedly this step is for the very security conscious (read: paranoid) as I was quite certain at this point that the passwords on the system at this time had not been compromised. This however is to be the first step in a regularly scheduled series of password rotations that the system will handle on my behalf as a part of standard system maintenance.

Step #7 – Reinstall all server-side software

Once all of the base security measures was in place and tested, I set up the application software I wanted to run on the web server. The key here is to do the set up using copies of the software obtained only from trusted sources. What a trusted source is will vary from software package to software package, but typically the main project site for an open-source project (not a mirror) or the vendor website are good places to start. In this case downloading the latest stable WordPress release from the main website <link>.
I made sure not to rely on previously downloaded installation packages, getting the newest most up-to-date version I could lay my hands on.

Step #8 – Configure server-side software

Each software package is different, but going through all the configuration steps for your software package is important: don’t try to short-cut the process. In the case of WordPress we have to set up a MySQL database, set a number of hey/hash values which are used for authorization and cookies and finally set up the user accounts. I wanted to make sure that any passwords, keys or salt values were set using long randomly-generated strings. In my case I used the password generation function in LastPass. Other options would include tools like 1Password, RoboForm or Perfect Paper Passwords <Links>. The longer and more random the string is, the more difficult it will be to crack. I have been using values from 24 to 64 characters in length depending on the purpose.
If you have a system that assigns default passwords for new user accounts, be sure to change those default system-generated passwords and replace them with your own strong credentials at this stage.

Step #9 – Set up extensions and themes for server-side software

Once I got the base configuration is in place it was time to add in the additional features I required for these sites. In my case it was a collection of WordPress plugins and themes. It is easy to forget that each extension, plugin or theme that you add to your website’s software package is in fact additional software that will be executed when the website is used. Just as with the base software package it is important to trust the source of your plugins and themes. If you are suspicious as to the origins of the software, choose something else.
I also added the plugins and themes one at a time confirming after each step that there were no immediately visible defects.

Step 10 – Automated backup

The next step was to add a backup script for both the website and the associated database. By building this as a shell script it was possible to schedule full backups of the various sites and have them run on a set schedule. For now the script is very simple:

1. Extract the contents of the database
2. Zip the website and extracted database into a single archive
3. Send that file over SFTP to a location off-site from the server

There are other ideas for automation as well, but this post is long enough as it is. I will save those for later.

Lessons Learned

This could have been much worse. In many ways I count myself very lucky. I could have had all of my data wiped out, I could potentially have seen malware/scripts injected into my websites to capture login credentials or other sensitive information. This attack served as a warning and though I have had to spend a number of hours rethinking the way my websites are set up and managed, at the end of the day I will have better control over the sites I manage, better practices in place for dealing with security, and with any luck, better personal habits for dealing with information security.

Last, but certainly not least, a big thank you to the folks at DreamHost for confirming my initial diagnosis, helping to find the  possible entry vectors, providing guidance on cleanup and purging, and just generally doing that great customer service thing that they do.

During a recent vacation I had the opportunity to help my brother-in-law with a project. It took a few iterations but we managed to get some code written which fulfilled the need in just a few nights’ work. Though the complexity was about the same as my own small personal projects, this project was different.  It was for a client — albeit a client who I wasn’t charging money for the work.  Someone else was going to read and maintain this code so I knew that I had to take some measures that I hadn’t been used to taking recently.

The first was refactoring. I refactored the crap out of this application — and for the most part things worked out nicely. I was able to turn all my poorly named and one-character variable names into elements and objects with meaningful monikers.

Commenting was the next step.  After the refactoring was completed I could move on to the next step of the application which was to put in some descriptive comments in the applications.  These comments describe not the what, but the why.   This is the approach I’ve tried to take relying on good naming within the application to describe the what, and comments to provide rationale.

The final thing I did was take my brother-in-law through the code.  He has a background in software but has never done much in the way of OO (he writes Cobol) and hasn’t ever worked with the .NET languages.  In doing the walkthrough we came across a few minor issues and a couple of small enhancements for the application.

All told I spent about 25 hours working on this project, and in that short amount of time, I really did begin to regain my programming chops.  I’ve got a ways to go yet before I get to a level that I’d be comfortable with, but it’s a start.

I was wanting to get some source control in place for a few of my personal projects.  I chose to go with Subversion for my source control server for a few reasons, not the least of which was that my hosting company supports auto-configuration of SVN repositories, so I was able to get that set up in just a couple of minutes.  That left me some time to contemplate how I would access the repository from the client.

I’m running Visual Studio 2008 on my development machine and this gives me the ability to use plugins for the IDE, a feature that is sadly missing from the express editions.  There were a couple of good options available for SVN plugins, VisualSVN which is the 800lb gorilla in this space, and the open-source option CollabNet’s AnkhSVN.  Given the fact that this was for personal exploration of the toolset, the open source (free) option was the obvious choice.

The setup for AnkhSVN was quick and painless, and when the IDE opened up it put options for source control right in the menus where they were nice and easy to find.  I created a project, and selected the “add to Subversion” checkbox, entered the necessary credentials and created the project in my SVN repository.

When in Visual Studio, the AnkhSVN controls are located on a tab at the bottom of the IDE, alongside other solution-wide functionality like the To-do list, output window etc.  This pane tracks all of the changes (adds, deletes and updates) that you’ve made to the solution files.  This is extra handy as a review when you’re ready to make your commits back to the repository.  By quickly scanning the list of changes you’re able to write solid commit comments to provide some decent documentation for you, or those who come after you.

I’m still relatively new to Subversion and AnkhSVN, but I’m looking forward to exploring them in more detail — maybe I’ll even do a podcast episode about it!