This past spring I made an attempt to move myself out of the shackles of the
commercial software world and truly embrace open-source. I tried to move my
primary machine off Windows 7, and onto Ubuntu
Linux. I knew the transition wouldn’t be seamless but
I’d heard so many good things about living in a Linux universe that I decided
it was time.
The experiment did not go as well as I might have hoped, and despite my
efforts to stick with it for some time, I eventually had to cut the experiment
short. As I was preparing to re-image my system I started a blog post which I
decided not to post at the time. I’ve included a short excerpt which shows my
state of mind back in May, just after the experiment concluded.
I told myself I was going to stick it out for at least 3 months. But here I
sit, not 3 weeks after making the decision to migrate my primary machine to
Ubuntu, with the Windows 7 installation disk in hand. What could possibly have
brought me to this point? Primarily, time.
It’s going to take me about 8 hours of work to prep all the data on my system
for the transition, wipe the linux partition, re-install windows, re-install
the applications, re-install VMWare, re-install my Linux VMs (I do still have
a use for them!). The problem is, things on linux generally have taken longer
than they should. Some of this is due to the fact that I’m learning, and I’ve
tried to ignore those. Others are generally due to the fit and finish of
Ubuntu.
So what went wrong?
Problem #1 - 10.10 or 11.04?
I generally resist the temptation to move to the latest OS release, but when I
tried setting up a Windows VM under VirtualBox in Ubuntu 10.10 the audio was
mucked up. It seemed a bit slow too, but that may have been my imagination. So
I tried installing the newly minted 11.04. The VM now worked like a charm, but
that was a long multi-step process.
Problem #2 - Virtualization
Trying to set up a virtual machine that would start up at boot time (like a
Windows service or any number of linux daemons) proved a nearly impossible
task. After several hours of searching, tweaking, testing, and ultimately
failing, I decided to abandon the effort and live with manually starting my
VMs.
Problem #3 - File Sharing
Setting up network shares was probably one of the better experiences I had. I
was able to set up a “public” share on the linux machine and access it from
anywhere on the network… as long as I didn’t want to protect it with a
username and password. That was going to require more voodoo and black magic
than I was prepared to endure for such a simple task. Overall, not a bad
experience.
Problem #4 - Flash in Browsers
Like it or not Flash is still an integral part of the web, and Flash in the
browser was just one of those things that never quite worked right. When I
talk about fit and finish of a product, this is what I mean. Blocky artifacts
showing up on video players was the most common issue, though there were other
things like playback and audio problems as well.
Problem #5 - Lack of Air Support
The fact that I felt compelled to write a blog post calling attention to a
tutorial for getting Adobe Air installed under Ubuntu
11.04 speaks
to just how difficult this didn’t need to be. On any other major platform, you
can go to a website and simply click the install button. The rest is
automatic. Not here though.
Problem #6 - Button Clicks
I constantly had problems just clicking on buttons. Sometimes in an
application (Chromium comes to mind) but sometimes just within the Ubuntu
environment itself. This kind of thing makes you start to question the faith
you have in your OS.
Problem #7 - Learning Curve
I suppose it’s a bit unfair to put this here as it’s undoubtedly the same
issue that would come up moving between any two major operating systems. The
bottom line is that I have a young family with whom I like to spend the
majority of my day. That means that when I decide to sit down at the computer
to do something, I don’t really have the time to spend learning how to do
things all over again.
There were a few things that were also pleasant surprises during this whole
thing. Mostly to do with 3rd party applications.
CrashPlan support
CrashPlan was able to seamlessly match up my Windows backup to the Linux file
system. This made it very easy to move everything over. I just hope it works
as well in reverse.
AcidRip
Digitizing DVDs has never been easier. It took a couple of tries to get the
quality settings just where I wanted them, but the process worked out really
well.
Shell
I love the *nix shell, Bash in particular. This is the one thing I will truly
miss when I move back to Windows. Having commands like rsync at my disposal,
and built in SSH support are also fantastic. While this is something that has
to be hacked into a Windows installation, it is available by default on OS X.
In summary…
The availability of good software to do most tasks is one of the key benefits
of moving to an open source experience, but the truth is that the experience
really didn’t live up to my hopes or my expectations. I’m getting to the point
where I want my computing time to be spent creating, not just experimenting
with different ways that I could set up my tool sets. And as time moves on,
the number of free or open-source applications available on the major
commercial platforms like Windows and OS X is growing. Once either of those
operating systems is installed I can do everything I want to do without having
to pay a license for another piece of software – and in many cases the
applications are as good or better than the open-source tools available for
the Linux platforms. Add to that the growing number of applications which
reside in the cloud and are completely browser and platform agnostic and it
starts to become a simple equation for me.
Is it worth the $150 or so that it costs to get my new computer preloaded with
a commercial OS? Yes.
With the latest release of Firefox, Mozilla has decided that we don’t need to
see the “http://” at the beginning of a URL. While this may be true for day-
to-day browsing, it makes copying and pasting URLs a bloody nightmare.
Most applications detect a URL based on it starting with some sort of protocol
directive (http://, https://, ftp://, mailto:). By removing that directive
from the beginning of the URL Mozilla now forces us to type them in as we go,
reducing productivity and generally being a pain in the behind.
For the record, this portion of the URL is still visible for https:// URLs to
help everyone know that pages are encrypted using SSL/TLS. This somehow makes
it even worse in my eyes, since this non-security related behaviour is
different based on whether or not the application is encrypted.
You can, however, correct this abhorrent behaviour with a trip to the Firefox
about:config page.
DISCLAIMER: Read the disclaimer on the about:config page.
Go to the about:config page in Firefox [](http://kdmurray.net/wp-
content/uploads/2011/10/trimUrls–1.png)
In the filter box, type in: browser.urlbar.trimURLs
[](http://kdmurray.net/wp-
content/uploads/2011/10/trimUrls–2.png)
Double-click on the value to change from true to false
[](http://kdmurray.net/wp-
content/uploads/2011/10/trimUrls–3.png)
After making the change, that line will show up in bold to indicate that it
has been changed from the default setting. This is helpful if you want to
restore settings to their default at some point in the future – though in
this case I can’t imagine why.
This site was hacked. While it’s still unclear exactly how it happened, or
precisely when, sometime in the past 6 weeks my blog, at least 2 other
websites and possibly my DreamHost
shell account were all hacked. I’m generally a pretty security conscious
person, but even I get lazy from time to time. It wasn’t clear to me just how
dangerous that laziness could be until this week. I’m going to outline a bit
below some of the issues which may have led to my problems, and talk about the
steps that have now been taken to help prevent them from occurring again in
the future.
Many of us take the time to make sure our operating systems and browsers are
up-to-date and fully patched; but do we take the necessary time to make sure
that all of our software is patched? Particularly things which don’t reside on
our home computers? If you run your own blog, forum or other website and are
responsible for your own updates can you say unequivocally that you are
currently running the latest and greatest version? Software that is out of
date by as little as one revision may have critical vulnerabilities which
could allow for disruption of your site, or even execution of commands on your
web server.
([aside: If you don’t use Secunia’s PSI
product on your home PC
at least once a month, you should.])
Error #2 - Abandoned Web Properties
This goes hand-in-hand with the out-of-date software but is, in some ways, a
bit trickier to prevent. It is far easier to remember to update software on
sites which you update and monitor on a regular basis. It’s far more difficult
to monitor sites which have been, for lack of a better term, abandoned. In my
case there were three separate sites on my account which were running versions
of their software which were more than 12 months out-of-date. The reason was
that I was no longer maintaining these sites and had, in essence, forgotten
they were still there. I had hidden a couple of them by renaming the homepage
which made it look (to the casual observer) like the sites weren’t there but
of course all of the other pages were still in their normal locations and were
full of holes.
Error #3 - Shared User Accounts
Sharing is good, right? Not in this case… I have a several different domains
hosted under a single hosting account. DreamHost is really generous allowing
customers to register any number of domains and attach them to the account. I
host sites for myself, for family and for a couple of organizations I’m
affiliated with. This in and of itself does not cause a problem. The security
hole in my plan was that most of these domains were hosted on a single user
account. This means that if that shared user account gets compromised, all of
the domains which are run on that user account are potentially at risk.
Error #4 - Lack of Backups
The websites had no viable backups. Because no regular backups were being run
of the account, it was virtually impossible to trace when the hack initially
occurred. If there had been regular full or differential backups being made of
the various websites it may have been possible to determine when the initial
attack took place and roll all of the sites back to the way they were before
they were compromised. In addition, if there had been any data loss (there
does not appear to have been) the lack of backups could have meant the loss of
many hours of work.
Error #5 - Reused Credentials
We hear it all the time – do not reuse usernames and passwords on your various
accounts, particularly accounts you care about or are important. Account reuse
increases the chances that a hack on one site can do more wide-spread damage
than the initial compromised password should really allow. My main SSH
credentials were a username and password that I had used on over 100 different
sites and services. I know for sure that one of the web properties I use had
these particular credentials released into the wild. Why didn’t I change the
password? I don’t know. If that was the entry vector, it is quite possible
that a number of other accounts of mine have also been compromised.
Overall Impact
The impact was
(thankfully) minimal. Only two sites of value were compromised, and it appears
that all of the data for those sites is undamaged. A number of other obsolete
sites were compromised as well but as they are no longer actively used they
are of no great loss. It also appears that some sort of mass-mailing script
was being run from the account as well. My server-side user account had
received over 27K “Message Undeliverable” replies from various web servers. I
hate to think how many it was able to send successfully.
The very first step was to ensure that my local machine had not been infected
or compromised. I was pretty sure that it was clean as scans are run every
night, but it would be like trying to wash a car with mud. No amount of
scrubbing with the muddy sponge would get it clean. The machine checked out.
The second step was to change the passwords for all of the users on my hosting
account, and change the main password for the account itself.
Next, data from the websites that needed to be saved was exported. None of the
code for the software running those sites was saved, only the data. There was
no way to tell if the software was clean or compromised so I decided to take
no chances. The application software is not that difficult to install, and I
was willing to take the hit on setting up modules, components and themes anew.
Once the data was backed up I wiped out all of the data on the user accounts
which were being preserved. This meant a full wipe from the file-system from
the operating-system shell on the server. All files and directories including
“hidden” and “special” folders were wiped out. Some of these operations
required the assistance of a DreamHost technician.
Step #1 - Remove all unused or obsolete websites
This was taken care of as part of the cleanup activities mentioned above.
Simply removing the affected websites greatly decreased the attack surface of
the account and reduced the number of attack vectors which could be used to
attack the websites and/or the account.
Step #2 – Remove all un-needed user accounts
In the case of any obsolete sites, test accounts or test databases, these were
removed directly from the hosting provider’s control panel as they would no
longer be needed. Much like response #1, there is no sense in keeping any old
files or data hanging around where they might later become a liability.
Step #3 - Change the passwords again
Once all of the files, scripts, data, databases, directories, logs and
anything else I could think of were removed from the sites, the passwords were
rotated again. This was done in the off-chance that there were cached
credentials or some other form of persistent authentication lurking somewhere
in the ether.
Step #4 - Create new per-domain user accounts
For each of the domains that would be remaining active, a new user account was
created specifically for that user. These accounts would be used to connect to
and install the necessary software on the websites, as well as to run backup
and maintenance scripts. Passwords for these accounts were set to extremely
long strings of random characters as they would not be required for day-to-day
access and maintenance.
Step #5 - Set up public key authentication
For regular access to these sites, I decided to go with public key
authentication. By requiring a private key (stored in an encrypted volume on
my main desktop) and a lengthy but easy-to-remember passphrase I could fairly
safely rely on the same public/private key pair to secure access to all of the
websites. I found out during this step that both PuTTY’s puttygen application
and my hosting provider’s implementation of OpenSSH have an upper-limit on the
length of the passphrase. It is still a very long upper limit, but I was
surprised to find it. If you share access to a website keys can be installed
for each trusted user using the same method.
Step #6 – Change passwords again (optional)
Once the public-key authentication is in place the account passwords can be
changed at will without affecting the state of the affected keys. This means
that I have effectively made the public keys the only viable way of accessing
the site over SSH short of having access to the main hosting provider account
to do a password reset. Admittedly this step is for the very security
conscious (read: paranoid) as I was quite certain at this point that the
passwords on the system at this time had not been compromised. This however is
to be the first step in a regularly scheduled series of password rotations
that the system will handle on my behalf as a part of standard system
maintenance.
Step #7 - Reinstall all server-side software
Once all of the base security measures was in place and tested, I set up the
application software I wanted to run on the web server. The key here is to do
the set up using copies of the software obtained only from trusted sources.
What a trusted source is will vary from software package to software package,
but typically the main project site for an open-source project (not a mirror)
or the vendor website are good places to start. In this case downloading the
latest stable WordPress release from the main website . I made sure not
to rely on previously downloaded installation packages, getting the newest
most up-to-date version I could lay my hands on.
Step #8 - Configure server-side software
Each software package is different, but going through all the configuration
steps for your software package is important: don’t try to short-cut the
process. In the case of WordPress we have to set up a MySQL database, set a
number of hey/hash values which are used for authorization and cookies and
finally set up the user accounts. I wanted to make sure that any passwords,
keys or salt values were set using long randomly-generated strings. In my case
I used the password generation function in LastPass. Other options would
include tools like 1Password, RoboForm or Perfect Paper Passwords . The
longer and more random the string is, the more difficult it will be to crack.
I have been using values from 24 to 64 characters in length depending on the
purpose. If you have a system that assigns default passwords for new user
accounts, be sure to change those default system-generated passwords and
replace them with your own strong credentials at this stage.
Step #9 - Set up extensions and themes for server-side software
Once I got the base configuration is in place it was time to add in the
additional features I required for these sites. In my case it was a collection
of WordPress plugins and themes. It is easy to forget that each extension,
plugin or theme that you add to your website’s software package is in fact
additional software that will be executed when the website is used. Just as
with the base software package it is important to trust the source of your
plugins and themes. If you are suspicious as to the origins of the software,
choose something else. I also added the plugins and themes one at a time
confirming after each step that there were no immediately visible defects.
Step 10 – Automated backup
The next step was to add a backup script for both the website and the
associated database. By building this as a shell script it was possible to
schedule full backups of the various sites and have them run on a set
schedule. For now the script is very simple:
Extract the contents of the database
Zip the website and extracted database into a single archive
Send that file over SFTP to a location off-site from the server
There are other ideas for automation as well, but this post is long enough as
it is. I will save those for later.
Lessons Learned
This could have been much worse. In many ways I count myself very lucky. I
could have had all of my data wiped out, I could potentially have seen
malware/scripts injected into my websites to capture login credentials or
other sensitive information. This attack served as a warning and though I have
had to spend a number of hours rethinking the way my websites are set up and
managed, at the end of the day I will have better control over the sites I
manage, better practices in place for dealing with security, and with any
luck, better personal habits for dealing with information security.
Last, but certainly not least, a big thank you to the folks at DreamHost for
confirming my initial diagnosis, helping to find the possible entry vectors,
providing guidance on cleanup and purging, and just generally doing that great
customer service thing that they do.
[](http://kdmurray.net/wp-
content/uploads/2010/12/lg_optimus_71.jpg)Today Windows Phone handsets around
the world [begin to receive](http://windowsteamblog.com/windows_phone/b/window
sphone/archive/2011/09/27/windows-phone–7–5-mango-update-begins.aspx) their
official Windows Phone 7.5 “Mango” software updates. If you have a WP7 device
check the [Where’s my Update](http://mango.microsoft.com/windowsphone/en-
us/features/update-schedules.aspx)? page to see if your carrier is delivering
“Mango”.
At this point it’s unclear whether the accelerated update
process that worked
for many of us when the “NoDo” update was released will be effective with
“Mango”.
The
latest episode of the AGP is in the feed. This week Dave and I go over a few
things that took place during our long absence and discuss news from Google,
Twitter and Linux. We also have some fantastic apps for mobile, tablet and web
and an illuminating story of recycling and ingenuity.
Dave and I had a great time putting this show together, and I had the
opportunity to produce an episode of the AGP for the first time in well over a
year. It felt good to get my hands dirty again, and I think Dave appreciated
the break. ;)
Over the past year my personal life as undergone some fairly major changes. I
started a new job a little over a year back and there were the obvious changes
that go along with that. But more importantly my wife and I welcomed our first
child into the world and that was a life changing moment. Now, most of you
know that I don’t talk about my personal life in the blog so suffice to say
that we have thoroughly enjoyed our first year as parents. It is a wonderful
experience and we eagerly await every new day to see what will happen next.
One of the things that changes when you have a new baby is the amount of time
you can spend on yourself and your own hobbies and pursuits. I used to spend
upwards of 4–6 hours every day outside of work on the computer blogging,
coding, or otherwise toiling in one digital adventure or another. Now I find
that the number ranges somewhere in the range of 0–2 hours per day. That is a
pretty drastic reduction no matter how you slice it (about 80% for those of
you scoring at home).
There are a number of projects that I have started and stopped over the past
few years each of them trying to build a better mousetrap, or re-make
something from scratch just to see if I could do it. With the limited time
available to me now, I have become more focused on wanting to actually do more
with the time I have – this means not reinventing the wheel every chance I
get.
My wife and I have both found that we have become far more effective with our
time, getting more done with less time than we ever have before. In the past
couple of months I have started to extend that to my digital life as well.
Gone are the days when I focused on a writing a to-do list, a backup utility,
a blogging engine, a photo manager or a disk-erasing tool. There are lots of
great (free) tools out there which can handle those tasks very well, even if
they don’t satisfy all my neurotic desires (like how my historic completed
work tasks should be handled, cataloged and stored for reporting purposes (you
know, for when I will pull metrics on my completed work)).
I have also decided that diving in to learn a new, modern programming language
is probably something that would realistically take more time than I’m willing
to devote to the enterprise. Python, Ruby, Java, and the ASP.NET MVC framework
are all on my list, but are undergoing changes and enhancements so frequently
that I’m having trouble keeping up with what’s out there, nevermind trying to
actually learn the stuff. But I do want to become a productive programmer in
some language outside the rather constrained, and somewhat self-imposed, .NET
bubble in which I have spent the majority of my professional career. Ideally I
would like to write in something that I can port between operating systems
without too much headache. Being able to produce code that will run on
anyone’s machine is a great asset – especially when you have Windows, Mac and
Linux machines in your own house to start with.
So the question is what can I learn that will allow me to:
write code for multiple platforms
grow as a developer
not have to keep up with constant enhancements
The answer I came to was 42 C. It seems to satisfy all of the
criteria above for me in a way that other languages don’t.
C is by nature intended to be a multi-platform system. If you’re able to
confine your applications to CGI or the command-line this is made even easier.
C also requires developers to know much more about how computers and compilers
work than more contemporary languages like C#, Java or Python. Though it
arguably makes programming more difficult, I think it will help me become a
better programmer over time as I learn some of the trickier parts of getting a
computer to do what I want it to do.
The current ANSI standard specification for C was introduced in 1999. This
means that for the past 12 years, the standard for C programming has remained
essentially unchanged. This makes C a good choice for someone who doesn’t have
a great deal of time to keep up with changes and enhancements in the
specification.
For all these reasons, and my own simple curiosity I’m embarking on an
adventure to learn and become proficient in C. I make no assertions that I’m
trying to master the language as I can’t see myself getting beyond the
hobbyist or perhaps open-source contributor stages. I do have some ideas for
the first couple of projects I would like to tackle once I get the basics out
of the way. Hopefully I’ll be able to release some source code back into the
world over the next year or two – after all, I’m in no hurry.
I was having a devil of a time trying to get TweetDeck to even install under
Ubuntu 11.04 until I came across this fantastic
tutorial by
deepok1968 on YouTube. The problem
boils down to the Adobe Air installer not being all that happy with x64
systems in its native state.
If you visit the video page you can get a copy of the commands that he uses. I
stuck them in a bash script so that I can re-use them if I need to do the
install again.
Now I just need to figure out how he got that wicked looking OSX style dock…
I don’t do this kind of thing lightly, but it might be a good idea to post
this on your wall:
Facebook found a problem in the way that it was authenticating applications.
Any time you used an application a token was created that would allow the application to do it’s thing – including posting on your wall, accessing photos or whatever other permissions it requested.
The tokens did not expire and were being “leaked” through normal operation on Facebook.
Anyone who found a token would be able to use it to do the same things that you allowed the application to do – including posting on your wall, accessing photos or whatever other permissions it requested.
It is important to note that Facebook has said there is no evidence that this
has been exploited – yet.
The problem has now been fixed, but all the old tokens could still be usable
until September 2011. You can re-secure your account by simply changing your
Facebook password. This will invalidate any of the existing tokens.
Here’s a quick and dirty implementation of “IsNumeric” in C#. This is one of
those methods that just seems to be missing from C# which appears in so many
other languages.
UPDATE 12-Apr–2011: After some fantastic discussion elsewhere I’ve modified the code to handle a number of additional scenarios. A point was also raised that a combination of Int64.TryParse() _and _ Decimal.TryParse() would accomplish the same thing. They would, almost, but those methods test for valid 64-bit integers and valid 64-bit decimals – they don’t test whether a string is numeric. Feed them a long enough string of numbers and they’ll return false. It’s a pretty fine distinction, I grant that, but I figured since I was writing the code I might as well make it as robust as possible.
public static bool IsNumeric(string s)
{
return IsNumeric(s, false);
}
public static bool IsNumeric(string s, bool allowDecimal)
{
bool result = true;
if (String.IsNullOrEmpty(s))
{
return false;
}
if (s.StartsWith("-"))
{
s = s.Substring(1);
}
char[] chars = s.ToCharArray();
if (allowDecimal)
{
bool decimalFound = false;
foreach (char c in chars)
{
if (c == '.' && !decimalFound)
{
decimalFound = true;
}
else
{
result = result & (char.IsNumber(c));
}
}
}
else
{
foreach (char c in chars)
{
result = result & char.IsNumber(c);
}
}
return result;
}
I built 1439 unit tests for this on the project I built it for
throwing all sorts of weird and null data at it, and it seems to run fairly
well and reasonably quickly. Any comments/suggestions are welcome.
One of the things that has slowed the recent publication of episodes of blog
posts (among other things) has been a desire to spend a good chunk of my on-
the-computer time writing code. I’ve done a few little projects which I’ve put
to use, started a half-dozen others which died on the drawing board, and have
managed to get a couple working just the way I want them to help me with
something that I actually want to get done.
It seems to me that some of the best pieces of code I’ve ever written are the
ones I wrote to solve a problem that I had. The two WordPress plugins I wrote
([Random Image Selector](http://wordpress.org/extend/plugins/random-image-
selector/) and [Admin Links Widget](http://wordpress.org/extend/plugins/admin-
links-sidebar-widget/)) have met with far more success and way more downloads
than I ever would have thought possible.
In that same vein of ‘solving my own problems with code’ I bring you
FeedPublisher. This does only one thing, and it does it reasonably well: It
downloads the contents of an RSS feed and creates an EPUB book out of it,
suitable for transferring to your favourite eBook reader.
The bulk of the idea for this project came from my experiences with an
application called Calibre which was shown to me
by Knightwise. I’m paraphrasing a bit, but he had
said it was the best application to come along for dealing with
eBooks that he’d ever used.
High praise coming from someone who prides himself on finding and using the
best free cross-platform software that the Internet has to offer.
So I tried it out and really did like most of what it did. It can manage
conversions to and from a number of different formats, it deals with PDFs like
nobody’s business, and it had a really cool feature: download and convert
blogs to eBooks.
The first thing I thought was that this would be a brilliant way to handle
eBooks. But as I used it, a couple of things that just weren’t quite what I
wanted. The created eBooks were a bit on the ugly side (and had notices that
they’d been created with Calibre). But the larger problem was that it wasn’t
really possible to schedule the download of the feeds. Calibre had to be
running all the time for you to pick them up. This scheduling problem proved
to be the final push I needed to get off my butt and create FeedPublisher.
FeedPublisher itself is a
command-line app (for Windows only at the moment) which downloads the contents
of a selected list of RSS (or Atom) feeds. Each feed becomes its own ePUB book
containing the contents of the blog posts. This first version works quite well
and rarely crashes but it’s definitely ‘beta’ quality software. Chances are it
will work for you. :)
The project is still in its infancy, but if you’re interested, check it out!
](http://kdmurray.net/wp-content/uploads/2011/09/folder.jpg){kind=link}