My mentor as regards data backup is David Cairns, Director of Computer Services at the University of Prince Edward Island.

Many years ago David and I were talking about backup systems, and David suggested that backups should be designed to address the situation when hard disk storage systems fail, not if they fail. In other worlds, design as though you expect failure to happen. Any minute now. David’s wisdom has stayed with me, and I’ve always tried to keep an extra eye on how I’m backing up my data.

For the last year I’ve been doing a full nightly backup of the Reinvented servers to Strongspace. They offer a network-based backup solution that’s cheap and simply to use; I posted about my positive experiences in dealing with them back in August.

The problem with using Strongspace for a nightly backup was that it was hard to do a simple incremental backup (i.e. only sending over what changed in the previous 24 hours) because I was sending the data as encrypted, compressed tar archives (encrypted because no matter how secure they claim to be, I didn’t want to leave my data, in the clear, on somebody else’s servers; compressed because I wanted to limited bandwidth usage). While Strongspace supports the use of tools like rsync, that really wouldn’t work in my situation, so I was ending up sending everything, every night. About 2GB worth.

Eventually my friends over at ISN noticed that I had suddenly started using an extra 60GB of bandwidth every month, and politely suggested that I either cut down or pay more.

I opted for the “cut down” approach, and resolved to find a new backup solution.

What I settled on was a Linksys NSLU2 and a 300GB Maxtor OneTouch USB/Firewire hard drive.

The NSLU2 is a neat little Linux-based device with the sole purpose of allowing USB external hard drives to be connected to an Ethernet LAN. While I could have plugged the new 300GB drive directly into my iMac, I wanted it tucked away in the server room, and I wanted it to be available to all the workstations and servers on our LAN here without depending on my iMac to be up and running.

I started off trying to use the NSLU2 in its stock condition, using the web interface to set up SAMBA shares that could be accessed from my Linux and OS X machines. While this worked fairly well, and was rock-solid on the iMac-NSLU2 side, I ran into reliability problems with using smbmount on the Linux servers to mount the NSLU2 shares, so I decided to upgrade the unit’s firmware using the open source Unslung replacement: this was a fairly painless process, similar to installing OpenWRT on my Linksys router. In about 30 minutes I was ready to go with SSH into the NSLU2.

In terms of the actual backing up, I settled on a simple Perl script from Norway called rsyncbackup that lets me use rsync over SSH to sync selected files and folders from my Linux and OS X machines to a copy on the Maxtor drive. After an initial sync that took many hours to complete for each machine — there was a lot of data to transfer over! — the daily sync now happens in only a couple of minutes because rsyncbackup is only moving the changed files, not everything.

I’m not going to give up on Strongspace entirely — it’s still nice to have an off-site backup in case the whole server room goes up in flames. But I’ll only run the data up there every week or so, which will make ISN happy.

The only hitch I’ve run into so far with the system is that my old IBM PC 300GL, which is the “old webserver” that’s still in operation as a DNS server, backup webserver, and general purpose workhorse, started to crash every time the backup routine started. Crash like “needed to be hard rebooted.” I eventually traced the problem back to the Promise-brand 20268 IDE hard drive controller that, it seems from what I’ve found in Google, has problems in DMA mode in certain situations with certain Linux kernels. Rather than trying to hack my kernel, I settled for simply turning DMA off; the drive got a lot slower (about 10x), but it’s stable now, and I don’t really need the speed given what the server is tasked for.

Otherwise, backups are purring along silently every night, and I can sleep a little easier know that when a disk fails, I’ll not have lost client web pages, photos of Oliver, or pithy posts in this space.