Jackhammered

I was out for most of the morning today – a quick coffee with a friend I hadn’t seen for a while turned into a 2-hour long coffee-and-lunch – and when I returned to the office I noticed that my email was offline and there was a Tweet from Timothy Cullen alerting me that this website was offline:

Sure enough, not only was the site down, but our phone service and email (hosted on the same server) were offline as well.

I placed a quick call, on my cell phone, to Keith over at silverorange where our server lives, and asked him to reboot the server, as I couldn’t reach it from here over the network. He mentioned they’d had a brief power outage earlier that had taken everything down – a rare occurrence for silverorange, which prides itself on crazy-redundancy – and that was likely why things were awry.

Unfortunately a reboot didn’t do it, so I got my coat on and hiked up to Fitzroy Street to look at things from the inside.

Along the way I stopped in at Computer Dynamics, the folks who built the box that was offline, to pick up a replacement fan, as Keith had let me know a few weeks ago that the fan was shot. This gave me a chance to finally meet Marty MacLeod, who I’ve been buying computers and accessories from for more than a decade, whose shop is 3 minutes walk from my office, but whom, until today, I’d never met face-to-face.

Once I arrived at silverorange I set about replacing the fan, and then once everything was plugged back in found that I was still offline. Scratching our heads together, Keith and I tried various things until I remembered that the OpenWRT router I use as a gateway for the server, which has two outside-facing IP addresses, sometimes – not for many years now, but sometimes – gets itself into a state where I need to switch the primary and secondary IP addresses to get everything upstream to work. As soon as I did that, blamo, everything started to flow and I had web, email and phone back.

The cause of the brief power outage? There’s construction happening in the basement at silverorange, and somehow the vibrations of a jackhammer were enough to cause a circuit breaker to blow.

In the end analysis, then, a jackhammer in a basement in Charlottetown causes Timothy Cullen to tweet. We live in a strange world.