The Roy Johnstone Nightmare

This weekend we started to get alerts from our server monitoring system that our mail server here at Reinvented HQ was timing out. On Sunday I started to look into the issue, and it turned out to be a simple matter of thousands of spam messages suddenly showing up. Looking deeper, it turned out that almost every one of the messages — there were a couple of hundred thousand by the time it was all over — was for an address in the RoyJohnstone.com domain.

Roy is a client, and we host a single email address for him. The spam, however, wasn’t for Roy, bur rather for Alex, Benny, Clarisa, David, Edgar, Felix, and thousands upon thousands of other names — there was obviously a dictionary somewhere that was just churning the stuff out.

It would have been a simple matter of dropping all packets from the offending host. Except that the flood was coming from a seemingly infinite variety of hosts, from Taiwan to Poland.

None of the spam was actually getting anywhere — we have the server set up to automatically drop any email that comes in to an invalid address — but all those connections were taxing our poor old SMTP server to its limits.

In the end I solved the problem by pointing Roy’s email at a new Gmail account through a Google Apps setup for the domain; almost as soon as I changed the MX records in our DNS the spam stopped. Since the switch to GMail, one spam email has made its way to Roy and 75 have been dumped into the Gmail spam folder.

This is a perplexing kind of problem to deal with when you’re running a small mail server like we are: it’s one thing to filter out the spam (and Spamassassin and Apple’s Mail.app do a decent job of that), it’s another thing entirely to deal with the simple torrent of spam flooding into the server. I’ve tuned up Sendmail a little more since the flood stopped, adding some of its own spam-fighting tools like connection rate and bad recipient throttling. But these are all imperfect solutions.

I’m starting to wonder whether, in the spam-drenched world, running ones own mail server will be tenable any longer. If anyone has additional recommendations for fighting this sort of thing, I welcome them.