We have a network monitoring system here at Reinvented. It’s not exotic, but it does allow us to keep track of services running on both our own machines, and on our clients’ machines in several sites (for those of you that are curious, we use the open-source Nagios package).
The first step in checking to see whether a remote server is “alive” is to send it something called a “ping” which is the electronic equivalent of sending a “hello, are you there?” message over the wire and listening for the “yes, I’m here” message back.
If our monitor can send a ping and receive back a response, it knows that the remote server is online, and then proceeds to send it more complex messages like “is your webserver running?” and “is your disk full?”
Without a successful response to the initial ping, however, our monitor assumes that the remote server is dead in the water, and it proceeds to do all sort of crazy emergency-like behaviour like emailing us and paging us and generally waving its arms in the air to grab our attention to the problem.
It’s a system that’s worked well for several years, and we rely on it to offer our clients good service.
Today, without telling anyone, Aliant, our upstream bandwidth provider, decided to turn off the ability for us to generate outgoing or incoming pings. In essense, they are filtering out all “ping traffic.”
So when our network monitor tries to send a ping to anywhere on the Internet, the entire network appears to be “dead in the water.” And so the system starts emailing us and paging us and generally waving its arms in the air to grab our attention to the problem.
I called Aliant’s technical “support” line this afternoon, and was told that this move was taken because of the various Microsoft-related viruses and worms that were released last week — apparently the increase in network traffic caused by the viruses and worms prompted them to filter network traffic to try and deal with the problem.
But they neither told me, their customer relying on this service that they were going to do this, nor can they offer any estimated timing on the removal of this filtering beyond “when the virus problem has cleared up.”
I’ve asked them to remove the ping filtering from my subnet, but they claim to not be able to do this.
And they seem perplexed that anyone would actually rely on the ability to ping as a business tool.
I’m so goddammed angry at these idiots at Aliant that I want to scream. Fortunately I don’t need to scream: I’m switching bandwidth providers this week, part of a gradual and determined de-Aliant-ification of my life.
Another month or two, once I’ve switched cell phone and land-line providers, I’ll be totally free of Aliant’s unique approach to customer dis-service, and able to conduct business without worrying about crap like this.
Sorry about the strong words, but from PEINet to Island Tel to Aliant, I’ve spent hundreds if not thousands of hours banging my head against faceless technologists who have neither the skill to execute their duties, nor the compassion to admit this. I count myself extremely lucky to have an alternative bandwidth provider to fall back on, one where a real person answers the phone, and where I can go camp on the owner’s doorstep until problems are solved.