We have a network monitoring system here at Reinvented. It’s not exotic, but it does allow us to keep track of services running on both our own machines, and on our clients’ machines in several sites (for those of you that are curious, we use the open-source Nagios package).
The first step in checking to see whether a remote server is “alive” is to send it something called a “ping” which is the electronic equivalent of sending a “hello, are you there?” message over the wire and listening for the “yes, I’m here” message back.
If our monitor can send a ping and receive back a response, it knows that the remote server is online, and then proceeds to send it more complex messages like “is your webserver running?” and “is your disk full?”
Without a successful response to the initial ping, however, our monitor assumes that the remote server is dead in the water, and it proceeds to do all sort of crazy emergency-like behaviour like emailing us and paging us and generally waving its arms in the air to grab our attention to the problem.
It’s a system that’s worked well for several years, and we rely on it to offer our clients good service.
Until today.
Today, without telling anyone, Aliant, our upstream bandwidth provider, decided to turn off the ability for us to generate outgoing or incoming pings. In essense, they are filtering out all “ping traffic.”
So when our network monitor tries to send a ping to anywhere on the Internet, the entire network appears to be “dead in the water.” And so the system starts emailing us and paging us and generally waving its arms in the air to grab our attention to the problem.
I called Aliant’s technical “support” line this afternoon, and was told that this move was taken because of the various Microsoft-related viruses and worms that were released last week — apparently the increase in network traffic caused by the viruses and worms prompted them to filter network traffic to try and deal with the problem.
Fair enough.
But they neither told me, their customer relying on this service that they were going to do this, nor can they offer any estimated timing on the removal of this filtering beyond “when the virus problem has cleared up.”
I’ve asked them to remove the ping filtering from my subnet, but they claim to not be able to do this.
And they seem perplexed that anyone would actually rely on the ability to ping as a business tool.
I’m so goddammed angry at these idiots at Aliant that I want to scream. Fortunately I don’t need to scream: I’m switching bandwidth providers this week, part of a gradual and determined de-Aliant-ification of my life.
Another month or two, once I’ve switched cell phone and land-line providers, I’ll be totally free of Aliant’s unique approach to customer dis-service, and able to conduct business without worrying about crap like this.
Sorry about the strong words, but from PEINet to Island Tel to Aliant, I’ve spent hundreds if not thousands of hours banging my head against faceless technologists who have neither the skill to execute their duties, nor the compassion to admit this. I count myself extremely lucky to have an alternative bandwidth provider to fall back on, one where a real person answers the phone, and where I can go camp on the owner’s doorstep until problems are solved.
About This Blog
I am Peter Rukavina and this is my blog. I am a writer, letterpress printer, and a curious person.
To learn more about me, read my /now, look at my bio, listen to audio I’ve posted, read presentations and speeches I’ve written, or get in touch (peter@rukavina.net is the quickest way).
You can subscribe to an RSS feed of posts, an RSS feed of comments, or a podcast RSS feed that just contains audio posts. You can also receive a daily digests of posts by email.
Comments
I think I’ve mentioned this
I think I’ve mentioned this book before on Reinvented, but you are such a perfect example of this it’s worth repeating. Albert O. Hirschman is an economist and he was travelling in Nigeria in the late 60’s. He was travelling by train and he found it odd that there were trucks that travelled the same 800 mile trip that he did (the train track and railroad ran parallel). He found it odd that people would move goods by truck when the train would presumably be cheaper. He asked around and found out that the train people had caused this situation - if you sent goods by train you had no idea when the train would leave, when it would arrive or if the goods would get there in one piece. The truckers could guarantee all of this even if they cost more. Hirschman developed a theory from this which is in his book Exit, Voice and Loyalty. You maintain your loyalty to an organization when your voice is heard and if you exit the organization the place where you go next will have to mess up really, really badly for you ever to consider switching back. There is a phone company here in the Southwest which has a simple business plan - people will come to us who can’t stand dealing with Southwest Bell anymore. They have no technicians, no service people, they may not even have anyone who could fix a telephone or install a phone jack in a house, but they have a steady stream of customers simply by being the anti-SBC. Oh, Hirschman points out one other things - the people who care most about the organization are also the ones most likely to exit when their voice is not heard but it is precisely those people who could make the organization better. The parents who pull their kids out of public school because they don’t like what they see are exactly the parents whose input could make the schools better.
Peter, I think I know why
Peter, I think I know why they filtered pings. You will get a kick out of this.
I called tech support after my firewall emailed me 67 times to report ping flooding on the Aliant network. All the flood attempts were from Aliant IP addresses. When I spoke with the “technician” they had no idea what a ping flood was. I explained what a ping was, what a ping flood was, what a denial of service attack was. and alerted them to why it could be a problem.
The lady on the other end put me on hold for about 15-20 minutes and then came back to inform me she had spoken with a senior tech and would need to have some of the offending IP addresses. I gave her a lists of about 12 and she assured me that something would be done to correct the problem.
I am afraid that I may be part of the cause of your woes. Nice solution they came up!
Has Aliant mentioned which
Has Aliant mentioned which virus(es) they are attempting to gaurd against?
Ah the good old Aliant
Ah the good old Aliant service.
We recently upgraded our hosting package to upgrade our site to MySQL and PHP technologies.
In addition to paying 5 to 6 times the price of the reliable hosting companies in the US, I also signed up to the “where is waldo” Help desk feature.
Anyhow, to make a long story short after uploading all the files to the new server I noticed that my forms we’re not working… Ah, a minor problem with my CGI script. Tried to fix it didn’t work (was told it was my code), hum… work on my hosting plan in the states. Second attempt, well I’m going to figure out how to process my forms using PHP. One quick tutorial and an hour later bingo the form works and I can send the data to any email address EXCEPT for the emails ending with my own domain name LOL. (that’s very nifty)
Well 14 days later and about 5hours of waiting for tech support help I was told that is was indeed their error and something was wrong with the configuration of their server.
Aliant’s solution, use hotmail to send all your form mail to your aliant email address. now this is what I get when I get a request from a client on the net
____________________________________________
From: Mail Delivery Subsystem <mailer-daemon>
To: XXXXXXXX
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)
The original message was received at Tue, 26 Aug 2003 15:56:35 -0300
from XXXXXX@localhost
–— The following addresses had permanent fatal errors –—
andre@XXXXX.com
(reason: 550 5.1.1 User unknown)
–— Transcript of session follows –—
550 5.1.1 andre@XXXXXXX.com… User unknown
Reporting-MTA: dns; tianb.com
Arrival-Date: Tue, 26 Aug 2003 15:56:35 -0300
Final-Recipient: RFC822; andre@XXXXXXXX.com
Action: failed
Status: 5.1.1
Diagnostic-Code: X-Unix; 550 5.1.1 User unknown
Last-Attempt-Date: Tue, 26 Aug 2003 15:56:35 -0300
Return-Path: <xxxxxxxxx>
Received: (from XXXXXXXX@localhost)
by tianb.com (8.12.9/8.11.2) id h7QIuZAO029143;
Tue, 26 Aug 2003 15:56:35 -0300
Date: Tue, 26 Aug 2003 15:56:35 -0300
Message-Id: <200308261856.h7QIuZAO029143@XXXXXXXXXX.com>
To: andre@XXXXXX.com
Subject: Form Results
From: andre <andreleger@hotmail.com>
Content-Type: text/plain; x-avg-checked=avg-ok-88F31B9; charset=us-ascii
Content-Transfer-Encoding: 8bit
andre filled out the form and
selected a favorite color of Green
________________________
Great work Aliant. What would you do without Hotmail!
Now I’m going to send a box of bananas to their developers
Add new comment