Network Outage 4/29/2006

[dvanburen]

I am sure you are all aware of this, based on the 100's of calls that we received. Here is the official explanation promised.

From Dean's post in another thread:

I'll try to explain what happened. An access list on the router was being updated to reflect new IP assignments. When the new list was being applied, a newline character got munged. On Ciscos, this will cause the remainder of the list to be ignored. When that happened, the access list was incomplete and started blocking all traffic. Since all traffic includes our own connection to the router, we rebooted it and brought it back up on its saved config.

Upon reboot, the router re-established its BGP connections to our upstream providers. The second issue that occurred was that we could get out of our router, but no traffic was coming back in. We were eventually able to track it down to a BGP filter list that was not working. We removed the filter list and reset our BGP sessions. The reason why the filter list never affected us in the past is because it is only applied when the BGP session is completely severed and brought back up, like when you reboot the router.

I appologize for this inconvenience.

[PinkyBrain]

I appeciate the explaination Dean. Any plan for an offsite status update? thanks.

[Dean]

We have an offsite monitor, but its one of those things we only go to when there is a problem. We found today that the offsite monitor was not available for some reason. I'll find out why as soon as possible.

[ideatek]

May I suggest a more timely update to the network status option on your phone menu, may have reduced the 100's of calls going into the support queue because the network status said "all servers are up" at least the couple times I checked.
Thanks for the quick repair and honest posting.
Daniel

[soky]

Funny... I didn't call, couldn't write and just tried all my domains for a few minutes and figured it was something. I wasn't worried a bit. Even though it's Saturday, I had every confidence that all that could be done was being done.

So... I went home and took a nap. Came back and here you were with an explanation and everything.

Keep up the hard work folks, you're doing a great job!!!

[generic]

maybe a maintenance post or email like you have done in the past would be a good idea before you do any work on the main router. Thought you used to send out things like that.

looks like webhostingtalk.com will be our offline forum for now, not really a good place to do that though because lots of people will begin seeing..."is vortech down again?" posts. one of us could easily put a simple forum on a free host account somewhere.

[admin]

As far as I am aware this was unplanned.. But thats as much as I know at this time. If it was planned we would of had a post up here I am sure dean would not have for gotten..

[hulahost.com]

This is really horrible, can I ask why this was going on in the middle of the day? Shouldn't something that could be so harmful be done at say 2am!??! I don't know about other people here, but I am seriosusly considering moving eveything to another provider. I spend a lot of money for advetisements on a daily basis, which are now lost, not to mention keeping my users in the dark for over 2 hours make us look cheap, and unprofessional. Just one incident like this can ruin your reputation.

I think most people who've been with you guys for a few years know this is extraordinary and that on the whole, the service and availability provided are far above average.

If we can get this forum offsite and get the Kayako support (www.vortechsupport.com) offsite, I think this will cut loads of headache.

[admin]

Originally Posted by hulahost.com

This is really horrible, can I ask why this was going on in the middle of the day? Shouldn't something that could be so harmful be done at say 2am!??! I don't know about other people here, but I am seriosusly considering moving eveything to another provider. I spend a lot of money for advetisements on a daily basis, which are now lost, not to mention keeping my users in the dark for over 2 hours make us look cheap, and unprofessional. Just one incident like this can ruin your reputation.

Did you read my post? Not planned..

[TicoGrande]

The service here is really very, very good. After a few years of doing business with Vortech, I am almost always a very happy camper... but something I STILL do not understand is... why are these Forums still in the Vortech network?

Though it does not happen often, when a major shut-down occurs effecting everything, even the DNS, there is just no way to (easily) get an status report.

I called yesterday, but no answer, not even an answering device. The #4 choice on the voice mail system cheerfully told me that the network was up and running without problems. LOL.

Trying to login to these forums failed of course as they are on the Vortech network.

Therefore... nothing! The customer calls begin and we are unable to explain anything other than we are working on the problem. Fortunately, yesterday was a Saturday so we got only one call Still... that is not the point.

Can I please suggest you move these Forums to another geographic location entirely outside the Vortech network? Then, if there IS a real crisis, you can easily communicate with hundreds of us resellers at once. I just do not see why this cannot be done! It would alleviate the number of TT's you would need to respond to, lessen the number of phone calls to Support (In fact I always use it before opening a network problem related TT.) I just see NO downside whatever for Vortech!

There used to be a separate Forum as I recall, and I do not know if it still exists nor did I bookmark it (my error) as this forum almost always is sufficient to get updates as these Forums are only down during a REAL crisis... but it is exactly those real crisis' that require the most communication!

In any case... thanks for getting it fixed...

And regarding planned/unplanned work or maintenance....so what if this was not announced.

Do all resellers contact all their customers every time Vortech wants to do some change or upgrade? I sure don't as about 99% of the time, this work goes smoothly and is totally transparent to the client. Why get someone's undies in a bunch over something that has not happened yet?

Abour the only benefit is we have something to say if something does go wrong, but even then, it is NEVER enough as the customer wants the details like time frame, etc.

So again...

We do need to have a solid, reliable way to keep the resellers updated, and to me, these Forums is the easiest and most efficient tool to do so. The only way it becomes a reliable tool is if it is moved off-network.

Just my thoughts.

TG

PS... If the bandwidth/MB's used for these forums is not too out of line, I'd volunteer to host them at one of my other locations for nada!

[admin]

We can ether move this forum off network or we can setup a new off site forum, what do you guys think is better? The problem with moving this forum forum.vortechhosting.com as the name servers for all vortechhosting.com is here. So it would be kind of point less to move this one.

I purchased Kayako and a support domain (like you guys vortechsupport.com) for my hosting reselling clients. The idea was to have my own support domain and email isolated so that should something major happen to my service or other accounts, I'd still have this one. But the fundamental flaw still was that this was all in one (albiet reliable) space.

Yesterday's experience, and the wrath of the few but valued clients I have, has caused me to (urgh, I hated to do it) get another host JUST for my support site/email/Kayako. After some hours of migration (pain pain), I now have a completely off-Vortech domain/email/support site for my clients.

Should another complete outage like this occur, my clients will still be able to contact me and feel the love.

So, yes, I submit 1 vote for the offsite location of the Kayako support, Forum, and a support email.

The chances of Vortech and whoever they would choose for offsite, going down simultaneously should be nill and even though there might be some SCHEDULED downtime during migration, the end result would be worth it in my opinion.

[admin]

I don't think having our main ticket system and forum off network is a good idea I think maybe a new domain or one of the domains we already own using out side DNS and Networks would be good.

I will have this started tomorrow with dean/aaron.

[impsnake]

Although I don`t know it`s a good suggestion or not,you can use DNS fail over service such as dnsmadeeasy.com for this matter,